[Bug]: Output includes tokens from other prompts when host kv cache enabled

[pathorn]

System Info

• CPU architecture (e.g., x86_64, aarch64): x86_64
• CPU/Host memory size (if known): 2T
• GPU properties: NVIDIA H100 and NVIDIA B200
• Docker image: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
  (Also tried 2aade46)
• NVIDIA driver version

H100 System:
| NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2 |

B200 System:
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |

• OS (Ubuntu 24.04, CentOS 8): Ubuntu 22.04 (H100) and 24.04 (B200)

Who can help?

@eopXD

This is a follow-up to issue #8274.
This issue was easier to reproduce prior to nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1, but it still happens after the fix.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. docker run -it --rm --shm-size=250g --gpus "device=4" -p 8000:8000 --entrypoint /bin/bash -v /data:/data nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
2. Use the following extra_llm.yaml

enable_chunked_prefill: true
cuda_graph_config:
    max_batch_size: 60
enable_attention_dp: false
kv_cache_config:
    free_gpu_memory_fraction: 0.85
    host_cache_size: 2000000000

3. trtllm-serve nvidia/Meta-Llama-3.1-8B-Instruct-FP8 --tp_size=1 --backend=pytorch --host=0.0.0.0 --port=8000 --max_seq_len=131072 --max_batch_size=60 --max_num_tokens=2048 --extra_llm_api_options extra_llm.yaml
4. Open two terminal windows or screen. In the first window, do this:
   pkill -f curl
for i in {0..10000}; do ( curl -Z -s http://localhost:8000/v1/chat/completions'?[0-5]' -H "Content-Type: application/json" -d '{"model": "none", "messages": [{"role":"system","content":"Be a helpful assistant #'$RANDOM'. Respond concisely in one sentence."}, {"role": "user","content": "The quick brown fox jumps"}],"reasoning_effort":"high","response_format":{"type":"text"},"temperature":0,"top_k":1,"max_tokens":100,"n":1}' | tr } '\n' | grep --color=never -o 'content":".*' & ); sleep 0.51; done
   This will spam a bunch of curl processes. Make sure to pkill -f curl between tests as some get stuck.
5. In the second terminal window, use the following python script and txt file. Tested with pip install --user httpx==0.28.1

client_bench.py
war_and_peace_excerpt.txt

Run python client_bench.py - it prints lots of httpx.ReadError errors. The exceptions will be ignored, and this is normal. It will create many cancelled requests, but this is fine and may help reproduce the error.
6. Keep both commands running for a while. It may take 5-10 minutes to reproduce the error. Since version 1.2.0rc1, the error happens far less frequently and usually only a few bad responses will show up. When done, make sure to pkill -f curl
7. Test again with host_cache_size: 0. There will be no bad responses, no matter how many times the test is run.

Expected behavior

The responses should all be a sentence, something like this:

content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"That's a well-known pangram, a sentence that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"That is a well-known pangram, a sentence that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The sentence \"The quick brown fox jumps\" is a well-known pangram, a phrase that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The sentence \"The quick brown fox jumps\" is a well-known pangram, a phrase that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]

There should be no repeated words or # or repeated whitespace characters.

actual behavior

The prompt in the above testcase is "Be a helpful assistant #'$RANDOM'. Respond concisely in one sentence. ... The quick brown fox jumps" - so seeing these words verbatim in the output is unexpected.

Most responses are good. Bad responses will include mostly or entirely words from the prompt + chat template like some combination of the following:

content":"#     assistant#   assistant# assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#             #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#        ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"assistant #quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661. Respond concisely in one sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#           #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#      ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox","reasoning_content":null,"reasoning":null,"tool_calls":[]

Note in particular the above: My script picks a random number "You are assistant #12345" and performs 5 requests per random number. If there are 6 responses that include the same assistant number, this is proof that prompts are being mixed up across multiple requests, which is a security issue.

I think in general it seems like when the issue happens, it includes a subset of prompt tokens, so running some string comparison against the prompt might be a way to automate detecting this bug.

additional notes

This has been very difficult to get a reproduction for, but it has been an issue for a long time, at least August, probably longer. The issue is also confirmed in 1.1.0x and so on.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[eopXD]

Thank you for the clear bug issue @pathorn.

Regarding your configuration of the KV cache manager, do you have both enable_block_reuse and enable_partial_reuse enabled? If partial reuse is enabled, may you try disabling it and see if the problem goes away? Thank you.

[pathorn]

Thank you, @eopXD
The default for enable_partial_reuse and enable_block_reuse is true, so I was using them.

I added enable_partial_reuse: false to kv_cache_config, and I can confirm at least in my reproduction case, that the issue no longer happens when using host_cache_size.

The issue being related to partial reuse seems consistent with observing bad tokens from the last partial block of the prompt.

[josephrocca]

I've also recently discovered that a related issue can still occur, despite the fix. It seems to be slightly different this time.

• In 1.2.0rc0 the issue was that the LLM was using context tokens from other prompts in the dataset when host_cache_size is enabled.
• But in 1.2.0rc1 the issue I'm getting seems to be that the LLM only sees the last few tokens of some prompts. E.g. if the prompt is 1000 tokens long, then it seems as though sometimes the LLM only sees the last 50 tokens. So the tokens that it generates aren't based on other prompts, but are instead based on an incomplete view of the prompt.

I'm not certain about this, but it definitely seems like that's what's happening - very noticably different issue to the one in 1.2.0rc0.

At low host_cache_size values like @pathorn is using, the issue goes away, but at higher values (like 64GB) the issue exists.

The issue seems to only occur under very high concurrency (such that each of the concurrent requests is receiving about 1 token per second).

Here's my extra_llm_api_options config:

stream_interval: 2
cuda_graph_config: null
enable_attention_dp: false
kv_cache_config:
  enable_partial_reuse: false
  enable_block_reuse: true
  dtype: fp8
  free_gpu_memory_fraction: 0.85
  host_cache_size: 68719476736

And here's my start command (running on 4xB200):

docker run -d --shm-size=8g --ulimit memlock=-1 --ulimit stack=67108864 --gpus '"device=0,1,2,3"' -p 4000:4000 -v /root/data:/root/data -w /root --log-opt max-size=10m nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1 bash -c "trtllm-serve /root/data/nvidia___DeepSeek-R1-0528-FP4 --port 4000 --host 0.0.0.0 --backend pytorch --max_seq_len 9216 --max_batch_size 2048 --max_num_tokens 32768 --tp_size 4 --trust_remote_code --extra_llm_api_options /root/trtllm-config.yml"

And here are the things I've tried, based on the above config (where ❌ means it did not fix it, and ✅ means it seems to have fixed it):

• ❌ enable_partial_reuse: true
• ❌ removed dtype: fp8
• ❌ enable_attention_dp: false
• ❌ enable_chunked_prefill: true
• ❌ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
• ❌ PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (this seems to cause a memory leak, eventually consuming all system RAM and crashing the server. unsure if this is related to this issue)
• ❌ PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:8192
• ✅ reduce host_cache_size from 64GB to 1GB (but such a small host_cache_size defeats the purpose for my use case. also, perhaps this just makes the issue proportionally rare to reproduce)
• ❌ add --ep_size 4
• ❌ add moe_config.backend: TRTLLM

@eopXD Unfortunately I haven't yet been able to create a minimal repro, but for now please feel free to let me know some tests to try, if you have any ideas.

[pathorn]

@josephrocca The default for enable_partial_reuse is already true.

Are you sure you tried setting enable_partial_reuse: false in the kv_cache_config? That seems to solve the issue I am seeing. Can you try again with the initial configuration and only add enable_partial_reuse: false

reduce host_cache_size from 64GB to 1GB (

Yeah I am also unable to reproduce the issue, with extremely small host cache sizes. I can reproduce it with 2GB, but then I typed too few 0's and did 200MB instead of 2GB and that does not reproduce the issue.

My guess is smaller amount of host kv cache just makes the problem significantly less likely to reproduce but not impossible, and we just haven't hit the exact conditions to observe it. Plus, such a small amount of host kv cache is not useful.

[josephrocca]

Are you sure you tried setting enable_partial_reuse: false in the kv_cache_config?

Yes.

My guess is smaller amount of host kv cache just makes the problem significantly less likely to reproduce but not impossible

Yep, this is my guess too. There does seem to be a trend of larger size = faster to hit the issue.

Note though that your issue seems slightly different to mine. In my case it's not about prompt "mixing" (which I did observe in 1.2.0rc0). Instead, my issue with 1.2.0rc1 seems to be that sometimes prompts are being truncated so that only the last few tokens are visible to the LLM when it's generating the response. I haven't yet observed any "wrong prompt/context retrieved" type outputs in 1.2.0rc1 - only "missing prompt/context".

[josephrocca]

@eopXD This change (by Codex) seems to fix the issue for me:

• Edited ~/TensorRT-LLM/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp (+4 -0)
    238      copyBlock(offloadBlock, block, pools, false, numTokensToCopy, mode, directory);
    239 +    // Ensure the primary compute stream does not consume the block before the host->device copy finishes.
    240 +    tr::CudaEvent onboardEvent;
    241 +    mOnboardManager.getStream().record(onboardEvent);
    242 +    mBufferManager.getStream().wait(onboardEvent);
    243  }

I cloned the repo at v1.2.0rc1, made that change, built it, and am now not able to replicate the bug, even after several runs of my client script (previously I hit the bug on every client run without fail).

Unsure if this is the correct or most performant way to fix the issue though. Still playing around with it - will update this thread if I learn anything new.

[eopXD]

@pathorn : The issue being related to partial reuse seems consistent with observing bad tokens from the last partial block of the prompt.

Yes, I am seeing partial reuse causing the problem and investigating on it now. Will keep you and @josephrocca in-touch on fixing this.

@josephrocca: In 1.2.0rc0 #8274 was that the LLM was using context tokens from other prompts in the dataset when host_cache_size is enabled.

Your observation is correct. The offloading mechanism was flawed, which I fixed while supporting for SWA in the KV cache manager.

@josephrocca: But in 1.2.0rc1 the issue I'm getting seems to be that the LLM only sees the last few tokens of some prompts. E.g. if the prompt is 1000 tokens long, then it seems as though sometimes the LLM only sees the last 50 tokens. So the tokens that it generates aren't based on other prompts, but are instead based on an incomplete view of the prompt.

I think this depends on whether you have set the correct setting of the KV cache manager depending on the model you are running on. In 1.2.0rc1, sliding window attention is supported in the KV cache manager. What you have described seems like the behavior of running sliding window attention. May you double check you are setting the KV cache manager configuration correctly?

For example, for model like GPT-OSS with hybrid attention layers (18 full+18 SWA), setting for the KV cache manager should have:

kv_cache_config:
  max_attention_window: [128, 4096]

For Llama4 which is of all full attention layers, setting for the KV cache manager should have (setting the singular window to something big):

kv_cache_config:
  max_attention_window: [32768]

[josephrocca]

@eopXD As shown in my repro code above I'm using DeepSeek R1 0528, which I believe doesn't use SWA?

[eopXD]

@josephrocca You are right, let me try reproduce your situation locally.

[josephrocca]

@eopXD FWIW, this was an alternative fix by Codex that seems to have fixed the issue:

• https://gist.github.com/josephrocca/28698c417c31c9fa13d0dc9cd21ea774

(The previous fix is included in that patch, but it's commented out. Codex gave this alternate solution when I asked about increasing performance/throughput.)

But again, I'm not experienced enough here to know whether this is a correct/performant solution to the issue. All I can say for sure is that after that fix I am no longer able to reproduce the bug.

[eopXD]

@josephrocca The fix in gist is very concise after doing some trace. Let me fast track it with you as author to fix this. We also have a fix on-going for partial reuse.