[TRTLLM-12317][feat] Match V1 KV event hashes for V2 cache events - #13624
Merged
peihu-nv merged 1 commit intoApr 30, 2026
Merged
Conversation
peihu-nv
force-pushed
the
review/pr-13589-v1-hash-fix
branch
from
April 29, 2026 23:34
11407fd to
282568c
Compare
Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
peihu-nv
force-pushed
the
review/pr-13589-v1-hash-fix
branch
from
April 30, 2026 16:08
282568c to
c9d6d9a
Compare
peihu-nv
marked this pull request as ready for review
April 30, 2026 17:26
pcastonguay
approved these changes
Apr 30, 2026
lfr-0531
pushed a commit
that referenced
this pull request
May 7, 2026
…3624) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
lfr-0531
pushed a commit
that referenced
this pull request
May 14, 2026
…3624) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
zheyuf
added a commit
to zheyuf/TensorRT-LLM
that referenced
this pull request
May 30, 2026
…k hashing The disaggregated orchestrator computes KV-cache block hashes per request for prefix-/conversation-aware routing. hash_v1_block_key is a pure-Python per-token loop; for long prompts (~38k tokens) it dominates the single asyncio event loop's CPU (py-spy: ~89% of one saturated core) and, being GIL-bound, serializes concurrent requests, so TTFT degrades super-linearly with concurrency. Route block_key_hasher's common path (cache_salt_id is None) to the existing nanobind-exposed C++ BlockKeyHasher, which implements the identical algorithm, removing the per-token Python loop from the orchestrator hot path. Bit-exact with hash_v1_block_key (verified 550/550 incl. the chained per-block pattern), no new dependency. Falls back to the Python implementation only when a cache_salt_id is set (the C++ hash() binding takes no salt) -- which does not occur on the unsalted agent/chat completion path. Only the serve router's routing-time hashing is changed; the worker-side KV-cache event hashing that Dynamo consumes (kept V1-compatible by NVIDIA#13624) is untouched, so the V1 hash contract is preserved. Impact (DSv4-Pro 1P1D disagg, GB300, AgentPerf-V2): c128 TTFT p95 3.74s, within SLO60 (p95<5s); c256 p95 26.5s -> 9.5s; orchestrator event-loop core de-saturated. Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf
added a commit
to zheyuf/TensorRT-LLM
that referenced
this pull request
May 30, 2026
…k hashing The disaggregated orchestrator computes KV-cache block hashes per request for prefix-/conversation-aware routing. hash_v1_block_key is a pure-Python per-token loop; for long prompts (~39k tokens) it dominates the single asyncio event loop's CPU (py-spy: ~89% of one saturated core) and, being GIL-bound, serializes concurrent requests, so TTFT degrades super-linearly with concurrency. Route block_key_hasher's common path (cache_salt_id is None) to the existing nanobind-exposed C++ BlockKeyHasher, which implements the identical algorithm, removing the per-token Python loop from the orchestrator hot path. Bit-exact with hash_v1_block_key (verified 550/550 incl. the chained per-block pattern), no new dependency. Falls back to the Python implementation only when a cache_salt_id is set (the C++ hash() binding takes no salt) -- which does not occur on the unsalted agent/chat completion path. Only the serve router's routing-time hashing is changed; the worker-side KV-cache event hashing that Dynamo consumes (kept V1-compatible by NVIDIA#13624) is untouched, so the V1 hash contract is preserved. Impact (DSv4-Pro 1P1D disagg, DEP8, GB300, AgentPerf-V2): c128 TTFT p95 10.71s -> 3.13s (-71%), throughput +50%, req/s +63% at fixed concurrency; SLO60 ceiling (p95<=5s and SSE p25>=60 tok/s) ~c80 -> c160 (~5 -> 10 users/GPU). Post-fix the bottleneck shifts to prefill queueing (ctx_queue), not the orchestrator. Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 10, 2026
…IDIA#13624) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 11, 2026
…IDIA#13624) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 12, 2026
…IDIA#13624) Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
This PR is stacked on top of #13589 and tightens KV cache event hash compatibility for Dynamo to support DeepSeek V4. It makes V2 KV cache events use the V1
BlockKeyHashercontract by default throughkv_cache_event_hash_algo = auto, including parent chaining, LoRA task ID, and cache salt ID. This lets Dynamo consume V2 KV cache events without changing its hash expectations or adding a new config override.Dynamo currently consumes V1-style KV event block hashes. Without this compatibility path, enabling V2 KV cache events would require Dynamo-side hash changes or could produce mismatched routing/cache state. Matching V1 hashes for supported text-token blocks, including salted cache namespaces, lets #13589 land with lower integration risk. Native V2 hashes remain available as explicit opt-in via
v2_sha256orv2_sha256_64.Changes:
kv_cache_event_hash_algo = autoresolve tov1_block_keyfor V2 KV cache events.cache_salt_idthrough request handling, V2 cache creation, root blocks, and event hash generation.Test Coverage
7 passedtest_llm_kv_cache_events.pycases:4 passed4 passed, 2 skippedPR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.