[NVBug 5969206][fix] Break circular dependency in disagg KV transfer timeout cleanup - #12313
Closed
nv-yna wants to merge 1 commit into
Closed
[NVBug 5969206][fix] Break circular dependency in disagg KV transfer timeout cleanup#12313nv-yna wants to merge 1 commit into
nv-yna wants to merge 1 commit into
Conversation
…timeout cleanup In disaggregated serving, cancelled requests stuck in state 21 (kDISAGG_CONTEXT_TRANS_IN_PROGRESS) leak KV cache blocks permanently. The kv_transfer_timeout_ms mechanism has a two-phase design: - Phase 1 (_check_kv_transfer_timeout): flags timed-out requests every iteration - Phase 2 (_check_disagg_ctx_cache_transfer_status): cleans up flagged requests Phase 2 only ran inside _send_kv_async, which requires new context requests to be scheduled. Once the KV pool is exhausted by leaked blocks, no new requests can be scheduled, so Phase 2 never runs — creating an unrecoverable circular dependency. Move the Phase 2 call to the main executor loop so it runs every iteration regardless of whether new requests are being scheduled. This allows timed-out requests to be cleaned up even when the pool is fully exhausted. Reproducer: 1P1D disagg with Qwen/Qwen3-0.6B on L40 PCIe, 4+ cancel cycles with concurrency=300. The prefill worker enters a permanent timeout loop with "Timed out waiting for context KV cache transfer after 1000 milliseconds" and GPU utilization drops to 0%. Also adds diagnostic logging to the cancel request path for debuggability. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
This was referenced Mar 18, 2026
yifjiang
added a commit
to yifjiang/TensorRT-LLM
that referenced
this pull request
Apr 15, 2026
… side checkContextTransferStatus retries stuck prefill-side KV cache transfers indefinitely using only the per-iteration kv_transfer_sender_future_timeout_ms. The per-request total timeout kv_transfer_timeout_ms is plumbed through config (CacheTransceiverConfig::getKvTransferTimeoutMs) but never read in batch_manager code — it is dead code. Under concurrent load with constrained cache, stuck transfers hold KV blocks forever, exhausting the pool. The prefill worker becomes permanently unresponsive while health probes continue returning 200 OK. Fix: After each per-iteration timeout in checkContextTransferStatus, check total elapsed time (via LlmRequest::getKvCacheTransferStart, already set by sendAsync) against kv_transfer_timeout_ms. When exceeded, mark the request as DISAGG_TRANS_ERROR, best-effort cancel via CacheSender, and remove from mSenderFutures so blocks can be freed. Reproducer: 1P1D disagg with Qwen3-0.6B, free_gpu_memory_fraction=0.2, NIXL over TCP, concurrency 16 with ISL 8000. Server hangs after ~2 minutes and never recovers. Related: NVIDIA#12249 (set default kv_transfer_timeout_ms=60s — config only) Related: NVIDIA#12313, NVIDIA#12314 (Python-level fixes — cannot fire due to race condition with C++ transfer completion removing requests from Python tracking before the 60s timeout elapses) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
yifjiang
added a commit
to yifjiang/TensorRT-LLM
that referenced
this pull request
Apr 15, 2026
… side checkContextTransferStatus retries stuck prefill-side KV cache transfers indefinitely using only the per-iteration kv_transfer_sender_future_timeout_ms. The per-request total timeout kv_transfer_timeout_ms is plumbed through config (CacheTransceiverConfig::getKvTransferTimeoutMs) but never read in batch_manager code — it is dead code. Under concurrent load with constrained cache, stuck transfers hold KV blocks forever, exhausting the pool. The prefill worker becomes permanently unresponsive while health probes continue returning 200 OK. Fix: After each per-iteration timeout in checkContextTransferStatus, check total elapsed time (via LlmRequest::getKvCacheTransferStart, already set by sendAsync) against kv_transfer_timeout_ms. When exceeded, mark the request as DISAGG_TRANS_ERROR, best-effort cancel via CacheSender, and remove from mSenderFutures so blocks can be freed. Reproducer: 1P1D disagg with Qwen3-0.6B, free_gpu_memory_fraction=0.2, NIXL over TCP, concurrency 16 with ISL 8000. Server hangs after ~2 minutes and never recovers. Related: NVIDIA#12249 (set default kv_transfer_timeout_ms=60s — config only) Related: NVIDIA#12313, NVIDIA#12314 (Python-level fixes — cannot fire due to race condition with C++ transfer completion removing requests from Python tracking before the 60s timeout elapses) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a permanent hang in disaggregated serving where cancelled requests leak KV cache blocks until the pool is exhausted and the prefill worker becomes unresponsive.
Root cause: The
kv_transfer_timeout_mscleanup mechanism (Phase 2) only ran inside_send_kv_async, which requires new context requests to be scheduled. Once leaked blocks exhaust the KV pool, no new requests can be scheduled, so Phase 2 never runs — an unrecoverable circular dependency.Fix: Call
_check_disagg_ctx_cache_transfer_status(0)every iteration in the main executor loop, breaking the circular dependency. Timed-out requests are cleaned up regardless of whether new requests are being scheduled.Reproduction
1P1D disaggregated serving with Qwen/Qwen3-0.6B on L40 PCIe (or any slow-interconnect setup):
Symptoms:
[WARNING] Timed out waiting for context KV cache transfer after 1000 millisecondsrepeating every 1shas 2048 active_requests, scheduled 0 context requests— pool exhaustedConfirmed on: L40 PCIe (localhost), H100 PCIe (cross-node), Dynamo 1.0.0 + TRT-LLM. Does NOT reproduce on H100 NVLink (transfers complete before cancel arrives).
Evidence
With the timeout workaround, Phase 1 correctly flags stuck requests (776 "Terminating by timeout"), and the engine recovers to 4 active_requests — but only when new requests trigger Phase 2. Without new requests (full pool exhaustion), recovery is impossible.
Test plan
🔗 Full analysis: NVBugs 5969206