[https://nvbugs/6104831][fix] Disaggregated KV transfer: lifecycle, cancellation, and quiescence hardening by chienchunhung · Pull Request #13713 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-05-03T06:20:23Z

Summary by CodeRabbit

Bug Fixes
- Improved reliability of disaggregated KV cache transfers with enhanced cancellation and timeout handling across context and generation phases.
- Enhanced resource cleanup and lifecycle safety for asynchronous transfer operations to prevent memory leaks during errors or cancellations.
- Better error handling and state management during KV cache transfer failures and request termination.

Summary

For Disaggregated Serving under long-prompt bursts with client-side cancellations, the existing cleanup path could leave the deployment alive but permanently non-responsive (wedge): workers up, /health returns 200, but every post-burst request timed out.

This PR fixes the permanent wedge for the PyTorch backend. Root cause is a stack of cleanup/lifetime invariant gaps across C++ KV transfer, Python termination, NIXL cancel, and block reuse. Cancelled requests now stop scheduling immediately, but KV resources release only after C++ transfer is terminal — and deferred termination is retried exactly once when it becomes safe.

Operator default: mid-flight NIXL cancel + fail-closed-on-unquiesced is opt-in via TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1. Default unset → no behavioral change vs. pre-PR baseline for that surface. The orthogonal lifetime, idempotency, RAII, and rc13-block-reuse fixes ship always-on.
Two late additions during the upstream/main merge cycle close two further failure modes that became deterministic in CI after this PR's shared_ptr<LlmRequest> lifetime extension turned previously-loud per-rank divergences into silent hangs:

Commits bdfdf8be02 (gen side) and 53a0692aa4 (ctx side) close the sig update aarch64 batch manager libraries to main #9 cross-collective ABBA on the helix pp1dp2cp2 / pp2tp1cp2 parametrizations by removing three rank-asymmetric Python gates from disagg gen-side scheduling and two from ctx-side scheduling so every rank reaches the C++ gatherRequestIds collective together. See sig update aarch64 batch manager libraries to main #9 and the rank-asymmetric-gates row in the change table.
Commit 9a1a0329fb (Option C) caps the per-iteration wait_for slice in checkContextTransferStatus / checkGenTransferStatus at 50 ms so the polling thread always cycles back to drive MPI/UCX forward progress. Without the cap, when neither kv_transfer_sender_future_timeout_ms nor kv_transfer_timeout_ms is configured, effectiveSliceMs fell through to INT64_MAX — and wait_for(INT64_MAX) on a not-yet-ready future blocked the polling thread indefinitely on transports without their own progress thread (notably mpi_kvcache over UCX-over-shm). See the cpp-gtest-wedge row in the failure-signatures and change tables.

Co-worked with @yifjiang.

Failure signatures

Sig	Symptom	Root cause
#1	Sender-side `Broken promise`	cancel-after-ready erased the promise without fulfilling it
#4	Generation event loop blocked forever	`checkGenTransferStatus(atLeastNum=1)` called `future.get()` on an unready future
#5	Receiver-side `Broken promise`	queued cancel erased the receiver promise without fulfilling it
#6	Receiver buffer-pool wedge	`requestSync()` early-return leaked a recv-buffer slot
#7	First-request SIGSEGV / mutex wedge	lifetime + eval-order bugs after moving from raw pointer to `shared_ptr<LlmRequest>`
rc13	No recovery even after #1–#7 fixed	rc13's default-on block reuse: Python deferred termination during in-flight context transfer, then skipped cleanup because it assumed early termination had already succeeded
#9	Helix CI hang: gen ranks asymmetrically enter the C++ `gatherRequestIds` collective vs. proceed to ADP `tp_allgather` / PP step-boundary collective	rank-local Python gates over per-rank state (`fitting_disagg_gen_init_requests`, `_disagg_gen_kv_recv_started_ids`, `any(req.is_disagg_generation_transmission_in_progress)`) before the only call chain into the C++ cross-rank collective. Pre-PR raw-pointer lifetime crashed loudly on the same divergence; this PR's `shared_ptr<LlmRequest>` closes that UAF and makes the divergence silent, turning it into an ABBA deadlock
cpp-gtest-wedge	`test_asymmetric_executor[llama-4proc-mpi_kvcache-103]` deterministically hits gtest's 300 s internal timeout on every PR build	`effectiveSliceMs` in `checkContextTransferStatus` / `checkGenTransferStatus` fell through to `INT64_MAX` when neither `kv_transfer_sender_future_timeout_ms` nor `kv_transfer_timeout_ms` was configured (the cpp gtest's default `CacheTransceiverConfig` sets neither). `wait_for(INT64_MAX)` on a not-yet-ready future blocked the polling thread; transports without a dedicated progress thread (notably `mpi_kvcache` over UCX-over-shm) then never advanced the in-flight transfer → future never became ready → death spiral. Test passes in ~35 s with the 50 ms cap.

Correctness invariant

A request can be cancelled at the API level immediately,
but its KV resources cannot release/reuse until:
  1. C++ KV transfer is terminal, or transfer/buffer state is fail-closed;
  2. transfer-buffer ownership has been released or poisoned;
  3. block-reuse / prefix-cache references are accounted for;
  4. Python resource cleanup runs exactly once.

Python request state alone is insufficient: with block reuse, a cancelled request may still have KV blocks referenced by the reuse tree or pinned by an in-flight context transfer. Freeing too early → memory corruption. Forgetting to free → permanent wedge.

What this PR changes

#	Change	Default	Closes
1	C++ transceiver uses `std::shared_ptr<LlmRequest>` across async send/receive	always-on	sig #7 UAF (Python termination racing C++ workers)
2	Cancellation paths fulfil promises before dropping queued/ready work; recv-side per-request idempotency guards `std::promise` against double-fulfilment	always-on (idempotency); gated (cancel propagation)	sigs #1, #5 (`Broken promise`)
3	`checkGenTransferStatus(atLeastNum=1)` skips unready entries in non-blocking polling	always-on	sig #4 (gen loop blocked on unready future)
4	NIXL `TransferStatus::release()` → `nixlAgent::releaseXferReq()` on cancel	gated	wedged workers under terminal peer failure (NVBug 6104831 customer report)
5	Materialize `reqId` before `std::move(resp)` in `sendAndRemoveResponse` (C++ eval-order UAF after the `shared_ptr` migration in #1)	always-on	sig #7 variant
6	Python `py_request_id`-keyed idempotency on disagg gen init + KV recv setup	always-on	duplicate `addSequence()` / `request_and_receive_async()` calls
7	Fail-closed on unquiesced transfer: `BufferIndexHolder` RAII (always-on); `poison()` on cancel + tri-state ready-signal + Python `_fail_closed_for_unquiesced_disagg_transfer`	RAII always-on; poison + fail-closed gated	silent buffer-pool corruption under cancel races
8	Deferred cleanup after timeout cancel: do not free Python resources until C++ transfer status reports terminal	always-on (the deferred-cleanup pattern); gated (the `cancel_request()` call)	premature free while NIXL/C++ may still touch buffers
9	rc13 block-reuse-safe deferred termination: `RequestTransferMetadata{termination_requested, resources_freed}` + `EndTransferResult{completed, needs_termination}`; `_end_transfer_and_maybe_terminate()` retries termination if an earlier attempt was deferred	always-on	rc13 regression (dual-path lost cleanup obligation)
10	Remove rank-asymmetric Python gates in disagg gen-side scheduling (`_prepare_disagg_gen_init`, `_recv_disagg_gen_cache`, `_check_disagg_gen_transfer_status` — commit `bdfdf8be02`) AND ctx-side scheduling (`_executor_loop_pp` + `_executor_loop`'s `if num_fitting_reqs == 0 and not fitting_disagg_gen_init_requests` gates — commit `53a0692aa4`) so every rank reaches the corresponding C++ `gatherRequestIds` cross-rank collective together. C++ side handles empty `mSenderFutures` / `mRequesterFutures` cheaply (one empty allgather, no per-request loop work).	always-on	sig #9 (helix CI hang: cross-collective ABBA against `_can_queue::tp_allgather` under attention-DP / PP step-boundary collective under `gen_pp > 1`)
11	Cap `effectiveSliceMs` in `checkContextTransferStatus` / `checkGenTransferStatus` at `kMaxPollSliceMs = 50` ms (commit `9a1a0329fb` — Option C) so the polling thread always cycles back to drive MPI/UCX forward progress. The deadline-check observability above is unchanged — if `kvTransferTimeoutMs` is set, the deadline still fires within `kMaxPollSliceMs` of true expiry; cancellation semantics (gated by `TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL`) unchanged.	always-on	cpp-gtest-wedge (transport-progress starvation on `mpi_kvcache` under default `CacheTransceiverConfig`)

12. Opt-in flag: `TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL` (new)

Gates the {cancel propagation, poison signal, fail-closed} triple — sections 4, the cancel-propagation half of 2, and 7 — behind a single env var. Default unset → no behavioral change vs. pre-PR baseline for that triple.

Mode	`cancel_request`	C++ cancel chain	`has_poisoned_transfer_buffer`	Layer-5 fail-closed
Default (unset / `=0`)	no-op, returns `False`	dormant	returns `False`	never fires
`=1`	full path	active	reports `true` on poison	fires on poison

Tests

New / updated unit coverage:

sender cancel-after-ready promise fulfillment
receiver queued-cancel promise fulfillment
non-blocking checkGenTransferStatus(atLeastNum=1)
deferred termination surviving until async transfer completion
partial-reuse path still skipping termination when none requested
successful early free preventing duplicate termination
direct EndTransferResult.needs_termination coverage
Integration regression coverage:
TestDeepSeekV3Lite::test_auto_dtype_with_helix[fifo_v2-cudagraph:with_padding-pp1dp2cp2] and […-pp2tp1cp2]. Both wedged at the 300 s hang-detector on CI build #39529 (commit 5234311); both PASS under the CI-default configuration (default kv_transfer_timeout_ms=60000, no TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL) post-fix, with GSM8K accuracy 63.87 vs the 61.54 hypothesis-test threshold — i.e. the model actually serves the workload, not a no-op pass.
test_asymmetric_executor[llama-4proc-mpi_kvcache-103] (cpp gtest): deterministically wedged at gtest's 300 s internal timeout on every pre-Option-C PR build. Passes in ~35 s post-Option-C (item add git-lfs dependency for binaries (release) #11), without any CacheTransceiverConfig timeout being configured.

Notes / limitations

The direct UCX path has a separate throughput/backpressure boundary under high concurrency. This PR primarily fixes the NIXL customer path and the shared disagg cleanup invariants.
MLA send-path hardening included for parity; stress validation uses Qwen3-0.6B which primarily exercises the non-MLA formatter.
The local CuTe DSL import guard used during development is not part of this PR.
TestQwen3NextInstruct::test_auto_dtype[use_py_transceiver=False] (gen pp=2, NIXL → C++ transceiver, listed in l0_dgx_b200.yml with TIMEOUT (60)) matches the sig update aarch64 batch manager libraries to main #9 trigger condition exactly (multi-rank gen + downstream unconditional collective + C++ transceiver). The extended timeout most plausibly masks the same hang. Worth re-running with the TIMEOUT (60) annotation removed once this PR lands; tracked as a follow-up in the investigation report's 08-next-steps-and-pr-map.md item 12.
Cross-rank consensus enforcement (follow-up work, NOT in this PR) — a follow-up implementation of doc 12's Path B (explicit horizontal-consistency layer) is prepared and locally validated. It ports the V2 transceiver's _consensus_outcome pattern into V1's C++ checkContextTransferStatus / checkGenTransferStatus as a four-pass pipeline (readiness consensus → local classify+cache → 2 outcome allgathers → state transitions), gated behind a new env var TRTLLM_DISAGG_USE_CONSENSUS_OUTCOME (default OFF, byte-identical to this PR's default behaviour). Local validation across 11 runs on 4 test families, 4 topologies, 3 transports, 4 models — all PASSED, 3 of 4 helix-class tests caught real cross-rank divergence events the consensus mechanism correctly deferred, cache mechanism survived multi-iteration deferral. See investigation doc 14-cross-rank-consistency-enforcement.md for the full design + empirical results + updated landing plan. Decision on whether to fold this into this PR or land as a follow-up is gated on this PR's CI signal.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Adds 00-tldr.md as the front door to the investigation. Targets a 10-minute read with two inline Mermaid figures: - The disagg architecture + cancellation flow (where the bugs live). - The L1-L8 → combo PR NVIDIA#13713 fix mapping (which piece closes which layer). Written under the assumption that PR NVIDIA#13713 (combo: PR NVIDIA#13056 + PR NVIDIA#13495 + eval-order fix + Python idempotency guards) is the chosen final solution. Doesn't compare approaches A/B/C — that comparison lives in 06-fix-approaches/README.md for readers who want depth on the landing decision. Section structure: 1. What is broken (the wedge symptom). 2. Where the bugs live (architecture figure). 3. Root cause: eight invariant gaps (L1-L8 table). 4. The fix (combo figure + per-piece breakdown). 5. Does it work (empirical recovery table). 6. Why this took 8 days to find (cascade pattern). 7. What is left to do. 8. Where to go from here (reader paths to deeper files). README.md updated to put 00-tldr.md at the top of the navigation table and add a 10-minute reading path entry. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

…rt, and L9 invariant Update the NVBug 6104831 investigation report to reflect the current state of the combo fix path (PR NVIDIA#13713): - Fold PR NVIDIA#13728 (fail-closed on unquiesced disagg KV transfer) into the combo as a fifth piece, plus the MLA-formatter port that closes the same hazard for DeepSeek-style models. - Add L9 (transport quiescence on unsafe exit) to the defect-class stack as a defense-in-depth memory-safety invariant. L1-L8 are wedge-prevention; L9 is the rip-cord that prevents silent buffer-pool corruption on cancel/exception when transport quiescence is unknown. - Update the empirical results tables across README.md, 00-tldr.md, 06-fix-approaches/D-combo.md, and 06-fix-approaches/README.md to show CONC=128 (3-pair) review-fix-v3 5/5 PASS post-fold-in plus the prior pre-fold-in stress runs through CONC=256. - Add MLA-stress validation as a follow-up item and motivate the RAII-audit task with the MLA-formatter finding. - Update the PR map to include NVIDIA#13728 and the chained sig regression PRs that the combo retains as test scaffolding. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…r queues Follow-up to the prior commit. The worker queues inside CacheSender::Impl and CacheReceiver::Impl still held raw LlmRequest* pointers — meaning the executor's mSenderFutures / mRequesterFutures pinned the lifetime but the worker thread that actually dereferences the request had only a raw observer. If the executor erased its tracking entry while the worker was mid-flight, the LlmRequest could be freed under the worker. This commit closes that surface: * CacheSender::Impl::Response::mRequest → std::shared_ptr<LlmRequest>. * CacheReceiver::Impl::RequestAndPromise::mRequest → std::shared_ptr<LlmRequest>. Move/copy semantics simplified now that the field is a smart pointer. * CacheSender::sendAsync(LlmRequest&) → sendAsync(std::shared_ptr<LlmRequest>). * CacheReceiver::receiveAsync(LlmRequest&) → receiveAsync(std::shared_ptr<LlmRequest>). * CacheReceiver::Impl::requestAndReceiveAsyncMultiThreads similarly. * CacheReceiver::Impl::receiveAsync now captures the shared_ptr by value in the std::async lambda so the worker thread pins the LlmRequest independently of the caller. * CacheTransceiver::respondAndSendAsync/-LayerWise/requestAndReceive* pass the shared_ptr (no longer .get()-strip it) and move into the TrackedFuture entry where appropriate. Also fixes the eval-order UAF in CacheSender::Impl::handleAsyncSend that PR NVIDIA#13713 / PR NVIDIA#13728 called out: once Response::mRequest is a shared_ptr, the one-liner sendAndRemoveResponse(resp.mRequest->mRequestId, std::move(resp)); becomes undefined behaviour because C++ argument evaluation order is unspecified — the compiler may evaluate std::move(resp) first, leaving resp.mRequest empty when reading mRequestId. Materialise the id into a local before the move. The PyCacheTransceiver trampoline's bool cancelRequest override is updated to match the new shared_ptr signature. The doc at docs/source/features/disagg-kv-transfer-session-lifecycle.md spells out the full ownership chain: executor tracking + worker queue both hold shared_ptr, and TransferSession::mRequest is an ephemeral observer used only inside the worker frame where the shared_ptr is already held. Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>

…g, structured cancel, bounded quarantine The C++ cache transceiver and Python executor cooperate on disaggregated KV transfer cancellation, timeout, and process health. Earlier attempts (PR NVIDIA#13706, PR NVIDIA#13713, PR NVIDIA#13728) each fixed one slice of the problem but still hung in the field: * PR NVIDIA#13706 with Python changes restarted workers on every routine per-request timeout (Python fail-close was too aggressive). * PR NVIDIA#13713 / PR NVIDIA#13728 hung even with Python cleanup removed because the C++ status polling could still call future.get() on an unready worker future, freezing the executor event loop while NIXL/UCX was wedged. This PR keeps each layer's responsibility separate, as described in docs/source/features/disagg-kv-transfer-hang-restart-analysis.md and the new docs/source/features/disagg-kv-transfer-session-lifecycle.md: * checkContextTransferStatus / checkGenTransferStatus are now strictly non-blocking. They poll with wait_for(0ms) and only call future.get() when the future is already ready. atLeastRequestNum > 0 still admits additional ready entries but never selects an unready one to satisfy the count. drainContextTransferStatus / drainGenTransferStatus are the only blocking variants and are intended for shutdown drain. * cancelRequestStructured returns a TransferCancelResult enum with six outcomes (NotFound, AlreadyComplete, CancelledBeforeAdvertise, CancelRequestedInFlight, BackendUnhealthy, NotCancellable). Only the first three permit Python to free request resources; the others keep the request owned by C++ until the worker future reaches a final state. The historical bool cancelRequest is preserved as a backward-compatible wrapper. * The transceiver maintains a bounded quarantine counter and a global progress deadline. A per-request timeout marks the entry quarantined but keeps the future pinned so NIXL/UCX cannot write into freed memory. If quarantined entries exceed mQuarantineBudget or no worker has reached a final state for longer than mGlobalProgressDeadlineMs, isHealthy() flips false and getHealth() surfaces the snapshot for orchestration. * PyExecutor adds _can_terminate_request_now / _inflight_cancel_- requested_ids so _do_terminate_request defers freeing resources for any request that is still in disagg transfer state or whose cancel is in flight. Per-request timeouts no longer clear active_requests, the waiting queue, or set is_shutdown — that was the PR NVIDIA#13706 restart-loop trigger and is explicitly forbidden by the analysis doc. * The ADP synchronized pending-response flush from PR NVIDIA#13112 is ported here (the base commit precedes that merge). Transfer- completion responses created in _end_transfer_and_maybe_terminate are buffered and flushed at synchronised loop points so every DP rank participates in the tp_gather collective. Test coverage: * tests/unittest/disaggregated/test_kv_transfer_session_lifecycle.py — 14 unit tests for the deferred-terminate guard, the structured- cancel decision tree, and the ADP flush symmetry. They run without GPU or MPI so they fail fast in pre-merge CI. References: * analysis: docs/source/features/disagg-kv-transfer-hang-restart-analysis.md * contract: docs/source/features/disagg-kv-transfer-session-lifecycle.md Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>

…r queues Follow-up to the prior commit. The worker queues inside CacheSender::Impl and CacheReceiver::Impl still held raw LlmRequest* pointers — meaning the executor's mSenderFutures / mRequesterFutures pinned the lifetime but the worker thread that actually dereferences the request had only a raw observer. If the executor erased its tracking entry while the worker was mid-flight, the LlmRequest could be freed under the worker. This commit closes that surface: * CacheSender::Impl::Response::mRequest → std::shared_ptr<LlmRequest>. * CacheReceiver::Impl::RequestAndPromise::mRequest → std::shared_ptr<LlmRequest>. Move/copy semantics simplified now that the field is a smart pointer. * CacheSender::sendAsync(LlmRequest&) → sendAsync(std::shared_ptr<LlmRequest>). * CacheReceiver::receiveAsync(LlmRequest&) → receiveAsync(std::shared_ptr<LlmRequest>). * CacheReceiver::Impl::requestAndReceiveAsyncMultiThreads similarly. * CacheReceiver::Impl::receiveAsync now captures the shared_ptr by value in the std::async lambda so the worker thread pins the LlmRequest independently of the caller. * CacheTransceiver::respondAndSendAsync/-LayerWise/requestAndReceive* pass the shared_ptr (no longer .get()-strip it) and move into the TrackedFuture entry where appropriate. Also fixes the eval-order UAF in CacheSender::Impl::handleAsyncSend that PR NVIDIA#13713 / PR NVIDIA#13728 called out: once Response::mRequest is a shared_ptr, the one-liner sendAndRemoveResponse(resp.mRequest->mRequestId, std::move(resp)); becomes undefined behaviour because C++ argument evaluation order is unspecified — the compiler may evaluate std::move(resp) first, leaving resp.mRequest empty when reading mRequestId. Materialise the id into a local before the move. The PyCacheTransceiver trampoline's bool cancelRequest override is updated to match the new shared_ptr signature. The doc at docs/source/features/disagg-kv-transfer-session-lifecycle.md spells out the full ownership chain: executor tracking + worker queue both hold shared_ptr, and TransferSession::mRequest is an ephemeral observer used only inside the worker frame where the shared_ptr is already held. Signed-off-by: Yifan Jiang <19356972+yifjiang@users.noreply.github.com>

…sig NVIDIA#8, Phase 15 The combo (PR NVIDIA#13713 + NVIDIA#13728 fold + MLA port) regressed when applied to rc13: with rc13's default-on block reuse, `_handle_responses`'s early-termination branch and `_end_transfer_and_maybe_terminate` can each refuse termination under the right timing, leaving the request with no cleanup owner. Server hangs on scenarios that succeeded on rc11. Captures this in the investigation report: - 03-defect-class-stack.md: add L10 (redundant block-reuse cleanup mechanism on the disagg path) with code sites, customer-visible symptom, and the latent symptoms the Phase 1 stop-gap leaves open (pin leak on cancel/timeout, PP > 1 disagg without block reuse, eviction race in the unpin → release window, regression risk on adjacent code). - 03-defect-class-stack.md: extend the layer-to-signature mermaid with L10 → sig NVIDIA#8 plus a dotted edge to the latent-symptom set. - 02-failure-signatures.md: add sig NVIDIA#8 (rc13 server hang under disagg + block reuse + in-flight cancel) with full root-cause, short-term stop-gap, and medium-term Phase 2 plan. - 05-investigation-timeline.md: add Phase 15 documenting the rc13 regression discovery, the two competing fix proposals, and the recommended staged plan. - 08-next-steps-and-pr-map.md: split item 2 into "land combo with rc13 stop-gap" and a new item 2a "land Phase 2 of the block-reuse-overlap-scheduler design". - README.md: add the rc13 caveat callout in the status section; update navigation language from L1-L8/seven-sig to L1-L10/eight-sig. Adds incentive cross-reference in the existing design doc: - docs/design/block-reuse-overlap-scheduler/README.md: promote Phase 2 status from "deprioritised" to "load-bearing for stable disagg block reuse" with cross-link to the rc13 evidence. - docs/design/block-reuse-overlap-scheduler/phase2-unify-reuse-mechanisms.md: add an "Empirical confirmation: the rc13 regression" section at the top, framing Phase 2 as the architectural answer the rc13 bug predicted. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Adds 09-executive-summary-rc11-to-rc13.md as the 15-minute read for someone briefing the full journey: original rc11 wedge, how PR NVIDIA#13713 solved it, why it regressed on rc13 (block reuse → L10 dual-path), how the short-term stop-gap unblocks rc13, and why the design doc's Phase 2 is the right long-term answer. Five Mermaid figures: - L1-L9 layer stack as the rc11 root cause framework - PR NVIDIA#13713's four-piece composition with L1-L9 mapping - The L10 dual-path: both cleanup paths refusing termination on rc13 - Stop-gap coverage vs latent symptoms it leaves open - Add-coordination vs delete-redundancy comparison + staged plan Updates README.md navigation table and reading paths to point at the new file with a "15 minutes / brief someone on the full journey" entry. The file extends 00-tldr.md (10-minute read on rc11 only). Where 00-tldr stops at "the rc11 wedge is fixed", 09 picks up at the rc13 regression discovery and walks through to the architectural follow-up. Designed for an exec briefing or a teammate joining mid-investigation. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

…cancellation/cleanup invariants Two changes to 09-executive-summary-rc11-to-rc13.md per review feedback: 1. Treat the file as an independent report. Removed the "extends 00-tldr.md with the rc13 chapter" framing; removed the inline "> Detail: <other-file>.md" pointers from each section; restated the ten invariants in-place rather than referring readers to 03-defect-class-stack.md to follow along; moved the cross-file pointers to a clearly-marked "Optional deep-dive pointers" appendix at the end. A reader can now walk away with a complete understanding of the bug class, the fix, the regression, and the plans without needing to consult any other file. 2. Add a new section 2 "The invariants for correct request cancellation and cleanup" that names the ten invariants the bug class violates, organised into four architectural categories: - Lifetime invariants (L2/L7/L9): what stays alive while transfers are in flight. - Resource invariants (L5/L6): every acquired resource must release on every exit. - Synchronization invariants (L1/L3/L4): no thread waits forever. - Coordination invariants (L8/L10): exactly one owner, no implicit handoffs. Each invariant is named, defined, and explained in terms of why it matters (with the rc11/rc13 evidence that violated it). Section 3 onwards then describes PR NVIDIA#13713, the rc13 regression, the stop-gap, and the architectural fix in terms of which invariants they enforce or fail to enforce. A new Mermaid figure visualises the four invariant categories so a reader can see the architectural hierarchy at a glance: lifetime governs who is alive, resource governs what is held, synchronization governs who is waiting, coordination governs who is responsible. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

…p-dive pointers Per review feedback: - Title now just mentions the NVBug number: "09 — NVBug 6104831 Executive Summary". - Preface trimmed to one sentence that briefly mentions PR NVIDIA#13713 and the rc11/rc13 contrast. Audience block, reading-time block, and self-contained note removed. - "Optional deep-dive pointers" appendix removed entirely. The file is now strictly the rc11 → rc13 narrative with the four- category invariant model, no meta-framing or cross-reference appendix. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-05-07T06:30:48Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-05-07T06:38:52Z

PR_Github #47132 [ run ] triggered by Bot. Commit: 484655e Link to invocation

tensorrt-cicd · 2026-05-07T22:24:36Z

PR_Github #47132 [ run ] completed with state SUCCESS. Commit: 484655e
/LLM/main/L0_MergeRequest_PR pipeline #37097 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ve conflicts Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

… 50ms Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-05-27T23:35:08Z

/bot run --disable-fail-fast --stage-list "DGX_H100-4_GPUs-CPP-1"

tensorrt-cicd · 2026-05-27T23:40:44Z

PR_Github #50629 [ run ] triggered by Bot. Commit: 9a1a032 Link to invocation

…orcement (V2 _consensus_outcome port to V1 C++) Documents the implementation and empirical validation of doc 12 § 5.2 Path B (explicit horizontal consistency layer). The V1 C++ checkContextTransferStatus / checkGenTransferStatus now have a four-pass pipeline (readiness consensus → local classify+cache → 2 outcome allgathers → state transitions) gated behind a new TRTLLM_DISAGG_USE_CONSENSUS_OUTCOME env var (default OFF, so the PR NVIDIA#13713 default behaviour is byte-identical to current HEAD). Local validation: 11 valid runs across 4 test families (cpp gtest asymmetric_executor, helix DeepSeek-V3-Lite, helix Qwen3-8B, disagg TinyLlama ctxpp2_genpp2), 4 topologies, 3 transports (MPI, NIXL, UCX), 4 models — all PASSED. 3 of 4 helix-class tests caught real cross-rank divergence events the consensus mechanism correctly deferred. The cache mechanism (mSenderLocalOutcomes / mRequesterLocalOutcomes) survived multi-iteration deferral (TinyLlama: same request deferred across iters 2, 3, 4 — would have crashed with std::future_error on iter 3 without the cache). Overhead measured at ~22% wall time on a flake-prone helix test that catches 2 divergence events, ~0% on tests with no divergence to handle. Includes the updated landing plan (§ 5) that supersedes doc 12 § 6's Path A → Path B sequence: Path B was implemented in ~2 days (not the 1-2 weeks doc 12 estimated, because the V2 transceiver provided a working pattern to port) and works on every test family tried, so Path A (lifetime flag wrapper) is no longer needed as an interim. Also refines doc 13's framing of asymmetric_executor[mpi_kvcache] (§ 3.4): doc 13's narrow claim (no divergence visible at gather points) remains correct, but the broader inference (therefore not a consistency issue) is incomplete. The cpp gtest is fixed by consensus (348s+FAIL → 46s+PASS on the same test), so it IS partly a consistency issue at the outcome-resolution layer, even if the divergence isn't visible at the gather-readiness layer. Both a transport-progress issue (doc 13 / Option C) and a consistency-amplifier issue (this doc / Path B). Also updates the README file table to add entries for docs 11, 12, 13, and 14 (the table had stopped at 10). Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

tensorrt-cicd · 2026-05-28T06:02:35Z

PR_Github #50629 [ run ] completed with state SUCCESS. Commit: 9a1a032
/LLM/main/L0_MergeRequest_PR pipeline #40124 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-05-28T06:41:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-28T06:47:38Z

PR_Github #50714 [ run ] triggered by Bot. Commit: 9a1a032 Link to invocation

tensorrt-cicd · 2026-05-28T20:40:49Z

PR_Github #50714 [ run ] completed with state FAILURE. Commit: 9a1a032
/LLM/main/L0_MergeRequest_PR pipeline #40198 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-05-28T21:16:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-28T21:22:13Z

PR_Github #50890 [ run ] triggered by Bot. Commit: 9a1a032 Link to invocation

tensorrt-cicd · 2026-05-29T00:41:47Z

PR_Github #50890 [ run ] completed with state FAILURE. Commit: 9a1a032
/LLM/main/L0_MergeRequest_PR pipeline #40357 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ent inventory Add 13-pr-decomposition-plan/README.md documenting: 1. Component inventory: 18 functionally-distinct components in PR NVIDIA#13713 (A1-A10 always-on, A11 reverted, C1 standalone polling cap, G1-G8 gated) with code names, short mnemonics, scope, files, line estimates, risk. 2. Six-tier submission sequence with strict hard dependencies and parallel possibilities. Tier 1 (standalone bug fixes) -> Tier 2 (observability + Python idempotency) -> Tier 3 (state-transition invariant, isolated for attribution) -> CHECKPOINT -> Tier 4 (shared_ptr foundation, optional) / Tier 5 (horizontal-consistency, conditional) / Tier 6 (cancel surface, gated, last). 3. Dependency graph showing parallel authoring opportunities and strict precedence rules. 4. Checkpoint criteria after Tier 3 with a decision matrix for the four observed failure classes (decoder assertion, Class B RuntimeError, V2 setup race, Class A wedge). Motivation: PR NVIDIA#13713 has burned 8+ failed CI cycles over ~4 weeks with overlapping bug surfaces that prevent attribution. Decomposition gives each landing a tight blast radius and clean revertable scope. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

…gle-process test to prevent daemon thread leak Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-05-29T07:16:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-29T07:22:31Z

PR_Github #51001 [ run ] triggered by Bot. Commit: f362223 Link to invocation

tensorrt-cicd · 2026-05-29T17:30:01Z

PR_Github #51001 [ run ] completed with state SUCCESS. Commit: f362223
/LLM/main/L0_MergeRequest_PR pipeline #40450 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…dedup tracing Adds per-rank dedup-set tracing on the Python gen side and counterpart tracing on the C++ ctx/gen sides to identify which mechanism in PR NVIDIA#13713 causes gen-side ranks to disagree on whether a reqId is "already started," producing a 1-of-N sendRequestInfo wedge on the ctx side. Python (_torch/pyexecutor/py_executor.py): - DIAG-GEN-PREPARE-INPUT at _prepare_disagg_gen_init entry: rank, iter, fitting_py_ids, KV-manager prepared-id snapshot. - DIAG-GEN-RECV-INPUT at _recv_disagg_gen_cache entry: rank, iter, input_py_ids, full _disagg_gen_kv_recv_started_ids snapshot. - DIAG-GEN-DEDUP-DECISION for every request in the dedup filter: rank, py_id, state, already_in_set, action (SKIP/CALL_RECV). - DIAG-GEN-DEDUP-DISCARD-ROLLBACK-{SYNC,ASYNC} in the except blocks that roll back the dedup id when request_and_receive_{sync,async} throws. - DIAG-GEN-DEDUP-DISCARD-TERMINATE in _do_terminate_request when discarding the dedup id at request termination. Together these pin which of M1 (schedule-broadcast divergence), M2 (A3 per-rank dedup divergence), or M3 (state mismatch) caused the observed 1-of-2 send. C++ (batch_manager/dataTransceiver.cpp): - DIAG-CTX-COUNTERPARTS in recvRequestInfo: ctx side logs allCounterparts and the peerSelfIdx of the rank that just sent a request-info, so the silent counterpart is identifiable as the entry that never shows up. - DIAG-GEN-SEND-INFO-COUNTERPARTS once before the per-peer send loop and DIAG-GEN-SEND-INFO-DEST per loop iteration so we can confirm each gen rank's view of the dest set and which dest each notifySyncMessage targets. Pure logging additions; no behavior change. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-06-03T22:14:38Z

PR#14768 will replace this PR (PR#13713) as a safer subset of the changes with smaller blast radius: it only includes the changes required for resolving UAF memory corruption, segmentation fault, ..etc.

… cancel-poison design Adds a new investigation doc 19 capturing the external forensic A/B (fengyul/dynamo-disagg exp 4) that decomposes the decode-side wedge into three independent failures: F1 (Broken promise UAF), F2 (engine- loop freeze on unbounded future.get()), and F3 (eager-free poisons UCX progress thread → permanent wedge). Maps F1/F2/F3 onto the existing layer model (L1, L3, L4+L5) and signatures (#1, #4), and onto the design doc's C4 invariant. Strategic conclusions surfaced from the experiment: * PR NVIDIA#14979's shared_ptr lifetime port closes F1 (a co-occurring crash class) but does not recover the field wedge on NIXL by itself. * Bounded polling alone (F2 fix) does not progress a stuck transfer on NIXL because UCX runs its own background progress thread — engine-freeze and transfer-stall are separate concerns. * The load-bearing fix is F3 done safely = quiescence-gated freeing, which is structurally tangled across py_executor.py and the transfer manager API and not cleanly cherry-pickable. The deployable that passes the reproducer is PR NVIDIA#13713 in full. Updates to investigation README: * Adds doc 19 to the 'How to read' index. * Adds a 'Can we ship a small subset of NVIDIA#13713?' suggested reading path. Updates to disagg-inflight-cancel-poison design README: * Adds the exp 4 evidence to 'Empirical evidence motivating this work'. * Adds a 'load-bearing for recovery, not optional polish' note on Phase 2 (deferred un-poison) — the polling-until-quiescence mechanism realizes C4 on the V1 + C++ path and is direct empirical answer to the field wedge, not just operability improvement on top of the existing fail-closed surface. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

github-actions Bot assigned chienchunhung May 3, 2026

chienchunhung force-pushed the pr13056-pr13495-combo-nvbug6104831 branch 4 times, most recently from 631975c to 262796c Compare May 4, 2026 06:57

This was referenced May 4, 2026

Disagg request cancellation fix (PR #13713 rebased onto v1.3.0rc13) #13727

Draft

[codex] Fail closed on unquiesced disagg KV transfer #13728

Draft

Disagg cancellation + fail-closed (PR #13728 rebased onto v1.3.0rc13) #13741

Draft

chienchunhung changed the title ~~[https://nvbugs/6104831][fix] Disagg request cancellation fix~~ [https://nvbugs/6104831][fix] Disaggregated KV transfer: lifecycle, cancellation, and quiescence hardening May 5, 2026

yifjiang mentioned this pull request May 6, 2026

[https://nvbugs/6104831][fix] Disagg KV transfer: non-blocking polling, structured cancel, bounded quarantine #13796

Draft

5 tasks

chienchunhung force-pushed the pr13056-pr13495-combo-nvbug6104831 branch 2 times, most recently from 87c4bb8 to 484655e Compare May 7, 2026 06:22

chienchunhung force-pushed the pr13056-pr13495-combo-nvbug6104831 branch from 484655e to 5719008 Compare May 7, 2026 06:43

pcastonguay requested a review from Tabrizian May 7, 2026 21:27

yifjiang mentioned this pull request May 7, 2026

PR #13713 (8-commit consolidation) rebased onto v1.3.0rc13 yifjiang/TensorRT-LLM#2

Draft

chienchunhung mentioned this pull request May 7, 2026

[https://nvbugs/3140325][fix] 6140325 test time out #13868

Draft

1 task

yifjiang mentioned this pull request May 8, 2026

[https://nvbugs/6104831][draft] PR 13713 rebased onto v1.3.0rc13 — v7 #13913

Draft

chienchunhung marked this pull request as ready for review May 8, 2026 19:46

chienchunhung added 2 commits May 27, 2026 16:03

Merge upstream/main into pr13056-pr13495-combo-nvbug6104831 and resol…

7d6c158

…ve conflicts Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6104831][fix] Cap disagg KV polling wait_for slice at…

9a1a032

… 50ms Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung disabled auto-merge May 28, 2026 03:24

[https://nvbugs/6104831][fix] explicit V2 transceiver shutdown in sin…

f362223

…gle-process test to prevent daemon thread leak Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung closed this Jun 3, 2026

Uh oh!

Conversation

chienchunhung commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Summary

Failure signatures

Correctness invariant

What this PR changes

12. Opt-in flag: TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL (new)

Tests

Notes / limitations

PR Checklist

GitHub Bot Help

Uh oh!

chienchunhung commented May 7, 2026

Uh oh!

tensorrt-cicd commented May 7, 2026

Uh oh!

tensorrt-cicd commented May 7, 2026

Uh oh!

chienchunhung commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

chienchunhung commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

chienchunhung commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

chienchunhung commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

chienchunhung commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

chienchunhung commented May 3, 2026 •

edited

Loading

12. Opt-in flag: `TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL` (new)