[TRTLLM-12721][feat] Enable disaggregated in-flight cancellation by default

[chienchunhung]

Summary

This draft enables disaggregated in-flight KV-transfer cancellation by default only for the qualified C++ NIXL/UCX configuration. Unsupported configurations continue without coordinated in-flight cancellation unless TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 explicitly requests validation, in which case startup fails.

The branch intentionally has two DCO-signed commits:

1. one squashed snapshot of [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup #15238, [https://nvbugs/6342844][fix] Prevent disaggregated KV transfer stalls #15737, [TRTLLM-12721][fix] Harden disagg cancellation consensus and buffer ownership #15794, and [TRTLLM-12721][fix] Harden PyExecutor disagg cancellation lifecycle #15795 plus prerequisite fixes found during self-review;
2. the focused [TRTLLM-12721][feat] Enable disaggregated in-flight cancellation by default #15738 policy/default-on commit.

The source parent PRs remain independently reviewable. The Python lifecycle fixes are backported to #15795 and this snapshot is refreshed from that head. The remaining C++ prerequisite fixes in commit 1 still need to be backported to #15794.

Effective policy

TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL	Qualified configuration	Unqualified configuration
unset	enabled automatically	coordinated in-flight cancellation remains disabled
0	disabled	disabled
1	enabled and required	startup fails with a configuration error

Any other value is rejected.

The qualified configuration is:

• Standard PyTorch PyExecutor, with no simultaneous KV connector.
• Asynchronous C++ cache transceiver.
• NIXL with its UCX plugin.
• Positive max_tokens_in_buffer and kv_transfer_timeout_ms.
• Ordinary TP only: PP=1, CP=1, attention DP disabled, and DWDP disabled.
• Staging-buffer transfer with overlap enabled; layerwise and zero-copy disabled.

AutoDeploy, legacy TensorRT execution, Python transceiver, direct UCX, MPI, Mooncake, NIXL-libfabric, synchronous receive, PP, CP, attention DP, DWDP, connector, layerwise, zero-copy, no-deadline, and unbounded-admission paths do not auto-enable coordinated in-flight cancellation.

Self-review fixes

C++ protocol and ownership

• Distinguish all-ready, all-rejected, and mixed peer readiness; mixed readiness quarantines advertised receive slots.
• Make public split-phase TransferSession calls own and release/quarantine their preassigned NIXL receive slots.
• Track the ready-commit boundary so cancellation before ready=true remains a clean queued rejection.
• Complete request-info fanout after the first peer notification so every sender reaches a terminal response count.
• Quarantine only when buffer advertisement may have occurred; definite pre-notify errors release normally.
• Preserve one-time NIXL metadata across a cancelled pre-notify attempt.
• Destroy connection managers before transfer buffers and the dynamically loaded plugin.
• Preserve DISAGG_TRANS_ERROR for default-off synchronous rejection instead of rewriting it as success.

Python policy and fatal cleanup

• Reject DWDP from automatic/required qualification.
• Reject explicit =1 when no transceiver is configured, including AutoDeploy and the legacy TensorRT executor.
• Track active and AsyncTransferManager-only context sends through fatal cleanup without duplicate responses or double termination.
• Use fixed-order CTX/GEN polling plus a TP-wide cleanup-state vote before releasing resources.
• Add an independent watchdog and unsafe-executor quarantine so blocked cleanup wakes shutdown without freeing transfer-owned buffers.
• Preserve generic-fatal shutdown sentinel behavior and serialize control enqueue against shutdown.
• Narrow timeout/configuration descriptions so unsupported paths are not promised coordinated cancellation.

Coverage

• Add real C++ NIXL/UCX active-mode cancellation coverage and a default-off synchronous-rejection regression.
• Put both tests in one-GPU sanity and B200 pre-merge test lists.
• Add C++ coverage for receive-slot lifecycle, mixed-ready reduction, notification ambiguity, and metadata retry.
• Add focused DWDP, missing-transceiver, AutoDeploy, detached-owner, watchdog, unsafe-shutdown, and queue/control-race tests.

Readiness

This PR remains draft and not merge-ready.

Blocking items:

• Backport the remaining C++ protocol/ownership self-review fixes in prerequisite commit 1 to [TRTLLM-12721][fix] Harden disagg cancellation consensus and buffer ownership #15794, then refresh this snapshot again.
• Land and integrate [TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode #15798 after resolving its mixed-version new enabled GEN -> legacy CTX settlement and adding serialization, TP>1, and C++ NIXL integration coverage.
• Implement the [TRTLLM-12721][docs] Design disaggregated peer cancellation protocol #15799 peer cancellation/terminal protocol, or obtain explicit approval for a bounded fail/restart policy that provides equivalent no-peer and post-commit memory safety.
• Run after-ready=true cancellation/poison fault injection and multi-rank TP consensus/mixed-ready coverage.
• Compile and run bufferIndexHolderTest, cacheTransferStateTest, peerProtocolTest, and agentCommTest.
• Run the C++/NIXL synchronous and asynchronous suites plus the original B200/B300 gen-only reproducer.
• Run full pre-merge/post-merge-stage CI on the final stacked head.

Until #15798 is complete, upgrades must be atomic: set TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=0 on every CTX and GEN worker, upgrade and verify all workers, then remove the override. Default-on also remains gated on #15799 or the approved bounded fail/restart alternative.

Validation

Passed locally on f303901b7c (prerequisite snapshot 06ece0c791):

• Full changed-file pre-commit suite for the refreshed [TRTLLM-12721][fix] Harden PyExecutor disagg cancellation lifecycle #15795 files: isort, YAPF, Ruff, Ruff format, codespell, YAML, legacy lint, and repository checks.
• Test-list duplicate and AST validation under Python 3.13.
• Python syntax compilation.
• git diff --check.
• Two-commit history and DCO sign-offs.
• Range-diff confirms the focused default-on commit is patch-identical to the previous 36751c2c62 head.

Not available locally:

• Focused pytest collection (transformers is not installed).
• Native C++/CUDA build and distributed GPU execution.

The previous full CI run targeted a superseded head and must be replaced for this commit.

Dependency graph

Arrows point from prerequisite to dependent. #15238 and #15737 are parallel, independently mergeable inputs to #15794. All PRs target NVIDIA/main; stacked child diffs are cumulative until their parents merge.

#15798 and #15799 are production/default-on hardening follow-ups. They do not block the default-off merges of #15238, #15794, or #15795. #15798 is required for #15738. #15799 may be replaced only by an explicitly approved bounded fail/restart policy that satisfies the same memory-safety requirements.

flowchart LR
    PR15238["#15238<br/>gated cancellation<br/>(default off)"]
    PR15737["#15737<br/>CacheSender lost-wakeup fix"]
    PR15794["#15794<br/>C++ protocol and buffer safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor lifecycle safety<br/>(default off)"]
    PR15798["#15798<br/>CTX↔GEN protocol/mode negotiation"]
    PR15799["#15799<br/>peer cancel/terminal protocol<br/>(design scaffold)"]
    PR15738["#15738<br/>qualified default-on policy"]

    PR15238 --> PR15794
    PR15737 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or bounded fail/restart policy| PR15738

Loading

[chienchunhung]

/bot run

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56696 [ run ] triggered by Bot. Commit: de0eb2b Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56708 [ run ] triggered by Bot. Commit: 36751c2 Link to invocation

[tensorrt-cicd]

PR_Github #56708 [ run ] completed with state FAILURE. Commit: 36751c2
/LLM/main/L0_MergeRequest_PR pipeline #45531 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56759 [ run ] triggered by Bot. Commit: 83a0565 Link to invocation

[tensorrt-cicd]

PR_Github #56759 [ run ] completed with state SUCCESS. Commit: 83a0565
/LLM/main/L0_MergeRequest_PR pipeline #45578 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation