[TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode

[chienchunhung]

Summary

Negotiate cancellation compatibility between the C++ NIXL context and generation services before cache DMA.

• Append a versioned protocol descriptor to the opaque NIXL agent identity without changing the positional DataTransceiverState or RequestInfo wire layout.
• Advertise protocol versions, capabilities, effective cancellation mode, and a process-session token.
• Validate every local and remote rank. Compatible peers proceed; protocol-aware mismatches settle through the existing request/ready=false exchange without DMA.

The negotiation is scoped to the C++ NIXL cache-transceiver path. Python transceiver, direct UCX, MPI, Mooncake, and descriptor-less AgentConnectionManager construction remain unchanged.

Compatibility and rollout

Local / peer state	Result
new baseline / new baseline	proceed
new baseline / descriptor-less legacy	proceed only while cancellation is known to be off on the legacy deployment
new enabled / compatible new enabled	proceed
new enabled / new baseline, either direction	reject with ready=false; no DMA
new enabled CTX / legacy GEN	CTX rejects with ready=false; no DMA
new enabled GEN / legacy CTX	GEN rejects before advertisement; legacy CTX may require drain/restart to settle

A descriptor-less peer cannot prove its effective mode. Rolling upgrades must keep cancellation disabled until both services are protocol-aware. Enabling by default remains blocked on #15799 or an explicitly approved drain/restart policy for the final row.

Safety properties

• Local ranks must advertise one exact mode, version range, and capability set; session tokens remain rank-specific.
• Active mode requires in-flight cancellation and pre-DMA rejection capabilities. The selected common version is bound to the transfer session.
• The advertised CommState[selfIdx] identity and address must match the live NIXL notification sender.
• GEN validates unsafe legacy/malformed peers before reserving receive buffers and latches local rejection. Unexpected readiness or incomplete settlement poisons advertised slots; unanimous ready=false releases them.
• The terminal v1 envelope prevents hostname text from impersonating metadata and rejects unversioned field extensions.

Validation

Local checks passed: DCO, repository pre-commit, git diff --check, and a self-contained C++17 -Wall -Wextra -Werror protocol check.

Coverage includes descriptor parsing and spoof resistance, capability/version/rank matrices, serialization and live-sender binding, rejected-session buffer ownership, baseline NIXL propagation, and real 1P+1D NIXL mode mismatches in both directions with future settlement, no-DMA sentinels, and receive-pool reuse checks.

Remaining verification:

• Run the full TensorRT-LLM CI pipeline after this history rewrite.
• Add TP>1 live-NIXL mismatch rank/order coverage.
• Verify latest-main [None][feat] Handle empty-block ranks with Helix CP #15383 Helix empty-block short/long cases and run disaggregated stress/performance testing.
• Resolve enabled-GEN → legacy-CTX settlement through [TRTLLM-12721][docs] Design disaggregated peer cancellation protocol #15799 or the deployment policy above.

Dependency graph

Arrows point from prerequisite to dependent. All PRs target NVIDIA/main; stacked child diffs remain cumulative until their parents merge. #15798 is a single unique commit on the current #15794 head.

#15798 and #15799 harden production/default-on readiness; they do not block the default-off merges of #15238, #15794, or #15795. #15799 may be replaced only by an explicitly approved bounded fail/restart policy with equivalent memory-safety guarantees.

flowchart LR
    PR15238["#15238<br/>gated cancellation<br/>(default off)"]
    PR15737["#15737<br/>CacheSender lost-wakeup fix"]
    PR15794["#15794<br/>C++ protocol and buffer safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor lifecycle safety<br/>(default off)"]
    PR15798["#15798<br/>CTX↔GEN protocol/mode negotiation"]
    PR15799["#15799<br/>peer cancel/terminal protocol<br/>(design scaffold)"]
    PR15738["#15738<br/>qualified default-on policy"]

    PR15238 --> PR15794
    PR15737 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or bounded fail/restart policy| PR15738

Loading

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56743 [ run ] triggered by Bot. Commit: 25aac38 Link to invocation

[tensorrt-cicd]

PR_Github #56743 [ run ] completed with state FAILURE. Commit: 25aac38
/LLM/main/L0_MergeRequest_PR pipeline #45562 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation