[TRTLLM-12721][docs] Design disaggregated peer cancellation protocol

[chienchunhung]

Summary

Adds the design scaffold for the generation-safe peer cancellation and terminal protocol needed to qualify default-on disaggregated in-flight cancellation after #15798.

This PR changes documentation only: it adds no runtime control messages, leaves cancellation default-off, and does not treat a released backend handle or received ACK as proof that NIXL memory is quiescent or reusable.

This branch is stacked on #15798. Review the single #15799 commit after the current #15798 head for the PR-specific delta.

Initial qualified scope

The proposed active protocol is limited to the C++ transceiver with the CacheTransceiver NIXL backend over NIXL-UCX, preallocated buffers, a positive transfer timeout, overlap enabled, PP=1, CP=1, attention-DP off, layerwise off, and zero-copy off. Unsupported and unqualified configurations remain on their existing paths.

Protocol decisions

• A transfer is identified by a generation-safe context ticket, CTX/GEN attempt IDs, endpoint epochs, rank namespaces, and a canonical topology digest; request ID alone never authorizes work.
• Phase B2 extends [TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode #15798 with a v2 descriptor and deterministic topology-wide selection of the control-protocol version, capability set, and serialization ABI. A dynamic OPEN remains a proposal and cannot by itself authorize buffers, READY, or DMA.
• CTX and GEN use default-deny registries with parent attempt leases, per-edge evidence, ABA-safe slot generations, and cleanup-once ownership. One completed component cannot release a partial parent lease.
• Admission, cancellation, and completion decisions use unconditional ordered topology-local rank agreement. Timeout and transport callbacks record intent only; response threads do not enter collectives or reclaim resources.
• READY follows prepare → topology decision → installation barrier → publication. Cancellation is parent-scoped, so cancelling one child edge marks all sibling edges pending.
• Active OPEN carries the complete RequestAndBufferInfo through a capability-gated control notification variant and requires bounded structural validation before native decoding or state mutation.
• CANCEL and TERMINAL are sent only for an exact installed session. Terminal evidence is role-specific, and a receipt acknowledges bytes—not memory quiescence or topology-wide completion.
• UNKNOWN quarantines still-owned, unproven lease generations and invokes bounded admission-stop/process-restart handling. Only topology-wide NOT_STARTED permits cancellation-related early release; the authoritative success fast path remains unchanged.
• Typed ticket identity is preserved through createResult(), Nanobind, HTTP/base64, worker handoff, and retries, with total opaque-state bounds and ABI validation.

Implementation and qualification gates

• Provide a timeout-capable collective or process watchdog for every local decision round, and prove control-dispatch lock ordering.
• Complete the C++ API/ABI review for typed ContextPhaseParams ticket storage and preserve the ticket through every qualified handoff.
• Implement the control carrier plus checked, bounded decoding and fuzzing for DataTransceiverState and the complete RequestAndBufferInfo.
• Prove authoritative bundle-wide completion and enforce parent-lease, slot-generation, terminal-retention, and bounded-restart ownership.
• Add deterministic TP=2 admission/cancel/complete/publication races, multi-peer tests, exact-cell B200/B300 stress and performance qualification, and full pre-merge/post-merge-stage CI.

Validation

• Rechecked the design against the current [TRTLLM-12721][fix] Negotiate disaggregated cancellation protocol and mode #15798 head, including its fixed v1 descriptor, capability bits, selected-version result, and pre-DMA rejection contract.
• Completed $trtllm-review lifecycle, collective, serialization, ownership, retry, and dynamic-peer passes; the final stop-condition found no remaining P0/P1 contradiction.
• Passed DCO, codespell, whitespace, and all applicable local pre-commit hooks. The two unrelated test-list hooks were skipped locally because this host uses Python 3.9.
• Added the document to the Sphinx toctree; a local Sphinx build was unavailable because Sphinx is not installed on this host.

Dependency graph

Solid arrows are required prerequisites. #15238 and #15737 are independent inputs to #15794. #15798 and #15799 are default-on hardening work and do not block the default-off merges of #15238, #15794, or #15795. The dotted edge means #15799 is required for #15738 unless an explicitly approved bounded fail/restart policy provides equivalent memory safety.

flowchart LR
    PR15238["#15238<br/>gated cancellation<br/>(default off)"]
    PR15737["#15737<br/>CacheSender lost-wakeup fix"]
    PR15794["#15794<br/>C++ protocol and buffer safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor lifecycle safety<br/>(default off)"]
    PR15798["#15798<br/>CTX↔GEN protocol/mode negotiation"]
    PR15799["#15799<br/>peer cancel/terminal protocol<br/>(design scaffold)"]
    PR15738["#15738<br/>qualified default-on policy"]

    PR15238 --> PR15794
    PR15737 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or bounded fail/restart policy| PR15738

Loading

[chienchunhung]

/bot run --disable-fail-fast

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56740 [ run ] triggered by Bot. Commit: 63009f4 Link to invocation

[tensorrt-cicd]

PR_Github #56739 [ run ] triggered by Bot. Commit: 63009f4 Link to invocation

[tensorrt-cicd]

PR_Github #56740 [ run ] completed with state ABORTED. Commit: 63009f4

Link to invocation

[tensorrt-cicd]

PR_Github #56739 [ run ] completed with state FAILURE. Commit: 63009f4
/LLM/main/L0_MergeRequest_PR pipeline #45559 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation