[TRTLLM-12721][fix] Harden PyExecutor disagg cancellation lifecycle

[chienchunhung]

Summary

This draft adds the PyExecutor and IPC lifecycle support required by disaggregated in-flight KV-transfer cancellation. It depends on #15794 and keeps the feature disabled by default.

#15794 owns timeout detection, rank consensus, native cancellation, quiescence, and the topology-poison latch. This PR consumes that contract without rank-local cancellation, duplicate terminal responses, unsafe teardown, or hung proxy callers.

Changes

• Make C++ the sole owner of active-transfer timeout cancellation; Python no longer performs a second TP vote or rank-local timeout cancellation.
• Preserve terminal-response ownership:
  • cancellation wins over a racing success only when the feature is active;
  • native cancellation is retried through the status loop;
  • exactly one FinishReason.CANCELLED response is emitted after native and AsyncTransferManager owners release the request;
  • transfer-only cancellation progresses even when no model forward is scheduled.
• Treat topology poison as process-fatal:
  • close and drain request ingress and pending control actions;
  • emit one terminal error per request;
  • retain active and detached transfer owners;
  • perform one rank-symmetric best-effort cancellation/status pass;
  • quarantine the PyExecutor and outer worker instead of inferring remote quiescence from local futures or freeing transfer-owned resources.
• Propagate fatal worker state across IPC:
  • bypass postprocessing with WorkerFatalError and fail all proxy-owned results;
  • reject racing and future submissions;
  • retire worker_main without cross-thread request-ZMQ access;
  • stop postprocessors locally with nonblocking FatalStop messages;
  • use bounded MPI-session shutdown/abort instead of unbounded future waits.
• Serialize control enqueue against shutdown.

Activation scope

Opt-in is accepted only for standard PyTorch PyExecutor with the C++ transceiver, NIXL/UCX, asynchronous preallocated staging buffers, positive admission and timeout bounds, ordinary TP (PP=CP=1, no attention DP), no KV connector, and no layerwise or zero-copy transfer.

The IPC executor must also own disposable MpiPoolSession worker processes so poison can be resolved by starting fresh processes. Reusable MpiCommSession, remote management-leader, Ray, RPC, and single-process paths fail opt-in preflight. With the flag unset, existing paths retain legacy behavior.

Validation

Passed on 198c2efaac:

• DCO and the full changed-file pre-commit suite.
• git diff --check and Python syntax compilation.
• Three independent code-owner-style reviews covering distributed lifecycle, upstream integration, default-off compatibility, and tests; no remaining P0-P2 findings.
• Post-rebase patch comparison against [TRTLLM-12721][fix] Harden disagg cancellation consensus and buffer ownership #15794; upstream postprocessor-hook and late-response fixes are preserved.

Not available locally: focused pytest collection because transformers is not installed, real MpiPoolSession plus postprocess-parallel fatal IPC teardown, and distributed GPU poison fault injection.

Stack follow-ups

#15798 and #15799 do not block this default-off PR. #15798 is required before #15738 enables the feature for qualified deployments. #15799 must be implemented unless an explicitly approved bounded fail/restart policy provides equivalent no-peer and ambiguous-transfer safety.

Dependency graph

Arrows point from prerequisite to dependent. #15238 and #15737 are parallel inputs to #15794. Stacked child diffs remain cumulative until their parents merge.

flowchart LR
    PR15238["#15238<br/>gated cancellation<br/>(default off)"]
    PR15737["#15737<br/>CacheSender lost-wakeup fix"]
    PR15794["#15794<br/>C++ protocol and buffer safety<br/>(default off)"]
    PR15795["#15795<br/>PyExecutor + IPC lifecycle safety<br/>(default off)"]
    PR15798["#15798<br/>CTX↔GEN protocol/mode negotiation"]
    PR15799["#15799<br/>peer cancel/terminal protocol<br/>(design scaffold)"]
    PR15738["#15738<br/>qualified default-on policy"]

    PR15238 --> PR15794
    PR15737 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or approved bounded fail/restart policy| PR15738

Loading

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56694 [ run ] triggered by Bot. Commit: d85793b Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56711 [ run ] triggered by Bot. Commit: d7ae0fa Link to invocation

[tensorrt-cicd]

PR_Github #56694 [ run ] completed with state ABORTED. Commit: d85793b

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #56741 [ run ] triggered by Bot. Commit: 198c2ef Link to invocation

[tensorrt-cicd]

PR_Github #56745 [ run ] triggered by Bot. Commit: 198c2ef Link to invocation

[tensorrt-cicd]

PR_Github #56741 [ run ] completed with state ABORTED. Commit: 198c2ef

Link to invocation

[tensorrt-cicd]

PR_Github #56745 [ run ] completed with state FAILURE. Commit: 198c2ef
/LLM/main/L0_MergeRequest_PR pipeline #45564 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation