[NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: trace disaggregated transfer lifecycle

[chienchunhung]

Diagnostic only

This is a CI-only experiment and is not intended to merge. It is a direct child of the exact PR #16566 treatment head, with no rebase or current-main code mixed into the experiment tree.

Why

The matched #15139 / #15737-controlled A/B produced:

• Control [NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: control lifecycle change with sender wakeup fix #16565: 512/512 requests, 1557.83 output tok/s.
• Treatment [NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: test lifecycle change with sender wakeup fix #16566: 481/512 successful requests; the raw 598.42 tok/s value is censored and invalid because 31 requests failed after the 5400-second harness deadline.

The common #15737 sender wakeup fix therefore does not remove the #15139-specific lifecycle stall. The treatment repeatedly reports CTX KV-transfer future wait timeouts while GEN continues decoding.

This run distinguishes among:

1. local NIXL futures remaining pending;
2. TP/PP consensus entry skew or a slow PP rendezvous;
3. a completed consensus decision that is delayed before Python resource/KV release.

What is instrumented

TRTLLM_NVBUG_6448152_TRACE=1 is enabled only in the exact GB300 perf-sanity YAML. The trace records transition-only NVBUG6448152_DIAG markers for:

• CTX/GEN transfer enqueue and local future terminal state;
• the first CTX future wait timeout per request;
• TP and PP consensus enter/exit, local duration, rank, communicator rank, and call sequence;
• slow consensus calls and bounded 512-call checkpoints while transfers are active;
• global commit, Python call source/sequence, termination dispatch, and resource release.

The diagnostic path adds no collective, request-state decision, cancellation, timeout, or workload change. The default-off path retains the original two-communicator reduction call. Absolute steady-clock timestamps are not compared across nodes.

Experiment identity

• Parent: c58a3ac9be5ca30d69a5d326f250768b021f97f2 (PR [NVBUG-6448152][test] TEST ONLY; DO NOT REVIEW: test lifecycle change with sender wakeup fix #16566 exact head)
• Sender-fix blob: dataTransceiver.cpp = b3eb1c0f984394e798993708bebcabe885ea1644
• Exact selector: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL
• Exact stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

Local validation

• clang-format, YAPF, YAML parsing, Python byte-compilation, git diff --check, codespell, and relevant lint hooks pass.
• A local C++ build is unavailable, so CI compilation is the first compile validation.
• Two unrelated historical-tree test-list hooks crash under Python 3.9 on str | None; they were skipped for the commit after all patch-relevant hooks passed.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60126 [ run ] triggered by Bot. Commit: ad79abd Link to invocation

[tensorrt-cicd]

PR_Github #60126 [ run ] completed with state FAILURE. Commit: ad79abd
/LLM/main/L0_MergeRequest_PR pipeline #48509 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation