TEST ONLY; DO NOT REVIEW — [NVBUG-6448152][test] trace current-tree disaggregated phases

[chienchunhung]

Purpose

This diagnostic is a direct child of the exact current implementation commit a2fca780. It localizes the later-tree gap before context KV-transfer submission without changing the coordinator, scheduler decisions, admission policy, transfer traffic, or timeouts.

Instrumentation

• Adds env-gated, transition-only markers for request fetch and activation, PP schedule production/send/receive/apply, forward entry, CUDA enqueue/observed timing, Python-to-C++ transfer submission, local future terminal state, coordinator vote/commit, and Python cleanup/resource release.
• Includes world rank, PP rank, request, executor iteration where available, and monotonic time.
• Bounds request and transition cardinality. Idle, retry, unavailable-event, and future-poll counts emit only at powers of two plus a final summary.
• Reuses existing profiler timing values and the existing nonblocking GPU-elapsed observation. It creates, queries, synchronizes, retains, or reallocates no CUDA event.
• Enables TRTLLM_NVBUG_6448152_PHASE_TRACE=1 only in the exact perf-sanity workload YAML.

Verification

• Explicit-file pre-commit suite passed, including Python and C++ formatting, YAML validation, lint, and test-list validators.
• Python syntax compilation and a focused bounded-marker/CUDA-metadata smoke test passed.
• The exact GB300 workload is the required runtime validation.

Targeted workload

Stage: GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1

Selector: disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL

This is an instrumentation run. Its traced throughput is diagnostic and must not be compared directly with uninstrumented throughput. Any timeout or failed request censors the throughput result.

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

[tensorrt-cicd]

PR_Github #60811 [ run ] triggered by Bot. Commit: d11e755 Link to invocation

[tensorrt-cicd]

PR_Github #60811 [ run ] completed with state FAILURE. Commit: d11e755
/LLM/main/L0_MergeRequest_PR pipeline #49087 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation