[https://nvbugs/6481034][test] Trace Kimi gen-only KV transfer timeout

[chienchunhung]

Summary

• instrument the Python cache-transceiver path with opt-in, structured per-request KV_TRANSFER_TRACE lifecycle events
• validate the exact unwaived GB300 Kimi K2.5 gen-only test on a base containing [https://nvbugs/6503299][fix] Default fabric memory KV pool for Python cache transceiver #16832
• keep the test at main's existing GEN free_gpu_memory_fraction: 0.8 (CTX remains 0.6); this PR does not raise the KV-cache fraction
• align the multi-node perf launcher with pytest's least_duration shard selection so both launch and collect artifacts for the same test

Validation result

The targeted run used the Python transceiver with transfer overlap disabled, kv_transfer_timeout_ms=60000, and GEN free_gpu_memory_fraction: 0.8.

The workload itself passed:

• 4,096/4,096 requests completed successfully; 0 client failures
• benchmark duration: 360.63 seconds
• the benchmark, GEN, and CTX pytest roles reported success
• no num_fitting_reqs=0, insufficient-KV-cache, KV-transfer-timeout, OOM, failed AgentResult, or failed NIXL outcome was observed
• all 4,096 requests were represented in the dedicated traces across all expected 4 CTX and 16 GEN ranks

The captured transfer latencies were safely below the existing 60-second timeout:

Phase	p50	p99	Maximum
complete GEN synchronous transfer	2.856 ms	20.068 ms	6.890 s
GEN blocking wait	2.166 ms	12.388 ms	6.860 s
CTX ready to receiver request	88.884 ms	5.220 s	8.221 s
CTX worker queue	0.205 ms	3.281 ms	15.544 ms
NIXL operation	0.660 ms	1.452 ms	1.437 s

The GEN logs also confirm that #16832's Python-transceiver fabric-memory default was active (TRTLLM_KVCACHE_POOL_USE_FABRIC_MEMORY=1). The original NVBUG#6481034 KV-capacity/transfer-timeout signature therefore appears fixed by #16832 on the current 0.8 configuration. This run provides no evidence for raising the memory fraction or transfer timeout.

Targeted stage:
GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3

• CI completion comment
• parent pipeline
• targeted stage
• collected artifact

Further issue exposed by the run

Although the workload completed and wrote benchmark_status=Done, the overall stage remained red. All GEN test logic and MPI workers exited cleanly, but the multi-node GEN srun did not finish reaping and therefore never created gen_server_0.done. The benchmark role waited for that sentinel until one Slurm step exited nonzero; --kill-on-bad-exit=1 then cancelled the remaining job.

The evidence points to a post-benchmark launcher/teardown problem in the GEN-log completion-sentinel flow introduced by #16717, not a KV-transfer failure. The logs did not capture the exact lingering launcher task, so its identity still needs a teardown-focused diagnostic before assigning a definitive code-level cause.

There is also a separate ruff-format failure in jenkins/scripts/perf/submit.py, and this draft currently conflicts with newer main.

Current PR scope

Beyond tracing/instrumentation, the diff contains the pytest-shard alignment in jenkins/scripts/perf/submit.py and its unit coverage. This was needed because an earlier run had the launcher and pytest selecting different tests/output directories.

The net diff contains:

• opt-in transfer lifecycle events and dedicated, line-buffered per-rank trace files
• propagation and startup verification of trace settings for all external executor ranks
• a trace snapshot manifest for artifact completeness
• trace enablement in the Kimi and colocated DeepSeek stage configurations
• perf-launcher/pytest least_duration shard alignment and tests

It contains no net change to KV-cache memory fractions, transfer timeout, or test waivers, and no runtime product fix beyond diagnostics.

Local validation

• YAML parsed and verified: Kimi GEN 0.8, CTX 0.6
• python3 -m py_compile passed for all modified Python files
• git diff --check passed

[chienchunhung]

/bot run --disable-fail-fast --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

[tensorrt-cicd]

PR_Github #62028 [ run ] triggered by Bot. Commit: 6ffa538 Link to invocation

[coderabbitai]

Walkthrough

Updates one perf-sanity waiver and increases the generation worker’s free_gpu_memory_fraction from 0.8 to 0.85 for a GB300 Kimi-K2.5 configuration.

Changes

Perf sanity updates

Layer / File(s)	Summary
Waiver and KV-cache configuration tests/integration/test_lists/waives.txt, tests/scripts/perf-sanity/disaggregated/...yaml	Removes one perf/test_perf_sanity.py::test_e2e waiver and raises the generation KV-cache GPU memory fraction from 0.8 to 0.85.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16920: Removes related GB300 Kimi-K2.5 perf-sanity waiver entries and adjusts KV-transfer timeout settings.

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed but contradicts the diff by claiming no waiver or memory-fraction changes and describing absent tracing work.	Rewrite the description to match the diff: remove the NVBUG 6490049 waiver, change GEN free_gpu_memory_fraction from 0.8 to 0.85, and document relevant validation.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title relates to the Kimi gen-only timeout issue, but it does not identify the waiver removal or GEN KV-cache fraction change.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chienchunhung]

/bot run --post-merge --disable-fail-fast --only-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

[tensorrt-cicd]

PR_Github #62037 [ run ] triggered by Bot. Commit: 6ffa538 Link to invocation

[tensorrt-cicd]

PR_Github #62037 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4"

[tensorrt-cicd]

PR_Github #62039 [ run ] triggered by Bot. Commit: 6ffa538 Link to invocation

[tensorrt-cicd]

PR_Github #62028 [ run ] completed with state ABORTED. Commit: 6ffa538

Link to invocation

[tensorrt-cicd]

PR_Github #62039 [ run ] completed with state SUCCESS. Commit: 6ffa538
/LLM/main/L0_MergeRequest_PR pipeline #50224 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Targeted CI result: the Kimi 0.85 change was not exercised because the perf launcher and pytest selected different shard entries.

• Pipeline: L0_MergeRequest_PR #50224
• Targeted SBSA multi-node job: L0_Test-SBSA-Multi-GPU #2958
• submit.py --split-group 4 positionally selected the fourth rendered entry (Kimi) for Slurm resource sizing, stage-level environment, and wrapper output-directory setup.
• The actual pytest command independently used --splitting-algorithm least_duration --splits 4 --group 4. Based on the committed duration cache, pytest group 4 contains the DeepSeek gen-only test, while the Kimi gen-only test is in group 3.
• Pytest therefore generated and ran the DeepSeek configuration, then failed with missing 'prev_device_step_time' in gen_server_*.log. The artifacts contain the DeepSeek runtime files; Kimi free_gpu_memory_fraction: 0.85 was not tested.

This is a deterministic positional-vs-duration shard mismatch, not a failure of the proposed 0.85 setting. A follow-up run should target shard 3 to reach the Kimi pytest case under the current duration mapping; its wrapper will still derive stage-level environment from the positional DeepSeek entry, although the 20-GPU/5-node resource shape is identical.

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[tensorrt-cicd]

PR_Github #62561 [ run ] triggered by Bot. Commit: 827abc5 Link to invocation

[tensorrt-cicd]

PR_Github #62561 [ run ] completed with state SUCCESS. Commit: 827abc5
/LLM/main/L0_MergeRequest_PR pipeline #50706 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[tensorrt-cicd]

PR_Github #62618 [ run ] triggered by Bot. Commit: 2191139 Link to invocation

[tensorrt-cicd]

PR_Github #62618 [ run ] completed with state SUCCESS. Commit: 2191139
/LLM/main/L0_MergeRequest_PR pipeline #50758 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Latest targeted result (#50758) changes the diagnosis:

• With main including [https://nvbugs/6503299][fix] Default fabric memory KV pool for Python cache transceiver #16832, the Kimi gen-only benchmark itself completed 4096/4096 requests with 0 client failures and did not reproduce the prior KV-transfer timeout/OOM signature.
• The reported CI failure happened after the benchmark. submit.py --split-group 3 selected the third test positionally (DeepSeek), while pytest-split group 3 selected Kimi by cached duration. As a result, launcher status/sentinel/trace paths were under DeepSeek while pytest waited and archived under Kimi. This caused the post-benchmark hang and excluded the trace files from the artifact.
• Commit 49c05edde makes the launcher mirror pytest-split's least_duration assignment, fails immediately if the trace directory differs from the pytest-selected output directory, and writes an fsynced manifest proving that all expected 4 CTX + 16 GEN rank traces are in the archived test tree.

Therefore the remaining product question is no longer “can this workload complete?” in the latest sample. It is whether #16832 is the actual fix and whether this PR's 0.85 fraction is still necessary. The next targeted run should finally provide the complete per-request trace needed to answer that; an eventual 0.8/fabric-memory A/B can isolate the memory-fraction contribution.

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[tensorrt-cicd]

PR_Github #62854 [ run ] triggered by Bot. Commit: 49c05ed Link to invocation

[chienchunhung]

/bot cancel

[github-actions]

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label. Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the ci: post-merge approved PR label applied by an active member of NVIDIA/trt-llm-ci-approvers. The approval label remains in place when new commits are pushed.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label.

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[chienchunhung]

/bot kill

[chienchunhung]

/bot run --disable-fail-fast --add-multi-gpu-test --stage-list "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3"

[tensorrt-cicd]

PR_Github #62859 [ run ] triggered by Bot. Commit: cf5afef Link to invocation

[tensorrt-cicd]

PR_Github #62854 [ run ] completed with state ABORTED. Commit: 49c05ed

Link to invocation

[tensorrt-cicd]

PR_Github #62859 [ run ] completed with state FAILURE. Commit: cf5afef
/LLM/main/L0_MergeRequest_PR pipeline #50982 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

Closing because no product or test-list change remains to merge. The exact GB300 Kimi gen-only test was already unwaived by #16717. The targeted validation on a base containing #16832, with GEN free_gpu_memory_fraction=0.8 and kv_transfer_timeout_ms=60000, completed 4,096/4,096 requests successfully; all captured transfers completed below the timeout and the original KV-capacity/transfer-timeout signature did not reproduce. The remaining red stage outcome was a separate post-benchmark GEN srun/completion-sentinel teardown issue and should be tracked independently. Diagnostic tracing and the generic pytest-shard alignment can be split into separate PRs if independently desired.