Skip to content

[None][test] Increase kv_transfer_timeout_ms for b200 deepseek-r1 disagg gen_only perf test - #15205

Merged
chenfeiz0326 merged 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:dev-chenfeiz-unwaive-6016528
Jun 10, 2026
Merged

[None][test] Increase kv_transfer_timeout_ms for b200 deepseek-r1 disagg gen_only perf test#15205
chenfeiz0326 merged 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:dev-chenfeiz-unwaive-6016528

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

The disagg-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL perf-sanity test was failing under high concurrency (1536): all benchmark requests failed and the run exceeded its wall-clock limit. The KV cache transfer elapsed time crept just above the 60000 ms default (kv_transfer_timeout_ms), tripping the transfer timeout.

This raises kv_transfer_timeout_ms to 600000 ms in both the ctx and gen cache_transceiver_config blocks of the config, matching the value already used by the gb200/gb300 disagg perf-sanity configs, and unwaives the test.

Changes

  • tests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml: add kv_transfer_timeout_ms: 600000 to the ctx and gen cache_transceiver_config.
  • tests/integration/test_lists/waives.txt: remove the disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_...-NIXL SKIP entry for https://nvbugs/6016528.

Test plan

  • CI: perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL] on B200 passes with the increased timeout.

Links

Summary by CodeRabbit

Release Notes

  • Tests

    • Removed test waiver to re-enable previously skipped performance benchmark.
  • Chores

    • Updated timeout configuration for improved stability in distributed cache operations.

…eek-r1 disagg gen_only test

The disagg-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL
test hit the KV cache transfer timeout (elapsed slightly above the 60000 ms
default) under high concurrency (1536), causing all benchmark requests to fail.

Raise kv_transfer_timeout_ms to 600000 in both the ctx and gen
cache_transceiver_config blocks, matching the value already used by the
gb200/gb300 disagg perf-sanity configs, and unwaive the test in waives.txt.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR configures KV cache transfer timeout settings and removes the corresponding test waiver. Two timeout values are added to the NIXL backend block's cache transceiver configurations, and a single waiver entry is removed from the test skip list.

Changes

Performance Test Timeout Configuration

Layer / File(s) Summary
KV transfer timeout config and waiver removal
tests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml, tests/integration/test_lists/waives.txt
Adds kv_transfer_timeout_ms: 600000 to both generator and context cache transceiver configurations within the NIXL backend, and removes the waiver that was previously skipping the corresponding performance test.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

  • kaiyux
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: increasing kv_transfer_timeout_ms for a specific performance test configuration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description clearly explains the issue, the solution, test coverage, and includes all required sections from the template.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@chenfeiz0326
chenfeiz0326 requested a review from xinhe-nv June 10, 2026 07:26
@chenfeiz0326

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-4"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53266 [ run ] triggered by Bot. Commit: 5a7ec63 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53266 [ run ] completed with state SUCCESS. Commit: 5a7ec63
/LLM/main/L0_MergeRequest_PR pipeline #42458 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@chenfeiz0326

Copy link
Copy Markdown
Collaborator

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

@chenfeiz0326
chenfeiz0326 enabled auto-merge (squash) June 10, 2026 09:26
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53289 [ skip ] triggered by Bot. Commit: 5a7ec63 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #53289 [ skip ] completed with state SUCCESS. Commit: 5a7ec63
Skipping testing for commit 5a7ec63

Link to invocation

@chenfeiz0326
chenfeiz0326 merged commit 2d196f7 into NVIDIA:main Jun 10, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants