[None][test] Increase kv_transfer_timeout_ms for b200 deepseek-r1 disagg gen_only perf test - #15205
Conversation
…eek-r1 disagg gen_only test The disagg-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL test hit the KV cache transfer timeout (elapsed slightly above the 60000 ms default) under high concurrency (1536), causing all benchmark requests to fail. Raise kv_transfer_timeout_ms to 600000 in both the ctx and gen cache_transceiver_config blocks, matching the value already used by the gb200/gb300 disagg perf-sanity configs, and unwaive the test in waives.txt. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
📝 WalkthroughWalkthroughThis PR configures KV cache transfer timeout settings and removes the corresponding test waiver. Two timeout values are added to the NIXL backend block's cache transceiver configurations, and a single waiver entry is removed from the test skip list. ChangesPerformance Test Timeout Configuration
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/bot run --disable-fail-fast --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-4" |
|
PR_Github #53266 [ run ] triggered by Bot. Commit: |
|
PR_Github #53266 [ run ] completed with state |
|
/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline" |
|
PR_Github #53289 [ skip ] triggered by Bot. Commit: |
|
PR_Github #53289 [ skip ] completed with state |
Summary
The
disagg-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXLperf-sanity test was failing under high concurrency (1536): all benchmark requests failed and the run exceeded its wall-clock limit. The KV cache transfer elapsed time crept just above the 60000 ms default (kv_transfer_timeout_ms), tripping the transfer timeout.This raises
kv_transfer_timeout_msto 600000 ms in both thectxandgencache_transceiver_configblocks of the config, matching the value already used by thegb200/gb300disagg perf-sanity configs, and unwaives the test.Changes
tests/scripts/perf-sanity/disaggregated/b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml: addkv_transfer_timeout_ms: 600000to the ctx and gencache_transceiver_config.tests/integration/test_lists/waives.txt: remove thedisagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_...-NIXLSKIP entry for https://nvbugs/6016528.Test plan
perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-b200_deepseek-r1-fp4_8k1k_con1536_ctx1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL]on B200 passes with the increased timeout.Links
Summary by CodeRabbit
Release Notes
Tests
Chores