[None][test] Update DSV32 32k4k config to avoid timeout issue by chenfeiz0326 · Pull Request #14856 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-06-02T07:47:55Z

… con2048 disagg

The disagg test for deepseek-v32-fp4 32k4k con2048 (ctx1_dep4_gen1_dep32 with eplb288 mtp1) was timing out at the 5400s benchmark deadline because the ctx worker (max_batch_size=1) cannot keep up with concurrency=2048 * multi_round=5 (10240 prompts). Trailing requests miss the default ~30s NIXL kv-transfer timeout, so ~9% of requests fail and the benchmark phase exceeds 5400s.

Recipe (verified on lyris GB300 against TRT-LLM main TOT):

multi_round: 5 -> 2 (4096 prompts; benchmark phase ~46m, well under 5400s)
cache_transceiver_config.kv_transfer_timeout_ms: 600000 (ctx + gen, both NIXL and UCX backends)

After the recipe: 0 failed requests, throughput slightly improved.

Applied to all 6 con2048 v32 32k4k disagg configs (gb200/gb300 x NIXL/UCX under tests/scripts/perf/disaggregated, plus the two gb200/gb300 NIXL configs under tests/scripts/perf-sanity/disaggregated). Unwaives the now-passing e2e-gb300-NIXL and gen_only-gb200-NIXL entries in waives.txt.

Bug: https://nvbugs/6085022

Summary by CodeRabbit

Tests
- Updated benchmark configurations for disaggregated scenarios, reducing benchmark rounds and adding explicit transfer timeout settings for improved test reliability.
- Removed test waivers for specific disaggregated upload benchmark variants.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-06-02T07:52:20Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c0290c12-a872-492e-ad26-dca9226ccca2

📥 Commits

Reviewing files that changed from the base of the PR and between 460adc7 and 5c39da6.

📒 Files selected for processing (7)

tests/integration/test_lists/waives.txt
tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml
tests/scripts/perf/disaggregated/gb300_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml
tests/scripts/perf/disaggregated/gb300_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR updates disaggregated deepseek-v32 benchmark configurations across six YAML files and removes specific test skip waivers. All benchmark configs are adjusted to reduce iteration rounds and add explicit KV transfer timeout configuration.

Changes

Disaggregated deepseek-v32 benchmark configuration and test waiver updates

Layer / File(s)	Summary
Test waivers for disagg_upload variants `tests/integration/test_lists/waives.txt`	Removed skip waivers for `disagg_upload-e2e-gb300-deepseek-v32` and `disagg_upload-gen_only-gb200-deepseek-v32`, while retaining `disagg_upload-gen_only` entries for `deepseek-r1` on GB200 and B200.
Benchmark parameter updates `tests/scripts/perf-sanity/disaggregated/gb200_deepseek-v32--NIXL.yaml`, `tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v32--NIXL.yaml`, `tests/scripts/perf/disaggregated/gb200_deepseek-v32--NIXL.yaml`, `tests/scripts/perf/disaggregated/gb200_deepseek-v32--UCX.yaml`, `tests/scripts/perf/disaggregated/gb300_deepseek-v32--NIXL.yaml`, `tests/scripts/perf/disaggregated/gb300_deepseek-v32--UCX.yaml`	Reduced `multi_round` from 5 to 2 across all six benchmark YAML files. Added `kv_transfer_timeout_ms: 600000` to both `worker_config.gen.cache_transceiver_config` and `worker_config.ctx.cache_transceiver_config` in each file, applying the same configuration pattern to both perf-sanity and perf directories.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14787: Also modifies tests/integration/test_lists/waives.txt to adjust integration test skip waivers.

Suggested reviewers

ruodil
StanleySun639
dongfengy

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main purpose: updating DSV32 32k4k configuration to address a timeout issue, which aligns with the core changes.
Description check	✅ Passed	The PR description provides a clear explanation of the issue, the applied recipe (multi_round reduction and timeout increase), verification details, and scope across all affected config files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

fredricz-20070104

Approve. This one is used to resolve the timeout issue.

chenfeiz0326 · 2026-06-02T08:02:37Z

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4"

tensorrt-cicd · 2026-06-02T08:09:10Z

PR_Github #51558 [ run ] triggered by Bot. Commit: 7aba22f Link to invocation

tensorrt-cicd · 2026-06-02T10:53:11Z

PR_Github #51558 [ run ] completed with state SUCCESS. Commit: 7aba22f
/LLM/main/L0_MergeRequest_PR pipeline #40951 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

… con2048 disagg The disagg test for deepseek-v32-fp4 32k4k con2048 (ctx1_dep4_gen1_dep32 with eplb288 mtp1) was timing out at the 5400s benchmark deadline because the ctx worker (max_batch_size=1) cannot keep up with concurrency=2048 * multi_round=5 (10240 prompts). Trailing requests miss the default ~30s NIXL kv-transfer timeout, so ~9% of requests fail and the benchmark phase exceeds 5400s. Recipe (verified on lyris GB300 against TRT-LLM main TOT): - multi_round: 5 -> 2 (4096 prompts; benchmark phase ~46m, well under 5400s) - cache_transceiver_config.kv_transfer_timeout_ms: 600000 (ctx + gen, both NIXL and UCX backends) After the recipe: 0 failed requests, throughput slightly improved. Applied to all 6 con2048 v32 32k4k disagg configs (gb200/gb300 x NIXL/UCX under tests/scripts/perf/disaggregated, plus the two gb200/gb300 NIXL configs under tests/scripts/perf-sanity/disaggregated). Unwaives the now-passing e2e-gb300-NIXL and gen_only-gb200-NIXL entries in waives.txt. Bug: https://nvbugs/6085022 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-06-02T13:52:57Z

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-3,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-4"

tensorrt-cicd · 2026-06-02T14:13:36Z

PR_Github #51614 [ run ] triggered by Bot. Commit: 33dc98c Link to invocation

tensorrt-cicd · 2026-06-02T15:51:42Z

PR_Github #51614 [ run ] completed with state SUCCESS. Commit: 33dc98c
/LLM/main/L0_MergeRequest_PR pipeline #40998 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-06-03T02:19:50Z

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-06-03T02:25:20Z

PR_Github #51732 [ skip ] triggered by Bot. Commit: 33dc98c Link to invocation

tensorrt-cicd · 2026-06-03T02:30:53Z

PR_Github #51732 [ skip ] completed with state SUCCESS. Commit: 33dc98c
Skipping testing for commit 33dc98c

Link to invocation

github-actions Bot assigned chenfeiz0326 Jun 2, 2026

chenfeiz0326 requested a review from fredricz-20070104 June 2, 2026 07:48

fredricz-20070104 approved these changes Jun 2, 2026

View reviewed changes

chenfeiz0326 and others added 2 commits June 2, 2026 13:52

update

33dc98c

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 force-pushed the fix/dsv32-fp4-32k4k-con2048-disagg-recipe branch from 7aba22f to 33dc98c Compare June 2, 2026 13:52

chenfeiz0326 merged commit aa4276d into NVIDIA:main Jun 3, 2026
7 checks passed

coderabbitai Bot mentioned this pull request Jun 9, 2026

[https://nvbugs/6108994][fix] add kv_transfer_timeout_ms to avoid timeout #15152

Merged

1 task

Uh oh!

Conversation

chenfeiz0326 commented Jun 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jun 2, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

chenfeiz0326 commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

chenfeiz0326 commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

chenfeiz0326 commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chenfeiz0326 commented Jun 2, 2026 •

edited by coderabbitai Bot

Loading