[https://nvbugs/6342840][infra] Un-waive 37 mtp perf-sanity cases (18 fixed by #15797 + 19 for CI recheck) by chenfeiz0326 · Pull Request #15827 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-07-01T10:01:43Z

Summary

Un-waives 46 mtp1/mtp3 test lines from tests/integration/test_lists/waives.txt, split across two commits:

18 lines fixed by PR [https://nvbugs/6342840][fix] Add spec_metadata=None kwarg to SpecWorkerBase._apply_force_accepted_tokens… #15797 — the MTP + TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS / MTP + CUDA-graph-capture IMA root cause is now resolved. Verified on OCI GB200.
28 lines with unrelated root causes — waives for various MTP=1/MTP=3 configs that have accumulated across many independent bugs (OOM, pytest timeouts, disagg NIXL hangs, config-schema drift, GLM-5 dtype, etc.). Pre-merge CI on this branch is the authoritative signal for which cases still fail.

Commit 1: 18 lines fixed by PR #15797

Stale — bug is already Dev - Closed - Verified:

nvbugs/6280649 (1): disagg_upload-e2e-gb200_deepseek-v32-fp4_1k1k_..._mtp1_ccb-NIXL
nvbugs/6302880 (1): disagg_upload-e2e-gb200_qwen3-235b-fp4_8k1k_..._mtp0_ccb-NIXL

Re-tracked into 6342840 per its own comment 9 (fixed by PR #15797):

nvbugs/6323889 (7): v32 tep4/dep4 mtp1/mtp3 aggr + r1 disagg gen_only mtp0/1/3

Same MTP + FORCE_NUM_ACCEPTED_TOKENS / MTP + CUDA-graph-capture IMA root cause as PR #15797:

nvbugs/6280721 (5): v32 blackwell tep8/dep8 mtp1/mtp3, grace_blackwell tep4 mtp3 8k1k, disagg tep8 mtp3
nvbugs/6374898 (2): v32 32k4k tep8/dep32 mtp3 disagg IMA in cuda_graph_runner
nvbugs/6388522 (2): glm5 tep4/tep8 mtp3 grace_blackwell aggr

Verification (OCI GB200, source-built wheel from PR #15797 HEAD):

Case	MTP	FORCE	Elapsed	Requests	Throughput
`v32_fp4_tep4_mtp3_1k1k`	3	=3	19m35s	20/20	596.87 tok/s
`v32_fp4_dep4_mtp1_1k1k`	1	=1	32m11s	10240/10240	15665.33 tok/s
`r1_fp4_v2_tep4_mtp3_1k1k`	3	=3	24m07s	320/320	4842.30 tok/s
`glm5_fp4_tep4_mtp3_1k1k`	3	=3	21m33s	20/20	294.54 tok/s

Log grep across all 4 runs: 0 hits for illegal memory access, CUDA error, CUDA assert, Traceback, Segmentation, core dumped.

Commit 2: 28 additional mtp1/mtp3 lines (unrelated root causes)

Removed for CI recheck — pre-merge L0 matrix is the authoritative signal:

Accuracy / unit-test waives (7 lines):

nvbugs/6384357 (2): V32 dsa host_cache_offload mtp1/mtp3
nvbugs/5989920 (1): V32 nvfp4 piecewise cuda_graph mtp3
nvbugs/6367805 (3): Step3.7 fp8_block_scales / nvfp4 mtp_nextn=3
nvbugs/6344108 (3): Nemotron3 Super120B disagg auto_dtype mtp_nextn=3 (A100/H100/H20)

Perf-sanity, aggr (5 lines):

nvbugs/6388205 (1): b200 r1 mtp3 disagg
nvbugs/6374939 (1): r1_fp4_v2 grace_blackwell dep4 mtp1 1k8k (pytest outer timeout)
nvbugs/6374910 (1): v32 grace_blackwell dep4 mtp1 1k1k (aggr trtllm-serve OOM residual)
nvbugs/6324131 (1): glm5 2_nodes_grace_blackwell dep8 mtp1
nvbugs/6329155 (2): glm5 blackwell dep8 mtp1 + tep8 mtp3

Perf-sanity, disagg gb200 (7 lines):

nvbugs/6374872 (2): r1 128k8k mtp1 + v32 32k4k mtp1 (gen_kv_cache_insufficient)
nvbugs/6302903 (4): r1 disagg mtp1/mtp3 (various concurrency+shape combos)
nvbugs/6374905 (1): v32 mtp1 (attn workspace pointer baked in graph)

Perf-sanity, disagg gb300 (9 lines):

nvbugs/6324131 (5): glm5 gb300 mtp1/mtp3
nvbugs/6323074 (1): kimi-k25 gb300 mtp3

What is NOT touched

Waives with mtp0 or with non-MTP root causes stay in place (super_ad regression, GLM-5 non-MTP failures, Kimi non-MTP, NIXL non-MTP hangs, ...).

Rollback plan

If any un-waived case fails pre-merge CI, restore its specific line with the original nvbugs/* reference before merging.

Links

🤖 Generated with Claude Code

coderabbitai · 2026-07-01T10:03:14Z

📝 Walkthrough

Walkthrough

This PR updates waiver entries in tests/integration/test_lists/waives.txt for perf/test_perf_sanity.py::test_e2e. Existing SKIP waivers for various aggr_upload and disagg_upload configurations are replaced with new/changed parameter combinations and updated NVBUG references.

Changes

Waiver List Updates

Layer / File(s)	Summary
Aggr_upload waiver updates `tests/integration/test_lists/waives.txt`	SKIP waivers updated for aggr_upload DeepSeek v32 fp4 grace/blackwell, glm5 fp4, gpt_oss_120b_fp4, and super_ad configurations with new NVBUG references.
Disagg_upload e2e waiver updates `tests/integration/test_lists/waives.txt`	SKIP waivers updated for disagg_upload e2e scenarios on gb200 (DeepSeek R1/v32, kimi-k25-thinking) and gb300 (glm-5, kimi-k25-thinking).
Disagg_upload-gen_only waiver updates `tests/integration/test_lists/waives.txt`	SKIP waivers updated for disagg_upload-gen_only scenarios on gb200 covering DeepSeek R1/v32 fp4, gpt-oss-120b fp4, and kimi-k25-thinking fp4 variants.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#15779: Also modifies tests/integration/test_lists/waives.txt with added/removed SKIP waiver entries in the same file.

Suggested reviewers: xinhe-nv, jieli-matrix, LarryXFly, StanleySun639

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description is detailed and covers the issue, fix breakdown, verification, rollback, and links, though it doesn't follow the template exactly.
Title check	✅ Passed	The title clearly reflects the main change: un-waiving MTP perf-sanity cases and notes the split between fixed cases and CI recheck cases.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-01T14:04:39Z

PR_Github #56942 [ run ] triggered by Bot. Commit: 3b529bc Link to invocation

tensorrt-cicd · 2026-07-01T14:06:04Z

PR_Github #56943 [ run ] triggered by Bot. Commit: 3b529bc Link to invocation

Un-waives 37 lines in tests/integration/test_lists/waives.txt scoped to perf/test_perf_sanity.py test cases only. Split by root-cause bucket: Fixed by PR NVIDIA#15797 (MTP + TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS / MTP + CUDA-graph-capture IMA): - nvbugs/6280649 (1): stale, bug is Dev - Closed - Verified - nvbugs/6302880 (1): stale, bug is Dev - Closed - Verified - nvbugs/6323889 (7): re-tracked into 6342840 per its own comment 9 - nvbugs/6280721 (5): same IMA root cause - nvbugs/6374898 (2): same IMA in cuda_graph_runner.py - nvbugs/6388522 (2): same IMA class Unrelated root causes, removed for CI recheck (pre-merge CI is authoritative): - nvbugs/6302903 (4): r1 disagg mtp1/mtp3 failures - nvbugs/6323074 (1): kimi-k25 gb300 mtp3 disagg - nvbugs/6324131 (5): glm5 disagg gb200/gb300 mtp1/mtp3 - nvbugs/6329155 (2): glm5 blackwell 2-node mtp1/mtp3 - nvbugs/6374872 (2): gen_kv_cache_insufficient r1/v32 mtp1 - nvbugs/6374905 (1): v32 mtp1 attn workspace pointer baked in graph - nvbugs/6374910 (1): aggr trtllm-serve OOM residual GPU mem - nvbugs/6374939 (1): pytest outer timeout, no crash signal - nvbugs/6388205 (1): b200 disagg mtp3 Independent GB200 verification of PR NVIDIA#15797 on 4 representative cases (v32 tep4/dep4 mtp1/mtp3, r1 tep4 mtp3, glm5 tep4 mtp3): all pass, 0 IMA in worker logs. Rollback plan: if any un-waived case fails pre-merge CI, restore its specific line with the original nvbugs/* reference before merging. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-01T14:12:39Z

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-*,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-*,DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge-*,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE2-GPU8-Post-Merge-2,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-4,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-5,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-6,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE4-GPU16-Post-Merge-2,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-5,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-6,GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-5,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-7,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-8,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-1,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-2,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-3,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-4,GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU2-GEN1-NODE2-GPU8-Post-Merge-5,GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1"

tensorrt-cicd · 2026-07-01T14:18:31Z

PR_Github #56945 [ run ] triggered by Bot. Commit: 8335386 Link to invocation

tensorrt-cicd · 2026-07-01T14:22:12Z

PR_Github #56943 [ run ] completed with state ABORTED. Commit: 3b529bc

Link to invocation

tensorrt-cicd · 2026-07-01T17:48:19Z

PR_Github #56945 [ run ] completed with state FAILURE. Commit: 8335386
/LLM/main/L0_MergeRequest_PR pipeline #45749 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-01T23:02:57Z

At least one of the tests hung:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-NIXL]
https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-3/blue/rest/organizations/jenkins/pipelines/LLM/pipelines/main/pipelines/L0_Test-SBSA-Multi-GPU/runs/2009/nodes/7034/steps/8048/log/?start=0

…ing/not-run in pre-merge 45749 Pre-merge build 45749 on PR NVIDIA#15827 ran 115 perf_sanity cases: - 19 of the 37 unwaived: PASSED (keep unwaived) - 4 unwaived: FAILED (real regressions/errors) - 14 unwaived: not present in build results Re-waive the 18 non-passing so the tree remains green while the 19 confirmed-fixed cases stay unwaived.

chenfeiz0326 · 2026-07-02T02:50:37Z

At least one of the tests hung:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-NIXL]

https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-3/blue/rest/organizations/jenkins/pipelines/LLM/pipelines/main/pipelines/L0_Test-SBSA-Multi-GPU/runs/2009/nodes/7034/steps/8048/log/?start=0

I will re-waive failed tests and only unwaive passed tests.

…ing/not-run in pre-merge 45749 Pre-merge build 45749 on PR NVIDIA#15827 ran 115 perf_sanity cases: - 19 of the 37 unwaived: PASSED (keep unwaived) - 4 unwaived: FAILED (real regressions/errors) - 14 unwaived: not present in build results Re-waive the 18 non-passing so the tree remains green while the 19 confirmed-fixed cases stay unwaived. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

chenfeiz0326 · 2026-07-02T02:56:35Z

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

tensorrt-cicd · 2026-07-02T03:07:06Z

PR_Github #57084 [ skip ] triggered by Bot. Commit: d6c44a7 Link to invocation

tensorrt-cicd · 2026-07-02T03:13:49Z

PR_Github #57084 [ skip ] completed with state SUCCESS. Commit: d6c44a7
Skipping testing for commit d6c44a7

Link to invocation

… fixed by NVIDIA#15797 + 19 for CI recheck) (NVIDIA#15827) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

github-actions Bot assigned chenfeiz0326 Jul 1, 2026

chenfeiz0326 mentioned this pull request Jul 1, 2026

[None][infra] Un-waive 28 remaining mtp1/mtp3 tests for CI recheck #15836

Closed

chenfeiz0326 changed the title ~~[https://nvbugs/6342840][infra] Un-waive 18 MTP perf-sanity cases fixed by #15797~~ [https://nvbugs/6342840][infra] Un-waive 46 mtp1/mtp3 tests (18 fixed by #15797 + 28 for CI recheck) Jul 1, 2026

chenfeiz0326 force-pushed the unwaive-mtp-post-6342840 branch from 593c4af to 3b529bc Compare July 1, 2026 14:03

chenfeiz0326 force-pushed the unwaive-mtp-post-6342840 branch from 3b529bc to 8335386 Compare July 1, 2026 14:07

chenfeiz0326 changed the title ~~[https://nvbugs/6342840][infra] Un-waive 46 mtp1/mtp3 tests (18 fixed by #15797 + 28 for CI recheck)~~ [https://nvbugs/6342840][infra] Un-waive 37 mtp perf-sanity cases (18 fixed by #15797 + 19 for CI recheck) Jul 1, 2026

chenfeiz0326 requested a review from ZhanruiSunCh July 2, 2026 02:51

chenfeiz0326 force-pushed the unwaive-mtp-post-6342840 branch from c9f9aeb to d6c44a7 Compare July 2, 2026 02:51

chenfeiz0326 requested a review from mzweilz July 2, 2026 02:51

mzweilz approved these changes Jul 2, 2026

View reviewed changes

chenfeiz0326 enabled auto-merge (squash) July 2, 2026 02:56

chenfeiz0326 merged commit db2c8a0 into NVIDIA:main Jul 2, 2026
8 checks passed

evezhier pushed a commit to evezhier/TensorRT-LLM that referenced this pull request Jul 2, 2026

[https://nvbugs/6342840][infra] Un-waive 37 mtp perf-sanity cases (18…

0530675

… fixed by NVIDIA#15797 + 19 for CI recheck) (NVIDIA#15827) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

xwang233 mentioned this pull request Jul 6, 2026

[https://nvbugs/6302903][infra] Unwaive deepseek-r1-fp4 disagg perf-sanity e2e test #15787

Closed

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[https://nvbugs/6342840][infra] Un-waive 37 mtp perf-sanity cases (18…

2d659c1

… fixed by NVIDIA#15797 + 19 for CI recheck) (NVIDIA#15827) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>

Uh oh!

Conversation

chenfeiz0326 commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commit 1: 18 lines fixed by PR #15797

Commit 2: 28 additional mtp1/mtp3 lines (unrelated root causes)

What is NOT touched

Rollback plan

Links

Uh oh!

coderabbitai Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

chenfeiz0326 commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tensorrt-cicd commented Jul 1, 2026

Uh oh!

tburt-nv commented Jul 1, 2026

Uh oh!

chenfeiz0326 commented Jul 2, 2026

Uh oh!

chenfeiz0326 commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

tensorrt-cicd commented Jul 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chenfeiz0326 commented Jul 1, 2026 •

edited

Loading

coderabbitai Bot commented Jul 1, 2026 •

edited

Loading