Skip to content

[None][infra] Un-waive 28 remaining mtp1/mtp3 tests for CI recheck - #15836

Closed
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:unwaive-mtp-all-remaining
Closed

[None][infra] Un-waive 28 remaining mtp1/mtp3 tests for CI recheck#15836
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:unwaive-mtp-all-remaining

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Un-waives 28 remaining mtp1/mtp3 test lines from tests/integration/test_lists/waives.txt. Companion to PR #15827 which removed the 18 lines fixed by PR #15797.

Rationale

Waives for MTP=1 / MTP=3 configurations have accumulated across many independent bugs (OOM, pytest timeouts, disagg NIXL hangs, GLM-5 config drift, ...). Some are stale — the codebase has fixed the underlying issue via other PRs. Running the full test set on this branch is the authoritative signal for which cases still fail.

Removed lines by root cause bucket

Accuracy / unit-test waives (7 lines):

  • nvbugs/6384357 (2): V32 dsa host_cache_offload mtp1/mtp3
  • nvbugs/5989920 (1): V32 nvfp4 piecewise cuda_graph mtp3
  • nvbugs/6367805 (3): Step3.7 fp8_block_scales/nvfp4 mtp_nextn=3
  • nvbugs/6344108 (3): Nemotron3 Super120B disagg auto_dtype mtp_nextn=3 (A100/H100/H20)

Perf-sanity waives, aggr (5 lines):

  • nvbugs/6388205 (1): b200 r1 mtp3 disagg
  • nvbugs/6374939 (1): r1_fp4_v2 grace_blackwell dep4 mtp1 1k8k (pytest outer timeout)
  • nvbugs/6374910 (1): v32 grace_blackwell dep4 mtp1 1k1k (aggr trtllm-serve OOM residual)
  • nvbugs/6324131 (1): glm5 2_nodes_grace_blackwell dep8 mtp1
  • nvbugs/6329155 (2): glm5 blackwell dep8 mtp1 + tep8 mtp3

Perf-sanity waives, disagg gb200 (7 lines):

  • nvbugs/6374872 (2): r1 128k8k mtp1 + v32 32k4k mtp1 (gen_kv_cache_insufficient)
  • nvbugs/6302903 (4): r1 disagg mtp1/mtp3 (various concurrency+shape combos)
  • nvbugs/6374905 (1): v32 mtp1 (attn workspace pointer baked in graph)

Perf-sanity waives, disagg gb300 (9 lines):

  • nvbugs/6324131 (5): glm5 gb300 mtp1/mtp3
  • nvbugs/6323074 (1): kimi-k25 gb300 mtp3

Verification

The 4-case GB200 verification of the underlying MTP fix (see PR #15827) confirms the _apply_force_accepted_tokens code path is now safe. For the 28 lines in this PR, pre-merge CI on this branch is the test — I will trigger it manually.

Rollback plan

If any un-waived case fails pre-merge CI, restore its specific line with the original nvbugs/* reference before merging. The failure signal points directly at which underlying bug still has an open issue.

Links

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Updated the list of skipped test cases to reflect current expected coverage.
    • Removed several outdated waivers and added a few new ones for accuracy, performance, and multimodal scenarios.
    • Adjusted skip coverage across multiple hardware and configuration combinations, including bulk updates for end-to-end performance checks.

Removes 28 waives that reference MTP=1 or MTP=3 configurations. Companion
to PR NVIDIA#15827, which un-waived 18 lines fixed by PR NVIDIA#15797. This PR covers
the remaining mtp1/mtp3 waives whose underlying bugs have other root
causes; pre-merge CI on this branch is the authoritative signal for
whether each case still fails.

Rationale: waives.txt entries for MTP configs have accumulated across
many independent bugs (OOM, timeouts, disagg hangs, config-schema drift,
etc.). Running the full test set now catches which ones the codebase
has already fixed as a side effect.

Bugs referenced (13 distinct):
- nvbugs/5989920, 6367805        (accuracy: V32 nvfp4 piecewise cuda_graph, Step3.7 fp8/nvfp4)
- nvbugs/6344108, 6384357        (accuracy: Nemotron3 disagg, V32 dsa host_cache_offload)
- nvbugs/6302903, 6323074        (perf: R1/Kimi disagg failures)
- nvbugs/6324131, 6329155        (perf: GLM-5 disagg + blackwell 2-node)
- nvbugs/6374872, 6374893        (perf: gen_kv_cache_insufficient / ctx_kv_cache_transfer_timeout)
- nvbugs/6374905                 (perf: attn workspace pointer baked in graph)
- nvbugs/6374910, 6374939        (perf: aggr trtllm-serve OOM residual GPU mem, pytest outer timeout)
- nvbugs/6388205                 (perf: b200 disagg mtp3)

Rollback plan: if any un-waived case fails pre-merge CI, restore its
specific line with the original bug ID reference before merging.

Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 594f7079-73c1-41b5-990d-eaeac6911b6b

📥 Commits

Reviewing files that changed from the base of the PR and between 98d9d18 and 2c933a2.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

<review_stack_artifact_context>

</review_stack_artifact_context>

Walkthrough

This PR updates the test waiver list (waives.txt) by removing and adding SKIP entries for accuracy tests (DeepSeekV32, TestStep3_7, TestMistralLarge3_675B) and reworking perf sanity test_e2e parameterizations across multiple configurations and platforms.

Changes

Waiver list updates

Layer / File(s) Summary
Accuracy test waiver adjustments
tests/integration/test_lists/waives.txt
Removes DeepSeekV32 dsa_host_cache_offload and nvfp4_multi_gpus_piecewise_cuda_graph waivers, removes TestStep3_7 fp8_block_scales/nvfp4 waivers for mtp_nextn=3, and adds a TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm] waiver.
Perf sanity waiver adjustments
tests/integration/test_lists/waives.txt
Replaces a block of perf/test_perf_sanity.py::test_e2e waivers with new aggr_upload/disagg_upload/gen_only parameter combinations, and trims disagg_upload-gen_only-gb300 entries for glm-5-fp4 and kimi-k25-thinking variants.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#15778: Both PRs modify the same waiver file to add/remove specific parametrized test SKIPs for different accuracy/perf cases.
  • NVIDIA/TensorRT-LLM#15764: Both PRs modify waives.txt for accuracy/test_llm_api_pytorch.py DeepSeek FP8/NVFP4/MTP test cases.
  • NVIDIA/TensorRT-LLM#15774: Both PRs adjust accuracy waiver entries for test_llm_api_pytorch_multimodal.py parametrized test cases.

Suggested reviewers: jieli-matrix, xinhe-nv, StanleySun639

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title follows the required [None][type] format and accurately summarizes the main change: un-waiving mtp1/mtp3 tests for CI recheck.
Description check ✅ Passed The description includes summary, rationale, verification, rollback, and links; only the Test Coverage and checklist sections are not filled in.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

Consolidating into #15827 — bundling all 46 mtp1/mtp3 un-waives into a single PR per request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant