[None][infra] Un-waive 28 remaining mtp1/mtp3 tests for CI recheck - #15836
[None][infra] Un-waive 28 remaining mtp1/mtp3 tests for CI recheck#15836chenfeiz0326 wants to merge 1 commit into
Conversation
Removes 28 waives that reference MTP=1 or MTP=3 configurations. Companion to PR NVIDIA#15827, which un-waived 18 lines fixed by PR NVIDIA#15797. This PR covers the remaining mtp1/mtp3 waives whose underlying bugs have other root causes; pre-merge CI on this branch is the authoritative signal for whether each case still fails. Rationale: waives.txt entries for MTP configs have accumulated across many independent bugs (OOM, timeouts, disagg hangs, config-schema drift, etc.). Running the full test set now catches which ones the codebase has already fixed as a side effect. Bugs referenced (13 distinct): - nvbugs/5989920, 6367805 (accuracy: V32 nvfp4 piecewise cuda_graph, Step3.7 fp8/nvfp4) - nvbugs/6344108, 6384357 (accuracy: Nemotron3 disagg, V32 dsa host_cache_offload) - nvbugs/6302903, 6323074 (perf: R1/Kimi disagg failures) - nvbugs/6324131, 6329155 (perf: GLM-5 disagg + blackwell 2-node) - nvbugs/6374872, 6374893 (perf: gen_kv_cache_insufficient / ctx_kv_cache_transfer_timeout) - nvbugs/6374905 (perf: attn workspace pointer baked in graph) - nvbugs/6374910, 6374939 (perf: aggr trtllm-serve OOM residual GPU mem, pytest outer timeout) - nvbugs/6388205 (perf: b200 disagg mtp3) Rollback plan: if any un-waived case fails pre-merge CI, restore its specific line with the original bug ID reference before merging. Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
📝 Walkthrough<review_stack_artifact_context> </review_stack_artifact_context> WalkthroughThis PR updates the test waiver list (waives.txt) by removing and adding SKIP entries for accuracy tests (DeepSeekV32, TestStep3_7, TestMistralLarge3_675B) and reworking perf sanity test_e2e parameterizations across multiple configurations and platforms. ChangesWaiver list updates
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Consolidating into #15827 — bundling all 46 mtp1/mtp3 un-waives into a single PR per request. |
Summary
Un-waives 28 remaining mtp1/mtp3 test lines from
tests/integration/test_lists/waives.txt. Companion to PR #15827 which removed the 18 lines fixed by PR #15797.Rationale
Waives for MTP=1 / MTP=3 configurations have accumulated across many independent bugs (OOM, pytest timeouts, disagg NIXL hangs, GLM-5 config drift, ...). Some are stale — the codebase has fixed the underlying issue via other PRs. Running the full test set on this branch is the authoritative signal for which cases still fail.
Removed lines by root cause bucket
Accuracy / unit-test waives (7 lines):
nvbugs/6384357(2): V32 dsa host_cache_offload mtp1/mtp3nvbugs/5989920(1): V32 nvfp4 piecewise cuda_graph mtp3nvbugs/6367805(3): Step3.7 fp8_block_scales/nvfp4 mtp_nextn=3nvbugs/6344108(3): Nemotron3 Super120B disagg auto_dtype mtp_nextn=3 (A100/H100/H20)Perf-sanity waives, aggr (5 lines):
nvbugs/6388205(1): b200 r1 mtp3 disaggnvbugs/6374939(1): r1_fp4_v2 grace_blackwell dep4 mtp1 1k8k (pytest outer timeout)nvbugs/6374910(1): v32 grace_blackwell dep4 mtp1 1k1k (aggr trtllm-serve OOM residual)nvbugs/6324131(1): glm5 2_nodes_grace_blackwell dep8 mtp1nvbugs/6329155(2): glm5 blackwell dep8 mtp1 + tep8 mtp3Perf-sanity waives, disagg gb200 (7 lines):
nvbugs/6374872(2): r1 128k8k mtp1 + v32 32k4k mtp1 (gen_kv_cache_insufficient)nvbugs/6302903(4): r1 disagg mtp1/mtp3 (various concurrency+shape combos)nvbugs/6374905(1): v32 mtp1 (attn workspace pointer baked in graph)Perf-sanity waives, disagg gb300 (9 lines):
nvbugs/6324131(5): glm5 gb300 mtp1/mtp3nvbugs/6323074(1): kimi-k25 gb300 mtp3Verification
The 4-case GB200 verification of the underlying MTP fix (see PR #15827) confirms the
_apply_force_accepted_tokenscode path is now safe. For the 28 lines in this PR, pre-merge CI on this branch is the test — I will trigger it manually.Rollback plan
If any un-waived case fails pre-merge CI, restore its specific line with the original
nvbugs/*reference before merging. The failure signal points directly at which underlying bug still has an open issue.Links
🤖 Generated with Claude Code
Summary by CodeRabbit