[https://nvbugs/6427411][chore] Waive additional PP sampler regressions - #16146
Conversation
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot skip --comment "test waiving to unblock CI" |
|
PR_Github #58304 [ skip ] triggered by Bot. Commit: |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
|
/bot skip --comment "test waiving to unblock CI" |
|
PR_Github #58304 [ skip ] completed with state |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR adds new SKIP waiver entries to the integration test waives list, covering additional ChangesTest waivers update
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #58306 [ skip ] triggered by Bot. Commit: |
|
PR_Github #58306 [ skip ] completed with state |
…ns (NVIDIA#16146) Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109 so the restored pre-NVIDIA#15920 sampler path is exercised in CI. Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109, plus the PP2 waiver added by NVIDIA#16169, so the restored pre-NVIDIA#15920 sampler path is exercised in CI. Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109, plus the PP2 waiver added by NVIDIA#16169, so the restored pre-NVIDIA#15920 sampler path is exercised in CI. Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
Summary
Temporarily waive five exact pipeline-parallel test variants affected by NVBug 6427411:
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False]accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False]accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False]accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False]unittest/llmapi/test_llm_multi_gpu_pytorch.py::test_llm_get_stats_pp4[False-True-True]Only these test IDs are waived; their broader GPU4 suites remain enabled.
Background
These cases surfaced or were identified after the earlier waivers in #16127 landed. A failing PP test can terminate its pytest/MPI process before later parameterizations run or write JUnit results, masking additional affected cases.
The observed failures share the pipeline-parallel sampler regression introduced by #15920. Confirmed cases fail in
finish_if_reasonwithIndexError: list index out of range, and the remaining queued variant exercises the same PP sampler path. The corrective change is tracked by #16096.This PR contains no runtime changes. These waivers can be removed after #16096 or an equivalent fix lands.
Validation
python3 scripts/check_test_list.py --check-duplicate-waivespython3 scripts/check_test_list.py --validatepre-commit run --files tests/integration/test_lists/waives.txtSummary by CodeRabbit