Skip to content

[https://nvbugs/6427411][chore] Waive additional PP sampler regressions - #16146

Merged
chienchunhung merged 2 commits into
NVIDIA:mainfrom
chienchunhung:codex/waive-additional-pp-sampler-regressions
Jul 8, 2026
Merged

[https://nvbugs/6427411][chore] Waive additional PP sampler regressions#16146
chienchunhung merged 2 commits into
NVIDIA:mainfrom
chienchunhung:codex/waive-additional-pp-sampler-regressions

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Temporarily waive five exact pipeline-parallel test variants affected by NVBug 6427411:

  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False]
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False]
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp2pp2-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False]
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False]
  • unittest/llmapi/test_llm_multi_gpu_pytorch.py::test_llm_get_stats_pp4[False-True-True]

Only these test IDs are waived; their broader GPU4 suites remain enabled.

Background

These cases surfaced or were identified after the earlier waivers in #16127 landed. A failing PP test can terminate its pytest/MPI process before later parameterizations run or write JUnit results, masking additional affected cases.

The observed failures share the pipeline-parallel sampler regression introduced by #15920. Confirmed cases fail in finish_if_reason with IndexError: list index out of range, and the remaining queued variant exercises the same PP sampler path. The corrective change is tracked by #16096.

This PR contains no runtime changes. These waivers can be removed after #16096 or an equivalent fix lands.

Validation

  • python3 scripts/check_test_list.py --check-duplicate-waives
  • python3 scripts/check_test_list.py --validate
  • pre-commit run --files tests/integration/test_lists/waives.txt

Summary by CodeRabbit

  • Bug Fixes
    • Updated automated test exclusions to better account for known multi-GPU coverage gaps.
    • Added additional safeguards for specific distributed inference and statistics scenarios, helping keep validation runs stable.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "test waiving to unblock CI"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58304 [ skip ] triggered by Bot. Commit: a5ef292 Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "test waiving to unblock CI"

@chienchunhung
chienchunhung marked this pull request as ready for review July 8, 2026 22:04
@chienchunhung
chienchunhung enabled auto-merge (squash) July 8, 2026 22:06
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58304 [ skip ] completed with state SUCCESS. Commit: a5ef292
Skipping testing for commit a5ef292

Link to invocation

@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9d828680-ce47-4ebb-8cf2-09befadd73d0

📥 Commits

Reviewing files that changed from the base of the PR and between 5ce293f and b2c2831.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR adds new SKIP waiver entries to the integration test waives list, covering additional TestDeepSeekV3Lite accuracy test configurations for test_fp8_block_scales_4gpus and test_nvfp4_4gpus, plus a new waiver for test_llm_get_stats_pp4 in the unittest multi-GPU pytorch tests.

Changes

Test waivers update

Layer / File(s) Summary
Add new SKIP waiver entries
tests/integration/test_lists/waives.txt
Adds three new test_fp8_block_scales_4gpus SKIP entries and one test_nvfp4_4gpus SKIP entry for TestDeepSeekV3Lite, plus a test_llm_get_stats_pp4[False-True-True] SKIP entry.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#15774: Both PRs extend SKIP waivers in waives.txt for TestDeepSeekV3Lite fp8kv=True cases including test_nvfp4.
  • NVIDIA/TensorRT-LLM#16127: Both PRs add related SKIP waivers for TestDeepSeekV3Lite::test_fp8_block_scales_4gpus and test_llm_get_stats_pp4.
  • NVIDIA/TensorRT-LLM#16103: Both PRs adjust SKIP waivers for the same class of LLM API multi-GPU test cases including DeepSeekV3Lite parameters.

Suggested reviewers: shuyixiong, mingyangHao, brb-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately reflects the PR's temporary test-waiver change.
Description check ✅ Passed The description covers summary, background, and validation, and is sufficiently complete despite missing explicit Test Coverage and PR Checklist sections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58306 [ skip ] triggered by Bot. Commit: b2c2831 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58306 [ skip ] completed with state SUCCESS. Commit: b2c2831
Skipping testing for commit b2c2831

Link to invocation

@chienchunhung
chienchunhung merged commit 4cd00bb into NVIDIA:main Jul 8, 2026
15 checks passed
BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026
…ns (NVIDIA#16146)

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
mingyangHao added a commit to mingyangHao/TensorRT-LLM that referenced this pull request Jul 9, 2026
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109 so the restored pre-NVIDIA#15920 sampler path is exercised in CI.

Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
mingyangHao added a commit to mingyangHao/TensorRT-LLM that referenced this pull request Jul 9, 2026
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109, plus the PP2 waiver added by NVIDIA#16169, so the restored pre-NVIDIA#15920 sampler path is exercised in CI.

Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
mingyangHao added a commit to mingyangHao/TensorRT-LLM that referenced this pull request Jul 9, 2026
Remove the PP sampler waivers introduced by NVIDIA#16103, NVIDIA#16105, NVIDIA#16127, and NVIDIA#16146. Also remove overlapping PP and gpu2 waivers from NVIDIA#16109, plus the PP2 waiver added by NVIDIA#16169, so the restored pre-NVIDIA#15920 sampler path is exercised in CI.

Signed-off-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants