[https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D by xxi-nv · Pull Request #16621 · NVIDIA/TensorRT-LLM

xxi-nv · 2026-07-20T11:13:30Z

Summary

Unwaive TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] on RTXPro6000D, tracked by NVBug 5948435.

Why

The test was waived with the generic symptom "Test terminated unexpectedly" (a crash, not an accuracy failure). Investigation shows it is not reproducible on a correctly-built binary:

Local repro on RTX Pro 6000 Blackwell (SM120) × 4, clean builds from source:
- single-arch -a 120-real → PASS (GSM8K accuracy 63.99, ref 63.71)
- CI's multi-arch -a '90-real;100-real;103-real;120-real' → PASS (GSM8K accuracy 63.50)
- The multi-arch build contains the full SM120 kernel set (381 sm_120 SASS kernels), including the FP8 MLA-generation FMHA kernel — identical to the single-arch build, so multi-arch compilation does not drop SM120 kernels.
Repair Bot (2026-07-17): could not reproduce; all attempts passed on current main.
OpenSearch: no failures for this test in the last 90+ days.

The one crash I initially observed was traced to an incremental build that reused a stale cpp/build directory pinned to 100-real (B200), producing an SM120-incomplete .so — a local build artifact, not the product/CI behavior. CI builds include 120-real (jenkins/Build.groovy, jenkins/L0_Test.groovy), so the CI binary is SM120-complete.

The original 2026-03-03 failure was therefore most likely a genuine transient/flaky event (or has since been fixed). Removing the waive so post-merge CI can confirm on the real CI binary.

Test Coverage

Removing a waive; the test itself re-enters CI. Post-merge stage: RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1.

PR Checklist

PR title follows [JIRA/NVBUG][type] description
DCO sign-off included
pre-commit run passes (incl. waives.txt duplication + AST test-list validation)
One concern per PR (only removes the 5948435 waive line; sibling 5961814 entries left untouched)

Summary by CodeRabbit

Tests
- Updated tests/integration/test_lists/waives.txt to remove the nvbugs/5948435 SKIP waiver for full:RTXPro6000D/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus (CUTLASS, EP4, FP8KV, torch_compile=False parameterization).
- Left the corresponding torch_compile=True waiver in place (still pointing at nvbugs/5961814), so the configuration can resume running in RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1.

Dev Engineer Review

Correctness/consistency
- Verified the specific nvbugs/5948435 entry is no longer present in tests/integration/test_lists/waives.txt.
- Kept the intended alternate waiver coverage for the same test/config under torch_compile=True (nvbugs/5961814), without broadening waiver scope to unrelated parameterizations.
Config hygiene
- Change is limited to the single waiver entry in the test list (no public/API/config surface changes).

QA Engineer Review

Files touched
- tests/integration/test_lists/waives.txt
  - Removed: full:RTXPro6000D/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[...]torch_compile=False SKIP entry referencing nvbugs/5948435.
  - Retained: the torch_compile=True SKIP entry for the same RTXPro6000D test/config (referencing nvbugs/5961814).
Verdict
- needs follow-up

…us CUTLASS ep4 fp8kv on RTXPro6000D The test was waived as flaky ('Test terminated unexpectedly'). Local repro on RTX Pro 6000 Blackwell (SM120) x4 with a correctly-built binary (both single-arch '120-real' and CI's multi-arch '90-real;100-real;103-real;120-real' clean builds) passes deterministically (GSM8K accuracy ~63.5-64.0, matching reference 63.71). The SM120 FP8 MLA generation FMHA kernel is present in a correct build (381 sm_120 SASS kernels). Repair Bot (2026-07-17) and OpenSearch (90+ days) also show no reproduction. Removing the waive so post-merge CI can confirm. Signed-off-by: XingFei Xi <xxi@nvidia.com>

xxi-nv · 2026-07-20T11:13:47Z

/bot run --add-multi-gpu-test --stage-list "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

coderabbitai · 2026-07-20T11:15:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 98d3236d-6c25-4d20-9175-4cabd2b2ded5

📥 Commits

Reviewing files that changed from the base of the PR and between 513dc2e and 9a82429.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

Walkthrough

Changes

DeepSeekV3Lite integration waiver

Layer / File(s)	Summary
Update `torch_compile` waiver `tests/integration/test_lists/waives.txt`	Removes the `torch_compile=False` waiver while retaining the matching `torch_compile=True` waiver and its NVBug reference.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#7698: Modifies related DeepSeekV3Lite nvfp4 waivers in the same file.
NVIDIA/TensorRT-LLM#16398: Adjusts overlapping DeepSeekV3Lite torch_compile waiver entries.
NVIDIA/TensorRT-LLM#16585: Updates related test_nvfp4 waiver parameterizations.

Suggested reviewers: schetlur-nv, brnguyen2, yuanjingx87

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR's main change and uses the required ticket/type format.
Description check	✅ Passed	The description includes the summary, rationale, test coverage, and checklist items requested by the template.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-20T11:20:06Z

PR_Github #60360 [ run ] triggered by Bot. Commit: 513dc2e Link to invocation

tensorrt-cicd · 2026-07-20T11:35:16Z

PR_Github #60360 [ run ] completed with state FAILURE. Commit: 513dc2e
/LLM/main/L0_MergeRequest_PR pipeline #48702 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-20T23:50:16Z

/bot run --extra-stage "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-20T23:56:07Z

PR_Github #60502 [ run ] triggered by Bot. Commit: 513dc2e Link to invocation

tensorrt-cicd · 2026-07-21T00:10:22Z

PR_Github #60502 [ run ] completed with state FAILURE. Commit: 513dc2e
/LLM/main/L0_MergeRequest_PR pipeline #48827 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-21T05:51:24Z

/bot run --extra-stage "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-21T05:57:10Z

PR_Github #60601 [ run ] triggered by Bot. Commit: 513dc2e Link to invocation

tensorrt-cicd · 2026-07-21T06:12:32Z

PR_Github #60601 [ run ] completed with state FAILURE. Commit: 513dc2e
/LLM/main/L0_MergeRequest_PR pipeline #48909 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-21T10:35:12Z

/bot run --extra-stage "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-21T10:41:17Z

PR_Github #60676 [ run ] triggered by Bot. Commit: 231ad53 Link to invocation

tensorrt-cicd · 2026-07-21T12:56:32Z

PR_Github #60676 [ run ] completed with state FAILURE. Commit: 231ad53
/LLM/main/L0_MergeRequest_PR pipeline #48971 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-22T03:25:52Z

/bot run --extra-stage "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-22T03:33:26Z

PR_Github #60869 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-22T05:50:22Z

PR_Github #60869 [ run ] completed with state SUCCESS. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49142 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-22T07:16:01Z

/bot run --disable-fail-fast --extra-stage "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-22T07:26:14Z

PR_Github #60914 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-22T10:51:26Z

PR_Github #60914 [ run ] completed with state FAILURE. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49182 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-07-23T00:25:28Z

/bot run --stage-list "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-23T00:34:45Z

PR_Github #61129 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-23T15:45:20Z

PR_Github #61129 [ run ] completed with state SUCCESS. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49383 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

xxi-nv · 2026-07-23T23:48:24Z

/bot run --stage-list "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2"

xxi-nv · 2026-07-24T13:02:51Z

/bot run --stage-list "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2"

tensorrt-cicd · 2026-07-24T13:08:28Z

PR_Github #61576 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-24T15:53:05Z

PR_Github #61576 [ run ] completed with state SUCCESS. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49787 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

xxi-nv · 2026-07-24T23:55:17Z

/bot run --stage-list "RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2" --disable-fail-fast

tensorrt-cicd · 2026-07-25T00:00:47Z

PR_Github #61664 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-25T00:28:40Z

PR_Github #61664 [ run ] completed with state SUCCESS. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49869 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

xxi-nv · 2026-07-25T00:47:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-25T00:53:46Z

PR_Github #61668 [ run ] triggered by Bot. Commit: 9a82429 Link to invocation

tensorrt-cicd · 2026-07-25T01:38:07Z

PR_Github #61668 [ run ] completed with state SUCCESS. Commit: 9a82429
/LLM/main/L0_MergeRequest_PR pipeline #49874 completed with status: 'SUCCESS'

CI Report

Link to invocation

…us CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) Signed-off-by: XingFei Xi <xxi@nvidia.com>

…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (54 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

…nnahz/dep-1082-shared-mnnvl-moe-lifecycle * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (142 commits) [NVIDIA#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 (NVIDIA#12705) [None][test] Adjust timeout cases in QA perf test (NVIDIA#16894) [https://nvbugs/6157892][fix] Mistral format refactor (NVIDIA#15123) [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe (NVIDIA#16830) [https://nvbugs/6501376][fix] Test-only fix — drop the `if hidden_size % 2 != 0: with pytest.raises(...)`… (NVIDIA#16844) [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support (NVIDIA#16355) [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert `bfloat16` under… (NVIDIA#16203) [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft (NVIDIA#16831) [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test (NVIDIA#16878) [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests (NVIDIA#16838) [None][infra] Waive 21 failed cases for main in post-merge 2862 (NVIDIA#16882) [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host (NVIDIA#16791) [None][perf] Optimize Blackwell fused MHC half-MMA kernel (NVIDIA#16799) [None][infra] Auto-update test durations from OpenSearch (last 7 days) [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path (NVIDIA#16789) [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks (NVIDIA#16774) [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer (NVIDIA#16505) [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue (NVIDIA#16775) [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats (NVIDIA#16571) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

xxi-nv requested review from a team as code owners July 20, 2026 11:13

xxi-nv requested review from brnguyen2, schetlur-nv, yiqingy0, yuanjingx87 and yufeiwu-nv July 20, 2026 11:13

github-actions Bot assigned xxi-nv Jul 20, 2026

coderabbitai Bot mentioned this pull request Jul 20, 2026

[None][test] Waive 1 failed cases for main in QA CI #16622

Merged

coderabbitai Bot mentioned this pull request Jul 20, 2026

[None][test] Waive hang issues #16625

Merged

1 task

QiJune approved these changes Jul 21, 2026

View reviewed changes

yufeiwu-nv removed their request for review July 21, 2026 02:57

yiqingy0 approved these changes Jul 21, 2026

View reviewed changes

Merge branch 'main' into unwaive-5948435-dsv3lite-nvfp4-cutlass

231ad53

Merge branch 'main' into unwaive-5948435-dsv3lite-nvfp4-cutlass

9a82429

xxi-nv merged commit b8e4594 into NVIDIA:main Jul 25, 2026
10 checks passed

xxi-nv deleted the unwaive-5948435-dsv3lite-nvfp4-cutlass branch July 25, 2026 01:42

yuanjingx87 pushed a commit to yuanjingx87/TensorRT-LLM that referenced this pull request Jul 26, 2026

[https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gp…

4a613c8

…us CUTLASS ep4 fp8kv on RTXPro6000D (NVIDIA#16621) Signed-off-by: XingFei Xi <xxi@nvidia.com>

Uh oh!

Conversation

xxi-nv commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Test Coverage

PR Checklist

Summary by CodeRabbit

Dev Engineer Review

QA Engineer Review

Uh oh!

xxi-nv commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

xxi-nv commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

xxi-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

xxi-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

xxi-nv commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

xxi-nv commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

xxi-nv commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

xxi-nv commented Jul 23, 2026

Uh oh!

xxi-nv commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 24, 2026

Uh oh!

xxi-nv commented Jul 24, 2026

Uh oh!

tensorrt-cicd commented Jul 25, 2026

Uh oh!

tensorrt-cicd commented Jul 25, 2026

Uh oh!

xxi-nv commented Jul 25, 2026

Uh oh!

tensorrt-cicd commented Jul 25, 2026

Uh oh!

tensorrt-cicd commented Jul 25, 2026

Uh oh!

Uh oh!

xxi-nv commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading