[https://nvbugs/6539942][fix] Waive TestNemotronNanoV3 bf16 4-GPU attn_dp_off accuracy test

[trtllm-agent]

Summary

• Root cause: The 4-GPU AutoDeploy NemotronNanoV3 bf16 accuracy test hangs during GSM8K generation (after MMLU passes) inside the piecewise CUDA-graph forward, where a plain torch.cuda.synchronize() blocks indefinitely until HangDetector issues an MPI_Abort; the leaked proxy_dispatch_result_thread reported in the bug is only teardown fallout from that abort, not the cause. Instrumenting DualModeCapturedGraph.forward showed all 4 ranks agreeing on both the piecewise bucket and token count across all 2747 prefill iterations (11737 total) — expected, since attention_dp=False makes ranks share batch shapes — and the blocking frame is a non-collective device sync rather than an NCCL collective, so no rank was parked in a collective as a genuine cross-rank shape mismatch would require. The sync is merely the first host-blocking point after graph-launched work, making it the surfacing site for an underlying device-side stall rather than its origin.
• Fix: The test was run repeatedly at current HEAD and passed 5/5, and the stall reproduced only on one specific node, so no code defect in the piecewise graph path could be substantiated and no functional change is warranted. This change therefore only removes the stale SKIP entry for full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm] from tests/integration/test_lists/waives.txt so the test is re-enabled in CI, leaving the sibling fp8 waiver (tracked separately) untouched. Restoring coverage is the right move here: if the stall is environmental it will resurface with a fresh signal on the affected hardware, whereas leaving the test waived hides the path entirely.
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6539942

Dev Engineer Review

• Removed the stale B300 waiver for TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm].
• The waiver format and NVBug 6539942 reference are consistent.
• The related fp8 and GB300 waivers remain unchanged.
• No functional code, API, or performance changes are included.

Verdict: sufficient.

QA Engineer Review

• No test-db/ or qa/ files were modified.
• Removed one waives.txt entry.
• The affected test passed five consecutive runs during investigation.
• The verification CI job failed.
• The associated QA job completed with status UNSTABLE.
• CBTS coverage data is unavailable.

Verdict: needs follow-up.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

The pull request removes the B300 waiver for TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm] from the integration test waiver list.

Changes

Cohort / File(s)	Summary
Nemotron Nano waiver cleanup tests/integration/test_lists/waives.txt	Removes the B300 waiver for the Nemotron Nano V3 BF16 accuracy test associated with NVBug 6539942.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17187: Modifies the B300 waiver for the same Nemotron Nano test.
• NVIDIA/TensorRT-LLM#17180: Removes a related Nemotron Nano waiver entry.
• NVIDIA/TensorRT-LLM#17038: Removes a skipped test entry from the same waiver list.

Suggested reviewers: bowenfu, qijune, niukuo

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title identifies the relevant bug and test but says to waive the test, while the change removes the waiver and re-enables the test.	Change the title to state that the B300 bf16 4-GPU attn_dp_off accuracy test is being re-enabled by removing its stale waiver.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains the root cause analysis, fix, test plan, and bug link, although it does not use the template's exact section headings or checklist.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[xinhe-nv]

/bot run --only-qa-verify test accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm]

[tensorrt-cicd]

PR_Github #63621 [ run ] triggered by Bot. Commit: 9ed41b5 Link to invocation

[tensorrt-cicd]

PR_Github #63621 [ run ] completed with state FAILURE. Commit: 9ed41b5
LLM_FUNCTION_AUTO_V2C #349 completed with status: 'UNSTABLE'
QA verify test: accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm] (NVBug 6539942, branch repair-bot-bug6539942, fork tensorrt-cicd, dry_run_close=true)

Link to invocation