From d82e64ef8d348f939e0457889d637eabb7a0a711 Mon Sep 17 00:00:00 2001 From: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Date: Mon, 3 Aug 2026 10:31:22 -0700 Subject: [PATCH] [nvbugs/6539942][fix] Unwaive B300 NemotronNanoV3 bf16 attn_dp_off trtllm accuracy test The hang was not reproducible at HEAD on healthy B300 hardware. The test passed 5/5 consecutive runs (MMLU 74.05-74.22, GSM8K 68.46-69.75 against a 66.17 threshold). All four ranks' hang-detector stacks ended at the same plain, non-collective torch.cuda.synchronize() in PiecewiseCapturedGraph.forward, with no rank inside an NCCL collective. Instrumenting DualModeCapturedGraph.forward confirmed every rank agreed on the graph path and token bucket for all 2747 prefill iterations, ruling out cross-rank divergence in piecewise CUDA graph dispatch. That synchronize is the first host-blocking point after graph-launched work, so a device-side stall surfaces there regardless of cause. Only the B300 bf16 entry is removed; the GB200 fp8 and GB300 bf16 entries filed under the same bug ID are left in place since that hardware was not available to verify. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> --- tests/integration/test_lists/waives.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 71b5595714c3..c6a0049ca77e 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -170,7 +170,6 @@ full:B200/test_e2e.py::test_multi_nodes_eval[Qwen3/Qwen3-235B-A22B-tp16-mmlu] SK full:B200/test_e2e.py::test_multi_nodes_eval[Qwen3/saved_models_Qwen3-235B-A22B_nvfp4_hf-tp16-mmlu] SKIP (https://nvbugs/6424188) full:B200/test_e2e.py::test_qwen_e2e_cpprunner_large_new_tokens[DeepSeek-R1-Distill-Qwen-1.5B-DeepSeek-R1-Distill-Qwen-1.5B] SKIP (https://nvbugs/6414760) full:B300/accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype_with_helix[fifo_v2-cudagraph:with_padding-pp1tp2cp2] SKIP (https://nvbugs/6410881) -full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm] SKIP (https://nvbugs/6539942) full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[fp8-4-attn_dp_off-trtllm] SKIP (https://nvbugs/6483370) full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[bf16_ws4_180gb-flashinfer] SKIP (https://nvbugs/6478645) full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[bf16_ws4_180gb-trtllm] SKIP (https://nvbugs/6478645)