From cf4d9662e2537a6e04f52af81c0cea76508b664a Mon Sep 17 00:00:00 2001 From: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> Date: Sun, 31 May 2026 08:11:49 -0700 Subject: [PATCH] [nvbugs/6211189][fix] Lower EXAONE-4.5-33B MMMU reference to match observed score The MMMU reference for LGAI-EXAONE/EXAONE-4.5-33B was set to 51.22 in PR #12873 but never validated by pre-merge CI -- the test was added directly to the QA test list, so the bogus reference was never caught. The model in fact scores ~46.5 on every GPU we run it on (H20, B200, B300, GB200, GB300), and the bug confirms "PASSED commit: N/A": the test has never passed since the day it was added. Drop the reference from 51.22 to 46.5 (matches the cross-GPU empirical mean 46.444-46.778) and remove the two waiver entries that were suppressing both parametrize variants. Verified locally on GB300: both full_budget (eval=46.444) and forced_chunked_prefill variants pass against the new threshold (42.623). Run wall time 39 min. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com> --- tests/integration/defs/accuracy/references/mmmu.yaml | 5 ++++- tests/integration/test_lists/waives.txt | 2 -- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/integration/defs/accuracy/references/mmmu.yaml b/tests/integration/defs/accuracy/references/mmmu.yaml index 5e3a2ba8f4c9..f25b73aa0d30 100644 --- a/tests/integration/defs/accuracy/references/mmmu.yaml +++ b/tests/integration/defs/accuracy/references/mmmu.yaml @@ -24,7 +24,10 @@ google/gemma-3-12b-it: kv_cache_quant_algo: FP8 accuracy: 50.11 LGAI-EXAONE/EXAONE-4.5-33B: - - accuracy: 51.22 + # Empirically achieved ~46.5 across H20/B200/B300/GB200/GB300 since the + # model was added (PR #12873). The original 51.22 was committed without + # pre-merge CI validation and was never reproduced. See nvbugs/6211189. + - accuracy: 46.5 Qwen/Qwen2-VL-7B-Instruct: - accuracy: 48.44 Qwen/Qwen2.5-VL-7B-Instruct: diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 9237ef23835d..230c3442fb0b 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -137,8 +137,6 @@ accuracy/test_llm_api_pytorch.py::TestQwen3_5_9B::test_bf16[mtp_on] SKIP (https: accuracy/test_llm_api_pytorch.py::TestQwen3_8B::test_fp8_block_scales_early_first_token_response SKIP (https://nvbugs/6200128) accuracy/test_llm_api_pytorch.py::TestStep3_7::test_fp8_block_scales[tp_size=4-ep_size=4-mtp_nextn=0] SKIP (https://nvbugs/6274932) accuracy/test_llm_api_pytorch.py::TestStep3_7::test_fp8_block_scales[tp_size=4-ep_size=4-mtp_nextn=3] SKIP (https://nvbugs/6274932) -accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[forced_chunked_prefill] SKIP (https://nvbugs/6211189) -accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget] SKIP (https://nvbugs/6211189) accuracy/test_llm_api_pytorch_multimodal.py::TestGemma3_27BInstruct::test_fp8_prequantized SKIP (https://nvbugs/6215689) accuracy/test_llm_api_pytorch_multimodal.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm] SKIP (https://nvbugs/6181383) accuracy/test_llm_api_pytorch_multimodal.py::TestStep3_7::test_fp8_block_scales SKIP (https://nvbugs/6274932)