[https://nvbugs/6211189][fix] Lower the reference to 46.5 (matching cross-GPU empirical mean) and remove the t by tensorrt-cicd · Pull Request #14799 · NVIDIA/TensorRT-LLM

tensorrt-cicd · 2026-05-31T15:49:44Z

Summary

Root cause: MMMU reference 51.22 for LGAI-EXAONE/EXAONE-4.5-33B in references/mmmu.yaml was set in PR [None][feat] EXAONE-4.5 Support #12873 without pre-merge CI validation; actual score is ~46.5 on every GPU.
Fix: Lower the reference to 46.5 (matching cross-GPU empirical mean) and remove the two waiver lines for this NVBug; verified both parametrize variants now PASS.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6211189

Summary by CodeRabbit

Tests
- Removed test waivers enabling stricter validation for multimodal model tests.
Chores
- Updated accuracy baseline measurement for EXAONE-4.5-33B model.

coderabbitai · 2026-05-31T15:56:43Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 67b84557-0d42-4000-bfe2-e6b4ee062199

📥 Commits

Reviewing files that changed from the base of the PR and between 0fb9a36 and 1e03c39.

📒 Files selected for processing (2)

tests/integration/defs/accuracy/references/mmmu.yaml
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR updates the MMMU accuracy baseline for EXAONE-4.5-33B from 51.22 to 46.5 with empirical measurement notes, and removes two corresponding test waivers from the skip list to reflect the corrected expectation.

Changes

EXAONE-4.5-33B Test Expectations

Layer / File(s)	Summary
MMMU accuracy baseline update `tests/integration/defs/accuracy/references/mmmu.yaml`	LGAI-EXAONE/EXAONE-4.5-33B accuracy reference value is changed from 51.22 to 46.5 with added inline comments documenting the measured baseline and referencing the prior unreproduced issue.
Remove accuracy test waivers `tests/integration/test_lists/waives.txt`	Two waived entries for `TestExaone4_5_33B::test_auto_dtype` are removed from the skip list for the `forced_chunked_prefill` and `full_budget` parameterizations.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14523: Updates waiver entries in the same waives.txt file to adjust accuracy test skip behavior.
NVIDIA/TensorRT-LLM#14788: Modifies waives.txt to add waivers for accuracy multi-GPU test variants.

Suggested reviewers

xinhe-nv
jieli-matrix
yechank-nvidia

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title is truncated/incomplete; it states '[https://nvbugs/6211189][fix] Lower the reference to 46.5 (matching cross-GPU empirical mean) and remove the t' but cuts off mid-word at 'the t', making it unclear what is being removed.	Complete the title by finishing the sentence—likely should be 'remove the two waiver lines' or similar. Ensure the full title is clear and concise without truncation.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The PR description covers the root cause, fix, test plan, and links, but does not follow the repository's required template structure with explicit Description and Test Coverage sections as specified in the template.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

2ez4bz · 2026-06-04T22:08:09Z

/bot run

tensorrt-cicd · 2026-06-04T22:13:53Z

PR_Github #52167 [ run ] triggered by Bot. Commit: 5244922 Link to invocation

tensorrt-cicd · 2026-06-05T01:26:44Z

PR_Github #52167 [ run ] completed with state SUCCESS. Commit: 5244922
/LLM/main/L0_MergeRequest_PR pipeline #41488 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yechank-nvidia · 2026-06-05T07:40:12Z

/bot run

tensorrt-cicd · 2026-06-05T07:46:02Z

PR_Github #52298 [ run ] triggered by Bot. Commit: 5244922 Link to invocation

tensorrt-cicd · 2026-06-05T09:52:31Z

PR_Github #52298 [ run ] completed with state SUCCESS. Commit: 5244922
/LLM/main/L0_MergeRequest_PR pipeline #41606 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

2ez4bz · 2026-06-05T18:04:13Z

/bot run

tensorrt-cicd · 2026-06-05T18:09:59Z

PR_Github #52405 [ run ] triggered by Bot. Commit: 5244922 Link to invocation

tensorrt-cicd · 2026-06-05T18:50:30Z

PR_Github #52405 [ run ] completed with state SUCCESS. Commit: 5244922
/LLM/main/L0_MergeRequest_PR pipeline #41698 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…served score The MMMU reference for LGAI-EXAONE/EXAONE-4.5-33B was set to 51.22 in PR NVIDIA#12873 but never validated by pre-merge CI -- the test was added directly to the QA test list, so the bogus reference was never caught. The model in fact scores ~46.5 on every GPU we run it on (H20, B200, B300, GB200, GB300), and the bug confirms "PASSED commit: N/A": the test has never passed since the day it was added. Drop the reference from 51.22 to 46.5 (matches the cross-GPU empirical mean 46.444-46.778) and remove the two waiver entries that were suppressing both parametrize variants. Verified locally on GB300: both full_budget (eval=46.444) and forced_chunked_prefill variants pass against the new threshold (42.623). Run wall time 39 min. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

2ez4bz · 2026-06-08T16:25:50Z

/bot run

tensorrt-cicd · 2026-06-08T16:32:00Z

PR_Github #52814 [ run ] triggered by Bot. Commit: cf4d966 Link to invocation

tensorrt-cicd · 2026-06-08T21:41:00Z

PR_Github #52814 [ run ] completed with state SUCCESS. Commit: cf4d966
/LLM/main/L0_MergeRequest_PR pipeline #42067 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

2ez4bz · 2026-06-08T21:45:32Z

/bot run

tensorrt-cicd · 2026-06-08T21:51:31Z

PR_Github #52862 [ run ] triggered by Bot. Commit: cf4d966 Link to invocation

tensorrt-cicd · 2026-06-09T01:04:12Z

PR_Github #52862 [ run ] completed with state SUCCESS. Commit: cf4d966
/LLM/main/L0_MergeRequest_PR pipeline #42113 completed with status: 'SUCCESS'

CI Report

Link to invocation

…ross-GPU empirical mean) and remove the t (NVIDIA#14799) Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>

tensorrt-cicd requested a review from a team as a code owner May 31, 2026 15:49

tensorrt-cicd assigned yechank-nvidia May 31, 2026

github-actions Bot assigned tensorrt-cicd May 31, 2026

crazydemo approved these changes Jun 2, 2026

View reviewed changes

tensorrt-cicd force-pushed the repair-bot-bug6211189 branch from 1e03c39 to 5244922 Compare June 2, 2026 03:32

2ez4bz enabled auto-merge (squash) June 4, 2026 22:08

yechank-nvidia approved these changes Jun 5, 2026

View reviewed changes

tensorrt-cicd force-pushed the repair-bot-bug6211189 branch from 5244922 to cf4d966 Compare June 8, 2026 07:13

2ez4bz merged commit c1e9b00 into NVIDIA:main Jun 9, 2026
7 checks passed

Uh oh!

Conversation

tensorrt-cicd commented May 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 31, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 inconclusive)

Uh oh!

2ez4bz commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

yechank-nvidia commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

2ez4bz commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

2ez4bz commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

2ez4bz commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tensorrt-cicd commented May 31, 2026 •

edited by coderabbitai Bot

Loading