Skip to content

[https://nvbugs/6478692][fix] Pass max_workers=16 to _build in EPDVariant.nano_omni_fp8 (mirroring… - #16672

Merged
2ez4bz merged 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6478692
Jul 31, 2026
Merged

[https://nvbugs/6478692][fix] Pass max_workers=16 to _build in EPDVariant.nano_omni_fp8 (mirroring…#16672
2ez4bz merged 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6478692

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Root cause: nano_omni_fp8 did not override max_workers, defaulting to VideoMME.MAX_BATCH_SIZE=128, which oversubscribed the CPU-side multimodal preprocessing/hashing path on H100 and tripped a native-library SIGABRT.
  • Fix: Pass max_workers=16 to _build in EPDVariant.nano_omni_fp8 (mirroring qwen3vl_2b), and remove the corresponding entry from waives.txt.
  • Automated fix generated by repair-bot

Test plan

  • Verify fix on the same GPU type as the original failure
  • Check for regressions in related tests

Links

Summary by CodeRabbit

  • Bug Fixes
    • Improved stability for multimodal processing on H100 hardware by limiting parallel workers.
    • Re-enabled the disaggregated video multimodal accuracy test so it runs as part of validation.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 043d0d47-ae7f-426b-9db5-15ebe72f8662

📥 Commits

Reviewing files that changed from the base of the PR and between d75cb89 and 537c3eb.

📒 Files selected for processing (2)
  • tests/integration/defs/accuracy/test_epd_disagg_multimodal.py
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

The nano omni FP8 multimodal EPD test now explicitly limits its worker pool to 16, and the corresponding VideoMME test waiver is removed.

Changes

Multimodal EPD test enablement

Layer / File(s) Summary
Worker limit and waiver removal
tests/integration/defs/accuracy/test_epd_disagg_multimodal.py, tests/integration/test_lists/waives.txt
The nano_omni_fp8 variant sets max_workers=16, and its disaggregated VideoMME test is removed from the waiver list.

Estimated code review effort: 2 (Simple) | ~5 minutes

Possibly related PRs

Suggested reviewers: 2ez4bz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main fix and follows the ticket/type pattern, with specific context about max_workers in nano_omni_fp8.
Description check ✅ Passed The description covers the issue, fix, test plan, and bug link, though it doesn't use the template's exact section headings.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@xinhe-nv

Copy link
Copy Markdown
Collaborator

/bot run --only-qa-verify test accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8]

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60696 [ run ] triggered by Bot. Commit: 537c3eb Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60696 [ run ] completed with state FAILURE. Commit: 537c3eb
LLM_FUNCTION_AUTO_V2C #299 completed with status: 'UNSTABLE'
QA verify test: accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8] (NVBug 6478692, branch repair-bot-bug6478692, fork tensorrt-cicd, dry_run_close=true)

Link to invocation

@mzweilz mzweilz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if the tests pass.

@StanleySun639

Copy link
Copy Markdown
Collaborator

LGTM if the test can pass.

The nano_omni_fp8 EPDVariant did not override max_workers and defaulted
to VideoMME.MAX_BATCH_SIZE=128, which oversubscribes the CPU-side
multimodal preprocessing / hashing path on H100 and trips a
native-library abort (SIGABRT) with many threads inside
_media_to_raw_chw and apply_mm_hashes. The passing qwen3vl_2b variant
already caps max_workers at 16 for the same reason.

Cap the submission concurrency at 16 to match the working sibling
variant, and remove the waiver so the test runs in CI again.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@2ez4bz
2ez4bz force-pushed the repair-bot-bug6478692 branch from 537c3eb to 57ade4d Compare July 31, 2026 18:22
@2ez4bz
2ez4bz requested review from a team as code owners July 31, 2026 18:22
@2ez4bz
2ez4bz requested a review from ruodil July 31, 2026 18:22
@2ez4bz

2ez4bz commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

/bot run

@2ez4bz
2ez4bz enabled auto-merge (squash) July 31, 2026 18:24
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63095 [ run ] triggered by Bot. Commit: 57ade4d Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #63095 [ run ] completed with state SUCCESS. Commit: 57ade4d
/LLM/main/L0_MergeRequest_PR pipeline #51187 completed with status: 'SUCCESS'

CI Report

Link to invocation

@2ez4bz
2ez4bz merged commit 78a1e6a into NVIDIA:main Jul 31, 2026
7 of 8 checks passed
yuanjingx87 pushed a commit that referenced this pull request Aug 1, 2026
…PDVariant.nano_omni_fp8` (mirroring… (#16672)

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants