[None][feat] EXAONE-4.5 Support - #12873
Merged
Merged
Conversation
yechank-nvidia
requested review from
QiJune,
byshiue,
dongjiyingdjy,
nv-guomingz,
qixiang-99 and
syuoni
April 9, 2026 02:53
Contributor
📝 WalkthroughWalkthroughThe PR adds EXAONE-4.5 multimodal VLM support with new model implementations and weight mappers. Vision models are refactored to derive dtype/device directly from tensors instead of transformers utilities. Qwen vision models undergo significant RoPE computation and positional embedding pipeline updates. Model loader APIs are updated for kosmos-2, and test infrastructure gains skip mechanisms for conditional test execution. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant InputProc as Input Processor
participant VisionEnc as Vision Encoder
participant TextEmbed as Text Embedding
participant LLM as Language Model
participant Output
Client->>InputProc: Text + Images
InputProc->>InputProc: Preprocess text & images
InputProc->>InputProc: Fuse multimodal placeholders
InputProc->>VisionEnc: pixel_values + grid_thw
VisionEnc->>VisionEnc: Compute windowed RoPE (cos, sin)
VisionEnc->>VisionEnc: Apply vision attention with position_ids
VisionEnc-->>InputProc: Vision embeddings
InputProc->>TextEmbed: Fused input_ids + multimodal_data
TextEmbed->>TextEmbed: Embed text tokens
TextEmbed->>LLM: Fused embeddings (text + vision)
LLM->>LLM: Causal language modeling
LLM-->>Output: Logits / Tokens
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (2)
tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py (1)
19-19: Missing return type annotation.The
preprocess_weightsmethod is missing a return type hint. Per coding guidelines, functions should be annotated with type hints.✏️ Proposed fix
- def preprocess_weights(self, weights: dict): + def preprocess_weights(self, weights: dict) -> dict:🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py` at line 19, The method preprocess_weights(self, weights: dict) is missing a return type annotation; update its signature to include an explicit return type such as -> Dict[str, Any] (or Mapping[str, Any] if preferred) and ensure you import the corresponding typing symbols (Dict and Any or Mapping) at the top of the module so the signature reads e.g. def preprocess_weights(self, weights: Dict[str, Any]) -> Dict[str, Any]: while keeping the existing behavior in the preprocess_weights implementation.tests/unittest/_torch/modeling/test_modeling_exaone4_5.py (1)
183-186: Hardcoded local path for test weights.The test config contains a hardcoded developer-specific path (
/code/yechan-models/exaone45_beta_2026-03-19_bf16). While theskip_testproperty handles missing paths gracefully, consider using an environment variable (e.g.,EXAONE_4_5_MODEL_PATH) for configurability:♻️ Suggested improvement
+import os + +_EXAONE_4_5_DEFAULT_PATH = "/code/yechan-models/exaone45_beta_2026-03-19_bf16" + EXAONE_4_5_TEST_CONFIG = { # ... other config ... - "_name_or_path": str( - os.path.join("/code/yechan-models", "exaone45_beta_2026-03-19_bf16") - ), # str(os.path.join(llm_models_root(), "Qwen2.5-VL-7B-Instruct")) + "_name_or_path": os.environ.get("EXAONE_4_5_MODEL_PATH", _EXAONE_4_5_DEFAULT_PATH), }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/unittest/_torch/modeling/test_modeling_exaone4_5.py` around lines 183 - 186, Replace the hardcoded developer path in the test config for "_name_or_path" with an environment-configurable value: read os.environ.get("EXAONE_4_5_MODEL_PATH") and fall back to the existing os.path.join("/code/yechan-models", "exaone45_beta_2026-03-19_bf16") if the env var is not set; keep the str(...) cast and preserve the existing skip_test behavior that already handles missing paths. Update the assignment where "_name_or_path" is set in tests/unittest/_torch/modeling/test_modeling_exaone4_5.py to use this env var fallback.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/models/core/exaone/README.md`:
- Around line 102-106: Replace the unresolved TODOs in the EXAONE-4.5 README:
set the actual HuggingFace repo name in the git clone command (replace `<TODO:
FILL>` in the git clone URL that follows HF_MODEL_DIR) and update the example's
expected output section (the lines labeled `TODO: FILL` around the expected
output) with the real output produced by the model so the README contains the
correct repository path and sample result.
In `@tensorrt_llm/_torch/models/modeling_exaone4_5.py`:
- Around line 235-237: Replace the runtime type check in load_weights to avoid
using assert: explicitly verify that weight_mapper is an instance of
Exaone4_5HfWeightMapper and if not raise a TypeError with a clear message (e.g.,
indicating expected Exaone4_5HfWeightMapper but got type(weight_mapper)); then
proceed to call weight_mapper.preprocess_weights(weights). Ensure you reference
the load_weights method and the Exaone4_5HfWeightMapper/BaseWeightMapper types
when making the change.
- Around line 175-189: Exaone4_5_VLModel is processing all multimodal_params
including text-only entries; call the parent filter to only keep requests with
actual multimodal data before calling the encoder. Modify the block handling
multimodal_params to first call
self._get_requests_with_mm_data(multimodal_params) (or assign its return to a
local filtered list) and use that filtered list when deciding to call
get_multimodal_embeddings, then pass the filtered mm_embeds into
find_input_mm_embeds; ensure you still raise NotImplementedError for
disaggregated mode in the same place.
In `@tensorrt_llm/_torch/models/modeling_qwen2vl.py`:
- Around line 914-916: The prepare_attn_metadata method in modeling_qwen2vl.py
shadows the incoming batch_size parameter by reassigning it to len(seq_lens);
remove that reassignment so the passed batch_size is used, mirroring the fix
from modeling_qwen3vl.py; then ensure any callers that currently rely on the old
behavior (e.g., the call sites referenced around lines 976-980) are updated to
pass the correct batch_size value (or compute len(seq_lens) before calling) so
prepare_attn_metadata(batch_size: int, seq_lens: List[int], attn_metadata:
AttentionMetadata) uses its batch_size argument as intended.
In `@tensorrt_llm/_torch/models/modeling_qwen3vl.py`:
- Around line 768-771: The prepare_attn_metadata function currently shadows its
batch_size parameter with batch_size = len(seq_lens); remove the parameter from
prepare_attn_metadata's signature and update every call site that passes
batch_size (e.g., where prepare_attn_metadata(...) is invoked) to stop supplying
that argument, or alternatively keep the parameter and delete the reassignment
so the passed batch_size is used; modify the function signature and all
references consistently (function name: prepare_attn_metadata, local variable:
seq_lens) so there is no shadowing or dead parameter.
In `@tensorrt_llm/serve/chat_utils.py`:
- Around line 289-290: The line initializing MultimodalDataTracker is
misformatted causing pre-commit/yapf failures; reformat the statement that
constructs MultimodalDataTracker(type(model_config).model_type,
multimodal_server_config) to satisfy the project's formatter (wrap arguments
across lines or adjust spacing consistent with other calls), then run the
project's pre-commit hooks or `yapf` to enforce line wrapping for this and any
adjacent function calls; ensure you update the call sites referencing
MultimodalDataTracker, model_config, and multimodal_server_config so they
conform to the repository's line-length and formatting rules.
- Around line 305-307: Remove the dead local assignment "model_type =
model_config.model_type" since subsequent calls (e.g.,
MULTIMODAL_PLACEHOLDER_REGISTRY.get_content_format(type(model_config).model_type))
use type(model_config).model_type instead; delete the unused "model_type"
variable or alternatively replace other uses to reference the local variable
consistently, ensuring references around model_config and
MULTIMODAL_PLACEHOLDER_REGISTRY.get_content_format remain correct.
---
Nitpick comments:
In `@tensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.py`:
- Line 19: The method preprocess_weights(self, weights: dict) is missing a
return type annotation; update its signature to include an explicit return type
such as -> Dict[str, Any] (or Mapping[str, Any] if preferred) and ensure you
import the corresponding typing symbols (Dict and Any or Mapping) at the top of
the module so the signature reads e.g. def preprocess_weights(self, weights:
Dict[str, Any]) -> Dict[str, Any]: while keeping the existing behavior in the
preprocess_weights implementation.
In `@tests/unittest/_torch/modeling/test_modeling_exaone4_5.py`:
- Around line 183-186: Replace the hardcoded developer path in the test config
for "_name_or_path" with an environment-configurable value: read
os.environ.get("EXAONE_4_5_MODEL_PATH") and fall back to the existing
os.path.join("/code/yechan-models", "exaone45_beta_2026-03-19_bf16") if the env
var is not set; keep the str(...) cast and preserve the existing skip_test
behavior that already handles missing paths. Update the assignment where
"_name_or_path" is set in
tests/unittest/_torch/modeling/test_modeling_exaone4_5.py to use this env var
fallback.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a4fc1689-50d7-46db-9958-52340c2e4a19
📒 Files selected for processing (18)
examples/models/core/exaone/README.mdrequirements.txttensorrt_llm/_torch/models/__init__.pytensorrt_llm/_torch/models/checkpoints/hf/exaone4_5_weight_mapper.pytensorrt_llm/_torch/models/modeling_clip.pytensorrt_llm/_torch/models/modeling_exaone4_5.pytensorrt_llm/_torch/models/modeling_exaone_moe.pytensorrt_llm/_torch/models/modeling_llama.pytensorrt_llm/_torch/models/modeling_qwen2vl.pytensorrt_llm/_torch/models/modeling_qwen3vl.pytensorrt_llm/_torch/models/modeling_siglip.pytensorrt_llm/_torch/visual_gen/models/wan/transformer_wan.pytensorrt_llm/models/gpt/convert.pytensorrt_llm/serve/chat_utils.pytensorrt_llm/serve/openai_server.pytensorrt_llm/tools/multimodal_builder.pytests/unittest/_torch/modeling/test_modeling_exaone4_5.pytests/unittest/_torch/modeling/test_modeling_multimodal.py
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #42598 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #42598 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #42996 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #42996 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #43013 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #43013 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #43026 [ run ] triggered by Bot. Commit: |
Two follow-ups to the Qwen2.5-VL transformers-5.x fix: * `Qwen3VisionModel.__init__` was hard-coding `self.config.max_position_embeddings = 8192`. Match Qwen2.5-VL: read the value from `text_config.max_position_embeddings` (falling back to the top-level pretrained_config) and mirror it onto vision_config. * `Qwen3VLVisionAttention.__init__` was mutating `model_config.pretrained_config.max_position_embeddings` from `text_config.max_position_embeddings`. The parent `Qwen2_5_VLVisionAttention.__init__` already performs that fallback, so the mutation is redundant. Keep the `vision_config.torch_dtype` mirror — that one is still required because Qwen3-VL only carries dtype on `text_config`. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
Addresses @venkywonka's review comment on the EXAONE 4.5 PR: the new file under tensorrt_llm/_torch/models/checkpoints/hf/ was missing the standard SPDX Apache-2.0 header. Adds the canonical NVIDIA header matching the rest of the repo (2026 copyright year). Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
…ites `setup_scenario` rebuilds `trtllm_model` with `disable_fuse_rope=True` when the scenario asks for it, but does not undo the rebuild afterwards (the parent `setup_scenario` only resets runtime_features and attn_metadata). With the previous ordering, the rebuilt model leaked into the chunked-prefill and kv-cache-reuse scenarios that ran after `no_fuse_rope`, where it surfaced as a cos/sin vs. q/k seq-len mismatch inside `MRotaryEmbedding.forward` (the rotary path expected the fused-rope wiring rather than the explicit `apply_rotary_pos_emb` codepath). Move the `disable_fuse_rope=True` scenario to the tail of every Qwen-VL test suite (Qwen2.5-VL, Qwen3-VL, Qwen3-VL-MoE) so the rebuilt model is the last one used and never seen by another scenario. Only one extra engine is built (for that scenario) per file; the four upstream scenarios still share a single fused-rope-enabled model. Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
yechank-nvidia
force-pushed
the
exaone_4_5
branch
from
May 20, 2026 01:03
c36cb8b to
36d9309
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49292 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #49292 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49334 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #49334 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49378 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #49378 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49433 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #49433 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49536 [ run ] triggered by Bot. Commit: |
juney-nvidia
approved these changes
May 21, 2026
Collaborator
|
PR_Github #49536 [ run ] completed with state
|
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #49669 [ run ] triggered by Bot. Commit: |
yechank-nvidia
enabled auto-merge (squash)
May 21, 2026 11:17
Collaborator
|
PR_Github #49669 [ run ] completed with state |
xxi-nv
pushed a commit
to xxi-nv/TensorRT-LLM
that referenced
this pull request
May 22, 2026
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
KleinBlueC
pushed a commit
to KleinBlueC/TensorRT-LLM
that referenced
this pull request
May 26, 2026
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
bmarimuthu-nv
pushed a commit
to nv-auto-deploy/TensorRT-LLM
that referenced
this pull request
May 28, 2026
Signed-off-by: yechank <161688079+yechank-nvidia@users.noreply.github.com>
2 tasks
tensorrt-cicd
added a commit
to tensorrt-cicd/TensorRT-LLM
that referenced
this pull request
Jun 2, 2026
…served score The MMMU reference for LGAI-EXAONE/EXAONE-4.5-33B was set to 51.22 in PR NVIDIA#12873 but never validated by pre-merge CI -- the test was added directly to the QA test list, so the bogus reference was never caught. The model in fact scores ~46.5 on every GPU we run it on (H20, B200, B300, GB200, GB300), and the bug confirms "PASSED commit: N/A": the test has never passed since the day it was added. Drop the reference from 51.22 to 46.5 (matches the cross-GPU empirical mean 46.444-46.778) and remove the two waiver entries that were suppressing both parametrize variants. Verified locally on GB300: both full_budget (eval=46.444) and forced_chunked_prefill variants pass against the new threshold (42.623). Run wall time 39 min. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is adding Day-0 support of LG AI's new VLM model, EXAONE-4.5.
This PR includes Text + Multimodal support.
Prerequisite
Sample command
@codderabbit summary