[https://nvbugs/6405760][fix] Do not load the multimodal encoder for text-only trtllm-bench runs - #16250
Conversation
…text-only trtllm-bench runs Since NVIDIA#15249/NVIDIA#14599, Qwen3.5 dense/MoE checkpoints (arch Qwen3_5ForConditionalGeneration / Qwen3_5MoeForConditionalGeneration) resolve to the VLM wrapper, which unconditionally instantiates and loads the vision tower (~0.86 GiB for Qwen3.5-9B). For text-only serving this memory is pure overhead: it shrinks the KV cache pool sized from free GPU memory (22.55 -> 21.69 GiB on L40S for qwen3.5_9b) and drops 500/2000 ISL/OSL throughput by ~7.5% by reducing the number of concurrently schedulable requests. Add a prototype TorchLlmArgs flag, disable_mm_encoder, that skips instantiating and loading the multimodal encoder of a VLM checkpoint (mirroring the existing MM E/P disagg encoder-skip path), and set it in trtllm-bench throughput runs when --modality is not given. Raw image/video requests are rejected with a clear error when the encoder is disabled (pre-existing guard). KV cache capacity profiling skips the dummy encoder pass when no local encoder is present. Verified on H100 with Qwen3.5-9B bf16 (maxbs 512, maxnt 2048, max_seq 2500): KV pool 53.13 GiB with the tower loaded vs 54.04 GiB with disable_mm_encoder=1 or pre-NVIDIA#15249 code; weights-load memory returns from 17.87 GiB to 16.94 GiB. Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com>
|
/bot run |
|
PR_Github #58698 [ run ] triggered by Bot. Commit: |
📝 WalkthroughWalkthroughAdds a ChangesMultimodal encoder configuration and runtime
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Benchmark
participant ModelLoader
participant Qwen3VLModelBase
participant PyExecutor
Benchmark->>ModelLoader: set disable_mm_encoder for text-only PyTorch run
ModelLoader->>Qwen3VLModelBase: load configuration
Qwen3VLModelBase-->>ModelLoader: omit local multimodal encoder
PyExecutor->>Qwen3VLModelBase: request dummy encoder inputs
Qwen3VLModelBase-->>PyExecutor: return empty batch
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tensorrt_llm/llmapi/llm_args.py (1)
5286-5290: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a regression test for the mutually exclusive modes.
The existing Qwen3.5 tests exercise
ModelConfigdirectly, so they do not verify thatTorchLlmArgs(disable_mm_encoder=True, mm_encoder_only=True)raisesValueError. Add or extendtests/unittest/llmapi/test_llm_args.pyto cover this invalid combination and the defaultFalsebehavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/llmapi/llm_args.py` around lines 5286 - 5290, Add regression coverage in test_llm_args.py for TorchLlmArgs: assert that constructing it with disable_mm_encoder=True and mm_encoder_only=True raises ValueError, and verify both options default to False when omitted.Source: Path instructions
tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.py (1)
168-190: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for downstream encoder-disabled behavior.
These tests sufficiently verify constructor state, but coverage is insufficient for the new contracts: disabled models must reject raw image/video requests, and text-only PyTorch throughput must default the flag while preserving an explicit
disable_mm_encoder=Falseoverride.Add focused cases to
tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.pyand the throughput command test module, for exampletests/unittest/bench/benchmark/test_throughput.py; if that suite is integration-only, track the benchmark case as follow-up outside this PR.As per path instructions, this test review identifies what is covered, what is insufficient, and the concrete follow-up files.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.py` around lines 168 - 190, Add focused tests beyond constructor state: verify a Qwen3_5VLModel created with disable_mm_encoder=True rejects raw image and video requests, and update the PyTorch text-only throughput command tests to confirm the flag defaults to disabled while an explicit disable_mm_encoder=False is preserved. Use the existing Qwen3_5VLModel test helpers and throughput command test symbols; if throughput coverage is integration-only, record that benchmark case as a follow-up.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/bench/benchmark/throughput.py`:
- Around line 452-454: Normalize options.backend to a consistent case before the
automatic-default condition in the benchmark dispatch logic, then compare the
normalized value with "pytorch" so mixed-case PyTorch invocations also set
disable_mm_encoder=True when appropriate.
---
Nitpick comments:
In `@tensorrt_llm/llmapi/llm_args.py`:
- Around line 5286-5290: Add regression coverage in test_llm_args.py for
TorchLlmArgs: assert that constructing it with disable_mm_encoder=True and
mm_encoder_only=True raises ValueError, and verify both options default to False
when omitted.
In `@tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.py`:
- Around line 168-190: Add focused tests beyond constructor state: verify a
Qwen3_5VLModel created with disable_mm_encoder=True rejects raw image and video
requests, and update the PyTorch text-only throughput command tests to confirm
the flag defaults to disabled while an explicit disable_mm_encoder=False is
preserved. Use the existing Qwen3_5VLModel test helpers and throughput command
test symbols; if throughput coverage is integration-only, record that benchmark
case as a follow-up.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 702af6a7-b035-41b6-a377-3844cb326c88
📒 Files selected for processing (9)
tensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/models/modeling_qwen3_5.pytensorrt_llm/_torch/models/modeling_qwen3vl.pytensorrt_llm/_torch/pyexecutor/_util.pytensorrt_llm/_torch/pyexecutor/model_loader.pytensorrt_llm/bench/benchmark/throughput.pytensorrt_llm/llmapi/llm_args.pytests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.pytests/unittest/api_stability/references/llm.yaml
thorjohnsen
added
the
api-compatible
|
PR_Github #58698 [ run ] completed with state
|
…nifest for disable_mm_encoder The disable_mm_encoder TorchLlmArgs field added by this PR is captured by the LLM API telemetry manifest, so the committed golden manifest must include it. Regenerated tensorrt_llm/usage/llm_args_golden_manifest.json, which adds the single disable_mm_encoder row. Fixes the premerge privacy-gate failure in tests/unittest/usage/test_llmapi_config_telemetry_docs.py (test_build_capture_manifest_matches_committed_golden). Signed-off-by: Thor Johnsen <tjohnsen@nvidia.com>
|
/bot run |
…-mm-encoder Signed-off-by: Thor Johnsen <tjohnsen@nvidia.com> # Conflicts: # tensorrt_llm/usage/llm_args_golden_manifest.json
|
/bot run |
|
PR_Github #58997 [ run ] triggered by Bot. Commit: |
|
PR_Github #58997 [ run ] completed with state
|
…l opt-in Address review nits: clarify in both the ModelConfig comment and the TorchLlmArgs field description that disable_mm_encoder is not applied auto-magically - each model implementation must honor it (currently the Qwen3-VL / Qwen3.5-VL family), and it is a no-op for models that do not check it. Docstring/comment only; no schema change (telemetry golden manifest and api_stability reference unaffected). Signed-off-by: Thor Johnsen <tjohnsen@nvidia.com>
|
/bot run |
|
PR_Github #59021 [ run ] triggered by Bot. Commit: |
|
PR_Github #59021 [ run ] completed with state
|
|
/bot run |
|
PR_Github #59082 [ run ] triggered by Bot. Commit: |
|
PR_Github #59082 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #59216 [ run ] triggered by Bot. Commit: |
DomBrown
left a comment
DomBrown
left a comment
There was a problem hiding this comment.
LGTM for the API review part
|
PR_Github #59216 [ run ] completed with state
|
|
/bot run |
|
PR_Github #59267 [ run ] triggered by Bot. Commit: |
|
PR_Github #59267 [ run ] completed with state
|
|
/bot run |
|
PR_Github #59316 [ run ] triggered by Bot. Commit: |
|
PR_Github #59316 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #59484 [ run ] triggered by Bot. Commit: |
|
PR_Github #59484 [ run ] completed with state |
8 participants
Description
Fixes the qwen3.5_9b L40S perf regression tracked in NVBug 6405760 / TRTLLM-13979 that persisted after the deep_gemm import-context fix (#15632).
Root cause: since #15249 (dense) and #14599 (MoE), Qwen3.5 checkpoints with arch
Qwen3_5ForConditionalGeneration/Qwen3_5MoeForConditionalGenerationresolve to the VLM wrapper, which unconditionally instantiates and loads the vision tower (~0.86 GiB for Qwen3.5-9B) — also for text-only serving. The KV cache pool is sized from free GPU memory, so on L40S it shrank from 22.55 to 21.69 GiB, dropping the number of concurrently schedulable 2500-token requests and costing ~7.5% total token throughput on the 500/2000 ISL/OSL perf test (this workload sits right at a capacity threshold: QA history shows 22.55 GiB ↔ ~3.1k tok/s fast regime, ≤22.34 GiB ↔ ~2.87k tok/s slow regime).Fix:
TorchLlmArgs.disable_mm_encoder: skip instantiating and loading the multimodal encoder of a VLM checkpoint (mirrors the existing MM E/P disagg encoder-skip path). Raw image/video requests are rejected by the pre-existing guard.trtllm-bench throughputsets it automatically for the PyTorch backend when--modalityis not given (overridable viaextra_llm_api_options), restoring pre-[TRTLLM-13383][feat] Add support for Qwen3.5 VL Dense #15249 behavior for text-only benchmarks without QA test changes.Verification (H100, Qwen3.5-9B bf16, maxbs 512 / maxnt 2048 / max_seq_len 2500, identical binaries, Python-only pinned-binary bisect):
End-to-end run (init, KV sizing, CUDA graph warmup, generate) passes with the flag enabled.
Likely also resolves the re-occurrence risk for the qwen3.5 MoE perf tests (nvbugs 6418453/6419078 family) since the gate lives in the shared
Qwen3VLModelBase.Test Coverage
tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.py::test_qwen35_dense_vl_disable_mm_encoder_skips_vision_towertests/unittest/_torch/modeling/test_modeling_qwen3_5_vl.py::test_qwen35_dense_vl_default_keeps_vision_towerPR Checklist
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes