[TRTLLM-14054][perf] Qwen3.5-VL: pass the inner LM's normalized model_config to the weight mapper#16353
Conversation
…to the weight mapper _Qwen3_5VLModel.load_weights handed Qwen3_5MoeHfWeightMapper the VLM wrapper's model_config, but only the inner LM's config goes through the Qwen3.5 quant-dict normalization applied in the inner LM's __init__ (HF->TRT-LLM key translation + synthesis of the fused in_proj_qkvz FP8 entry). With the wrapper's un-normalized copy the mapper misses that FP8 entry, dequantizes the GDN in_proj projections to bf16 and drops their calibrated scales; the FP8-built module then re-casts the weights with weight_scale=1.0 and quantizes activations dynamically in front of every GDN in_proj GEMM. Passing self.llm.model_config keeps GDN in_proj FP8 with its calibrated scales, so the projection runs on the static per-tensor FP8 path and the per-call dynamic-quant chain disappears (prefill and decode). Measured with trtllm-serve + benchmark_serving on Qwen3.6-35B-A3B-NVFP4 (hybrid gated-delta-net + MoE) on 2x B200 (TP2 + EP2), ISL=1024/ OSL=6144, concurrency sweep 1..512: output throughput improves 10.3% on average across the 10-point curve (up to +16.4% at low concurrency, flat within noise at the saturated top end), and median TTFT at concurrency 512 improves 28%. Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthrough
ChangesQwen3.5 weight loading
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
2ez4bz
left a comment
2ez4bz
left a comment
There was a problem hiding this comment.
Out of curiosity, how did you find this issue?
The agent found this when optimizing performance for Qwen-3.6 model : ) |
|
/bot run --disable-fail-fast |
|
PR_Github #59127 [ run ] triggered by Bot. Commit: |
|
PR_Github #59127 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
1 similar comment
|
/bot run --disable-fail-fast |
|
PR_Github #59178 [ run ] triggered by Bot. Commit: |
|
PR_Github #59178 [ run ] completed with state |
3 participants
Description
_Qwen3_5VLModel.load_weightshandedQwen3_5MoeHfWeightMapperthe VLM wrapper'smodel_config, but only the inner LM's config goes through the Qwen3.5 quant-dict normalization applied in the inner LM's__init__(HF→TRT-LLM key translation + synthesis of the fusedin_proj_qkvzFP8 entry).With the wrapper's un-normalized copy, the mapper misses that FP8 entry, dequantizes the GDN
in_projprojections to bf16 and drops their calibrated scales; the FP8-built module then re-casts the weights withweight_scale=1.0and quantizes activations dynamically in front of every GDNin_projGEMM.Fix: pass
self.llm.model_config. The GDNin_projweights stay FP8 with their calibrated scales, the projection runs on the static per-tensor FP8 path (static_quantize_e4m3_per_tensor+ scaled-mm), and the per-call dynamic-quant chain disappears from both prefill and decode. One functional line plus a comment.Test Coverage
tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl_moe.py -k "test_construction_and_weight_loading_smoke or test_qwen35_moe_vl_resolves_model_and_mapper"— 2 passed.in_projshards verified to hold FP8 weights with the checkpoint's calibrated scales.trtllm-serve+benchmark_servingon Qwen3.6-35B-A3B-NVFP4 (hybrid GDN + MoE), 2× B200 with TP2 + EP2, ISL 1024 / OSL 6144, random dataset with--ignore-eos, one point per concurrency (paired num_prompts 8…2048) over one shared server launch per side. Measured at base8ef7190ebfstacked on [TRTLLM-14054][perf] Reduce per-step host preparation overhead in the PyTorch executor decode path #16313 + [TRTLLM-14054][perf] Load Qwen3-Next GDN in_proj in dense layout and add a multi-row gated RMSNorm #16314 + the LMHead TP2 load fix [TRTLLM-14054][fix] LMHead: pass true full dims to Linear for quantized TP>1 #16352 (needed to bring the TP2 server up at all), under a tuned serving config (TRTLLM MoE backend, CUDA-graph padding with max_batch_size 512); the A/B differs only by this PR's change. No shared files with any of those PRs — the change is independent at the code level.Output throughput improves +10.32% on average across the 10-point curve — up to +16.4% at low concurrency, flat within run-to-run noise at the saturated top end. No latency trade-away at the top point: median TPOT/ITL flat, median TTFT at concurrency 512 improves 28% (6835 ms → 4911 ms; large-M prefill GEMMs no longer pay the per-call dynamic-quant chain).
PR Checklist
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code
Summary by CodeRabbit