[None][feat] Add Qwen3 MoE support to TensorRT backend - #6470
Conversation
📝 WalkthroughWalkthroughSupport for the "qwen3_moe" model variant is added across the model mapping, configuration, conversion, and model implementation. Logic and mappings are updated to handle "qwen3_moe" alongside "qwen2_moe", with specific differentiation for shared expert handling, activation functions, and weight loading procedures. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant ModelLoader
participant QWenForCausalLM
participant Config
participant Converter
User->>ModelLoader: Request to load 'Qwen3MoeForCausalLM'
ModelLoader->>QWenForCausalLM: Instantiate with config
QWenForCausalLM->>Config: Check qwen_type ('qwen3_moe')
Config-->>QWenForCausalLM: Set hidden_act to 'swiglu'
QWenForCausalLM->>QWenForCausalLM: Setup MoE expert mappings (no shared expert)
QWenForCausalLM->>Converter: Convert HF weights (MoE logic for 'qwen3_moe')
Converter-->>QWenForCausalLM: Return processed weights
QWenForCausalLM-->>ModelLoader: Model ready
ModelLoader-->>User: Return loaded model
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
6c3ca76 to
d1de243
Compare
svc-trtllm-gh-bot
added
the
Community want to contribute
|
Huge thanks for making this @gkswns0531. Does this support FP8 quantisation and serving for Qwen 3's MOEs? |
|
I’ve confirmed that TensorRT-LLM/examples/models/core/qwen/convert_checkpoint.py is running correctly. However, I currently only have access to an L4, and since the smallest Qwen 3 MoE model is 30B, the conversion process takes quite a while. Because of that, I haven’t been able to run inference yet. From my side, I believe I’ve addressed all the logical issues that needed fixing, but there’s always a chance I may have missed something. I’ve also requested H100 access, but I’m not sure when it will be approved. If you happen to be able to run an inference test based on my PR in the meantime, I’d really appreciate it. Thanks! 
|
|
Hi @gkswns0531, I tried the fp8 quantisation on the Qwen3 30BA3B model with H100 SXM with this command:
It failed and I ran into this issue below: |
|
@Dramilio |
ed9e784 to
527d33a
Compare
|
Hi @gkswns0531 thanks for the fix, but I still ran to that same error using H100 SXM and TensorRT-LLM v0.20.0, wondering if there's any version compatibility issues between TensorRT and ModelOpt, should I create an image from the main branch? |
|
@Dramilio |
de7d7b3 to
895e676
Compare
gkswns0531
changed the title
|
@Dramilio |
|
Hi @gkswns0531, once again, huge thanks for the work you've put in. I've successfully quantised and converted the model into a TensorrtLLM FP8 checkpoint, but ran into this error when building the engine. Here is the error: And here is my engine building command: I'm using TensorrtLLM 0.20.0, ModelOpt 0.33.0 and cherry-picked your commits. |
|
@Dramilio Thanks! ref) Try running the following commands: |
|
@byshiue @nv-yilinf @shaharmor98 Thank you! |
|
/bot run |
|
PR_Github #13895 [ run ] triggered by Bot |
|
PR_Github #13895 [ run ] completed with state |
0b04348 to
e84ab20
Compare
|
/bot run |
|
PR_Github #13969 [ run ] triggered by Bot |
|
PR_Github #13969 [ run ] completed with state |
Signed-off-by: gkswns0531 <gkswns0531@gmail.com>
Signed-off-by: gkswns0531 <gkswns0531@gmail.com>
- Add Qwen3MoeForCausalLM to model registry - Fix MLP block mapping for quantization compatibility - Add mlp_only_layers and decoder_sparse_step config support - Preserve existing Qwen2 MoE functionality Signed-off-by: gkswns0531 <gkswns0531@gmail.com>
|
PR_Github #14053 [ run ] triggered by Bot |
|
PR_Github #14053 [ run ] completed with state |
|
/bot run |
|
PR_Github #14063 [ run ] triggered by Bot |
|
PR_Github #14063 [ run ] completed with state |
|
/bot run |
|
PR_Github #14095 [ run ] triggered by Bot |
|
PR_Github #14095 [ run ] completed with state |
|
/bot run |
|
PR_Github #14197 [ run ] triggered by Bot |
|
PR_Github #14197 [ run ] completed with state |
Signed-off-by: gkswns0531 <gkswns0531@gmail.com> Signed-off-by: hanjuncho <gkswns0531@gmail.com> Co-authored-by: bhsueh_NV <11360707+byshiue@users.noreply.github.com>
|
@gkswns0531 how do i exactly use the pre release version of tensorrt llm? Should I build the whole triton server image from scratch using the base image as nvcr.io/nvidia/tritonserver:25.07-py3-min or is there any other way? Thank you for your work. |
|
@vivek-guda |
|
Hello, I have tried the awq int4 quantization, and both the quantization and compilation passed normally. However, when the model uses the TensorRT-LLM backend for inference, it repeatedly crashes and cannot be stopped |
|
I tried the quantized versions of fp8 and int4 here, and there were instances of duplicate answers. I used convert_checkpoint.by and quantize.by respectively @gkswns0531 @Dramilio |
|
Hello, |
|
Hello, when I use the Qwen3-30B-A3B engine built with TensorRT-LLM to answer questions, the model keeps repeating the same phrase. I don’t know what’s wrong. Could you take a look? Thank you. root@tensorrt-llm# python3 ../../../run.py --input_text "Hello, what's your name?" --max_output_len=50 --tokenizer_dir engine/bf16/qwen3_30b_a3b/trt_engines/1-gpu/ --engine_dir=engine/bf16/qwen3_30b_a3b/trt_engines/1-gpu/ |
|
Could you try running with improved sampling parameters to resolve the token repetition issue? python3 ../../../run.py \
--input_text "Hello, what's your name?" \
--max_output_len=50 \
--tokenizer_dir engine/bf16/qwen3_30b_a3b/trt_engines/1-gpu/ \
--engine_dir=engine/bf16/qwen3_30b_a3b/trt_engines/1-gpu/ \
--temperature=0.7 \
--top_p=0.8 \
--top_k=20 \
--repetition_penalty=1.5 \
--presence_penalty=1.5The default greedy decoding settings ( |
|
@gkswns0531 root@tensorrt-llm-yy-0:/lpai/volumes/lpai-yharnam-vol-ga/yuanyu/tensorrt_yy/TensorRT-LLM/examples/models/core/qwen# python3 What do you think about this question? Okay let me try to explain |
Using this configuration model may result in a smaller repetition rate, but there are still issues temperature=0.7 --top_p=0.8 top_k=20 repetition_penalty=1.5 presence_penalty=1.5 Tested 2 quantification methods Specific commands: convert_checkpoint.py int4: quantize.py int4: env -u NCCL_IB_HCA -u UCX_NET_DEVICES OMPI_MCA_pml=^ucx OMPI_MCA_btl=self,tcp,vader UCX_TLS= NCCL_IB_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 trtllm-build --checkpoint_dir Qwen3-30B-A3B-Thinking-2507-awq-tp1 quantize.py fp8： env -u NCCL_IB_HCA -u UCX_NET_DEVICES OMPI_MCA_pml=^ucx OMPI_MCA_btl=self,tcp,vader UCX_TLS= NCCL_IB_DISABLE=1 trtllm-build --checkpoint_dir Qwen3-30B-A3B-Thinking-2507-fp8 --output_dir Qwen3-30B-A3B-Thinking-2507-fp8-by --gemm_plugin disable --max_batch_size 8 --max_num_tokens 32768 --multiple_profiles enable --use_fp8_context_fmha enable --use_paged_context_fmha enable |
|
Hello? any update @gkswns0531 @byshiue |
|
Hi @gkswns0531 — I was impressed by your Qwen3 MoE TensorRT backend work: handling the expert-structure differences from Qwen2, updating weight conversion, and validating FP16/FP8 engine creation with vLLM vs. TensorRT latency and throughput data on H100. I’m Daniel, a user researcher with a product research team exploring a developer-owned local AI system for inference/runtime engineers. We’re researching what would materially improve model-porting and performance-validation work like this—stable low-level APIs, profiler/compiler visibility, hardware access, or reproducible end-to-end environments. Would you be open to a Zoom conversation of up to 30 minutes? We’re at the prototype stage and may later provide test units or invite relevant participants as early users or technical advisors. This is research, not sales. If interested, what private contact channel would you prefer? |
9 participants
Implements TensorRT-LLM support for Qwen3 MoE models with architectural differences from Qwen2 MoE:
Implement
Testing
ref)
vllm vs tensorrt-llm latency (Qwen3-30B-A3B MoE, H100 PCIe GPU, 2048 input_length, 3072 max_length, batch 16)
Dear Maintainers,
I would like to kindly submit a pull request that adds support for building the Qwen3-MoE model as a TensorRT engine.
I would be truly grateful if you could take the time to review it at your convenience.
Thank you very much for your consideration.
Summary by CodeRabbit
Summary by CodeRabbit