[https://nvbugs/5836830][fix] Fix DeepSeekV3 W4A8 mixed precision quantization for MoE experts by Barry-Delaney · Pull Request #12149 · NVIDIA/TensorRT-LLM

Barry-Delaney · 2026-03-12T08:13:10Z

Resolve per-expert quant config (e.g. W4A8_AWQ) from the global MIXED_PRECISION config so that MoE backends receive the correct quantization strategy. Previously the ambiguous MIXED_PRECISION config was passed through, causing assertion failures and incorrect weight loading.

Summary by CodeRabbit

Bug Fixes
- Improved quantization configuration handling for mixture of expert layers, enabling more precise per-module quantization strategies instead of global settings.
Refactor
- Centralized quantization configuration resolution to provide better flexibility in applying different precision modes across model components.

coderabbitai · 2026-03-12T08:18:31Z

📝 Walkthrough

Walkthrough

The changes implement per-expert and per-module quantization override propagation in DeepSeek-v3 model construction and configurable MoE layers. Quantization configuration resolution is centralized at per-layer and per-expert initialization stages, replacing previous per-call recomputation with dedicated override configuration storage and application logic.

Changes

Cohort / File(s)	Summary
MoE Quantization Configuration `tensorrt_llm/_torch/models/modeling_deepseekv3.py`	Introduces per-expert quantization resolution for MIXED_PRECISION via expert_quant_config override. Defers weight loading mode decision to use expert_quant_config.layer_quant_mode.has_nvfp4(). Replaces hard assertion on MIXED_PRECISION with conservative per-module quant configuration approach for nvfp4 support determination.
Configurable MoE Backend `tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py`	Stores and applies per-module override_quant_config in init. Updates _get_quant_config_dict to prefer _override_quant_config fallback to model_config.quant_config. Expands quant mode extraction with additional boolean flags (has_nvfp4, has_w4afp8, has_fp8_block_scales) for precise quantization capability reporting.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the issue (ambiguous MIXED_PRECISION config causing failures) and solution (per-expert quant config resolution), but lacks Test Coverage section and PR Checklist required by the template.	Add the Test Coverage section listing relevant tests, and complete the PR Checklist with checkboxes to demonstrate review of NVIDIA coding guidelines and test coverage requirements.
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically identifies the main change: fixing W4A8 mixed precision quantization for MoE experts in DeepSeekV3, which aligns with the changeset's focus on resolving per-expert quantization configuration.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

📝 Coding Plan

Generate coding plan for human review comments

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can use Trivy to scan for security misconfigurations and secrets in Infrastructure as Code files.

Add a .trivyignore file to your project to customize which findings Trivy reports.

Barry-Delaney · 2026-03-12T08:18:56Z

/bot run

tensorrt-cicd · 2026-03-12T08:25:39Z

PR_Github #38699 [ run ] triggered by Bot. Commit: 4e029cf Link to invocation

litaotju

LGTM — clean fix, good comments. One question: are there any test cases in CI or post-merge that exercise the MIXED_PRECISION path through MoE init (e.g. the W4AFP8 quickstart test from the bug)? Want to make sure this gets regression-covered going forward.

tensorrt-cicd · 2026-03-12T10:56:24Z

PR_Github #38699 [ run ] completed with state SUCCESS. Commit: 4e029cf
/LLM/main/L0_MergeRequest_PR pipeline #30018 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Barry-Delaney · 2026-03-13T05:39:21Z

/bot run

tensorrt-cicd · 2026-03-13T05:46:02Z

PR_Github #38831 [ run ] triggered by Bot. Commit: 474ec37 Link to invocation

Barry-Delaney · 2026-03-13T05:54:53Z

Are there any test cases in CI or post-merge that exercise the MIXED_PRECISION path through MoE init (e.g. the W4AFP8 quickstart test from the bug)? Want to make sure this gets regression-covered going forward.

Thanks on the review @litaotju! It seems the test is only covered in QA, I added one into post-merge in the latest commit.

Barry-Delaney · 2026-03-13T05:55:37Z

/bot kill

tensorrt-cicd · 2026-03-13T06:01:24Z

PR_Github #38832 [ kill ] triggered by Bot. Commit: 801cb77 Link to invocation

tensorrt-cicd · 2026-03-13T06:02:07Z

PR_Github #38832 [ kill ] completed with state SUCCESS. Commit: 801cb77
Successfully killed previous jobs for commit 801cb77

Link to invocation

Barry-Delaney · 2026-03-13T06:02:45Z

/bot run

tensorrt-cicd · 2026-03-13T06:08:39Z

PR_Github #38834 [ run ] triggered by Bot. Commit: f21ad31 Link to invocation

tensorrt-cicd · 2026-03-13T08:45:22Z

PR_Github #38834 [ run ] completed with state SUCCESS. Commit: f21ad31
/LLM/main/L0_MergeRequest_PR pipeline #30144 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Barry-Delaney · 2026-03-20T03:28:12Z

/bot run

tensorrt-cicd · 2026-03-20T03:33:57Z

PR_Github #39677 [ run ] triggered by Bot. Commit: b30bd67 Link to invocation

tensorrt-cicd · 2026-04-08T17:10:06Z

PR_Github #42317 [ run ] completed with state SUCCESS. Commit: ca8c7c5
/LLM/main/L0_MergeRequest_PR pipeline #33108 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Barry-Delaney · 2026-04-09T02:30:18Z

/bot run

Barry-Delaney · 2026-04-09T07:22:31Z

/bot run

tensorrt-cicd · 2026-04-09T07:32:26Z

PR_Github #42498 [ run ] triggered by Bot. Commit: 8343239 Link to invocation

tensorrt-cicd · 2026-04-09T13:45:24Z

PR_Github #42498 [ run ] completed with state SUCCESS. Commit: 8343239
/LLM/main/L0_MergeRequest_PR pipeline #33245 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Barry-Delaney · 2026-04-14T06:33:48Z

/bot run

tensorrt-cicd · 2026-04-14T06:41:11Z

PR_Github #43196 [ run ] triggered by Bot. Commit: 7cacaf2 Link to invocation

tensorrt-cicd · 2026-04-14T06:41:11Z

PR_Github #43196 [ run ] completed with state DISABLED
Freeze main and open the PR merge only after CI is back to healthy https://nvidia.slack.com/archives/C059LSY62BT/p1776141760843319?thread_ts=1775985925.442509&cid=C059LSY62BT

Link to invocation

Barry-Delaney · 2026-04-14T07:05:44Z

/bot kill

tensorrt-cicd · 2026-04-14T07:11:41Z

PR_Github #43201 [ kill ] triggered by Bot. Commit: 7cacaf2 Link to invocation

tensorrt-cicd · 2026-04-14T07:11:43Z

PR_Github #43201 [ kill ] completed with state SUCCESS. Commit: 7cacaf2
Successfully killed previous jobs for commit 7cacaf2

Link to invocation

Barry-Delaney · 2026-04-16T02:20:53Z

/bot run

tensorrt-cicd · 2026-04-16T02:28:15Z

PR_Github #43612 [ run ] triggered by Bot. Commit: 4041fe3 Link to invocation

tensorrt-cicd · 2026-04-16T03:24:58Z

PR_Github #43612 [ run ] completed with state FAILURE. Commit: 4041fe3
/LLM/main/L0_MergeRequest_PR pipeline #34104 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Barry-Delaney · 2026-04-16T07:40:46Z

/bot run

tensorrt-cicd · 2026-04-16T07:46:26Z

PR_Github #43713 [ run ] triggered by Bot. Commit: a55f022 Link to invocation

tensorrt-cicd · 2026-04-16T21:02:01Z

PR_Github #43713 [ run ] completed with state FAILURE. Commit: a55f022
/LLM/main/L0_MergeRequest_PR pipeline #34196 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Resolve per-expert quant config (e.g. W4A8_AWQ) from the global MIXED_PRECISION config so that MoE backends receive the correct quantization strategy. Previously the ambiguous MIXED_PRECISION config was passed through, causing assertion failures and incorrect weight loading. For non-MIXED_PRECISION cases (e.g. nvfp4, unquantized MTP layers), pass override_quant_config through as-is since it already encodes the correct exclusions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Barry-Delaney · 2026-04-22T08:57:36Z

/bot run

tensorrt-cicd · 2026-04-22T09:03:16Z

PR_Github #44944 [ run ] triggered by Bot. Commit: 88a2e8d Link to invocation

Barry-Delaney self-assigned this Mar 12, 2026

Barry-Delaney requested review from a team as code owners March 12, 2026 08:13

Barry-Delaney requested review from QiJune, dongxuy04, symphonylyh and xxi-nv March 12, 2026 08:13

Barry-Delaney force-pushed the fix_w4a8 branch from 54f0ee9 to 4e029cf Compare March 12, 2026 08:18

litaotju approved these changes Mar 12, 2026

View reviewed changes

Barry-Delaney changed the title ~~[https://nvbugs/5955173][fix] Fix DeepSeekV3 W4A8 mixed precision quantization for MoE experts~~ [https://nvbugs/5836830][fix] Fix DeepSeekV3 W4A8 mixed precision quantization for MoE experts Mar 13, 2026

Barry-Delaney force-pushed the fix_w4a8 branch from 4e029cf to 474ec37 Compare March 13, 2026 05:33

Barry-Delaney force-pushed the fix_w4a8 branch from 801cb77 to f21ad31 Compare March 13, 2026 06:02

Barry-Delaney force-pushed the fix_w4a8 branch 2 times, most recently from c1f1080 to b30bd67 Compare March 20, 2026 03:27

Barry-Delaney force-pushed the fix_w4a8 branch from ca8c7c5 to fdc83cc Compare April 9, 2026 02:30

Barry-Delaney force-pushed the fix_w4a8 branch 2 times, most recently from c0d799b to 8343239 Compare April 9, 2026 07:22

Barry-Delaney force-pushed the fix_w4a8 branch from 8343239 to 7cacaf2 Compare April 14, 2026 06:33

Barry-Delaney force-pushed the fix_w4a8 branch from 7cacaf2 to 4041fe3 Compare April 16, 2026 02:20

Barry-Delaney force-pushed the fix_w4a8 branch from 4041fe3 to a55f022 Compare April 16, 2026 07:40

Barry-Delaney and others added 3 commits April 22, 2026 16:57

Fix incorrect override in NVFP4+MTP

94c48eb

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Update CI

88a2e8d

Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>

Barry-Delaney force-pushed the fix_w4a8 branch from a55f022 to 88a2e8d Compare April 22, 2026 08:57

lfr-0531 mentioned this pull request Jul 15, 2026

[None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints #16433

Merged

1 task

Uh oh!

Conversation

Barry-Delaney commented Mar 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (2 warnings)

Uh oh!

Barry-Delaney commented Mar 12, 2026

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

litaotju left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Mar 12, 2026

Uh oh!

Barry-Delaney commented Mar 13, 2026

Uh oh!

tensorrt-cicd commented Mar 13, 2026

Uh oh!

Barry-Delaney commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Barry-Delaney commented Mar 13, 2026

Uh oh!

tensorrt-cicd commented Mar 13, 2026

Uh oh!

tensorrt-cicd commented Mar 13, 2026

Uh oh!

Barry-Delaney commented Mar 13, 2026

Uh oh!

tensorrt-cicd commented Mar 13, 2026

Uh oh!

tensorrt-cicd commented Mar 13, 2026

Uh oh!

Barry-Delaney commented Mar 20, 2026

Uh oh!

tensorrt-cicd commented Mar 20, 2026

Uh oh!

tensorrt-cicd commented Apr 8, 2026

Uh oh!

Barry-Delaney commented Apr 9, 2026

Uh oh!

Barry-Delaney commented Apr 9, 2026

Uh oh!

tensorrt-cicd commented Apr 9, 2026

Uh oh!

tensorrt-cicd commented Apr 9, 2026

Uh oh!

Barry-Delaney commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

Barry-Delaney commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

tensorrt-cicd commented Apr 14, 2026

Uh oh!

Barry-Delaney commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

Barry-Delaney commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

tensorrt-cicd commented Apr 16, 2026

Uh oh!

Barry-Delaney commented Apr 22, 2026

Barry-Delaney commented Mar 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 12, 2026 •

edited

Loading

Barry-Delaney commented Mar 13, 2026 •

edited

Loading