[None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints#16433
Conversation
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
📝 WalkthroughWalkthroughDeepSeek v4 decoder layers now handle mixed-precision quantization without asserting, disabling layer-level NVFP4 when appropriate. A unit test verifies the decoder flag, MoE quantization override, and routed-expert quantization configuration. ChangesDeepSeek v4 mixed-precision quantization
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/unittest/_torch/modeling/test_modeling_deepseekv4.py (1)
447-503: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCoverage is sufficient for the changed gating behavior.
The test correctly isolates and validates the new
is_nvfp4mixed-precision gating and the interaction with_get_experts_quant_config's per-expert override, using the module-attribute monkeypatch trick so the directDeepseekV4MoE._get_experts_quant_configcall at line 503 still exercises the real implementation. This is sufficient coverage for the code change inmodeling_deepseekv4.py.One untested edge case (pre-existing, not touched by this PR): the MTP-excluded-layer branch (
is_module_excluded_from_quantization) combined with a MIXED_PRECISION global config. Low risk since that path already resolvesquant_algo=Noneregardless, but could be added as a follow-up if maintainers want explicit regression coverage for that combination.As per path instructions, "Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM... suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modeling/test_modeling_deepseekv4.py` around lines 447 - 503, Coverage for the mixed-precision expert gating and per-expert override is sufficient; no code changes are required. Optionally record the MTP-excluded-layer combined with MIXED_PRECISION scenario as follow-up coverage outside this PR, targeting the existing DeepseekV4 modeling test file.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/unittest/_torch/modeling/test_modeling_deepseekv4.py`:
- Around line 447-503: Coverage for the mixed-precision expert gating and
per-expert override is sufficient; no code changes are required. Optionally
record the MTP-excluded-layer combined with MIXED_PRECISION scenario as
follow-up coverage outside this PR, targeting the existing DeepseekV4 modeling
test file.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 03219f51-8187-4120-ac91-0ad72132a39b
📒 Files selected for processing (2)
tensorrt_llm/_torch/models/modeling_deepseekv4.pytests/unittest/_torch/modeling/test_modeling_deepseekv4.py
|
/bot run |
|
PR_Github #59640 [ run ] triggered by Bot. Commit: |
|
PR_Github #59640 [ run ] completed with state
|
|
/bot run |
|
PR_Github #59743 [ run ] triggered by Bot. Commit: |
|
PR_Github #59743 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60211 [ run ] triggered by Bot. Commit: |
|
PR_Github #60211 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60295 [ run ] triggered by Bot. Commit: |
|
PR_Github #60295 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60541 [ run ] triggered by Bot. Commit: |
|
PR_Github #60541 [ run ] completed with state
|
Treat the non-expert weights in DeepSeek V4 NVFP4 checkpoints as FP8 block-scale weights while retaining the per-layer NVFP4 expert configuration. Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #60718 [ run ] triggered by Bot. Commit: |
|
PR_Github #60718 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60837 [ run ] triggered by Bot. Commit: |
|
PR_Github #60837 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60904 [ run ] triggered by Bot. Commit: |
|
PR_Github #60904 [ run ] completed with state |
…nnahz/dep-1083-port-flashinfer-stable-va-lifecycle-for-native-all-reduce * 'main' of https://github.com/NVIDIA/TensorRT-LLM: (83 commits) [None][feat] Bind SourceIdentity to checkpoint artifacts (NVIDIA#16159) [None][infra] Waive 4 failed cases for main in pre-merge 49550 (NVIDIA#16798) [None][fix] Make FlashInfer sampling op wrappers opaque to Dynamo (NVIDIA#16732) [None][feat] top-k: route decode to CuTe DSL GVR top-k in e2e (NVIDIA#16420) [None][feat] Default GLM-5 to the Python KV-cache transceiver (NVIDIA#16524) [None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor (NVIDIA#16422) [https://nvbugs/6426850][test] Unwaive Qwen3.5 397B NVFP4 ADP4 TRTLLM test (NVIDIA#16348) [https://nvbugs/6445456][fix] Restore inplace ops for functionalization v2 (NVIDIA#16410) [None][infra] Waive 1 failed cases for main in pre-merge 49229 (NVIDIA#16786) [None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints (NVIDIA#16433) [None][feat] ADP conversation router: configurable least-queued placement for new conversations (NVIDIA#16294) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16780) [None][infra] Waive 1 failed cases for main in pre-merge 49424 (NVIDIA#16781) [TRTLLMINF-188][infra] Require approval for PerfSanity wildcard runs (NVIDIA#16777) [TRTLLM-13969][feat] Support MiniMax M3 for Disaggregated Serving (NVIDIA#16017) [NVIDIA#11932][fix] Filter CUTLASS MoE GEMM tile configs by device shared memory on SM121 (NVIDIA#12704) [None][chore] Remove attention backend test waivers (NVIDIA#16723) [TRTLLM-14540][perf] Skip fp32 state round-trip in FlashInfer GDN pre… (NVIDIA#16703) [TRTLLM-13694][feat] Add IBDB recipe provenance and refresh configs (NVIDIA#16254) [None][infra] Preview/bump/main (NVIDIA#16758) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
…A#16433) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com> Co-authored-by: Fanrong Li <lfr-0531@users.noreply.github.com>
3 participants
@coderabbitai summary
Description
The
nvidia/DeepSeek-V4-Pro-NVFP4Hugging Face checkpoint uses a globalMIXED_PRECISIONquantization configuration, FP8 block-scale weights for non-routed-expert modules, and layer-specific NVFP4 configurations for routed experts. DeepSeek V4 previously rejected the global configuration during decoder initialization. After removing that rejection, loading the actual checkpoint also showed that the global mixed-precision setting caused FP8 base modules to be constructed without their required block-scale parameters.Following the per-module resolution approach in #12149, this PR keeps the routed experts' layer-specific NVFP4 configuration while disabling ambiguous layer-level NVFP4 fusions. For DeepSeek V4 checkpoints whose inline Hugging Face configuration declares FP8 128x128 block scales, it resolves the non-expert/base configuration to
FP8_BLOCK_SCALES, preserves runtime-populated quantization fields, and retains the NVFP4 expert entries inquant_config_dict.Regression tests cover both decoder construction with mixed-precision NVFP4 experts and FP8 base-config resolution, including cached quantization modes and the runtime
torch.dtypecache field. The change is model-specific and does not alter public APIs or add dependencies.Related PR: #12149
Test Coverage
python scripts/build_wheel.py --trt_root /usr/local/tensorrt --micro_benchmarks --use_ccache --cuda_architectures '90-real;100-real' --configure_cmake --job_count 64 --install --yes(passed)tensorrt_llm,modeling_deepseekv4.py, and rebuiltbindings.cpython-312-x86_64-linux-gnu.so.pytest -q tests/unittest/_torch/modeling/test_modeling_deepseekv4.py -k 'mixed_precision or fused_a_weight_scale or routed_moe_quant_config'(7 passed, 26 deselected)pytest -q tests/unittest/_torch/modeling/test_modeling_deepseekv4.py(33 passed)pytest -c tests/unittest/pytest.ini tests/unittest/llmapi/test_llm_quant.py -k "quant_cfg_from_quant_cfg_json or quant_cfg_top_level_overlay or quant_cfg_from_hf_quant_config or mixed_precision_kv_cache_conflict" -q(4 passed)nvidia/DeepSeek-V4-Pro-NVFP4, Hugging Face revision1449d1e641023406daf6b432361486c768aad74064/64safetensors shards and2686/2686parameters loaded on every rank.LLM(...)completed withLOAD_OK elapsed_seconds=687.9, followed bySHUTDOWN_OK.pre-commit run --files tensorrt_llm/_torch/models/modeling_deepseekv4.py tests/unittest/_torch/modeling/test_modeling_deepseekv4.py(passed)PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.