Skip to content

[None][feat] Support NVFP4 dsv4 - #14026

Merged
lfr-0531 merged 7 commits into
NVIDIA:feat/deepseek_v4from
Tracin:nvfp4_dsv4
May 13, 2026
Merged

[None][feat] Support NVFP4 dsv4#14026
lfr-0531 merged 7 commits into
NVIDIA:feat/deepseek_v4from
Tracin:nvfp4_dsv4

Conversation

@Tracin

@Tracin Tracin commented May 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Tracin and others added 3 commits May 11, 2026 20:19
@Tracin
DSV4 support NVFP4.
bb8d935 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Barry-Delaney @Tracin
DSv4 NVFP4 routed-experts all-zero: skip swiglu_limit on TRTLLM-Gen F…
28e62af 
…P4 block-scale kernel without bias

The TRTLLM-Gen fp4-block-scale fused-MoE kernel (`run_fp4_block_scale_moe`
in `fused_moe_trtllm_gen.py:760`, used for NVFP4 / W4A16-MXFP4 /
W4A8-MXFP4-MXFP8) only implements the GPT-OSS-style SwiGLU clamping
path, which expects `bias`, `swiglu_alpha`, `swiglu_beta` to be set
alongside `swiglu_limit`. DeepSeek-V4 has no FFN bias, so passing
`swiglu_limit` through that kernel collapses the routed-experts output
to all zeros.

`tests/unittest/_torch/modules/test_fused_moe.py::test_fused_moe_nvfp4`
only exercises `swiglu_limit` via `gptoss_style=True` (which also sets
the bias and alpha/beta tensors), so this bias-free combination is
uncovered by tests.

Minimal repro (single-rank, layer-0 only):
  python scripts/dsv4_nvfp4_one_layer_repro.py --routing dsv4-hashed
  python scripts/dsv4_nvfp4_one_layer_repro.py --routing dsv4-hashed \
                                               --swiglu-limit 10.0
The only difference between these two invocations is the
`--swiglu-limit` flag, and that single change flips the routed-MoE
output absmax from ~0.17 to 0.

Workaround applied here: when `moe_cls` resolves to TRTLLMGenFusedMoE
or WideEPMoE and the experts quant mode uses any FP4 block-scale
variant, do not construct the `moe_swiglu_limit` tensor. The shared
experts still apply the limit via the standard `GatedMLP` linear stack
below, so only the routed-experts pre-activation clamping is dropped.

Verified on a 2-layer truncated `flash-nvfp4-experts-v3.5` checkpoint
via the DSv4 dump infra: `layer.0.routed_output` went from absmax=0,
nnz=0/32768 to absmax=0.64, nnz=32768/32768.

The proper long-term fix is on the kernel side -- either add a no-bias
clamping variant to `run_fp4_block_scale_moe`, have the kernel skip the
clamp when bias is None, or enforce that all four (bias, swiglu_alpha,
swiglu_beta, swiglu_limit) be set together and gate the construction at
the Python level so the silent-zero failure mode becomes impossible.

Signed-off-by: Barry Kang <jinshik@nvidia.com>
@Tracin
DSV4 support NVFP4.
1994149 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Tracin
Tracin requested review from a team as code owners May 12, 2026 03:20
@Tracin
Tracin requested review from HuiGao-NV and syuoni and removed request for a team May 12, 2026 03:20
Tracin added 4 commits May 11, 2026 20:24
@Tracin
Rebase branch.
9c0b9f9 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Tracin
Make code clean.
fa6226a 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Tracin
Make code clean.
864191d 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Tracin
Make code clean.
e8ce5c7 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
@Tracin

Tracin commented May 13, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@lfr-0531

lfr-0531 commented May 13, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48076 [ run ] triggered by Bot. Commit: e8ce5c7 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48077 [ run ] triggered by Bot. Commit: e8ce5c7 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48076 [ run ] completed with state ABORTED. Commit: e8ce5c7

Link to invocation

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48077 [ run ] completed with state SUCCESS. Commit: e8ce5c7
/LLM/main/L0_MergeRequest_PR pipeline #37907 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@Tracin

Tracin commented May 13, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48115 [ run ] triggered by Bot. Commit: e8ce5c7 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented May 13, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #48115 [ run ] completed with state SUCCESS. Commit: e8ce5c7
/LLM/main/L0_MergeRequest_PR pipeline #37943 completed with status: 'SUCCESS'

CI Report

Link to invocation

@lfr-0531
lfr-0531 merged commit d9aba31 into NVIDIA:feat/deepseek_v4 May 13, 2026
5 checks passed
lfr-0531 pushed a commit that referenced this pull request May 14, 2026
@Tracin @Barry-Delaney @heyuhhh
[None][feat] Support NVFP4 dsv4 (#14026)
13888d2 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
@lancelly lancelly mentioned this pull request May 18, 2026
Merged
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request May 29, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
cdf2e71 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 1, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
74b6a2c 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 3, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
6a23e08 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 7, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
368ccf1 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
@Tracin
Tracin deleted the nvfp4_dsv4 branch June 9, 2026 09:56
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 10, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
bb3629d 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 11, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
8f89870 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 12, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
fd75cdc 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 12, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
60c08d6 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 13, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
2fdb74a 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 16, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
8bee6f8 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 17, 2026
@Tracin @Barry-Delaney @lfr-0531
[None][feat] Support NVFP4 dsv4 (NVIDIA#14026)
8faaeb4 
Signed-off-by: Tracin <10434017+Tracin@users.noreply.github.com>
Signed-off-by: Barry Kang <jinshik@nvidia.com>
Co-authored-by: Barry Kang <jinshik@nvidia.com>
(cherry picked from commit 13888d2)
Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@syuoni syuoni Awaiting requested review from syuoni syuoni is a code owner automatically assigned from NVIDIA/trt-llm-llmapi-devs
@HuiGao-NV HuiGao-NV Awaiting requested review from HuiGao-NV HuiGao-NV was automatically assigned from NVIDIA/trt-llm-torch-modules

Assignees

@Tracin Tracin

@lfr-0531 lfr-0531

Labels

deepseek-v4

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Tracin @lfr-0531 @tensorrt-cicd @Barry-Delaney