[None][fix] Restore DSV4 routed-expert swiglu_limit on TRTLLM-Gen#14238
Merged
lfr-0531 merged 1 commit intoMay 19, 2026
Merged
Conversation
Barry-Delaney
force-pushed
the
user/jinshik/fix_dsv4_swiglu_limit
branch
2 times, most recently
from
May 18, 2026 08:25
d9f9367 to
b578955
Compare
Barry-Delaney
marked this pull request as ready for review
May 18, 2026 09:06
…ideEP mxfp4 paths The pre-existing kernel_requires_bias_for_swiglu_limit guard was over-broad: it blocked swiglu_limit on the NVFP4 / W4A16_MXFP4 / W4A8_MXFP4_MXFP8 routed paths. The MXFP4 variants do not actually need bias — the GPT-OSS unit test test_moe_backend[backend=TRTLLM, quant=W4A8_MXFP4_MXFP8, ..., limit=7.0] already exercises the cubin clamp slot end-to-end for an MXFP4 quant. Narrow the guard to NVFP4 only, so MXFP4 routed experts now pass per-expert swiglu_limit through to the cubin clamp slot, matching the CUTLASS path and the shared-expert GatedMLP. NVFP4 stays guarded: empirically the TRTLLM-Gen fp4-block-scale cubin variant used for the NVFP4 routed path does not produce DSV4's no-bias clamped SwiGLU when bias is nullptr even with swiglu_limit set (GSM8K on DeepSeek-V4-Flash NVFP4 with TP=4/EP=4: 95.34% with the guard vs 0.76% with the guard fully removed). This means the cubin's actual behaviour for the NVFP4 slot diverges from the nullptr-default contract documented in cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/ trtllmGen_bmm_export/KernelParamsDecl.h:244-261; until that path grows a no-bias clamp variant (or the swiglu_limit / fc31_alpha rescale in tensorrt_llm/_torch/modules/fused_moe/quantization.py:3679-3680 is re-derived for NVFP4), keep dropping the limit on the NVFP4 routed path. The shared-expert GatedMLP keeps applying the clamp via the standard linear stack, so NVFP4 retains accuracy. Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Barry-Delaney
force-pushed
the
user/jinshik/fix_dsv4_swiglu_limit
branch
from
May 18, 2026 09:06
b578955 to
780faba
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #48883 [ run ] triggered by Bot. Commit: |
lfr-0531
approved these changes
May 18, 2026
Collaborator
|
PR_Github #48883 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #48942 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #48942 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #48970 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #48970 [ run ] completed with state
|
Collaborator
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #49023 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #49023 [ run ] completed with state |
Barry-Delaney
added a commit
to Barry-Delaney/TensorRT-LLM
that referenced
this pull request
May 28, 2026
…a rescale into BaseMethod The TRTLLM-Gen NVFP4 fp4-block-scale cubin's fused-activation clamp slot expects ptrClampLimit in the same quantized domain as the pre-dequant GEMM output (i.e. divided by fc31_alpha). Previously only the gptoss subclass NVFP4TRTLLMGenFusedMoEMethod applied that rescale; the DSv4 routed-expert path (default SwiGLU, no custom alpha/beta) dispatches to NVFP4TRTLLMGenFusedMoEBaseMethod which skipped it. Cubin therefore saw the raw limit (e.g. 10.0) and produced near-zero outputs, tanking GSM8K from 95.34% to 0.76% (PR NVIDIA#14238 / commit 780faba data). Fix: hoist the swiglu_limit rescale into the BaseMethod's process_weights_after_loading so all NVFP4 TRTLLM-Gen subclasses pick it up via super(). The subclass keeps its own bias / w2_bias / swiglu_beta rescales (gptoss-only) so the W4A8 sibling's behavior is unchanged. The kernel_requires_bias_for_swiglu_limit guard in modeling_deepseekv4.py is narrowed to WideEPMoE only, since TRTLLMGenFusedMoE no longer needs it. Unit coverage: add test_fused_moe_nvfp4_dsv4_clamp_only (bias=off, swiglu_limit ∈ {1.0, 7.0, 10.0}, no custom alpha/beta) which hits the bias=off BaseMethod path that test_fused_moe_nvfp4_gptoss_style never reaches. Without the fix the test fails at 67-86 percent element match; with the fix it passes at >95 percent. The gptoss-style cases (8 combos) still pass with the rescale hoisted, confirming no regression. Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Barry-Delaney
added a commit
to Barry-Delaney/TensorRT-LLM
that referenced
this pull request
May 28, 2026
…a rescale into BaseMethod The TRTLLM-Gen NVFP4 fp4-block-scale cubin's fused-activation clamp slot expects ptrClampLimit in the same quantized domain as the pre-dequant GEMM output (i.e. divided by fc31_alpha). Previously only the gptoss subclass NVFP4TRTLLMGenFusedMoEMethod applied that rescale; the DSv4 routed-expert path (default SwiGLU, no custom alpha/beta) dispatches to NVFP4TRTLLMGenFusedMoEBaseMethod which skipped it. Cubin therefore saw the raw limit (e.g. 10.0) and produced near-zero outputs, tanking GSM8K from 95.34% to 0.76% (PR NVIDIA#14238 / commit 780faba data). Fix: hoist the swiglu_limit rescale into the BaseMethod's process_weights_after_loading so all NVFP4 TRTLLM-Gen subclasses pick it up via super(). The subclass keeps its own bias / w2_bias / swiglu_beta rescales (gptoss-only) so the W4A8 sibling's behavior is unchanged. The kernel_requires_bias_for_swiglu_limit guard in modeling_deepseekv4.py is narrowed to WideEPMoE only, since TRTLLMGenFusedMoE no longer needs it. Unit coverage: add test_fused_moe_nvfp4_dsv4_clamp_only (bias=off, swiglu_limit ∈ {1.0, 7.0, 10.0}, no custom alpha/beta) which hits the bias=off BaseMethod path that test_fused_moe_nvfp4_gptoss_style never reaches. Without the fix the test fails at 67-86 percent element match; with the fix it passes at >95 percent. The gptoss-style cases (8 combos) still pass with the rescale hoisted, confirming no regression. Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
May 29, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 1, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 3, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 7, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 10, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 11, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 12, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 12, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 13, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 16, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
lfr-0531
pushed a commit
to lfr-0531/TensorRT-LLM
that referenced
this pull request
Jun 17, 2026
…IDIA#14238) Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com> (cherry picked from commit 552de9e) Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Drop the kernel_requires_bias_for_swiglu_limit guard so DSV4 routed experts pass swiglu_limit through on the TRTLLM-Gen / WideEP fp4 + mxfp4 paths, matching the CUTLASS path and the shared-expert GatedMLP.
Details
The guard was added on the assumption that the trtllm-gen fp4 block-scale fused-activation cubin needed the full GPT-OSS contract (bias + swiglu_alpha + swiglu_beta + swiglu_limit) and would emit all-zero outputs without bias. The exported cubin header documents sane nullptr defaults (cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParamsDecl.h:244-261):
ptrGatedActAlpha == nullptr -> alpha = 1.0
ptrGatedActBeta == nullptr -> beta = 0.0
ptrBias == nullptr -> bias = 0
ptrClampLimit == nullptr -> clamp = +inf
With those defaults the fused activation evaluates silu(x_glu.clamp(max=L)) * x_linear.clamp(-L, L), i.e. DSV4's clamped no-bias SwiGLU. The cubin slot is already covered end-to-end by test_moe_backend[backend=TRTLLM, quant=W4A8_MXFP4_MXFP8, ..., limit=7.0].
Test plan