[TRTLLM-14822][feat] deprecate WIDEEP MoE backend by xxi-nv · Pull Request #17119 · NVIDIA/TensorRT-LLM

xxi-nv · 2026-07-31T09:11:59Z

Summary

Deprecate the WIDEEP MoE backend by raising an explicit error in both get_moe_cls and create_moe_backend, so it cannot be selected via create_moe nor by calling create_moe_backend directly. Wide expert parallelism and EPLB stay available through the remaining backends (DEEPGEMM for FP8 block-scale checkpoints, TRTLLM / CUTEDSL / CUTLASS otherwise).

AUTO never resolves to WIDEEP, so no implicit user path is affected.

Why the FP8 EPLB tests can move off WIDEEP

TestDeepSeekV4FlashBase FP8 EPLB cases used to require WIDEEP because the default CUTLASS FP8 block-scale path (CutlassFp8BlockScaleGemmRunner::moeGemm) takes a Hopper-only wgmma route with no SM100/SM103 implementation. WIDEEP avoided this via an is_sm_100f() check that switches to DeepGEMM. DEEPGEMM reaches the same kernel directly, so those cases now use DEEPGEMM.

Other updates

Examples, docs and CLI choices that offered WIDEEP now point at CUTEDSL / DEEPGEMM.
Test list entries parametrized on WIDEEP are remapped to an equivalent case rather than deleted; see Test list handling below.
WideEPMoE is dropped from the mirrored swiglu_limit capability tuple in modeling_deepseekv4.py to keep it consistent with create_moe.py.
The layer_wise_benchmarks --scaled-from test is skipped: that feature is implemented by monkeypatching WideEPMoE.select_alltoall_method_type.

Test list handling

Entries are remapped to an equivalent case rather than deleted, so no coverage dimension is lost. In llm_function_core.txt the WIDEEP entries were the only coverage of mtp_nextn=0 and fp8kv=False, so plain deletion would have dropped those dimensions entirely.

List	Original `WIDEEP` entry	Now
`llm_function_core.txt`	`test_bfloat16_4gpus_online_eplb[mtp_nextn=0-...]`	`[mtp_nextn=0-moe_backend=CUTLASS]`
`llm_function_core.txt`	`test_bfloat16_4gpus_online_eplb[mtp_nextn=2-...]`	`[mtp_nextn=2-moe_backend=TRTLLM]`
`llm_function_core.txt`	`test_nvfp4_4gpus_online_eplb[fp8kv=False-...]`	`[fp8kv=False-moe_backend=TRTLLM]`
`llm_function_core.txt`	`test_nvfp4_4gpus_online_eplb[fp8kv=True-...]`	dropped, the `TRTLLM` variant was already listed
`llm_function_rtx6k.txt`	`test_bfloat16_4gpus_online_eplb[mtp_nextn=0/2-...]`	`[...-moe_backend=CUTLASS]`
`llm_function_rtx6k.txt`	`test_nvfp4_4gpus_online_eplb[fp8kv=False/True-...]`	dropped, no SM120 equivalent (below)
`l0_dgx_b200.yml`	commented `test_*_8gpus_static_eplb[moe_backend=WIDEEP]`	commented `test_nvfp4_4gpus_static_eplb[moe_backend=TRTLLM]` and `test_fp8_4gpus_static_eplb[moe_backend=DEEPGEMM]`; the `8gpus` variants do not exist in the source
`waives.txt`	`test_bfloat16_4gpus_online_eplb[mtp_nextn=2-...] SKIP (nvbugs/6313993)`	removed, the waiver targeted the `WIDEEP` combination only

rtx6k has no equivalent for nvfp4 online EPLB. With WIDEEP gone, that test's parametrize list is TRTLLM-only, and TRTLLMGenFusedMoE explicitly rejects SM120 (fused_moe_trtllm_gen.py: "does not support SM120 and above"). WIDEEP was the only backend that could run this combination on SM120, so the two entries are dropped rather than remapped. The test is additionally gated by @pytest.mark.skip_device_not_contain(["GB200"]), so those entries never executed on that platform.

The gb200_wideep_* / gb300_wideep_* perf entries are unrelated to this deprecation and are kept. There the wideep token is only a filename prefix describing the wide-EP deployment shape (dep32 + eplb288); the configs under tests/scripts/perf/disaggregated/ already set moe_config.backend: CUTEDSL. An earlier revision of this PR removed those ten entries from llm_perf_disagg.yml and llm_perf_multinode.txt by mistake; they are restored.

Test plan

Ran the full TestDeepSeekV4FlashBase class on 4x B300 (SM103), wheel built with -a "100-real;103-real", using TRTLLM_ACCURACY_NO_REFERENCE=1 (this model has no registered MMLU/GSM8K reference entries):

test_fp8_4gpus_static_eplb[moe_backend=DEEPGEMM] - PASSED (15m09s)
test_auto_dtype[moe_backend=DEEPGEMM] - PASSED (7m46s)
test_auto_dtype[moe_backend=TRTLLM] - PASSED (7m58s)
test_fp8_chunked_prefill - PASSED (7m18s)
test_fp8_4gpus_online_eplb[moe_backend=DEEPGEMM] - fails on a pre-existing incompatibility, see below

Note on the online EPLB case

test_fp8_4gpus_online_eplb fails with AssertionError: t.is_contiguous() must be True at moe_load_balancer.py:151. This is not introduced by this PR:

On SM100/SM103, WideEPMoE._get_quant_method returns the same DeepSeekFP8BlockScalesFusedMoEMethodDeepGemm that DeepGemmFusedMoE returns, so switching the backend does not change this code path.
_needs_e8m0_resmooth() is true for all Blackwell architectures, and the resulting scale tensor from transform_sf_into_required_layout is non-contiguous, which the online EPLB host tensor sharer rejects. static EPLB passes layer_updates_per_iter=0 and short-circuits that requirement.
The class is gated by @skip_pre_blackwell, so every platform that can run this case hits the same assertion, including the previous WIDEEP parametrization.
The case is not referenced by any CI test list, so it was never covered by CI.

Fixing it requires deciding whether forcing contiguity is safe for the DeepGEMM scale layout, which is a separate concern from this deprecation.

Dev Engineer Review

WIDEEP deprecation is applied across code, configuration, examples, documentation, benchmarks, and tests.
get_moe_cls and create_moe_backend now reject WIDEEP with an explicit error.
Supported backends retain wide expert parallelism and EPLB support.
Configuration values and test-list formats are consistent.
The online FP8 EPLB test remains blocked by a pre-existing non-contiguous scale tensor incompatibility.

QA Engineer Review

Updated EPLB backend coverage in test_llm_api_pytorch.py.
Updated backend selections in test_disaggregated_serving.py and test_e2e.py.
Updated LoRA validator cases in test_moe_lora_validator.py.
Skipped test_deepseek_r1_gen_scaled_from_16_dep because --scaled-from lacks a compatible all-to-all selection hook.
Updated llm_function_core.txt, llm_function_rtx6k.txt, and l0_dgx_b200.yml.
Removed the obsolete waiver from waives.txt.
The modified test code has corresponding CI or QA test-list coverage where applicable.
Static FP8 EPLB, automatic dtype, and FP8 chunked prefill tests passed on 4x B300.
The online FP8 EPLB test remains affected by the known scale tensor issue.

Verdict: sufficient

tensorrt-cicd · 2026-08-03T20:07:38Z

PR_Github #63543 [ run ] completed with state FAILURE. Commit: 6ef6efe
/LLM/main/L0_MergeRequest_PR pipeline #51509 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-08-03T20:10:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T20:16:58Z

PR_Github #63551 [ run ] triggered by Bot. Commit: 6ef6efe Link to invocation

tensorrt-cicd · 2026-08-03T20:42:47Z

PR_Github #63551 [ run ] completed with state FAILURE. Commit: 6ef6efe
/LLM/main/L0_MergeRequest_PR pipeline #51517 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-08-03T20:44:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-03T20:49:50Z

PR_Github #63556 [ run ] triggered by Bot. Commit: 6ef6efe Link to invocation

tensorrt-cicd · 2026-08-03T21:16:13Z

PR_Github #63556 [ run ] completed with state FAILURE. Commit: 6ef6efe
/LLM/main/L0_MergeRequest_PR pipeline #51522 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-08-04T00:06:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-04T00:12:34Z

PR_Github #63587 [ run ] triggered by Bot. Commit: 5dab9bf Link to invocation

tensorrt-cicd · 2026-08-04T03:55:36Z

PR_Github #63587 [ run ] completed with state FAILURE. Commit: 5dab9bf
/LLM/main/L0_MergeRequest_PR pipeline #51551 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-08-04T04:11:07Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-04T04:18:14Z

PR_Github #63652 [ run ] triggered by Bot. Commit: 5dab9bf Link to invocation

tensorrt-cicd · 2026-08-04T07:23:12Z

PR_Github #63652 [ run ] completed with state FAILURE. Commit: 5dab9bf
/LLM/main/L0_MergeRequest_PR pipeline #51607 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xxi-nv · 2026-08-04T07:36:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-04T07:41:38Z

PR_Github #63705 [ run ] triggered by Bot. Commit: 5dab9bf Link to invocation

tensorrt-cicd · 2026-08-04T11:12:31Z

PR_Github #63705 [ run ] completed with state SUCCESS. Commit: 5dab9bf
/LLM/main/L0_MergeRequest_PR pipeline #51657 completed with status: 'SUCCESS'

CI Report

Link to invocation

Raise an explicit error for the WIDEEP MoE backend in both get_moe_cls and create_moe_backend, so the backend cannot be selected through create_moe or by calling create_moe_backend directly. Wide expert parallelism and EPLB remain available through the other backends. TestDeepSeekV4FlashBase FP8 EPLB cases previously required WIDEEP because CutlassFp8BlockScaleGemmRunner takes a Hopper-only wgmma path with no SM100/SM103 implementation. DEEPGEMM routes the same block-scale weights through DeepGEMM, so those cases now use DEEPGEMM. Examples, docs and test lists that referenced WIDEEP are updated to CUTEDSL or DEEPGEMM accordingly. The layer_wise_benchmarks --scaled-from test is skipped because that feature is implemented by monkeypatching WideEPMoE.select_alltoall_method_type. Signed-off-by: xxi <xxi@nvidia.com>

…tion The perf disagg entries named gb200_wideep_* / gb300_wideep_* were removed by mistake. The wideep token is only a filename prefix describing the wide-EP deployment shape (dep32 + eplb288); those configs already set moe_config.backend: CUTEDSL and are unaffected by the WIDEEP deprecation. All ten entries are restored in llm_perf_disagg.yml and llm_perf_multinode.txt. For the DeepSeekV3Lite online EPLB cases the WIDEEP entries were the only coverage of mtp_nextn=0 and fp8kv=False, so deleting them dropped those dimensions entirely. They are now replaced with the equivalent backends still present in each test parametrize list rather than deleted. rtx6k keeps no nvfp4 online EPLB entry: that test parametrize is now TRTLLM-only and TRTLLMGenFusedMoE rejects SM120, so no equivalent backend exists on that platform. The commented-out DeepSeek-V4 EPLB sanity entries now point at the tests that actually exist (4gpus variants) instead of being dropped. Signed-off-by: xxi <xxi@nvidia.com>

WIDEEP dispatched FP8 block-scale MoE on is_sm_100f() internally: Cutlass for SM90/SM120, DeepGEMM for SM100/SM103. Hardcoding DEEPGEMM after the deprecation dropped both Cutlass platforms, because DeepGemmFusedMoE only implements SM100/SM103 and the DEEPGEMM dispatch branch has no SM fallback. Select the backend at runtime with the same is_sm_100f() condition so the test keeps its original per-platform coverage. Signed-off-by: xxi <xxi@nvidia.com>

…uide ep_load_balancer/README.md kept three runnable YAML snippets on the WIDEEP backend. They were doubly invalid: the top-level moe_backend key is not an LlmArgs field (BaseLlmArgs sets extra=forbid) and WideEP does not match the strict Literal on MoeConfig.backend. Drop the stray key and move the snippets to CUTEDSL, which supports FP8 block scales on SM>=90 and is in the EPLB backend allowlist. layer_wise_benchmarks/README.md advertised WIDEEP in six commands and two support notes. TRTLLM_FORCE_ALLTOALL_METHOD is read only by WideEPMoE, so the DeepEP examples now use TRTLLM_FORCE_COMM_METHOD, which CommunicationFactory reads on the ConfigurableMoE path. The supported-backend list is corrected to match runner.py. MOE_DEVELOPER_GUIDE.md described WideEPMoE as Deprecating; it is now rejected outright. Signed-off-by: xxi <xxi@nvidia.com>

…eference Fold in the fix from the closed PR NVIDIA#17210. The checkpoint was missing from references/gsm8k.yaml, so the spec lookup raised Not registered specs and test_fp8_4gpus_static_eplb ended up waived on all four Blackwell platforms. The three measurements recorded there came from the WIDEEP variant this branch deprecates. A DEEPGEMM run on 4x B300 scored 91.13, inside that spread, which is expected because both backends share the DeepGEMM FP8 block-scale path on SM100/SM103. The four waivers go away now that the reference resolves. Signed-off-by: xxi <xxi@nvidia.com>

xxi-nv · 2026-08-04T12:49:27Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-04T12:54:58Z

PR_Github #63756 [ run ] triggered by Bot. Commit: e7e9da5 Link to invocation

xxi-nv requested review from a team as code owners July 31, 2026 09:12

xxi-nv requested review from Shixiaowei02, arysef, aswinvisva, dc3671, hyukn, reasonsolo, rosong11, ruodil, sunnyqgg and xinhe-nv July 31, 2026 09:12

github-actions Bot assigned xxi-nv Jul 31, 2026

xxi-nv requested review from JunyiXu-nv, QiJune, allisonlim-nv, schetlur-nv, yingguo-trt and yufeiwu-nv July 31, 2026 09:12

xxi-nv force-pushed the deprecate-wideep-moe branch from 265fc80 to 5dab9bf Compare August 4, 2026 00:03

xxi-nv requested a review from yingguo-trt August 4, 2026 01:27

QiJune approved these changes Aug 4, 2026

View reviewed changes

yingguo-trt approved these changes Aug 4, 2026

View reviewed changes

yufeiwu-nv removed their request for review August 4, 2026 02:32

xxi-nv enabled auto-merge (squash) August 4, 2026 10:10

xxi-nv added 5 commits August 4, 2026 12:42

xxi-nv force-pushed the deprecate-wideep-moe branch from 5dab9bf to e7e9da5 Compare August 4, 2026 12:47

Uh oh!

Conversation

xxi-nv commented Jul 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why the FP8 EPLB tests can move off WIDEEP

Other updates

Test list handling

Test plan

Note on the online EPLB case

Dev Engineer Review

QA Engineer Review

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

xxi-nv commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

xxi-nv commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

xxi-nv commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

xxi-nv commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

xxi-nv commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

xxi-nv commented Aug 4, 2026

Uh oh!

tensorrt-cicd commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

xxi-nv commented Jul 31, 2026 •

edited by coderabbitai Bot

Loading