[None][feat] Support MARLIN MoE with MTP and attention DP + EP - #16597
Conversation
1344440 to
76711d2
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #60283 [ run ] triggered by Bot. Commit: |
WalkthroughMarlin MoE selection now falls back to Cutlass for unsupported quantization and uses precomputed routing for external communication. LM-head and per-layer KV-cache quantization handling are updated, with focused unit and integration coverage. ChangesMarlin backend and routing flow
Quantization configuration consistency
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ModelConfig
participant get_moe_cls
participant ExternalCommMoEScheduler
participant routing_method
participant MarlinFusedMoE
ModelConfig->>get_moe_cls: provide MARLIN and quantization configuration
get_moe_cls->>MarlinFusedMoE: select NVFP4 Marlin backend or Cutlass fallback
ExternalCommMoEScheduler->>routing_method: precompute routing for active communication
routing_method-->>ExternalCommMoEScheduler: return routing and scales
ExternalCommMoEScheduler->>MarlinFusedMoE: dispatch routed tokens
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8)tests/integration/test_lists/test-db/l0_dgx_h100.ymlTraceback (most recent call last): tests/integration/test_lists/test-db/l0_dgx_h200.ymlTraceback (most recent call last): Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
tests/unittest/_torch/modules/moe/test_moe_module.py (1)
1380-1381: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid shadowing the helper inputs.
Lines 1380-1381 overwrite
backend_typeandquant_algopassed to the helper. Rename the generated case values and use those for the per-case skip check and parameter tuple.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/moe/test_moe_module.py` around lines 1380 - 1381, Rename the generated case values in the test helper instead of overwriting its backend_type and quant_algo inputs. Use the renamed values consistently for the per-case skip check and the parameter tuple.Source: Coding guidelines
tests/unittest/_torch/modules/moe/test_moe_backend.py (1)
334-338: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAnnotate the new reusable helper interfaces.
Both helpers are new non-test APIs and should use precise parameter and return types, including built-in parameterized generics.
tests/unittest/_torch/modules/moe/test_moe_backend.py#L334-L338: annotatequant_algoand returnModelConfig.tests/unittest/_torch/modules/moe/test_moe_module.py#L1346-L1353: annotate all inputs, use a preciseCallable[...] | Noneforshould_skip_fn, and replace bareListwith a parameterized built-in generic.As per coding guidelines, “Annotate every function” and “prefer built-in generic types.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/moe/test_moe_backend.py` around lines 334 - 338, Annotate _marlin_model_config in tests/unittest/_torch/modules/moe/test_moe_backend.py:334-338 with the precise QuantAlgo | None parameter type and ModelConfig return type. In tests/unittest/_torch/modules/moe/test_moe_module.py:1346-1353, annotate every helper parameter, use a precise Callable[...] | None type for should_skip_fn, and replace bare List with an appropriate parameterized built-in generic.Source: Coding guidelines
tests/unittest/_torch/test_model_config.py (1)
137-166: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd return annotations to the new helper and tests.
Annotate
_make_mixed_precision_model_config()and both test functions (-> ModelConfig/-> None). Coverage intests/unittest/_torch/test_model_config.pyis otherwise sufficient for the explicit override andautopaths. As per coding guidelines, “Annotate every function.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/test_model_config.py` around lines 137 - 166, Add return annotations to _make_mixed_precision_model_config, returning ModelConfig, and to both test_validate_and_set_kv_cache_quant_propagates_to_quant_config_dict and test_validate_and_set_kv_cache_quant_auto_keeps_quant_config_dict, returning None. Preserve their existing behavior and assertions.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/models/modeling_utils.py`:
- Around line 390-418: Clear lm_head_quant_config when both
config.mapping.enable_attention_dp and config.mapping.enable_lm_head_tp_in_adp
are enabled, before constructing LMHead, so LM-head TP under attention DP always
uses a dense weight. Preserve the existing exclusion and tied-embedding checks
for other modes.
---
Nitpick comments:
In `@tests/unittest/_torch/modules/moe/test_moe_backend.py`:
- Around line 334-338: Annotate _marlin_model_config in
tests/unittest/_torch/modules/moe/test_moe_backend.py:334-338 with the precise
QuantAlgo | None parameter type and ModelConfig return type. In
tests/unittest/_torch/modules/moe/test_moe_module.py:1346-1353, annotate every
helper parameter, use a precise Callable[...] | None type for should_skip_fn,
and replace bare List with an appropriate parameterized built-in generic.
In `@tests/unittest/_torch/modules/moe/test_moe_module.py`:
- Around line 1380-1381: Rename the generated case values in the test helper
instead of overwriting its backend_type and quant_algo inputs. Use the renamed
values consistently for the per-case skip check and the parameter tuple.
In `@tests/unittest/_torch/test_model_config.py`:
- Around line 137-166: Add return annotations to
_make_mixed_precision_model_config, returning ModelConfig, and to both
test_validate_and_set_kv_cache_quant_propagates_to_quant_config_dict and
test_validate_and_set_kv_cache_quant_auto_keeps_quant_config_dict, returning
None. Preserve their existing behavior and assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 4326179f-562e-41ad-8dad-e469f66934b0
📒 Files selected for processing (10)
examples/llm-api/quickstart_advanced.pytensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.mdtensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.pytensorrt_llm/_torch/modules/fused_moe/moe_scheduler.pytensorrt_llm/_torch/pyexecutor/model_loader.pytests/unittest/_torch/modules/moe/test_moe_backend.pytests/unittest/_torch/modules/moe/test_moe_module.pytests/unittest/_torch/test_model_config.py
💤 Files with no reviewable changes (1)
- tensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.py
|
need to add an accuracy guard for the feature (MARLIN moe + MTP + ADP) |
|
You may add end-to-end accuracy test for NT models in https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/integration/defs/accuracy/test_llm_api_pytorch.py with the new ADP and MTP features. Also update https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/integration/test_lists/qa/llm_function_core.txt and https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/integration/test_lists/test-db/l0_dgx_h200.yml if you add new test methods. |
BowenFu
left a comment
BowenFu
left a comment
There was a problem hiding this comment.
LGTM — MARLIN fallback + attention-DP/EP wiring are MARLIN-scoped; the lm_head quant hoist and per-layer kv-cache propagation are gated on quant_config_dict (MIXED_PRECISION checkpoints, None for normal models); and moe_scheduler.py's 'or moe.comm is not None' only affects the external-comm flow (which already requires precomputed routing). No default Cutlass/TRTLLM MoE path change.
|
PR_Github #60685 [ run ] completed with state
|
14f32b3 to
4903f2a
Compare
|
/bot run --disable-fail-fast |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/integration/defs/accuracy/test_llm_api_pytorch.py (1)
7026-7027: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
max_draft_lenhere.
num_nextn_predict_layersis deprecated/internal and only kept for compatibility; passmax_draft_len=mtp_nextninstead.Test coverage summary:
test_nvfp4_marlin_adp_4gpus[mtp_nextn=3]is listed intests/integration/test_lists/qa/llm_function_core.txtandtests/integration/test_lists/test-db/l0_dgx_h200.yml, but not in the H100 test-db list. Coverage verdict: insufficient for H100 CI.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py` around lines 7026 - 7027, Update the MTPDecodingConfig construction to pass mtp_nextn through the public max_draft_len parameter instead of the deprecated num_nextn_predict_layers parameter, while preserving mtp_eagle_one_model=True. Add the corresponding test case to the H100 test-db list so test_nvfp4_marlin_adp_4gpus[mtp_nextn=3] is covered in H100 CI.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/test-db/l0_dgx_h100.yml`:
- Around line 171-178: Add the new accuracy-guard test entry to the H100 CI list
alongside the existing CUTLASS and MARLIN filters, using the same pytest target
and filter registration used in QA or l0_dgx_h200.yml. Keep the Hopper-only
backend selection unchanged and ensure the accuracy guard runs in the 4x H100
test matrix.
---
Nitpick comments:
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py`:
- Around line 7026-7027: Update the MTPDecodingConfig construction to pass
mtp_nextn through the public max_draft_len parameter instead of the deprecated
num_nextn_predict_layers parameter, while preserving mtp_eagle_one_model=True.
Add the corresponding test case to the H100 test-db list so
test_nvfp4_marlin_adp_4gpus[mtp_nextn=3] is covered in H100 CI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: feace091-0e1e-496c-9cec-b943712a61c3
📒 Files selected for processing (13)
tensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.mdtensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.pytensorrt_llm/_torch/modules/fused_moe/moe_scheduler.pytensorrt_llm/_torch/pyexecutor/model_loader.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/l0_dgx_h100.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/unittest/_torch/modules/moe/test_moe_backend.pytests/unittest/_torch/modules/moe/test_moe_module.pytests/unittest/_torch/test_model_config.py
🚧 Files skipped from review as they are similar to previous changes (8)
- tensorrt_llm/_torch/modules/fused_moe/moe_scheduler.py
- tensorrt_llm/_torch/modules/fused_moe/create_moe.py
- tensorrt_llm/_torch/pyexecutor/model_loader.py
- tensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.py
- tensorrt_llm/_torch/models/modeling_utils.py
- tests/unittest/_torch/modules/moe/test_moe_backend.py
- tests/unittest/_torch/test_model_config.py
- tests/unittest/_torch/modules/moe/test_moe_module.py
|
PR_Github #60896 [ run ] triggered by Bot. Commit: |
|
PR_Github #60896 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60992 [ run ] triggered by Bot. Commit: |
|
PR_Github #60992 [ run ] completed with state
|
4903f2a to
e10d405
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tensorrt_llm/_torch/modules/fused_moe/moe_scheduler.py (1)
377-384: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a regression test for external-communication routing.
This broadens separated routing to every
moe.commpath, but the supplied tests do not exerciseExternalCommMoESchedulerwith Marlin plus attention-DP/EP. Add a focused test asserting that routing is precomputed and its expert IDs/scales are passed to dispatch andrun_moe.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/fused_moe/moe_scheduler.py` around lines 377 - 384, Add a focused regression test for ExternalCommMoEScheduler covering MarlinFusedMoE with attention-DP and EP enabled. Assert that routing is precomputed and that the resulting expert IDs and scales are forwarded to both the external dispatch step and run_moe, using the existing scheduler test helpers and mocks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tensorrt_llm/_torch/modules/fused_moe/moe_scheduler.py`:
- Around line 377-384: Add a focused regression test for
ExternalCommMoEScheduler covering MarlinFusedMoE with attention-DP and EP
enabled. Assert that routing is precomputed and that the resulting expert IDs
and scales are forwarded to both the external dispatch step and run_moe, using
the existing scheduler test helpers and mocks.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 20782a06-900a-461e-8edf-39ef2e5ec13f
📒 Files selected for processing (13)
tensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.mdtensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.pytensorrt_llm/_torch/modules/fused_moe/moe_scheduler.pytensorrt_llm/_torch/pyexecutor/model_loader.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/l0_dgx_h100.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/unittest/_torch/modules/moe/test_moe_backend.pytests/unittest/_torch/modules/moe/test_moe_module.pytests/unittest/_torch/test_model_config.py
🚧 Files skipped from review as they are similar to previous changes (9)
- tests/integration/test_lists/qa/llm_function_core.txt
- tensorrt_llm/_torch/modules/fused_moe/create_moe.py
- tests/integration/test_lists/test-db/l0_dgx_h100.yml
- tensorrt_llm/_torch/models/modeling_utils.py
- tests/unittest/_torch/test_model_config.py
- tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md
- tensorrt_llm/_torch/pyexecutor/model_loader.py
- tests/integration/defs/accuracy/test_llm_api_pytorch.py
- tests/unittest/_torch/modules/moe/test_moe_module.py
Enable the MARLIN (SM90 W4A16-NVFP4) MoE backend to run with MTP speculative decoding and attention DP + expert parallelism: - create_moe.get_moe_cls: fall back to CutlassFusedMoE (warn instead of raise) for non-NVFP4 layers, so MIXED_PRECISION checkpoints with unquantized MTP draft layers keep main-model experts on MARLIN. - fused_moe_marlin: allow attention DP (drop the ConfigurableMoE rejection); document the EP non-local masking perf follow-up. - moe_scheduler: require separated routing whenever an external MoE communication strategy is attached, so scheduler-precomputed routing feeds the dispatch under attention DP + EP. - model_loader.validate_and_set_kv_cache_quant: propagate an explicit kv_cache_config.dtype override into the per-layer quant configs of MIXED_PRECISION checkpoints (KV pool allocated with the overridden dtype while attention layers used the checkpoint dtype caused out-of-bounds access); fix the return annotation to None. - modeling_utils.DecoderModelForCausalLM: resolve the per-layer lm_head quant entry once and pass it to both LMHead constructions (ADP-replicated and TP); exclude_modules, tie_word_embeddings, custom LoRA lm_head, and lm_head TP in ADP all keep lm_head dense. Tests: unit tests for per-layer kv propagation and MARLIN backend selection/fallback; a focused MARLIN DEP+ALLGATHER multi-GPU matrix selected in H100 pre-merge (l0_dgx_h100.yml); and an end-to-end accuracy guard TestNemotronV3Super::test_nvfp4_marlin_adp_4gpus (MMLU+GSM8K with tp4/ep4 + attention DP + MTP3 + fp8 KV) registered in l0_dgx_h200.yml and llm_function_core.txt. MoE developer guide updated. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
e10d405 to
dd26aa7
Compare
|
/bot run --disable-fail-fast |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unittest/_torch/modules/moe/test_moe_backend.py`:
- Around line 334-368: Add type annotations to _marlin_model_config, including
its quant_algo parameter and return type, and mark the three new test functions
in test_moe_backend.py as returning None. In
tests/unittest/_torch/modules/moe/test_moe_module.py lines 1346-1353, annotate
_generate_focused_multi_gpu_test_params parameters and return value, replacing
bare List with an appropriate parameterized built-in generic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3c35228a-a371-405f-a274-c636288881e5
📒 Files selected for processing (13)
tensorrt_llm/_torch/models/modeling_utils.pytensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.mdtensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.pytensorrt_llm/_torch/modules/fused_moe/moe_scheduler.pytensorrt_llm/_torch/pyexecutor/model_loader.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/l0_dgx_h100.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/unittest/_torch/modules/moe/test_moe_backend.pytests/unittest/_torch/modules/moe/test_moe_module.pytests/unittest/_torch/test_model_config.py
🚧 Files skipped from review as they are similar to previous changes (10)
- tests/integration/test_lists/qa/llm_function_core.txt
- tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md
- tensorrt_llm/_torch/modules/fused_moe/moe_scheduler.py
- tests/integration/test_lists/test-db/l0_dgx_h200.yml
- tensorrt_llm/_torch/modules/fused_moe/create_moe.py
- tensorrt_llm/_torch/models/modeling_utils.py
- tensorrt_llm/_torch/modules/fused_moe/fused_moe_marlin.py
- tensorrt_llm/_torch/pyexecutor/model_loader.py
- tests/unittest/_torch/test_model_config.py
- tests/integration/defs/accuracy/test_llm_api_pytorch.py
|
PR_Github #61154 [ run ] triggered by Bot. Commit: |
|
PR_Github #61154 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61252 [ run ] triggered by Bot. Commit: |
|
PR_Github #61252 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61458 [ run ] triggered by Bot. Commit: |
|
PR_Github #61458 [ run ] completed with state |
…A#16597) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
10 participants
Features
Enable serving NemotronH MTP-graft MIXED_PRECISION checkpoints with the MARLIN (W4A16 NVFP4, SM90) MoE backend:
Verified on 4x H100: multi-GPU module tests vs single-GPU reference and end-to-end trtllm-serve with tp4/ep4 attention DP + MARLIN + MTP + fp8 KV cache. Dynamic EPLB remains unsupported for Marlin.
Dev Engineer Review
get_moe_clsnow warns and falls back toCutlassFusedMoEwhen the backend isMARLINbut the layer/quant config is missing NVFP4 capability (instead of failing fast).MarlinFusedMoE.validate_configurable_moe, eliminating the prior rejection of MoE setups needing external all-to-all dispatch/combine whenmoe.use_dpanddp_size > 1.ExternalCommMoESchedulernow broadensrequires_separated_routingwhenmoe.commis active, forcing host-precomputed/separated routing and affecting whether/whenmoe.routing_method.apply(...)runs and the dispatch tensors produced.validate_and_set_kv_cache_quantnow mutatesmodel_config(returnsNone) and propagates mapped KV cache quantization into each per-layerkv_cache_quant_algowhenquant_config_dictis present."auto"leaving per-layerkv_cache_quant_algounset.lm_headquant-config handling for attention-DP:lm_head_quant_configonce and pass it into the attention-DPLMHead(...).quant_config_dictis used,lm_head_quant_configis read fromquant_config_dict.get("lm_head")and explicitly cleared when (a) the module is excluded from quantization, (b)tie_word_embeddingsis enabled, and (c) attention-DP withenable_attention_dp && enable_lm_head_tp_in_adp.lm_headis detected,lm_head_quant_configis forced toNoneto avoid unintended quantization of the custom head.QA Engineer Review
Test code changes
Unit tests (MoE backend selection):
tests/unittest/_torch/modules/moe/test_moe_backend.pyget_moe_cls(NVFP4 selectsMarlinFusedMoE; non-NVFP4/None/FP8 falls back toCutlassFusedMoE; per-layeroverride_quant_configcan force Cutlass fallback).Unit tests (multi-GPU test generation):
tests/unittest/_torch/modules/moe/test_moe_module.pyUnit tests (KV cache quant propagation):
tests/unittest/_torch/test_model_config.pyvalidate_and_set_kv_cache_quant()propagation intoquant_config_dictand"auto"behavior.Integration accuracy test (4 GPUs):
tests/integration/defs/accuracy/test_llm_api_pytorch.pyTestNemotronV3Ultra.test_nvfp4_marlin_adp_4gpus(self, mtp_nextn)(Hopper-only) for MARLIN MoE under attention-DP + EP with MTP, assertingllm.args.quant_config.quant_algo == QuantAlgo.MIXED_PRECISIONand evaluating MMLU and GSM8K.mtp_nextn=3.Test-list-only changes
Modified
tests/integration/test_lists/qa/llm_function_core.txtaccuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_marlin_adp_4gpus[mtp_nextn=3]Modified
tests/integration/test_lists/test-db/l0_dgx_h100.ymlunittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu -k "MARLIN"Modified
tests/integration/test_lists/test-db/l0_dgx_h200.ymlaccuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_marlin_adp_4gpus[mtp_nextn=3]Coverage verdict
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.