[https://nvbugs/6221450][fix] AutoDeploy: Qwen3.5 400B NVFP4 accuracy regression fix by taylor-yb-lee · Pull Request #14667 · NVIDIA/TensorRT-LLM

taylor-yb-lee · 2026-05-28T04:23:13Z

Summary by CodeRabbit

Release Notes

New Features
- Added configurable sharding exclusion filter enabling selective replication of model components during distributed sharding operations.
Bug Fixes
- Fixed operation ordering in sparse MoE expert output combination to correctly handle replicated components.
Chores
- Updated MoE model sharding configuration with new hint-driven strategy.
- Re-enabled comprehensive model inference test coverage.

Description

Background / Motivation

Qwen3.5-397B-A17B-NVFP4 accuracy dropped to near 0 after PR#13478.

Root cause

The model was migrated to the hint-based IR sharding path in #13478
(92c5030, "Switch ... to sharding-IR canonical models"). That changed how the
MoE shared expert is handled:

Pre-IR (detect_sharding): the shared expert was replicated — it was
intentionally left out of the manual tp_plan and excluded by
simple_shard_filter, so it was never TP-sharded.
IR path (apply_sharding_hints): the shared expert is matched into a
fused torch_nvfp4_swiglu_mlp op and unconditionally TP-sharded. The
NVFP4 swiglu scale-sharding additionally deswizzled every projection's
per-block weight_scale with the gate projection's shape, which is wrong for
the differently-shaped down_proj scale — corrupting the shared-expert output.

A second, independent regression from the same migration: the IR path reads
allreduce_strategy from apply_sharding_hints, but the model only declared
SYMM_MEM under the (now-disabled) detect_sharding, so all-reduces silently
fell back to NCCL — a perf regression vs. the pre-IR path.

Summary of changes

Replicate the shared expert (correctness). Add a shard_exclude_filter
option to the IR sharder: nodes whose weight-parameter keys match a substring
(e.g. shared_expert) are left replicated instead of TP-sharded — the same
convention as the legacy simple_shard_filter. The Qwen3.5-MoE config uses
shard_exclude_filter: ["shared_expert"], restoring the pre-IR behavior. The
MoE all-reduce is reordered so the replicated shared output is added after
the routed all-reduce (adding before would scale it by the TP world size).
Restore SYMM_MEM all-reduce (perf). Set allreduce_strategy: SYMM_MEM
under apply_sharding_hints so the IR path uses it, and remove the dead
detect_sharding block (disabled on the IR path; superseded by in-model
sharding hints).

Validation

accuracy/test_llm_api_autodeploy.py::TestQwen3_5_397B_MoE::test_nvfp4[8]
(8×B200):

MMLU 26.8% → 86.4%, GSM8K 96.0% — PASS.
FX graph dumps confirm the shared expert is replicated and all all-reduces use
SYMM_MEM (0 NCCL), matching the pre-IR baseline.

Impact

shard_exclude_filter defaults to None (no behavior change for other
models); only Qwen3.5-MoE opts in.

Test Coverage

Accuracy test is being covered by post-merge accuracy check for Qwen3.5.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

taylor-yb-lee · 2026-05-28T04:27:46Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-05-28T04:33:21Z

PR_Github #50691 [ run ] triggered by Bot. Commit: 3f237b4 Link to invocation

taylor-yb-lee · 2026-05-28T04:44:42Z

/bot help

github-actions · 2026-05-28T04:44:49Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

taylor-yb-lee · 2026-05-28T04:45:53Z

/bot run --stage-list "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-05-28T04:52:04Z

PR_Github #50696 [ run ] triggered by Bot. Commit: 3f237b4 Link to invocation

tensorrt-cicd · 2026-05-28T04:55:34Z

PR_Github #50691 [ run ] completed with state ABORTED. Commit: 3f237b4

Link to invocation

tensorrt-cicd · 2026-05-28T07:21:27Z

PR_Github #50696 [ run ] completed with state SUCCESS. Commit: 3f237b4
/LLM/main/L0_MergeRequest_PR pipeline #40182 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

taylor-yb-lee · 2026-05-29T05:56:58Z

/bot run --stage-list "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

coderabbitai · 2026-05-29T06:00:27Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ddbed1ed-6bf2-4186-bafa-33b2c7c7732c

📥 Commits

Reviewing files that changed from the base of the PR and between 7bdd835 and b99e762.

📒 Files selected for processing (4)

examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5_moe.py
tensorrt_llm/_torch/auto_deploy/transform/library/sharding_ir.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR refactors Qwen 3.5 MoE tensor-parallel sharding to use hint-driven exclusion for shared experts. It adds a new shard_exclude_filter configuration option to the sharding IR transform, reorders MoE output combination to all-reduce before summing, and applies the feature in the model config.

Changes

Qwen MoE Shared Expert Sharding Exclusion

Layer / File(s)	Summary
MoE output combination and all-reduce ordering `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5_moe.py`	Reorders routed expert output and shared expert combination: all-reduce the routed expert output before adding shared expert output, preventing TP scaling of the replicated shared expert.
Sharding exclusion filter infrastructure `tensorrt_llm/_torch/auto_deploy/transform/library/sharding_ir.py`	Adds `shard_exclude_filter` field to `IRShardingConfig` and implements filtering logic in `ApplyShardingHints._apply`. Helper function `_node_weight_keys_match` checks whether FX node parameter keys match configured substring patterns and skips sharding for matching nodes.
Configuration and test waiver update `examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml`, `tests/integration/test_lists/waives.txt`	Updates Qwen 3.5 MoE 400B config from manual detection-based sharding to hint-driven sharding with `shard_exclude_filter: ["shared_expert"]`. Removes test waiver for NVFP4[8], confirming the test now passes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#13478: Both PRs modify tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5_moe.py MoE output merge and all-reduce semantics for shared vs. routed experts.
NVIDIA/TensorRT-LLM#14596: Both PRs modify tests/integration/test_lists/waives.txt to remove test waiver entries for integration tests.
NVIDIA/TensorRT-LLM#14393: Both PRs modify tests/integration/test_lists/waives.txt by changing which integration tests are marked as SKIP.

Suggested reviewers

suyoggupta
nvchenghaoz
shaharmor98
brb-nv
StanleySun639

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: a fix for an accuracy regression in Qwen3.5 400B NVFP4 model deployment, with proper formatting including the NVBugs ticket reference and [fix] type.
Description check	✅ Passed	The PR description comprehensively covers motivation, root cause analysis, summary of changes, validation results, and impact assessment. All required template sections are filled with substantive content.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-05-29T06:02:32Z

PR_Github #50981 [ run ] triggered by Bot. Commit: b99e762 Link to invocation

taylor-yb-lee · 2026-05-29T06:43:17Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-05-29T06:48:47Z

PR_Github #50993 [ run ] triggered by Bot. Commit: b99e762 Link to invocation

tensorrt-cicd · 2026-05-29T06:52:16Z

PR_Github #50981 [ run ] completed with state ABORTED. Commit: b99e762

Link to invocation

greg-kwasniewski1 · 2026-05-29T13:37:41Z

@taylor-yb-lee
No need to overcomplicate it with new field shard_exclude_filter. There is already a field for that (but not a blacklist, but a whitelist): shard_layers .

What I suggest to do:

In class Qwen3_5MoeMLP, change layer_type="moe" to ` layer_type="shared_expert",
in [qwen3.5_moe_400b.yaml](https://github.com/NVIDIA/TensorRT-LLM/pull/14667/changes#diff-5ec3491444accaea22ac702e365a8d8dc1363f3375015320f024199608ae4721) , instead of shard_exclude_filter: ["shared_expert"], use shard_layers = ["moe", "delta", "mha"]

Sure you can argue that sometimes, blacklist is more convenient than whitelist, but I'd prefer to avoid having both - it's not clear what if these two contradict each other.

tensorrt-cicd · 2026-05-29T16:28:12Z

PR_Github #50993 [ run ] completed with state SUCCESS. Commit: b99e762
/LLM/main/L0_MergeRequest_PR pipeline #40442 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

taylor-yb-lee · 2026-05-30T06:51:52Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

taylor-yb-lee · 2026-05-30T06:53:24Z

/bot run --extra-stage "DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1" --disable-fail-fast

tensorrt-cicd · 2026-05-30T06:58:26Z

PR_Github #51171 [ run ] triggered by Bot. Commit: 5130806 Link to invocation

tensorrt-cicd · 2026-05-30T06:58:55Z

PR_Github #51172 [ run ] triggered by Bot. Commit: 5130806 Link to invocation

tensorrt-cicd · 2026-05-31T06:15:01Z

PR_Github #51215 [ run ] completed with state SUCCESS. Commit: 2e7db8b
/LLM/main/L0_MergeRequest_PR pipeline #40639 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

greg-kwasniewski1

Good for now to fix the bug, but let's follow-up with a more structured approach for passing sharding IR info

taylor-yb-lee · 2026-06-01T19:16:20Z

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1,DGX_H100-4_GPUs-AutoDeploy-1,DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

…d nodes under IR sharding Add a shard_exclude_filter config so nodes whose weight keys match (e.g. the Qwen3.5-MoE shared expert) are replicated instead of TP-sharded, and reorder the MoE all-reduce so the replicated shared output is added after it (not scaled by world size). Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

…M all-reduce Exclude the shared expert from TP sharding via shard_exclude_filter (cheaper replicated at this size), set SYMM_MEM all-reduce on the IR path, and drop the now-dead detect_sharding block. test_nvfp4[8] MMLU 86.4% (was ~27%). Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

…_layers whitelist - Tag shared-expert linears layer_type="shared_expert" (was "moe"). - Drop shard_exclude_filter blacklist; reuse existing shard_layers whitelist. - yaml: shard_layers=["moe","delta","mha"] omits shared_expert -> replicated. Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

- SwiGLU fusion dropped the layer_type op-arg, so hint-driven sharding (shard_layers) saw lt=None and TP-sharded nodes meant to be replicated. - Add optional layer_type to the unfused SwiGLU ops and propagate the source linears' layer_type onto the fused node in the matchers. gate/up/down linear(layer_type=X) --fuse--> swiglu_mlp() [hint lost] gate/up/down linear(layer_type=X) --fuse--> swiglu_mlp(layer_type=X) [kept] Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

taylor-yb-lee · 2026-06-01T20:16:19Z

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1,DGX_H100-4_GPUs-AutoDeploy-1,DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-01T20:22:57Z

PR_Github #51413 [ run ] triggered by Bot. Commit: 4fa4379 Link to invocation

tensorrt-cicd · 2026-06-02T04:54:24Z

PR_Github #51413 [ run ] completed with state SUCCESS. Commit: 4fa4379
/LLM/main/L0_MergeRequest_PR pipeline #40823 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

taylor-yb-lee · 2026-06-02T06:32:27Z

/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1,DGX_H100-4_GPUs-AutoDeploy-1,DGX_B200-8_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-02T06:38:40Z

PR_Github #51534 [ run ] triggered by Bot. Commit: 4fa4379 Link to invocation

tensorrt-cicd · 2026-06-02T08:56:30Z

PR_Github #51534 [ run ] completed with state SUCCESS. Commit: 4fa4379
/LLM/main/L0_MergeRequest_PR pipeline #40930 completed with status: 'SUCCESS'

CI Report

Link to invocation

…ccuracy fix) - Shared-expert NVFP4 TP-sharding corrupts deswizzled weight scales (GSM8K 92.27); tag layer_type="shared_expert", propagate it across the SwiGLU fusion, and exclude it via shard_layers => replicated. GSM8K 94.77@1319 (legacy 95.03), 119 tps. - Reorder MoE merge so the replicated shared output is added AFTER the routed all-reduce (adding before would scale shared by the TP world size). - Ports PR NVIDIA#14667 (Qwen3.5 fix) to the DeepSeek NVFP4 IR path. NOTE: swiglu.py + fuse_swiglu.py here duplicate PR NVIDIA#14667 generic infra (already in upstream/main); drop them on rebase, keep modeling_deepseek.py + the IR config. MoE merge (before -> after): add(routed_partial, shared_partial) --all_reduce("moe")--> [shared TP-sharded] all_reduce("moe")(routed_partial) --> + shared_full [shared replicated] Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

…rites AutoDeploy pattern matchers that collapse N fine-grained ops into one (e.g. the SwiGLU matchers) dropped the per-op layer_type sharding hint. The hint-driven sharder (apply_sharding_hints / shard_layers) then read layer_type=None on the fused op and, because the whitelist is fail-open, TP-sharded nodes meant to be replicated -- the root cause of the Qwen3.5-MoE NVFP4 shared-expert accuracy regression fixed point-wise in NVIDIA#14667. Generalize the fix into the matcher infra: add an optional layer_type arg to the intermediate SwiGLU ops, add collect_classification_hints / stamp_hints helpers in node_utils, and propagate the consensus layer_type from the matched nodes onto the replacement op in ADReplacementPatternEntry. Per-weight mechanics (tp_mode, output_sizes, ...) are intentionally not propagated: the fused op's ShardableNode re-derives them structurally and they legitimately differ across constituents (an MLA layer mixes none/colwise/rowwise yet shares one layer_type), so generic propagation of the sharding decision is ill-posed. Tested: GPU-free unit tests for the helpers (consensus / conflict-drop / schema-guarded stamping), an NVFP4 end-to-end regression mirroring the Qwen3.5-MoE bug, and the full transformations/library suite (329 passed). Fixes NVIDIA#14834 Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>

…_oss IR port Closes several pre-existing gaps in the sharding-IR pipeline that together prevented IR-marked modeling files from actually exercising the IR path, and adds the supporting infrastructure for GPT-OSS-style models. Core fixes (transform/library/sharding_ir.py): * IR-vs-legacy dispatch bug: ``has_sharding_ir_markers`` compared ``node.target == torch.ops.auto_deploy.all_reduce`` (OpOverloadPacket), which always returned False because FX nodes carry the ``.default`` OpOverload as their target. Renamed to ``is_shardingIR_enabled`` and fixed to use ``is_op(...)`` so the comparison matches both forms. * Rowwise-bias double-counting: ``LinearShardableNode`` was leaving the bias intact for rowwise sharding. Mirrored the legacy ``detect_column_row_shard`` "drop bias on all but one rank" pattern (reusing ``_load_hook_remove``) so the bias appears exactly once after reduction. * ``torch_attention`` registered to ``WeightedParamShardableNode`` (with a new ``enable_sharding: bool = False`` schema kwarg). Slices per-head free Parameters passed directly to ``torch_attention`` (e.g. GPT-OSS's ``sinks``) along the head dim. * New ``DenseMLPMoEShardableNode`` registered for ``torch.ops.auto_deploy.torch_moe_dense_mlp``. Stacked-tensor MoE variants other than ``triton_mxfp4_moe`` were previously unregistered (left replicated). Verified end-to-end on gpt_oss-ep-only: 8 collectives inserted, rel_rmse=0.0049 (tol=0.02). Modeling: * modeling_gpt_oss.py: IR-port the attention block (Q/K/V/O linears -> torch_linear_simple, head reshapes -> auto_deploy.view with tp_scaled_dim=2, o_proj rowwise + all_reduce, ``enable_sharding=True`` on the ``torch_attention`` call so ``sinks`` is sliced). * modeling_qwen3_5_moe.py: revert the all_reduce-vs-add ordering inversion from NVIDIA#14667 (Taylor's NVFP4 fix). The current ``all_reduce(routed) + shared_partial`` order is mathematically wrong for BF16 sharded routed+shared MoE: routed gets reduced TP-fold while shared stays partial, so the sum is full+partial. Restoring the canonical "add first, all_reduce once" pattern matches the other IR-ported MoE models. NVFP4 path behavior may need a separate investigation. Test harness: * Renamed ``test_sharding_ir_equivalence`` -> ``test_sharding_num_correctness``. Deduplicated to import the production ``is_shardingIR_enabled`` helper instead of a parallel implementation. Dropped an erroneous ``pytest.skip`` inside a ``spawn_multiprocess_job`` worker (would surface as ``Process exited with code 1`` and mask the actual outcome). Added a small ``_KNOWN_FAILING_MODELING_FILES`` skip list with ``granite_moe_hybrid`` as the sole entry (pre-existing modeling-side sharding bug, unchanged from origin/main). * custom_ops/attention/torch_attention.py: added ``enable_sharding: bool = False`` to the op schema (real impl + fake). Skill: * .claude/skills/ad-sharding-ir-port/SKILL.md: documented the new per-head Parameter sharding pattern, corrected stale Pitfall #9. Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com>

…ccuracy fix) - Shared-expert NVFP4 TP-sharding corrupts deswizzled weight scales (GSM8K 92.27); tag layer_type="shared_expert", propagate it across the SwiGLU fusion, and exclude it via shard_layers => replicated. GSM8K 94.77@1319 (legacy 95.03), 119 tps. - Reorder MoE merge so the replicated shared output is added AFTER the routed all-reduce (adding before would scale shared by the TP world size). - Ports PR NVIDIA#14667 (Qwen3.5 fix) to the DeepSeek NVFP4 IR path. NOTE: swiglu.py + fuse_swiglu.py here duplicate PR NVIDIA#14667 generic infra (already in upstream/main); drop them on rebase, keep modeling_deepseek.py + the IR config. MoE merge (before -> after): add(routed_partial, shared_partial) --all_reduce("moe")--> [shared TP-sharded] all_reduce("moe")(routed_partial) --> + shared_full [shared replicated] Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

github-actions Bot assigned taylor-yb-lee May 28, 2026

taylor-yb-lee force-pushed the taylor/unwaive branch from 2225ea9 to b99e762 Compare May 29, 2026 05:39

taylor-yb-lee changed the title ~~[None][infra] Try unwaive Qwen3.5 NVFP4 acc test~~ [https://nvbugs/6221450][fix] AutoDeploy: Qwen3.5 400B NVFP4 accuracy regression fix May 29, 2026

taylor-yb-lee marked this pull request as ready for review May 29, 2026 05:56

taylor-yb-lee requested a review from a team as a code owner May 29, 2026 05:56

taylor-yb-lee requested a review from tcherckez-nvidia May 29, 2026 05:56

taylor-yb-lee changed the title ~~[https://nvbugs/6221450][fix] AutoDeploy: Qwen3.5 400B NVFP4 accuracy regression fix~~ [https://nvbugs/6221450][fix] AutoDeploy: Qwen3.5 400B NVFP4 accuracy regression fix May 29, 2026

greg-kwasniewski1 reviewed May 29, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5_moe.py Outdated

taylor-yb-lee force-pushed the taylor/unwaive branch from 7b9fb18 to f0ed2b9 Compare May 30, 2026 06:52

taylor-yb-lee force-pushed the taylor/unwaive branch from f0ed2b9 to 5130806 Compare May 30, 2026 06:54

greg-kwasniewski1 approved these changes Jun 1, 2026

View reviewed changes

taylor-yb-lee added 5 commits June 1, 2026 13:08

Try unwaive Qwen3.5

6fbc738

Signed-off-by: Taylor Yeonbok Lee <249374542+taylor-yb-lee@users.noreply.github.com>

taylor-yb-lee force-pushed the taylor/unwaive branch from 2e7db8b to 4fa4379 Compare June 1, 2026 20:16

This was referenced Jun 1, 2026

[Feature Request] AutoDeploy: propagate layer_type hint across pattern-matcher rewrites #14834

Closed

[None][feat] AutoDeploy: propagate layer_type hint across pattern-matcher rewrites #14835

Merged

taylor-yb-lee enabled auto-merge (squash) June 2, 2026 16:03

taylor-yb-lee merged commit 6ef7b38 into NVIDIA:main Jun 2, 2026
8 checks passed

taylor-yb-lee mentioned this pull request Jun 5, 2026

[https://nvbugs/6272668][infra] Unwaive DSR1 and Qwen3.5 again #15010

Merged

1 task

coderabbitai Bot mentioned this pull request Jun 10, 2026

[None][feat] AutoDeploy: Qwen3.5: Apply whielist based sharding and apply lm_head sharding #15185

Merged

1 task

Uh oh!

Conversation

taylor-yb-lee commented May 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Description

Background / Motivation

Root cause

Summary of changes

Validation

Impact

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

taylor-yb-lee commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

taylor-yb-lee commented May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

taylor-yb-lee commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

tensorrt-cicd commented May 28, 2026

Uh oh!

taylor-yb-lee commented May 29, 2026

Uh oh!

coderabbitai Bot commented May 29, 2026

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

taylor-yb-lee commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

Uh oh!

greg-kwasniewski1 commented May 29, 2026

Uh oh!

tensorrt-cicd commented May 29, 2026

Uh oh!

taylor-yb-lee commented May 30, 2026

Uh oh!

taylor-yb-lee commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 30, 2026

Uh oh!

tensorrt-cicd commented May 31, 2026

Uh oh!

greg-kwasniewski1 left a comment

Choose a reason for hiding this comment

Uh oh!

taylor-yb-lee commented Jun 1, 2026

Uh oh!

taylor-yb-lee commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

taylor-yb-lee commented May 28, 2026 •

edited by coderabbitai Bot

Loading