[#14287][feat] Add Qwen3.5/3.6 MoE NVFP4 + MTP support for SM120/SM121 - #15680
Closed
mihai-chiorean wants to merge 13 commits into
Closed
[#14287][feat] Add Qwen3.5/3.6 MoE NVFP4 + MTP support for SM120/SM121#15680mihai-chiorean wants to merge 13 commits into
mihai-chiorean wants to merge 13 commits into
Conversation
Four changes required to load nvidia/Qwen3.6-35B-A3B-NVFP4 on SM121: 1. quantization/mode.py: Add W4A16_NVFP4 to QuantAlgo enum as an alias for NVFP4. modelopt labels the Qwen3.6 NVFP4 checkpoint with "W4A16_NVFP4" (a naming convention artifact -- both weight and activations are quantized to FP4 / W4A4). Map it to the same QuantMode bits as NVFP4 so the CUTLASS kernel dispatch is unchanged. 2. _torch/model_config.py: In _build_modelopt_quant_config, normalize per-layer W4A16_NVFP4 -> NVFP4 for kernel dispatch and move lm_head to exclude_modules so it loads as BF16 (LMHead bypasses Linear.create_weights and cannot carry NVFP4 scale Parameters). 3. _torch/pyexecutor/config_utils.py: Strip mrope_section and mrope_interleaved from _Qwen35ConfigCompat._flatten_rope for the text executor path. The Qwen3.6 VLM checkpoint includes mRoPE fields intended for the vision path; the text executor never builds 3D position_ids so leaving type="mrope" causes MRotaryEmbedding to produce silently wrong cos/sin from 2D position_ids. 4. _torch/models/modeling_speculative.py: Extend MTPDraftModel (two-engine mode) to dispatch qwen3_5_text and qwen3_5_moe_text to Qwen3NextMTP. The one-engine path (MTPOneDraftModel) already included these model types; this brings the two-engine path into parity. The MTP weight key normalization (mtp.* -> model.layers.40.*), the NVFP4 MoE scale loading, and the MTP unquantized-sublayer backend fallback are all already handled by existing code in Qwen3NextHfWeightMapper and Qwen3NextMTP. Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
The mRoPE strip block worked on a local copy of rope_scaling but only wrote back when the dict was non-empty. When the strip cleared the dict, text_config["rope_scaling"] retained the pre-strip mrope_section / mrope_interleaved values, silently defeating the strip and producing wrong cos/sin from MRotaryEmbeddings 2D fallback at inference time. Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
…NVFP4 dequant Scalar FP8 scale tensors (weight_scale, input_scale) have ndim == 0 and cannot be split or stacked across q/k/v/z projections. Forward them directly to the fused qkvz/ba key instead of routing through the regular split/pack path (_SCALAR_SCALE_SUFFIXES early-exit in _pack_split_projections). Add _dequantize_nvfp4_weight + _dequantize_nvfp4_excluded_weights: modules in quant_config.exclude_modules (e.g. lm_head) are expected to load as BF16, but NVFP4 checkpoints store them as packed uint8 with per-block fp8 scales and a global fp32 scale. Detect by dtype, dequant to BF16, and drop the orphaned scale tensors before the module loader runs. Call site added in preprocess_weights for MIXED_PRECISION quant. Also add NVIDIA copyright header (file previously had none). Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
…r dispatch Two correctness fixes for loading Qwen3.6-35B-A3B-NVFP4 on the PyTorch backend: 1. model_config.py: hf_quant_config.json keys use the VLM-wrapper prefix "model.language_model.<...>", but TRT-LLM module names produced by named_modules() (after the _Qwen35ConfigCompat shim strips the vision wrapper) use "model.<...>". Normalize keys in _build_modelopt_quant_config() so apply_layerwise_quant_config() actually matches. Without this, every Linear/MoE module sees the global MIXED_PRECISION config and ignores its per-layer NVFP4 quant_algo. 2. modeling_qwen3_next.py: Qwen3NextSparseMoeBlock.__init__ now looks up the per-layer MoE NVFP4 config and passes it as override_quant_config to create_moe(). Without this, get_moe_cls() sees MIXED_PRECISION and falls back to CutlassFusedMoE (BF16 expectations), which then fails with a 2048 vs 512 NVFP4 tensor shape mismatch when loading expert weights. Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
…ing in NVFP4 Qwen3.6-35B-A3B-NVFP4 stores linear-attention in_proj_qkv and in_proj_z as FP8 per-tensor-scale tensors (float8_e4m3fn + scalar weight_scale). These are packed into in_proj_qkvz by _pack_split_projections. With a MIXED_PRECISION global quant_algo, the in_proj_qkvz Linear receives UnquantizedLinearMethod (BF16 weight buffer) because there is no matching per-layer FP8 entry under the packed TRT-LLM module name. Previously, copy_weight cast the packed FP8 tensor to BF16 numerically — interpreting raw FP8 bit-patterns without applying the per-tensor weight_scale. This inflated each weight value by ~1/weight_scale (~1000x), corrupting the linear-attention hidden states in all 30 GatedDeltaNet layers. Fix: add _dequantize_fp8_pertensor_excluded_split_weights, called in preprocess_weights before _pack_split_projections for MIXED_PRECISION. It detects split linear-attention FP8 weight tensors with a scalar weight_scale and dequantizes each component individually: bf16 = fp8.to(float32) * weight_scale Orphaned scale tensors (weight_scale, input_scale) are removed so _pack_split_projections does not forward a now-stale scalar scale. Smoke test on Qwen3.6-35B-A3B-NVFP4: raw: "The capital of France is" -> " Paris, a city renowned..." chat: "What is the capital of France?" -> coherent English reasoning Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Two defensive fixes from codex review:
1. qwen3_5_weight_mapper: the scalar weight_scale / input_scale
bypass for FP8 attention projections was firing for any matching
suffix regardless of dimensionality. FP8_PER_CHANNEL_PER_TOKEN
checkpoints have 1-D weight_scale tensors (after
_normalize_scale_names squeezes the trailing axis), which would
trip the ndim==0 assertion and abort weight loading. Require
all candidate tensors to be 0-dim before taking the bypass so
per-channel scales fall through to the regular split/pack path.
2. _Qwen35ConfigCompat._flatten_rope: also pop the legacy "type"
key alongside "rope_type" / "mrope_section" / "mrope_interleaved".
HF Qwen2.5-VL configs used {"type": "mrope", ...}; the new
rope_type form (HF transformers 5.x) is what Qwen3.5/3.6 ship,
but defending against the legacy form is cheap and avoids
resurrecting the broken MRotaryEmbedding path on older configs.
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
The text-only path normalizes to model_type=qwen3_5_moe_text and is exercised by the smoke test. The VLM-composite checkpoint format publishes model_type=qwen3_5_moe at the top level (before the _Qwen35ConfigCompat shim strips the wrapper), which would otherwise fall through to the unsupported-model-type error on the MTP draft engine path. Both MTPForCausalLM (one-model) and MTPDraftModel (separate engine) dispatches now cover the composite form for forward compatibility. Codex pre-PR review flag. Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
mihai-chiorean
requested review from
2ez4bz,
HuiGao-NV,
aswinvisva,
byshiue and
moraxu
June 27, 2026 02:53
Contributor
📝 WalkthroughWalkthroughAdds a ChangesQwen3.5 mixed-precision support
Sequence Diagram(s)Weight preprocessing flowsequenceDiagram
participant preprocess_weights
participant dequantize_fp8 as _dequantize_fp8_pertensor_excluded_split_weights
participant pack_split_projections as _pack_split_projections
participant dequantize_nvfp4 as _dequantize_nvfp4_excluded_weights
preprocess_weights->>dequantize_fp8: dequantize scalar-scale split FP8 weights
preprocess_weights->>pack_split_projections: pack split projections and forward scalar scales
preprocess_weights->>dequantize_nvfp4: dequantize excluded NVFP4 weights
MoE override selectionsequenceDiagram
participant moe_init as Qwen3NextSparseMoeBlock.__init__
participant quant_config_dict as model_config.quant_config_dict
participant create_moe as create_moe
moe_init->>quant_config_dict: look up layer_idx-specific expert config
moe_init->>create_moe: pass override_quant_config during MIXED_PRECISION
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/model_config.py`:
- Around line 435-453: After the language-model prefix normalization in
model_config handling, prefixed LM head keys like model.lm_head are not being
excluded or moved to BF16 because the current lm_head check only matches the
exact unprefixed name. Update the mixed_quant_configs /
quant_config.exclude_modules logic in the model configuration path to recognize
the normalized LM head key alongside the original lm_head name, and remove the
matching entry from mixed_quant_configs so it bypasses the NVFP4 path and loads
as BF16.
In `@tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py`:
- Around line 459-480: Avoid using tensor truthiness when picking scalar
fallback values in the Qwen3 weight mapper. In the scalar-scale handling inside
the branch that builds packed weights, update the representative selection in
the qkv/q/k/v path and the b/a path so it chooses by key presence rather than
`or`, since 0-D tensors with value 0 can be treated as false. Use the existing
`tensors` mapping in `qwen3_5_weight_mapper.py` to explicitly prefer
`"qkv"`/`"b"` when present, otherwise fall back to the other candidate keys
without relying on boolean evaluation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1eed451b-5e0d-4b5d-a64f-b4c0fae54cc8
📒 Files selected for processing (6)
tensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.pytensorrt_llm/_torch/models/modeling_qwen3_next.pytensorrt_llm/_torch/models/modeling_speculative.pytensorrt_llm/_torch/pyexecutor/config_utils.pytensorrt_llm/quantization/mode.py
Comment on lines
+435
to
+453
| # Normalize "model.language_model." prefix to "model." so that | ||
| # quant_config_dict keys match TRT-LLM module names produced by | ||
| # named_modules() (which don't include the "language_model" level). | ||
| _LM_PREFIX = "model.language_model." | ||
| _MODEL_PREFIX = "model." | ||
| mixed_quant_configs = { | ||
| (_MODEL_PREFIX + k[len(_LM_PREFIX):] if k.startswith(_LM_PREFIX) else k): v | ||
| for k, v in mixed_quant_configs.items() | ||
| } | ||
| # LMHead bypasses Linear.create_weights (manual Parameter), | ||
| # so NVFP4 weight scales are never allocated there. Move | ||
| # lm_head to exclude_modules so it loads as BF16. | ||
| if mixed_quant_configs and "lm_head" in mixed_quant_configs: | ||
| if quant_config.exclude_modules is None: | ||
| quant_config.exclude_modules = [] | ||
| if "lm_head" not in quant_config.exclude_modules: | ||
| quant_config.exclude_modules = list( | ||
| quant_config.exclude_modules) + ["lm_head"] | ||
| del mixed_quant_configs["lm_head"] |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Handle prefixed lm_head keys after language-model normalization.
Line 441 can rewrite model.language_model.lm_head to model.lm_head, but Lines 447-453 only remove/exclude exact lm_head. That leaves prefixed LM heads in the per-layer NVFP4 path and prevents excluded-weight dequantization because lm_head does not match model.lm_head.
Proposed fix
- if mixed_quant_configs and "lm_head" in mixed_quant_configs:
+ lm_head_keys = [
+ k for k in mixed_quant_configs
+ if k == "lm_head" or k.endswith(".lm_head")
+ ]
+ if lm_head_keys:
if quant_config.exclude_modules is None:
quant_config.exclude_modules = []
- if "lm_head" not in quant_config.exclude_modules:
- quant_config.exclude_modules = list(
- quant_config.exclude_modules) + ["lm_head"]
- del mixed_quant_configs["lm_head"]
+ lm_head_excludes = ["lm_head", *lm_head_keys]
+ quant_config.exclude_modules = list(
+ dict.fromkeys(list(quant_config.exclude_modules) + lm_head_excludes)
+ )
+ for key in lm_head_keys:
+ del mixed_quant_configs[key]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # Normalize "model.language_model." prefix to "model." so that | |
| # quant_config_dict keys match TRT-LLM module names produced by | |
| # named_modules() (which don't include the "language_model" level). | |
| _LM_PREFIX = "model.language_model." | |
| _MODEL_PREFIX = "model." | |
| mixed_quant_configs = { | |
| (_MODEL_PREFIX + k[len(_LM_PREFIX):] if k.startswith(_LM_PREFIX) else k): v | |
| for k, v in mixed_quant_configs.items() | |
| } | |
| # LMHead bypasses Linear.create_weights (manual Parameter), | |
| # so NVFP4 weight scales are never allocated there. Move | |
| # lm_head to exclude_modules so it loads as BF16. | |
| if mixed_quant_configs and "lm_head" in mixed_quant_configs: | |
| if quant_config.exclude_modules is None: | |
| quant_config.exclude_modules = [] | |
| if "lm_head" not in quant_config.exclude_modules: | |
| quant_config.exclude_modules = list( | |
| quant_config.exclude_modules) + ["lm_head"] | |
| del mixed_quant_configs["lm_head"] | |
| # Normalize "model.language_model." prefix to "model." so that | |
| # quant_config_dict keys match TRT-LLM module names produced by | |
| # named_modules() (which don't include the "language_model" level). | |
| _LM_PREFIX = "model.language_model." | |
| _MODEL_PREFIX = "model." | |
| mixed_quant_configs = { | |
| (_MODEL_PREFIX + k[len(_LM_PREFIX):] if k.startswith(_LM_PREFIX) else k): v | |
| for k, v in mixed_quant_configs.items() | |
| } | |
| # LMHead bypasses Linear.create_weights (manual Parameter), | |
| # so NVFP4 weight scales are never allocated there. Move | |
| # lm_head to exclude_modules so it loads as BF16. | |
| lm_head_keys = [ | |
| k for k in mixed_quant_configs | |
| if k == "lm_head" or k.endswith(".lm_head") | |
| ] | |
| if lm_head_keys: | |
| if quant_config.exclude_modules is None: | |
| quant_config.exclude_modules = [] | |
| lm_head_excludes = ["lm_head", *lm_head_keys] | |
| quant_config.exclude_modules = list( | |
| dict.fromkeys(list(quant_config.exclude_modules) + lm_head_excludes) | |
| ) | |
| for key in lm_head_keys: | |
| del mixed_quant_configs[key] |
🧰 Tools
🪛 Ruff (0.15.18)
[warning] 451-452: Consider [*list(quant_config.exclude_modules), "lm_head"] instead of concatenation
Replace with [*list(quant_config.exclude_modules), "lm_head"]
(RUF005)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/_torch/model_config.py` around lines 435 - 453, After the
language-model prefix normalization in model_config handling, prefixed LM head
keys like model.lm_head are not being excluded or moved to BF16 because the
current lm_head check only matches the exact unprefixed name. Update the
mixed_quant_configs / quant_config.exclude_modules logic in the model
configuration path to recognize the normalized LM head key alongside the
original lm_head name, and remove the matching entry from mixed_quant_configs so
it bypasses the NVFP4 path and loads as BF16.
Comment on lines
+459
to
+480
| if suffix in _SCALAR_SCALE_SUFFIXES and all( | ||
| t.ndim == 0 for t in tensors.values()): | ||
| qkvz_candidates = {"qkv", "q", "k", "v", "z"} & tensors.keys() | ||
| if qkvz_candidates: | ||
| # Use the scalar from "qkv" if present, else fall back to | ||
| # any available candidate (q/k/v all hold the same value). | ||
| representative = tensors.get("qkv") or next( | ||
| v for k, v in tensors.items() if k in {"q", "k", "v"} | ||
| ) | ||
| assert representative.ndim == 0, ( | ||
| f"Expected scalar (ndim=0) for {prefix}.in_proj_qkv.{suffix}, " | ||
| f"got shape {representative.shape}" | ||
| ) | ||
| packed_name = f"{prefix}.in_proj_qkvz.{suffix}" | ||
| assert packed_name not in packed_weights, ( | ||
| f"Packed projection {packed_name} already exists" | ||
| ) | ||
| packed_weights[packed_name] = representative | ||
|
|
||
| ba_candidates = {"b", "a"} & tensors.keys() | ||
| if ba_candidates: | ||
| representative_ba = tensors.get("b") or tensors.get("a") |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
path = Path("tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py")
lines = path.read_text().splitlines()
for start, end in [(450, 490)]:
print(f"\n--- {path}:{start}-{end} ---")
for i in range(start - 1, min(end, len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 2826
🏁 Script executed:
python3 - <<'PY'
try:
import torch
except Exception as exc:
print(f"torch_import_error: {type(exc).__name__}: {exc}")
else:
for value in [0.0, 1.0]:
t = torch.tensor(value)
try:
print(value, bool(t), (t or torch.tensor(2.0)).item())
except Exception as exc:
print(value, f"{type(exc).__name__}: {exc}")
PYRepository: NVIDIA/TensorRT-LLM
Length of output: 222
Avoid tensor truthiness when selecting scalar fallbacks. In tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py:465 and :480, or on a 0-D tensor treats a valid zero-valued scale as false, so the code can skip an available candidate or raise StopIteration when it is the only one. Select by key presence instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py` around
lines 459 - 480, Avoid using tensor truthiness when picking scalar fallback
values in the Qwen3 weight mapper. In the scalar-scale handling inside the
branch that builds packed weights, update the representative selection in the
qkv/q/k/v path and the b/a path so it chooses by key presence rather than `or`,
since 0-D tensors with value 0 can be treated as false. Use the existing
`tensors` mapping in `qwen3_5_weight_mapper.py` to explicitly prefer
`"qkv"`/`"b"` when present, otherwise fall back to the other candidate keys
without relying on boolean evaluation.
The scalar weight_scale / input_scale lookup used "or" chains:
representative = tensors.get("qkv") or next(...)
representative_ba = tensors.get("b") or tensors.get("a")
For a 0-D torch tensor, bool(tensor) reduces to bool(tensor.item()),
so a legitimate zero-valued scalar would silently fall through to
the next candidate. Replace with explicit membership checks so
the value of the tensor never affects the dispatch. CodeRabbit
review nit.
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
6 tasks
mihai-chiorean
marked this pull request as draft
June 28, 2026 14:52
Collaborator
|
FYI, I'm refactoring some common utils in #14599 |
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
Contributor
Author
@moraxu sweet! how about I rebase on your branch since you have a bunch of stamps and that way I only keep what's relevant on top of yours? and... shameless plug... I have #12301 approved it just needs CI. any chance you can spare a minute to trigger the bot? I don't think it starts if I tell it :) |
|
Data point: this also affects consumer RTX 5090 (sm_120). On |
Contributor
Author
I have a few PRs that will get this going. I've only tested on sm121 and this is only a draft right now because I want to rebase once #14599 merges. hopefully I can get some reviews |
Signed-off-by: Mihai Chiorean <mihai.v.chiorean@gmail.com>
mihai-chiorean
changed the title
Contributor
Author
|
superseded upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds PyTorch-backend support for the
Qwen3_5MoeForCausalLM/ Qwen3.6 MoE family on consumer Blackwell (SM120 RTX PRO 6000, SM121 GB10 / DGX Spark), including the defaultTRTLLMattention path with MTP speculative decoding and CUDA graphs enabled.The target validation model is
nvidia/Qwen3.6-35B-A3B-NVFP4. This PR intentionally stays on the working default runtime path and does not include the experimental FlashInfer fixes.The checkpoint uses the modelopt label
W4A16_NVFP4, but the payload behaves as full NVFP4: expert weights and input activations are both 4-bit, per-expert input scales are present, and modelopt exposes this through the NVFP4 config family. This PR treats that label as an alias forQuantAlgo.NVFP4and routes the model through existing CUTLASS NVFP4 kernels.Changes
W4A16_NVFP4as NVFP4 and normalize Qwen3.5/Qwen3.6 per-layer quant config keys, includingmodel.language_model.prefix removal.lm_headfrom NVFP4 module loading so it can be dequantized and loaded separately.lm_headhandling, and BF16 dequantization before packing fused linear-attention buffers with different scales.lm_headexclusion, and FP8 per-tensor linear-attention dequant-before-pack behavior.Dependencies and related issues
1.3.0rc15on sm_121a / GB10: MTP path fails — CUTLASS sm_120 grouped-GEMM init → SMEM shortfall → Triton fallback emits.rsPTX rejected by PTXAS #14575 covers a broader GB10 MTP failure stack; this PR does not claim to close the Triton.rsPTX issue from that report.Smoke test result: DGX Spark GB10 / SM121
Configuration: PyTorch backend, single GPU,
attn_backend="TRTLLM", default CUDA graph behavior,max_batch_size=8,max_seq_len=4096, anddisable_flashinfer_sampling=Truefor the local Spark environment.29 ms TPOTon the validation prompt set.22.5-22.7 ms TPOT, about44 tok/s, roughly1.28xfaster than no-MTP.Representative raw prompt,
"The capital of France is":Out of scope
Qwen3_5MoeForConditionalGeneration)..rsPTX rejection from1.3.0rc15on sm_121a / GB10: MTP path fails — CUTLASS sm_120 grouped-GEMM init → SMEM shortfall → Triton fallback emits.rsPTX rejected by PTXAS #14575 item 3.Test plan
LLM_MODELS_ROOT=/models PYTHONPATH=$PWD pytest -q tests/unittest/llmapi/test_llm_quant.py::test_quant_cfg_qwen35_nvfp4_alias_and_prefix_normalization tests/unittest/_torch/models/test_qwen3_5_moe_weight_mapper.py::test_fp8_pertensor_linear_attn_weights_are_dequantized_before_pack/bot run --extra-stage "GB10-PyTorch-Post-Merge-1, RTXPro6000D-PyTorch-Post-Merge-1"temperature=0PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why.
PR follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths where practical.
If PR introduces API changes, an appropriate PR label is added. This PR does not intend a public API change.
Any new dependencies have been scanned for license and vulnerabilities.
CODEOWNERS updated if ownership changes.
Documentation updated as needed.
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for this PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.