Run Nemotron-3.5-Lightning-30B-A3B-NVFP4 end to end: NemotronH hybrid (23 Mamba2 / 6 GQA / 23 non-gated relu² MoE) with a MIXED_PRECISION checkpoint

[localai-bot]

Goal

Run nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 end to end, token-gated
against the pinned vLLM oracle on GB10.

Row: MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm (INVENTORIED,
model-matrix.md:248). Arch NemotronHForCausalLM, model_type: nemotron_h.

What the checkpoint is

52 layers from layers_block_type: 23 mamba (Mamba2) / 6 GQA attention / 23
MoE. hidden_size=2688, vocab=131072, max_position_embeddings=1048576.

• Mamba2: mamba_num_heads=64, mamba_head_dim=64, n_groups=8,
  ssm_state_size=128, conv_kernel=4, chunk_size=128, silu,
  use_conv_bias=true, use_bias=false, mamba_ssm_cache_dtype=float32.
• Attention (idx 5,12,19,26,33,42): 32 q / 2 kv heads, head_dim=128,
  rope_theta=10000, no sliding window.
• MoE: 128 routed experts, top-6, non-gated relu² (up_proj and
  down_proj only, no gate_proj), 1 shared expert (intermediate=3712),
  sigmoid scoring with e_score_correction_bias, n_group=1/topk_group=1,
  norm_topk_prob=true, routed_scaling_factor=2.5 applied to the OUTPUT.
• MTP: num_nextn_predict_layers=1, mtp_layers_block_type=["attention","moe"],
  DeepSeek-style eh_proj/enorm/hnorm, bf16 (unquantized).
• Quantization: ModelOpt quant_algo: MIXED_PRECISION, quant_method: modelopt, producer modelopt 0.44.0rc5, with a 5981-entry quantized_layers
  map — NVFP4 W4A16 group_size 16 for routed experts, shared experts and
  lm_head; FP8 W8A8 static (with input_scale) for the mamba
  in_proj/out_proj only (46 targets); bf16 attention q/k/v/o; fp8
  kv_cache_scheme with k_scale/v_scale. Weight prefix is backbone..

20.1 GiB on disk, fits one GB10. Downloaded to
$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4 at pinned revision
29f2d1746d8f41e316523194b19018707749b1b1.

Upstream (pinned oracle 555967922, vLLM 0.26.0.dev0)

registry.py:179 -> models/nemotron_h.py::NemotronHForCausalLM;
MambaMixer2 at nemotron_h.py:373; NemotronHMoEDecoderLayer at :317
(FusedMoE with ckpt_names=("up_proj","down_proj",""), use_grouped_topk,
scoring_func="sigmoid", activation=activation_without_mul(mlp_hidden_act),
apply_routed_scale_to_output=True); MTP at models/nemotron_h_mtp.py;
ModelOptMixedPrecisionConfig at layers/quantization/modelopt.py:2280.

Blockers and owed work

1. KERNEL-SSM-MAMBA Mamba2 SSD core is unported: KERNEL-SSM-MAMBA has only the GDN arm, so every hybrid-SSM arch (Nemotron-H/3.5, Bamba, Granite-4, Zamba2, Falcon-H1, Mamba2) is blocked #496 — Mamba2 SSD. Hard blocker.
   spec.
   W1 (CPU host references) is in fresh review on row/KERNEL-SSM-MAMBA-SSD-W1;
   the CUDA arm is W2.
2. Non-gated relu² MoE. Every grouped-MoE op we have is SwiGLU-shaped
   (kMoeGateUpSwiGLUGrouped, kMoeGroupedGemmBf16GateUpSilu, kMoeSiluMul).
   This arch has no gate half at all: up_proj -> relu² -> down_proj, and it
   must route through vt::MergedGemmGroup / the NVFP4 Marlin grouped path
   rather than a hand-rolled sibling.
3. ModelOpt MIXED_PRECISION loading. We handle uniform modelopt NVFP4
   (Laguna, MiniMax-H3, Qwen3.5) and have no per-module quant_algo resolution
   at all — quantization_config is read ad-hoc in exactly two model weight
   files today (kimi_k3_weights.cpp:171, deepseek_v2_weights.cpp:365).
   Mirror get_quant_algo_for_layer's direct-then-shard-prefix lookup.
4. Model plumbing. nemotron_h.cpp / _weights.cpp / _registry.cpp,
   layers_block_type -> layer dispatch, het-KV group construction (one
   MambaSpec group + one full-attention group covering only 6 layers), the
   backbone. prefix, and the MTP head (our MTP spec-decode machinery already
   exists — qwen3_5_mtp.cpp).

Gate

Token-exact greedy vs the pinned oracle on GB10, identical prompts, token
counts, batching and sampling; oracle identity asserted before any number is
believed. Note the oracle venv trap: $HOME/venvs/vllm-oracle on dgx symlinks
to vllm-oracle-v0.25.0-stage (vLLM 0.25.0), which predates the NemotronH MoE
decoder layer entirely. The pin is vllm-oracle-next
(555967922, transformers 5.14.1, flashinfer 0.6.15.post1).

Speed is a separate, later question and no ratio is claimed by this row until
correctness lands.

Not in scope

NemotronHPuzzleForCausalLM heterogeneous per-layer configs, moe_latent_size
(null here), TP sharding of n_groups, ReplaySSM.