spec(MODEL-TEXT-nemotron-h): the first hybrid Mamba2 model and the first MIXED_PRECISION checkpoint (#517) - #518
Merged
Conversation
…rst MIXED_PRECISION checkpoint (#517) FOLLOWING_AGENTS_PROTOCOL Row spec for NemotronHForCausalLM, driven by nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 at pinned revision 29f2d174, now staged on the NAS at 20.1 GiB. Two things make this more than another model file, and both are the first of their kind here. It is the first true hybrid Mamba2 model: 52 layers, 23 Mamba2, 6 GQA attention at indices 5/12/19/26/33/42, 23 MoE. Our GDN hybrids supply the het-KV machinery but not the recurrence, which is why this row is hard-blocked on KERNEL-SSM-MAMBA (#496). It is the first MIXED_PRECISION checkpoint. The repo name says NVFP4 and most parameters are, but that is not what the file contains: modelopt writes a 5981-entry quantized_layers map with NVFP4 W4A16 group_size 16 on the experts, shared experts and lm_head, FP8 W8A8 static on the mamba in_proj/out_proj (46 targets, with input_scale), bf16 attention q/k/v/o, and an fp8 kv_cache_scheme. We have never resolved a quant algorithm per module; quantization_config is read ad-hoc in exactly two weight files today. Reading this as uniform NVFP4 gets the loader wrong in the way a token gate cannot see: still numerically fine, still matching tokens, moving the wrong bytes. A third difference is smaller but has no local precedent either. The MoE is non-gated: there is no gate_proj anywhere in the checkpoint, the expert is up_proj -> relu^2 -> down_proj, and every grouped-MoE op we have is SwiGLU-shaped. The spec requires that to go through MlpGateUpMethodBase and vt::MergedGemmGroup, or to extend the seam, never a hand-rolled sibling. Six W's, each its own delegated task. W1 (the MIXED_PRECISION resolver, gated on the real config.json with no weights needed) and W2 (the non-gated relu^2 MoE) are independent of #496 and can run beside it. W3 onward cannot, and the spec says so rather than implying a forward path is testable today. Recorded because it will otherwise be rediscovered the expensive way: the dgx $HOME/venvs/vllm-oracle symlink points at vllm-oracle-v0.25.0-stage, vLLM 0.25.0 with transformers 5.13.1, which predates NemotronHMoEDecoderLayer entirely. A run through it fails on this checkpoint and reads as "the model is unsupported". The pin is vllm-oracle-next, 555967922 with transformers 5.14.1 and flashinfer 0.6.15.post1, and every oracle run asserts all three and aborts on mismatch. No lifecycle state moves and no measurement is claimed. The row stays INVENTORIED: spec committed, implementation not started. Records and docs only, no product code touched. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row spec for
MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm, tracking #517.Hard-blocked on #496 (Mamba2 SSD), whose W1 is in fresh review.
No lifecycle state moves. No measurement is claimed. No product code is
touched. The row stays
INVENTORIED.Why this arch is not just another model file
First true hybrid Mamba2 model. 52 layers: 23 Mamba2, 6 GQA attention
(indices 5/12/19/26/33/42), 23 MoE. Our GDN hybrids give us the het-KV
machinery but not the recurrence.
First
MIXED_PRECISIONcheckpoint. The repo name says NVFP4, and mostparameters are — but the file is not uniform. ModelOpt writes a 5981-entry
quantized_layersmap:lm_headgroup_size=16in_proj/out_proj(46 targets)input_scalek_scale/v_scale)We have never resolved a quant algorithm per module —
quantization_configis read ad-hoc in exactly two weight files today. Reading this as uniform NVFP4
is the failure a token gate cannot see: still numerically fine, still matching
tokens, moving the wrong bytes.
Non-gated MoE. No
gate_projexists in the checkpoint; the expert isup_proj → relu² → down_proj, and every grouped-MoE op we have isSwiGLU-shaped. The spec requires this to go through
MlpGateUpMethodBase/vt::MergedGemmGroup, or to extend the seam — never a hand-rolled sibling.Structure
Six W's, each its own delegated task with a fresh implementer and fresh
reviewer. W1 (MIXED_PRECISION resolver, gated on the real
config.jsonwith no weights needed) and W2 (non-gated
relu²MoE) are independent of#496 and can run beside it. W3 onward cannot, and the spec says so rather than
implying a forward path is testable today.
Recorded so it is not rediscovered the expensive way
$HOME/venvs/vllm-oracleon dgx symlinks tovllm-oracle-v0.25.0-stage—vLLM 0.25.0, transformers 5.13.1 — which predates
NemotronHMoEDecoderLayerentirely. A run through it fails on this checkpointand reads as "the model is unsupported". The pin is
vllm-oracle-next(
555967922, transformers 5.14.1, flashinfer 0.6.15.post1). Every oracle runasserts all three and aborts on mismatch.
The checkpoint is staged on the NAS at pinned revision
29f2d174(20.1 GiB);the oracle smoke run is queued behind
$HOME/gpu.lock.scripts/agent-preflight.sh --staged: green. (test_cpu_x86_llamacpp_floorred once under box contention — it is a wait-for-quiet harness self-test,
passes standalone, and this diff touches no script.)
🤖 Generated with Claude Code