From cf1716e584f073d20687751a552d2699d03fbf23 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 18:44:43 +0000 Subject: [PATCH] spec(MODEL-TEXT-nemotron-h): the first hybrid Mamba2 model and the first MIXED_PRECISION checkpoint (#517) FOLLOWING_AGENTS_PROTOCOL Row spec for NemotronHForCausalLM, driven by nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 at pinned revision 29f2d174, now staged on the NAS at 20.1 GiB. Two things make this more than another model file, and both are the first of their kind here. It is the first true hybrid Mamba2 model: 52 layers, 23 Mamba2, 6 GQA attention at indices 5/12/19/26/33/42, 23 MoE. Our GDN hybrids supply the het-KV machinery but not the recurrence, which is why this row is hard-blocked on KERNEL-SSM-MAMBA (#496). It is the first MIXED_PRECISION checkpoint. The repo name says NVFP4 and most parameters are, but that is not what the file contains: modelopt writes a 5981-entry quantized_layers map with NVFP4 W4A16 group_size 16 on the experts, shared experts and lm_head, FP8 W8A8 static on the mamba in_proj/out_proj (46 targets, with input_scale), bf16 attention q/k/v/o, and an fp8 kv_cache_scheme. We have never resolved a quant algorithm per module; quantization_config is read ad-hoc in exactly two weight files today. Reading this as uniform NVFP4 gets the loader wrong in the way a token gate cannot see: still numerically fine, still matching tokens, moving the wrong bytes. A third difference is smaller but has no local precedent either. The MoE is non-gated: there is no gate_proj anywhere in the checkpoint, the expert is up_proj -> relu^2 -> down_proj, and every grouped-MoE op we have is SwiGLU-shaped. The spec requires that to go through MlpGateUpMethodBase and vt::MergedGemmGroup, or to extend the seam, never a hand-rolled sibling. Six W's, each its own delegated task. W1 (the MIXED_PRECISION resolver, gated on the real config.json with no weights needed) and W2 (the non-gated relu^2 MoE) are independent of #496 and can run beside it. W3 onward cannot, and the spec says so rather than implying a forward path is testable today. Recorded because it will otherwise be rediscovered the expensive way: the dgx $HOME/venvs/vllm-oracle symlink points at vllm-oracle-v0.25.0-stage, vLLM 0.25.0 with transformers 5.13.1, which predates NemotronHMoEDecoderLayer entirely. A run through it fails on this checkpoint and reads as "the model is unsupported". The pin is vllm-oracle-next, 555967922 with transformers 5.14.1 and flashinfer 0.6.15.post1, and every oracle run asserts all three and aborts on mismatch. No lifecycle state moves and no measurement is claimed. The row stays INVENTORIED: spec committed, implementation not started. Records and docs only, no product code touched. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/model-matrix.md | 2 +- .agents/roadmap_v1.md | 1 + .agents/specs/nemotron-h-model.md | 211 ++++++++++++++++++++++++++++++ docs/FEATURES.md | 2 +- 4 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 .agents/specs/nemotron-h-model.md diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 9ae59607a..17278916a 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -245,7 +245,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-TEXT-mimo-v2-mi-mo-v2-flash-for-causal-lm` | `MiMoV2FlashForCausalLM` | `registry.py:166`; `vllm/model_executor/models/mimo_v2.py::MiMoV2FlashForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-mimo-v2-mi-mo-v2-for-causal-lm` | `MiMoV2ForCausalLM` | `registry.py:167`; `vllm/model_executor/models/mimo_v2.py::MiMoV2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-nemotron-nemotron-for-causal-lm` | `NemotronForCausalLM` | `registry.py:168`; `vllm/model_executor/models/nemotron.py::NemotronForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10) | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: row spec committed ([#517](https://github.com/mudler/vllm.cpp/issues/517), [spec](specs/nemotron-h-model.md)); BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10) | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-olmo-olmo-for-causal-lm` | `OlmoForCausalLM` | `registry.py:171`; `vllm/model_executor/models/olmo.py::OlmoForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-olmo2-olmo2-for-causal-lm` | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | `registry.py:172-173`; `vllm/model_executor/models/olmo2.py::Olmo2ForCausalLM` | causal generation / text | OLMo-2-0425-1B dense LANDED (W0-W4). Impl: `include/vllm/model_executor/models/olmo2.h` + `src/vllm/model_executor/models/{olmo2,olmo2_weights,olmo2_registry}.cpp` (TWO `REGISTER_VLLM_MODEL` for `Olmo2ForCausalLM`+`Olmo3ForCausalLM`, one shared factory; reuses the shared dense glue). ZERO new compute kernel — the two "distinctives" both reduce to WIRING over landed ops: (1) PURE POST-NORM (`norm_after`, `olmo2.py:261-277`) = standalone `vt::RmsNorm` on each sublayer OUTPUT + plain `vt::Add` residual re-join, NO pre-norm (the GLM-4 standalone-output-norm op, pre-norms dropped); (2) FULL-WIDTH QK-norm (`olmo2.py:113-117,160-172`) = two standalone `vt::RmsNorm` over `[T,q_size]`/`[T,kv_size]` before NeoX rope (NOT the per-head `kAttnQkNormRope`). Reuse: SiLU SwiGLU (`kSiluAndMul`), NeoX RoPE (`RopeFromCache`/`RopeNeox`), GQA/MHA paged glue (`dense_attn_block.h`), merged qkv/gate_up loader. Checkpoint is F32 on-disk → loader downcasts f32→bf16 (round-to-nearest-even, `vt::F32ToBF16`) to match vLLM-bf16. UNTIED lm_head (`tie_word_embeddings:false`). Loader 179 tensors, zero missing/unmapped. Tokenizer: ONE shared-TU touch — `tokenizer.cpp` accepts the OLMo-2 Split `behavior=Removed,invert=true` encoding (equiv. to `Isolated` for the full-cover cl100k regex → `kLlama3`); guarded additive branch, no other checkpoint uses it. NO-BOS-verified real ByteLevel gate. Runs EAGER (bf16, no decode graph). Gate: 16/16 vs vLLM 0.25.0 (STRICT 13/16 + near-tie 3/16, max gap 0.094 nats, 0 forward-divergent) — speed pending. **OLMo-3 W5 (batch3) IMPLEMENTED** (guarded additive edits to `olmo2.{h,cpp,weights}`, diff-inert for OLMo-2 — re-run OLMo-2 gate 16/16 UNCHANGED): per-layer interleaved routing off `config.layer_types` — sliding_attention layers use plain NeoX rope (theta 500000) + finite window (masked at the FA kernel, inert for short contexts), full_attention layers use a precomputed YaRN cos/sin cache (get_rope yarn: factor 8, original 8192, mscale=yarn_get_mscale(8)=1.2079=config attention_factor, indexed by real positions); dtype-aware loader (OLMo-3 is BF16 on-disk vs OLMo-2 F32). **BUT the pinned vLLM 0.25.0 oracle CANNOT run `allenai/OLMo-3-1025-7B`** — its transformers version predates OLMo-3's nested per-layer-type rope schema: `olmo2.py:143` does `rope_parameters["rope_theta"]` → `KeyError: 'rope_theta'` (top-level rope_theta not folded into standardized rope_parameters), and forcing it surfaces `Unrecognized keys {'sliding_attention','full_attention'}` → `TypeError: unhashable type: 'dict'`. So there is NO pinned-oracle SACRED bar for OLMo-3 (DEP-blocked, spec D5); our engine LOADS + RUNS it (bootstrap loaded clean). W5 SACRED gate pending an oracle that can construct OLMo-3's rope config. | ✅ [sweep-olmo2](specs/sweep-olmo2.md) | `PARTIAL` | `test_olmo2_paged_engine` 16/16 (dgx, 92 assertions, batch3 re-run UNCHANGED); `test_olmo3_paged_engine` present (oracle-blocked, skips); loader 179 tensors zero-unmapped; registry resolves both arch strings; registration `src/vllm/model_executor/models/olmo2_registry.cpp:126`; test `tests/vllm/models/test_model_registry.cpp:118` | `CLAIM-SWEEP-OLMO2` (Claude Code opus-4-8) | | `MODEL-TEXT-olmo-hybrid-olmo-hybrid-for-causal-lm` | `OlmoHybridForCausalLM` | `registry.py:174`; `vllm/model_executor/models/olmo_hybrid.py::OlmoHybridForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; Mamba/SSM state; GDN/linear-attention state | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index d938f39cc..865de9075 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -88,6 +88,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#206](https://github.com/mudler/vllm.cpp/issues/206) | `KERNEL-SSM-MAMBA` | RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT and VRAM gaps vs vLLM — owns the sm_120 post-conv token tile and the K=4 causal-conv arm (PR #155) | feature | | [#305](https://github.com/mudler/vllm.cpp/issues/305) | `KERNEL-SSM-MAMBA` | GDN causal-conv: the `conv_state` initial-state read races the final-state write across blocks (`VT_CONV_REG` + exact chunks, both default ON) | bug | | [#496](https://github.com/mudler/vllm.cpp/issues/496) | `KERNEL-SSM-MAMBA` | Mamba2 SSD core is unported: the row carries only the GDN arm, so every hybrid-SSM arch (Nemotron-H/3.5, Granite-4, Zamba2, Falcon-H1, Mamba2) is blocked | feature | +| [#517](https://github.com/mudler/vllm.cpp/issues/517) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | Run Nemotron-3.5-Lightning-30B-A3B-NVFP4 e2e: NemotronH hybrid (23 Mamba2 / 6 GQA / 23 non-gated relu2 MoE), first MIXED_PRECISION checkpoint | feature | | [#352](https://github.com/mudler/vllm.cpp/issues/352) | `KV-EVENTS` | The batch envelope is never published and `kv_cache_report_mode` does not exist — events can only be observed by reaching into the `BlockPool` | feature | | [#353](https://github.com/mudler/vllm.cpp/issues/353) | `KV-EVENTS` | `KVEventsConfig` has no `__post_init__`: an enabled config with an unset publisher reports `unknown event publisher ''` instead of resolving to zmq | bug | | [#396](https://github.com/mudler/vllm.cpp/issues/396) | `KV-EXTERNAL-CACHE` | `test_lmcache_connector` data race under TSan: `MockLmcacheServer` writes non-atomic `listen_fd_` before joining its accept thread | bug | diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md new file mode 100644 index 000000000..cbccc7948 --- /dev/null +++ b/.agents/specs/nemotron-h-model.md @@ -0,0 +1,211 @@ +# Nemotron-H — the first hybrid Mamba2 model, and the first MIXED_PRECISION checkpoint + +**Claim:** `CLAIM-MODEL-NEMOTRON-H`. **Model row:** +`MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` (existing, stays `INVENTORIED` +at this spec commit). **Issue:** [#517](https://github.com/mudler/vllm.cpp/issues/517). + +**Hard blocker:** `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496) +([spec](mamba2-ssd.md)). W1 (CPU host references) is in fresh review on +`row/KERNEL-SSM-MAMBA-SSD-W1`; the CUDA arm is W2 of that row. **No forward +path in this spec can be gated before that lands**, and W3 below says so +explicitly rather than pretending otherwise. + +**Base:** `main` HEAD `66deca15`. +**Pinned oracle:** `${VLLM_SOURCE}` = `/home/mudler/_git/vllm` @ +`5559679229bc961848b121ccdeaa8fa5d79bec98` (vLLM 0.26.0.dev0). +**Driver checkpoint:** `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` +at pinned revision `29f2d1746d8f41e316523194b19018707749b1b1`, staged on the +NAS at `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4` (20.1 GiB, fits one +GB10). + +--- + +## 0. Scope (headline verdict) + +Two things make this arch more than "another model file", and both are the +first of their kind in this tree: + +1. **It is the first true hybrid Mamba2 model.** 52 layers: 23 Mamba2, 6 GQA + attention, 23 MoE. Our GDN hybrids (Qwen3.5, Kimi-Linear) supply the het-KV + machinery, but not the recurrence — see [mamba2-ssd.md](mamba2-ssd.md). +2. **It is the first `MIXED_PRECISION` checkpoint.** One file carries NVFP4 + W4A16 group-16 experts, FP8 W8A8 static mamba projections, bf16 attention, + and an fp8 KV scheme. We have never resolved a quant algorithm *per module*; + `quantization_config` is read ad-hoc in exactly two weight files today + (`kimi_k3_weights.cpp:171`, `deepseek_v2_weights.cpp:365`). + +A third thing is smaller but has no local precedent either: the MoE is +**non-gated**. There is no `gate_proj` anywhere in the checkpoint — the expert +is `up_proj -> relu² -> down_proj`. Every grouped-MoE op we have is +SwiGLU-shaped. + +**Out of scope, explicitly:** `NemotronHPuzzleForCausalLM` heterogeneous +per-layer configs (`get_nemotron_h_config_for_layer`), `moe_latent_size` +(null in this checkpoint, so `fc1_latent_proj`/`fc2_latent_proj` are absent), +TP sharding of `n_groups`, ReplaySSM, and any speed claim. Correctness first, +always; no ratio is recorded by this row until the token gate passes. + +## 1. The checkpoint, exactly + +`architectures: ["NemotronHForCausalLM"]`, `model_type: nemotron_h`, +`hidden_size=2688`, `vocab_size=131072`, `max_position_embeddings=1048576`, +`tie_word_embeddings=false`, weight prefix `backbone.`. + +`layers_block_type` (52 entries) gives 23 `mamba`, 23 `moe`, 6 `attention` +at indices **5, 12, 19, 26, 33, 42**. + +| Block | Parameters | +|---|---| +| Mamba2 | `mamba_num_heads=64`, `mamba_head_dim=64`, `n_groups=8`, `ssm_state_size=128`, `conv_kernel=4`, `chunk_size=128`, `mamba_hidden_act=silu`, `use_conv_bias=true`, `use_bias=false`, `mamba_proj_bias=false`, `mamba_ssm_cache_dtype=float32`, `time_step_min/max/floor = 1e-3 / 1e-1 / 1e-4` | +| Attention | 32 q / 2 kv heads, `head_dim=128`, `rope_theta=10000`, `partial_rotary_factor=1.0`, `attention_bias=false`, `sliding_window=null` | +| MoE | `n_routed_experts=128`, `num_experts_per_tok=6`, `moe_intermediate_size=1856`, `n_shared_experts=1`, `moe_shared_expert_intermediate_size=3712`, `mlp_hidden_act=relu2`, `norm_topk_prob=true`, `n_group=1`, `topk_group=1`, `routed_scaling_factor=2.5`, `moe_shared_expert_overlap=true` | +| MTP | `num_nextn_predict_layers=1`, `mtp_layers_block_type=["attention","moe"]`, weights `mtp.layers.0.{eh_proj,enorm,hnorm,final_layernorm,norm,mixer.*}`, unquantized | + +**Quantization** — `quant_method: modelopt`, `quant_algo: MIXED_PRECISION`, +`producer: modelopt 0.44.0rc5`, two `config_groups` plus a 5981-entry +`quantized_layers` map and an `ignore` list: + +| Target | Scheme | Tensors | +|---|---|---| +| routed experts, shared experts, `lm_head` | `W4A16_NVFP4`, **`group_size=16`** | `weight`, `weight_scale` (e4m3 per-16-block), `weight_scale_2` (fp32 global) | +| mamba `in_proj` / `out_proj` (46 targets) | FP8 W8A8 static | `weight`, `weight_scale`, `input_scale` | +| attention `q/k/v/o_proj`, `conv1d`, gates, norms, embeddings | unquantized bf16 | — | +| KV cache | `kv_cache_scheme` fp8 | `k_proj.k_scale`, `v_proj.v_scale` | + +Note the polarity trap: the repo name says NVFP4, and most of the *parameters* +are, but the mamba projections are FP8 and the attention tower is bf16. Reading +it as uniform NVFP4 gets the loader wrong in a way a token gate can still pass +while moving the wrong bytes — see [porting.md](../porting.md) on checking the +memory format against the oracle explicitly. + +## 2. Upstream chain (`file:line` @ `555967922`) + +| What | Anchor | +|---|---| +| registry | `registry.py:179` -> `models/nemotron_h.py::NemotronHForCausalLM` | +| layer dispatch (`ALL_DECODER_LAYER_TYPES`, `M`/`*`/`E`/`-`) | `nemotron_h.py:531-536`, model `:546-600` | +| Mamba2 layer | `nemotron_h.py:373-389` (`MambaMixer2`, fed `mamba_num_heads * mamba_head_dim`) | +| MoE | `nemotron_h.py:126-256` (`NemotronHMoE`), decoder layer `:317` | +| non-gated activation | `activation_without_mul(config.mlp_hidden_act)` -> `ReLUSquaredActivation` (`layers/activation.py`) | +| expert ckpt naming | `ckpt_names=("up_proj", "down_proj", "")` (`nemotron_h.py:220`) | +| routed scale applied to OUTPUT | `apply_routed_scale_to_output=True` (`nemotron_h.py:246`) | +| router dtype | `GateLinear(..., out_dtype=torch.float32, force_fp32_compute=True)` (`nemotron_h.py:150-156`) | +| state shape / dtype | `mamba_utils.py:174-199`, `:73-81` | +| MTP | `models/nemotron_h_mtp.py::NemotronHMTP` (`registry.py:638`) | +| MIXED_PRECISION resolution | `layers/quantization/modelopt.py:2280-2450`, per-layer lookup `:2416-2445` | + +The `quantized_layers` lookup is **direct name first, then shard prefix** +(`modelopt.py:2426`, `:2437`). Mirror both, in that order; a merged/sharded +local name that resolves only by prefix is the case that will bite. + +Config note: `nemotron_h.py` reads `config.hybrid_override_pattern`, which +newer transformers exposes as a property derived from `layers_block_type` +(`_list_to_pattern`, mapping `mamba->M`, `moe->E`, `attention->*`). Our loader +reads `layers_block_type` directly and does not reconstruct the char pattern. + +## 3. Our baseline — reuse vs new + +### REUSE (landed) + +- Het-KV: `MambaSpec` + `HybridKVCacheCoordinator` + per-group managers + (porting-inventory.md:78-79,109). Only 6 of 52 layers hold a paged KV group. +- Causal conv1d, all three arms (`kCausalConv1dFwd/Update/SpecUpdate`). +- Grouped-topk sigmoid routing with `e_score_correction_bias` and + `routed_scaling_factor` — the DeepSeek-V2/V4 path (`kMoeRouterTopK`). +- Shared experts (`kSharedExpertGate`, `kMoeCombineGate`). +- NVFP4 W4A16 grouped MoE Marlin (`kMoeGroupedGemmNvfp4Marlin`), FP8 W8A8 + linear, fp8 KV (`kReshapeAndCacheFp8`). +- MTP spec-decode machinery (`qwen3_5_mtp.cpp`, `SPEC-MTP`). +- Dense attention + rope (`dense_attn::AttnBlock`, `kAttnQkNormRope`). + +### NEW + +- `vt::Mamba2ChunkScan` / `Mamba2StateUpdate` / `RmsNormGatedGroup` — **owned + by #496, not by this row.** +- A non-gated `relu²` grouped-MoE arm on the merged-GEMM seam. +- A ModelOpt `MIXED_PRECISION` per-module quant resolver. +- `nemotron_h.cpp` / `nemotron_h_weights.cpp` / `nemotron_h_registry.cpp`, + and the MTP head. + +## 4. W-breakdown + +Each W is one delegated task with its own fresh implementer and fresh reviewer. +W1 and W2 are independent of #496 and can run in parallel with it; W3 onward +cannot. + +| W | Content | Gate | Depends on | +|---|---|---|---| +| **W1** | ModelOpt `MIXED_PRECISION` resolver: parse `quantization_config`, resolve per-module `quant_algo` (direct then shard-prefix), expose it to weight loading. Refuse an unknown algo by name | unit tests on the REAL `config.json` (committed as a fixture, weights not needed): every one of the 5981 entries resolves, the `ignore` list resolves to unquantized, an unknown algo refuses | — | +| **W2** | Non-gated `relu²` grouped MoE through `MlpGateUpMethodBase` / `vt::MergedGemmGroup`; bf16 arm then NVFP4 W4A16 g16 | byte/tolerance tests vs a host reference; `relu²` mutation caught; routed scale applied to the OUTPUT, not the logits | — | +| **W3** | `nemotron_h_weights.cpp` + `_registry.cpp`: `layers_block_type` dispatch, `backbone.` prefix, het-KV group construction (1 Mamba group + 1 full-attn group over 6 layers), enumeration gate vs the released index | enumeration: every tensor in `model.safetensors.index.json` is claimed or explicitly refused; KV spec shapes match `mamba2_state_shape` | #496 W1 | +| **W4** | `nemotron_h.cpp` forward: hybrid layer loop, Mamba2 mixer wiring, 6 attention layers, MoE layers | CPU forward runs; per-layer activations vs a dumped oracle reference | #496 W1, W2, W3 | +| **W5** | MTP head (`mtp.layers.0`, `eh_proj`/`enorm`/`hnorm`) on the existing spec-decode seam | draft acceptance non-zero; spec-off and spec-on token-identical | W4 | +| **W6** | **GB10 e2e token gate vs the pinned oracle** | token-exact greedy, identical prompts/counts/batching/sampling; oracle identity asserted | #496 W2 (CUDA), W4, W5 | + +## 5. Gates + +**Correctness first, always.** No throughput number is recorded by this row +until W6 passes. When speed is measured later, the denominator is vLLM's +production configuration, never `--enforce-eager`. + +**Oracle identity is asserted, not assumed.** `$HOME/venvs/vllm-oracle` on the +dgx host symlinks to `vllm-oracle-v0.25.0-stage` — vLLM **0.25.0**, transformers +5.13.1 — which predates `NemotronHMoEDecoderLayer` entirely. A run through that +venv fails on this checkpoint and reads as "the model is unsupported". The pin +is `vllm-oracle-next`: `0.23.1rc1.dev1511+g555967922`, transformers 5.14.1, +flashinfer 0.6.15.post1. Every oracle run asserts all three and ABORTS on +mismatch before producing a number. + +**The fixture must be the checkpoint the changed path loads.** Pin the +revision (`29f2d174`) explicitly; a repo silently re-quantized under the same +name has cost this project a full campaign before. + +**A token gate cannot see a dtype that is too wide.** Every f32 buffer on this +path owes a one-line reason, and the loaded memory format is checked against +the oracle explicitly, not inferred from matching tokens. + +**GPU discipline on dgx:** `flock $HOME/gpu.lock`, `local-ai-worker` parked, +never a large oracle alongside `ctest` — `gpu_memory_utilization` reserves HOST +RAM on GB10 and has OOM-rebooted the box. + +## 6. Risks / decisions + +- **Non-gated MoE must not become a parallel path.** If + `MlpGateUpMethodBase` / `vt::MergedGemmGroup` cannot represent a + gate-half-absent expert, extend the seam or record one exact tracked + exception. Never hand-roll a sibling. +- **`group_size=16` NVFP4.** Confirm our Marlin grouped path actually supports + 16 and does not silently assume another group size. Prove it on the real + tensors, not on a synthetic fixture. +- **Router in f32.** Upstream forces fp32 router compute + (`force_fp32_compute=True`). Mirror the polarity; do not inherit the model + dtype here. +- **`routed_scaling_factor` position.** Applied to the OUTPUT + (`apply_routed_scale_to_output=True`), not folded into the router weights. A + mis-placed scale is exactly the class of error a token gate catches late and + a unit test catches immediately. +- **6 attention layers out of 52** means KV is small and the 1M context is + cheap — but it also means an attention-side defect is diluted across 46 + non-attention layers and may not move tokens on short prompts. Gate with a + long-prompt arm, not only a 6-token one. + +## 7. Now + +**State at this commit:** spec committed, implementation **not started**. The +row stays `INVENTORIED`; this commit changes no lifecycle state. The checkpoint +is staged on the NAS and the oracle smoke run is queued behind the GPU lock. + +**Next action:** dispatch fresh implementers for **W1** and **W2** (both +independent of #496) as soon as `row/KERNEL-SSM-MAMBA-SSD-W1` clears review, +so the ops-header churn does not collide. + +## 8. Stop conditions + +- The pinned oracle cannot be made to load and run this checkpoint on GB10 → + stop and report; without a running oracle there is no gateable denominator + and the row does not proceed on source inspection alone. +- A `quantized_layers` entry names an algorithm we do not implement → refuse by + name and record it as owed. Never silently dequantize to a supported path; + that is invisible to a token gate. +- The non-gated expert cannot be expressed on the shared merged-GEMM seam → + `NEEDS_DECISION`, do not fork a parallel MoE path. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1f25791cd..c368654a2 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -166,7 +166,7 @@ Enumerated in `.agents/model-matrix.md`, not registered, no runnable GB10 gate: | `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | 671B, ~642 GiB fp8 vs 119 GiB unified; V3.2 also DSA-indexer dep-blocked | | `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | ~1404 GiB bf16; dep-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) | | `MiniMaxM2ForCausalLM` | MiniMax-M2 | ~230B, ~428 GiB bf16, ~4x over the unified pool | -| `NemotronHForCausalLM` | Nemotron-H / Nemotron-3.5-Lightning-30B-A3B | 20.1 GiB NVFP4 fits the pool, so capability-blocked, not HW-blocked: Mamba2 SSD is unported ([#496](https://github.com/mudler/vllm.cpp/issues/496)), plus non-gated `relu²` MoE and ModelOpt `MIXED_PRECISION` | +| `NemotronHForCausalLM` | Nemotron-H / Nemotron-3.5-Lightning-30B-A3B | capability-blocked, not HW-blocked (20.1 GiB fits the pool): Mamba2 SSD unported ([#496](https://github.com/mudler/vllm.cpp/issues/496)), MoE and loader owed; spec [#517](https://github.com/mudler/vllm.cpp/issues/517) | 27 of the 31 registered text-generation architectures carry a passing correctness gate today; the rest are honestly marked scaffold or blocked above.