diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 9ae59607a..17278916a 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -245,7 +245,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-TEXT-mimo-v2-mi-mo-v2-flash-for-causal-lm` | `MiMoV2FlashForCausalLM` | `registry.py:166`; `vllm/model_executor/models/mimo_v2.py::MiMoV2FlashForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-mimo-v2-mi-mo-v2-for-causal-lm` | `MiMoV2ForCausalLM` | `registry.py:167`; `vllm/model_executor/models/mimo_v2.py::MiMoV2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-nemotron-nemotron-for-causal-lm` | `NemotronForCausalLM` | `registry.py:168`; `vllm/model_executor/models/nemotron.py::NemotronForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10) | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: row spec committed ([#517](https://github.com/mudler/vllm.cpp/issues/517), [spec](specs/nemotron-h-model.md)); BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10) | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-olmo-olmo-for-causal-lm` | `OlmoForCausalLM` | `registry.py:171`; `vllm/model_executor/models/olmo.py::OlmoForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-olmo2-olmo2-for-causal-lm` | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | `registry.py:172-173`; `vllm/model_executor/models/olmo2.py::Olmo2ForCausalLM` | causal generation / text | OLMo-2-0425-1B dense LANDED (W0-W4). Impl: `include/vllm/model_executor/models/olmo2.h` + `src/vllm/model_executor/models/{olmo2,olmo2_weights,olmo2_registry}.cpp` (TWO `REGISTER_VLLM_MODEL` for `Olmo2ForCausalLM`+`Olmo3ForCausalLM`, one shared factory; reuses the shared dense glue). ZERO new compute kernel — the two "distinctives" both reduce to WIRING over landed ops: (1) PURE POST-NORM (`norm_after`, `olmo2.py:261-277`) = standalone `vt::RmsNorm` on each sublayer OUTPUT + plain `vt::Add` residual re-join, NO pre-norm (the GLM-4 standalone-output-norm op, pre-norms dropped); (2) FULL-WIDTH QK-norm (`olmo2.py:113-117,160-172`) = two standalone `vt::RmsNorm` over `[T,q_size]`/`[T,kv_size]` before NeoX rope (NOT the per-head `kAttnQkNormRope`). Reuse: SiLU SwiGLU (`kSiluAndMul`), NeoX RoPE (`RopeFromCache`/`RopeNeox`), GQA/MHA paged glue (`dense_attn_block.h`), merged qkv/gate_up loader. Checkpoint is F32 on-disk → loader downcasts f32→bf16 (round-to-nearest-even, `vt::F32ToBF16`) to match vLLM-bf16. UNTIED lm_head (`tie_word_embeddings:false`). Loader 179 tensors, zero missing/unmapped. Tokenizer: ONE shared-TU touch — `tokenizer.cpp` accepts the OLMo-2 Split `behavior=Removed,invert=true` encoding (equiv. to `Isolated` for the full-cover cl100k regex → `kLlama3`); guarded additive branch, no other checkpoint uses it. NO-BOS-verified real ByteLevel gate. Runs EAGER (bf16, no decode graph). Gate: 16/16 vs vLLM 0.25.0 (STRICT 13/16 + near-tie 3/16, max gap 0.094 nats, 0 forward-divergent) — speed pending. **OLMo-3 W5 (batch3) IMPLEMENTED** (guarded additive edits to `olmo2.{h,cpp,weights}`, diff-inert for OLMo-2 — re-run OLMo-2 gate 16/16 UNCHANGED): per-layer interleaved routing off `config.layer_types` — sliding_attention layers use plain NeoX rope (theta 500000) + finite window (masked at the FA kernel, inert for short contexts), full_attention layers use a precomputed YaRN cos/sin cache (get_rope yarn: factor 8, original 8192, mscale=yarn_get_mscale(8)=1.2079=config attention_factor, indexed by real positions); dtype-aware loader (OLMo-3 is BF16 on-disk vs OLMo-2 F32). **BUT the pinned vLLM 0.25.0 oracle CANNOT run `allenai/OLMo-3-1025-7B`** — its transformers version predates OLMo-3's nested per-layer-type rope schema: `olmo2.py:143` does `rope_parameters["rope_theta"]` → `KeyError: 'rope_theta'` (top-level rope_theta not folded into standardized rope_parameters), and forcing it surfaces `Unrecognized keys {'sliding_attention','full_attention'}` → `TypeError: unhashable type: 'dict'`. So there is NO pinned-oracle SACRED bar for OLMo-3 (DEP-blocked, spec D5); our engine LOADS + RUNS it (bootstrap loaded clean). W5 SACRED gate pending an oracle that can construct OLMo-3's rope config. | ✅ [sweep-olmo2](specs/sweep-olmo2.md) | `PARTIAL` | `test_olmo2_paged_engine` 16/16 (dgx, 92 assertions, batch3 re-run UNCHANGED); `test_olmo3_paged_engine` present (oracle-blocked, skips); loader 179 tensors zero-unmapped; registry resolves both arch strings; registration `src/vllm/model_executor/models/olmo2_registry.cpp:126`; test `tests/vllm/models/test_model_registry.cpp:118` | `CLAIM-SWEEP-OLMO2` (Claude Code opus-4-8) | | `MODEL-TEXT-olmo-hybrid-olmo-hybrid-for-causal-lm` | `OlmoHybridForCausalLM` | `registry.py:174`; `vllm/model_executor/models/olmo_hybrid.py::OlmoHybridForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; Mamba/SSM state; GDN/linear-attention state | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index d938f39cc..865de9075 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -88,6 +88,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#206](https://github.com/mudler/vllm.cpp/issues/206) | `KERNEL-SSM-MAMBA` | RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT and VRAM gaps vs vLLM — owns the sm_120 post-conv token tile and the K=4 causal-conv arm (PR #155) | feature | | [#305](https://github.com/mudler/vllm.cpp/issues/305) | `KERNEL-SSM-MAMBA` | GDN causal-conv: the `conv_state` initial-state read races the final-state write across blocks (`VT_CONV_REG` + exact chunks, both default ON) | bug | | [#496](https://github.com/mudler/vllm.cpp/issues/496) | `KERNEL-SSM-MAMBA` | Mamba2 SSD core is unported: the row carries only the GDN arm, so every hybrid-SSM arch (Nemotron-H/3.5, Granite-4, Zamba2, Falcon-H1, Mamba2) is blocked | feature | +| [#517](https://github.com/mudler/vllm.cpp/issues/517) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | Run Nemotron-3.5-Lightning-30B-A3B-NVFP4 e2e: NemotronH hybrid (23 Mamba2 / 6 GQA / 23 non-gated relu2 MoE), first MIXED_PRECISION checkpoint | feature | | [#352](https://github.com/mudler/vllm.cpp/issues/352) | `KV-EVENTS` | The batch envelope is never published and `kv_cache_report_mode` does not exist — events can only be observed by reaching into the `BlockPool` | feature | | [#353](https://github.com/mudler/vllm.cpp/issues/353) | `KV-EVENTS` | `KVEventsConfig` has no `__post_init__`: an enabled config with an unset publisher reports `unknown event publisher ''` instead of resolving to zmq | bug | | [#396](https://github.com/mudler/vllm.cpp/issues/396) | `KV-EXTERNAL-CACHE` | `test_lmcache_connector` data race under TSan: `MockLmcacheServer` writes non-atomic `listen_fd_` before joining its accept thread | bug | diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md new file mode 100644 index 000000000..cbccc7948 --- /dev/null +++ b/.agents/specs/nemotron-h-model.md @@ -0,0 +1,211 @@ +# Nemotron-H — the first hybrid Mamba2 model, and the first MIXED_PRECISION checkpoint + +**Claim:** `CLAIM-MODEL-NEMOTRON-H`. **Model row:** +`MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` (existing, stays `INVENTORIED` +at this spec commit). **Issue:** [#517](https://github.com/mudler/vllm.cpp/issues/517). + +**Hard blocker:** `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496) +([spec](mamba2-ssd.md)). W1 (CPU host references) is in fresh review on +`row/KERNEL-SSM-MAMBA-SSD-W1`; the CUDA arm is W2 of that row. **No forward +path in this spec can be gated before that lands**, and W3 below says so +explicitly rather than pretending otherwise. + +**Base:** `main` HEAD `66deca15`. +**Pinned oracle:** `${VLLM_SOURCE}` = `/home/mudler/_git/vllm` @ +`5559679229bc961848b121ccdeaa8fa5d79bec98` (vLLM 0.26.0.dev0). +**Driver checkpoint:** `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` +at pinned revision `29f2d1746d8f41e316523194b19018707749b1b1`, staged on the +NAS at `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4` (20.1 GiB, fits one +GB10). + +--- + +## 0. Scope (headline verdict) + +Two things make this arch more than "another model file", and both are the +first of their kind in this tree: + +1. **It is the first true hybrid Mamba2 model.** 52 layers: 23 Mamba2, 6 GQA + attention, 23 MoE. Our GDN hybrids (Qwen3.5, Kimi-Linear) supply the het-KV + machinery, but not the recurrence — see [mamba2-ssd.md](mamba2-ssd.md). +2. **It is the first `MIXED_PRECISION` checkpoint.** One file carries NVFP4 + W4A16 group-16 experts, FP8 W8A8 static mamba projections, bf16 attention, + and an fp8 KV scheme. We have never resolved a quant algorithm *per module*; + `quantization_config` is read ad-hoc in exactly two weight files today + (`kimi_k3_weights.cpp:171`, `deepseek_v2_weights.cpp:365`). + +A third thing is smaller but has no local precedent either: the MoE is +**non-gated**. There is no `gate_proj` anywhere in the checkpoint — the expert +is `up_proj -> relu² -> down_proj`. Every grouped-MoE op we have is +SwiGLU-shaped. + +**Out of scope, explicitly:** `NemotronHPuzzleForCausalLM` heterogeneous +per-layer configs (`get_nemotron_h_config_for_layer`), `moe_latent_size` +(null in this checkpoint, so `fc1_latent_proj`/`fc2_latent_proj` are absent), +TP sharding of `n_groups`, ReplaySSM, and any speed claim. Correctness first, +always; no ratio is recorded by this row until the token gate passes. + +## 1. The checkpoint, exactly + +`architectures: ["NemotronHForCausalLM"]`, `model_type: nemotron_h`, +`hidden_size=2688`, `vocab_size=131072`, `max_position_embeddings=1048576`, +`tie_word_embeddings=false`, weight prefix `backbone.`. + +`layers_block_type` (52 entries) gives 23 `mamba`, 23 `moe`, 6 `attention` +at indices **5, 12, 19, 26, 33, 42**. + +| Block | Parameters | +|---|---| +| Mamba2 | `mamba_num_heads=64`, `mamba_head_dim=64`, `n_groups=8`, `ssm_state_size=128`, `conv_kernel=4`, `chunk_size=128`, `mamba_hidden_act=silu`, `use_conv_bias=true`, `use_bias=false`, `mamba_proj_bias=false`, `mamba_ssm_cache_dtype=float32`, `time_step_min/max/floor = 1e-3 / 1e-1 / 1e-4` | +| Attention | 32 q / 2 kv heads, `head_dim=128`, `rope_theta=10000`, `partial_rotary_factor=1.0`, `attention_bias=false`, `sliding_window=null` | +| MoE | `n_routed_experts=128`, `num_experts_per_tok=6`, `moe_intermediate_size=1856`, `n_shared_experts=1`, `moe_shared_expert_intermediate_size=3712`, `mlp_hidden_act=relu2`, `norm_topk_prob=true`, `n_group=1`, `topk_group=1`, `routed_scaling_factor=2.5`, `moe_shared_expert_overlap=true` | +| MTP | `num_nextn_predict_layers=1`, `mtp_layers_block_type=["attention","moe"]`, weights `mtp.layers.0.{eh_proj,enorm,hnorm,final_layernorm,norm,mixer.*}`, unquantized | + +**Quantization** — `quant_method: modelopt`, `quant_algo: MIXED_PRECISION`, +`producer: modelopt 0.44.0rc5`, two `config_groups` plus a 5981-entry +`quantized_layers` map and an `ignore` list: + +| Target | Scheme | Tensors | +|---|---|---| +| routed experts, shared experts, `lm_head` | `W4A16_NVFP4`, **`group_size=16`** | `weight`, `weight_scale` (e4m3 per-16-block), `weight_scale_2` (fp32 global) | +| mamba `in_proj` / `out_proj` (46 targets) | FP8 W8A8 static | `weight`, `weight_scale`, `input_scale` | +| attention `q/k/v/o_proj`, `conv1d`, gates, norms, embeddings | unquantized bf16 | — | +| KV cache | `kv_cache_scheme` fp8 | `k_proj.k_scale`, `v_proj.v_scale` | + +Note the polarity trap: the repo name says NVFP4, and most of the *parameters* +are, but the mamba projections are FP8 and the attention tower is bf16. Reading +it as uniform NVFP4 gets the loader wrong in a way a token gate can still pass +while moving the wrong bytes — see [porting.md](../porting.md) on checking the +memory format against the oracle explicitly. + +## 2. Upstream chain (`file:line` @ `555967922`) + +| What | Anchor | +|---|---| +| registry | `registry.py:179` -> `models/nemotron_h.py::NemotronHForCausalLM` | +| layer dispatch (`ALL_DECODER_LAYER_TYPES`, `M`/`*`/`E`/`-`) | `nemotron_h.py:531-536`, model `:546-600` | +| Mamba2 layer | `nemotron_h.py:373-389` (`MambaMixer2`, fed `mamba_num_heads * mamba_head_dim`) | +| MoE | `nemotron_h.py:126-256` (`NemotronHMoE`), decoder layer `:317` | +| non-gated activation | `activation_without_mul(config.mlp_hidden_act)` -> `ReLUSquaredActivation` (`layers/activation.py`) | +| expert ckpt naming | `ckpt_names=("up_proj", "down_proj", "")` (`nemotron_h.py:220`) | +| routed scale applied to OUTPUT | `apply_routed_scale_to_output=True` (`nemotron_h.py:246`) | +| router dtype | `GateLinear(..., out_dtype=torch.float32, force_fp32_compute=True)` (`nemotron_h.py:150-156`) | +| state shape / dtype | `mamba_utils.py:174-199`, `:73-81` | +| MTP | `models/nemotron_h_mtp.py::NemotronHMTP` (`registry.py:638`) | +| MIXED_PRECISION resolution | `layers/quantization/modelopt.py:2280-2450`, per-layer lookup `:2416-2445` | + +The `quantized_layers` lookup is **direct name first, then shard prefix** +(`modelopt.py:2426`, `:2437`). Mirror both, in that order; a merged/sharded +local name that resolves only by prefix is the case that will bite. + +Config note: `nemotron_h.py` reads `config.hybrid_override_pattern`, which +newer transformers exposes as a property derived from `layers_block_type` +(`_list_to_pattern`, mapping `mamba->M`, `moe->E`, `attention->*`). Our loader +reads `layers_block_type` directly and does not reconstruct the char pattern. + +## 3. Our baseline — reuse vs new + +### REUSE (landed) + +- Het-KV: `MambaSpec` + `HybridKVCacheCoordinator` + per-group managers + (porting-inventory.md:78-79,109). Only 6 of 52 layers hold a paged KV group. +- Causal conv1d, all three arms (`kCausalConv1dFwd/Update/SpecUpdate`). +- Grouped-topk sigmoid routing with `e_score_correction_bias` and + `routed_scaling_factor` — the DeepSeek-V2/V4 path (`kMoeRouterTopK`). +- Shared experts (`kSharedExpertGate`, `kMoeCombineGate`). +- NVFP4 W4A16 grouped MoE Marlin (`kMoeGroupedGemmNvfp4Marlin`), FP8 W8A8 + linear, fp8 KV (`kReshapeAndCacheFp8`). +- MTP spec-decode machinery (`qwen3_5_mtp.cpp`, `SPEC-MTP`). +- Dense attention + rope (`dense_attn::AttnBlock`, `kAttnQkNormRope`). + +### NEW + +- `vt::Mamba2ChunkScan` / `Mamba2StateUpdate` / `RmsNormGatedGroup` — **owned + by #496, not by this row.** +- A non-gated `relu²` grouped-MoE arm on the merged-GEMM seam. +- A ModelOpt `MIXED_PRECISION` per-module quant resolver. +- `nemotron_h.cpp` / `nemotron_h_weights.cpp` / `nemotron_h_registry.cpp`, + and the MTP head. + +## 4. W-breakdown + +Each W is one delegated task with its own fresh implementer and fresh reviewer. +W1 and W2 are independent of #496 and can run in parallel with it; W3 onward +cannot. + +| W | Content | Gate | Depends on | +|---|---|---|---| +| **W1** | ModelOpt `MIXED_PRECISION` resolver: parse `quantization_config`, resolve per-module `quant_algo` (direct then shard-prefix), expose it to weight loading. Refuse an unknown algo by name | unit tests on the REAL `config.json` (committed as a fixture, weights not needed): every one of the 5981 entries resolves, the `ignore` list resolves to unquantized, an unknown algo refuses | — | +| **W2** | Non-gated `relu²` grouped MoE through `MlpGateUpMethodBase` / `vt::MergedGemmGroup`; bf16 arm then NVFP4 W4A16 g16 | byte/tolerance tests vs a host reference; `relu²` mutation caught; routed scale applied to the OUTPUT, not the logits | — | +| **W3** | `nemotron_h_weights.cpp` + `_registry.cpp`: `layers_block_type` dispatch, `backbone.` prefix, het-KV group construction (1 Mamba group + 1 full-attn group over 6 layers), enumeration gate vs the released index | enumeration: every tensor in `model.safetensors.index.json` is claimed or explicitly refused; KV spec shapes match `mamba2_state_shape` | #496 W1 | +| **W4** | `nemotron_h.cpp` forward: hybrid layer loop, Mamba2 mixer wiring, 6 attention layers, MoE layers | CPU forward runs; per-layer activations vs a dumped oracle reference | #496 W1, W2, W3 | +| **W5** | MTP head (`mtp.layers.0`, `eh_proj`/`enorm`/`hnorm`) on the existing spec-decode seam | draft acceptance non-zero; spec-off and spec-on token-identical | W4 | +| **W6** | **GB10 e2e token gate vs the pinned oracle** | token-exact greedy, identical prompts/counts/batching/sampling; oracle identity asserted | #496 W2 (CUDA), W4, W5 | + +## 5. Gates + +**Correctness first, always.** No throughput number is recorded by this row +until W6 passes. When speed is measured later, the denominator is vLLM's +production configuration, never `--enforce-eager`. + +**Oracle identity is asserted, not assumed.** `$HOME/venvs/vllm-oracle` on the +dgx host symlinks to `vllm-oracle-v0.25.0-stage` — vLLM **0.25.0**, transformers +5.13.1 — which predates `NemotronHMoEDecoderLayer` entirely. A run through that +venv fails on this checkpoint and reads as "the model is unsupported". The pin +is `vllm-oracle-next`: `0.23.1rc1.dev1511+g555967922`, transformers 5.14.1, +flashinfer 0.6.15.post1. Every oracle run asserts all three and ABORTS on +mismatch before producing a number. + +**The fixture must be the checkpoint the changed path loads.** Pin the +revision (`29f2d174`) explicitly; a repo silently re-quantized under the same +name has cost this project a full campaign before. + +**A token gate cannot see a dtype that is too wide.** Every f32 buffer on this +path owes a one-line reason, and the loaded memory format is checked against +the oracle explicitly, not inferred from matching tokens. + +**GPU discipline on dgx:** `flock $HOME/gpu.lock`, `local-ai-worker` parked, +never a large oracle alongside `ctest` — `gpu_memory_utilization` reserves HOST +RAM on GB10 and has OOM-rebooted the box. + +## 6. Risks / decisions + +- **Non-gated MoE must not become a parallel path.** If + `MlpGateUpMethodBase` / `vt::MergedGemmGroup` cannot represent a + gate-half-absent expert, extend the seam or record one exact tracked + exception. Never hand-roll a sibling. +- **`group_size=16` NVFP4.** Confirm our Marlin grouped path actually supports + 16 and does not silently assume another group size. Prove it on the real + tensors, not on a synthetic fixture. +- **Router in f32.** Upstream forces fp32 router compute + (`force_fp32_compute=True`). Mirror the polarity; do not inherit the model + dtype here. +- **`routed_scaling_factor` position.** Applied to the OUTPUT + (`apply_routed_scale_to_output=True`), not folded into the router weights. A + mis-placed scale is exactly the class of error a token gate catches late and + a unit test catches immediately. +- **6 attention layers out of 52** means KV is small and the 1M context is + cheap — but it also means an attention-side defect is diluted across 46 + non-attention layers and may not move tokens on short prompts. Gate with a + long-prompt arm, not only a 6-token one. + +## 7. Now + +**State at this commit:** spec committed, implementation **not started**. The +row stays `INVENTORIED`; this commit changes no lifecycle state. The checkpoint +is staged on the NAS and the oracle smoke run is queued behind the GPU lock. + +**Next action:** dispatch fresh implementers for **W1** and **W2** (both +independent of #496) as soon as `row/KERNEL-SSM-MAMBA-SSD-W1` clears review, +so the ops-header churn does not collide. + +## 8. Stop conditions + +- The pinned oracle cannot be made to load and run this checkpoint on GB10 → + stop and report; without a running oracle there is no gateable denominator + and the row does not proceed on source inspection alone. +- A `quantized_layers` entry names an algorithm we do not implement → refuse by + name and record it as owed. Never silently dequantize to a supported path; + that is invisible to a token gate. +- The non-gated expert cannot be expressed on the shared merged-GEMM seam → + `NEEDS_DECISION`, do not fork a parallel MoE path. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1f25791cd..c368654a2 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -166,7 +166,7 @@ Enumerated in `.agents/model-matrix.md`, not registered, no runnable GB10 gate: | `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | 671B, ~642 GiB fp8 vs 119 GiB unified; V3.2 also DSA-indexer dep-blocked | | `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | ~1404 GiB bf16; dep-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) | | `MiniMaxM2ForCausalLM` | MiniMax-M2 | ~230B, ~428 GiB bf16, ~4x over the unified pool | -| `NemotronHForCausalLM` | Nemotron-H / Nemotron-3.5-Lightning-30B-A3B | 20.1 GiB NVFP4 fits the pool, so capability-blocked, not HW-blocked: Mamba2 SSD is unported ([#496](https://github.com/mudler/vllm.cpp/issues/496)), plus non-gated `relu²` MoE and ModelOpt `MIXED_PRECISION` | +| `NemotronHForCausalLM` | Nemotron-H / Nemotron-3.5-Lightning-30B-A3B | capability-blocked, not HW-blocked (20.1 GiB fits the pool): Mamba2 SSD unported ([#496](https://github.com/mudler/vllm.cpp/issues/496)), MoE and loader owed; spec [#517](https://github.com/mudler/vllm.cpp/issues/517) | 27 of the 31 registered text-generation architectures carry a passing correctness gate today; the rest are honestly marked scaffold or blocked above.