diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 17278916a..71ae250cb 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -19,12 +19,20 @@ a practical unit that one agent can spike without silently dropping aliases. ## Architecture-support checklist At-a-glance view of which architectures we have actually engaged, and how far. -**326 architecture rows are inventoried at the pin, plus 3 beyond-pin rows -(`KimiK3ForConditionalGeneration` and `MuseGlimmerForConditionalGeneration`, both -released after the pin, and `MiniMaxH3DiTModel`, -which is a DIFFUSION architecture living in the separate `vllm-omni` repository) -= 329 total**; 45 are -past `INVENTORIED` (engaged), the remaining 284 are known-but-not-started long tail. Every mark +**326 architecture rows are inventoried at the pin, plus 10 rows that the pinned +registry does not contain = 336 total.** Those 10 are, by why they are not at the +pin: `KimiK3ForConditionalGeneration` and `MuseGlimmerForConditionalGeneration`, +both released after the pin; `BailingMoeV3ForCausalLM`, a pin-lag row registered +on vLLM `main` and absent only at the pin (#609); `MiniMaxH3DiTModel`, a DIFFUSION architecture living in the +separate `vllm-omni` repository; the four TTS / audio-generation architectures +registered in that same `vllm-omni` repository — `MossTTSDelayModel`, +`MossTTSRealtime`, `Qwen3TTSForConditionalGeneration` and +`HiggsMultimodalQwen3ForConditionalGeneration`; and two whose upstream target is +still pending because the exact `config.json` architecture string is registered +in neither core vLLM `main` nor `vllm-omni` — +`VoxtralRealtimeForConditionalGeneration` and +`BailingMMNativeForConditionalGeneration` (#610). Of the 336, 45 are +past `INVENTORIED` (engaged), the remaining 291 are known-but-not-started long tail. Every mark below is grounded in the row's lifecycle `State` cell plus its ledger evidence, and this section is CI-enforced against those rows by [`scripts/check-model-checklist.py`](../scripts/check-model-checklist.py): a mark @@ -44,7 +52,7 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | State | Rows | |---|---| -| INVENTORIED | 314 | +| INVENTORIED | 321 | | PARTIAL | 20 | | ACTIVE | 9 | | SPIKE | 7 | @@ -52,9 +60,9 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | DONE | 3 | | READY | 3 | | GATING | 1 | -| **Total** | **362** | +| **Total** | **369** | -Engaged architectures (the 49 non-`INVENTORIED` rows): +Engaged architectures (the 48 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| @@ -172,6 +180,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-TEXT-bailing-moe-bailing-moe-for-causal-lm` | `BailingMoeForCausalLM` | `registry.py:78`; `vllm/model_executor/models/bailing_moe.py::BailingMoeForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-bailing-moe-bailing-moe-v2-for-causal-lm` | `BailingMoeV2ForCausalLM` | `registry.py:79`; `vllm/model_executor/models/bailing_moe.py::BailingMoeV2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-bailing-moe-linear-bailing-moe-v25-for-causal-lm` | `BailingMoeV2_5ForCausalLM` | `registry.py:80`; `vllm/model_executor/models/bailing_moe_linear.py::BailingMoeV25ForCausalLM` | causal generation / text | model loader/forward; FusedMoE/grouped GEMM; Mamba/SSM state; GDN/linear-attention state; MLA/latent KV | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-TEXT-bailing-moe-v3-bailing-moe-v3-for-causal-lm` | `BailingMoeV3ForCausalLM` | **BEYOND-PIN (pin-lag) — NOT registered in `555967922`**; present on vLLM `main` at `registry.py:82` (`_TEXT_GENERATION_MODELS`) → module `vllm/model_executor/models/bailing_moe_v3.py`, class `BailingMoeV3ForCausalLM` (1495 lines, 9 classes), with its own MTP draft registered at `main` `registry.py:654` → module `bailing_moe_v3_mtp.py`, class `BailingMoeV3MTPModel`. Module and class are given as separate fields rather than in the compact colon-pair form: that form marks a target inside the PINNED registry inventory, and this row is beyond the pin — exactly how the `MuseGlimmerForConditionalGeneration`, `KimiK3ForConditionalGeneration` and `MiniMaxH3DiTModel` rows already render theirs. Succeeds `BailingMoeV2_5ForCausalLM` but is a **NEW MODULE, not a class rename in the old one**: V2.5 at the pin (`bailing_moe_linear.py`, 821 lines) mixes MLA full attention with Bailing linear attention (`vllm/model_executor/layers/mamba/linear/bailing_linear_attn`, imported `bailing_moe_linear.py:32`), whereas V3 keeps MLA but swaps the linear arm for **Kimi Delta Attention** (`vllm/third_party/flash_linear_attention/ops/kda` — `chunk_kda`, `fused_kda_gate`, `fused_recurrent_kda`, imported `bailing_moe_v3.py:74-79`) with a per-projection causal `conv1d` (`:650-671`), adds grouped-topk MoE through `config.n_group` (`:1140`), a clamped `SwigluStepAndMul` where a per-layer `swiglu_limit` is configured (`:387`, `:1107-1135`), an fp8 quant-config hook `_configure_ling_fp8_quant_config` (`:215`, called `:1354`), and REFUSES a checkpoint without `no_kda_lora=True` (`:603-604`). Recipe `inclusionAI/Ling-3.0-flash` declares `architectures: ["BailingMoeV3ForCausalLM"]`. Tracked by [#609](https://github.com/mudler/vllm.cpp/issues/609) | causal generation / text | model loader/forward; FusedMoE/grouped GEMM; KDA/linear-attention state; causal conv1d; MLA/latent KV; fp8 | ☐ required | `INVENTORIED` | none — cannot advance past this state until the pin advances to a revision that registers it | unassigned | | `MODEL-TEXT-bloom-bloom-for-causal-lm` | `BloomForCausalLM` | `registry.py:81`; `vllm/model_executor/models/bloom.py::BloomForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-chatglm-chat-glmfor-causal-lm` | `ChatGLMModel`, `ChatGLMForConditionalGeneration` | `registry.py:82-83`; `vllm/model_executor/models/chatglm.py::ChatGLMForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; legacy ChatGLM2/3 lineage — hand-written attention with `multi_query_group_num` (`chatglm.py:60-63`) and a HARDCODED partial rotary factor 0.5 (`:103`, applied `:131`), on an out-of-tree config requiring `trust_remote_code` (`:34`). Shares only the NEW partial-rotary primitive with the rest of GLM. Smallest genuine checkpoint `zai-org/chatglm3-6b` 6.2B / 23.3 GiB fp16 — FITS GB10. Lowest priority in the campaign (spike W9), deferrable without blocking anything | [glm-dsa-latest-deepseek spike](specs/glm-dsa-latest-deepseek.md) | `SPIKE` | none | `CLAIM-GLM-DSA-LATEST-DEEPSEEK` | | `MODEL-TEXT-commandr-cohere-for-causal-lm` | `CohereForCausalLM`, `Cohere2ForCausalLM` | `registry.py:84-85`; `vllm/model_executor/models/commandr.py::CohereForCausalLM` | causal generation / text | impl `include/vllm/model_executor/models/commandr.h` + `src/vllm/model_executor/models/commandr{,_weights,_registry}.cpp` (weight-only Cohere `LayerNorm` grounded `commandr.py:76-87,65-73`; GPT-J full-width RoPE `is_neox_style=false` `commandr.py:174-179`; PARALLEL residual `commandr.py:257-273`; `logit_scale` `commandr.py:376`; tied embeds `commandr.py:372`; ZERO new kernel, one REGISTER line); test `tests/parity/test_commandr_paged_engine.cpp` (SACRED gate, dgx-only, SKIPs — no runnable vehicle). CPU `-Werror` build compiles+links+registers. use_qk_norm/sliding rejected at parse (Cohere2 arch). | [recent-dense-batch](specs/sweep-recent-dense-batch.md) | `BLOCKED` | W0 oracle RUN-VERIFIED (tiny-random builds+runs, golden captured, `CohereForCausalLM`≠`Cohere2`); gate BLOCKED — real ckpts HF-gated (no dgx token) + only ungated are tiny-random head_dim 8/2 (outside validated attn) + dgx disk-full | `CLAIM-SWEEP-RECENT-DENSE` | @@ -413,6 +422,12 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | `MuseGlimmerForConditionalGeneration` | **BEYOND-PIN — NOT in `555967922`** (Muse Glimmer released 2026-08-08, after the pin, and NOT on vLLM `main` either); the only upstream implementation is the OPEN, approved-but-CI-red PR [vllm#51655](https://github.com/vllm-project/vllm/pull/51655) at head `075d645af` — `vllm/model_executor/models/muse_glimmer.py` (text attn `:1083`, decoder layer `:1212`, vision encoder `:692`, adapter `:1036`, MM processor `:262`), `vllm/transformers_utils/configs/muse_glimmer.py`, `vllm/transformers_utils/processors/muse_glimmer.py`, DFlash hooks in `qwen3_dflash.py:75-94` + `v1/spec_decode/dflash.py:83-95` | conditional generation / image+video | model loader/forward; MM processor; vision encoder + merge; video path; speculative decoding (DFlash); reasoning + tool parsers | [muse-glimmer spec](specs/muse-glimmer.md), issue [#268](https://github.com/mudler/vllm.cpp/issues/268) | `SPIKE` | **W-SPEC (2026-08-10, `row/MODEL-MUSE-GLIMMER`, records+spec only, NO code).** Traced end-to-end from PR #51655 @ `075d645af`. `meta-models/Muse-Glimmer-30B`, Apache-2.0, bf16, `model_type: muse_glimmer`; registry maps BOTH `MuseGlimmerForConditionalGeneration` and `MuseGlimmerForCausalLM` onto one class. **Text tower** L=52, H=6656, 32 q-heads / 2 kv-heads (GQA 16:1), head_dim 128, vocab 202048, 131k ctx, rope theta 5e5: Gemma2-style SANDWICH norms (input/post-attn/pre-ffn/post-ffn) computed in fp32 with a BAKED `+1` weight offset and SPLIT eps (pre-norms `rms_norm_eps`, post-norms the smaller `post_norm_eps`) (`muse_glimmer.py:1236-1247`); **iRoPE** — `no_rope_layers[i]==1` ⇒ RoPE AND sliding-window, `==0` ⇒ NoPE AND full attention, sliding×3 then full every 4th (`:1114-1116,:1167-1168`) — the ONE mechanism with no local analogue (we have no Llama-4); WEIGHTLESS QK-norm over head_dim in fp32 applied BEFORE RoPE (`:1189-1196`); post-QK-norm QUERY PRE-SCALE `scale_query_by`≈3.87 with softmax scaling left at `head_dim**-0.5` (`:1112,:1192`); per-head sigmoid ATTENTION OUTPUT GATE whose gate reads the LAYER INPUT, not the attn output (`:1203-1206`); no logit softcapping. **Two named correctness traps:** (a) the query pre-scale ships under two schemas — native raw `qk_scale_factor`≈43.784 vs modular PRE-FOLDED ≈3.87 — disambiguated upstream BY MAGNITUDE against `sqrt(head_dim)` (`:472-517`); mis-reading it scales every query by 11.3×; (b) `use_qk_norm` / `use_attn_output_gate` read as `None` (not `True`) in the modular schema and only an explicit `False` disables them (`:456-469`), so a naive `getattr(...,False)` silently drops BOTH mechanisms while still emitting plausible text. **Perception encoder** L=50, H=1536, 16 heads (head_dim 96), patch 14×14, `patch_temporal`=2, 32×32 learned pos-emb grid, interleaved window/full per `layer_types`, projector 4096→6144, image tok 200092 / video tok 200091, placeholders `<\|patch\|>`/`<\|image\|>`/`<\|video\|>`: LINEAR patchify (`conv1_linear`, NOT a conv) (`:696,:710`), BILINEAR pos-emb interpolation with per-corner validity masking and a half-pixel `+0.5/-0.5` convention (`:761-820`), 2D RoPE with **width-first** `cat([freq_w,freq_h])` (`:741-759`), block-windowed attention via a `-1`-padded permutation whose per-block valid counts become `seq_lens` (`:844-867`), pixel-shuffle downsample asserting `output_dim == hidden*merge^2` (`:822-842`), plain `LayerNorm` (not RMSNorm) at ln_pre/ln_post. **DFlash** adds NO new drafter — PR #51655 only recognises `MuseGlimmerAssistantModel` as method `dflash` and threads the TARGET's `is_neox_style` into the draft config, because a draft/target RoPE-layout mismatch is SILENT (acceptance collapses, nothing errors, output stays correct). **REUSE MAP:** sandwich norms → `gemma2.cpp`/`gemma4.cpp`; output gate → `qwen3_5.cpp`; weightless QK-norm → Qwen3 family; windowed vision tower → `qwen3_vl_vision.cpp`; gated MLP → `layers::MlpGateUpMethodBase`+`vt::MergedGemmGroup`; decode → `ModelRegistry::Forward`+`dense_attn::AttnBlock`; fusion → `vt::FusedChain`; DFlash → the existing speculator row. **HONESTY:** the pinned oracle CANNOT load `muse_glimmer` (and the checkpoint wants transformers 5.15.0.dev0 vs the pin's 5.14.1), so there is NO gateable oracle and **every performance axis is an OPEN GAP by construction** — no parity/throughput claim may be made from this row until #51655 merges and the pin advances. Correctness gates against the HF reference instead, with per-mechanism RED-first mutation carrying the weight the missing oracle would have. Weights (~60 GiB bf16) NOT downloaded; GB10 fit needs a quantized arm. W-plan W0-W7 in the spec. **W0 CPU SCAFFOLDING LANDED (2026-08-10, `CLAIM-MUSE-GLIMMER-W0`, CPU-only, no weights, no GPU):** additive TUs register BOTH `MuseGlimmerForCausalLM` and `MuseGlimmerForConditionalGeneration` onto ONE factory (mirroring registry.py @ #51655); `ParseMuseGlimmerParams` descends the canonical NESTED layout AND normalizes the older FLAT layout (configs/muse_glimmer.py:186-305 — without it a flat config silently deserializes to an ALL-DEFAULT text config, ignoring every checkpoint value with no error); `NormalizeMuseGlimmerWeightName` ports the `hf_to_vllm_mapper` (:1389-1425) for BOTH checkpoint conventions; `EnumerateMuseGlimmerTensors` is the structural name map, deliberately OMITTING the three WEIGHTLESS modules (`embed_norm` :1286, per-head `qk_norm` :1121, `perception_emb_norm` :1470) that ship no tensor; forward REFUSES-by-name. Gate `tests/vllm/models/test_muse_glimmer_scaffold.cpp` **11/11 cases, 73/73 assertions**, clean CPU `-Werror` build. **RED-FIRST MUTATION-PROVEN, all four named traps** (each mutated in-tree, gate went RED, tree restored byte-identical): (1) treating the native raw `qk_scale_factor` as pre-folded → 3 RED (the 11.3x query blow-up); (2) defaulting the absent `use_qk_norm`/`use_attn_output_gate` to FALSE → 4 RED; (3) counting the iRoPE mask FORWARD instead of backward from the last layer → 5 RED; (4) applying the legacy-guac sandwich-norm renames in the wrong order (which SWAPS post-attention with pre-feedforward) → 1 RED. KV-cache spec is a documented W0 placeholder (one full-attention group; the real sliding/full split rides the Gemma-4 per-layer seam at W1). **W1-W7 FOLLOWED ON THE SAME BRANCH (`row/MODEL-MUSE-GLIMMER`, PR #279, NOT merged), and this paragraph's "config/name-map/registry only, NO forward" is superseded:** W1 the 52-layer text forward (sandwich norms w/ split eps, iRoPE, weightless QK-norm before RoPE, query pre-scale, attn output gate, SwiGLU, untied lm_head, output_multiplier before the soft-cap) gated vs an independent fp32 transcription of `075d645af` plus a property test per mechanism (`test_muse_glimmer_text`, 21 cases / 487 assertions, and the SAME binary re-run under `VT_FUSED_CHAIN_ADOPT=0` so the non-FusedChain fallback arm is gated too); W3 the 50-layer perception encoder (`test_muse_glimmer_vision`); W4 the mm wiring — tower → adapter → projection → `perception_emb_norm` → masked scatter onto the image/video placeholder rows, with the released 30B's 1436 tensors accounted 1436/1436 (`test_muse_glimmer_wiring`, 9/10316); W7 the ATEM reasoning + tool parsers (34 cases / 251 assertions). **REAL-WEIGHT EVIDENCE, and its exact limits:** on real 30B tensors at **reduced depth 4 of 52** our forward's 5 prefill argmax positions are identical to our torch transcription AND to HF's own `muse_glimmer` (`exportable-muse` @`a9e337e8`); max abs diff 0.0889745, cosine 0.999981. NOT established, and not to be implied: the full-depth 52-layer arm NEVER RAN; these are prefill argmax positions, NOT generated tokens, so multi-step decode and the sliding window across steps are untested; the perception encoder has NO reference check of any kind (the wiring gate proves reachability and placement, not that an image produces the right tokens); nothing has run end to end through the server; and the ATEM parsers' channel scoping DOES NOT WORK at server defaults, because the ToolParser/ReasoningParser seam has no `adjust_request` dispatch site while `skip_special_tokens` defaults true and the checkpoint marks `<\|start\|>`/`<\|message\|>`/`<\|eom\|>`/`<\|eot\|>` special (open gap, spec §6.7). **NO vLLM SPEED AXIS ON ANY DIMENSION** — unchanged and unchangeable until #51655 merges and the pin advances. A SECONDARY, quant-matched llama.cpp bar now EXISTS and is measured (#333, 2026-08-11, `row/MUSE-BENCH-2`, spec §14): same 16.76 GB Q4_K_M file both engines on an idle GB10 CPU, 128-token prefill **tie at 0.997x** (0.898x at 20 threads), decode **0.232x**, 512-token prefill 0.170x, peak RSS **1.92x MORE** (30.29 vs 15.74 GiB, the §10.2 dequantized qkv/lm_head/embed). It is a secondary reference and is NEVER the denominator; the vLLM cell stays an open gap by construction. Row token stays `SPIKE` deliberately: advancing it owes `docs/STATUS.md` and `.agents/NOW.md` rows and both surfaces sit byte-exactly at their shrink-only ratchet/budget, so the advance belongs to the landing commit | `CLAIM-MUSE-GLIMMER-SPEC`, `CLAIM-MUSE-GLIMMER-W0`, `CLAIM-MUSE-GLIMMER-W1`, `CLAIM-MUSE-GLIMMER-W3`, `CLAIM-MUSE-GLIMMER-W4`, `CLAIM-MUSE-GLIMMER-W7`, `CLAIM-MUSE-GLIMMER-FIX` | | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | `KimiK3ForConditionalGeneration` | **BEYOND-PIN — NOT in `555967922`** (K3 released 2026-07-27, after the pin); closest registered = its literal text backbone `KimiLinearForCausalLM` (`registry.py:140`; `vllm/model_executor/models/kimi_linear.py`) + the K2.5 vision wrapper `kimi_k25.py:290` / tower `kimi_k25_vit.py` | conditional generation / image (text-first) | model loader/forward; FusedMoE/grouped GEMM; GDN/linear-attention state (KDA); MLA/latent KV; MXFP4 compressed-tensors quant; MM processor + MoonViT-V2 encoder/merge | [kimi-k3 spike](specs/kimi-k3.md) | `SPIKE` | **W0 SCOPE (2026-07-28, `CLAIM-KIMI-K3-SCOPE`, DERIVE-AND-SHIP, records-only).** From the HF `config.json` (fetch-derived): `architectures:["KimiK3ForConditionalGeneration"]`, `text_config.architectures:["KimiLinearForCausalLM"]` — the text backbone IS the pinned Kimi-Linear hybrid, MASSIVELY scaled: **H=7168, L=93 (69 KDA + 24 MLA full-attn), 896 experts / top-16 / 2 shared, `moe_intermediate_size=3072`**; MLA geometry `kv_lora=512`/`q_lora=1536`/`qk_nope=128`/`qk_rope=64` (= our landed DeepSeek-V3 dims); KDA `head_dim=128`/`num_heads=96`/`short_conv=4`/`gate_lower_bound=-5.0`; quant **`mxfp4-pack-quantized` (compressed-tensors, group 32, e8m0) + MXFP8 acts (QAT)**; vision **MoonViT-V2** (~401M, patch 14, 27L). **HEAVY REUSE** — GDN (KDA's parent, `cuda_gdn.cu`/`gdn_attn.cpp`), DeepSeek MLA (`deepseek_v2.cpp`/`mla_attention.*`, exact geometry), DeepSeek-style MoE (`qwen3_moe.cpp`/`cuda_moe.cu`, scale to 896), and the Qwen3.6-35B GDN-hybrid-MoE model skeleton (`qwen3_5_moe.cpp`) are the structural twins; Kimi-K2 tokenizer/tool parser (`parser/kimi_k2.cpp`) reused. **NET-NEW:** the KDA kernel delta (per-channel `[H,D]` low-rank decay `f_a_proj`/`f_b_proj` + sigmoid-gated output norm + 3 q/k/v convs — already scoped on the Kimi-Linear row), **MXFP4** (we have NVFP4 group-16, not MXFP4 group-32/e8m0), **AttnRes** (report-only, UNCONFIRMED — not in config.json nor pinned `kimi_linear.py`), and the **MoonViT-V2 tower**. **HW-fit: DOES NOT FIT GB10** — 2.8T MXFP4 ≈ **1.56 TB ≈ ~12× over the 119 GiB pool**; no small K3 exists. **DERIVE-AND-SHIP** (no on-box golden, like the beyond-vLLM CUDA bricks): (a) REAL proxy gate of KDA+MLA+MoE on the FITTING `Kimi-Linear-48B-A3B` (~89–91 GiB) vs the pinned oracle, (b) build-verify + structural review for the K3 scale-up. The pinned oracle has NO `kimi_k3` ⇒ even HW-rich users need a pin advance to oracle-gate K3 itself. CORRECTS the 2026-07-25 sweep note ("loads as `DeepseekV3ForCausalLM`" — true for K2, NOT K3). W-plan W1-W8 in the spec. **W2/W5 CPU SCAFFOLDING LANDED (2026-07-28, `CLAIM-KIMI-K3-W2-W5`, DERIVED+BUILD-VERIFIED, NOT pushed):** additive registry TU registers `KimiK3ForConditionalGeneration` (info: text-gen + `is_hybrid` + `has_inner_state` + `supports_multimodal`); config-descent `ParseKimiK3Params` reads the nested `text_config` (KimiLinear KDA+MLA+MoE scalars — note upstream key `num_experts_per_token`), `vision_config` (MoonViT-V2 PARTIAL), `quantization_config` (MXFP4 detect) grounded in `configs/kimi_linear.py:11-148`; pure `EnumerateKimiK3TextBackboneTensors` is the 93-layer KDA/MLA + 896-expert MoE structural name-map grounded 1:1 in `kimi_linear.py:104-378,460-554` + `kimi_gdn_linear_attn.py:102-226` (per-layer KDA vs MLA-with-qLoRA vs MoE-vs-dense branching); forward REFUSE-by-name (`VT_CHECK(false)`, mirrors `deepseek_v4.cpp`); loader REFUSES MXFP4 (a real K3 checkpoint's dtype) deferring to the shared DeepSeek-V4 MXFP4 row. **NOT-YET-BUILDABLE (correctly deferred):** MXFP4 materialization (→ `CLAIM-DEEPSEEK-V4-*` / quantization-matrix MXFP4), the KDA kernel delta (→ Kimi-Linear row `MODEL-TEXT-kimi-linear-*`), MoonViT-V2 vision (W7), K3 multimodal-wrapper weight prefix (post-pin). Code: `src/vllm/model_executor/models/kimi_k3{,_registry,_weights}.cpp` + `include/vllm/model_executor/models/kimi_k3.h`. Test: `tests/vllm/models/test_kimi_k3_scaffold.cpp` (6/6, 63 assertions — registry-resolve + config-descent + split logic + enumeration + reject + MXFP4-refuse). Clean CPU build (`-DVLLM_CPP_CUDA=OFF`). Row stays `SPIKE` (no on-box e2e; forward not implemented); registration `src/vllm/model_executor/models/kimi_k3_registry.cpp:126`; test `tests/vllm/models/test_kimi_k3_scaffold.cpp:123` | `CLAIM-KIMI-K3-SCOPE`, `CLAIM-KIMI-K3-W2-W5` | | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | `MiniMaxH3DiTModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (H3 released after the pin) and not in the vLLM repository at all: it lives in `vllm-project/vllm-omni`, `vllm_omni/diffusion/models/minimax_h3/` (`minimax_h3_transformer.py`, `packed_sequence.py`, `packed_tokens.py`, `scheduling_minimax_h3_euler_ancestral.py`, `denoise_loop.py`, `vae.py`, `encoder.py`, `pipeline_minimax_h3.py`) | diffusion generation / video + audio (text/image/video/audio in) | flow-matching denoise loop; packed varlen NON-CAUSAL attention; AdaLN modulation; 3D MM-RoPE; video VAE + audio VAE (checkpoint REMOTE CODE); Qwen3-VL-derived encoder; MP4 muxing; Ulysses sequence parallelism | [minimax-h3 spike](specs/minimax-h3.md) | `PARTIAL` | **W0-W2 LANDED (2026-08-03, `CLAIM-MINIMAX-H3-W0-W2`, DERIVE-AND-SHIP).** H3 is NOT an autoregressive LLM: it is a CFG-distilled joint video+audio DIFFUSION transformer (50 blocks, H=5376, 56 MHA heads x 128, SwiGLU 14336, AdaLN 6x3xH, 3D RoPE rotating 96 of 128 dims, video row width 96, audio latent 32) forwarded ONCE PER STEP of a 50-step flow-matching loop — no KV cache, no sampler, no logits, so the SACRED token-exact methodology does not apply. **HW VERDICT: e2e is IMPOSSIBLE on this project's hardware** — ~354 GB checkpoint (DiT 66.3 GB + Qwen3-VL-derived encoder 51.5 GB + video VAE ~10 GB + audio VAE ~0.6 GB), upstream validates on **4x NVIDIA B300 at ~133 GB peak per rank**, vs ONE GB10 with 119 GiB UNIFIED memory; CPU offload does not help because the pool IS host RAM. **WHAT IS GATED (and it is exact):** upstream's modules are pure Python, so they are executed at REDUCED DIMENSIONS on CPU as the oracle (`scripts/gen-minimax-h3-goldens.py` imports them by file path and freezes their outputs; both sides rebuild weights/inputs from an identical FNV-1a + splitmix64 stream, so no weight byte is checked in). Results: fl2va + ref2va packed layouts EXACT including the **fp64 position grid BIT-EXACT** (it feeds RoPE — the port reproduces numpy's `linspace(endpoint=False)` evaluation order and upstream's deliberately-split pairwise vs sequential span summations, `packed_sequence.py:101-113`); patchify/unpatchify/audio pack EXACT + round-trip; scheduler EXACT; **full DiT forward max abs diff 1.6e-7 (video) / 1.5e-7 (audio)** — f32 round-off. **REUSE:** the packed varlen non-causal attention routes through the SHARED `vt::DFlashBlockAttention(causal=false)` (its per-document bidirectional contract IS upstream's varlen FA call) and every projection through `vt::MatmulBT` — NO new kernel was added. **NOT-YET-BUILT (honest):** device-resident/bf16 forward + fusion folds (W2b), H3-Encoder on our existing Qwen3-VL tower (W3), the two VAEs — which are **checkpoint REMOTE CODE** under `trust_remote_code` and must be reimplemented in C++, not adapted (W4/W5) — pipeline/tasks (W6), `/v1/videos` + MP4 muxing, which needs a NEW dependency decision (W7), and USP multi-GPU (W8). No speed number is claimed; upstream reports the DiT at 88% of request latency. Code: [minimax_h3.h](../include/vllm/model_executor/models/minimax_h3.h#L1-L333), [minimax_h3.cpp](../src/vllm/model_executor/models/minimax_h3.cpp#L410-L640) (`MiniMaxH3DitForward`), [minimax_h3_packing.cpp](../src/vllm/model_executor/models/minimax_h3_packing.cpp#L259-L400) (`BuildMiniMaxH3PackedSequence`). Test: [test_minimax_h3.cpp](../tests/vllm/models/test_minimax_h3.cpp#L376-L470) (DiT forward parity; 10/10 cases / 2539 assertions, clean CPU build) + generator [gen-minimax-h3-goldens.py](../scripts/gen-minimax-h3-goldens.py#L1-L60); [ledger](parity-ledger.md#L889). **W6A+W9 LANDED (2026-08-03, `CLAIM-MINIMAX-H3-W6A-W9`) + HW VERDICT CORRECTED.** (a) the **bf16 PRODUCTION dtype policy** now runs (upstream's cast points with the fp32 islands preserved; gated vs a bf16 upstream golden at max abs diff 2.4e-3); (b) **request planning** — 17n+5 frame snapping, video/audio latent shapes, the rectified-flow time-shift sigma schedule, canvas resolution and t2va/fl2va/ref2va dispatch — ported and EXACT vs `time_request.py` + `pipeline_minimax_h3.py:121-122,207-222,374-434`; (c) the **ComfyUI-GGUF arm**: the name map is the IDENTITY and every one of the **535 real tensors** of `MiniMax-H3-FL2VA-Q3_K_M.gguf` resolves onto our contract, with the geometry derived from SHAPES ALONE equal to the shipped config (gated on a manifest read from the file's own header by range request — no payload downloaded). Two shape rules recorded: GGUF `ne` is reversed vs torch, and `comfy.gguf.orig_shape.` overrides it where ComfyUI reshaped a tensor for quant-block alignment (the 50 AdaLN projections: logical [96768, 2688], 2688 not a multiple of the 256-element Q3_K block). **★ HARDWARE VERDICT CORRECTED — the earlier 'e2e is IMPOSSIBLE on this hardware' was WRONG** because it reasoned from the bf16 release alone: quantized H3 checkpoints exist and FIT (GGUF DiT Q3_K_M 15.6 GB + Qwen3-VL encoder Q4_K_M 14.6 GB + VAEs ~11 GB ~= **41 GB** in a 119 GiB pool; `lilcheaty/MiniMax-H3-NVFP4` likewise). So e2e AND a speed comparison are REACHABLE; NVFP4 is the likely speed path (sm_121 native FP4 tensor cores + our tuned NVFP4 stack). **W5 LANDED (audio VAE)**: H3's VAEs are checkpoint REMOTE CODE under `trust_remote_code`, so a no-Python engine must REIMPLEMENT them — the DAC-lineage BigVGAN audio decoder (weight-norm materialization (w = g*v divided by the row norm), anti-aliased SnakeBeta with kaiser-sinc up/down resampling, replicate padding, final clamp) is ported and gated against the checkpoint's OWN modules at **max abs diff 4.2e-9**. The VIDEO VAE (`klvae.py` ~48 KB + CNN/ViT + tiling) is the largest remaining brick. **W10 GROUNDED + W4 SCOPED from REAL manifests** (safetensors headers captured by range request, no payload downloaded): the NVFP4 checkpoint's 1051 tensors are textbook compressed-tensors NVFP4 (U8 packed 2-per-byte + E4M3 `weight_scale` at group 16 + F32 scalar `weight_scale_2`; 258 quantized projections; fp32/bf16 islands left unquantized; names identical to our contract) — i.e. EXACTLY the layout our tuned NVFP4 stack already consumes, so W10 is loader wiring, not a new quant scheme. The video VAE's 560 tensors show its ENCODER is the 3D CNN (rank-5 Conv3d) while its DECODER — the half generation needs — is a 36-block TRANSFORMER (to_qkv/to_out, ff.w1/w2, 2 norms + 2 learned residual scales per block, x_embedder/mask_token/register_tokens/proj_out), materially smaller than klvae.py's 48 KB suggested. **W4 BLOCK LANDED**: the video-VAE decoder's repeated `TransformerBlock` is ported and gated at **6.0e-8** against the checkpoint's OWN remote code — RMSNorm + per-head RMS qk-norm (no affine) + full attention + gated-SiLU FF + LEARNED PER-CHANNEL residual scales, and critically the PER-HEAD-INTERLEAVED qkv layout ([head][q,k,v], NOT the DiT's [q_all,k_all,v_all]) that would otherwise produce a plausible-but-wrong image. **W4 DECODER DONE**: the FULL ViT3D video-VAE decoder — pack, x_embedder, register/cls tokens, 3D RoPE (RotaryEmbeddingND, length-normalized ids, angle scale 2pi, tiled freq blocks), the 36-block stack, LayerNorm norm_out, proj_out and unpatchify — is ported and gated at **8.9e-8** against the checkpoint's own `ViT3DDecoder`, at its real hyperparameters (36 layers, 32 heads x 64, rope_theta 100, rope_dim_ratio 0.75). BOTH VAE decoders are now done (audio 4.2e-9). **W3 TEXT TOWER DONE**: the H3-Encoder's truncated Qwen3-VL text tower — gated at **1.2e-7** vs upstream — with all three H3 deltas exercised: layer truncation (min(num_hidden_layers, 50)), the UNNORMALIZED layer-49 output (NO final RMSNorm, unlike stock Qwen3-VL — applying one silently shifts every conditioning vector), and DeepStack visual injection into the first N layers; plus interleaved M-RoPE, fused QKV, per-head q/k RMSNorm, causal GQA and the gated-SiLU MLP. **W6 t2va ASSEMBLED — the WHOLE PATH COMPOSES**: `MiniMaxH3GenerateT2va` wires packed layout -> rectified-flow sigma schedules -> the multi-step denoise loop of DiT forwards -> unpatchify + audio unpack -> per-channel denormalize -> BOTH VAE decoders, producing correctly-shaped, finite frames and a stereo waveform in [-1, 1]; gated by a structural end-to-end test at reduced dimensions with random weights (NOT a quality result). Assembling it also caught a real gap: the audio decode needed the checkpoint's `dec_in_proj` (Conv1d k=1, vae_latent_channels -> num_mels) ahead of BigVGAN. **W9 GGUF ARM DONE**: `LoadMiniMaxH3DitFromGguf` materializes the DiT from a ComfyUI-format GGUF — dequantizing every tensor through the SHARED GGUF dequant entry point (so the Q2_K/Q3_K/Q4_K families the H3 GGUFs use are covered by the same code every other GGUF model uses), recovering the geometry from shapes alone, and binding the forward's views with missing tensors throwing BY NAME rather than reading as zeros; gated by a synthetic-file load-AND-RUN test (a real DiT forward executes off the loaded weights). **W10 LOADER DONE**: `LoadMiniMaxH3DitFromNvfp4` materializes the DiT from an NVFP4 compressed-tensors checkpoint — the U8-packed [out, in/2] weight plus its E4M3 group-16 `weight_scale` and F32 scalar `weight_scale_2` go through the project's EXISTING NVFP4 dequant (no new quant code), sidecars are excluded from the model tensor set, and the logical [out, in] shape is recovered; gated by a synthetic-file load-AND-RUN test. BOTH quantized loaders are now done. **W3 VISION BLOCK DONE** (6.0e-8): the repeated unit of the encoder's Qwen3-VL vision tower, which differs from the text tower in every way that matters numerically — LayerNorm WITH BIAS (not RMSNorm), a [q_all, k_all, v_all] qkv layout (not the video VAE ViT's per-head interleave), fp32 rotary, NON-CAUSAL attention segmented by `cu_seqlens` (the test asserts a perturbation in one packed image leaves the other's outputs BIT-IDENTICAL), and the TANH-approximate GELU. **W3 ENCODER COMPLETE**: the FULL vision tower also lands — Conv3d patch embed (kernel == stride, so a linear over the flattened patch), BILINEAR resampling of the learned position grid into spatial-merge order, the 2D rotary table, per-frame `cu_seqlens`, and both merger flavours (the final merger norms the PRE-shuffle width while the DeepStack mergers norm the POST-shuffle width, and both use exact-erf GELU unlike the block MLP's tanh approximation), gated over a RAGGED two-image batch. Only the MM processor remains on the encoder. **CONDITION-NOISE augmentation DONE** (fl2va/ref2va): the noised-anchor mix plus its ROW ACCOUNTING — each visual condition draws noise of length `target_latent_t + imgvid_cond_num_frames` and slices the PREFIX matching its own latent_t, every condition restarts the SAME seed (so concatenating and drawing once would differ for multi-reference requests), and rows advance by that condition's own patchified count. Gated EXACT with the noise supplied, so the comparison isolates the accounting from torch's RNG. **REFERENCE-VIDEO geometry + FRAME SCHEDULE DONE** (the pure-math half of `reference_video.py`): the canvas pipeline (aspect clamp -> 768 short edge -> max-pixel rescale -> nearest multiple of 32) and the 24-to-2 FPS frame resample with per-temporal-patch block timestamps, both EXACT. NOTE the rest of that module (probe, transcode, frame extraction, audio decode) shells out to ffmpeg and is blocked on the SAME external dependency decision as `/v1/videos` MP4 muxing — one decision unlocks reference-video INPUT decode and generated-video OUTPUT encode together. **VIDEO VAE TILING DONE**: the tile plan (smallest tile count whose MINIMUM overlaps still cover the axis, leftover slack distributed in whole `vae_ratio` units ROUND-ROBIN across the seams) plus the linear seam cross-fade, both EXACT. Shipped config tile 256 / overlap 64 / vae_ratio 16 (= prod(space_down), the 'f16' in f16t4). Getting the slack distribution wrong shifts every tile after the first and surfaces as seam artifacts rather than an error. **PRESENTATION TOKEN TAGS DONE** — the fl2va vision-span override the denoise loop requires callers to have applied: a vision block is `vision_start + pad*count + vision_end` and the WHOLE block, MARKERS INCLUDED, is tagged VIDEO; tagging only the pads would leave two markers as TEXT and shift every AdaLN modulation index after them. Gated EXACT, with the test proving each VIDEO run is a whole vision block. **VAE 3D-CNN ENCODER PRIMITIVES DONE**: causal Conv3d (all temporal padding on the LEFT so a frame never sees the future, `reflect` spatial padding), GroupNorm3D (32 groups, eps 1e-6, statistics spanning TIME as well as space) and ResnetBlock3D, gated EXACT — with CAUSALITY proven directly on the bare convolution (a change to the last frame provably cannot reach earlier frames). **Downsample3D DONE** too: the strided inter-level conv, whose subtlety is the ASYMMETRIC pre-pad — one pixel on the RIGHT of W and the BOTTOM of H before a stride-2 conv with padding (1,0,0); padding symmetrically instead shifts the whole sampling lattice by half a pixel, which is a silent wrong latent rather than an error. Only the EncoderFCN3D level-loop assembly remains on the VAE encoder. **VIDEO VAE COMPLETE — encoder AND decoder**: the whole 3D-CNN encoder level loop (conv_in -> per level [ResnetBlock3D x N then Downsample3D or a 1x1x1 channel match] -> GroupNorm -> SiLU -> conv_out) is gated EXACT. **MM PROCESSOR = REUSE, gated**: H3's `FL2VA/processor` is a stock `Qwen3VLProcessor`, so the multimodal front end is this project's EXISTING Qwen3-VL processor rather than a new port; H3's own config is parsed and driven through it (patch 16 / temporal 2 / merge 2, **0.5 normalization rather than CLIP statistics**, a 32-pixel grid, the 768x1344 default canvas proven an IDENTITY under smart_resize, and VIDEO bounds deliberately looser than the image ones). **With this every PORTABLE piece of the lane is done**; what remains is one dependency decision (ffmpeg media I/O + MP4 muxing) and GPU-blocked work (the device-resident FP4 forward and any speed number). **WAV OUTPUT** added: the decoded stereo waveform serializes to RIFF/WAVE 16-bit PCM, converting the VAE's CHANNEL-MAJOR layout to INTERLEAVED (getting that backwards yields audio that plays but with the channels time-smeared) and clamping rather than wrapping. Deliberately dependency-free, and required under EITHER outcome of the open MP4/muxer decision. **VIDEO OUTPUT PATH DONE**: PPM frame serialization (planar [C,T,H,W] -> row-major interleaved RGB, [-1,1] -> [0,255] clamped) plus the MP4 mux argv (h264/yuv420p + AAC, `-shortest`, `+faststart`). The built argv was RUN through real ffmpeg 6.1.1 and produced a VALID MP4 (ffprobe: h264 yuv420p video + AAC stereo at 32 kHz). The library never spawns a process — `src/vllm/` has no subprocess precedent — so it builds the artifacts and the command while the example/server layer invokes it. **`/v1/videos` API LOGIC DONE**: the request contract (H3 defaults — 50 steps, flow shift 12 video / 3 audio; both the vLLM-Omni `extra_params` nesting and a flat top-level spelling accepted; malformed input rejected with a reason rather than silently defaulted) and the job store (queued -> running -> succeeded/failed, illegal transitions throw, unknown ids reported so the route can 404, status JSON omits fields that do not apply, and concurrent creation is thread-safe). Remaining is mechanical glue: route registration + runner injection, with the ffmpeg call in `examples/` per the ratified boundary. Test: 34/34 + video-api 4/4 (9233 + 63 assertions). OPEN: there is no vllm-omni parity PIN — the upstream-sync protocol covers only the vLLM repo | `CLAIM-MINIMAX-H3-W0-W2`, `CLAIM-MINIMAX-H3-W6A-W9` | +| `MODEL-MM-moss-tts-moss-tts-delay-talker-for-generation` | `MossTTSDelayModel` | **OUT-OF-REPO** — not in `555967922` and not in the vLLM repository at all (absent from every dictionary of `registry.py` on `main` as well): it is registered by the separate `vllm-project/vllm-omni` repository, `vllm_omni/model_executor/models/registry.py:350-354` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py`, class `MossTTSDelayTalkerForGeneration`, with the shared stage-1 codec decoder at `registry.py:368-372` → module `moss_tts/modeling_moss_tts_codec.py`, class `MossTTSCodecDecoder`. Module and class are given as separate fields rather than in the compact colon-pair form, as the `MiniMaxH3DiTModel` row does: that form marks a target inside the PINNED vLLM registry inventory, and this target is in another repository entirely. Documented at `vllm-omni` `docs/models/supported_models.md:78`. Official recipes: `OpenMOSS-Team/MOSS-TTS`, `MOSS-TTSD-v1.0`, `MOSS-SoundEffect`, `MOSS-VoiceGenerator` — four of the 157 recipes at `vllm-project/recipes` `86c7777a`, all four declaring `architectures: ["MossTTSDelayModel"]`. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / codec decoder; delay-pattern multi-codebook decoding; reference-audio speaker encoder; streaming audio output. **We serve no audio-generation output modality at all today** | ☐ required | `INVENTORIED` | none. There is no `vllm-omni` parity PIN — the upstream-sync protocol covers only the vLLM repository — so an oracle for this row has to be established before it can advance | unassigned | +| `MODEL-MM-moss-tts-moss-tts-realtime-talker-for-generation` | `MossTTSRealtime` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:356-360` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py`, class `MossTTSRealtimeTalkerForGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention) — the SAME module as `MossTTSDelayModel` but a different class, so the two are separate targets rather than aliases. Documented at `vllm-omni` `docs/models/supported_models.md:79`. Official recipe: `OpenMOSS-Team/MOSS-TTS-Realtime` (1.7B). Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+audio in, AUDIO OUT (realtime/streaming TTS) | MM processor; audio detokenizer / codec decoder; realtime full-duplex streaming; incremental audio decode | ☐ required | `INVENTORIED` | none. Same missing-`vllm-omni`-pin blocker as the other omni rows | unassigned | +| `MODEL-MM-qwen3-tts-qwen3-tts-talker-for-conditional-generation` | `Qwen3TTSForConditionalGeneration`, `Qwen3TTSTalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:152-156` (and the explicit `Qwen3TTSTalkerForConditionalGeneration` spelling at `:157`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py`, class `Qwen3TTSTalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with the code2wav stage at `registry.py:162-166` → module `qwen3_tts/qwen3_tts_code2wav.py`, class `Qwen3TTSCode2Wav` and 12 Hz / 25 Hz tokenizers under `qwen3_tts/tokenizer_12hz/` and `tokenizer_25hz/`. Documented at `vllm-omni` `docs/models/supported_models.md:72-74`. Official recipe: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` (published at `recipes.vllm.ai`). The two architecture strings resolve to the identical `(module, class)` target so they are grouped on one row, per the alias rule | conditional generation / text in, AUDIO OUT (TTS, custom voice) | MM processor; audio detokenizer / code2wav; speaker-embedding conditioning; word timestamps; streaming audio output | ☐ required | `INVENTORIED` | none. Its text tower is Qwen3-derived, which is the one piece of this row we already ship; everything downstream of the talker (code2wav, audio tokenizer, speaker embeddings) is unported | unassigned | +| `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index ce96bdad7..c32d864a8 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -39,6 +39,7 @@ issue is not yet placed. Keyed record: update in place, never append. | Issue | Row | Title | Kind | |---:|---|---|---| | [#168](https://github.com/mudler/vllm.cpp/issues/168) | `BACKEND-CUDA-SM110` | Jetson AGX Thor (sm_110): 32B NVFP4, Tekken tokenizer blocker | feature | +| [#618](https://github.com/mudler/vllm.cpp/issues/618) | `BACKEND-GATE-CPU-LLAMACPP` | `test_cpu_x86_llamacpp_floor`'s contended-leg case is load-dependent: at loadavg 63 the harness exits `NO_QUIET_WINDOW` (4) instead of `GIVING_UP` (2), so the guarantee goes untested and the red reads as a defect in whatever diff is in flight | bug | | [#433](https://github.com/mudler/vllm.cpp/issues/433) | `BACKEND-GATE-CPU-LLAMACPP` | No x86_64 arm: the closed CPU floor is Arm/i8mm-only and every lever that closed it is Arm-specific | perf | | [#199](https://github.com/mudler/vllm.cpp/issues/199) | `BACKEND-METAL-MLX` | macOS MLX build fails on `-Werror` in MLX headers | bug | | [#41](https://github.com/mudler/vllm.cpp/issues/41) | `BACKEND-ROCM` | ROCm (AMD GPU) backend | feature | @@ -113,6 +114,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#395](https://github.com/mudler/vllm.cpp/issues/395) | `LORA-RUNTIME` | `main` is RED on `sanitize-cpu (address,undefined)`: `test_punica_cpu` `RefShrink` reads past `a_stacked` for an out-of-range slot | bug | | [#400](https://github.com/mudler/vllm.cpp/issues/400) | `LORA-RUNTIME` | `test_punica_cpu` does not cover the out-of-range slot guard in `BgmvShrink` or `BgmvExpandSlice`: dropping either leaves the suite green | bug | | [#432](https://github.com/mudler/vllm.cpp/issues/432) | `MODEL-MM-voxtral-voxtral-for-conditional-generation` | Voxtral audio TTFT is ~17x vLLM: the Whisper encoder attention is a scalar warp kernel where vLLM runs FA-2 | enhancement | +| [#610](https://github.com/mudler/vllm.cpp/issues/610) | `MODEL-MM-moss-tts-moss-tts-delay-talker-for-generation` | Six `vllm-omni` recipe architectures had no model-matrix row: the TTS / audio-generation output modality was entirely unplaced (rows added `INVENTORIED`; two are `target-pending`) | bug | | [#268](https://github.com/mudler/vllm.cpp/issues/268) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer (Meta, 30B agentic multimodal): text tower, perception encoder, DFlash drafter | feature | | [#329](https://github.com/mudler/vllm.cpp/issues/329) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer GGUF k-quants: text arm lands; mmproj blocked by a converter axis drop | feature | | [#333](https://github.com/mudler/vllm.cpp/issues/333) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer speed: no number on any axis; benchmark vs llama.cpp, HF, and ourselves | perf | @@ -120,6 +122,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#359](https://github.com/mudler/vllm.cpp/issues/359) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer GGUF k-quant generates INCOHERENT text where llama.cpp on the same file is coherent | bug | | [#412](https://github.com/mudler/vllm.cpp/issues/412) | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | Muse Glimmer config defaults are neutral values, not the architecture's constants: the released GGUF's sandwich post-norms ran at 1e-5 instead of 1e-8 | bug | | [#489](https://github.com/mudler/vllm.cpp/issues/489) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | GDN `output_gate_type` is never parsed: a "sigmoid" checkpoint silently computes silu gating, and no token gate can see it | bug | +| [#609](https://github.com/mudler/vllm.cpp/issues/609) | `MODEL-TEXT-bailing-moe-v3-bailing-moe-v3-for-causal-lm` | `BailingMoeV3ForCausalLM` (recipe `inclusionAI/Ling-3.0-flash`) is registered on vLLM `main` but absent at the pin and had no model-matrix row. The issue's other architecture, `Qwen3_5MoeForCausalLM`, is owned by [#490](https://github.com/mudler/vllm.cpp/issues/490) / PR #601, which registers it rather than only rowing it | bug | | [#298](https://github.com/mudler/vllm.cpp/issues/298) | `PERF-27B-LMHEAD-DSR` | `main` is RED on the `device-leakage` DSR ratchet: `PrepareLmHeadResident` added a 33rd `vt_ifdef` to the shared layer | bug | | [#213](https://github.com/mudler/vllm.cpp/issues/213) | `PERF-27B-LMHEAD-FP4` | Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity | perf | | [#339](https://github.com/mudler/vllm.cpp/issues/339) | `PERF-27B-LMHEAD-FP4` | 27B c1: every fp8 input projection asks for an f32 output, selecting the slower nvjet template family where vLLM emits bf16 (48 f32-out projections 18.51 ms vs 48 bf16-out 7.05 ms). The merged GDN `in_proj` arm is built DEFAULT OFF as `VT_GDN_FP8_IN_BF16`, spec [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) §Attempt 4 — UNMEASURED: no committed gate loads the fp8 tower (`row/GATE-27B-FP8-TOWER-GOLDEN` builds that arm) | perf | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index dad6332c4..dcb69a3cd 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -185,6 +185,7 @@ on the committed fixture); reranking/classify models are not yet registered. | Video | ✅ correctness-gated | ✅ | ✅ | ☐ | | Audio | ✅ correctness-gated | ✅ | ◐ | ◐ | | Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, unpruned AND pruned (§8.21); ABI v12 `vllm_video_*` | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ | +| Speech / audio GENERATION (TTS, vLLM-Omni lane) | ☐ not started. The six architectures reached by the nine official `vllm-project/recipes` TTS models are inventoried in `.agents/model-matrix.md` (#610); an inventoried row is not a supported one | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS) | not assessed | not assessed | | Multimodal over the OpenAI server | ◐ image request path wired, forward pending | ✅ | ✅ | ◐ | Image, video and audio are correct through the CLI and library. Over the HTTP @@ -192,7 +193,9 @@ API the image **request** path is wired end to end (`ROAD-V1-MM` W1-W3): the production server attaches the seam at `server_main.cpp:826`. Two residuals keep it from ✅: the model runner has no mm-forward consuming `Request.mm_features`, and no image codec is vendored (raw RGB only). Video, audio and multi-image over -HTTP are not started. +HTTP are not started. Audio **in** is gated; audio **out** does not exist: we +ship no TTS or speech-generation path on any surface, which is why that row is +the only ☐ in our column here. ## Speculative decoding diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 00053742c..ced9050f9 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -39,7 +39,25 @@ # delegates it. Its only upstream implementation is the still-OPEN # vllm#51655; see porting-inventory.md §9 deviation 16. Bumped because a new # row EXISTS, never to make a transition pass. - "MODEL": (AGENTS / "model-matrix.md", 362), + # 369 since 2026-08-13: +7 rows for the architectures behind official + # `vllm-project/recipes` models that had no row at all (#609, #610). One is + # pin-lag — `BailingMoeV3ForCausalLM` is registered on vLLM `main` and + # absent only at `555967922`. Six are out-of-repo: `MossTTSDelayModel`, + # `MossTTSRealtime`, `Qwen3TTSForConditionalGeneration` and + # `HiggsMultimodalQwen3ForConditionalGeneration` are registered by + # `vllm-project/vllm-omni`, and `VoxtralRealtimeForConditionalGeneration` + # and `BailingMMNativeForConditionalGeneration` are target-pending — their + # exact `config.json` architecture strings are registered in neither core + # vLLM `main` nor `vllm-omni`, so the rows record what was searched instead + # of an invented anchor. SEVEN, not eight: the audit's eighth architecture + # `Qwen3_5MoeForCausalLM` is rowed by #490 / PR #601, which registers it + # rather than only inventorying it. Two branches ADDING the same keyed row + # merge without a conflict and define it twice, so the row is left to its + # owner. None of the seven touches the at-the-pin model inventory below + # (324/373/356/310/261 is unchanged), because like the MuseGlimmer, KimiK3 + # and MiniMaxH3DiT rows they carry no pinned-registry target. Bumped + # because seven new rows EXIST, never to make a transition pass. + "MODEL": (AGENTS / "model-matrix.md", 369), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 53c29c846..394b347b8 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -274,9 +274,10 @@ def test_model_row_ratchet_is_load_bearing(self) -> None: Mirrors the ENGINE ratchet above, for the same reason and with more force: the MODEL count is the one that actually moves, because every new - architecture re-pins it by hand. Muse Glimmer took it 361 -> 362. Without - this, bumping the number to silence a failure is indistinguishable from - bumping it because a row really landed. + architecture re-pins it by hand. Muse Glimmer took it 361 -> 362; the + seven recipe architectures that had no row at all took it 362 -> 369 + (#609, #610). Without this, bumping the number to silence a failure is + indistinguishable from bumping it because a row really landed. """ clean: list[str] = [] agent_record.check_matrices(clean) @@ -299,6 +300,103 @@ def test_model_row_ratchet_is_load_bearing(self) -> None: agent_record.check_matrices(errors) require(errors, r"\d+ MODEL rows; expected \d+") + def test_recipe_backfill_rows_are_inside_the_model_ratchet(self) -> None: + """The #609/#610 rows and the 362 -> 369 bump are one semantic change. + + Mirrors `test_windows_release_row_is_inside_the_engine_ratchet`: name + the rows the bump was taken FOR, so a count raised to silence a broken + parse cannot look identical to a count raised because rows landed. Two + of the seven are pinned, one per issue; seven near-identical assertions + would add repetition, not force. + """ + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + self.assertEqual([error for error in errors if "MODEL rows" in error], []) + + for item_id in ( + "MODEL-TEXT-bailing-moe-v3-bailing-moe-v3-for-causal-lm", + "MODEL-MM-moss-tts-moss-tts-delay-talker-for-generation", + ): + found = [row for row in rows if row.item_id == item_id] + self.assertEqual(len(found), 1, item_id) + self.assertEqual(found[0].path.name, "model-matrix.md") + self.assertEqual( + found[0].field("state").strip().strip("`"), "INVENTORIED", item_id + ) + + def test_beyond_pin_rows_stay_out_of_the_at_pin_model_inventory(self) -> None: + """A beyond-pin row must not inflate the AT-THE-PIN model inventory. + + `check_model_invariants` pins what vLLM's OWN registry holds at + `555967922`: 324 category/target rows, 373 memberships, 356 + architectures, 310 targets, 261 modules. It counts a row only when the + Upstream cell carries a backticked `module`-colon-colon-`class` token. + So a row for an architecture that is NOT at the pin -- MuseGlimmer, + KimiK3, MiniMaxH3DiT, and the seven recipe architectures rowed for + #609/#610 -- deliberately spells its module and class as separate + fields instead, and contributes nothing. + + That is a convention, and an unenforced convention drifts. The moment + someone "helpfully" anchors one of those rows the pinned inventory + silently gains a target vLLM does not register at the pin, and the next + person to re-pin the counts bakes the error in. This proves the + omission is load-bearing rather than an oversight. + """ + clean: list[str] = [] + agent_record.check_model_invariants(clean) + self.assertEqual(clean, []) + + beyond_pin = ( + "MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation", + "MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation", + "MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit", + "MODEL-TEXT-bailing-moe-v3-bailing-moe-v3-for-causal-lm", + "MODEL-MM-moss-tts-moss-tts-delay-talker-for-generation", + "MODEL-MM-moss-tts-moss-tts-realtime-talker-for-generation", + "MODEL-MM-qwen3-tts-qwen3-tts-talker-for-conditional-generation", + "MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation", + "MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation", + "MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation", + ) + matrix = agent_record.AGENTS / "model-matrix.md" + lines = matrix.read_text(encoding="utf-8").splitlines(keepends=True) + for item_id in beyond_pin: + rows = [line for line in lines if line.startswith(f"| `{item_id}` |")] + self.assertEqual(len(rows), 1, item_id) + upstream = agent_record.split_cells(rows[0])[2] + self.assertEqual( + [v for v in re.findall(r"`([^`]+)`", upstream) if "::" in v], + [], + item_id, + ) + + # Anchoring exactly ONE of them the at-the-pin way must move the + # inventory off its pin, in the file rather than in a stub. + victim = "MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit" + mutated: list[str] = [] + for line in lines: + if line.startswith(f"| `{victim}` |"): + cells = line.split("|") + cells[3] += " `vllm/model_executor/models/minimax_h3.py::MiniMaxH3DiTModel` " + line = "|".join(cells) + mutated.append(line) + self.assertNotEqual(mutated, lines) + + errors: list[str] = [] + with tempfile.TemporaryDirectory() as tmp: + agents = Path(tmp) + (agents / "model-matrix.md").write_text("".join(mutated), encoding="utf-8") + # ROOT moves with AGENTS: the checker reports the path relative to + # it, so leaving ROOT pointing at the real tree raises instead of + # producing the error we are asserting on. + with ( + mock.patch.object(agent_record, "AGENTS", agents), + mock.patch.object(agent_record, "ROOT", agents), + ): + agent_record.check_model_invariants(errors) + + require(errors, r"model inventory .*expected") + def test_engine_summary_rejects_stale_area_rollup(self) -> None: source = agent_record.ENGINE_MATRIX.read_text(encoding="utf-8") current = next(