mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | DeepSeek-V4-Flash decode | **Closed: BEATS ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact). Phase-2 routed-expert residency NEGATIVE (−3.4%), default-OFF | — |
 | f32-out GEMV audit | Only laguna + deepseek_v4 bf16 tower affected; gate/on-framework dense unaffected | Re-verify deepseek_v4 tower same-tool |
 | Invocation-parity prevention | CI guard (`check-gemv-invocation-consistency.py`) + AGENTS.md checklist landing | Review + merge; CUDA build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | **LANDED on main (#26)**: portable path complete; e2e video on real weights (Thor); clean CPU build+ctest gated | FP4 speed path on GB10 (sm_121) |
+| MiniMax-H3 lane | **W-FP4a fp4 routing CPU-landed** (`row/H3-FP4-SPEED`): NVFP4 kept packed -> Marlin W4A16, no new quant code; gate 62/62 | GB10 CUDA gate; real e2e disk-blocked |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -13093,6 +13093,54 @@ regenerate the GGUF manifest by range-fetching the first 4 MiB of the .gguf and
 **Next gate:** download a quantized checkpoint and close the e2e loop (encoder -> VAEs -> pipeline), then W2b/W10 for speed. See
 [.agents/specs/minimax-h3.md](../.agents/specs/minimax-h3.md).
 
+## MiniMax-H3 W-FP4a — fp4-RESIDENT Marlin-W4A16 routing wired; CPU wiring-gate GREEN; GB10 fp4-vs-bf16 delta + s/step PENDING (2026-08-06, `row/H3-FP4-SPEED`, `ROAD-V1-H3`)
+
+**What changed.** The NVFP4 arm dequantized every packed FP4 projection to bf16 and
+ran `vt::MatmulBT`, so the sm_121a FP4 tensor-core route had NEVER run for H3. This
+keeps the FP4 packed and routes each quantized projection through
+`dense_nvfp4::MatmulNvfp4W4A16D` (Marlin W4A16, forced by vLLM's own a16 selection —
+the checkpoint is weight-only NVFP4, `IsTrueW4A4()==false`). No new quant code; it
+reuses the same dispatcher as Laguna routed-experts + dense Qwen3-32B NVFP4.
+
+**Per-shape routing (real geometry, spec §8.1):** all quantized projections are
+uniformly Marlin W4A16 — `qkv_proj [21504,5376]`, `out_proj [5376,7168]`, `fc1
+[28672,5376]`(+SiluAndMul, pre-merged so no fused-pair), `fc2 [5376,14336]`, block
+`adaln [96768,2688]`, `condition_proj [5376,5120]`, `final_adaln [10752,2688]`,
+refiner qkv/out/fc1/fc2. Islands + norms/biases stay bf16/f32. All N,K %16==0 and
+%128==0 — no Marlin shape blocker. Peak device memory ~1/4 of bf16 (~16 GB packed
+vs ~66 GB bf16).
+
+**CPU gate (verified, this box has no nvcc):** `test_minimax_h3` 62/62 cases /
+30039 assertions, 0 failed (Release, gcc-13). The synthetic-NVFP4 case now streams
+BOTH arms on the SAME file, asserts the fp4 loader kept projections PACKED (fp4 slot
+set / bf16 slot Empty; inverse for the bf16 loader), and asserts the W4A16
+dispatcher executed all 11 quantized GEMMs (the `Nvfp4W4A16Stats` this-path-ran
+counter == 11). fp4-vs-bf16 bounded <= 2e-3. On CPU the dispatcher has no Marlin op,
+so it falls back to the bf16 arm's own dequant+matmul: this is a WIRING gate here,
+proving the loader sets the fp4 slots and the forward routes them, NOT the kernel
+numerics. The Marlin W4A16 kernel itself is CUDA-gated independently by
+`test_ops_nvfp4_matmul` (odd shapes, 2e-3 f32-out / 4e-3 bf16-out vs bf16 ref) and
+`test_linear_method`.
+
+**PENDING (GB10, handoff recipe):** git-archive the row to dgx.casa, configure with
+`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, build ONLY
+`test_minimax_h3` (targeted — not a whole-archive), run
+`test_minimax_h3 --test-case="*NVFP4 checkpoint loads*"` — on CUDA the same case
+exercises Marlin (`stats.marlin_gemms==11`) and reports the real fp4-vs-bf16 delta;
+add a steady per-step timer around `MiniMaxH3DitForwardDevice`. Blocked on a safe
+disk/build window (dgx 33 G free vs 15 G floor; 570 G `.cache` is shared, not ours).
+Real-checkpoint t2va e2e is DISK-BLOCKED (~41 GB NVFP4 working set to download vs
+~18 GB usable, encoder required for a real render). `benchmark_binding=false`.
+
+**Comparability (mission #3):** vLLM-Omni CANNOT serve a quantized H3 on one GPU —
+BF16-only in practice (source-audited at `a4ea67a2`/v0.26.0: no quantized H3
+checkpoint exists; the fp32-island guard `minimax_h3_transformer.py:898-904` aborts
+a naive quant; text encoder hard-bf16 `encoder.py:930`; GGUF not wired into H3).
+Single-GPU IS supported but as BF16 + `--enable-cpu-offload`. -> HW/loader-forced-
+indirect (DeepSeek-GGUF precedent). The "DiT = 88% of latency" figure is NOT in the
+vllm-omni source; the documented anchor is the recipe's 4×B300 BF16 evidence (FL2VA
+209f 1248×768 = 86.964 s; two-video Ref2VA = 784.394 s, `recipe:298-311`).
+
 
 ## Laguna-S-2.1-NVFP4 decode — router top-k KERNEL-EFFICIENCY (`VT_LAGUNA_TOPK_SHFL`), BYTE-EXACT, SigmoidTopK 1.67×, −0.57% decode-step GPU (2026-08-03, `CLAIM-LAGUNA-TOPK-SHFL`)
 Measured results for vllm.cpp, against the reference engine each workload

diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md
diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md
@@ -913,3 +913,4 @@ Columns:
 | 2026-08-04 (`HARDEN-DETECTOR-LANES` PR #28 CI repair; `CLAIM-HARDEN-SANITIZER-DISK`; Ordino task `t-e19dc73f`; CPU-only, lifecycle unchanged; closing commit: this checkpoint) | Repairs the hosted ASan+UBSan build's filesystem exhaustion without weakening detector coverage. Sanitizer tests share one internal fully instrumented engine image instead of force-linking another static copy into every executable, and `-g1` retains file/line traces without full type/local-variable DWARF. CI enables the existing `VT_POOL_BYPASS=1` exact-allocation/real-free detector mode. The newly reachable leak survey removes a real minja `MacroNode` ownership cycle by weakly capturing the context that owns the callable. It also closes the two Nix-only suite portability gaps: resolve `true` and the active Python executable instead of assuming `/usr/bin`, and remove inherited `PYTHONHASHSEED` only from the unconfigured control suite. Default build linkage and packaged C ABI exports remain unchanged. | No vLLM behavioral counterpart: this is local build/test infrastructure plus a vendored minja lifetime repair. Anchors: `CMakeLists.txt`, `tests/CMakeLists.txt`, `.github/workflows/ci.yml`, `third_party/minja/minja.hpp`, `tests/tools/test_gdn_packed_component.py`, and `tests/tools/test_online_gate_client.py`. Remote root-cause evidence: GitHub run `30819266647`, job `91704728276`, 99 MiB free then `ld: No space left on device`. | **PASS.** GCC 15.2.0 ASan+UBSan full suite **331/331** under leak detection and pool bypass; TSan full suite **331/331** under pool bypass; affected plain GCC 15 `-Werror` suites **3/3**. ASan+UBSan tree **93 GiB -> 5.6 GiB** (about 94% smaller); TSan tree **1.9 GiB**. `benchmark_binding=false`, performance **NOT APPLICABLE**. Hosted PR confirmation is the next external gate; `continue-on-error` stays until that confirmation. |
 | 2026-08-06 (`row/KERNEL-FA2-GQA-SWAP`; `CLAIM-KERNEL-FA2-GQA-SWAP`; kernel `KERNEL-ATTN-FA2`; gated default-OFF, lifecycle unchanged) | Ports vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization into the d128 varlen decode launcher (`LaunchDecodeVarlenFA2Bf16`, gate `VT_FA2_DECODE_GQA_SWAP`): the Qwen3-dense decode grid becomes `(batch, kv_heads)` not `(batch, hq)` — the ngroups query heads pack into seqlen_q, KV read once/group, presented WITHOUT a materialized transpose via kv-major-group-minor strides (a 1:1 mirror of the already-shipped d256 `LaunchDecodeFA2Bf16` swap). OFF path byte-identical to the prior plain-varlen reduction; ON is non-byte-exact only when num_splits>1 (split reduction order → near-tie, toward vLLM's own numerics). | Mirrors `flash-attention @ 2c839c33` `mha_fwd_kvcache` seqlenq_ngroups_swapped + `set_params_splitkv` and vLLM v0.25.0 `flash_attn.py flash_attn_varlen_func` decode (#47 measured vLLM's swapped grid `(1,6,16)` = batch×kv_heads vs ours `(1,3,64)` = batch×query_heads). The vendored `flash_fwd_kernel.h` `get_lse_tile`/combine already honor the flag in both the num_splits==1 direct-write and >1 combine paths (the d256 arm is the proof). | GB10 sm_121a CUDA 13.0: op RED-first test 280/280 (both GQA ratios × batch{1,2,4,8} × short+long ctx; `swap_launches==1` proves the grid engaged; swap-vs-plain near-tie; MHA-inert) — RED proven (wrong swapped stride → 26,528 violations); full binary 28/28·454,679 no regression; compute-sanitizer 0-err/0-leak; #44 MXFP4 e2e smoke swap-ON 3/3 deterministic TOKEN-EXACT + coherent, byte-identical to swap-OFF. `benchmark_binding=false` (c1-c8 x3 re-bench + default flip = recorded next step; #47 projects flash ~28%@c2 / ~55%@c8 of the gap). |
 | 2026-08-06 (`row/KERNEL-MARLIN-DENSE-PORT`; `CLAIM-KERNEL-MARLIN-DENSE-PORT`; kernel `KERNEL-GEMM-MARLIN-W4A16`; gated default-OFF, lifecycle unchanged) | Vendors vLLM's OWN dense marlin W4A16 GEMM as a new `vt::MarlinDenseGemm` op (`VT_MARLIN_DENSE`, default OFF) and routes the E=1 dense NVFP4/MXFP4 projections (`dense_nvfp4_gemm.h` `MatmulNvfp4MarlinD`/`MatmulMxfp4W4A16D`/`GateUpFusedMarlinD`) through it. The dense kernel is direct-A + tile-per-CTA with vLLM's OWN dense fp32-C_tmp reduce, so at M<=8 it runs the sms-wide (48-CTA) grid WITHOUT the one-bf16-ULP shift the `VT_MARLIN_E1_PAR1` MoE-route par-regroup costs (#54: that ULP flips a strict 32B-NVFP4A16 token). Reuses the EXISTING marlin resident + workspace (same `marlin_permute` repack for dense and MoE — confirmed, no shim); rank-2 operand views, no moe_align gather. | 1:1 lift of vLLM @ `555967922` `csrc/libtorch_stable/quantization/marlin/`: `marlin.cu:326-541` (`marlin::marlin_mm` + config helpers) → `marlin_mm_dense.cu`; the torch::stable `marlin_gemm` wrapper (`:545-894`) → torch-free `cuda_marlin_dense.cu` launcher (mirrors `cuda_moe_marlin.cu`, dense c_tmp sizing `:713-716`); `kernel.h`/`marlin_template.h:1-2081` verbatim (the DENSE kernel — DISTINCT from the moe one, but SAME 12-param `Marlin<>` template so the generated `kernel_selector.h`+`sm80_*.cu` instantiation set is shared, namespace `marlin` from the local kernel.h). Shared `marlin.cuh`/`marlin_dtypes.cuh`/`dequant.h`/`marlin_mma.h` diff-verified byte-identical. Forced-Marlin a16 selection `kernels/linear/__init__.py:879-881`. | CPU `-fsyntax-only` CLEAN (`ops.cpp` + the `VT_MARLIN_NVFP4` routing header). GPU compile: all 3 new dense `.cu` compile CLEAN on dgx GB10 sm_121a under exact production flags (`-Werror=all-warnings`, `-static-global-template-stub=false`, `--generate-code=…sm_121a`). RED-first unit battery WRITTEN (`test_ops_moe_grouped.cpp`: NVFP4+MXFP4, M=1..8 × 3 shapes, dense-vs-CPU-ref AND dense-vs-grouped-route, row-shifted stride RED-injection). `benchmark_binding=false`; GPU EXEC gates (unit run + strict token battery dense-ON vs oracle incl. 32B-NVFP4A16:344 + launch-counter + nsys 48-CTA + binding c1..c8 x3) are the scoped dgx follow-up; default stays OFF until the strict battery proves oracle byte-match and the binding beats the MoE route (state `KERNEL-MARLIN-DENSE-PORT`). |
+| 2026-08-06 (`row/H3-FP4-SPEED`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; lifecycle unchanged) | **MiniMax-H3 W-FP4a — fp4-RESIDENT NVFP4 routing for the device DiT forward (NO new quant code).** Until now both NVFP4 loaders dequantized the packed FP4 projections to bf16 and the device forward ran `vt::MatmulBT`, so the sm_121a FP4 tensor-core route never ran for H3. Adds `Nvfp4Weight` carriers to `MiniMaxH3DitBlockWeights`/`MiniMaxH3DitWeights`, a fp4-resident streamer `StreamMiniMaxH3Nvfp4ToDeviceFp4` (keeps the compressed-tensors triple host-resident; the shared dispatcher uploads+repacks lazily then frees the fp4 originals, so peak device memory is ~1/4 of the bf16 arm), and a `LinearDev` dispatch that routes a non-Empty fp4 projection through `dense_nvfp4::MatmulNvfp4W4A16D`. | The routing is vLLM's OWN forced-Marlin-for-a16 selection: the checkpoint is weight-only NVFP4 (no `input_activations`, `IsTrueW4A4()==false`), so `kernels/linear/__init__.py:879-881` forces the Marlin W4A16 kernel, mirrored by `include/vllm/model_executor/models/dense_nvfp4_gemm.h:12-22,505-549` (`MatmulNvfp4W4A16D` -> single-expert `vt::MoeGroupedGemmNvfp4Marlin`, the SAME kernel Laguna routed-experts + dense Qwen3-32B NVFP4 use). Not cutlass-fp4/W4A4 (needs fp4 activations, private to `qwen3_5.cpp`). fc1 is already merged `[gate;up]` -> one W4A16 GEMM + `vt::SiluAndMul`. | **CPU-GATED (wiring), verified**: `test_minimax_h3` 62/62 cases / 30039 assertions, 0 failed. The synthetic-NVFP4 case streams the fp4 twin, asserts the loader kept the projections PACKED (fp4 slot set / bf16 slot Empty, and the inverse for the bf16 loader), runs fp4 + bf16 device forwards on the SAME file, asserts the W4A16 dispatcher executed ALL 11 quantized GEMMs (the `Nvfp4W4A16Stats` this-path-ran counter), and bounds fp4-vs-bf16 <= 2e-3. On CPU the dispatcher has no Marlin op so it falls to the bf16 arm's own dequant+matmul (hence a WIRING gate here); the Marlin kernel numerics are CUDA-gated independently by `test_ops_nvfp4_matmul` / `test_linear_method` (2e-3 f32-out / 8e-3 bf16-out vs a bf16 reference). `benchmark_binding=false`. PENDING: GB10 CUDA build + the fp4-vs-bf16 numeric delta and steady per-step timing (disk/build window); real-checkpoint t2va e2e DISK-BLOCKED (~41 GB working set). Comparability: vLLM-Omni serves NO quantized H3 (BF16-only in practice; source-audited `a4ea67a2`, spec §8.3) -> HW/loader-forced-indirect. |