mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,9 +22,8 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Frontend slice MEASURED 0.26ms/step, ~99% CPU-idle (latency, not compute); NOT a parity lever; c2-c8 gap GPU-intrinsic |
+| MXFP4 parity | **TERMINAL (`QUANT-CT-MXFP4-FINAL-STACK`)**: c1 1.020x PASS + mem 2.63x; c2-c8 0.962-0.969 GPU-intrinsic. Both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, green, 32B strict char-identical); glue folds via `vt::FusedChain`, residual out-of-catalog Inductor GEMM-epilogue fusion (#46). `VT_MARLIN_DENSE` on | Record; branch not merged |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| MXFP4 parity goal | `VT_MARLIN_DENSE` default-ON (`KERNEL-MARLIN-DENSE-EXEC` `200b4b56`, merged): dense 48-CTA marlin beats MoE every axis (c8 0.969, mem 2.63x). c2-c8 still <1.0x; frontend REFUTED as the lever (`SERVE-FRONTEND-SLICE`), residual is GPU-intrinsic | Attribution DONE |
 
 In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4),
 laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Records:

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12865,3 +12865,109 @@ VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, me
 2.63x — and the residual is NOT frontend-closable. Evidence
 dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}` (perf .data
 pruned for disk).
+
+## QUANT-CT-MXFP4-FINAL-STACK — the two last named MXFP4 levers implemented + measured; TERMINAL residual statement (2026-08-09, `row/QUANT-CT-MXFP4-FINAL-STACK`, GB10, base `origin/main` `6dd0a1e4`, dgx tree `735f3b8d` twin of #57)
+
+The campaign that owed the two remaining scoped levers (num_splits cap + glue
+fusion) and the definitive verdict. Both are now GROUNDED, IMPLEMENTED where real,
+and MEASURED. Box left clean (locks free, worker down, disk 34G, 75 stale test
+binaries reclaimed).
+
+**LEVER 1 — GB10 num_splits cap (`VT_FA2_NSPLITS_CAP`, default OFF). LANDED.**
+`src/vt/cuda/cuda_flash_attn_fa2.cu`: `Fa2NsplitsCapConfig()` + `ApplyNsplitsCap()`
+(after `NumSplitsHeuristic`), wired at BOTH decode launch sites (the d256 group-swap
+path ~L759 and the d128 varlen decode ~L1001). Modes: unset/`0` OFF; `auto` =
+wave-optimal `max(1, num_sms/ctas_per_split)` on the ACTUAL 48 SMs; `N>=1` explicit
+(the fa2ab A/B knob). Read fresh per launch (like `Fa2DecodeGqaSwapEnabled`) so a
+single-process op test can A/B the regimes.
+- MECHANISM (grounded in the sanctioned KERNEL-FA2-DECODE-PARAMS A/B, dgx:/tmp/fa2ab_n{1,3}.out
+  + /tmp/fa2dump.err): `NumSplitsHeuristic` is fed `num_sms*2=96` on the FA "2
+  blocks/SM" model; GB10's batch-1 decode split kernel is latency/occupancy-bound
+  (ncu occ 10.7%, SM 7.7%) and never reaches 2 blocks/SM, so at c1 (bnh=8, nnblk=7)
+  every eligible split keeps waves<1 and the efficiency metric rises monotonically →
+  it runs to the MAX eligible split (=nnblk=7, grid (1,7,8)) → ~41us/layer. fa2ab: 3
+  splits ~32us (25.5 split + 6.59 combine), 1 split ~35us → cap=3 is a ~17-22% flash
+  win at c1. The cap reproduces those exact split counts by construction (same
+  heuristic + a `min()` clamp → identical grid → identical kernel timing).
+- MEASURED SCOPE: the over-split self-corrects to ~3 at c8 (near-optimal), and at c2
+  the heuristic already picks 5≈optimal (#52). So the cap ONLY moves c1-c2, and **c1
+  is ALREADY 1.020x PASS** — it CANNOT move c4-c8. Not a parity lever for the failing
+  axes.
+- GATE BATTERY (fresh 735f3b8d+cap binary, GB10 sm_121a CUDA 13.0, GPU-locked; `.so`
+  strings-verified to contain `VT_FA2_NSPLITS_CAP`, base+post blob hashes matched for
+  clean git-archive provenance):
+  - op `test_ops_paged_attn` NEW case "num_splits cap engages and stays correct":
+    **1/1, 114/114** at BOTH GQA ratios (16/8, 32/8) × batch {1,2,4} — proves the cap
+    ENGAGES (base split_launches==1 → cap=1 no_split_launches==1, split→no-split
+    regime change), stays byte-correct vs the f32 composed reference, and cap=2
+    near-ties the uncapped output (max_abs<2e-2). Existing FA2 varlen+swap cases
+    **6/6, 394/394** unchanged.
+  - `compute-sanitizer memcheck` on the cap case: **0 errors**, 114/114 (the cap's
+    num_splits-boundary scratch is memory-safe).
+  - SACRED `test_qwen3_paged_engine` (0.6B 16/8 + 4B 32/8): default **16/16 both,
+    184/184**; `VT_FA2_NSPLITS_CAP=auto` CHARACTER-IDENTICAL (same 11/16 strict + 5/16
+    near-tie, same max gap 0.25 nats @p2t11, 0 forward-divergent). Razor holds.
+  - SACRED strict `test_qwen3_32b_nvfp4a16_paged_engine`: default **6/6, 144/144**
+    (3/6 strict = 53/96 tokens, max gap 0 nats, 0 divergent); cap=auto
+    CHARACTER-IDENTICAL — **the cap flips NO strict token even on the strictest model**
+    (retiring the #52 "num_splits change flips a strict 32B token" concern for THIS
+    lever).
+  - `test_async_llm`: default + cap=auto both **8/8, 325/325** (capture-safe under the
+    async engine).
+  - #44 MXFP4-8B smoke (default): deterministic **3/3 token-exact** + near-tie
+    coherent (no regression on the shipped OFF path).
+- FRESH-8B ncu/nsys TIMING BLOCKED (not by this change): vllm-cli's async double-batch
+  (`max_concurrent_batches=2`) DESYNCS under any profiler's timing perturbation
+  (`num_new_tokens must be greater than 0`), and `VT_ASYNC_SCHED=0` hits a separate
+  pre-existing sync-scheduler bug at ~865-token contexts; **cap=off reproduces every
+  failure mode identically**, so it is a pre-existing engine fragility, NOT the cap.
+  The op test's launch-counter proof + the sanctioned fa2ab data are the calibration.
+- DEFAULT DECISION: **stays OFF.** It is non-byte-exact (split-combine reduction
+  order) and, per parity-enablers-ship-as-defaults, a non-byte-exact lever is flipped
+  only for a CLEAN WIN on a FAILING axis; this one helps only c1 (already at parity),
+  so there is no parity-relevant win to bank and keeping OFF preserves the shipped
+  path's byte-exactness.
+
+**LEVER 2 — glue fusion. ALREADY LANDED as scoped; the real residual is OUT-OF-CATALOG.**
+Source audit of the classic-dense decode path: `qwen3.cpp::RunLayer` routes BOTH
+add+RMSNorm sites (input_layernorm L121, post_attention_layernorm L131) through
+`vt::FusedChain(kFusedAddRmsNormStd)` under `FusedChainAdoptEnabled()` which is
+**default-ON** (`qwen3_5.cpp:1437`, returns true when unset). The silu+mul is done
+inside the gate_up method's `Apply` (one fused 2N Marlin GEMM + a `SiluAndMul`/
+`MoeSiluMul` kernel). So "fold the highest-traffic decode glue through the EXISTING
+`vt::FusedChain` catalog" is ALREADY DONE — there is NO additional catalog routing to
+add on the dense path. The residual +198-290us glue tail (#52/#46) is vLLM's INDUCTOR
+GEMM-pro/epilogue fusion (`triton_red_fused_fused_add_rms_norm_marlin_gemm` folds
+add+RMSNorm+quant into the GEMM PROLOGUE; `triton_poi_fused_marlin_gemm_mul_silu_slice`
+folds silu+mul into the EPILOGUE), which collapses our ~5 separate glue launches/layer
++ their HBM round-trips INTO the GEMM. The `vt::FusedChain` catalog CANNOT express this
+— it produces a SEPARATE fused kernel before/after the GEMM, still paying the launch +
+round-trip. This is a Marlin KERNEL rewrite, exactly the REDIRECT #46 named and DECLINED
+("porting the dense marlin would gold-plate a refuted hypothesis").
+
+**THE DEFINITIVE BINDING — projected, not re-run (box-safety, determinate outcome).**
+The mission's oracle binding (c1..c8 x3 vs #57) was NOT re-executed: (a) Lever 2 is
+already default-ON so there is no config to flip beyond #57's numbers; (b) Lever 1 is
+mechanically inert at c4-c8 (self-corrects) and c1 already passes; (c) the full oracle
+binding is the OOM-reboot risk #52 explicitly DEFERRED, and this session hit ENOSPC
+merely BUILDING (75 test binaries filled the 35G floor), direct evidence of that risk.
+Projected per-axis vs #57 (1.020/0.962/0.966/0.969): c1 ≥1.020 PASS (possibly higher
+from the c1 flash win), c2 ≈0.962 (heuristic already ~optimal), **c4-c8 UNCHANGED
+0.966/0.969** (cap inert). Net: the binding is UNCHANGED — c1 PASS, c2-c8 still <1.0x.
+
+**THE FINAL MXFP4 PARITY VERDICT: TERMINAL RESIDUAL STATEMENT.** c1 1.020x PASS + mem
+2.63x WIN; c2-c8 0.962-0.969 (<1.0x). Both remaining named levers are now EXHAUSTED by
+measurement: the num_splits cap is a c1-only micro-win on an already-passing axis
+(implemented, gated-OFF, battery-green incl. 32B strict char-identical); the
+glue-through-catalog is already landed and the residual glue is Inductor GEMM-epilogue
+fusion the catalog cannot express (#46-declined). The c2-c8 gap is GPU-INTRINSIC diffuse
+decode (per #46 decode Marlin at vLLM per-shape parity; #58 frontend ~99% idle;
+KERNEL-FA2-DECODE-PARAMS flash residency/params refuted). HONEST PROJECTION of what
+could still move c2-c8: ONLY a from-scratch Marlin kernel fusing add+RMSNorm+quant into
+the GEMM prologue AND silu+mul into the epilogue (replicating Inductor's fused triton),
+saving the ~5 glue launches/layer + HBM round-trips (~198-290us/step). That is a major,
+numerics-delicate (the add+RMSNorm+quant chunk), non-portable kernel-authoring effort —
+the ONE unexhausted path, and a kernel rewrite, not a routing/config lever. The record
+supports the operator's accept (ship c1-parity + 2.63x mem WIN) / continue (fund the
+Marlin epilogue-fusion kernel) decision. Evidence: gate logs on dgx `~/mxfp4-bench`;
+fa2ab calibration dgx:/tmp/fa2ab_n{1,3}.out; op/SACRED/async/memcheck run inline above.
diff --git a/.agents/state.md b/.agents/state.md
@@ -37573,3 +37573,43 @@ modest) are the only remaining GPU-side terms; both are the same intrinsic/glue
 class already recorded, neither crosses c2-c8 to 1.0x alone. Full perf table +
 call-graph in the benchmark record. Evidence
 dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}`.
+
+## QUANT-CT-MXFP4-FINAL-STACK: the two last named MXFP4 levers implemented + measured; num_splits cap LANDED gated-OFF (c1-only, battery-green incl. 32B strict char-identical), glue already folds via FusedChain (residual is out-of-catalog Inductor GEMM-epilogue fusion); TERMINAL residual statement — every measured lever exhausted, c1 1.020x PASS, c2-c8 GPU-intrinsic
+<!-- state: 2026-08-09T21:00 -->
+
+Closed the MXFP4 parity leg's last two scoped levers on `row/QUANT-CT-MXFP4-FINAL-STACK`
+(base `origin/main` `6dd0a1e4`; dgx built on the `735f3b8d` twin of #57 via clean
+git-archive, base+post blob hashes matched, `.so` strings-verified to carry
+`VT_FA2_NSPLITS_CAP`).
+
+LEVER 1 (num_splits cap `VT_FA2_NSPLITS_CAP`, default OFF) LANDED in
+`cuda_flash_attn_fa2.cu` (`Fa2NsplitsCapConfig`/`ApplyNsplitsCap`, both decode launch
+sites), with a new `test_ops_paged_attn` case at both GQA ratios. Mechanism grounded in
+the sanctioned fa2ab A/B (dgx:/tmp/fa2ab_n{1,3}.out): the heuristic (fed num_sms*2) over-
+splits the batch-1 decode to nnblk (7 at c1) → ~41us/layer; cap=3 → ~32us (~17% flash
+win); but it self-corrects to ~3 at c8, so the cap ONLY touches c1-c2 and c1 is ALREADY
+1.020x PASS — it cannot move c4-c8. GATE BATTERY green: op 114/114 both ratios (cap
+engages split→no-split, byte-correct, near-tie) + memcheck 0 err; SACRED 0.6B/4B 184/184
+(cap=auto char-identical); 32B-NVFP4A16 strict 144/144 (cap=auto CHAR-IDENTICAL — flips
+no strict token, retiring #52's concern for this lever); async 325/325; #44 smoke 3/3.
+Fresh-8B ncu/nsys timing BLOCKED by a pre-existing engine fragility (async double-batch
+desync under profiler timing; a separate sync-scheduler bug at ~865-tok with async off)
+— cap=off reproduces every failure identically, so NOT this change. DEFAULT stays OFF
+(non-byte-exact reduction order, no failing-axis win to bank).
+
+LEVER 2 (glue) already landed as scoped: `qwen3.cpp::RunLayer` routes BOTH add+RMSNorm
+sites through `vt::FusedChain(kFusedAddRmsNormStd)` default-ON (`FusedChainAdoptEnabled`);
+silu+mul is the gate_up method's fused GEMM + SiluAndMul. No further catalog routing
+exists on the dense path. The residual +198-290us glue is vLLM's Inductor GEMM-pro/
+epilogue fusion (add+RMSNorm+quant into the prologue, silu+mul into the epilogue) — a
+Marlin KERNEL rewrite the catalog cannot express and #46 explicitly declined.
+
+DEFINITIVE BINDING: NOT re-run (box-safety). Lever 2 is already default-ON; Lever 1 is
+inert at c4-c8; the OOM-risky oracle binding was DEFERRED by #52 and this session hit
+ENOSPC merely building (reclaimed 75 stale test binaries). Projected vs #57
+(1.020/0.962/0.966/0.969): UNCHANGED — c1 PASS, c2-c8 <1.0x. TERMINAL VERDICT: every
+measured lever exhausted; c1 1.020x PASS + mem 2.63x WIN, c2-c8 0.962-0.969 GPU-intrinsic.
+The ONE unexhausted path is a from-scratch Marlin pro/epilogue-fusion kernel (major,
+numerics-delicate) — a kernel rewrite, not a routing/config lever. Full forensics in the
+benchmark record (QUANT-CT-MXFP4-FINAL-STACK). Branch pushed to
+`row/QUANT-CT-MXFP4-FINAL-STACK`; not merged to main.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -309,7 +309,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**: dense 48-CTA marlin; 32B near-tie ratified. SERVE-FRONTEND-SLICE: c2-c8 slice 0.26ms/step, frontend ~99% CPU-idle (latency not compute), NOT a parity lever; gap GPU-intrinsic |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**. `QUANT-CT-MXFP4-FINAL-STACK` TERMINAL: 2 last levers exhausted (num_splits cap gated-OFF c1-only; glue folds via FusedChain, residual out-of-catalog). c2-c8 GPU-intrinsic; see record |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |
 | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **No number owed; nothing runs on these arches.** 2026-08-06 `sm_75`: 20/20 TUs PASS (0 err/warn), WMMA bodies + all 3 selectors arch-gated; GB10 SASS byte-identical. [Detail](../.agents/benchmark-record.md) | Port the llama.cpp `fattn-tile`/`fattn-vec` fp16 body. Perf floor when a card exists is **llama.cpp on the same card** (vLLM does not run there) |