mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -23,7 +23,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS | Lever: batched Marlin decode GEMM (M=2..8); nsys c8 both engines |
+| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS. **Marlin-config lever REFUTED** (`row/QUANT-CT-MXFP4-M28-LEVER`): same-tool per-shape microbench = our MoE-E1 Marlin AT PARITY with vLLM dense `marlin_gemm` (moe/dense 0.99-1.08, all M); nsys c8 BOTH engines confirm Marlin at-parity, vLLM's edge = Inductor FUSION of norm/quant/act glue | REDIRECT: portable glue-fusion + SAME-tool c1-vs-c2 nsys diff (attn batching vs unfused glue vs async overlap); the gap is a STEP at c2, not the M-independent Marlin |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -81,6 +81,55 @@ this row (all additive; 27/35 byte-unchanged; CPU contract tests 45/45): `online
 `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`; `mxfp4_smoke_gate.py`;
 `dgx-online-serving.sh` q3mxfp4 branches; `mxfp4-online-serving-grid.sh` orchestrator.
 
+## QUANT-CT-MXFP4 c2-c8 gap — grouped-Marlin config lever REFUTED (2026-08-08, `row/QUANT-CT-MXFP4-M28-LEVER`, GB10, build `33e93608`==main)
+
+The binding (#45) attributed the c2-c8 TPOT deficit to "our per-expert/grouped Marlin
+tiling vs vLLM's Marlin as M grows 2->8" and named the fix as porting vLLM's M-dependent
+`thread_m_blocks`/tile selection. Profiled BOTH engines and ran the per-shape arbiter; the
+attribution is **wrong** — our Marlin already matches vLLM's per-shape.
+
+**Same-tool per-shape microbench** (`~/mxfp4-nsys/mxfp4_marlin_ubench.py`; oracle compiled
+ops, cuda-event timed; dense `ops.marlin_gemm` = what vLLM runs for a dense MXFP4 linear vs
+`ops.moe_wna16_marlin_gemm` E=1 = our production kernel's 1:1 upstream twin; Qwen3-8B decode
+shapes qkv 6144x4096, o 4096x4096, gate/up 12288x4096, down 4096x12288, gs=32):
+
+| M | dense/layer us | moe/layer us | moe/dense | dense step ms (x36) | moe step ms | delta ms |
+|---|---|---|---|---|---|---|
+| 1 | 226.6 | 231.7 | 1.023 | 8.16 | 8.34 | +0.18 |
+| 2 | 231.5 | 228.3 | 0.986 | 8.33 | 8.22 | -0.12 |
+| 4 | 223.7 | 240.8 | 1.077 | 8.05 | 8.67 | +0.62 |
+| 8 | 237.6 | 242.3 | 1.020 | 8.55 | 8.72 | +0.17 |
+
+Per-shape at M=8: qkv 1.111, o 1.026, gate 1.002, up 0.998, down 1.030. All deltas are noise
+(<0.6 ms/step; M=2 favors MoE). **Our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M**,
+so routing to vLLM's dense marlin config cannot close the ~4.3 ms/step (11%) gap. (Source:
+vLLM dense route `apply_fp4_marlin_linear`->`ops.marlin_gemm`, `marlin.cu:438`
+`m_block_size_8=prob_m<=8`; ours `MoeGroupedGemmNvfp4Marlin` E=1, `marlin_mm_moe.cu:363`
+`thread_m_blocks=div_ceil(moe_block_size,16)`, `MarlinMoeAlignBlockSizeSelect`=16 at M=8.
+The dispatchers differ but the cost does not.)
+
+**nsys BOTH engines at c8** (aggregation-trap separated; TPOT reproduced: ours 39.53 vs vLLM
+33.81 ms). OURS (`nsys -t cuda --cuda-graph-trace=node`, kern_sum): decode Marlin dominates
+(`marlin_moe_wna16::Marlin` 58,854 inst Med **123us**; prefill Marlin separated 3,240 @ 2.03ms);
+attention flash_fwd_splitkv<128,64,128,4> Med **218us**; glue = SEPARATE launches
+(RmsNormRow*, MoeSiluMul, QkvSplit, RopeFromCache, ReshapeAndCache), one per op per layer.
+VLLM (online `/start_profile` torch profiler, chrome-trace aggregate): decode marlin
+`void marlin::Marlin` 13,680 @ **112.9us**; flash same traits @ **151us**; and the structural
+tell — the glue is **FUSED into the GEMM by Inductor**:
+`triton_red_fused_fused_add_rms_norm_marlin_gemm_{0,2}` (add+RMSNorm+quant prologue) and
+`triton_poi_fused_marlin_gemm_mul_silu_slice` (silu+mul epilogue). vLLM collapses our ~5
+separate glue launches/layer into the GEMM pro/epilogue.
+
+VERDICT: the decode Marlin GEMM is at vLLM per-shape parity (REFUTED lever). The residual is
+diffuse decode; the robust structural divergence is vLLM's Inductor **glue fusion** (portable-
+fusion class). REDIRECT (no ceiling): (1) portable fusion of add+RMSNorm+quant into the Marlin
+prologue and silu+mul into the epilogue; (2) a SAME-tool c1-vs-c2 OURS nsys diff to localize
+the STEP at c2 (our Marlin is M-independent M1->M2 per the microbench, so the step is attention
+batching, unfused glue traffic, or the async `max_concurrent_batches=2` overlap — not the GEMM);
+(3) same-tool re-check of the cross-tool per-call flags (marlin 123 vs 113us; attention 218 vs
+151us on identical flash traits). No code shipped — porting the dense marlin would gold-plate a
+refuted hypothesis. Evidence: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`.
+
 ## DeepSeek-V4-Flash UD-IQ2_M — IQ2_S + MXFP4 CPU keep-quant bring-up (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`) — no throughput owed (off-GPU correctness bring-up)
 
 Off-GPU task (GB10 down, no nvcc on the dev box), so NO throughput is measured or owed. Adds the two per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint's last 4 routed-expert slabs use — **IQ2_S** (ggml type 22; 2.5625 bpw codebook, Q8_K activation) and **MXFP4** (type 39; OCP micro-scaling fp4, 32-elem blocks, Q8_0 activation) — as first-class vt block dtypes (`kIQ2_S`/`kMXFP4`): block traits + dequant + a keep-quant `vec_dot`, ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook copied verbatim + direct sign bytes; `kvalues_mxfp4` + `e8m0_to_fp32_half`). The memory point: these load COMPRESSED (keep-quant) instead of the ~17 GiB bf16 expansion that OOM-reboots the 119 GiB pool.

diff --git a/.agents/state.md b/.agents/state.md
@@ -36782,3 +36782,63 @@ non-empty evidence root); fixed in `mxfp4-online-serving-grid.sh` this change. R
 --configure-log <LOG>`. Evidence: `dgx:~/work/vllm.cpp-online-gate/evidence/33e93608...`
 (ratios.json + report.md + 24 raws + memory/thermal/cache-drop). local-ai-worker left as
 found (down); dgx /home build tree `~/mxfp4-bench` retained for the nsys continuation.
+
+## QUANT-CT-MXFP4-M28-LEVER: grouped-Marlin config lever REFUTED — our Marlin already at vLLM per-shape parity; the c2-c8 gap is diffuse decode (vLLM's edge = Inductor FUSION of the norm/quant/act glue)
+<!-- state: 2026-08-08T09:00 -->
+
+Chased the binding's named lever (#45 / NOW `QUANT-CT-MXFP4-BENCH`): "the c2-c8 TPOT
+gap is the batched grouped Marlin W4A16 decode GEMM as M grows 2->8; port vLLM's
+M-dependent thread_m_blocks/tile config". Ran the mandatory profile-BOTH + per-shape
+arbiter on dgx GB10 (build `33e93608` == main `7f4bbf88` MXFP4 code; both flock locks,
+single-load, box left clean/idle). **The lever is REFUTED by same-tool per-shape
+measurement.** Branch `row/QUANT-CT-MXFP4-M28-LEVER` off `origin/main` `7f4bbf88`.
+
+WHAT vLLM ACTUALLY RUNS (source, both sides cited): the dense MXFP4 W4A16 linear routes
+`MarlinMxFp4LinearKernel.apply_weights` -> `apply_fp4_marlin_linear` -> `ops.marlin_gemm`
+(the DENSE single-matrix marlin, `csrc/libtorch_stable/quantization/marlin/marlin.cu`;
+`m_block_size_8 = prob_m<=8 && a16` at marlin.cu:438, direct A addressing). OURS routes
+every dense projection through `MoeGroupedGemmNvfp4Marlin` E=1 (`cuda_moe_marlin.cu` ->
+`marlin_mm_moe.cu`; `thread_m_blocks=div_ceil(moe_block_size,16)`, `moe_block_size` from
+`MarlinMoeAlignBlockSizeSelect` = 8 at M=1/2/4, 16 at M=8, so `m_block_size_8` is FALSE at
+M=8; indirect `sorted_token_ids` gather; hardcoded fp32 C_tmp reduce). So the two engines
+DO run different marlin dispatchers — but that difference does NOT cost.
+
+PER-SHAPE ARBITER (same tool: both are the oracle's compiled torch ops, cuda-event timed;
+`~/mxfp4-nsys/mxfp4_marlin_ubench.py`, dense `ops.marlin_gemm` vs `ops.moe_wna16_marlin_gemm`
+E=1 at the 5 Qwen3-8B decode shapes x M in {1,2,4,8}). moe/dense per-STEP (sum of 5 GEMMs
+x36 layers): M=1 **1.023**, M=2 **0.986**, M=4 **1.077**, M=8 **1.020** (deltas <0.6ms, in
+noise; M=2 even favors MoE). At M=8 per-shape moe/dense: qkv 1.111, o 1.026, gate 1.002,
+up 0.998, down 1.030. => **our production MoE-E1 Marlin is at parity with vLLM's dense
+`marlin_gemm` at every M**; porting vLLM's dense M-config would NOT close the ~4.3ms/step
+(11%) gap. This is exactly the scan-hypothesis that per-shape MEASUREMENT kills
+(parity-lever-protocol lane #2) — recorded BEFORE any port, saving a large dead-end lift.
+
+nsys BOTH engines at c8 (aggregation-trap-separated by per-call time + count; OURS =
+`nsys -t cuda --cuda-graph-trace=node` on our server, kern_sum; VLLM = online
+`/start_profile` torch profiler on `vllm serve ... VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`,
+chrome-trace kernel aggregate). TPOT reproduced the binding: ours 39.53 vs vLLM 33.81 ms.
+- OURS decode: dominant = decode Marlin (`marlin_moe_wna16::Marlin`, 58,854 inst, Med
+  **123us**); prefill Marlin separated (3,240 inst @ 2.03ms). Attention flash_fwd_splitkv
+  <128,64,128,4> Med **218us**. Glue launched SEPARATELY: RmsNormRow*, MoeSiluMul, QkvSplit,
+  RopeFromCache, ReshapeAndCache — one kernel each, per layer.
+- VLLM decode: decode marlin (`void marlin::Marlin`, 13,680 inst @ **112.9us**); flash
+  same traits @ **151us**; and the structural tell — the glue is **FUSED by Inductor into
+  the GEMM**: `triton_red_fused_fused_add_rms_norm_marlin_gemm_{0,2}` (add+RMSNorm+quant
+  prologue, 3,600/3,500 inst @ 10.8/15.3us) and `triton_poi_fused_marlin_gemm_mul_silu_slice`
+  (silu+mul epilogue, 3,600 @ 25us). vLLM collapses our ~5 separate glue launches/layer into
+  the GEMM's pro/epilogue.
+
+CONCLUSION + REDIRECT (no ceiling): the Marlin GEMM kernel is NOT the lever — it is already
+at vLLM per-shape speed; the binding's attribution ("our grouped tiling vs vLLM's Marlin")
+was an inference the per-shape trace overturns. The residual c2-c8 gap is DIFFUSE decode.
+The one robust STRUCTURAL divergence in the trace is vLLM's Inductor **fusion of the
+norm/quant/activation glue** around at-parity marlin GEMMs (portable-fusion class, cf.
+[[fusion-must-be-portable-reuse-patterns]] — "modest ~3.5%@c1, tails 3-14%"). Cross-tool
+per-call flags to confirm SAME-tool next: decode marlin 123 vs 113us (~9%), decode
+attention 218 vs 151us on the identical flash traits — and a c1-vs-c2 OURS nsys diff to
+localize what grows M1->M2 (attention batching vs unfused glue vs the async
+`max_concurrent_batches=2` overlap), since the binding curve is a STEP at c2, not monotone
+in M (and our Marlin is M-independent M1->M2 per the microbench, so it cannot be the step).
+No code shipped (the named fix is refuted; shipping a dense-marlin port would gold-plate a
+dead hypothesis). Evidence text: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`.
+Box: both locks free, GPU idle, disk unchanged 23G, local-ai-worker left down, tmux clean.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -307,7 +307,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record) |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record). **Grouped-Marlin config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`, nsys c8 both engines + same-tool per-shape microbench):** our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M (moe/dense 0.99-1.08); decode Marlin is at-parity, so the residual is diffuse decode where vLLM's edge is Inductor FUSION of the norm/quant/act glue (`triton_..._add_rms_norm_marlin_gemm`, `..._marlin_gemm_mul_silu`). Redirect: portable glue-fusion + same-tool c1-vs-c2 diff |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |