mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW (c8 +1.1pp vs #49), mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | Residual: grouped→dense-direct-A; host slice NOT engine-loop |
+| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW, mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | c8 same-tool diff → goal row |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| MXFP4 parity goal | binding <1.0x on c2-c8 (best c8 0.953); c1 crosses to parity+. Residual = grouped-Marlin +7-9% (only real lever) + host slice (VT_LOOP_TRACE: NOT engine-loop) | NEXT: grouped→dense-direct-A E1 marlin |
+| MXFP4 parity goal | c8 same-tool diff (`QUANT-CT-MXFP4-C8-DIFF`) SETTLES #50: marlin grouped-5-GEMM (gate_up unfused, 180 vs 144) DOMINANT +0.9/+1.4ms; eager gap +0.88ms graph-closeable nets +1.3%; flash same-grid +11% | NEXT: grouped→dense-direct + gate_up fuse |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12615,3 +12615,50 @@ within the cross-tool attribution boundary (~0.7ms is ~24% of the c2 gap, near m
 Not a born-on-runner lever. (Caveat: the curl load under-saturated vs steady c8, so step_ms 25.5
 is not the c8 TPOT 37.6; the interval≈step finding is batch-independent and robust.)
 Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
+
+## QUANT-CT-MXFP4-C8-DIFF: fresh SAME-TOOL c8 decode-window per-step diff (POST-SLIVER `d3b412f5`==`4dd4e206`/#51) — marlin grouped-5-GEMM DOMINANT; eager launch-gap +880µs graph-closeable nets only −334µs; #50 grouped==dense SETTLED as isolated-shape ubench artifact; decode-graph opt-in = +1.3% byte-coherent (2026-08-09, GB10 sm_121a CUDA 13.0, RelWithDebInfo MARLIN=ON oracle-cutlass, vLLM oracle 0.25.0 `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`)
+
+SAME-TOOL both sides (the fix for the recorded cross-tool trap): nsys `--cuda-graph-trace=node`.
+OURS = online server, 24×128 window at c8. vLLM online-under-nsys does NOT capture the V1 EngineCore
+subprocess (empty rep at c8, MP-on AND MP-off) → captured vLLM via OFFLINE `LLM()` with
+`VLLM_ENABLE_V1_MULTIPROCESSING=0` (in-process) + cudaProfilerApi fence, IDENTICAL 8 corpus prompts
+(1024 tok) = clean M=8 batched decode. Kernel per-call times are frontend-independent. Steps segmented
+by lm_head anchor; steady window = modal-marlin-count; BOTH at flash gridZ=64 (matched M=8). Single
+node-trace per arm (per-step medians over 300+ steps are stable); binding c1–c8 x3 is #51's (0.953 c8).
+
+PER-STEP DECODE (median µs), OURS EAGER (as-shipped) vs vLLM (graphed):
+| class        | ours µs/step (calls) | vLLM µs/step (calls) | Δ           |
+|--------------|---------------------:|---------------------:|------------:|
+| marlin W4A16 | 17,183 (180=5×36)    | 16,286 (144=4×36)    | +897 (+5.5%)|
+| flash decode |  6,250 (36)          |  5,629 (36)          | +621 (+11%) |
+| glue         |    927 (291)         |    671 (299)         | +256        |
+| lm_head      |  5,413 (1)           |  5,395 (1)           | +18         |
+| flash_comb   |    130 (36)          |    140 (36)          | −10         |
+| sample+other |     36               |     23               | +13         |
+| GPU BUSY     | 30,027               | 28,190               | +1,837      |
+| GAP (idle)   | 1,184 (3.8%)         | 304 (1.1%)           | +880        |
+| STEP SPAN    | 31,211               | 28,491               | +2,720      |
+Measured median TPOT ours 37.56 / vLLM 34.58 ms (+2.98); GPU SPAN accounts 91%; ~0.26ms = async frontend.
+
+GAP LOCALIZATION (ours): uniform ~2.0–2.4µs idle between EVERY consecutive kernel + ~14 H2D + ~2 memset
+per step = host-launch-bound EAGER. vLLM ~0.4–0.76µs/gap = piecewise-CUDA-graph replay. Grids: ours
+marlin 144×1×1 (3 waves M=8), vLLM 48×1×1 (1 wave); flash IDENTICAL 1×3×64 both (GQA-swap parity).
+
+DECODE-GRAPH A/B (ours, `VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=1`, default-OFF opt-in, byte-identical-to-eager
+per source; SACRED gate unrun): the generic Qwen3-MXFP4 decode is EAGER by default (root-caused in
+`qwen3.cpp:DenseDecodeGraphForward`; no quant restriction). Graph ON: GAP 1,184→305µs (−880, =vLLM's,
+confirms the mechanism), BUSY 30,027→30,575µs (+547 bandwidth contention when packed), SPAN 31,211→30,877
+(−334), median TPOT 37.56→37.09ms (−1.3%), coherent. → c8 0.953→~0.965, still <1.0x. Self-limiting.
+
+FAIR graphed-vs-graphed (ours-graph vs vLLM): SPAN +2,386µs = marlin +1,377 (58%) + flash +658 (28%) +
+glue +290 (12%) + gap ~0. Dominant = MARLIN.
+
+VERDICT (a): dominant term = MARLIN — ours runs 5 grouped-MoE-E1 GEMM/layer (gate+up UNFUSED, 144-CTA)
+vs vLLM 4 dense (gate_up FUSED, 48-CTA) = +5.5%(eager)/+8.5%(graphed)/step, mostly the +25% GEMM count.
+SETTLES #50: its "grouped==dense parity" was vLLM-op-vs-vLLM-op at one ISOLATED shape, blind to OUR
+kernel's +36 GEMM/step + 3× CTAs; the residual is real per-step. Fix = grouped→dense-direct + gate_up
+fuse (delicate ~2000-line port, next dispatch). Secondary = decode-graph opt-in (+1.3%, SACRED owed).
+Tertiary = flash same-kernel/IDENTICAL-grid +11% (KV/splitkv), glue Inductor-fusion.
+Evidence dgx:~/mxfp4-nsys/{kern_sum_c8_dflt,vllm_offline_kern_c8,kern_sum_c8_graph}.txt +
+gpu_trace_c8_{dflt,graph}/vllm_offline_trace_c8_cuda_gpu_trace.csv + analyze_decode.py/gap_and_shape.py.
+Box left clean (both locks free, GPU idle, worker down, disk 20G, tmux gone).
diff --git a/.agents/state.md b/.agents/state.md
@@ -37180,3 +37180,81 @@ also runs) or within the cross-tool attribution boundary. NEXT lever (only real
 grouped→dense-direct-A marlin decode (#50 NO-GO'd the full port; a targeted E1 direct-A path is the
 open candidate). Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
 
+## QUANT-CT-MXFP4-C8-DIFF: fresh SAME-TOOL c8 decode-window diff on the POST-SLIVER binary — kernels are NOT identical; marlin grouped-5-GEMM is DOMINANT (+897us eager/+1377us graphed), eager launch-gap +880us is graph-closeable but nets only −334us (bandwidth contention); #50 "grouped==dense parity" SETTLED as an isolated-shape ubench artifact; decode-graph opt-in is a measured +1.3% byte-coherent secondary lever
+<!-- state: 2026-08-09T14:00 -->
+
+The deciding measurement the campaign owed: a FRESH same-tool (nsys `--cuda-graph-trace=node`
+BOTH sides) c8 decode-window per-step diff on the post-sliver binary (box build `d3b412f5` ==
+main `4dd4e206`/#51; block=8 sliver + GQA-swap default-ON both present, `ninja: no work to do`).
+Prior campaign per-kernel flags (#46/#47) were CROSS-TOOL (our nsys vs vLLM torch-profiler) — the
+old marlin 123-vs-113/flash 218-vs-151 were partly cross-tool bias. This is the first SAME-tool c8
+node-trace of BOTH. Branch `row/QUANT-CT-MXFP4-C8-DIFF`. Box left clean (locks free, GPU idle,
+worker down, disk 20G, tmux gone). NO code shipped (dominant fix not byte-small; decode-graph is an
+existing default-OFF opt-in whose per-model SACRED gate is unrun).
+
+METHOD (reliable same-tool capture — the crux): OURS = online server under nsys, 24×128 window at
+c8 (`nsys_ours_cN_dflt.sh`, reliable). vLLM under nsys is the known trap — the online api_server's
+V1 EngineCore subprocess is NOT captured (empty rep at c8, both MP-on AND MP-off). FIX: vLLM
+OFFLINE `LLM()` with `VLLM_ENABLE_V1_MULTIPROCESSING=0` (in-process → nsys captures) + cudaProfilerApi
+fence + `--cuda-graph-trace=node`, fed the IDENTICAL 8 corpus prompts (all 1024 tok) → clean M=8
+batched decode. Kernel per-call times are frontend-independent, so vLLM-offline decode == vLLM-online
+decode kernels. Steps segmented by lm_head anchor; steady window = modal-marlin-count steps; BOTH
+engines' steady decode is at flash gridZ=64 (matched M=8). Coherence: vLLM offline greedy correct;
+ours graph-on greedy "Paris…Rome" correct.
+
+PER-STEP DECODE (median over ~300–320 steady M=8 steps, µs), OURS EAGER (as-shipped) vs vLLM (graphed):
+| class          | ours µs/step (calls) | vLLM µs/step (calls) | Δ ours−vLLM |
+|----------------|---------------------:|---------------------:|------------:|
+| marlin W4A16   | 17,183 (180)         | 16,286 (144)         | +897 (+5.5%)|
+| flash decode   |  6,250 (36)          |  5,629 (36)          | +621 (+11%) |
+| glue           |    927 (291)         |    671 (299)         | +256        |
+| lm_head        |  5,413 (1)           |  5,395 (1)           | +18         |
+| flash_comb     |    130 (36)          |    140 (36)          | −10         |
+| sample+other   |     36               |     23               | +13         |
+| **GPU BUSY**   | **30,027**           | **28,190**           | **+1,837**  |
+| **GAP (idle)** | **1,184 (3.8%)**     | **304 (1.1%)**       | **+880**    |
+| **STEP SPAN**  | **31,211**           | **28,491**           | **+2,720**  |
+Measured median TPOT ours 37.56 vs vLLM 34.58 ms = +2.98ms; GPU SPAN accounts +2.72ms (91%); the
+~0.26ms residual = shared async frontend (consistent with #51 "host slice NOT engine-loop").
+
+GAP LOCALIZATION (ours, per-step): a UNIFORM ~2.0–2.4µs GPU-idle between EVERY consecutive kernel
+(marlin→glue 2.15, glue→glue 1.88, glue→marlin 2.15, flash→comb 2.37, …) + ~14 [CUDA memcpy H2D] and
+~2 [memset] per step = host-launch-latency-bound EAGER execution. vLLM's gaps are ~0.4–0.76µs (glue→glue
+0.76, glue→marlin 0.53) = piecewise-CUDA-graph REPLAY (node-to-node latency). Same GB10, so ours' 4× gap
+is ours-specific, not a HW floor.
+
+ROOT-CAUSED IN SOURCE: the generic Qwen3-8B-MXFP4 (`Qwen3ForCausalLM` dense) decode runs EAGER by
+default — `Qwen3DenseDecodeGraph` (`qwen3.cpp:DenseDecodeGraphForward`, shared decode CUDA-graph) is
+OPT-IN via `VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH` (default OFF, pending a per-model SACRED gate) and has NO
+quant restriction (only needs env-ON + pure_decode + CUDA). The binding c8=0.953 (#49/#51) was measured
+EAGER. TESTED the opt-in (env=1, byte-identical-to-eager per source; greedy "Paris…Rome" coherent):
+GAP 1,184→305µs (−880, = vLLM's, CONFIRMS the eager-launch mechanism) BUT BUSY 30,027→30,575µs (+547,
+marlin 17,183→17,663 — bandwidth contention when kernels pack tight on the GB10 unified bus), so SPAN
+31,211→30,877 (−334) and median TPOT 37.56→37.09ms (−1.3%). So the decode-graph lever is REAL but
+self-limiting: it closes the +880µs gap yet nets only ~−334µs; c8 tput 0.953→~0.965, still <1.0x.
+
+FAIR graphed-vs-graphed residual (ours-graph vs vLLM, the apples-to-apples since vLLM is graphed):
+SPAN +2,386µs = marlin +1,377 (58%) + flash +658 (28%) + glue +290 (12%) + gap ~0. So under a fair
+comparison the DOMINANT term is MARLIN, and the gap term dissolves.
+
+VERDICT (a) — the dominant divergent term ≥1% is MARLIN, named with mechanism + the #50 settlement:
+ours routes the 5 dense projections/layer through grouped-MoE-E1 marlin (`MoeGroupedGemmNvfp4Marlin`,
+grid 144×1×1 = 3 waves at M=8, gate+up UNFUSED = **180 GEMM/step**) vs vLLM's dense `marlin_gemm`
+(grid 48×1×1 = 1 wave, gate_up FUSED = **144/step**) → +5.5% (eager) / +8.5% (graphed) per step, driven
+PRIMARILY by the 25% higher GEMM count (gate_up not merged; per-call is at near-parity since W4A16 is
+memory-bound). **SETTLES the #50 contradiction:** the ubench "grouped==dense at parity once block=8"
+compared vLLM's OWN moe_wna16 vs dense op at a SINGLE isolated shape — it could not see the production
+structural divergence (OUR kernel, +36 GEMM/step, 3× CTAs). The residual is real at the per-step level.
+FIX = grouped→dense-direct marlin + gate_up fusion (the #50-scoped ~2000-line delicate port); NOT
+small/byte-safe → scoped for next dispatch. SECONDARY lever = decode-graph opt-in (+1.3%, byte-coherent,
+needs the per-model SACRED token gate). TERTIARY = flash decode: SAME kernel + IDENTICAL grid (1,3,64)
+BOTH engines (GQA-swap parity confirmed) yet +11% per-call (173.6 vs 156.3µs) — mechanism unresolved
+(KV-cache read layout / splitkv scratch), small; and glue +256µs = ours' separate norm/silu/rope/qkv
+launches vs vLLM's Inductor-fused triton (portable-fusion class, #46/#47).
+
+SHAs: worktree/main `4dd4e206` (#51) == box build `d3b412f5` (row pre-squash, content-identical).
+Evidence dgx:~/mxfp4-nsys/ — reps ours_c8_dflt/ours_c8_graph/vllm_offline_c8.nsys-rep; tables
+kern_sum_c8_dflt.txt, vllm_offline_kern_c8.txt, kern_sum_c8_graph.txt; trace CSVs
+gpu_trace_c8_{dflt,graph}_cuda_gpu_trace.csv, vllm_offline_trace_c8_cuda_gpu_trace.csv; analyzers
+analyze_decode.py + gap_and_shape.py; vLLM offline capture nsys_vllm_c8_v2.sh/run_vllm_offline_nsys.sh.
+
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -307,7 +307,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | grouped→dense-direct-A marlin; host slice NOT engine-loop (VT_LOOP_TRACE interval≈step) |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | c8 same-tool diff (`QUANT-CT-MXFP4-C8-DIFF`): marlin grouped-5-GEMM DOMINANT (gate_up unfused); eager gap +0.88ms graph-closeable (decode-graph opt-in +1.3%); flash same-grid +11%; #50 = isolated-shape artifact |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |