mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -23,7 +23,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS. **Marlin-config lever REFUTED** (`row/QUANT-CT-MXFP4-M28-LEVER`): same-tool per-shape microbench = our MoE-E1 Marlin AT PARITY with vLLM dense `marlin_gemm` (moe/dense 0.99-1.08, all M); nsys c8 BOTH engines confirm Marlin at-parity, vLLM's edge = Inductor FUSION of norm/quant/act glue | REDIRECT: portable glue-fusion + SAME-tool c1-vs-c2 nsys diff (attn batching vs unfused glue vs async overlap); the gap is a STEP at c2, not the M-independent Marlin |
+| QUANT-CT-MXFP4-BENCH | ~0.91x@c2-c8; marlin+glue levers **both REFUTED**; step = flash GQA group-swap | LEVER + detail: state `QUANT-CT-MXFP4-GLUE` (`cuda_flash_attn_fa2.cu:1075`) |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/state.md b/.agents/state.md
@@ -36842,3 +36842,78 @@ in M (and our Marlin is M-independent M1->M2 per the microbench, so it cannot be
 No code shipped (the named fix is refuted; shipping a dense-marlin port would gold-plate a
 dead hypothesis). Evidence text: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`.
 Box: both locks free, GPU idle, disk unchanged 23G, local-ai-worker left down, tmux clean.
+
+## QUANT-CT-MXFP4-GLUE: c1-vs-c2 localization — GLUE-fusion lever REFUTED; the c2-c8 step is the FLASH decode GQA group-swap (ours iterates 32 q-heads, vLLM 8 kv-heads)
+<!-- state: 2026-08-08T15:00 -->
+
+Ran the #46-prescribed discriminator (ours-only nsys c1-vs-c2 decode-window, same
+binary `33e93608` == main MXFP4 code) + the mandated SAME-tool re-check of the two
+cross-tool flags. **The prescribed GLUE-fusion lever is REFUTED by measurement; the
+real c2-c8 step is the FLASH decode attention launch config.** Branch
+`row/QUANT-CT-MXFP4-GLUE` off `origin/main` `024323e0`. Box left clean (both locks
+free, GPU idle, worker down, disk 22G, evidence 202M on /home).
+
+METHOD: ours nsys `-t cuda --cuda-graph-trace=node` at c1 (3274 decode steps) + c2
+(1573 steps, 2 tok/step), 24x128 window, GB10 48 SMs. vLLM SAME-tool where reachable
+(c2 nsys marlin; c2 torch `/start_profile` full breakdown; c8 torch from #46). Per-CALL
+Med, aggregation-trap-separated by count.
+
+PER-KERNEL c1->c2 DELTA (ours, per-STEP = Med x calls/step):
+| kernel            | c1 Med | c2 Med | c1 /step | c2 /step | grows? |
+|-------------------|-------:|-------:|---------:|---------:|--------|
+| decode marlin x180| 114.75us| 120.74us| 20,655us | 21,732us | +1,077us (+5.2% Med) |
+| decode flash x36  | 35.49us | 63.68us | 1,278us  | 2,292us  | **+1,014us (+79% Med)** |
+| lm_head cutlass_80| 5,395us | 5,390us | 5,445us  | 5,330us  | flat (memory-bound GEMV) |
+| glue (norm/silu/rope/qkv/reshape) | — | — | ~685us | ~776us | +90us (negligible) |
+So the c1->c2 STEP growth (+2.06ms GPU-sum ~ +2.57ms measured TPOT) is FLASH + marlin;
+GLUE barely moves. Glue is only ~2.5% of the step and the add+RMSNorm is ALREADY one
+kernel (RmsNorm-with-residual), so Tier-1 fusion (`VT_FUSED_TIER=1`) is a near-nop for
+graphed decode. **The binding's glue-fusion attribution is wrong.**
+
+CROSS-TOOL FLAG RE-CHECKS (the two #46 flags):
+- **MARLIN 123 vs 113 (~9%): ~PARITY, benign.** c2 nsys vLLM decode marlin rows 108-123us
+  ~ ours 120.7us; c8 torch vLLM 112.9 vs ours 123. Same-tool ubench already showed the two
+  dispatchers at per-shape parity. Real but small (+7-9%); the grouped-moe-E1 indirect gather
+  vs dense-marlin direct-A. The #46 dense-marlin-config lever STAYS refuted.
+- **FLASH 218 vs 151 (~44%): CONFIRMED REAL, and it is the primary lever.** Clean vLLM c2
+  torch decode flash = **41.7us** vs ours nsys c2 **63.7us (+53%)**; c8 vLLM torch 151 vs ours
+  218 (+44%). SAME vendored `flash_fwd_splitkv<128,64,128,4>` kernel BOTH sides -> a LAUNCH-
+  CONFIG divergence, not a kernel gap.
+- **GLUE fusion: REFUTED as the differential.** vLLM's fused-glue triton launches
+  (`triton_red_fused_fused_add_rms_norm_marlin_gemm` 5.3+4.3us, `..._mul_silu_slice` 6.9us,
+  reshape 4.2us) sum to ~910us/step >= OURS ~776us/step. vLLM's glue is NOT cheaper; the
+  #46 Inductor-fusion "structural edge" does not exist at the per-step level.
+
+ROOT CAUSE (grid dims, both sides, same workload): ours decode flash grid = **(1,3,64)**
+where gridZ=64 = batch(2)x**query_heads(32)**; vLLM = **(1,6,16)** where gridZ=16 =
+batch(2)x**kv_heads(8)**. vLLM applies the FA2 GQA **`seqlenq_ngroups_swapped`** decode
+optimization (h=kv_heads=8, the ngroups=4 query heads/group packed into seqlen_q, KV read
+once per group) -> 96 CTAs = 1 wave on 48 SMs. Ours iterates all 32 query heads separately
+-> c1 96 CTAs (1 wave, 35.5us, at PARITY) but c2 **192 CTAs = 2 waves (63.7us)** and c8 256
+CTAs (heuristic returns num_splits=1, ~5 waves, 218us). The over-wave at batch>=2 is EXACTLY
+the c1-parity / c2-c8-degradation shape. Our launcher hard-codes the swap OFF at
+`src/vt/cuda/cuda_flash_attn_fa2.cu:1075` (`p.seqlenq_ngroups_swapped = false; // the whole
+point: no group swap`) on a mistaken belief vLLM's varlen decode does not swap; the grid
+proves it does. The vendored kernel (`flash_fwd_kernel.h` get_lse_tile/combine) ALREADY
+supports the swap flag.
+
+DIFFERENTIAL ATTRIBUTION at c2 (ours-vLLM, per step): marlin +1,476us (7%, ambiguous cross-
+tool + #46-refuted), flash +792us (53%, CONFIRMED), lm_head +23us, glue -134us (ours less),
+combine -25us => GPU diff ~+2.13ms of the +2.82ms measured TPOT gap (~76%); ~0.7ms host/sched
+residual. At c8 flash is +2.4ms/step (~55% of the +4.34ms gap).
+
+LEVER (named, upstream-grounded, NOT yet shipped): port the FA2 GQA decode ngroups-swap into
+`LaunchDecodeVarlenFA2Bf16` (mirror flash-attn `mha_fwd_kvcache` seqlenq_ngroups_swapped:
+transpose q [nreq,32,d]->[nreq,ngroups=4,hkv=8,d], set h=hkv/seqlen_q=ngroups, recompute
+num_splits + LSE/oaccum scratch on the swapped dims, combine already supports the swapped LSE
+stride). NON-byte-exact (splitkv reduction order changes -> near-tie; it moves us TOWARD
+vLLM's numerics), touches the CUDA-graph-captured decode -> needs a RED-first op test +
+#44 smoke token-exact gate BEFORE a default flip; ship gated `VT_FA2_DECODE_GQA_SWAP` default
+OFF. Not shipped this session per byte-exact-first / SACRED-decode discipline (a rushed
+launcher rewrite of the graphed decode path is the wrong risk); scoped as the TDD next step
+with the exact site + the vLLM-grid proof. HONEST residual: flash alone closes ~28%(c2)/
+~55%(c8) of the gap; reaching >=1.0x also needs the marlin +7-9% (grouped->dense direct-A, a
+bigger delicate lift) and the ~0.7ms host/sched slice. No single lever reaches >=1.0x.
+Evidence: `dgx:~/mxfp4-nsys/{kern_sum_c1.txt,kern_sum_c2.txt,vllm_torch_c2.txt,vllm_kern_nsys_c2.txt,
+vllm_kern_sum.txt}` + reps `ours_c1/ours_c2/vllm_c2.nsys-rep`.
+Box: both locks free, GPU idle, disk 22G, local-ai-worker left down, tmux clean.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -307,7 +307,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record). **Grouped-Marlin config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`, nsys c8 both engines + same-tool per-shape microbench):** our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M (moe/dense 0.99-1.08); decode Marlin is at-parity, so the residual is diffuse decode where vLLM's edge is Inductor FUSION of the norm/quant/act glue (`triton_..._add_rms_norm_marlin_gemm`, `..._marlin_gemm_mul_silu`). Redirect: portable glue-fusion + same-tool c1-vs-c2 diff |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid (`33e93608`): 0.99x@c1 / ~0.91x@c2-c8, mem 2.6x LESS. Marlin-config + glue levers **both REFUTED**; c2-c8 step = flash decode GQA group-swap. Detail + lever: state `QUANT-CT-MXFP4-GLUE` |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |