mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| MXFP4 parity | **TERMINAL (`QUANT-CT-MXFP4-FINAL-STACK`)**: c1 1.020x PASS, mem 2.63x; c2-c8 0.962-0.969 GPU-intrinsic, both last levers exhausted (residual #46). `VT_MARLIN_DENSE` on | Record; branch not merged |
+| MXFP4 parity | **c1 1.020 PASS, c2-c8 0.962-0.969, mem 2.63x** (`VT_MARLIN_DENSE` banks +951us marlin). `FUSED-GLUE` W0: glue-into-marlin REFUTED at source; residual FLASH-dominant | flash same-tool audit OWED |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner |
 | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Op-dispatch profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist |
 

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -19,6 +19,129 @@ from relative link targets repointed for this file's location.
 
 # Benchmarks
 
+## QUANT-CT-MXFP4-FUSED-GLUE W0 — the funded glue-fusion-into-Marlin kernel target is SOURCE-REFUTED; vLLM does NOT fuse glue into the extern Marlin GEMM for W4A16; the c2-c8 residual is FLASH-dominant, not glue (2026-08-06, `row/QUANT-CT-MXFP4-FUSED-GLUE`, source-side on dev box; GPU dumps box-contended by `row/H3-FP4-GPU-E2E`, base `origin/main` `672fc760`)
+
+W0 of the funded MXFP4 kernel campaign. The chartered vehicle is `Yi30/Qwen3-8B-MXFP4`
+(dense `Qwen3ForCausalLM`, compressed-tensors **W4A16** Marlin keep-quant); oracle arm
+`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` (sm_121 cute-dsl mxf4 crashes). The
+prior FINAL-STACK (#59) terminal statement funded a from-scratch Marlin kernel fusing
+`add+RMSNorm+quant` into the GEMM PROLOGUE and `silu+mul` into the EPILOGUE, framed as
+mirroring vLLM's Inductor "GEMM pro/epilogue fusion" (`triton_red_fused_fused_add_rms_norm_marlin_gemm`
+/ `triton_poi_fused_marlin_gemm_mul_silu_slice`, #46/#52). W0's precondition is to DUMP
+vLLM's actual kernels and never trust the label. The GPU dumps (TORCH_LOGS=output_code,
+same-tool decode-window) are box-contended and OWED, but the vLLM COMPILATION SOURCE is
+decisive on its own and REFUTES the premise. Reading is at the parity pin `555967922`
+(`~/_git/vllm`, 0.26.0.dev0).
+
+**FINDING 1 (source-conclusive): the two named vLLM fusion passes DO NOT FIRE for a
+W4A16 MXFP4 model — they are ACTIVATION-QUANT fusions and W4A16 keeps bf16 activations.**
+- `PostGradPassManager.configure` adds `RMSNormQuantFusionPass` under `fuse_norm_quant`
+  and `ActivationQuantFusionPass` under `fuse_act_quant` (`vllm/compilation/passes/pass_manager.py:163-171`).
+  Those flags resolve via `enable_norm_fusion`/`enable_act_fusion` (`vllm/config/vllm.py:108-129`;
+  wired into `OPTIMIZATION_LEVEL_0x` at `vllm.py:232-233,255-256,278-279`).
+- But the PASSES only match norm/act followed by an **fp8/nvfp4 quant** op:
+  `RMSNormQuantFusionPass` registers ONLY `{FusedAdd,}RMSNorm{Static,Dynamic,Group}QuantPattern(..., FP8_DTYPE)`
+  (`rms_quant_fusion.py:629-669` — every pattern is `FP8_DTYPE`); `ActivationQuantFusionPass`
+  registers ONLY `SiluMulFp8StaticQuantPattern` / `SiluMulNvfp4QuantPattern` /
+  `SiluMulBlockQuantPattern(kFp8Dynamic*)` (`act_quant_fusion.py:296-320`).
+- W4A16 has NO activation quant: `apply_gptq_marlin_linear` calls `marlin_quant_input` ONLY
+  for `input_dtype==int8`/`float8_e4m3fn` (`marlin_utils.py:704-715`); for MXFP4 W4A16
+  `input_dtype=None`, so the **bf16** RMSNorm output is passed STRAIGHT to `ops.marlin_gemm`
+  (`marlin_utils.py:717`). MXFP4 is not nvfp4 (`is_nvfp4_quantized()` is False), so the nvfp4
+  clause of `enable_act_fusion` also does not apply. ⇒ There is no `rms_norm→quant` or
+  `silu_mul→quant` sub-graph to match; BOTH passes are no-ops on this model.
+
+**FINDING 2 (source-conclusive): `marlin_gemm` is a non-decomposable EXTERN custom op —
+Inductor structurally CANNOT fuse elementwise into it.** `ops.marlin_gemm` is registered
+with `register_fake("_C::marlin_gemm")` (`vllm/_custom_ops.py:1200-1247`); Inductor treats
+a fake-registered custom op as a FallbackKernel and emits an extern CALL, never Triton for
+its body. The observed `triton_*_marlin_gemm_*` kernels are therefore INDUCTOR-NATIVE
+elementwise fusions (a `triton_red` reduction doing residual-add + decomposed-RMSNorm; a
+`triton_poi` pointwise doing silu+mul+slice) that Inductor NAMES after the region containing
+the adjacent extern `marlin_gemm` call — NOT a prologue/epilogue baked into the Marlin CUDA
+kernel. vLLM's decode Marlin runs as a SEPARATE `void marlin::Marlin` CUDA kernel — the
+same-tool trace #57 shows it as 144 distinct launches, which it could not be if fused into a
+Triton kernel. **So "vLLM fuses add+RMSNorm+quant into the Marlin prologue and silu+mul into
+the epilogue" is a MISREAD of Inductor's region-naming. vLLM does no such fusion for W4A16.**
+
+**FINDING 3 (structural corroboration): vLLM runs MORE glue launches than we do, not
+fewer.** The same-tool #57 c8 decode-window (`nsys --cuda-graph-trace=node`, both engines)
+already recorded glue: **ours 866 us / 255 calls vs vLLM 671 us / 299 calls**. If vLLM had
+collapsed glue into the GEMM pro/epilogue it would show FEWER glue launches; it shows more
+(8.3 vs 7.1 per layer), individually cheaper. The +195 us glue delta is per-kernel
+efficiency (Inductor Triton elementwise vs our hand-written glue), NOT a fusion-count gap —
+directly contradicting the #46 "collapses our ~5 glue launches into 2" narrative (which came
+from vLLM's ONLINE torch-profiler, a cross-tool read AGENTS.md forbids for invocation parity).
+
+**FINDING 4 (the reframe that matters): the dominant marlin-tiling term #52/#57 measured was
+ALREADY CLOSED on `main` by the dense-direct default, AFTER the terminal binding.** #52/#57
+same-tool decode-window (MoE-grouped route, base `027af9b0`) decomposed the c8 gap as
+**marlin +1,177 us (52%) / flash +784 us (35%) / glue +195 us (9%)**, and named the marlin
+fix as grouped→dense-direct + gate_up-fuse (the opt-in "par1" arm closed marlin +1,177→+226).
+That lever then LANDED as the byte-exact DEFAULT: commit `efa6e40d` "perf(marlin):
+VT_MARLIN_DENSE default ON — dense route beats MoE on every axis (#57)" (in my base). On
+current `main` `MarlinDenseEnabled()` is default-ON (`include/vllm/model_executor/models/dense_nvfp4_gemm.h:106-127`),
+dense MXFP4 projections run vLLM's own dense marlin `vt::MarlinDenseGemm`
+(`src/vt/cuda/cuda_marlin_dense.cu:92`) at the 48-CTA grid (~86 us/call vs the MoE route's
+~118 us/call), gate_up merged into one 2N GEMM by default (`VT_MOE_FUSED_W13`,
+`dense_nvfp4_gemm.h:98-104,519-544`), 32B-NVFP4A16 SACRED max gap 0.000 nats. That is
+exactly why the terminal binding rose from #51's `0.925/0.939/0.953` (MoE) to #57's
+`0.962/0.966/0.969` (dense). **So the +951 us marlin lever is BANKED, and in the
+dense-direct regime the residual is FLASH-DOMINANT: at c8 0.969 the gap is ~+1,100 us of
+which flash ~+784 us (~71%), glue ~+195 us (~18%), residual-marlin ~+226 us or less.**
+
+**THE W0 ours-vs-vLLM per-span table (c8 decode-window, same-tool nsys, current dense-direct
+regime ≈ #57 par1 arm; medians over 300+ steps):**
+
+| span between GEMMs | our launches/layer (vt::) | what covers it in vLLM's step | ours us/step (calls) | vLLM us/step (calls) | Δ |
+|---|---|---|---|---|---|
+| decode Marlin GEMMs (qkv, o, gate_up[2N], down = 4/layer) | `vt::MarlinDenseGemm` ×4, 48-CTA dense-direct | `void marlin::Marlin` ×4 (SEPARATE CUDA kernel) | 16,512 (144) | 16,286 (144) | +226 |
+| flash decode attention | `vt::PagedAttention` (flash_fwd_splitkv 1×3×64) | flash_fwd_splitkv (IDENTICAL grid 1×3×64) | 6,436 (36) | 5,629 (36) | **+807** |
+| glue: add+RMSNorm ×2, qk-norm ×2, RoPE, QkvSplit, ReshapeAndCache, SiluAndMul | ~7 separate `vt::` kernels/layer | ~8 SEPARATE Inductor Triton elementwise kernels/layer (NOT fused into marlin) | 869 (255) | 671 (299) | +198 |
+| lm_head | cuBLAS GEMV | cuBLAS GEMV | 5,398 | 5,395 | ~0 |
+
+Our per-layer decode order (source-mapped): `FusedChain(kFusedAddRmsNormStd)` input-norm
+(`qwen3.cpp:121`) → `MarlinDenseGemm` qkv → `QkvSplit` → `FusedChain(kAttnQkNormRope)` q/k
+norm+RoPE (composite = 3 launches on CUDA) → `ReshapeAndCache` → `PagedAttention` →
+`MarlinDenseGemm` o → `FusedChain(kFusedAddRmsNormStd)` post-attn-norm (`qwen3.cpp:131`) →
+`GateUpFusedMarlinD` (`MarlinDenseGemm` over 2N, `dense_nvfp4_gemm.h:542`) → `SiluAndMul`
+(SEPARATE launch, `dense_nvfp4_gemm.h:544`) → `MarlinDenseGemm` down. The add+RMSNorm sites
+already route through `vt::FusedChain` default-ON; the `FusedChain` catalog (`include/vt/recipes.h`,
+`include/vt/fused_recipe.h`) has NO GEMM opcode and by design produces only standalone
+fused elementwise kernels before/after a GEMM — it cannot express a GEMM prologue/epilogue.
+
+**VERDICT — the funded glue-fusion-into-Marlin kernel is the WRONG lever; DO NOT BUILD it.**
+1. It would MIRROR a fusion vLLM does not perform (Findings 1-3): a from-scratch Marlin with
+   fused prologue/epilogue is genuinely BEYOND vLLM (the "surpass rung"), not a mirror.
+2. It targets the +195-198 us glue span = ~9% of the MoE-route c8 gap and ~18% of the current
+   dense-direct residual. Even a PERFECT glue fusion (glue→0) leaves c8 ≈ 0.977 — still <1.0.
+3. The dominant marlin-tiling lever it was conflated with is ALREADY the byte-exact default
+   (`efa6e40d`), banked in the 0.962-0.969 terminal.
+4. The dominant REMAINING term is FLASH (+784-807 us on an IDENTICAL grid/kernel) — a
+   same-kernel/different-time STRUCTURAL-lens context question the FA2 num_splits refutation
+   (#59 lever 1) did NOT address. That, not glue, is the unexhausted high-value lever.
+
+**COSTED surpass-rung proposal (scoped, NOT built, per the charter):** a dedicated
+`MarlinDenseGemm` variant with (a) a fused `add+RMSNorm` reduction PROLOGUE that produces the
+bf16 marlin input in-kernel, and (b) a `silu+mul` EPILOGUE folded into the gate_up 2N GEMM's
+store. Numerics: the prologue changes the RMSNorm reduction boundary (register vs HBM
+round-trip) → not byte-exact → near-tie razor + distributional gate + 32B strict + regen
+under the ratified-tie rule; the epilogue is byte-safe (pointwise on the GEMM output).
+Expected recovery ≈ the glue span it removes, ~150-198 us/step (the ~5 glue launches + their
+HBM round-trips), i.e. c8 ~0.969→~0.977 — a real but sub-parity, numerics-delicate,
+non-portable kernel. NOT RECOMMENDED as the parity path.
+
+**OWED GPU work (box-contended by H3; short + batched when the flock frees):** (1) capture
+`TORCH_LOGS=output_code` on the oracle to VISUALLY confirm the `triton_*_marlin_gemm_*`
+kernels are elementwise-only calling the extern marlin (Findings 1-2 predict this exactly);
+(2) a FRESH same-tool nsys decode-window on CURRENT `main` (dense-direct default) c1..c8 to
+re-decompose the residual and confirm the flash-dominant split; (3) the real next lever — a
+STRUCTURAL-lens same-tool audit of why identical-grid `flash_fwd_splitkv` runs +22 us/call
+slower than vLLM's (residency/context, per AGENTS.md's STRUCTURAL lens), the +784 us term.
+No code shipped on this row: building the glue Marlin would gold-plate a source-refuted
+hypothesis (#46's own standard). Box left as found (no GPU touched; both flock locks
+untouched; H3 campaign uninterrupted).
+
 ## QUANT-CT-MXFP4 W4 throughput bench — RAN on GB10, BELOW-FLOOR (2026-08-06, `row/QUANT-CT-MXFP4-BENCH` `33e93608`)
 
 The binding ours-vs-oracle online-serving grid on the SAME checkpoint

diff --git a/.agents/state.md b/.agents/state.md
@@ -39276,6 +39276,44 @@ render. Comparability: vLLM-Omni CANNOT serve a quantized H3 on one GPU
 (BF16-only in practice; source-audited `a4ea67a2`) -> HW/loader-forced-indirect,
 DeepSeek-GGUF precedent. Draft PR is the claim.
 
+## 2026-08-06T15:40 - QUANT-CT-MXFP4-FUSED-GLUE W0: the funded glue-fusion-into-Marlin kernel is SOURCE-REFUTED; residual is flash-dominant, not glue
+<!-- state: 2026-08-06T15:40 -->
+
+Row `row/QUANT-CT-MXFP4-FUSED-GLUE` (helper). The FINAL-STACK (#59) terminal funded a
+from-scratch Marlin fusing add+RMSNorm+quant into the prologue and silu+mul into the
+epilogue, framed as mirroring vLLM's Inductor "GEMM pro/epilogue fusion". W0's precondition
+is to dump vLLM's actual kernels and never trust the label. The GPU dumps are box-contended
+by the active `row/H3-FP4-GPU-E2E` (both flock locks) and are OWED, but the vLLM compilation
+SOURCE at the pin `555967922` is decisive and REFUTES the premise. Full forensic entry +
+per-span table in `.agents/benchmark-record.md` ("QUANT-CT-MXFP4-FUSED-GLUE W0").
+
+Findings: (1) `RMSNormQuantFusionPass`/`ActivationQuantFusionPass` are ACTIVATION-QUANT
+fusions — every registered pattern needs an fp8/nvfp4 quant op (`rms_quant_fusion.py:629-669`,
+`act_quant_fusion.py:296-320`). W4A16 MXFP4 keeps bf16 activations (`apply_gptq_marlin_linear`
+quantizes only int8/fp8, `marlin_utils.py:704-717`), so NEITHER pass fires. (2) `ops.marlin_gemm`
+is a `register_fake` extern op (`_custom_ops.py:1200-1247`) Inductor cannot fuse elementwise
+into; the `triton_*_marlin_gemm_*` kernels are Inductor-NATIVE elementwise fusions named after
+the adjacent extern call — vLLM's Marlin runs as a SEPARATE `void marlin::Marlin` CUDA kernel.
+(3) same-tool #57 already shows vLLM runs MORE glue launches than us (299 vs 255), not fewer —
+so there is no glue-into-GEMM collapse to mirror; the +195 us glue delta is per-kernel
+efficiency. (4) the dominant marlin-tiling term #52/#57 measured (+1,177 us) was ALREADY
+closed on `main` by the dense-direct default `efa6e40d` (`VT_MARLIN_DENSE` ON,
+`dense_nvfp4_gemm.h:106-127`, 48-CTA ~86 us/call), which is why the terminal rose #51
+0.925-0.953 → #57 0.962-0.969. In that regime the c8 residual is FLASH-dominant: ~+784 us
+flash (71%) / ~+195 us glue (18%) / ~+226 us residual-marlin.
+
+VERDICT: do NOT build the funded glue Marlin — it mirrors a fusion vLLM does not perform (it
+is the beyond-vLLM surpass rung), targets ~18% of the residual, and even glue→0 leaves c8
+~0.977 (<1.0). The real unexhausted lever is the flash +784 us term on an IDENTICAL grid — a
+STRUCTURAL-lens context/residency audit the FA2 num_splits refutation did not address.
+
+OWED (box-contended, short+batched when the flock frees): (a) TORCH_LOGS=output_code on the
+oracle to visually confirm the Triton kernels are elementwise-only over the extern marlin;
+(b) a fresh same-tool nsys decode-window on current `main` (dense-direct default) c1..c8 to
+re-decompose and confirm the flash-dominant split; (c) the flash same-kernel/different-time
+audit. Costed surpass-rung proposal (scoped, NOT built) recorded. No code shipped: building
+the glue Marlin would gold-plate a source-refuted hypothesis. Box untouched; H3 uninterrupted.
+
 ## QUANT-GGUF-CIQ-GEMM: the OWED fresh CPU op-dispatch profile is DONE and it re-ranks the levers away from G5; decode is 47% threadpool synchronisation and prefill is ~39% paged attention
 <!-- state: 2026-08-09T18:00 -->
 

diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -297,7 +297,7 @@ built on it rather than keeping the flattering one.
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
 | MiniMax-H3 FP4 speed (W-FP4a) | Pending. fp4-resident Marlin-W4A16 routing CPU-landed (62/62); GB10 delta + per-step unmeasured (disk window); real e2e disk-blocked; vLLM-Omni has no quantized H3 (BF16-only) | Build CUDA `test_minimax_h3` on dgx, run the NVFP4 case (Marlin via `marlin_gemms`), capture delta + s/step. Detail: benchmark-record + spec §8 |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON**. `QUANT-CT-MXFP4-FINAL-STACK` TERMINAL: 2 last levers exhausted (num_splits cap gated-OFF c1-only; glue folds via FusedChain, residual out-of-catalog). c2-c8 GPU-intrinsic; see record |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** banks +951us marlin. `FUSED-GLUE` W0: glue-into-marlin REFUTED (vLLM doesn't fuse into extern marlin); c8 residual FLASH-dominant, not glue. Flash same-tool audit OWED; see record |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |
 | Ampere consumer (`sm_86`, RTX 3090 class) | **No number owed; no such board here.** 2026-08-06 build-verify: 7/7 FA2 TUs 0-warn, real `sm_86` SASS. [Detail](../.agents/benchmark-record.md) | External RTX 3090 report. Floor is llama.cpp on that card (GGUF, not our Blackwell-only NVFP4 grid) |