mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -24,7 +24,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 | MXFP4 parity (Qwen3-8B) | **`MARLIN-STRUCT`: decode-graph + gate_up FUSION default-ON (180→144 GEMM/step); #44 3/3, 0.6B/4B 184/184, 32B 142/142** | residual = marlin CTA + flash |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| MXFP4 parity goal | graph+fuse default-ON; c8 dominant residual = marlin CTA 144 vs 48. **`KERNEL-MARLIN-DENSE-PORT` gated-OFF** (`VT_MARLIN_DENSE`): vLLM's own dense marlin = byte-preserving E=1; 3 dense `.cu` compile-clean dgx; unit WRITTEN | NEXT (dgx): strict dense-ON vs oracle + nsys + binding (state) |
+| MXFP4 parity goal | **`VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** (`row/KERNEL-MARLIN-DENSE-EXEC` `200b4b56`): dense E=1 marlin, 48-CTA vs 128. Ratified near-tie (32B razor gap 0.000, TIGHTER than MoE; goldens regen). Binding beats #51 every axis (c8 0.969, GPU mem 2.63x). Goal still <1.0x c2-c8 (state) | Merge row; next = decode-flash |
 
 In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4),
 laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Records:

diff --git a/.agents/state.md b/.agents/state.md
@@ -37446,3 +37446,62 @@ not attempted here):
 Default stays OFF until (b) proves byte-match vs the oracle and (d) beats the MoE route. SHA of this
 landing: `row/KERNEL-MARLIN-DENSE-PORT` (code commit + this records commit).
 
+## KERNEL-MARLIN-DENSE-EXEC: the four scoped gates RAN on GB10 — VT_MARLIN_DENSE FLIPS DEFAULT-ON (dense reduce IS vLLM's numerics: teacher-forced near-tie gap 0.000 nats, TIGHTER than MoE; 48-CTA grid beats the MoE route on EVERY binding throughput axis)
+<!-- state: 2026-08-09T18:00 -->
+
+Executed the operator dgx follow-up for `KERNEL-MARLIN-DENSE-PORT` (#55) on
+`row/KERNEL-MARLIN-DENSE-EXEC` off `origin/main` `828f6420`. Clean git-archive tree at the
+port sha built RelWithDebInfo on GB10 sm_121a (full production stack: marlin-nvfp4 [121a] +
+CUTLASS sm120a NVFP4 + FA2 [121a] + Triton-AOT vendored). **DECISION: FLIP VT_MARLIN_DENSE
+DEFAULT ON** (`200b4b56`; `=0` opts back out to the MoE route). Box left clean (gpu.lock free,
+GPU idle, worker down, disk 36G, tmux gone).
+
+(a) UNIT EXEC. First-ever execution of the #55 dense unit battery (`test_ops_moe_grouped`,
+NVFP4+MXFP4, M=1..8 × model shapes). **First run RED on NVFP4** — but the diagnostic proved it is a
+TEST-AUTHORING artifact, NOT a kernel bug: `max|dense-moe| == 0` at EVERY shape/M (dense output is
+BYTE-IDENTICAL to the validated single-expert MoE route), while the failing elements are all
+catastrophic-cancellation outputs (true |ref| 0.04-5, summands ~O(1e3)) where the bf16 tensor-core
+result parts from the naive fp32-accumulate CPU reference by O(1) — IDENTICALLY for the MoE route.
+The MXFP4 arm (looser 4e-2 + no cancellation in its RNG) passed. FIX (`200b4b56`): the NVFP4
+vs-reference checks use the standard cancellation-robust L2-relative metric; per-element
+byte-fidelity stays the EXACT dense==MoE gate; the row-shift RED injection still fires. Re-run
+**2/2 cases, 263/263 GREEN**. MEMCHECK (`compute-sanitizer memcheck --leak-check full`): **0
+invalid-access/race errors**; the 42 leak allocations are the graph-safe c_tmp retire-pool + static
+harness caches at process exit (same benign class as #54).
+
+(b) STRICT BATTERY dense-ON vs the ORACLE goldens. `test_qwen3_32b_nvfp4a16_paged_engine` (THE
+decider): baseline dense-OFF 6/6·143 reproduced (marlin_gemms=1536, dense_gemms=0). Dense-ON shifts
+the greedy anchor at two EXACT bf16 ties — prompt[2] tok0 264->304, prompt[3] tok3 374->702 — so the
+:344 hard REQUIRE fired (as it did for the #54 par1 clamp). **Near-tie razor (ratified method):**
+teacher-forcing the vLLM 0.25.0 oracle with OUR dense prefix, **every dense token == vLLM's
+teacher-forced argmax, max gap 0.000000 nats, 0 out-of-topK** — TIGHTER than the MoE route (which
+had 62 mnats at [2,1]). vLLM's greedy CAPTURE landed on 264/374; its teacher-forced run lands on
+304/702 (vLLM's OWN bf16 non-determinism at the tie) and dense tracks the tf-argmax. So the "dense =
+vLLM's own numerics" premise is CONFIRMED, and the shift is a RATIFIED tie -> regenerated the 32B
+`our_ids.npy`+`neartie_gap_mnats.npy` (all-zero gaps) under dense-ON. Re-run dense-ON **6/6·144
+graphed AND eager (byte-identical, capture-safe)**, counters `dense_gemms=2048, marlin_gemms=0,
+fused_gate_up=0, fallback_gemms=0` (gate-conditional counter assertion added, `d250893e`). Canaries
+(dense inert on bf16): `test_qwen3_paged_engine` 0.6B/4B 2/2·184; `test_qwen3_dense_async_serving`
+2/2·82. #44 MXFP4-8B smoke dense-ON **3/3 token-exact + near-tie coherent**.
+
+(c) NSYS c8 decode (same tool, MXFP4-8B, dense-OFF vs dense-ON A/B): the marlin kernel NAME changes
+`marlin_moe_wna16::Marlin` -> `marlin::Marlin` (the vendored dense template RAN), the decode grid
+drops **128x1x1 -> 48x1x1** (the designed sms-wide grid), and per-call median **117.8us -> 85.8us**
+(Avg 111.2us), total decode-marlin GPU -5%. Confirms the 48-CTA mechanism, now byte-safe.
+
+(d) BINDING c1..c8 x3 (`mxfp4-online-serving-grid.sh`, oracle vLLM 0.25.0
+`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, RelWithDebInfo strict-contract build in
+`~/mxfp4-bench/src` at the flip sha, single-load/arm, drop_caches+mincore, free>=90 gated,
+reps tight CoV~1%). total_token_throughput ours/vLLM **c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969**
+vs #51 [1.005/0.925/0.939/0.953] = **+0.015/+0.037/+0.027/+0.016 — dense-ON BEATS the MoE route on
+EVERY axis, no regression**; median TPOT 1.019/0.960/0.958/0.961 (all improve vs #51); peak GPU mem
+28055 vs 73795 MiB = **2.63x LESS**. Evidence
+`dgx:~/work/vllm.cpp-online-gate/evidence/6a992a59.../summary-q3mxfp4/`.
+
+PARITY VERDICT (MXFP4 goal) = still BELOW-FLOOR <1.0x at c2-c8 (best c8 0.969, gate_pass NO) but
+CLOSER than #51; c1 crosses to 1.02x. RESIDUAL MAP (per #54 fair graphed-vs-graphed): the marlin
+term is now byte-safe-recovered; the next-dominant is decode-flash (+658-784us) then the norm/quant/
+act GLUE tail (+195-290us) + ~0.7ms host/sched. FLIP conditions (parity-enablers) MET: correctness
+ratified + beats MoE every axis + no regression + memory win. Landing `row/KERNEL-MARLIN-DENSE-EXEC`
+(`200b4b56`): flip `dense_nvfp4_gemm.h`, regen 32B goldens, gate-conditional counters, L2 unit fix.
+
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -309,7 +309,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | #51 x3: c1 1.005, c2/c4/c8 0.925/0.939/0.953, mem 2.18x. `MARLIN-STRUCT`: decode-graph + gate_up FUSION default-ON, marlin 180->144 GEMM/step (vLLM-structural); #44 3/3, 32B-NVFP4A16 142/142 | nsys c8: marlin +1,177us (CTA 144 vs 48, dominant); `VT_MARLIN_E1_PAR1` E=1->48 CTAs near-parity but flips a strict 32B token (OFF). Byte-preserving `KERNEL-MARLIN-DENSE-PORT` landed gated-OFF; GPU binding pending |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE FLIPPED DEFAULT-ON**: dense marlin 48-CTA (vs MoE 128), nsys c8 117.8->85.8us/call; ratified 32B near-tie (razor gap 0.000, goldens regen). Goal <1.0x c2-c8, next decode-flash |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |
 | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **NO NUMBER OWED, nothing executes on these arches.** 2026-08-06 sm_75 compile audit (nvcc 13.0.88): 20 unconditional sm_80+ constructs enumerated; detail in .agents/benchmark-record.md | Port the llama.cpp `fattn-tile`/`fattn-vec` fp16 body. Perf floor when a card exists is **llama.cpp on the same card** (vLLM does not run there) |

diff --git a/docs/FEATURES.md b/docs/FEATURES.md
@@ -65,7 +65,7 @@ are our reading of their documented behavior, not measurements.
 | GGUF k-quants and i-quants | ✅ | ☐ | ☐ | ✅ |
 | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ |
 | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ |
-| MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.6x less. gate_up FUSION + decode-graph default-ON (180->144 GEMM/step); #44 3/3, 32B 142/142. Residual = marlin CTA; byte-preserving `KERNEL-MARLIN-DENSE-PORT` landed gated-OFF, GPU gates pending | ✅ | ✅ | ☐ |
+| MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ |
 | fp8 weights | ✅ | ✅ | ✅ | ☐ |
 | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ |
 | Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ |