mudler · localai-bot · Aug 6, 2026 · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -1,6 +1,6 @@
 # NOW — the one-Read resume surface
 
-<!-- now-updated: 2026-08-08 -->
+<!-- now-updated: 2026-08-09 -->
 
 Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live,
 the gate being chased, what to do next. Never a log — evidence lives in the
@@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| MXFP4 parity (Qwen3-8B) | c1 0.990/TTFT parity/mem 2.6x WIN; c2-c8 0.92-0.94 post-GQA-swap (#49). Dense-marlin port arbitrated **NO-GO** (#50: 4-run ubench, +7-9% was cross-tool bias; only real sliver = c8 block-16 padding ~0.33ms) | Closers: c8 block=8 tweak (~5 lines) + ws-memset drop + the ~0.7ms/step host/sched slice |
+| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW (c8 +1.1pp vs #49), mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | Residual: grouped→dense-direct-A; host slice NOT engine-loop |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| MXFP4 parity goal | swap-ON binding **still <1.0x** (best c8 0.942); residual grouped-Marlin +7-9% + ~0.7ms host/sched | NEXT: grouped→dense-direct-A marlin decode |
+| MXFP4 parity goal | binding <1.0x on c2-c8 (best c8 0.953); c1 crosses to parity+. Residual = grouped-Marlin +7-9% (only real lever) + host slice (VT_LOOP_TRACE: NOT engine-loop) | NEXT: grouped→dense-direct-A E1 marlin |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12554,3 +12554,64 @@ launch/GEMM, micro. (c) the **~0.7ms/step host/sched slice** — the real remain
 not a kernel port. No single lever reaches ≥1.0x; the MXFP4 parity goal stays BELOW-FLOOR (best c8
 0.942). Evidence: `dgx:~/mxfp4-nsys/{mxfp4_marlin_ubench.py,ubench.log,ubench_3x.log}`. Box left
 clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
+
+## QUANT-CT-MXFP4-CLOSERS: slivers (a)+(b) land BYTE-EXACT + default-ON; correctness gates GREEN, binding re-bench PENDING (2026-08-09, `row/QUANT-CT-MXFP4-CLOSERS`, GB10 sm_121a CUDA 13.0, `~/mxfp4-bench` overlay `1f446fd7`+swap-ON+3 files md5-matched)
+The two #50-arbitrated slivers, both in the SHARED header
+`include/vllm/model_executor/models/dense_nvfp4_gemm.h` (the qwen3_5.cpp twin serving the 27B/35B
+gate models is deliberately untouched): **(a)** `DenseAlignFor` forces `block=8` at M≤8 (:286),
+routing the M=8 dense E1 GEMM to vLLM's 8-row `m_block_size_8` tile instead of the padded 16-row
+tile (recovers the #50-measured ~0.33ms/step / ~0.8pp at c8); **(b)** `DenseMarlinWorkspace` zeroes
+the shared reduction workspace ONCE at alloc and the two per-call `Memset(ws)` are DROPPED — the
+fp32-reduce marlin barrier self-resets its locks (invariant cited: `use_atomic_add=false`
+cuda_moe_marlin.cu:141 ⇒ only the fp32 barrier path is reachable, whose last release zeroes the lock
+marlin_template.h:2170→:204; slice_count==1 never touches locks :2162; the atomic-add non-clearing
+path :614 is dead). GATES (both flock locks, ninja EXIT 0 no -Werror): OP RED-first
+`test_ops_moe_grouped` closers — block8-vs-block16 A/B at M=8 **BYTE-EXACT** (`bitdiff=0/32768
+max_abs=0`) on MXFP4 K=4096/N4096 + K=12288/N4096 AND NVFP4 K=4096/N4096; ws all-zero after a GEMM +
+reuse bit-identical; 15/15·2 cases. LAUNCH-CONFIG `test_qwen3_forward` `DenseAlignFor(d,8).block==8`
+(pre-fix ==16) 7/7. MEMCHECK 0 real errors (leaks = pre-existing static-cache harness artifacts; the
+unchanged prior test leaks MORE). #44 smoke (Yi30/Qwen3-8B-MXFP4, default async graphed, vllm-cli
+greedy vs golden) **3/3 deterministic TOKEN-EXACT** + near-tie coherent. Byte-exact ⇒ DEFAULT-ON
+unconditional (no `VT_*` gate). Blast radius (header consumers Qwen3-8B-MXFP4, Qwen3-32B-NVFP4A16,
+Laguna) closed by proving BOTH quant schemes byte-exact. Binding c1-c8 x3 (clean-checkout grid) + the
+substantive ~0.7ms/step host/sched slice (the #47 residual) remain the parity verdict's open terms.
+Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
+
+## QUANT-CT-MXFP4-CLOSERS BINDING: clean-checkout grid at d3b412f5 — slivers improve EVERY throughput axis vs #49 (c1 crosses to parity+); MXFP4 goal still <1.0x on c2-c8 (2026-08-09, GB10, vLLM oracle 0.25.0, evidence `dgx:~/work/vllm.cpp-online-gate/evidence/d3b412f5c191aace1f2960fa7940d8eef925762a`)
+Full c1/c2/c4/c8 x3 binding on the CLEAN-checkout build at the committed sha d3b412f5 (not an
+overlay), oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, single-load/arm,
+drop_caches+mincore, RelWithDebInfo+oracle-cutlass. #44 model gate re-passed inside the grid
+(mxfp4_smoke_battery). 24/24 legs `failed:0`; reps tight (ours c8 [1481.7,1475.1,1486.7] tok/s,
+CoV ~0.4%; vllm c8 CoV ~0.06%). The grid EXIT=1 is the gate-FAIL signal (`gate_pass:false`) + the
+single-model "cross-model summary waits for the other model", NOT a crash — all q3mxfp4 artifacts
+are complete. (q3mxfp4's sweep IS c1-c8, matching #48/#49; c16/c32 are other-model points.)
+
+| axis (ours/vLLM normalized ratio) | c1 | c2 | c4 | c8 | vs #49 (tput) |
+|---|---|---|---|---|---|
+| total_token_throughput | **1.005** | 0.925 | 0.939 | **0.953** | +0.015/+0.003/+0.009/+0.011 |
+| median_tpot_ms         | 1.002 | 0.922 | 0.915 | 0.939 | — |
+| mean_ttft_ms           | 1.034 | 0.962 | 1.004 | 0.999 | — |
+| median_itl_ms          | 1.004 | 0.919 | 0.920 | 0.929 | — |
+
+Peak host-mem footprint (peak_mem_available_drop): ours **35.2 GiB** vs vLLM **76.7 GiB** = **2.18x
+LESS** (WIN). VERDICT: **c1 PASSES every axis (tput 1.005, tpot 1.002, ttft 1.034, itl 1.004)**;
+c2-c8 BELOW on tput/tpot/itl (best c8 0.953 tput), TTFT at parity c4/c8 (1.004/0.999). gate_pass
+FALSE => MXFP4 parity goal **NOT DONE** (below-floor on c2-c8). The slivers moved EVERY throughput
+axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), exactly the
+byte-exact block=8 recovery (~0.8pp @c8, #50) plus the memset-drop; nothing regressed.
+
+RESIDUAL MAP (per-shape, c8 ~4.75% tput gap): (1) block=16 padding — **CLOSED this run** (sliver a).
+(2) grouped-Marlin decode **+7-9% per-call** (GPU): `MoeGroupedGemmNvfp4Marlin` E=1 indirect
+`sorted_token_ids` gather + fp32 `C_tmp` vs vLLM dense `marlin_gemm` direct-A (#46/#50; a delicate
+grouped->dense-direct-A port, per-shape parity at M<=8 so not a config lever). (3) the host slice.
+
+HOST-SLICE ATTRIBUTION (step 2, VT_LOOP_TRACE on our server under decode load, this session): in
+every 1 s window the engine-core loop shows **interval_ms ≈ step_ms** (mean 25.5 vs 25.5, delta
+<=0.02 ms; `admits=0` pure-decode windows show interval-step ≈ 0). So the born-on-runner
+engine-core decode loop (scheduler + drain + admit) carries **negligible** per-iteration host
+overhead — the #47 ~0.7ms/step "host/sched" residual is NOT in the engine loop; it lives in the
+shared-architecture async frontend (HTTP / output-processing / detokenize, which vLLM runs too) or
+within the cross-tool attribution boundary (~0.7ms is ~24% of the c2 gap, near measurement error).
+Not a born-on-runner lever. (Caveat: the curl load under-saturated vs steady c8, so step_ms 25.5
+is not the c8 TPOT 37.6; the interval≈step finding is batch-independent and robust.)
+Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
diff --git a/.agents/state.md b/.agents/state.md
@@ -37070,3 +37070,113 @@ grouped-Marlin decode **+7-9% per-call** — `MoeGroupedGemmNvfp4Marlin` E=1 ind
 at M≤8 per #46 → a delicate grouped→dense-direct-A port, not a config knob); (2) **~0.7ms/step
 host/sched**. No single lever ≥1.0x. NEXT lever candidate = grouped→dense-direct-A marlin decode.
 
+## QUANT-CT-MXFP4-CLOSERS: the two arbitrated slivers LAND byte-exact + default-ON — dense M≤8 8-row Marlin tile (#50 (a)) + per-call workspace re-zero dropped (#50 (b)); correctness-complete on GB10, binding re-bench pending
+<!-- state: 2026-08-09T10:00 -->
+
+Implemented the two #50-arbitrated closer slivers on `row/QUANT-CT-MXFP4-CLOSERS`
+off `origin/main` `b1ac444e`. **Both are BYTE-EXACT (proven, not asserted) so they
+ship DEFAULT-ON unconditionally — no gate, no near-tie razor.** MXFP4 parity goal
+still owes the c1-c8 x3 binding re-bench (the definitive per-axis verdict) + the
+substantive ~0.7ms/step host/sched slice. Box left clean (both locks free, GPU
+idle, worker down, disk 21G).
+
+WHAT SHIPPED (file:line, all in the SHARED dense header
+`include/vllm/model_executor/models/dense_nvfp4_gemm.h`; the qwen3_5.cpp twin that
+serves the 27B/35B gate models' dense attn + shared-expert is DELIBERATELY
+UNTOUCHED, so those SACRED gates cannot move):
+- (a) `DenseAlignFor` (:286): `c.block = (M<=8) ? 8 : MarlinMoeAlignBlockSizeSelect(M,1,1)`.
+  vLLM's DENSE marlin uses an 8-row tile for a16 at prob_m<=8 (`m_block_size_8 =
+  prob_m<=8`, marlin.cu:438) — NO padding; our grouped single-expert MoE-align
+  picked block=16 at M=8 (`8*1/1/8==1.0` fails the `<0.9` test,
+  cuda_marlin_repack.cu:362), padding 8 dummy rows into a 16-row tile
+  (m_block_size_8=false) = the reproducible +0.33ms/step at c8 (#50 arbiter). The
+  m_block_size_8=true 8-row kernels are vendored (kernel_selector.h:3-8 nvfp4,
+  :33-38 mxfp4); marlin_mm_moe.cu:363-364 maps block=8 -> thread_m_blocks=1 +
+  m_block_size_8=true. M>8 unchanged (already matches vLLM).
+- (b) `DenseMarlinWorkspace` (:294): zero the shared reduction workspace ONCE at
+  alloc; the two per-call `d.b.Memset(ws)` in `MatmulNvfp4MarlinD`/
+  `GateUpFusedMarlinD` DROPPED. Mirror of vLLM `marlin_make_workspace_new`
+  (`torch.zeros`, marlin_utils.py:399-407) reused without re-zeroing. INVARIANT
+  cited in source: our launch pins `use_atomic_add=false / use_fp32_reduce=true`
+  (cuda_moe_marlin.cu:141-142), so the ONLY reachable cross-CTA reduce is the
+  fp32 barrier, whose last slice-block release re-zeroes the lock
+  (marlin_template.h:2170 `barrier_release(...,last)` -> `lock[0]=0` :204);
+  slice_count==1 never touches locks (:2162). The non-self-clearing atomic-add
+  path (:614) is unreachable under the pinned config (comment restores the zero
+  if it ever flips).
+
+GATE VERDICTS (dgx GB10, sm_121a, CUDA 13.0, RelWithDebInfo, MARLIN=ON, oracle
+CUTLASS; `~/mxfp4-bench` overlay at `1f446fd7`+swap-ON+the 3 files, md5-matched;
+ninja EXIT 0, no -Werror; both flock locks held for GPU legs):
+- OP UNIT (RED-first, `tests/vt/test_ops_moe_grouped.cpp` new closers cases): the
+  block8-vs-block16 A/B at M=8 on REAL decode shapes is **BYTE-EXACT** —
+  `bitdiff=0/32768 max_abs=0` for MXFP4 K=4096/N=4096 AND K=12288/N=4096, and for
+  NVFP4 (group_blocks=1) K=4096/N=4096. So forcing block=8 does NOT flip an output
+  bit vs the production-validated block=16 route → zero risk to the SACRED token
+  gate. Sliver-(b) invariant proven DIRECTLY: workspace all-zero AFTER a GEMM
+  (`ws_nonzero==0`) + a second GEMM on the once-zeroed workspace bit-identical.
+  15/15 assertions, 2/2 cases GREEN.
+- LAUNCH-CONFIG (RED-first, `tests/vllm/models/test_qwen3_forward.cpp`):
+  `DenseAlignFor(d,8).block==8` (post-fix) with the pre-fix value documented
+  (`MarlinMoeAlignBlockSizeSelect(8,1,1)==16`); M in {1,2,4,8}==8, M in {12,16}
+  unchanged. 7/7 GREEN.
+- MEMCHECK: `compute-sanitizer --tool memcheck --leak-check full` on the closers
+  cases = **0 real memory errors** (no invalid access/race). The only leaks are
+  static-cache artifacts of the moe_grouped CUDA harness at process exit — the
+  UNCHANGED pre-existing MXFP4 test leaks MORE (2.95MB/7 vs 0.98MB/2), so the
+  memset-drop introduces no leak. Memory-clean.
+- #44 SMOKE (Yi30/Qwen3-8B-MXFP4, default async graphed config, vllm-cli greedy vs
+  golden_marlin_w4a16.json): **3/3 deterministic TOKEN-EXACT + near-tie coherent**
+  — the integrated model output is identical to golden, as byte-exactness predicts.
+
+DEFAULT DECISION = ON, unconditional (byte-exact bug-class refinement, not a
+numerics change → no `VT_*` gate, unlike the FA2 swap). BLAST RADIUS (header
+consumers): Qwen3-8B-MXFP4 (target), Qwen3-32B-NVFP4A16, Laguna dense projections
+— all exercise the same `MatmulNvfp4W4A16D`; byte-exactness proven for BOTH quant
+schemes closes it. The 32B-NVFP4 SACRED gate decodes at M=1 (sliver (a) is a
+no-op there); the NVFP4 unit arm covers the M=8 path it can't.
+
+PARITY VERDICT (MXFP4 goal) = binding re-bench PENDING (this pass shipped the
+correctness half). Projected: sliver (a) recovers ~0.33ms/step at c8 (~0.8pp,
+#50 arbiter) byte-exact; (b) removes ~120 per-step Memset launches (host micro).
+The ~0.7ms/step host/sched slice (the substantive term, #47 residual) is
+UNADDRESSED. So the honest projection stays <1.0x on c2-c8 (best c8 ~0.95x); the
+binding x3 confirms + attributes. NEXT: clean-checkout binding grid
+(`mxfp4-online-serving-grid.sh`) + host-slice nsys.
+
+## QUANT-CT-MXFP4-CLOSERS BINDING + host-slice attribution: slivers improve EVERY throughput axis vs #49 (c1 crosses to parity+); MXFP4 goal still <1.0x on c2-c8 → NOT DONE; the ~0.7ms host slice is NOT in the engine loop
+<!-- state: 2026-08-09T12:00 -->
+
+Ran the definitive clean-checkout binding grid at the committed sha `d3b412f5` (NOT an overlay:
+`git checkout -f d3b412f5` in `~/mxfp4-bench/src`, tree clean, binaries rebuilt) via
+`mxfp4-online-serving-grid.sh`, oracle vLLM 0.25.0
+`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, single-load/arm, drop_caches+mincore. #44 model
+gate re-passed inside the grid. 24/24 legs `failed:0`, reps tight (ours c8 CoV ~0.4%). The grid
+EXIT=1 is the gate-FAIL signal (`gate_pass:false`) + single-model "cross-model summary waits for the
+other model" — NOT a crash; all q3mxfp4 artifacts complete (q3mxfp4's sweep IS c1-c8, matching
+#48/#49). Evidence `dgx:~/work/vllm.cpp-online-gate/evidence/d3b412f5.../summary-q3mxfp4/ratios.json`.
+
+BINDING (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]:
+total_token_throughput c1 **1.005** / c2 0.925 / c4 0.939 / c8 **0.953** (Δ +0.015/+0.003/+0.009/
++0.011); median TPOT c1 1.002 / c2 0.922 / c4 0.915 / c8 0.939; mean TTFT c1 1.034 / c2 0.962 / c4
+1.004 / c8 0.999; median ITL c1 1.004 / c2 0.919 / c4 0.920 / c8 0.929. Peak host-mem 35.2 vs 76.7
+GiB = **2.18x LESS** (WIN).
+
+PARITY VERDICT (MXFP4 goal): **c1 PASSES every axis** (tput/tpot/ttft/itl all ≥1.0x); c2-c8 BELOW on
+tput/tpot/itl (best c8 0.953 tput), TTFT parity c4/c8. `gate_pass:false` → **NOT DONE** (below-floor
+on c2-c8). But the byte-exact slivers moved EVERY throughput axis up vs #49 (c1 +1.5pp crosses to
+parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression — exactly the #50 block=8 recovery
+(~0.8pp@c8) + the memset-drop.
+
+RESIDUAL MAP (c8 ~4.75% tput gap): (1) block=16 padding — CLOSED (sliver a). (2) grouped-Marlin
+decode +7-9% per-call (GPU: E=1 indirect `sorted_token_ids` gather + fp32 `C_tmp` vs vLLM dense
+`marlin_gemm` direct-A; #46/#50 — delicate grouped→dense-direct-A port, not a config knob). (3) the
+host slice. HOST-SLICE ATTRIBUTION (step 2, VT_LOOP_TRACE under decode this session): every window
+shows **interval_ms ≈ step_ms** (mean 25.5 vs 25.5, Δ≤0.02ms; `admits=0` pure-decode windows
+interval−step ≈ 0) → the born-on-runner engine-core loop (scheduler+drain+admit) has NEGLIGIBLE
+per-iteration host overhead. So the #47 ~0.7ms/step "host/sched" residual is NOT an engine-loop
+lever — it is in the shared-architecture async frontend (HTTP/output-proc/detokenize, which vLLM
+also runs) or within the cross-tool attribution boundary. NEXT lever (only real one left) =
+grouped→dense-direct-A marlin decode (#50 NO-GO'd the full port; a targeted E1 direct-A path is the
+open candidate). Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).
+
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -307,7 +307,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | Post-GQA-swap (#49): c1 0.990, c2-c8 0.92-0.94, TTFT parity, mem 2.6x WIN; dense-marlin port NO-GO (#50, cross-tool bias) | Closers: c8 block=8 tweak + ws-memset drop + ~0.7ms host slice |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | grouped→dense-direct-A marlin; host slice NOT engine-loop (VT_LOOP_TRACE interval≈step) |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |