mudler · localai-bot · Aug 5, 2026 · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,7 +22,8 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED + dgx-VERIFIED** (`f9c969ae`): #31 async mirror ported to classic dense `Qwen3ForCausalLM`. Async gate RED→GREEN 0.6B+4B, SACRED 184/184, memcheck 0, MXFP4 default e2e 3/4 + near-tie RATIFIED | Residual: **W4 throughput bench** (online_gate lacks Yi30/8B key); sibling scope one-liner |
+| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
+| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS | Lever: batched Marlin decode GEMM (M=2..8); nsys c8 both engines |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -19,6 +19,68 @@ from relative link targets repointed for this file's location.
 
 # Benchmarks
 
+## QUANT-CT-MXFP4 W4 throughput bench — RAN on GB10, BELOW-FLOOR (2026-08-06, `row/QUANT-CT-MXFP4-BENCH` `33e93608`)
+
+The binding ours-vs-oracle online-serving grid on the SAME checkpoint
+(`Yi30/Qwen3-8B-MXFP4`, dense `Qwen3ForCausalLM`, native Marlin W4A16 MXFP4
+keep-quant), oracle arm forced to Marlin via `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`
+(sm_121 cute-dsl mxf4 crashes). Production graphed vLLM 0.25.0 (torch.compile ON,
+Marlin W4A16), `--no-enable-prefix-caching` both arms, max-num-seqs 32, batched-tokens
+2048, c1/c2/c4/c8 x3 interleaved, single load/arm, drop_caches + memory-return
+between legs, both flock locks. Smoke model gate reproduced #44 (3/3 deterministic
+token-exact + coherent near-tie). **VERDICT: BELOW-FLOOR** (gate NO, 74/84 axes below);
+throughput ~0.91x at c2-c8, ~0.99x at c1; a large memory WIN.
+
+**Binding table (medians of 3 reps; per-rep spread ~1-3%):**
+
+| conc | total tok/s ours→vllm (ratio) | output tok/s ratio | med TTFT ms ours→vllm (ratio, ≥1 pass) | med TPOT ms ours→vllm (ratio, ≥1 pass) |
+|---|---|---|---|---|
+| c1 | 324.2 → 327.9 (**0.989**) | 0.989 | 287.5 → 295.2 (**1.027** PASS) | 25.70 → 25.34 (0.986) |
+| c2 | 569.2 → 624.7 (**0.911**) | 0.911 | 458.4 → 450.3 (0.982) | 28.27 → 25.45 (0.900) |
+| c4 | 951.4 → 1034.7 (**0.919**) | 0.919 | 862.1 → 867.5 (**1.006** PASS) | 31.03 → 28.07 (0.905) |
+| c8 | 1419.6 → 1554.8 (**0.913**) | 0.913 | 1412.2 → 1408.3 (0.997) | 39.64 → 35.30 (0.891) |
+| memory | peak GPU 28284 → 73723 MiB = **2.607 PASS** (ours 2.6x LESS) | — | — | — |
+
+Per-rep total tok/s — c1 ours 324.2/324.4/320.0 vllm 331.5/327.7/327.9; c2 ours
+569.2/569.9/558.9 vllm 624.2/624.9/624.7; c4 ours 951.4/958.9/945.8 vllm
+1040.9/1034.7/1034.3; c8 ours 1415.0/1433.9/1419.6 vllm 1560.9/1554.8/1554.7. Per-rep
+TPOT ms — c1 ours 25.68/25.70/26.08 vllm 25.02/25.34/25.34; c2 ours 28.21/28.27/28.67
+vllm 25.43/25.45/25.49; c4 ours 31.03/30.92/32.48 vllm 27.94/28.07/28.12; c8 ours
+39.64/39.23/39.77 vllm 35.15/35.34/35.30.
+
+**FIRST attribution (grounded in the per-concurrency curve — the gap is a BATCHED
+decode cost, NOT batch-1):** at c1 (batch-1 decode) ours is at parity (throughput
+0.989, TPOT 0.986, within run-noise). At c2-c8 (batched decode 2-8 seqs) ours is
+~0.91x throughput driven ENTIRELY by TPOT (~10-12% higher per-token: c2 0.900, c4
+0.905, c8 0.891), while TTFT/prefill is at parity (2/4 cells pass, others within
+noise). So the divergent hot path is the GROUPED Marlin W4A16 keep-quant decode GEMM
+as M grows 2→8 (our per-expert/grouped tiling vs vLLM's Marlin), NOT the batch-1
+GEMV nor prefill. Memory is a 2.6x WIN (keep-quant weights stay compressed: 28.3 vs
+73.7 GiB peak). NOT a ceiling. The precise NEXT step: same-tool nsys decode-window
+on the WORST cell (c8) BOTH engines (our server under nsys; vLLM under nsys, not the
+torch profiler — same tool) to name the exact divergent kernel/shape at M=8, then a
+lever on the batched Marlin decode GEMM.
+
+**Repro (one command, dgx, both flock locks free, free -g ≥ 90):**
+`scripts/mxfp4-online-serving-grid.sh --snapshot ~/.cache/huggingface/hub/models--Yi30--Qwen3-8B-MXFP4/snapshots/b3e7ab32f7225ca779b3dbf6ef4ecefeb6de9b47 --build-dir <PROD_BUILD> --configure-log <CONFIGURE_LOG>`.
+**Build contract (LEARNED THE HARD WAY — the online-serving `record-execution` is
+strict and DIFFERS from the generic build recipe):** the build MUST be on a REAL
+disk (NOT `/dev/shm` — tmpfs pages are always resident so the cache-drop
+`POSIX_FADV_DONTNEED`+mincore==0 proof fails on the server binary), and configured
+`-DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.0/bin/nvcc
+-DCMAKE_MAKE_PROGRAM=$HOME/venvs/vllm-oracle/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
+-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+-DVLLM_CPP_TRITON_REGEN=OFF -DVLLM_CPP_FLASH_ATTN=ON -DVLLM_CPP_SERVER=ON
+-DVLLM_CPP_BUILD_TESTS=ON -DVLLM_CPP_BENCH_PROFILE_CONTROL=OFF -DVLLM_CPP_CUTLASS_DIR=$HOME/venvs/vllm-oracle/lib/python3.12/site-packages/flashinfer/data/cutlass`
+(the oracle's flashinfer-bundled cutlass, which is 4.5.0 — `record-execution` pins
+`VLLM_CPP_CUTLASS_DIR` to the oracle's own cutlass tree, not `$HOME/cutlass-4.5.0`).
+The full build tree is only ~3 GiB (fits /home). nvcc 13.0.88. Evidence:
+`dgx:~/work/vllm.cpp-online-gate/evidence/33e936086a116c840518afcf732076c9917e96a5`
+(ratios.json + report.md + 24 raws + memory/thermal/cache-drop). Harness plumbing on
+this row (all additive; 27/35 byte-unchanged; CPU contract tests 45/45): `online_gate.py`
+`q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`; `mxfp4_smoke_gate.py`;
+`dgx-online-serving.sh` q3mxfp4 branches; `mxfp4-online-serving-grid.sh` orchestrator.
+
 ## DeepSeek-V4-Flash UD-IQ2_M — IQ2_S + MXFP4 CPU keep-quant bring-up (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`) — no throughput owed (off-GPU correctness bring-up)
 
 Off-GPU task (GB10 down, no nvcc on the dev box), so NO throughput is measured or owed. Adds the two per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint's last 4 routed-expert slabs use — **IQ2_S** (ggml type 22; 2.5625 bpw codebook, Q8_K activation) and **MXFP4** (type 39; OCP micro-scaling fp4, 32-elem blocks, Q8_0 activation) — as first-class vt block dtypes (`kIQ2_S`/`kMXFP4`): block traits + dequant + a keep-quant `vec_dot`, ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook copied verbatim + direct sign bytes; `kvalues_mxfp4` + `e8m0_to_fp32_half`). The memory point: these load COMPRESSED (keep-quant) instead of the ~17 GiB bf16 expansion that OOM-reboots the 119 GiB pool.

diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md
@@ -127,7 +127,7 @@ Registry source:
 | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - |
-| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) still owed — online_gate lacks a Yi30/8B model key (needs corpus + oracle-record plumbing). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` |
+| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. W4 THROUGHPUT bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH` `33e93608`, c1/c2/c4/c8x3, production graphed vLLM 0.25.0, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` -> Marlin W4A16): **BELOW-FLOOR** (gate NO). Total tok/s ours/vLLM 0.99x@c1 but ~0.91x@c2-c8 (c2 0.911, c4 0.919, c8 0.913); driven by TPOT +10-12%@c2-c8 (parity@c1); TTFT at parity; peak GPU mem **2.6x LESS** (28.3 vs 73.7 GiB). ATTRIBUTION (grounded, per-concurrency curve): the gap is a BATCHED-decode cost absent at batch-1 -> the grouped Marlin W4A16 decode GEMM at M=2..8, NOT batch-1 GEMV nor prefill; NOT a ceiling. NEXT: same-tool nsys decode-window on c8 both engines. Harness (all additive; 27/35 byte-unchanged; CPU tests 45/45): `online_gate` `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`, [`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py) (#44 battery), `dgx-online-serving.sh` q3mxfp4 branches, [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh). Full table + build contract (RelWithDebInfo + oracle flashinfer-cutlass + DISK build) in benchmark-record. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` |
 | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |