From 2f8312e04235a80660d50ff018943a1d41fe9588 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 02:08:17 +0000 Subject: [PATCH] =?UTF-8?q?record(quant):=20MXFP4=20c1-vs-c2=20localizatio?= =?UTF-8?q?n=20=E2=80=94=20GLUE=20lever=20REFUTED;=20step=20=3D=20flash=20?= =?UTF-8?q?decode=20GQA=20group-swap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measurement-first localization of the MXFP4 c2-c8 gap (#45/#46 continuation). The binding's prescribed glue-fusion lever is REFUTED by same-tool measurement (vLLM's fused glue ~910us/step >= ours ~776us/step; add+RMSNorm already one kernel). The real c2-c8 step is the FLASH decode launch config: ours iterates 32 query heads (grid 1,3,64 = 2 waves @c2), vLLM applies the FA2 GQA seqlenq_ngroups_swapped so it iterates 8 kv-heads (grid 1,6,16 = 1 wave). Ours c2 flash 63.7us vs vLLM 41.7us (+53%), c8 218 vs 151 (+44%), same vendored flash_fwd_splitkv kernel. Marlin ~parity same-tool (+7-9%). Named lever + exact site (cuda_flash_attn_fa2.cu:1075) recorded; no code shipped (a numerically-near-tie graphed-decode launcher port needs a RED-first TDD cycle, not a rushed change). Evidence dgx:~/mxfp4-nsys/. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/state.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- 4 files changed, 78 insertions(+), 3 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 77c659f1..11213aa9 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -23,7 +23,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS. **Marlin-config lever REFUTED** (`row/QUANT-CT-MXFP4-M28-LEVER`): same-tool per-shape microbench = our MoE-E1 Marlin AT PARITY with vLLM dense `marlin_gemm` (moe/dense 0.99-1.08, all M); nsys c8 BOTH engines confirm Marlin at-parity, vLLM's edge = Inductor FUSION of norm/quant/act glue | REDIRECT: portable glue-fusion + SAME-tool c1-vs-c2 nsys diff (attn batching vs unfused glue vs async overlap); the gap is a STEP at c2, not the M-independent Marlin | +| QUANT-CT-MXFP4-BENCH | ~0.91x@c2-c8; marlin+glue levers **both REFUTED**; step = flash GQA group-swap | LEVER + detail: state `QUANT-CT-MXFP4-GLUE` (`cuda_flash_attn_fa2.cu:1075`) | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/state.md b/.agents/state.md index 0647185f..7824df7f 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36842,3 +36842,78 @@ in M (and our Marlin is M-independent M1->M2 per the microbench, so it cannot be No code shipped (the named fix is refuted; shipping a dense-marlin port would gold-plate a dead hypothesis). Evidence text: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`. Box: both locks free, GPU idle, disk unchanged 23G, local-ai-worker left down, tmux clean. + +## QUANT-CT-MXFP4-GLUE: c1-vs-c2 localization — GLUE-fusion lever REFUTED; the c2-c8 step is the FLASH decode GQA group-swap (ours iterates 32 q-heads, vLLM 8 kv-heads) + + +Ran the #46-prescribed discriminator (ours-only nsys c1-vs-c2 decode-window, same +binary `33e93608` == main MXFP4 code) + the mandated SAME-tool re-check of the two +cross-tool flags. **The prescribed GLUE-fusion lever is REFUTED by measurement; the +real c2-c8 step is the FLASH decode attention launch config.** Branch +`row/QUANT-CT-MXFP4-GLUE` off `origin/main` `024323e0`. Box left clean (both locks +free, GPU idle, worker down, disk 22G, evidence 202M on /home). + +METHOD: ours nsys `-t cuda --cuda-graph-trace=node` at c1 (3274 decode steps) + c2 +(1573 steps, 2 tok/step), 24x128 window, GB10 48 SMs. vLLM SAME-tool where reachable +(c2 nsys marlin; c2 torch `/start_profile` full breakdown; c8 torch from #46). Per-CALL +Med, aggregation-trap-separated by count. + +PER-KERNEL c1->c2 DELTA (ours, per-STEP = Med x calls/step): +| kernel | c1 Med | c2 Med | c1 /step | c2 /step | grows? | +|-------------------|-------:|-------:|---------:|---------:|--------| +| decode marlin x180| 114.75us| 120.74us| 20,655us | 21,732us | +1,077us (+5.2% Med) | +| decode flash x36 | 35.49us | 63.68us | 1,278us | 2,292us | **+1,014us (+79% Med)** | +| lm_head cutlass_80| 5,395us | 5,390us | 5,445us | 5,330us | flat (memory-bound GEMV) | +| glue (norm/silu/rope/qkv/reshape) | — | — | ~685us | ~776us | +90us (negligible) | +So the c1->c2 STEP growth (+2.06ms GPU-sum ~ +2.57ms measured TPOT) is FLASH + marlin; +GLUE barely moves. Glue is only ~2.5% of the step and the add+RMSNorm is ALREADY one +kernel (RmsNorm-with-residual), so Tier-1 fusion (`VT_FUSED_TIER=1`) is a near-nop for +graphed decode. **The binding's glue-fusion attribution is wrong.** + +CROSS-TOOL FLAG RE-CHECKS (the two #46 flags): +- **MARLIN 123 vs 113 (~9%): ~PARITY, benign.** c2 nsys vLLM decode marlin rows 108-123us + ~ ours 120.7us; c8 torch vLLM 112.9 vs ours 123. Same-tool ubench already showed the two + dispatchers at per-shape parity. Real but small (+7-9%); the grouped-moe-E1 indirect gather + vs dense-marlin direct-A. The #46 dense-marlin-config lever STAYS refuted. +- **FLASH 218 vs 151 (~44%): CONFIRMED REAL, and it is the primary lever.** Clean vLLM c2 + torch decode flash = **41.7us** vs ours nsys c2 **63.7us (+53%)**; c8 vLLM torch 151 vs ours + 218 (+44%). SAME vendored `flash_fwd_splitkv<128,64,128,4>` kernel BOTH sides -> a LAUNCH- + CONFIG divergence, not a kernel gap. +- **GLUE fusion: REFUTED as the differential.** vLLM's fused-glue triton launches + (`triton_red_fused_fused_add_rms_norm_marlin_gemm` 5.3+4.3us, `..._mul_silu_slice` 6.9us, + reshape 4.2us) sum to ~910us/step >= OURS ~776us/step. vLLM's glue is NOT cheaper; the + #46 Inductor-fusion "structural edge" does not exist at the per-step level. + +ROOT CAUSE (grid dims, both sides, same workload): ours decode flash grid = **(1,3,64)** +where gridZ=64 = batch(2)x**query_heads(32)**; vLLM = **(1,6,16)** where gridZ=16 = +batch(2)x**kv_heads(8)**. vLLM applies the FA2 GQA **`seqlenq_ngroups_swapped`** decode +optimization (h=kv_heads=8, the ngroups=4 query heads/group packed into seqlen_q, KV read +once per group) -> 96 CTAs = 1 wave on 48 SMs. Ours iterates all 32 query heads separately +-> c1 96 CTAs (1 wave, 35.5us, at PARITY) but c2 **192 CTAs = 2 waves (63.7us)** and c8 256 +CTAs (heuristic returns num_splits=1, ~5 waves, 218us). The over-wave at batch>=2 is EXACTLY +the c1-parity / c2-c8-degradation shape. Our launcher hard-codes the swap OFF at +`src/vt/cuda/cuda_flash_attn_fa2.cu:1075` (`p.seqlenq_ngroups_swapped = false; // the whole +point: no group swap`) on a mistaken belief vLLM's varlen decode does not swap; the grid +proves it does. The vendored kernel (`flash_fwd_kernel.h` get_lse_tile/combine) ALREADY +supports the swap flag. + +DIFFERENTIAL ATTRIBUTION at c2 (ours-vLLM, per step): marlin +1,476us (7%, ambiguous cross- +tool + #46-refuted), flash +792us (53%, CONFIRMED), lm_head +23us, glue -134us (ours less), +combine -25us => GPU diff ~+2.13ms of the +2.82ms measured TPOT gap (~76%); ~0.7ms host/sched +residual. At c8 flash is +2.4ms/step (~55% of the +4.34ms gap). + +LEVER (named, upstream-grounded, NOT yet shipped): port the FA2 GQA decode ngroups-swap into +`LaunchDecodeVarlenFA2Bf16` (mirror flash-attn `mha_fwd_kvcache` seqlenq_ngroups_swapped: +transpose q [nreq,32,d]->[nreq,ngroups=4,hkv=8,d], set h=hkv/seqlen_q=ngroups, recompute +num_splits + LSE/oaccum scratch on the swapped dims, combine already supports the swapped LSE +stride). NON-byte-exact (splitkv reduction order changes -> near-tie; it moves us TOWARD +vLLM's numerics), touches the CUDA-graph-captured decode -> needs a RED-first op test + +#44 smoke token-exact gate BEFORE a default flip; ship gated `VT_FA2_DECODE_GQA_SWAP` default +OFF. Not shipped this session per byte-exact-first / SACRED-decode discipline (a rushed +launcher rewrite of the graphed decode path is the wrong risk); scoped as the TDD next step +with the exact site + the vLLM-grid proof. HONEST residual: flash alone closes ~28%(c2)/ +~55%(c8) of the gap; reaching >=1.0x also needs the marlin +7-9% (grouped->dense direct-A, a +bigger delicate lift) and the ~0.7ms host/sched slice. No single lever reaches >=1.0x. +Evidence: `dgx:~/mxfp4-nsys/{kern_sum_c1.txt,kern_sum_c2.txt,vllm_torch_c2.txt,vllm_kern_nsys_c2.txt, +vllm_kern_sum.txt}` + reps `ours_c1/ours_c2/vllm_c2.nsys-rep`. +Box: both locks free, GPU idle, disk 22G, local-ai-worker left down, tmux clean. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 4485352e..98ec5c4b 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record). **Grouped-Marlin config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`, nsys c8 both engines + same-tool per-shape microbench):** our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M (moe/dense 0.99-1.08); decode Marlin is at-parity, so the residual is diffuse decode where vLLM's edge is Inductor FUSION of the norm/quant/act glue (`triton_..._add_rms_norm_marlin_gemm`, `..._marlin_gemm_mul_silu`). Redirect: portable glue-fusion + same-tool c1-vs-c2 diff | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid (`33e93608`): 0.99x@c1 / ~0.91x@c2-c8, mem 2.6x LESS. Marlin-config + glue levers **both REFUTED**; c2-c8 step = flash decode GQA group-swap. Detail + lever: state `QUANT-CT-MXFP4-GLUE` | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/STATUS.md b/docs/STATUS.md index 2038a606..38eb120d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput grid RAN on GB10 (`33e93608`): ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x LESS; BELOW-FLOOR (batched-decode gap). **Marlin-config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`):** same-tool per-shape microbench shows our MoE-E1 Marlin is AT PARITY with vLLM's dense `marlin_gemm` at every M (moe/dense 0.99-1.08); nsys c8 both engines confirm the decode Marlin is at-parity and vLLM's structural edge is Inductor **fusion** of the norm/quant/activation glue around the GEMMs — the gap is diffuse decode, not the grouped-Marlin tiling. Detail in state.md | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput grid RAN on GB10 (`33e93608`): ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x LESS; BELOW-FLOOR (batched-decode gap). **Marlin-config + glue-fusion levers both REFUTED by measurement** (`row/QUANT-CT-MXFP4-M28-LEVER`, `row/QUANT-CT-MXFP4-GLUE`); the c2-c8 step is the flash decode GQA group-swap (lever site `cuda_flash_attn_fa2.cu:1075`). Detail in state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4-expert paths. CPU E8M0 dequant (`2^(byte-127)`, group 32, no global scale) unit-gated 5/5·1142 vs `dq_mxfp4_torch`. GPU W4A4 fp4 GEMM + MoE-expert e2e remain later bricks | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet |