diff --git a/.agents/NOW.md b/.agents/NOW.md index bab77717..77c659f1 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -23,7 +23,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS | Lever: batched Marlin decode GEMM (M=2..8); nsys c8 both engines | +| QUANT-CT-MXFP4-BENCH | **Grid RAN** (`33e93608`): BELOW-FLOOR — 0.99x@c1 / ~0.91x@c2-c8, TPOT +10-12%@c2-c8, GPU mem 2.6x LESS. **Marlin-config lever REFUTED** (`row/QUANT-CT-MXFP4-M28-LEVER`): same-tool per-shape microbench = our MoE-E1 Marlin AT PARITY with vLLM dense `marlin_gemm` (moe/dense 0.99-1.08, all M); nsys c8 BOTH engines confirm Marlin at-parity, vLLM's edge = Inductor FUSION of norm/quant/act glue | REDIRECT: portable glue-fusion + SAME-tool c1-vs-c2 nsys diff (attn batching vs unfused glue vs async overlap); the gap is a STEP at c2, not the M-independent Marlin | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 0f8a6c70..396f2ab2 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -81,6 +81,55 @@ this row (all additive; 27/35 byte-unchanged; CPU contract tests 45/45): `online `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`; `mxfp4_smoke_gate.py`; `dgx-online-serving.sh` q3mxfp4 branches; `mxfp4-online-serving-grid.sh` orchestrator. +## QUANT-CT-MXFP4 c2-c8 gap — grouped-Marlin config lever REFUTED (2026-08-08, `row/QUANT-CT-MXFP4-M28-LEVER`, GB10, build `33e93608`==main) + +The binding (#45) attributed the c2-c8 TPOT deficit to "our per-expert/grouped Marlin +tiling vs vLLM's Marlin as M grows 2->8" and named the fix as porting vLLM's M-dependent +`thread_m_blocks`/tile selection. Profiled BOTH engines and ran the per-shape arbiter; the +attribution is **wrong** — our Marlin already matches vLLM's per-shape. + +**Same-tool per-shape microbench** (`~/mxfp4-nsys/mxfp4_marlin_ubench.py`; oracle compiled +ops, cuda-event timed; dense `ops.marlin_gemm` = what vLLM runs for a dense MXFP4 linear vs +`ops.moe_wna16_marlin_gemm` E=1 = our production kernel's 1:1 upstream twin; Qwen3-8B decode +shapes qkv 6144x4096, o 4096x4096, gate/up 12288x4096, down 4096x12288, gs=32): + +| M | dense/layer us | moe/layer us | moe/dense | dense step ms (x36) | moe step ms | delta ms | +|---|---|---|---|---|---|---| +| 1 | 226.6 | 231.7 | 1.023 | 8.16 | 8.34 | +0.18 | +| 2 | 231.5 | 228.3 | 0.986 | 8.33 | 8.22 | -0.12 | +| 4 | 223.7 | 240.8 | 1.077 | 8.05 | 8.67 | +0.62 | +| 8 | 237.6 | 242.3 | 1.020 | 8.55 | 8.72 | +0.17 | + +Per-shape at M=8: qkv 1.111, o 1.026, gate 1.002, up 0.998, down 1.030. All deltas are noise +(<0.6 ms/step; M=2 favors MoE). **Our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M**, +so routing to vLLM's dense marlin config cannot close the ~4.3 ms/step (11%) gap. (Source: +vLLM dense route `apply_fp4_marlin_linear`->`ops.marlin_gemm`, `marlin.cu:438` +`m_block_size_8=prob_m<=8`; ours `MoeGroupedGemmNvfp4Marlin` E=1, `marlin_mm_moe.cu:363` +`thread_m_blocks=div_ceil(moe_block_size,16)`, `MarlinMoeAlignBlockSizeSelect`=16 at M=8. +The dispatchers differ but the cost does not.) + +**nsys BOTH engines at c8** (aggregation-trap separated; TPOT reproduced: ours 39.53 vs vLLM +33.81 ms). OURS (`nsys -t cuda --cuda-graph-trace=node`, kern_sum): decode Marlin dominates +(`marlin_moe_wna16::Marlin` 58,854 inst Med **123us**; prefill Marlin separated 3,240 @ 2.03ms); +attention flash_fwd_splitkv<128,64,128,4> Med **218us**; glue = SEPARATE launches +(RmsNormRow*, MoeSiluMul, QkvSplit, RopeFromCache, ReshapeAndCache), one per op per layer. +VLLM (online `/start_profile` torch profiler, chrome-trace aggregate): decode marlin +`void marlin::Marlin` 13,680 @ **112.9us**; flash same traits @ **151us**; and the structural +tell — the glue is **FUSED into the GEMM by Inductor**: +`triton_red_fused_fused_add_rms_norm_marlin_gemm_{0,2}` (add+RMSNorm+quant prologue) and +`triton_poi_fused_marlin_gemm_mul_silu_slice` (silu+mul epilogue). vLLM collapses our ~5 +separate glue launches/layer into the GEMM pro/epilogue. + +VERDICT: the decode Marlin GEMM is at vLLM per-shape parity (REFUTED lever). The residual is +diffuse decode; the robust structural divergence is vLLM's Inductor **glue fusion** (portable- +fusion class). REDIRECT (no ceiling): (1) portable fusion of add+RMSNorm+quant into the Marlin +prologue and silu+mul into the epilogue; (2) a SAME-tool c1-vs-c2 OURS nsys diff to localize +the STEP at c2 (our Marlin is M-independent M1->M2 per the microbench, so the step is attention +batching, unfused glue traffic, or the async `max_concurrent_batches=2` overlap — not the GEMM); +(3) same-tool re-check of the cross-tool per-call flags (marlin 123 vs 113us; attention 218 vs +151us on identical flash traits). No code shipped — porting the dense marlin would gold-plate a +refuted hypothesis. Evidence: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`. + ## DeepSeek-V4-Flash UD-IQ2_M — IQ2_S + MXFP4 CPU keep-quant bring-up (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`) — no throughput owed (off-GPU correctness bring-up) Off-GPU task (GB10 down, no nvcc on the dev box), so NO throughput is measured or owed. Adds the two per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint's last 4 routed-expert slabs use — **IQ2_S** (ggml type 22; 2.5625 bpw codebook, Q8_K activation) and **MXFP4** (type 39; OCP micro-scaling fp4, 32-elem blocks, Q8_0 activation) — as first-class vt block dtypes (`kIQ2_S`/`kMXFP4`): block traits + dequant + a keep-quant `vec_dot`, ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook copied verbatim + direct sign bytes; `kvalues_mxfp4` + `e8m0_to_fp32_half`). The memory point: these load COMPRESSED (keep-quant) instead of the ~17 GiB bf16 expansion that OOM-reboots the 119 GiB pool. diff --git a/.agents/state.md b/.agents/state.md index 44df48cc..0647185f 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36782,3 +36782,63 @@ non-empty evidence root); fixed in `mxfp4-online-serving-grid.sh` this change. R --configure-log `. Evidence: `dgx:~/work/vllm.cpp-online-gate/evidence/33e93608...` (ratios.json + report.md + 24 raws + memory/thermal/cache-drop). local-ai-worker left as found (down); dgx /home build tree `~/mxfp4-bench` retained for the nsys continuation. + +## QUANT-CT-MXFP4-M28-LEVER: grouped-Marlin config lever REFUTED — our Marlin already at vLLM per-shape parity; the c2-c8 gap is diffuse decode (vLLM's edge = Inductor FUSION of the norm/quant/act glue) + + +Chased the binding's named lever (#45 / NOW `QUANT-CT-MXFP4-BENCH`): "the c2-c8 TPOT +gap is the batched grouped Marlin W4A16 decode GEMM as M grows 2->8; port vLLM's +M-dependent thread_m_blocks/tile config". Ran the mandatory profile-BOTH + per-shape +arbiter on dgx GB10 (build `33e93608` == main `7f4bbf88` MXFP4 code; both flock locks, +single-load, box left clean/idle). **The lever is REFUTED by same-tool per-shape +measurement.** Branch `row/QUANT-CT-MXFP4-M28-LEVER` off `origin/main` `7f4bbf88`. + +WHAT vLLM ACTUALLY RUNS (source, both sides cited): the dense MXFP4 W4A16 linear routes +`MarlinMxFp4LinearKernel.apply_weights` -> `apply_fp4_marlin_linear` -> `ops.marlin_gemm` +(the DENSE single-matrix marlin, `csrc/libtorch_stable/quantization/marlin/marlin.cu`; +`m_block_size_8 = prob_m<=8 && a16` at marlin.cu:438, direct A addressing). OURS routes +every dense projection through `MoeGroupedGemmNvfp4Marlin` E=1 (`cuda_moe_marlin.cu` -> +`marlin_mm_moe.cu`; `thread_m_blocks=div_ceil(moe_block_size,16)`, `moe_block_size` from +`MarlinMoeAlignBlockSizeSelect` = 8 at M=1/2/4, 16 at M=8, so `m_block_size_8` is FALSE at +M=8; indirect `sorted_token_ids` gather; hardcoded fp32 C_tmp reduce). So the two engines +DO run different marlin dispatchers — but that difference does NOT cost. + +PER-SHAPE ARBITER (same tool: both are the oracle's compiled torch ops, cuda-event timed; +`~/mxfp4-nsys/mxfp4_marlin_ubench.py`, dense `ops.marlin_gemm` vs `ops.moe_wna16_marlin_gemm` +E=1 at the 5 Qwen3-8B decode shapes x M in {1,2,4,8}). moe/dense per-STEP (sum of 5 GEMMs +x36 layers): M=1 **1.023**, M=2 **0.986**, M=4 **1.077**, M=8 **1.020** (deltas <0.6ms, in +noise; M=2 even favors MoE). At M=8 per-shape moe/dense: qkv 1.111, o 1.026, gate 1.002, +up 0.998, down 1.030. => **our production MoE-E1 Marlin is at parity with vLLM's dense +`marlin_gemm` at every M**; porting vLLM's dense M-config would NOT close the ~4.3ms/step +(11%) gap. This is exactly the scan-hypothesis that per-shape MEASUREMENT kills +(parity-lever-protocol lane #2) — recorded BEFORE any port, saving a large dead-end lift. + +nsys BOTH engines at c8 (aggregation-trap-separated by per-call time + count; OURS = +`nsys -t cuda --cuda-graph-trace=node` on our server, kern_sum; VLLM = online +`/start_profile` torch profiler on `vllm serve ... VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, +chrome-trace kernel aggregate). TPOT reproduced the binding: ours 39.53 vs vLLM 33.81 ms. +- OURS decode: dominant = decode Marlin (`marlin_moe_wna16::Marlin`, 58,854 inst, Med + **123us**); prefill Marlin separated (3,240 inst @ 2.03ms). Attention flash_fwd_splitkv + <128,64,128,4> Med **218us**. Glue launched SEPARATELY: RmsNormRow*, MoeSiluMul, QkvSplit, + RopeFromCache, ReshapeAndCache — one kernel each, per layer. +- VLLM decode: decode marlin (`void marlin::Marlin`, 13,680 inst @ **112.9us**); flash + same traits @ **151us**; and the structural tell — the glue is **FUSED by Inductor into + the GEMM**: `triton_red_fused_fused_add_rms_norm_marlin_gemm_{0,2}` (add+RMSNorm+quant + prologue, 3,600/3,500 inst @ 10.8/15.3us) and `triton_poi_fused_marlin_gemm_mul_silu_slice` + (silu+mul epilogue, 3,600 @ 25us). vLLM collapses our ~5 separate glue launches/layer into + the GEMM's pro/epilogue. + +CONCLUSION + REDIRECT (no ceiling): the Marlin GEMM kernel is NOT the lever — it is already +at vLLM per-shape speed; the binding's attribution ("our grouped tiling vs vLLM's Marlin") +was an inference the per-shape trace overturns. The residual c2-c8 gap is DIFFUSE decode. +The one robust STRUCTURAL divergence in the trace is vLLM's Inductor **fusion of the +norm/quant/activation glue** around at-parity marlin GEMMs (portable-fusion class, cf. +[[fusion-must-be-portable-reuse-patterns]] — "modest ~3.5%@c1, tails 3-14%"). Cross-tool +per-call flags to confirm SAME-tool next: decode marlin 123 vs 113us (~9%), decode +attention 218 vs 151us on the identical flash traits — and a c1-vs-c2 OURS nsys diff to +localize what grows M1->M2 (attention batching vs unfused glue vs the async +`max_concurrent_batches=2` overlap), since the binding curve is a STEP at c2, not monotone +in M (and our Marlin is M-independent M1->M2 per the microbench, so it cannot be the step). +No code shipped (the named fix is refuted; shipping a dense-marlin port would gold-plate a +dead hypothesis). Evidence text: `dgx:~/mxfp4-nsys/{kern_sum.txt,vllm_kern_sum.txt,ubench.log}`. +Box: both locks free, GPU idle, disk unchanged 23G, local-ai-worker left down, tmux clean. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 8fe6aa46..4485352e 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record) | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid RAN (c1-c8x3, `33e93608`): total 0.99x@c1 / ~0.91x@c2-c8; TPOT +10-12%@c2-c8 (parity@c1); TTFT parity; GPU mem 2.6x LESS. BELOW-FLOOR (batched-decode gap; benchmark-record). **Grouped-Marlin config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`, nsys c8 both engines + same-tool per-shape microbench):** our MoE-E1 Marlin == vLLM dense `marlin_gemm` at every M (moe/dense 0.99-1.08); decode Marlin is at-parity, so the residual is diffuse decode where vLLM's edge is Inductor FUSION of the norm/quant/act glue (`triton_..._add_rms_norm_marlin_gemm`, `..._marlin_gemm_mul_silu`). Redirect: portable glue-fusion + same-tool c1-vs-c2 diff | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/STATUS.md b/docs/STATUS.md index cba63198..2038a606 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput grid RAN on GB10 (`33e93608`): ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x LESS; BELOW-FLOOR (batched-decode gap) | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Native Marlin W4A16 keep-quant compute PROVEN correct (#38); W4 throughput grid RAN on GB10 (`33e93608`): ~0.91x vLLM c2-c8 / 0.99x c1, peak GPU mem 2.6x LESS; BELOW-FLOOR (batched-decode gap). **Marlin-config lever REFUTED (`row/QUANT-CT-MXFP4-M28-LEVER`):** same-tool per-shape microbench shows our MoE-E1 Marlin is AT PARITY with vLLM's dense `marlin_gemm` at every M (moe/dense 0.99-1.08); nsys c8 both engines confirm the decode Marlin is at-parity and vLLM's structural edge is Inductor **fusion** of the norm/quant/activation glue around the GEMMs — the gap is diffuse decode, not the grouped-Marlin tiling. Detail in state.md | The shared quant path DeepSeek-V4-Flash (MXFP4 experts) and Kimi-K3 (its real checkpoint) both need. CPU weight unpack + E8M0 dequant to bf16/f32 (`2^(byte-127)`, group 32, no global scale — distinct from our NVFP4 group-16 fp8-scale path); unit-gated 5/5·1142 vs a double-precision `dq_mxfp4_torch` port (RED-first, bias-128 mutation fails 446 assertions). GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are named later bricks; the two model loaders keep their MXFP4 refusal until wired | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet |