diff --git a/.agents/NOW.md b/.agents/NOW.md index 7871368f..8a23d813 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW (c8 +1.1pp vs #49), mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | Residual: grouped→dense-direct-A; host slice NOT engine-loop | +| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW, mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | c8 same-tool diff → goal row | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| MXFP4 parity goal | binding <1.0x on c2-c8 (best c8 0.953); c1 crosses to parity+. Residual = grouped-Marlin +7-9% (only real lever) + host slice (VT_LOOP_TRACE: NOT engine-loop) | NEXT: grouped→dense-direct-A E1 marlin | +| MXFP4 parity goal | c8 same-tool diff (`QUANT-CT-MXFP4-C8-DIFF`) SETTLES #50: marlin grouped-5-GEMM (gate_up unfused, 180 vs 144) DOMINANT +0.9/+1.4ms; eager gap +0.88ms graph-closeable nets +1.3%; flash same-grid +11% | NEXT: grouped→dense-direct + gate_up fuse | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index fdf4dee6..fe36a8b3 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12615,3 +12615,50 @@ within the cross-tool attribution boundary (~0.7ms is ~24% of the c2 gap, near m Not a born-on-runner lever. (Caveat: the curl load under-saturated vs steady c8, so step_ms 25.5 is not the c8 TPOT 37.6; the interval≈step finding is batch-independent and robust.) Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). + +## QUANT-CT-MXFP4-C8-DIFF: fresh SAME-TOOL c8 decode-window per-step diff (POST-SLIVER `d3b412f5`==`4dd4e206`/#51) — marlin grouped-5-GEMM DOMINANT; eager launch-gap +880µs graph-closeable nets only −334µs; #50 grouped==dense SETTLED as isolated-shape ubench artifact; decode-graph opt-in = +1.3% byte-coherent (2026-08-09, GB10 sm_121a CUDA 13.0, RelWithDebInfo MARLIN=ON oracle-cutlass, vLLM oracle 0.25.0 `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`) + +SAME-TOOL both sides (the fix for the recorded cross-tool trap): nsys `--cuda-graph-trace=node`. +OURS = online server, 24×128 window at c8. vLLM online-under-nsys does NOT capture the V1 EngineCore +subprocess (empty rep at c8, MP-on AND MP-off) → captured vLLM via OFFLINE `LLM()` with +`VLLM_ENABLE_V1_MULTIPROCESSING=0` (in-process) + cudaProfilerApi fence, IDENTICAL 8 corpus prompts +(1024 tok) = clean M=8 batched decode. Kernel per-call times are frontend-independent. Steps segmented +by lm_head anchor; steady window = modal-marlin-count; BOTH at flash gridZ=64 (matched M=8). Single +node-trace per arm (per-step medians over 300+ steps are stable); binding c1–c8 x3 is #51's (0.953 c8). + +PER-STEP DECODE (median µs), OURS EAGER (as-shipped) vs vLLM (graphed): +| class | ours µs/step (calls) | vLLM µs/step (calls) | Δ | +|--------------|---------------------:|---------------------:|------------:| +| marlin W4A16 | 17,183 (180=5×36) | 16,286 (144=4×36) | +897 (+5.5%)| +| flash decode | 6,250 (36) | 5,629 (36) | +621 (+11%) | +| glue | 927 (291) | 671 (299) | +256 | +| lm_head | 5,413 (1) | 5,395 (1) | +18 | +| flash_comb | 130 (36) | 140 (36) | −10 | +| sample+other | 36 | 23 | +13 | +| GPU BUSY | 30,027 | 28,190 | +1,837 | +| GAP (idle) | 1,184 (3.8%) | 304 (1.1%) | +880 | +| STEP SPAN | 31,211 | 28,491 | +2,720 | +Measured median TPOT ours 37.56 / vLLM 34.58 ms (+2.98); GPU SPAN accounts 91%; ~0.26ms = async frontend. + +GAP LOCALIZATION (ours): uniform ~2.0–2.4µs idle between EVERY consecutive kernel + ~14 H2D + ~2 memset +per step = host-launch-bound EAGER. vLLM ~0.4–0.76µs/gap = piecewise-CUDA-graph replay. Grids: ours +marlin 144×1×1 (3 waves M=8), vLLM 48×1×1 (1 wave); flash IDENTICAL 1×3×64 both (GQA-swap parity). + +DECODE-GRAPH A/B (ours, `VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=1`, default-OFF opt-in, byte-identical-to-eager +per source; SACRED gate unrun): the generic Qwen3-MXFP4 decode is EAGER by default (root-caused in +`qwen3.cpp:DenseDecodeGraphForward`; no quant restriction). Graph ON: GAP 1,184→305µs (−880, =vLLM's, +confirms the mechanism), BUSY 30,027→30,575µs (+547 bandwidth contention when packed), SPAN 31,211→30,877 +(−334), median TPOT 37.56→37.09ms (−1.3%), coherent. → c8 0.953→~0.965, still <1.0x. Self-limiting. + +FAIR graphed-vs-graphed (ours-graph vs vLLM): SPAN +2,386µs = marlin +1,377 (58%) + flash +658 (28%) + +glue +290 (12%) + gap ~0. Dominant = MARLIN. + +VERDICT (a): dominant term = MARLIN — ours runs 5 grouped-MoE-E1 GEMM/layer (gate+up UNFUSED, 144-CTA) +vs vLLM 4 dense (gate_up FUSED, 48-CTA) = +5.5%(eager)/+8.5%(graphed)/step, mostly the +25% GEMM count. +SETTLES #50: its "grouped==dense parity" was vLLM-op-vs-vLLM-op at one ISOLATED shape, blind to OUR +kernel's +36 GEMM/step + 3× CTAs; the residual is real per-step. Fix = grouped→dense-direct + gate_up +fuse (delicate ~2000-line port, next dispatch). Secondary = decode-graph opt-in (+1.3%, SACRED owed). +Tertiary = flash same-kernel/IDENTICAL-grid +11% (KV/splitkv), glue Inductor-fusion. +Evidence dgx:~/mxfp4-nsys/{kern_sum_c8_dflt,vllm_offline_kern_c8,kern_sum_c8_graph}.txt + +gpu_trace_c8_{dflt,graph}/vllm_offline_trace_c8_cuda_gpu_trace.csv + analyze_decode.py/gap_and_shape.py. +Box left clean (both locks free, GPU idle, worker down, disk 20G, tmux gone). diff --git a/.agents/state.md b/.agents/state.md index c4edbfc5..636a76f7 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -37180,3 +37180,81 @@ also runs) or within the cross-tool attribution boundary. NEXT lever (only real grouped→dense-direct-A marlin decode (#50 NO-GO'd the full port; a targeted E1 direct-A path is the open candidate). Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). +## QUANT-CT-MXFP4-C8-DIFF: fresh SAME-TOOL c8 decode-window diff on the POST-SLIVER binary — kernels are NOT identical; marlin grouped-5-GEMM is DOMINANT (+897us eager/+1377us graphed), eager launch-gap +880us is graph-closeable but nets only −334us (bandwidth contention); #50 "grouped==dense parity" SETTLED as an isolated-shape ubench artifact; decode-graph opt-in is a measured +1.3% byte-coherent secondary lever + + +The deciding measurement the campaign owed: a FRESH same-tool (nsys `--cuda-graph-trace=node` +BOTH sides) c8 decode-window per-step diff on the post-sliver binary (box build `d3b412f5` == +main `4dd4e206`/#51; block=8 sliver + GQA-swap default-ON both present, `ninja: no work to do`). +Prior campaign per-kernel flags (#46/#47) were CROSS-TOOL (our nsys vs vLLM torch-profiler) — the +old marlin 123-vs-113/flash 218-vs-151 were partly cross-tool bias. This is the first SAME-tool c8 +node-trace of BOTH. Branch `row/QUANT-CT-MXFP4-C8-DIFF`. Box left clean (locks free, GPU idle, +worker down, disk 20G, tmux gone). NO code shipped (dominant fix not byte-small; decode-graph is an +existing default-OFF opt-in whose per-model SACRED gate is unrun). + +METHOD (reliable same-tool capture — the crux): OURS = online server under nsys, 24×128 window at +c8 (`nsys_ours_cN_dflt.sh`, reliable). vLLM under nsys is the known trap — the online api_server's +V1 EngineCore subprocess is NOT captured (empty rep at c8, both MP-on AND MP-off). FIX: vLLM +OFFLINE `LLM()` with `VLLM_ENABLE_V1_MULTIPROCESSING=0` (in-process → nsys captures) + cudaProfilerApi +fence + `--cuda-graph-trace=node`, fed the IDENTICAL 8 corpus prompts (all 1024 tok) → clean M=8 +batched decode. Kernel per-call times are frontend-independent, so vLLM-offline decode == vLLM-online +decode kernels. Steps segmented by lm_head anchor; steady window = modal-marlin-count steps; BOTH +engines' steady decode is at flash gridZ=64 (matched M=8). Coherence: vLLM offline greedy correct; +ours graph-on greedy "Paris…Rome" correct. + +PER-STEP DECODE (median over ~300–320 steady M=8 steps, µs), OURS EAGER (as-shipped) vs vLLM (graphed): +| class | ours µs/step (calls) | vLLM µs/step (calls) | Δ ours−vLLM | +|----------------|---------------------:|---------------------:|------------:| +| marlin W4A16 | 17,183 (180) | 16,286 (144) | +897 (+5.5%)| +| flash decode | 6,250 (36) | 5,629 (36) | +621 (+11%) | +| glue | 927 (291) | 671 (299) | +256 | +| lm_head | 5,413 (1) | 5,395 (1) | +18 | +| flash_comb | 130 (36) | 140 (36) | −10 | +| sample+other | 36 | 23 | +13 | +| **GPU BUSY** | **30,027** | **28,190** | **+1,837** | +| **GAP (idle)** | **1,184 (3.8%)** | **304 (1.1%)** | **+880** | +| **STEP SPAN** | **31,211** | **28,491** | **+2,720** | +Measured median TPOT ours 37.56 vs vLLM 34.58 ms = +2.98ms; GPU SPAN accounts +2.72ms (91%); the +~0.26ms residual = shared async frontend (consistent with #51 "host slice NOT engine-loop"). + +GAP LOCALIZATION (ours, per-step): a UNIFORM ~2.0–2.4µs GPU-idle between EVERY consecutive kernel +(marlin→glue 2.15, glue→glue 1.88, glue→marlin 2.15, flash→comb 2.37, …) + ~14 [CUDA memcpy H2D] and +~2 [memset] per step = host-launch-latency-bound EAGER execution. vLLM's gaps are ~0.4–0.76µs (glue→glue +0.76, glue→marlin 0.53) = piecewise-CUDA-graph REPLAY (node-to-node latency). Same GB10, so ours' 4× gap +is ours-specific, not a HW floor. + +ROOT-CAUSED IN SOURCE: the generic Qwen3-8B-MXFP4 (`Qwen3ForCausalLM` dense) decode runs EAGER by +default — `Qwen3DenseDecodeGraph` (`qwen3.cpp:DenseDecodeGraphForward`, shared decode CUDA-graph) is +OPT-IN via `VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH` (default OFF, pending a per-model SACRED gate) and has NO +quant restriction (only needs env-ON + pure_decode + CUDA). The binding c8=0.953 (#49/#51) was measured +EAGER. TESTED the opt-in (env=1, byte-identical-to-eager per source; greedy "Paris…Rome" coherent): +GAP 1,184→305µs (−880, = vLLM's, CONFIRMS the eager-launch mechanism) BUT BUSY 30,027→30,575µs (+547, +marlin 17,183→17,663 — bandwidth contention when kernels pack tight on the GB10 unified bus), so SPAN +31,211→30,877 (−334) and median TPOT 37.56→37.09ms (−1.3%). So the decode-graph lever is REAL but +self-limiting: it closes the +880µs gap yet nets only ~−334µs; c8 tput 0.953→~0.965, still <1.0x. + +FAIR graphed-vs-graphed residual (ours-graph vs vLLM, the apples-to-apples since vLLM is graphed): +SPAN +2,386µs = marlin +1,377 (58%) + flash +658 (28%) + glue +290 (12%) + gap ~0. So under a fair +comparison the DOMINANT term is MARLIN, and the gap term dissolves. + +VERDICT (a) — the dominant divergent term ≥1% is MARLIN, named with mechanism + the #50 settlement: +ours routes the 5 dense projections/layer through grouped-MoE-E1 marlin (`MoeGroupedGemmNvfp4Marlin`, +grid 144×1×1 = 3 waves at M=8, gate+up UNFUSED = **180 GEMM/step**) vs vLLM's dense `marlin_gemm` +(grid 48×1×1 = 1 wave, gate_up FUSED = **144/step**) → +5.5% (eager) / +8.5% (graphed) per step, driven +PRIMARILY by the 25% higher GEMM count (gate_up not merged; per-call is at near-parity since W4A16 is +memory-bound). **SETTLES the #50 contradiction:** the ubench "grouped==dense at parity once block=8" +compared vLLM's OWN moe_wna16 vs dense op at a SINGLE isolated shape — it could not see the production +structural divergence (OUR kernel, +36 GEMM/step, 3× CTAs). The residual is real at the per-step level. +FIX = grouped→dense-direct marlin + gate_up fusion (the #50-scoped ~2000-line delicate port); NOT +small/byte-safe → scoped for next dispatch. SECONDARY lever = decode-graph opt-in (+1.3%, byte-coherent, +needs the per-model SACRED token gate). TERTIARY = flash decode: SAME kernel + IDENTICAL grid (1,3,64) +BOTH engines (GQA-swap parity confirmed) yet +11% per-call (173.6 vs 156.3µs) — mechanism unresolved +(KV-cache read layout / splitkv scratch), small; and glue +256µs = ours' separate norm/silu/rope/qkv +launches vs vLLM's Inductor-fused triton (portable-fusion class, #46/#47). + +SHAs: worktree/main `4dd4e206` (#51) == box build `d3b412f5` (row pre-squash, content-identical). +Evidence dgx:~/mxfp4-nsys/ — reps ours_c8_dflt/ours_c8_graph/vllm_offline_c8.nsys-rep; tables +kern_sum_c8_dflt.txt, vllm_offline_kern_c8.txt, kern_sum_c8_graph.txt; trace CSVs +gpu_trace_c8_{dflt,graph}_cuda_gpu_trace.csv, vllm_offline_trace_c8_cuda_gpu_trace.csv; analyzers +analyze_decode.py + gap_and_shape.py; vLLM offline capture nsys_vllm_c8_v2.sh/run_vllm_offline_nsys.sh. + diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index a2117bdb..79366845 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | grouped→dense-direct-A marlin; host slice NOT engine-loop (VT_LOOP_TRACE interval≈step) | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | c8 same-tool diff (`QUANT-CT-MXFP4-C8-DIFF`): marlin grouped-5-GEMM DOMINANT (gate_up unfused); eager gap +0.88ms graph-closeable (decode-graph opt-in +1.3%); flash same-grid +11%; #50 = isolated-shape artifact | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/STATUS.md b/docs/STATUS.md index b8b23ddd..e91544c2 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); binding c1 0.99 / c2-c8 0.92-0.94 (#49 GQA-swap default-ON), mem 2.6x LESS, BELOW-FLOOR; dense-marlin port NO-GO (#50, cross-tool bias); Marlin-config + glue-fusion levers REFUTED by measurement. **Closer slivers LANDED byte-exact default-ON (`QUANT-CT-MXFP4-CLOSERS`, d3b412f5): dense M≤8 8-row Marlin tile + per-call ws re-zero dropped (block8≡block16 bitdiff=0 MXFP4+NVFP4, memcheck clean, #44 3/3). Binding x3: c1 1.005 PASSES, c2-c8 0.92-0.95 BELOW (best c8 0.953, +1.1pp vs #49), mem 2.18x WIN, gate NO; host slice NOT engine-loop (VT_LOOP_TRACE).** Detail in state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4-expert paths. CPU E8M0 dequant (`2^(byte-127)`, group 32, no global scale) unit-gated 5/5·1142 vs `dq_mxfp4_torch`. GPU W4A4 fp4 GEMM + MoE-expert e2e remain later bricks | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); dense-marlin NO-GO (#50); config+glue levers REFUTED. **Byte-exact closer slivers default-ON (`QUANT-CT-MXFP4-CLOSERS`, d3b412f5; #44 3/3). Binding x3: c1 1.005 PASSES, c2-c8 0.92-0.95 BELOW (best c8 0.953), mem 2.18x WIN, gate NO. c8 SAME-TOOL diff (`QUANT-CT-MXFP4-C8-DIFF`) SETTLES the residual: marlin grouped-5-GEMM (gate_up UNFUSED, 180 vs 144) DOMINANT (+0.9/+1.4ms step); eager launch-gap +0.88ms graph-closeable (`VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH` opt-in) nets +1.3% only; #50 grouped==dense = isolated-shape artifact.** Detail in state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4-expert paths. CPU E8M0 dequant (`2^(byte-127)`, group 32, no global scale) unit-gated 5/5·1142 vs `dq_mxfp4_torch`. GPU W4A4 fp4 GEMM + MoE-expert e2e remain later bricks | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index 515d920d..ffab14f6 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -321,7 +321,7 @@ def features_errors(text: str) -> list[str]: # Lowering these numbers as the page is compacted is the gate closing. STATUS = ROOT / "docs/STATUS.md" STATUS_RATCHET = { - "chars": 287_838, + "chars": 287_832, "h2_sections": 11, "long_paragraphs": 89, "oversized_cells": 47,