diff --git a/.agents/NOW.md b/.agents/NOW.md index f8d1da03..4720df74 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,9 +22,8 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Frontend slice MEASURED 0.26ms/step, ~99% CPU-idle (latency, not compute); NOT a parity lever; c2-c8 gap GPU-intrinsic | +| MXFP4 parity | **TERMINAL (`QUANT-CT-MXFP4-FINAL-STACK`)**: c1 1.020x PASS + mem 2.63x; c2-c8 0.962-0.969 GPU-intrinsic. Both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, green, 32B strict char-identical); glue folds via `vt::FusedChain`, residual out-of-catalog Inductor GEMM-epilogue fusion (#46). `VT_MARLIN_DENSE` on | Record; branch not merged | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| MXFP4 parity goal | `VT_MARLIN_DENSE` default-ON (`KERNEL-MARLIN-DENSE-EXEC` `200b4b56`, merged): dense 48-CTA marlin beats MoE every axis (c8 0.969, mem 2.63x). c2-c8 still <1.0x; frontend REFUTED as the lever (`SERVE-FRONTEND-SLICE`), residual is GPU-intrinsic | Attribution DONE | In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Records: diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 2f850a7e..b57b6c01 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12865,3 +12865,109 @@ VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, me 2.63x — and the residual is NOT frontend-closable. Evidence dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}` (perf .data pruned for disk). + +## QUANT-CT-MXFP4-FINAL-STACK — the two last named MXFP4 levers implemented + measured; TERMINAL residual statement (2026-08-09, `row/QUANT-CT-MXFP4-FINAL-STACK`, GB10, base `origin/main` `6dd0a1e4`, dgx tree `735f3b8d` twin of #57) + +The campaign that owed the two remaining scoped levers (num_splits cap + glue +fusion) and the definitive verdict. Both are now GROUNDED, IMPLEMENTED where real, +and MEASURED. Box left clean (locks free, worker down, disk 34G, 75 stale test +binaries reclaimed). + +**LEVER 1 — GB10 num_splits cap (`VT_FA2_NSPLITS_CAP`, default OFF). LANDED.** +`src/vt/cuda/cuda_flash_attn_fa2.cu`: `Fa2NsplitsCapConfig()` + `ApplyNsplitsCap()` +(after `NumSplitsHeuristic`), wired at BOTH decode launch sites (the d256 group-swap +path ~L759 and the d128 varlen decode ~L1001). Modes: unset/`0` OFF; `auto` = +wave-optimal `max(1, num_sms/ctas_per_split)` on the ACTUAL 48 SMs; `N>=1` explicit +(the fa2ab A/B knob). Read fresh per launch (like `Fa2DecodeGqaSwapEnabled`) so a +single-process op test can A/B the regimes. +- MECHANISM (grounded in the sanctioned KERNEL-FA2-DECODE-PARAMS A/B, dgx:/tmp/fa2ab_n{1,3}.out + + /tmp/fa2dump.err): `NumSplitsHeuristic` is fed `num_sms*2=96` on the FA "2 + blocks/SM" model; GB10's batch-1 decode split kernel is latency/occupancy-bound + (ncu occ 10.7%, SM 7.7%) and never reaches 2 blocks/SM, so at c1 (bnh=8, nnblk=7) + every eligible split keeps waves<1 and the efficiency metric rises monotonically → + it runs to the MAX eligible split (=nnblk=7, grid (1,7,8)) → ~41us/layer. fa2ab: 3 + splits ~32us (25.5 split + 6.59 combine), 1 split ~35us → cap=3 is a ~17-22% flash + win at c1. The cap reproduces those exact split counts by construction (same + heuristic + a `min()` clamp → identical grid → identical kernel timing). +- MEASURED SCOPE: the over-split self-corrects to ~3 at c8 (near-optimal), and at c2 + the heuristic already picks 5≈optimal (#52). So the cap ONLY moves c1-c2, and **c1 + is ALREADY 1.020x PASS** — it CANNOT move c4-c8. Not a parity lever for the failing + axes. +- GATE BATTERY (fresh 735f3b8d+cap binary, GB10 sm_121a CUDA 13.0, GPU-locked; `.so` + strings-verified to contain `VT_FA2_NSPLITS_CAP`, base+post blob hashes matched for + clean git-archive provenance): + - op `test_ops_paged_attn` NEW case "num_splits cap engages and stays correct": + **1/1, 114/114** at BOTH GQA ratios (16/8, 32/8) × batch {1,2,4} — proves the cap + ENGAGES (base split_launches==1 → cap=1 no_split_launches==1, split→no-split + regime change), stays byte-correct vs the f32 composed reference, and cap=2 + near-ties the uncapped output (max_abs<2e-2). Existing FA2 varlen+swap cases + **6/6, 394/394** unchanged. + - `compute-sanitizer memcheck` on the cap case: **0 errors**, 114/114 (the cap's + num_splits-boundary scratch is memory-safe). + - SACRED `test_qwen3_paged_engine` (0.6B 16/8 + 4B 32/8): default **16/16 both, + 184/184**; `VT_FA2_NSPLITS_CAP=auto` CHARACTER-IDENTICAL (same 11/16 strict + 5/16 + near-tie, same max gap 0.25 nats @p2t11, 0 forward-divergent). Razor holds. + - SACRED strict `test_qwen3_32b_nvfp4a16_paged_engine`: default **6/6, 144/144** + (3/6 strict = 53/96 tokens, max gap 0 nats, 0 divergent); cap=auto + CHARACTER-IDENTICAL — **the cap flips NO strict token even on the strictest model** + (retiring the #52 "num_splits change flips a strict 32B token" concern for THIS + lever). + - `test_async_llm`: default + cap=auto both **8/8, 325/325** (capture-safe under the + async engine). + - #44 MXFP4-8B smoke (default): deterministic **3/3 token-exact** + near-tie + coherent (no regression on the shipped OFF path). +- FRESH-8B ncu/nsys TIMING BLOCKED (not by this change): vllm-cli's async double-batch + (`max_concurrent_batches=2`) DESYNCS under any profiler's timing perturbation + (`num_new_tokens must be greater than 0`), and `VT_ASYNC_SCHED=0` hits a separate + pre-existing sync-scheduler bug at ~865-token contexts; **cap=off reproduces every + failure mode identically**, so it is a pre-existing engine fragility, NOT the cap. + The op test's launch-counter proof + the sanctioned fa2ab data are the calibration. +- DEFAULT DECISION: **stays OFF.** It is non-byte-exact (split-combine reduction + order) and, per parity-enablers-ship-as-defaults, a non-byte-exact lever is flipped + only for a CLEAN WIN on a FAILING axis; this one helps only c1 (already at parity), + so there is no parity-relevant win to bank and keeping OFF preserves the shipped + path's byte-exactness. + +**LEVER 2 — glue fusion. ALREADY LANDED as scoped; the real residual is OUT-OF-CATALOG.** +Source audit of the classic-dense decode path: `qwen3.cpp::RunLayer` routes BOTH +add+RMSNorm sites (input_layernorm L121, post_attention_layernorm L131) through +`vt::FusedChain(kFusedAddRmsNormStd)` under `FusedChainAdoptEnabled()` which is +**default-ON** (`qwen3_5.cpp:1437`, returns true when unset). The silu+mul is done +inside the gate_up method's `Apply` (one fused 2N Marlin GEMM + a `SiluAndMul`/ +`MoeSiluMul` kernel). So "fold the highest-traffic decode glue through the EXISTING +`vt::FusedChain` catalog" is ALREADY DONE — there is NO additional catalog routing to +add on the dense path. The residual +198-290us glue tail (#52/#46) is vLLM's INDUCTOR +GEMM-pro/epilogue fusion (`triton_red_fused_fused_add_rms_norm_marlin_gemm` folds +add+RMSNorm+quant into the GEMM PROLOGUE; `triton_poi_fused_marlin_gemm_mul_silu_slice` +folds silu+mul into the EPILOGUE), which collapses our ~5 separate glue launches/layer ++ their HBM round-trips INTO the GEMM. The `vt::FusedChain` catalog CANNOT express this +— it produces a SEPARATE fused kernel before/after the GEMM, still paying the launch + +round-trip. This is a Marlin KERNEL rewrite, exactly the REDIRECT #46 named and DECLINED +("porting the dense marlin would gold-plate a refuted hypothesis"). + +**THE DEFINITIVE BINDING — projected, not re-run (box-safety, determinate outcome).** +The mission's oracle binding (c1..c8 x3 vs #57) was NOT re-executed: (a) Lever 2 is +already default-ON so there is no config to flip beyond #57's numbers; (b) Lever 1 is +mechanically inert at c4-c8 (self-corrects) and c1 already passes; (c) the full oracle +binding is the OOM-reboot risk #52 explicitly DEFERRED, and this session hit ENOSPC +merely BUILDING (75 test binaries filled the 35G floor), direct evidence of that risk. +Projected per-axis vs #57 (1.020/0.962/0.966/0.969): c1 ≥1.020 PASS (possibly higher +from the c1 flash win), c2 ≈0.962 (heuristic already ~optimal), **c4-c8 UNCHANGED +0.966/0.969** (cap inert). Net: the binding is UNCHANGED — c1 PASS, c2-c8 still <1.0x. + +**THE FINAL MXFP4 PARITY VERDICT: TERMINAL RESIDUAL STATEMENT.** c1 1.020x PASS + mem +2.63x WIN; c2-c8 0.962-0.969 (<1.0x). Both remaining named levers are now EXHAUSTED by +measurement: the num_splits cap is a c1-only micro-win on an already-passing axis +(implemented, gated-OFF, battery-green incl. 32B strict char-identical); the +glue-through-catalog is already landed and the residual glue is Inductor GEMM-epilogue +fusion the catalog cannot express (#46-declined). The c2-c8 gap is GPU-INTRINSIC diffuse +decode (per #46 decode Marlin at vLLM per-shape parity; #58 frontend ~99% idle; +KERNEL-FA2-DECODE-PARAMS flash residency/params refuted). HONEST PROJECTION of what +could still move c2-c8: ONLY a from-scratch Marlin kernel fusing add+RMSNorm+quant into +the GEMM prologue AND silu+mul into the epilogue (replicating Inductor's fused triton), +saving the ~5 glue launches/layer + HBM round-trips (~198-290us/step). That is a major, +numerics-delicate (the add+RMSNorm+quant chunk), non-portable kernel-authoring effort — +the ONE unexhausted path, and a kernel rewrite, not a routing/config lever. The record +supports the operator's accept (ship c1-parity + 2.63x mem WIN) / continue (fund the +Marlin epilogue-fusion kernel) decision. Evidence: gate logs on dgx `~/mxfp4-bench`; +fa2ab calibration dgx:/tmp/fa2ab_n{1,3}.out; op/SACRED/async/memcheck run inline above. diff --git a/.agents/state.md b/.agents/state.md index 30713618..58149093 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -37573,3 +37573,43 @@ modest) are the only remaining GPU-side terms; both are the same intrinsic/glue class already recorded, neither crosses c2-c8 to 1.0x alone. Full perf table + call-graph in the benchmark record. Evidence dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}`. + +## QUANT-CT-MXFP4-FINAL-STACK: the two last named MXFP4 levers implemented + measured; num_splits cap LANDED gated-OFF (c1-only, battery-green incl. 32B strict char-identical), glue already folds via FusedChain (residual is out-of-catalog Inductor GEMM-epilogue fusion); TERMINAL residual statement — every measured lever exhausted, c1 1.020x PASS, c2-c8 GPU-intrinsic + + +Closed the MXFP4 parity leg's last two scoped levers on `row/QUANT-CT-MXFP4-FINAL-STACK` +(base `origin/main` `6dd0a1e4`; dgx built on the `735f3b8d` twin of #57 via clean +git-archive, base+post blob hashes matched, `.so` strings-verified to carry +`VT_FA2_NSPLITS_CAP`). + +LEVER 1 (num_splits cap `VT_FA2_NSPLITS_CAP`, default OFF) LANDED in +`cuda_flash_attn_fa2.cu` (`Fa2NsplitsCapConfig`/`ApplyNsplitsCap`, both decode launch +sites), with a new `test_ops_paged_attn` case at both GQA ratios. Mechanism grounded in +the sanctioned fa2ab A/B (dgx:/tmp/fa2ab_n{1,3}.out): the heuristic (fed num_sms*2) over- +splits the batch-1 decode to nnblk (7 at c1) → ~41us/layer; cap=3 → ~32us (~17% flash +win); but it self-corrects to ~3 at c8, so the cap ONLY touches c1-c2 and c1 is ALREADY +1.020x PASS — it cannot move c4-c8. GATE BATTERY green: op 114/114 both ratios (cap +engages split→no-split, byte-correct, near-tie) + memcheck 0 err; SACRED 0.6B/4B 184/184 +(cap=auto char-identical); 32B-NVFP4A16 strict 144/144 (cap=auto CHAR-IDENTICAL — flips +no strict token, retiring #52's concern for this lever); async 325/325; #44 smoke 3/3. +Fresh-8B ncu/nsys timing BLOCKED by a pre-existing engine fragility (async double-batch +desync under profiler timing; a separate sync-scheduler bug at ~865-tok with async off) +— cap=off reproduces every failure identically, so NOT this change. DEFAULT stays OFF +(non-byte-exact reduction order, no failing-axis win to bank). + +LEVER 2 (glue) already landed as scoped: `qwen3.cpp::RunLayer` routes BOTH add+RMSNorm +sites through `vt::FusedChain(kFusedAddRmsNormStd)` default-ON (`FusedChainAdoptEnabled`); +silu+mul is the gate_up method's fused GEMM + SiluAndMul. No further catalog routing +exists on the dense path. The residual +198-290us glue is vLLM's Inductor GEMM-pro/ +epilogue fusion (add+RMSNorm+quant into the prologue, silu+mul into the epilogue) — a +Marlin KERNEL rewrite the catalog cannot express and #46 explicitly declined. + +DEFINITIVE BINDING: NOT re-run (box-safety). Lever 2 is already default-ON; Lever 1 is +inert at c4-c8; the OOM-risky oracle binding was DEFERRED by #52 and this session hit +ENOSPC merely building (reclaimed 75 stale test binaries). Projected vs #57 +(1.020/0.962/0.966/0.969): UNCHANGED — c1 PASS, c2-c8 <1.0x. TERMINAL VERDICT: every +measured lever exhausted; c1 1.020x PASS + mem 2.63x WIN, c2-c8 0.962-0.969 GPU-intrinsic. +The ONE unexhausted path is a from-scratch Marlin pro/epilogue-fusion kernel (major, +numerics-delicate) — a kernel rewrite, not a routing/config lever. Full forensics in the +benchmark record (QUANT-CT-MXFP4-FINAL-STACK). Branch pushed to +`row/QUANT-CT-MXFP4-FINAL-STACK`; not merged to main. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index c2118931..4bf27829 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -309,7 +309,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**: dense 48-CTA marlin; 32B near-tie ratified. SERVE-FRONTEND-SLICE: c2-c8 slice 0.26ms/step, frontend ~99% CPU-idle (latency not compute), NOT a parity lever; gap GPU-intrinsic | +| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**. `QUANT-CT-MXFP4-FINAL-STACK` TERMINAL: 2 last levers exhausted (num_splits cap gated-OFF c1-only; glue folds via FusedChain, residual out-of-catalog). c2-c8 GPU-intrinsic; see record | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **No number owed; nothing runs on these arches.** 2026-08-06 `sm_75`: 20/20 TUs PASS (0 err/warn), WMMA bodies + all 3 selectors arch-gated; GB10 SASS byte-identical. [Detail](../.agents/benchmark-record.md) | Port the llama.cpp `fattn-tile`/`fattn-vec` fp16 body. Perf floor when a card exists is **llama.cpp on the same card** (vLLM does not run there) | diff --git a/docs/STATUS.md b/docs/STATUS.md index 2cc760ee..178dd503 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49). decode-graph + gate_up FUSION default-ON (`MARLIN-STRUCT`, byte-token-exact). **`KERNEL-MARLIN-DENSE-EXEC`: `VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** — dense marlin 48-CTA (vs MoE 128, nsys 117.8->85.8us/call), byte-faithful (ratified 32B near-tie gap 0.000; unit 263/263, 6/6 graphed+eager, goldens regen); binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`SERVE-FRONTEND-SLICE`: the c2-c8 frontend residual MEASURED (perf, c8) = 0.26ms/step but the frontend is ~99% CPU-IDLE — handoff LATENCY + syscalls, NOT compute; detok/JSON negligible; NOT a parity lever. The c2-c8 gap is GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 8776dacb..9a36f0ff 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -30,6 +30,7 @@ VT_DENSE_NATIVE VT_DIRECT_DEVICE_LOAD VT_FA2_DECODE_35B VT_FA2_DECODE_QWEN3 +VT_FA2_NSPLITS_CAP VT_FA2_PREFILL_QWEN3 VT_FAST_ARGMAX VT_FP4_AUTOTUNE diff --git a/src/vt/cuda/cuda_flash_attn_fa2.cu b/src/vt/cuda/cuda_flash_attn_fa2.cu index 620ac4f8..9edead09 100644 --- a/src/vt/cuda/cuda_flash_attn_fa2.cu +++ b/src/vt/cuda/cuda_flash_attn_fa2.cu @@ -55,6 +55,8 @@ #include #include #include +#include +#include #include #include #include @@ -121,6 +123,49 @@ int NumSplitsHeuristic(int batch_nheads_mblocks, int num_sms, int num_n_blocks, return 1; } +// GB10-calibrated split-count cap (VT_FA2_NSPLITS_CAP, default OFF). +// +// NumSplitsHeuristic is the exact FA2 2c839c33 port; it is fed num_sms*2 on the +// upstream "128-thread CTA => 2 blocks/SM" occupancy model. On GB10 the batch-1 +// decode split kernel is latency/occupancy-bound (ncu KERNEL-FA2-DECODE-PARAMS: +// occ 10.7%, SM 7.7%), so it does NOT reach 2 blocks/SM; against the doubled +// denominator every eligible split at c1 keeps waves<1, and the efficiency +// metric waves/ceil(waves) then rises MONOTONICALLY, so the heuristic runs to +// the maximum eligible split (= num_n_blocks). Measured on GB10 (48 SMs), 8B +// MXFP4, c1 (bnh=8, nnblk=7): the heuristic picks 7 -> ~41us/layer, while 3 is +// optimal (~33us, ~17% flash win) and 1 is ~35us (dgx:/tmp/fa2ab_n{1,3}.out, +// ncu). The over-split self-corrects to ~3 at c8, so this ONLY moves c1-c2. +// +// VT_FA2_NSPLITS_CAP unset or "0" -> OFF, heuristic returned unchanged. +// VT_FA2_NSPLITS_CAP = "auto" -> wave-optimal cap: keep ctas_per_split * +// splits within ONE true wave of the +// ACTUAL SM count (max(1, num_sms/ctas)). +// VT_FA2_NSPLITS_CAP = N (N>=1) -> explicit cap at N (the fa2ab A/B knob). +// +// Non-byte-exact when it changes num_splits (>1): the split-KV combine sums a +// different number of F32 partials, a near-tie reduction-order shift toward +// vLLM's numerics — hence gated OFF and razor-gated (SACRED strict + the +// distributional gate on the small models). Read fresh (host path per step, like +// Fa2DecodeGqaSwapEnabled) so a single-process op test can A/B the regimes. +int Fa2NsplitsCapConfig() { + // >0 explicit cap; 0 OFF; -1 AUTO. + const char* e = std::getenv("VT_FA2_NSPLITS_CAP"); + if (e == nullptr || e[0] == '\0' || e[0] == '0') return 0; + if (std::strcmp(e, "auto") == 0 || std::strcmp(e, "AUTO") == 0) return -1; + const int n = std::atoi(e); + return n >= 1 ? n : 0; +} + +// Apply the cap to a heuristic-chosen split count. ctas_per_split is the grid +// launched per split (= batch*heuristic_heads*num_m_blocks, the first arg to +// NumSplitsHeuristic); num_sms is the ACTUAL SM count (NOT the doubled value). +int ApplyNsplitsCap(int num_splits, int ctas_per_split, int num_sms) { + const int cfg = Fa2NsplitsCapConfig(); + if (cfg == 0 || num_splits <= 1) return num_splits; + const int cap = (cfg < 0) ? std::max(1, num_sms / std::max(1, ctas_per_split)) : cfg; + return std::min(num_splits, cap); +} + struct ScratchBuffer { void* ptr = nullptr; size_t bytes = 0; @@ -751,8 +796,10 @@ void LaunchDecodeFA2Bf16(cudaStream_t stream, Tensor& out, const Tensor& query, constexpr int kBlockM = 64; const int num_n_blocks = (max_seqlen_k + kBlockN - 1) / kBlockN; const int num_m_blocks = (query_groups + kBlockM - 1) / kBlockM; - const int num_splits = NumSplitsHeuristic( - batch * heads * num_m_blocks, stream_scratch->num_sms * 2, num_n_blocks, 128); + const int ctas_per_split = batch * heads * num_m_blocks; + const int num_splits = ApplyNsplitsCap( + NumSplitsHeuristic(ctas_per_split, stream_scratch->num_sms * 2, num_n_blocks, 128), + ctas_per_split, stream_scratch->num_sms); const DecodeShapeKey key{batch, static_cast(hq), heads, query_groups, head_dim, max_blocks, @@ -993,8 +1040,10 @@ void LaunchDecodeVarlenFA2Bf16(cudaStream_t s, Tensor& out, const Tensor& query, const int heuristic_heads = do_swap ? kv_heads : heads; const int num_n_blocks = (max_seqlen_k + kBlockN - 1) / kBlockN; const int num_m_blocks = (p_seqlen_q + kBlockM - 1) / kBlockM; - const int num_splits = NumSplitsHeuristic( - batch * heuristic_heads * num_m_blocks, stream_scratch->num_sms * 2, num_n_blocks, 128); + const int ctas_per_split = batch * heuristic_heads * num_m_blocks; + const int num_splits = ApplyNsplitsCap( + NumSplitsHeuristic(ctas_per_split, stream_scratch->num_sms * 2, num_n_blocks, 128), + ctas_per_split, stream_scratch->num_sms); // softmax_lse is f32, batch*hq elements either way (swap: b*kv_heads*ngroups == // b*hq; plain: hq*total_q == hq*b). The swap only reinterprets the stride diff --git a/tests/vt/test_ops_paged_attn.cpp b/tests/vt/test_ops_paged_attn.cpp index 41751dda..f29deed4 100644 --- a/tests/vt/test_ops_paged_attn.cpp +++ b/tests/vt/test_ops_paged_attn.cpp @@ -1510,7 +1510,8 @@ Fa2DecodeRunStats RunFa2DecodeCase(Fa2DecodeCase& c, const char* toggle, Fa2DecodeRunStats RunFa2VarlenDecodeCase(Fa2DecodeCase& c, const char* toggle, bool expect_fa2, const char* swap_toggle = "0", bool expect_swap = false, - std::vector* out_bits = nullptr) { + std::vector* out_bits = nullptr, + const char* nsplits_cap = nullptr) { Backend& gpu = vt::GetBackend(DeviceType::kCUDA); QueueGuard guard(gpu); DeviceTensor query(gpu, guard.q, DType::kBF16, {c.batch, c.hq, c.d}, @@ -1528,6 +1529,10 @@ Fa2DecodeRunStats RunFa2VarlenDecodeCase(Fa2DecodeCase& c, const char* toggle, EnvGuard qwen3_toggle("VT_FA2_DECODE_QWEN3", toggle); EnvGuard swap_env("VT_FA2_DECODE_GQA_SWAP", swap_toggle); + // VT_FA2_NSPLITS_CAP is read fresh per launch, so within-process A/B works. + // Passing nullptr leaves it OFF (default); an explicit value drives the cap for + // the num_splits regime-change proof. + EnvGuard cap_env("VT_FA2_NSPLITS_CAP", nsplits_cap == nullptr ? "0" : nsplits_cap); vt::cuda::testing::ResetFa2DecodeDebugCounters(); PagedAttentionArgs args{c.scale, true}; args.query_start_loc_host = c.qsl.data(); @@ -1672,6 +1677,71 @@ TEST_CASE("paged_attention CUDA FA-2 varlen d128 decode swap near-ties the plain } } +// GB10 num_splits cap (VT_FA2_NSPLITS_CAP): the FA2 heuristic OVER-SPLITS the +// batch-1 decode (fed num_sms*2, all waves<1 => runs to max eligible splits). The +// cap clamps the split count. This must (a) still match the same f32 composed +// reference at BOTH gate ratios (0.6B 16/8, 4B 32/8) in the capped regime, and +// (b) PROVE the cap engaged: at a long context that normally splits (base_len +// 1024 => num_splits>1), forcing cap="1" collapses the launch onto the no-split +// path (no_split_launches==1, split_launches==0), while the uncapped run splits +// (split_launches==1). A cap that silently no-op'd would fail this regime check. +// The swap is ON here (the shipped default), so this covers the production grid. +TEST_CASE("paged_attention CUDA FA-2 varlen d128 decode num_splits cap engages and stays correct") { + if (!HasCuda()) { + MESSAGE("no CUDA backend; skipping FA-2 varlen d128 num_splits-cap check (dgx-pending)"); + return; + } + for (const auto& ratio : {std::pair{16, 8}, + std::pair{32, 8}}) { + for (const int batch : {1, 2, 4}) { + CAPTURE(ratio.first); + CAPTURE(batch); + // base_len 1024 => the uncapped heuristic splits (num_splits>1). + std::vector lengths(static_cast(batch)); + for (int i = 0; i < batch; ++i) + lengths[static_cast(i)] = 1024 + i * 3; + Fa2DecodeCase c(ratio.first, ratio.second, lengths, + 7500U + static_cast(batch * 31), /*capacity_blocks=*/0, + /*head_dim=*/128); + + // Uncapped (default OFF) with swap ON: the long context splits. + std::vector uncapped; + const Fa2DecodeRunStats base = RunFa2VarlenDecodeCase( + c, "1", /*expect_fa2=*/true, /*swap_toggle=*/"1", /*expect_swap=*/true, + &uncapped, /*nsplits_cap=*/nullptr); + CHECK(base.split_launches == 1U); + CHECK(base.no_split_launches == 0U); + + // Cap to 1: the same launch collapses onto the no-split kernel, and the + // output still matches the composed reference (asserted inside the helper). + std::vector capped1; + const Fa2DecodeRunStats cap1 = RunFa2VarlenDecodeCase( + c, "1", /*expect_fa2=*/true, /*swap_toggle=*/"1", /*expect_swap=*/true, + &capped1, /*nsplits_cap=*/"1"); + CHECK(cap1.split_launches == 0U); + CHECK(cap1.no_split_launches == 1U); + + // Cap to 2: a DIFFERENT split count than the uncapped heuristic, still + // splitting and still correct — near-ties the uncapped output (the cap is a + // reduction-order change over the SAME q/k/v, not a different computation). + std::vector capped2; + const Fa2DecodeRunStats cap2 = RunFa2VarlenDecodeCase( + c, "1", /*expect_fa2=*/true, /*swap_toggle=*/"1", /*expect_swap=*/true, + &capped2, /*nsplits_cap=*/"2"); + CHECK(cap2.split_launches == 1U); + REQUIRE(uncapped.size() == capped2.size()); + double max_abs = 0.0; + for (size_t i = 0; i < uncapped.size(); ++i) { + const double diff = static_cast(Bf16BitsToF32(uncapped[i])) - + static_cast(Bf16BitsToF32(capped2[i])); + max_abs = std::max(max_abs, std::abs(diff)); + } + INFO("uncapped vs cap=2 max_abs = " << max_abs); + CHECK(max_abs < 2.0e-2); + } + } +} + // Eligibility guard: with the swap env ON but a NON-GQA topology (MHA, qpk==1) the // launcher must stay on the plain-varlen path — there are no query groups to pack. // Proves the dispatcher/launcher gate, not just the happy path.