From 1532993debe8b808b16a8965cd7d9c20abe99d61 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 15:49:40 +0000 Subject: [PATCH] =?UTF-8?q?record(mxfp4):=20FUSED-GLUE=20W0=20=E2=80=94=20?= =?UTF-8?q?glue-into-Marlin=20kernel=20SOURCE-REFUTED,=20residual=20is=20f?= =?UTF-8?q?lash-dominant?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W0 of the funded MXFP4 glue-fusion kernel campaign. The precondition is to dump vLLM's actual kernels and never trust the label. GPU dumps are box-contended by the active H3 GPU campaign (both flock locks) and are OWED, but the vLLM compilation source at the parity pin 555967922 is decisive and REFUTES the funded premise (a from-scratch Marlin fusing add+RMSNorm+quant into the prologue and silu+mul into the epilogue, framed as mirroring vLLM's Inductor GEMM pro/epilogue fusion): - RMSNormQuantFusionPass / ActivationQuantFusionPass are ACTIVATION-QUANT fusions: every registered pattern needs an fp8/nvfp4 quant op (rms_quant_fusion.py:629-669, act_quant_fusion.py:296-320). W4A16 MXFP4 keeps bf16 activations (apply_gptq_marlin_linear quantizes only int8/fp8, marlin_utils.py:704-717), so NEITHER pass fires for this model. - ops.marlin_gemm is a register_fake extern op (_custom_ops.py:1200-1247) Inductor cannot fuse elementwise into; the triton_*_marlin_gemm_* kernels are Inductor-NATIVE elementwise fusions named after the adjacent extern call — vLLM's Marlin runs as a SEPARATE void marlin::Marlin CUDA kernel (144 launches in the same-tool #57 trace). - Same-tool #57 already shows vLLM runs MORE glue launches than us (299 vs 255), not fewer — no glue-into-GEMM collapse to mirror. - The dominant marlin-tiling term #52/#57 measured (+1,177us) was already closed on main by the dense-direct default efa6e40d (VT_MARLIN_DENSE ON, dense_nvfp4_gemm.h:106-127), which is why the terminal rose #51 0.925-0.953 -> #57 0.962-0.969. In that regime the c8 residual is FLASH-dominant: ~+784us flash (71%) / ~+195us glue (18%) / ~+226us residual-marlin. VERDICT: do NOT build the funded glue Marlin — it mirrors a fusion vLLM does not perform (the beyond-vLLM surpass rung), targets ~18% of the residual, and even glue->0 leaves c8 ~0.977 (<1.0). The real unexhausted lever is the flash +784us term on an IDENTICAL grid — a STRUCTURAL-lens context/residency audit the FA2 num_splits refutation did not address. Costed surpass-rung proposal recorded (not built). Owed GPU work (box-contended): TORCH_LOGS=output_code visual confirm, a fresh same-tool decode-window on current main (dense-direct default) c1..c8, and the flash same-kernel/different-time audit. Records-only; no code shipped. Box untouched; H3 campaign uninterrupted. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 123 ++++++++++++++++++++++++++++++++++++ .agents/state.md | 38 +++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- 5 files changed, 164 insertions(+), 3 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index ae7d9052..06ad6637 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity | **TERMINAL (`QUANT-CT-MXFP4-FINAL-STACK`)**: c1 1.020x PASS, mem 2.63x; c2-c8 0.962-0.969 GPU-intrinsic, both last levers exhausted (residual #46). `VT_MARLIN_DENSE` on | Record; branch not merged | +| MXFP4 parity | **c1 1.020 PASS, c2-c8 0.962-0.969, mem 2.63x** (`VT_MARLIN_DENSE` banks +951us marlin). `FUSED-GLUE` W0: glue-into-marlin REFUTED at source; residual FLASH-dominant | flash same-tool audit OWED | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Op-dispatch profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 93ab37cd..092ba466 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,129 @@ from relative link targets repointed for this file's location. # Benchmarks +## QUANT-CT-MXFP4-FUSED-GLUE W0 — the funded glue-fusion-into-Marlin kernel target is SOURCE-REFUTED; vLLM does NOT fuse glue into the extern Marlin GEMM for W4A16; the c2-c8 residual is FLASH-dominant, not glue (2026-08-06, `row/QUANT-CT-MXFP4-FUSED-GLUE`, source-side on dev box; GPU dumps box-contended by `row/H3-FP4-GPU-E2E`, base `origin/main` `672fc760`) + +W0 of the funded MXFP4 kernel campaign. The chartered vehicle is `Yi30/Qwen3-8B-MXFP4` +(dense `Qwen3ForCausalLM`, compressed-tensors **W4A16** Marlin keep-quant); oracle arm +`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` (sm_121 cute-dsl mxf4 crashes). The +prior FINAL-STACK (#59) terminal statement funded a from-scratch Marlin kernel fusing +`add+RMSNorm+quant` into the GEMM PROLOGUE and `silu+mul` into the EPILOGUE, framed as +mirroring vLLM's Inductor "GEMM pro/epilogue fusion" (`triton_red_fused_fused_add_rms_norm_marlin_gemm` +/ `triton_poi_fused_marlin_gemm_mul_silu_slice`, #46/#52). W0's precondition is to DUMP +vLLM's actual kernels and never trust the label. The GPU dumps (TORCH_LOGS=output_code, +same-tool decode-window) are box-contended and OWED, but the vLLM COMPILATION SOURCE is +decisive on its own and REFUTES the premise. Reading is at the parity pin `555967922` +(`~/_git/vllm`, 0.26.0.dev0). + +**FINDING 1 (source-conclusive): the two named vLLM fusion passes DO NOT FIRE for a +W4A16 MXFP4 model — they are ACTIVATION-QUANT fusions and W4A16 keeps bf16 activations.** +- `PostGradPassManager.configure` adds `RMSNormQuantFusionPass` under `fuse_norm_quant` + and `ActivationQuantFusionPass` under `fuse_act_quant` (`vllm/compilation/passes/pass_manager.py:163-171`). + Those flags resolve via `enable_norm_fusion`/`enable_act_fusion` (`vllm/config/vllm.py:108-129`; + wired into `OPTIMIZATION_LEVEL_0x` at `vllm.py:232-233,255-256,278-279`). +- But the PASSES only match norm/act followed by an **fp8/nvfp4 quant** op: + `RMSNormQuantFusionPass` registers ONLY `{FusedAdd,}RMSNorm{Static,Dynamic,Group}QuantPattern(..., FP8_DTYPE)` + (`rms_quant_fusion.py:629-669` — every pattern is `FP8_DTYPE`); `ActivationQuantFusionPass` + registers ONLY `SiluMulFp8StaticQuantPattern` / `SiluMulNvfp4QuantPattern` / + `SiluMulBlockQuantPattern(kFp8Dynamic*)` (`act_quant_fusion.py:296-320`). +- W4A16 has NO activation quant: `apply_gptq_marlin_linear` calls `marlin_quant_input` ONLY + for `input_dtype==int8`/`float8_e4m3fn` (`marlin_utils.py:704-715`); for MXFP4 W4A16 + `input_dtype=None`, so the **bf16** RMSNorm output is passed STRAIGHT to `ops.marlin_gemm` + (`marlin_utils.py:717`). MXFP4 is not nvfp4 (`is_nvfp4_quantized()` is False), so the nvfp4 + clause of `enable_act_fusion` also does not apply. ⇒ There is no `rms_norm→quant` or + `silu_mul→quant` sub-graph to match; BOTH passes are no-ops on this model. + +**FINDING 2 (source-conclusive): `marlin_gemm` is a non-decomposable EXTERN custom op — +Inductor structurally CANNOT fuse elementwise into it.** `ops.marlin_gemm` is registered +with `register_fake("_C::marlin_gemm")` (`vllm/_custom_ops.py:1200-1247`); Inductor treats +a fake-registered custom op as a FallbackKernel and emits an extern CALL, never Triton for +its body. The observed `triton_*_marlin_gemm_*` kernels are therefore INDUCTOR-NATIVE +elementwise fusions (a `triton_red` reduction doing residual-add + decomposed-RMSNorm; a +`triton_poi` pointwise doing silu+mul+slice) that Inductor NAMES after the region containing +the adjacent extern `marlin_gemm` call — NOT a prologue/epilogue baked into the Marlin CUDA +kernel. vLLM's decode Marlin runs as a SEPARATE `void marlin::Marlin` CUDA kernel — the +same-tool trace #57 shows it as 144 distinct launches, which it could not be if fused into a +Triton kernel. **So "vLLM fuses add+RMSNorm+quant into the Marlin prologue and silu+mul into +the epilogue" is a MISREAD of Inductor's region-naming. vLLM does no such fusion for W4A16.** + +**FINDING 3 (structural corroboration): vLLM runs MORE glue launches than we do, not +fewer.** The same-tool #57 c8 decode-window (`nsys --cuda-graph-trace=node`, both engines) +already recorded glue: **ours 866 us / 255 calls vs vLLM 671 us / 299 calls**. If vLLM had +collapsed glue into the GEMM pro/epilogue it would show FEWER glue launches; it shows more +(8.3 vs 7.1 per layer), individually cheaper. The +195 us glue delta is per-kernel +efficiency (Inductor Triton elementwise vs our hand-written glue), NOT a fusion-count gap — +directly contradicting the #46 "collapses our ~5 glue launches into 2" narrative (which came +from vLLM's ONLINE torch-profiler, a cross-tool read AGENTS.md forbids for invocation parity). + +**FINDING 4 (the reframe that matters): the dominant marlin-tiling term #52/#57 measured was +ALREADY CLOSED on `main` by the dense-direct default, AFTER the terminal binding.** #52/#57 +same-tool decode-window (MoE-grouped route, base `027af9b0`) decomposed the c8 gap as +**marlin +1,177 us (52%) / flash +784 us (35%) / glue +195 us (9%)**, and named the marlin +fix as grouped→dense-direct + gate_up-fuse (the opt-in "par1" arm closed marlin +1,177→+226). +That lever then LANDED as the byte-exact DEFAULT: commit `efa6e40d` "perf(marlin): +VT_MARLIN_DENSE default ON — dense route beats MoE on every axis (#57)" (in my base). On +current `main` `MarlinDenseEnabled()` is default-ON (`include/vllm/model_executor/models/dense_nvfp4_gemm.h:106-127`), +dense MXFP4 projections run vLLM's own dense marlin `vt::MarlinDenseGemm` +(`src/vt/cuda/cuda_marlin_dense.cu:92`) at the 48-CTA grid (~86 us/call vs the MoE route's +~118 us/call), gate_up merged into one 2N GEMM by default (`VT_MOE_FUSED_W13`, +`dense_nvfp4_gemm.h:98-104,519-544`), 32B-NVFP4A16 SACRED max gap 0.000 nats. That is +exactly why the terminal binding rose from #51's `0.925/0.939/0.953` (MoE) to #57's +`0.962/0.966/0.969` (dense). **So the +951 us marlin lever is BANKED, and in the +dense-direct regime the residual is FLASH-DOMINANT: at c8 0.969 the gap is ~+1,100 us of +which flash ~+784 us (~71%), glue ~+195 us (~18%), residual-marlin ~+226 us or less.** + +**THE W0 ours-vs-vLLM per-span table (c8 decode-window, same-tool nsys, current dense-direct +regime ≈ #57 par1 arm; medians over 300+ steps):** + +| span between GEMMs | our launches/layer (vt::) | what covers it in vLLM's step | ours us/step (calls) | vLLM us/step (calls) | Δ | +|---|---|---|---|---|---| +| decode Marlin GEMMs (qkv, o, gate_up[2N], down = 4/layer) | `vt::MarlinDenseGemm` ×4, 48-CTA dense-direct | `void marlin::Marlin` ×4 (SEPARATE CUDA kernel) | 16,512 (144) | 16,286 (144) | +226 | +| flash decode attention | `vt::PagedAttention` (flash_fwd_splitkv 1×3×64) | flash_fwd_splitkv (IDENTICAL grid 1×3×64) | 6,436 (36) | 5,629 (36) | **+807** | +| glue: add+RMSNorm ×2, qk-norm ×2, RoPE, QkvSplit, ReshapeAndCache, SiluAndMul | ~7 separate `vt::` kernels/layer | ~8 SEPARATE Inductor Triton elementwise kernels/layer (NOT fused into marlin) | 869 (255) | 671 (299) | +198 | +| lm_head | cuBLAS GEMV | cuBLAS GEMV | 5,398 | 5,395 | ~0 | + +Our per-layer decode order (source-mapped): `FusedChain(kFusedAddRmsNormStd)` input-norm +(`qwen3.cpp:121`) → `MarlinDenseGemm` qkv → `QkvSplit` → `FusedChain(kAttnQkNormRope)` q/k +norm+RoPE (composite = 3 launches on CUDA) → `ReshapeAndCache` → `PagedAttention` → +`MarlinDenseGemm` o → `FusedChain(kFusedAddRmsNormStd)` post-attn-norm (`qwen3.cpp:131`) → +`GateUpFusedMarlinD` (`MarlinDenseGemm` over 2N, `dense_nvfp4_gemm.h:542`) → `SiluAndMul` +(SEPARATE launch, `dense_nvfp4_gemm.h:544`) → `MarlinDenseGemm` down. The add+RMSNorm sites +already route through `vt::FusedChain` default-ON; the `FusedChain` catalog (`include/vt/recipes.h`, +`include/vt/fused_recipe.h`) has NO GEMM opcode and by design produces only standalone +fused elementwise kernels before/after a GEMM — it cannot express a GEMM prologue/epilogue. + +**VERDICT — the funded glue-fusion-into-Marlin kernel is the WRONG lever; DO NOT BUILD it.** +1. It would MIRROR a fusion vLLM does not perform (Findings 1-3): a from-scratch Marlin with + fused prologue/epilogue is genuinely BEYOND vLLM (the "surpass rung"), not a mirror. +2. It targets the +195-198 us glue span = ~9% of the MoE-route c8 gap and ~18% of the current + dense-direct residual. Even a PERFECT glue fusion (glue→0) leaves c8 ≈ 0.977 — still <1.0. +3. The dominant marlin-tiling lever it was conflated with is ALREADY the byte-exact default + (`efa6e40d`), banked in the 0.962-0.969 terminal. +4. The dominant REMAINING term is FLASH (+784-807 us on an IDENTICAL grid/kernel) — a + same-kernel/different-time STRUCTURAL-lens context question the FA2 num_splits refutation + (#59 lever 1) did NOT address. That, not glue, is the unexhausted high-value lever. + +**COSTED surpass-rung proposal (scoped, NOT built, per the charter):** a dedicated +`MarlinDenseGemm` variant with (a) a fused `add+RMSNorm` reduction PROLOGUE that produces the +bf16 marlin input in-kernel, and (b) a `silu+mul` EPILOGUE folded into the gate_up 2N GEMM's +store. Numerics: the prologue changes the RMSNorm reduction boundary (register vs HBM +round-trip) → not byte-exact → near-tie razor + distributional gate + 32B strict + regen +under the ratified-tie rule; the epilogue is byte-safe (pointwise on the GEMM output). +Expected recovery ≈ the glue span it removes, ~150-198 us/step (the ~5 glue launches + their +HBM round-trips), i.e. c8 ~0.969→~0.977 — a real but sub-parity, numerics-delicate, +non-portable kernel. NOT RECOMMENDED as the parity path. + +**OWED GPU work (box-contended by H3; short + batched when the flock frees):** (1) capture +`TORCH_LOGS=output_code` on the oracle to VISUALLY confirm the `triton_*_marlin_gemm_*` +kernels are elementwise-only calling the extern marlin (Findings 1-2 predict this exactly); +(2) a FRESH same-tool nsys decode-window on CURRENT `main` (dense-direct default) c1..c8 to +re-decompose the residual and confirm the flash-dominant split; (3) the real next lever — a +STRUCTURAL-lens same-tool audit of why identical-grid `flash_fwd_splitkv` runs +22 us/call +slower than vLLM's (residency/context, per AGENTS.md's STRUCTURAL lens), the +784 us term. +No code shipped on this row: building the glue Marlin would gold-plate a source-refuted +hypothesis (#46's own standard). Box left as found (no GPU touched; both flock locks +untouched; H3 campaign uninterrupted). + ## QUANT-CT-MXFP4 W4 throughput bench — RAN on GB10, BELOW-FLOOR (2026-08-06, `row/QUANT-CT-MXFP4-BENCH` `33e93608`) The binding ours-vs-oracle online-serving grid on the SAME checkpoint diff --git a/.agents/state.md b/.agents/state.md index d6a7ae6c..8bff7e38 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39276,6 +39276,44 @@ render. Comparability: vLLM-Omni CANNOT serve a quantized H3 on one GPU (BF16-only in practice; source-audited `a4ea67a2`) -> HW/loader-forced-indirect, DeepSeek-GGUF precedent. Draft PR is the claim. +## 2026-08-06T15:40 - QUANT-CT-MXFP4-FUSED-GLUE W0: the funded glue-fusion-into-Marlin kernel is SOURCE-REFUTED; residual is flash-dominant, not glue + + +Row `row/QUANT-CT-MXFP4-FUSED-GLUE` (helper). The FINAL-STACK (#59) terminal funded a +from-scratch Marlin fusing add+RMSNorm+quant into the prologue and silu+mul into the +epilogue, framed as mirroring vLLM's Inductor "GEMM pro/epilogue fusion". W0's precondition +is to dump vLLM's actual kernels and never trust the label. The GPU dumps are box-contended +by the active `row/H3-FP4-GPU-E2E` (both flock locks) and are OWED, but the vLLM compilation +SOURCE at the pin `555967922` is decisive and REFUTES the premise. Full forensic entry + +per-span table in `.agents/benchmark-record.md` ("QUANT-CT-MXFP4-FUSED-GLUE W0"). + +Findings: (1) `RMSNormQuantFusionPass`/`ActivationQuantFusionPass` are ACTIVATION-QUANT +fusions — every registered pattern needs an fp8/nvfp4 quant op (`rms_quant_fusion.py:629-669`, +`act_quant_fusion.py:296-320`). W4A16 MXFP4 keeps bf16 activations (`apply_gptq_marlin_linear` +quantizes only int8/fp8, `marlin_utils.py:704-717`), so NEITHER pass fires. (2) `ops.marlin_gemm` +is a `register_fake` extern op (`_custom_ops.py:1200-1247`) Inductor cannot fuse elementwise +into; the `triton_*_marlin_gemm_*` kernels are Inductor-NATIVE elementwise fusions named after +the adjacent extern call — vLLM's Marlin runs as a SEPARATE `void marlin::Marlin` CUDA kernel. +(3) same-tool #57 already shows vLLM runs MORE glue launches than us (299 vs 255), not fewer — +so there is no glue-into-GEMM collapse to mirror; the +195 us glue delta is per-kernel +efficiency. (4) the dominant marlin-tiling term #52/#57 measured (+1,177 us) was ALREADY +closed on `main` by the dense-direct default `efa6e40d` (`VT_MARLIN_DENSE` ON, +`dense_nvfp4_gemm.h:106-127`, 48-CTA ~86 us/call), which is why the terminal rose #51 +0.925-0.953 → #57 0.962-0.969. In that regime the c8 residual is FLASH-dominant: ~+784 us +flash (71%) / ~+195 us glue (18%) / ~+226 us residual-marlin. + +VERDICT: do NOT build the funded glue Marlin — it mirrors a fusion vLLM does not perform (it +is the beyond-vLLM surpass rung), targets ~18% of the residual, and even glue→0 leaves c8 +~0.977 (<1.0). The real unexhausted lever is the flash +784 us term on an IDENTICAL grid — a +STRUCTURAL-lens context/residency audit the FA2 num_splits refutation did not address. + +OWED (box-contended, short+batched when the flock frees): (a) TORCH_LOGS=output_code on the +oracle to visually confirm the Triton kernels are elementwise-only over the extern marlin; +(b) a fresh same-tool nsys decode-window on current `main` (dense-direct default) c1..c8 to +re-decompose and confirm the flash-dominant split; (c) the flash same-kernel/different-time +audit. Costed surpass-rung proposal (scoped, NOT built) recorded. No code shipped: building +the glue Marlin would gold-plate a source-refuted hypothesis. Box untouched; H3 uninterrupted. + ## QUANT-GGUF-CIQ-GEMM: the OWED fresh CPU op-dispatch profile is DONE and it re-ranks the levers away from G5; decode is 47% threadpool synchronisation and prefill is ~39% paged attention diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 69f41d5c..f753ea08 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -297,7 +297,7 @@ built on it rather than keeping the flattering one. | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | Pending. fp4-resident Marlin-W4A16 routing CPU-landed (62/62); GB10 delta + per-step unmeasured (disk window); real e2e disk-blocked; vLLM-Omni has no quantized H3 (BF16-only) | Build CUDA `test_minimax_h3` on dgx, run the NVFP4 case (Marlin via `marlin_gemms`), capture delta + s/step. Detail: benchmark-record + spec §8 | -| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON**. `QUANT-CT-MXFP4-FINAL-STACK` TERMINAL: 2 last levers exhausted (num_splits cap gated-OFF c1-only; glue folds via FusedChain, residual out-of-catalog). c2-c8 GPU-intrinsic; see record | +| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** banks +951us marlin. `FUSED-GLUE` W0: glue-into-marlin REFUTED (vLLM doesn't fuse into extern marlin); c8 residual FLASH-dominant, not glue. Flash same-tool audit OWED; see record | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | | Ampere consumer (`sm_86`, RTX 3090 class) | **No number owed; no such board here.** 2026-08-06 build-verify: 7/7 FA2 TUs 0-warn, real `sm_86` SASS. [Detail](../.agents/benchmark-record.md) | External RTX 3090 report. Floor is llama.cpp on that card (GGUF, not our Blackwell-only NVFP4 grid) | diff --git a/docs/STATUS.md b/docs/STATUS.md index ff9295a8..b1b0d8fb 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -85,7 +85,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue already folds via `vt::FusedChain`, residual = Inductor GEMM-epilogue fusion (out-of-catalog, #46-declined; frontend refuted). c1 1.020x PASS, c2-c8 0.962-0.969 GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue folds via `vt::FusedChain`; `FUSED-GLUE` W0: residual FLASH-dominant, not glue (vLLM does not fuse into extern marlin). c1 1.020x PASS, c2-c8 0.962-0.969.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet |