diff --git a/.agents/NOW.md b/.agents/NOW.md index c0763a8ab..7871368f5 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -1,6 +1,6 @@ # NOW — the one-Read resume surface - + Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live, the gate being chased, what to do next. Never a log — evidence lives in the @@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity (Qwen3-8B) | c1 0.990/TTFT parity/mem 2.6x WIN; c2-c8 0.92-0.94 post-GQA-swap (#49). Dense-marlin port arbitrated **NO-GO** (#50: 4-run ubench, +7-9% was cross-tool bias; only real sliver = c8 block-16 padding ~0.33ms) | Closers: c8 block=8 tweak (~5 lines) + ws-memset drop + the ~0.7ms/step host/sched slice | +| MXFP4 parity (Qwen3-8B) | **Closers binding x3 (`QUANT-CT-MXFP4-CLOSERS`): c1 1.005 PASSES, c2-c8 0.925/0.939/0.953 BELOW (c8 +1.1pp vs #49), mem 2.18x WIN, gate NO. Byte-exact bitdiff=0, #44 3/3** | Residual: grouped→dense-direct-A; host slice NOT engine-loop | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| MXFP4 parity goal | swap-ON binding **still <1.0x** (best c8 0.942); residual grouped-Marlin +7-9% + ~0.7ms host/sched | NEXT: grouped→dense-direct-A marlin decode | +| MXFP4 parity goal | binding <1.0x on c2-c8 (best c8 0.953); c1 crosses to parity+. Residual = grouped-Marlin +7-9% (only real lever) + host slice (VT_LOOP_TRACE: NOT engine-loop) | NEXT: grouped→dense-direct-A E1 marlin | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index b62a8b2a2..fdf4dee61 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12554,3 +12554,64 @@ launch/GEMM, micro. (c) the **~0.7ms/step host/sched slice** — the real remain not a kernel port. No single lever reaches ≥1.0x; the MXFP4 parity goal stays BELOW-FLOOR (best c8 0.942). Evidence: `dgx:~/mxfp4-nsys/{mxfp4_marlin_ubench.py,ubench.log,ubench_3x.log}`. Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). + +## QUANT-CT-MXFP4-CLOSERS: slivers (a)+(b) land BYTE-EXACT + default-ON; correctness gates GREEN, binding re-bench PENDING (2026-08-09, `row/QUANT-CT-MXFP4-CLOSERS`, GB10 sm_121a CUDA 13.0, `~/mxfp4-bench` overlay `1f446fd7`+swap-ON+3 files md5-matched) +The two #50-arbitrated slivers, both in the SHARED header +`include/vllm/model_executor/models/dense_nvfp4_gemm.h` (the qwen3_5.cpp twin serving the 27B/35B +gate models is deliberately untouched): **(a)** `DenseAlignFor` forces `block=8` at M≤8 (:286), +routing the M=8 dense E1 GEMM to vLLM's 8-row `m_block_size_8` tile instead of the padded 16-row +tile (recovers the #50-measured ~0.33ms/step / ~0.8pp at c8); **(b)** `DenseMarlinWorkspace` zeroes +the shared reduction workspace ONCE at alloc and the two per-call `Memset(ws)` are DROPPED — the +fp32-reduce marlin barrier self-resets its locks (invariant cited: `use_atomic_add=false` +cuda_moe_marlin.cu:141 ⇒ only the fp32 barrier path is reachable, whose last release zeroes the lock +marlin_template.h:2170→:204; slice_count==1 never touches locks :2162; the atomic-add non-clearing +path :614 is dead). GATES (both flock locks, ninja EXIT 0 no -Werror): OP RED-first +`test_ops_moe_grouped` closers — block8-vs-block16 A/B at M=8 **BYTE-EXACT** (`bitdiff=0/32768 +max_abs=0`) on MXFP4 K=4096/N4096 + K=12288/N4096 AND NVFP4 K=4096/N4096; ws all-zero after a GEMM + +reuse bit-identical; 15/15·2 cases. LAUNCH-CONFIG `test_qwen3_forward` `DenseAlignFor(d,8).block==8` +(pre-fix ==16) 7/7. MEMCHECK 0 real errors (leaks = pre-existing static-cache harness artifacts; the +unchanged prior test leaks MORE). #44 smoke (Yi30/Qwen3-8B-MXFP4, default async graphed, vllm-cli +greedy vs golden) **3/3 deterministic TOKEN-EXACT** + near-tie coherent. Byte-exact ⇒ DEFAULT-ON +unconditional (no `VT_*` gate). Blast radius (header consumers Qwen3-8B-MXFP4, Qwen3-32B-NVFP4A16, +Laguna) closed by proving BOTH quant schemes byte-exact. Binding c1-c8 x3 (clean-checkout grid) + the +substantive ~0.7ms/step host/sched slice (the #47 residual) remain the parity verdict's open terms. +Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). + +## QUANT-CT-MXFP4-CLOSERS BINDING: clean-checkout grid at d3b412f5 — slivers improve EVERY throughput axis vs #49 (c1 crosses to parity+); MXFP4 goal still <1.0x on c2-c8 (2026-08-09, GB10, vLLM oracle 0.25.0, evidence `dgx:~/work/vllm.cpp-online-gate/evidence/d3b412f5c191aace1f2960fa7940d8eef925762a`) +Full c1/c2/c4/c8 x3 binding on the CLEAN-checkout build at the committed sha d3b412f5 (not an +overlay), oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, single-load/arm, +drop_caches+mincore, RelWithDebInfo+oracle-cutlass. #44 model gate re-passed inside the grid +(mxfp4_smoke_battery). 24/24 legs `failed:0`; reps tight (ours c8 [1481.7,1475.1,1486.7] tok/s, +CoV ~0.4%; vllm c8 CoV ~0.06%). The grid EXIT=1 is the gate-FAIL signal (`gate_pass:false`) + the +single-model "cross-model summary waits for the other model", NOT a crash — all q3mxfp4 artifacts +are complete. (q3mxfp4's sweep IS c1-c8, matching #48/#49; c16/c32 are other-model points.) + +| axis (ours/vLLM normalized ratio) | c1 | c2 | c4 | c8 | vs #49 (tput) | +|---|---|---|---|---|---| +| total_token_throughput | **1.005** | 0.925 | 0.939 | **0.953** | +0.015/+0.003/+0.009/+0.011 | +| median_tpot_ms | 1.002 | 0.922 | 0.915 | 0.939 | — | +| mean_ttft_ms | 1.034 | 0.962 | 1.004 | 0.999 | — | +| median_itl_ms | 1.004 | 0.919 | 0.920 | 0.929 | — | + +Peak host-mem footprint (peak_mem_available_drop): ours **35.2 GiB** vs vLLM **76.7 GiB** = **2.18x +LESS** (WIN). VERDICT: **c1 PASSES every axis (tput 1.005, tpot 1.002, ttft 1.034, itl 1.004)**; +c2-c8 BELOW on tput/tpot/itl (best c8 0.953 tput), TTFT at parity c4/c8 (1.004/0.999). gate_pass +FALSE => MXFP4 parity goal **NOT DONE** (below-floor on c2-c8). The slivers moved EVERY throughput +axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), exactly the +byte-exact block=8 recovery (~0.8pp @c8, #50) plus the memset-drop; nothing regressed. + +RESIDUAL MAP (per-shape, c8 ~4.75% tput gap): (1) block=16 padding — **CLOSED this run** (sliver a). +(2) grouped-Marlin decode **+7-9% per-call** (GPU): `MoeGroupedGemmNvfp4Marlin` E=1 indirect +`sorted_token_ids` gather + fp32 `C_tmp` vs vLLM dense `marlin_gemm` direct-A (#46/#50; a delicate +grouped->dense-direct-A port, per-shape parity at M<=8 so not a config lever). (3) the host slice. + +HOST-SLICE ATTRIBUTION (step 2, VT_LOOP_TRACE on our server under decode load, this session): in +every 1 s window the engine-core loop shows **interval_ms ≈ step_ms** (mean 25.5 vs 25.5, delta +<=0.02 ms; `admits=0` pure-decode windows show interval-step ≈ 0). So the born-on-runner +engine-core decode loop (scheduler + drain + admit) carries **negligible** per-iteration host +overhead — the #47 ~0.7ms/step "host/sched" residual is NOT in the engine loop; it lives in the +shared-architecture async frontend (HTTP / output-processing / detokenize, which vLLM runs too) or +within the cross-tool attribution boundary (~0.7ms is ~24% of the c2 gap, near measurement error). +Not a born-on-runner lever. (Caveat: the curl load under-saturated vs steady c8, so step_ms 25.5 +is not the c8 TPOT 37.6; the interval≈step finding is batch-independent and robust.) +Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). diff --git a/.agents/state.md b/.agents/state.md index 110531a7b..c4edbfc5d 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -37070,3 +37070,113 @@ grouped-Marlin decode **+7-9% per-call** — `MoeGroupedGemmNvfp4Marlin` E=1 ind at M≤8 per #46 → a delicate grouped→dense-direct-A port, not a config knob); (2) **~0.7ms/step host/sched**. No single lever ≥1.0x. NEXT lever candidate = grouped→dense-direct-A marlin decode. +## QUANT-CT-MXFP4-CLOSERS: the two arbitrated slivers LAND byte-exact + default-ON — dense M≤8 8-row Marlin tile (#50 (a)) + per-call workspace re-zero dropped (#50 (b)); correctness-complete on GB10, binding re-bench pending + + +Implemented the two #50-arbitrated closer slivers on `row/QUANT-CT-MXFP4-CLOSERS` +off `origin/main` `b1ac444e`. **Both are BYTE-EXACT (proven, not asserted) so they +ship DEFAULT-ON unconditionally — no gate, no near-tie razor.** MXFP4 parity goal +still owes the c1-c8 x3 binding re-bench (the definitive per-axis verdict) + the +substantive ~0.7ms/step host/sched slice. Box left clean (both locks free, GPU +idle, worker down, disk 21G). + +WHAT SHIPPED (file:line, all in the SHARED dense header +`include/vllm/model_executor/models/dense_nvfp4_gemm.h`; the qwen3_5.cpp twin that +serves the 27B/35B gate models' dense attn + shared-expert is DELIBERATELY +UNTOUCHED, so those SACRED gates cannot move): +- (a) `DenseAlignFor` (:286): `c.block = (M<=8) ? 8 : MarlinMoeAlignBlockSizeSelect(M,1,1)`. + vLLM's DENSE marlin uses an 8-row tile for a16 at prob_m<=8 (`m_block_size_8 = + prob_m<=8`, marlin.cu:438) — NO padding; our grouped single-expert MoE-align + picked block=16 at M=8 (`8*1/1/8==1.0` fails the `<0.9` test, + cuda_marlin_repack.cu:362), padding 8 dummy rows into a 16-row tile + (m_block_size_8=false) = the reproducible +0.33ms/step at c8 (#50 arbiter). The + m_block_size_8=true 8-row kernels are vendored (kernel_selector.h:3-8 nvfp4, + :33-38 mxfp4); marlin_mm_moe.cu:363-364 maps block=8 -> thread_m_blocks=1 + + m_block_size_8=true. M>8 unchanged (already matches vLLM). +- (b) `DenseMarlinWorkspace` (:294): zero the shared reduction workspace ONCE at + alloc; the two per-call `d.b.Memset(ws)` in `MatmulNvfp4MarlinD`/ + `GateUpFusedMarlinD` DROPPED. Mirror of vLLM `marlin_make_workspace_new` + (`torch.zeros`, marlin_utils.py:399-407) reused without re-zeroing. INVARIANT + cited in source: our launch pins `use_atomic_add=false / use_fp32_reduce=true` + (cuda_moe_marlin.cu:141-142), so the ONLY reachable cross-CTA reduce is the + fp32 barrier, whose last slice-block release re-zeroes the lock + (marlin_template.h:2170 `barrier_release(...,last)` -> `lock[0]=0` :204); + slice_count==1 never touches locks (:2162). The non-self-clearing atomic-add + path (:614) is unreachable under the pinned config (comment restores the zero + if it ever flips). + +GATE VERDICTS (dgx GB10, sm_121a, CUDA 13.0, RelWithDebInfo, MARLIN=ON, oracle +CUTLASS; `~/mxfp4-bench` overlay at `1f446fd7`+swap-ON+the 3 files, md5-matched; +ninja EXIT 0, no -Werror; both flock locks held for GPU legs): +- OP UNIT (RED-first, `tests/vt/test_ops_moe_grouped.cpp` new closers cases): the + block8-vs-block16 A/B at M=8 on REAL decode shapes is **BYTE-EXACT** — + `bitdiff=0/32768 max_abs=0` for MXFP4 K=4096/N=4096 AND K=12288/N=4096, and for + NVFP4 (group_blocks=1) K=4096/N=4096. So forcing block=8 does NOT flip an output + bit vs the production-validated block=16 route → zero risk to the SACRED token + gate. Sliver-(b) invariant proven DIRECTLY: workspace all-zero AFTER a GEMM + (`ws_nonzero==0`) + a second GEMM on the once-zeroed workspace bit-identical. + 15/15 assertions, 2/2 cases GREEN. +- LAUNCH-CONFIG (RED-first, `tests/vllm/models/test_qwen3_forward.cpp`): + `DenseAlignFor(d,8).block==8` (post-fix) with the pre-fix value documented + (`MarlinMoeAlignBlockSizeSelect(8,1,1)==16`); M in {1,2,4,8}==8, M in {12,16} + unchanged. 7/7 GREEN. +- MEMCHECK: `compute-sanitizer --tool memcheck --leak-check full` on the closers + cases = **0 real memory errors** (no invalid access/race). The only leaks are + static-cache artifacts of the moe_grouped CUDA harness at process exit — the + UNCHANGED pre-existing MXFP4 test leaks MORE (2.95MB/7 vs 0.98MB/2), so the + memset-drop introduces no leak. Memory-clean. +- #44 SMOKE (Yi30/Qwen3-8B-MXFP4, default async graphed config, vllm-cli greedy vs + golden_marlin_w4a16.json): **3/3 deterministic TOKEN-EXACT + near-tie coherent** + — the integrated model output is identical to golden, as byte-exactness predicts. + +DEFAULT DECISION = ON, unconditional (byte-exact bug-class refinement, not a +numerics change → no `VT_*` gate, unlike the FA2 swap). BLAST RADIUS (header +consumers): Qwen3-8B-MXFP4 (target), Qwen3-32B-NVFP4A16, Laguna dense projections +— all exercise the same `MatmulNvfp4W4A16D`; byte-exactness proven for BOTH quant +schemes closes it. The 32B-NVFP4 SACRED gate decodes at M=1 (sliver (a) is a +no-op there); the NVFP4 unit arm covers the M=8 path it can't. + +PARITY VERDICT (MXFP4 goal) = binding re-bench PENDING (this pass shipped the +correctness half). Projected: sliver (a) recovers ~0.33ms/step at c8 (~0.8pp, +#50 arbiter) byte-exact; (b) removes ~120 per-step Memset launches (host micro). +The ~0.7ms/step host/sched slice (the substantive term, #47 residual) is +UNADDRESSED. So the honest projection stays <1.0x on c2-c8 (best c8 ~0.95x); the +binding x3 confirms + attributes. NEXT: clean-checkout binding grid +(`mxfp4-online-serving-grid.sh`) + host-slice nsys. + +## QUANT-CT-MXFP4-CLOSERS BINDING + host-slice attribution: slivers improve EVERY throughput axis vs #49 (c1 crosses to parity+); MXFP4 goal still <1.0x on c2-c8 → NOT DONE; the ~0.7ms host slice is NOT in the engine loop + + +Ran the definitive clean-checkout binding grid at the committed sha `d3b412f5` (NOT an overlay: +`git checkout -f d3b412f5` in `~/mxfp4-bench/src`, tree clean, binaries rebuilt) via +`mxfp4-online-serving-grid.sh`, oracle vLLM 0.25.0 +`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, single-load/arm, drop_caches+mincore. #44 model +gate re-passed inside the grid. 24/24 legs `failed:0`, reps tight (ours c8 CoV ~0.4%). The grid +EXIT=1 is the gate-FAIL signal (`gate_pass:false`) + single-model "cross-model summary waits for the +other model" — NOT a crash; all q3mxfp4 artifacts complete (q3mxfp4's sweep IS c1-c8, matching +#48/#49). Evidence `dgx:~/work/vllm.cpp-online-gate/evidence/d3b412f5.../summary-q3mxfp4/ratios.json`. + +BINDING (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: +total_token_throughput c1 **1.005** / c2 0.925 / c4 0.939 / c8 **0.953** (Δ +0.015/+0.003/+0.009/ ++0.011); median TPOT c1 1.002 / c2 0.922 / c4 0.915 / c8 0.939; mean TTFT c1 1.034 / c2 0.962 / c4 +1.004 / c8 0.999; median ITL c1 1.004 / c2 0.919 / c4 0.920 / c8 0.929. Peak host-mem 35.2 vs 76.7 +GiB = **2.18x LESS** (WIN). + +PARITY VERDICT (MXFP4 goal): **c1 PASSES every axis** (tput/tpot/ttft/itl all ≥1.0x); c2-c8 BELOW on +tput/tpot/itl (best c8 0.953 tput), TTFT parity c4/c8. `gate_pass:false` → **NOT DONE** (below-floor +on c2-c8). But the byte-exact slivers moved EVERY throughput axis up vs #49 (c1 +1.5pp crosses to +parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression — exactly the #50 block=8 recovery +(~0.8pp@c8) + the memset-drop. + +RESIDUAL MAP (c8 ~4.75% tput gap): (1) block=16 padding — CLOSED (sliver a). (2) grouped-Marlin +decode +7-9% per-call (GPU: E=1 indirect `sorted_token_ids` gather + fp32 `C_tmp` vs vLLM dense +`marlin_gemm` direct-A; #46/#50 — delicate grouped→dense-direct-A port, not a config knob). (3) the +host slice. HOST-SLICE ATTRIBUTION (step 2, VT_LOOP_TRACE under decode this session): every window +shows **interval_ms ≈ step_ms** (mean 25.5 vs 25.5, Δ≤0.02ms; `admits=0` pure-decode windows +interval−step ≈ 0) → the born-on-runner engine-core loop (scheduler+drain+admit) has NEGLIGIBLE +per-iteration host overhead. So the #47 ~0.7ms/step "host/sched" residual is NOT an engine-loop +lever — it is in the shared-architecture async frontend (HTTP/output-proc/detokenize, which vLLM +also runs) or within the cross-tool attribution boundary. NEXT lever (only real one left) = +grouped→dense-direct-A marlin decode (#50 NO-GO'd the full port; a targeted E1 direct-A path is the +open candidate). Box left clean (both locks free, GPU idle, worker down, disk 21G, tmux gone). + diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 7b35a5aff..a2117bdbd 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Post-GQA-swap (#49): c1 0.990, c2-c8 0.92-0.94, TTFT parity, mem 2.6x WIN; dense-marlin port NO-GO (#50, cross-tool bias) | Closers: c8 block=8 tweak + ws-memset drop + ~0.7ms host slice | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Closers binding x3 (d3b412f5): c1 **1.005 PASSES**, c2/c4/c8 0.925/0.939/0.953 BELOW (+0.3/0.9/1.1pp vs #49), TTFT parity, mem 2.18x WIN; byte-exact slivers (bitdiff=0), #44 3/3, gate NO | grouped→dense-direct-A marlin; host slice NOT engine-loop (VT_LOOP_TRACE interval≈step) | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/STATUS.md b/docs/STATUS.md index 20f5d2d8c..b8b23dddb 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); binding c1 0.99 / c2-c8 0.92-0.94 post-GQA-swap (#49), mem 2.6x LESS, BELOW-FLOOR; dense-marlin port NO-GO (#50, cross-tool bias). **Marlin-config + glue-fusion levers both REFUTED by measurement** (`row/QUANT-CT-MXFP4-M28-LEVER`, `row/QUANT-CT-MXFP4-GLUE`); the c2-c8 step is the flash decode GQA group-swap. **Lever now IMPLEMENTED gated default-OFF (`VT_FA2_DECODE_GQA_SWAP`, `row/KERNEL-FA2-GQA-SWAP`): correctness-complete GB10 (op RED/GREEN, memcheck 0-err, swap-ON smoke token-exact); bench+flip pending.** Detail in state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4-expert paths. CPU E8M0 dequant (`2^(byte-127)`, group 32, no global scale) unit-gated 5/5·1142 vs `dq_mxfp4_torch`. GPU W4A4 fp4 GEMM + MoE-expert e2e remain later bricks | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); binding c1 0.99 / c2-c8 0.92-0.94 (#49 GQA-swap default-ON), mem 2.6x LESS, BELOW-FLOOR; dense-marlin port NO-GO (#50, cross-tool bias); Marlin-config + glue-fusion levers REFUTED by measurement. **Closer slivers LANDED byte-exact default-ON (`QUANT-CT-MXFP4-CLOSERS`, d3b412f5): dense M≤8 8-row Marlin tile + per-call ws re-zero dropped (block8≡block16 bitdiff=0 MXFP4+NVFP4, memcheck clean, #44 3/3). Binding x3: c1 1.005 PASSES, c2-c8 0.92-0.95 BELOW (best c8 0.953, +1.1pp vs #49), mem 2.18x WIN, gate NO; host slice NOT engine-loop (VT_LOOP_TRACE).** Detail in state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4-expert paths. CPU E8M0 dequant (`2^(byte-127)`, group 32, no global scale) unit-gated 5/5·1142 vs `dq_mxfp4_torch`. GPU W4A4 fp4 GEMM + MoE-expert e2e remain later bricks | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet | diff --git a/include/vllm/model_executor/models/dense_nvfp4_gemm.h b/include/vllm/model_executor/models/dense_nvfp4_gemm.h index fd23f434f..14021e6fa 100644 --- a/include/vllm/model_executor/models/dense_nvfp4_gemm.h +++ b/include/vllm/model_executor/models/dense_nvfp4_gemm.h @@ -270,7 +270,20 @@ inline DenseAlignCache& DenseAlignFor(Dev d, int M) { auto it = cache.find(M); if (it != cache.end()) return it->second; DenseAlignCache c; - c.block = vt::cuda::MarlinMoeAlignBlockSizeSelect(M, 1, 1); + // c8 sliver (#46/#50): vLLM's DENSE marlin uses an 8-row tile for the a16 + // path at prob_m<=8 (`m_block_size_8 = prob_m<=8 && a16`, + // csrc/libtorch_stable/quantization/marlin/marlin.cu:438) — NO padding. Our + // grouped single-expert MoE-align picks block_size_m=16 at M=8 + // (MarlinMoeAlignBlockSizeSelect: 8*1/1/8 == 1.0 fails the `< 0.9` test at + // cuda_marlin_repack.cu:362), padding 8 dummy rows into a 16-row tile + // (m_block_size_8=false) and wasting ~half the tile — the reproducible + // ~0.33ms/step at c8. The m_block_size_8=true 8-row kernels are vendored + // (kernel_selector.h:3-8 nvfp4, :33-38 mxfp4) and the fp32 C_tmp reduce is + // handled (marlin_mm_moe.cu:363-364 map block=8 -> thread_m_blocks=1 + + // m_block_size_8=true). Force block=8 for the single-expert dense case at + // M<=8 to match vLLM's dense tile exactly; M>8 is unchanged (already matches + // vLLM, which drops m_block_size_8 above 8). + c.block = (M <= 8) ? 8 : vt::cuda::MarlinMoeAlignBlockSizeSelect(M, 1, 1); vt::cuda::MarlinMoeAlignSizes(M, 1, 1, c.block, &c.max_tok, &c.max_blk); c.sorted = d.b.Alloc(static_cast(c.max_tok) * sizeof(int32_t)); c.expert = d.b.Alloc(static_cast(c.max_blk) * sizeof(int32_t)); @@ -289,8 +302,10 @@ inline DenseAlignCache& DenseAlignFor(Dev d, int M) { return cache.emplace(M, c).first->second; } -// Shared zeroed reduction workspace for the dense Marlin GEMMs (sms*4 i32 locks, -// mirror marlin_make_workspace_new). Memset to zero before each launch. +// Shared reduction workspace for the dense Marlin GEMMs (sms*4 i32 locks, mirror +// marlin_make_workspace_new). Zeroed ONCE at allocation; NOT re-zeroed per call +// (see the self-reset invariant below), exactly as vLLM allocates it with +// `torch.zeros` (marlin_utils.py:399-407) and reuses it across every call. inline void* DenseMarlinWorkspace(Dev d, int* out_sms) { static std::mutex mu; static void* ws = nullptr; @@ -299,6 +314,17 @@ inline void* DenseMarlinWorkspace(Dev d, int* out_sms) { if (!ws) { sms = vt::cuda::MarlinDeviceSms(d.q.device.index); ws = d.b.Alloc(static_cast(sms) * 4 * sizeof(int32_t)); + // Zero ONCE. The kernel self-resets its barrier locks: our launch pins + // use_atomic_add=false / use_fp32_reduce=true (cuda_moe_marlin.cu:141-142), + // so the ONLY reachable cross-CTA reduce is the fp32 barrier, whose LAST + // slice-block release re-zeroes the lock (marlin_template.h:2170 + // `barrier_release(&locks[locks_off], last)` -> `lock[0]=0` at :204); the + // slice_count==1 case never touches locks at all (:2162). So every completed + // GEMM leaves the workspace back at 0 and re-zeroing before each of the ~120 + // dense GEMMs/step is redundant host/launch work. (The non-self-clearing + // atomic-add path at :614 is unreachable under this pinned config; if that + // config ever flips, restore the per-call zero.) + d.b.Memset(d.q, ws, 0, static_cast(sms) * 4 * sizeof(int32_t)); } *out_sms = sms; return ws; @@ -312,8 +338,7 @@ inline DBuf MatmulNvfp4MarlinD(Dev d, const Tensor& x, const Nvfp4Weight& w, if (!mr.ready) BuildMarlinDenseResident(d, w, mr); DenseAlignCache& ac = DenseAlignFor(d, static_cast(M)); int sms = 0; - void* ws = DenseMarlinWorkspace(d, &sms); - d.b.Memset(d.q, ws, 0, static_cast(sms) * 4 * sizeof(int32_t)); + void* ws = DenseMarlinWorkspace(d, &sms); // zeroed once; kernel self-resets ++MutableW4A16Stats().marlin_gemms; // Marlin's output is bf16 (c_type=kBFloat16); an f32 result is the bf16 output @@ -429,8 +454,7 @@ inline DBuf GateUpFusedMarlinD(Dev d, const Tensor& x, const Nvfp4Weight& gw, if (!mr.ready) BuildMarlinDensePairResident(d, gw, uw, mr); DenseAlignCache& ac = DenseAlignFor(d, static_cast(M)); int sms = 0; - void* ws = DenseMarlinWorkspace(d, &sms); - d.b.Memset(d.q, ws, 0, static_cast(sms) * 4 * sizeof(int32_t)); + void* ws = DenseMarlinWorkspace(d, &sms); // zeroed once; kernel self-resets ++MutableW4A16Stats().fused_gate_up; DBuf gu(d, DType::kBF16, {M, 2 * N}); diff --git a/tests/vllm/models/test_qwen3_forward.cpp b/tests/vllm/models/test_qwen3_forward.cpp index 11b547ee5..ec3405c21 100644 --- a/tests/vllm/models/test_qwen3_forward.cpp +++ b/tests/vllm/models/test_qwen3_forward.cpp @@ -637,6 +637,41 @@ TEST_CASE("qwen3 dense forward: real Qwen3-0.6B CUDA prefill argmax (dgx-only)") CHECK(argmax == 12095); } +#ifdef VT_MARLIN_NVFP4 +// QUANT-CT-MXFP4-CLOSERS sliver (a): the shared dense NVFP4/MXFP4 W4A16 route +// forces moe_block_size=8 at M<=8 (dense_nvfp4_gemm.h::DenseAlignFor), matching +// vLLM's dense marlin m_block_size_8 = prob_m<=8, while leaving M>8 on the raw +// MoE-align selection. Direct launch-config assertion: RED against the pre-fix +// code, whose DenseAlignFor returned MarlinMoeAlignBlockSizeSelect(M,1,1) == 16 +// at M=8. DenseAlignFor allocs device memory + runs the moe_align kernel, so +// this is CUDA-only. Only dense_nvfp4_gemm.h is changed — the qwen3_5.cpp twin +// (27B/35B gate models' dense attn + shared-expert) is deliberately untouched. +TEST_CASE("dense_nvfp4 DenseAlignFor forces block=8 at M<=8 (dgx-only)") { + vt::Backend* cuda = nullptr; + try { + cuda = &vt::GetBackend(vt::DeviceType::kCUDA); + } catch (...) { + MESSAGE("SKIP: no CUDA backend registered"); + return; + } + vt::Queue q = cuda->CreateQueue(); + vllm::dense_nvfp4::Dev d{*cuda, q}; + // The pre-fix production route at M=8 was 16 (padded tile); the sliver forces 8. + CHECK(vt::cuda::MarlinMoeAlignBlockSizeSelect(8, 1, 1) == 16); + for (int M : {1, 2, 4, 8}) { + CAPTURE(M); + CHECK(vllm::dense_nvfp4::DenseAlignFor(d, M).block == 8); + } + // M>8 unchanged: still the MoE-align pick (16 at M in (8,16], 32 at M=16). + for (int M : {12, 16}) { + CAPTURE(M); + CHECK(vllm::dense_nvfp4::DenseAlignFor(d, M).block == + vt::cuda::MarlinMoeAlignBlockSizeSelect(M, 1, 1)); + } + cuda->DestroyQueue(q); +} +#endif // VT_MARLIN_NVFP4 + TEST_CASE("qwen3 dense forward: fusion-catalog ADOPT == hand-call fallback (byte-exact)") { const HfConfig c = TinyConfig(); const Qwen3DenseWeights w = TinyWeights(c); diff --git a/tests/vt/test_ops_moe_grouped.cpp b/tests/vt/test_ops_moe_grouped.cpp index e4a2e65e5..74fb64540 100644 --- a/tests/vt/test_ops_moe_grouped.cpp +++ b/tests/vt/test_ops_moe_grouped.cpp @@ -878,6 +878,245 @@ TEST_CASE("CUDA marlin MXFP4 W4A16 (group_blocks=2, E8M0) matches CPU dequant re } } +// --- QUANT-CT-MXFP4-CLOSERS slivers (a)+(b) -------------------------------- +// Sliver (a): the dense single-expert path forces moe_block_size=8 at M<=8 +// (dense_nvfp4_gemm.h::DenseAlignFor) so M=8 uses vLLM's 8-row m_block_size_8 +// tile instead of the padded 16-row tile MarlinMoeAlignBlockSizeSelect otherwise +// picks at M=8. This proves the forced block=8 route is CORRECT vs an +// independent CPU dequant reference at M=8, and MEASURES whether it is +// byte-exact vs the block=16 route production used before the fix (the near-tie +// arbiter). Sliver (b): the shared reduction workspace is zeroed ONCE, not +// per-call — the fp32-reduce marlin barrier self-resets its locks +// (marlin_template.h:2170 barrier_release(...,last) -> :204 lock[0]=0; the +// slice_count==1 case never touches locks at :2162; our launch pins +// use_atomic_add=false at cuda_moe_marlin.cu:141, so the non-self-clearing +// atomic-add path at :614 is unreachable). Proven directly: the workspace is +// all-zero AFTER a GEMM, and a second GEMM on the once-zeroed workspace is +// bit-identical to the first. +TEST_CASE("CUDA marlin MXFP4 W4A16 dense M=8: block=8 route + ws self-reset (closers)") { + if (!HasCuda()) { + MESSAGE("no CUDA backend registered; skipping"); + return; + } + Backend& gpu = vt::GetBackend(DeviceType::kCUDA); + + // Real Qwen3-8B decode projection shapes (K%128==0, K%32==0, N%64==0). + for (auto KN : std::vector>{{4096, 4096}, {12288, 4096}}) { + const int64_t K = KN.first, N = KN.second; + CAPTURE(K); + CAPTURE(N); + const int64_t M = 8, top_k = 1, E = 1, P = M * top_k; + + // Sanity: MarlinMoeAlignBlockSizeSelect DOES pick 16 at M=8 (the pre-fix + // production route this sliver overrides to 8). + CHECK(vt::cuda::MarlinMoeAlignBlockSizeSelect(static_cast(M), 1, 1) == 16); + + Mxfp4Weight w = MakeMxfp4Weight(N, K, 4321); + std::vector w_f32(static_cast(N * K)); + vllm::DequantMxfp4ToF32(w.packed.data(), w.scale.data(), N, K, w_f32.data()); + + const auto act_f = RandomF32(static_cast(M * K), 7700); + const auto act_bf16 = ToBf16(act_f); + std::vector act_r(act_f.size()); + for (size_t i = 0; i < act_r.size(); ++i) act_r[i] = vt::BF16ToF32(act_bf16[i]); + std::vector ref(static_cast(P * N), 0.0f); + for (int64_t m = 0; m < M; ++m) + for (int64_t n = 0; n < N; ++n) { + float acc = 0.0f; + for (int64_t k = 0; k < K; ++k) + acc += act_r[static_cast(m * K + k)] * w_f32[static_cast(n * K + k)]; + ref[static_cast(m * N + n)] = acc; + } + + QueueGuard gq(gpu); + void* stream = gq.q.handle; + const int dev = gq.q.device.index; + + // Repack ONCE (block-independent). + DeviceTensor dp(gpu, gq.q, DType::kI8, {N, K / 2}, w.packed.data()); + DeviceTensor ds(gpu, gq.q, DType::kI8, {N, K / 32}, w.scale.data()); + DeviceTensor wq(gpu, gq.q, DType::kI32, {E, K / 16, N * 2}); + DeviceTensor sc(gpu, gq.q, DType::kI8, {E, K / 32, N}); + vt::cuda::MarlinRepackExpertWeight(stream, dev, static_cast(wq.ptr()), + static_cast(dp.ptr()), + static_cast(K), static_cast(N)); + vt::cuda::MarlinProcessExpertScalesMxfp4(stream, static_cast(ds.ptr()), + static_cast(sc.ptr()), + static_cast(K), static_cast(N)); + float g_dummy = 1.0f; + DeviceTensor gg(gpu, gq.q, DType::kF32, {E}, &g_dummy); + DeviceTensor dact(gpu, gq.q, DType::kBF16, {M, K}, act_bf16.data()); + const int sms = vt::cuda::MarlinDeviceSms(dev); + + // One dense single-expert GEMM at `block`, sharing `wst` (the workspace). + auto run = [&](int block, Tensor wst, std::vector& out) { + std::vector topk_ids(static_cast(P), 0); + std::vector topk_w(static_cast(P), 1.0f); + int max_tok = 0, max_blk = 0; + vt::cuda::MarlinMoeAlignSizes(static_cast(M), static_cast(top_k), + static_cast(E), block, &max_tok, &max_blk); + DeviceTensor dtid(gpu, gq.q, DType::kI32, {M, top_k}, topk_ids.data()); + DeviceTensor dtw(gpu, gq.q, DType::kF32, {M, top_k}, topk_w.data()); + DeviceTensor sorted_ids(gpu, gq.q, DType::kI32, {max_tok}); + DeviceTensor expert_ids(gpu, gq.q, DType::kI32, {max_blk}); + DeviceTensor num_pad(gpu, gq.q, DType::kI32, {1}); + vt::cuda::MarlinMoeAlignBlockSize(stream, static_cast(dtid.ptr()), + static_cast(M), static_cast(top_k), + static_cast(E), block, + static_cast(sorted_ids.ptr()), + static_cast(expert_ids.ptr()), + static_cast(num_pad.ptr())); + DeviceTensor dout(gpu, gq.q, DType::kBF16, {P, N}); + vt::MoeMarlinArgs args{block, static_cast(top_k), static_cast(M), + static_cast(N), static_cast(K), false}; + args.group_size = 32; + args.mxfp4 = true; + vt::MoeGroupedGemmNvfp4Marlin(gq.q, dout.tensor(), dact.tensor(), wq.tensor(), sc.tensor(), + gg.tensor(), wst, sorted_ids.tensor(), expert_ids.tensor(), + num_pad.tensor(), dtw.tensor(), args); + out.assign(static_cast(P * N), 0); + dout.Download(gq.q, out.data()); + }; + auto to_f32 = [](const std::vector& b) { + std::vector f(b.size()); + for (size_t i = 0; i < b.size(); ++i) f[i] = vt::BF16ToF32(b[i]); + return f; + }; + + // Shared workspace zeroed ONCE (mirror the production DenseMarlinWorkspace). + DeviceTensor ws(gpu, gq.q, DType::kI32, {sms * 4}); + gpu.Memset(gq.q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + Tensor wst = MakeTensor(ws.ptr(), DType::kI32, gq.q.device, {sms * 4}); + + // Sliver (a): forced block=8 route is CORRECT vs the independent CPU ref. + std::vector out8; + run(8, wst, out8); + CheckClose(to_f32(out8), ref, 2e-2f, 2e-2f); + + // Sliver (b) invariant: a completed GEMM leaves the workspace all-zero. A + // non-zero lock here would mean the dropped per-call re-zero is load-bearing. + std::vector ws_host(static_cast(sms * 4), -1); + ws.Download(gq.q, ws_host.data()); + size_t ws_nonzero = 0; + for (int32_t v : ws_host) + if (v != 0) ++ws_nonzero; + CAPTURE(ws_nonzero); + CHECK(ws_nonzero == 0); + + // Sliver (b) reuse safety: a SECOND GEMM on the once-zeroed, not-re-zeroed + // workspace is bit-identical to the first. + std::vector out8b; + run(8, wst, out8b); + CHECK(out8b == out8); + + // Sliver (a) byte-exact arbiter: block=16 was the pre-fix M=8 route. Measure + // whether forcing block=8 changes the output bits (byte-exact => zero risk to + // the SACRED token gate; near-tie => the token gate arbitrates). + std::vector out16; + run(16, wst, out16); + CheckClose(to_f32(out16), ref, 2e-2f, 2e-2f); + size_t bitdiff = 0; + float max_abs = 0.0f; + { + const auto f8 = to_f32(out8); + const auto f16 = to_f32(out16); + for (size_t i = 0; i < out8.size(); ++i) { + if (out8[i] != out16[i]) ++bitdiff; + max_abs = std::max(max_abs, std::fabs(f8[i] - f16[i])); + } + } + MESSAGE("block8-vs-block16 M=8 K=" << K << " N=" << N << " bitdiff=" << bitdiff << "/" + << out8.size() << " max_abs=" << max_abs); + } +} + +// Sliver (a) blast-radius closure for the NVFP4 (group_blocks=1, fp8 scales + +// global) dense route — the OTHER header consumer (Qwen3-32B-NVFP4A16, Laguna). +// The m_block_size_8 tile controls the bf16 ACTIVATION/A-matrix layout, which is +// identical for NVFP4 and MXFP4 (the quant scheme only changes the B-weight scale +// decode along K), so byte-exactness is quant-agnostic; this proves it directly. +TEST_CASE("CUDA marlin NVFP4 W4A16 dense M=8: block=8 byte-exact vs block=16 (closers)") { + if (!HasCuda()) { + MESSAGE("no CUDA backend registered; skipping"); + return; + } + Backend& gpu = vt::GetBackend(DeviceType::kCUDA); + const int64_t K = 4096, N = 4096, M = 8, top_k = 1, E = 1, P = M * top_k; + + // The byte-exact A/B is self-validating: block=16 is the production-validated + // NVFP4 route (the Qwen3-32B-NVFP4A16 SACRED paged-engine gate runs on it), so + // block=8 == block=16 bit-for-bit proves block=8 is exactly as correct without + // needing an absolute CPU reference (whose sf/global-scale convention differs + // from the Marlin repack path — see the divisor-reciprocal note in + // test_qwen3_forward.cpp). + Nvfp4Weight w = MakeNvfp4Weight(N, K, 4321); + const auto act_f = RandomF32(static_cast(M * K), 7700); + const auto act_bf16 = ToBf16(act_f); + + QueueGuard gq(gpu); + void* stream = gq.q.handle; + const int dev = gq.q.device.index; + + std::vector bufs{w.scale.data()}; + std::vector lens{w.scale.size()}; + const float sf = vt::cuda::MarlinNvfp4CombinedScaleFactor(bufs, lens); + DeviceTensor dp(gpu, gq.q, DType::kI8, {N, K / 2}, w.packed.data()); + DeviceTensor dsx(gpu, gq.q, DType::kI8, {N, K / 16}, w.scale.data()); + DeviceTensor wq(gpu, gq.q, DType::kI32, {E, K / 16, N * 2}); + DeviceTensor sc(gpu, gq.q, DType::kI8, {E, K / 16, N}); + vt::cuda::MarlinRepackExpertWeight(stream, dev, static_cast(wq.ptr()), + static_cast(dp.ptr()), static_cast(K), + static_cast(N)); + vt::cuda::MarlinProcessExpertScales(stream, static_cast(dsx.ptr()), + static_cast(sc.ptr()), static_cast(K), + static_cast(N), sf); + float gsc = vt::cuda::MarlinNvfp4ProcessGlobalScale(w.scale2, sf); + DeviceTensor gg(gpu, gq.q, DType::kF32, {E}, &gsc); + DeviceTensor dact(gpu, gq.q, DType::kBF16, {M, K}, act_bf16.data()); + const int sms = vt::cuda::MarlinDeviceSms(dev); + DeviceTensor ws(gpu, gq.q, DType::kI32, {sms * 4}); + gpu.Memset(gq.q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + Tensor wst = MakeTensor(ws.ptr(), DType::kI32, gq.q.device, {sms * 4}); + + auto run = [&](int block, std::vector& out) { + std::vector topk_ids(static_cast(P), 0); + std::vector topk_w(static_cast(P), 1.0f); + int max_tok = 0, max_blk = 0; + vt::cuda::MarlinMoeAlignSizes(static_cast(M), static_cast(top_k), + static_cast(E), block, &max_tok, &max_blk); + DeviceTensor dtid(gpu, gq.q, DType::kI32, {M, top_k}, topk_ids.data()); + DeviceTensor dtw(gpu, gq.q, DType::kF32, {M, top_k}, topk_w.data()); + DeviceTensor sorted_ids(gpu, gq.q, DType::kI32, {max_tok}); + DeviceTensor expert_ids(gpu, gq.q, DType::kI32, {max_blk}); + DeviceTensor num_pad(gpu, gq.q, DType::kI32, {1}); + vt::cuda::MarlinMoeAlignBlockSize(stream, static_cast(dtid.ptr()), + static_cast(M), static_cast(top_k), + static_cast(E), block, + static_cast(sorted_ids.ptr()), + static_cast(expert_ids.ptr()), + static_cast(num_pad.ptr())); + DeviceTensor dout(gpu, gq.q, DType::kBF16, {P, N}); + vt::MoeMarlinArgs args{block, static_cast(top_k), static_cast(M), + static_cast(N), static_cast(K), false}; + args.group_size = 16; // NVFP4 + vt::MoeGroupedGemmNvfp4Marlin(gq.q, dout.tensor(), dact.tensor(), wq.tensor(), sc.tensor(), + gg.tensor(), wst, sorted_ids.tensor(), expert_ids.tensor(), + num_pad.tensor(), dtw.tensor(), args); + out.assign(static_cast(P * N), 0); + dout.Download(gq.q, out.data()); + }; + + std::vector out8, out16; + run(8, out8); + run(16, out16); + size_t bitdiff = 0; + for (size_t i = 0; i < out8.size(); ++i) + if (out8[i] != out16[i]) ++bitdiff; + MESSAGE("NVFP4 block8-vs-block16 M=8 K=" << K << " N=" << N << " bitdiff=" << bitdiff << "/" + << out8.size()); + CHECK(bitdiff == 0); +} + TEST_CASE("CUDA moe_align parallel == serial (expert_ids/num_pad exact, per-expert multiset)") { if (!HasCuda()) { MESSAGE("no CUDA backend registered; skipping");