diff --git a/.agents/NOW.md b/.agents/NOW.md index 3995dd8a..1a4fc739 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. #75: occupancy IDENTICAL 8.33%; matched vLLM reg+instr, STILL +10us = ptxas gap | c8: flash ptxas ~40%; try ptxas-lineage vendor | +| MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Op-dispatch profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | | Supported-models list (`row/DOCS-SUPPORTED-MODELS-MATRIX`) | **DRAFT PR**: FEATURES per-arch table CI-bound to registry (30 archs) | Reviewer merge | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 064121e1..73dcef21 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,33 @@ from relative link targets repointed for this file's location. # Benchmarks +## QUANT-CT-MXFP4-FLASH-PTXAS — the ptxas-lineage hypothesis is REFUTED three ways: vLLM's fa2 wheel ships NO sm_12x cubin (only CUDA-13.0 PTX-ISA-9.0 compute_80 PTX, driver-JIT'd on GB10), an "old CUDA 12.x ptxas" cannot even target sm_121a, and a same-params cuModule A/B shows our-PTX and vLLM's-OWN-PTX schedule the c8 decode kernel identically (~144 us) across driver-JIT / ptxas 13.0 / ptxas 13.2 — the +10 us/call engine gap is ENGINE CONTEXT, not flash codegen (2026-08-06, `row/QUANT-CT-MXFP4-FLASH-PTXAS`, base `362a3c99`, GB10 sm_121a, PR #82) + +#75 attributed the residual +10 us/call c8 flash gap (ours ~167 vs vLLM ~157) to "vLLM's wheel `ptxas` SASS-scheduling quality (older CUDA 12.x lineage)" and OWED obtaining that ptxas and A/B'ing it. This row did. **The lineage hypothesis dies at the premise, then again at the measurement.** + +**W1(a) — the wheel has no old lineage and no runnable GB10 cubin (`cuobjdump` `_vllm_fa2_C.abi3.so`, vLLM 0.25.0-stage).** 52 ELF cubins ALL `sm_80`; 52 PTX ALL `.target sm_80 .version 9.0`. PTX ISA 9.0 = **CUDA 13.0**, the SAME major lineage as our nvcc 13.0 — not an older 12.x. There is **NO sm_90/100/120/121 cubin**. On GB10 (sm_121a) the sm_80 SASS cannot execute, so the flash SASS that actually RUNS is **driver-JIT'd from the compute_80 PTX by the box CUDA-13.0 driver (580.159.03)** — produced by the DRIVER's JIT, never by a wheel `ptxas`. #75's mental model (a wheel `ptxas` baked vLLM's fast sm_121a cubin) is factually wrong; vLLM ships portable PTX and relies on the same driver JIT our own build can invoke. #75's own Build C already used that exact assembler (it compiled `code=compute_80` → driver-JIT) and matched vLLM's reg+instr — the assembler was never a variable we lacked. + +**W1(b) — an "old CUDA 12.x ptxas" cannot target the arch.** Box `ptxas`: 12.8 (triton), 13.0 (toolkit, our build default), 13.2 (nvidia/cu13 wheel). `ptxas 12.8` tops out at **sm_120a — no sm_121a** (first appears in 13.0), and it cannot read PTX ISA 9.0. So the mission's literal recipe (get the old ptxas, assemble our PTX for sm_121a) is **impossible by construction**: sm_121a SASS only exists from ptxas 13.0+. + +**W1(c) — same-params cuModule A/B (THE ARBITER), locked idle box, CUDA-event medians (100× back-to-back × 15 reps).** A standalone harness builds `Flash_fwd_params` for the exact c8 GQA-swap decode (b=8, kv_heads=8, ngroups=4, d=128, seqlen_k=1024, num_splits=3 → grid (1,3,64), block 128, dyn-smem 81 920 B) and times the SAME decode kernel (`flash_fwd_splitkv_kernel<…128,64,128,4…,0,0,0,0,1,0,1,0>`, mangled-identical in our build and vLLM PTX #30) loaded from our compute_80+fast-math PTX and from vLLM's extracted PTX #30, each via {driver-JIT, ptxas 13.0, ptxas 13.2}. All six cubins REG=241 (matched vLLM). Result: + +| decode-kernel arm (single split-launch) | module us/call | module/native | +|---|---|---| +| our compute_80+fm — driver-JIT | 144.96 | 0.998 | +| our compute_80+fm — ptxas 13.0 | 149.07 | 1.004 | +| our compute_80+fm — ptxas 13.2 | 150.78 | 1.009 | +| vLLM PTX #30 — driver-JIT | 144.55 | 1.007 | +| vLLM PTX #30 — ptxas 13.0 | 135.95 | 0.969 | +| vLLM PTX #30 — ptxas 13.2 | 140.54 | 1.013 | + +Native sm_121a anchors drifted 138.7–149.4 across the sequential arms (box clock drift), so the module/native RATIO is the drift-normalized read: all six ∈ [0.969, 1.013], i.e. **within ±1.3% — a tie.** No ptxas lineage (13.0/13.2/driver-JIT) moves our kernel, and vLLM's OWN PTX through the same assemblers runs the same as ours. The single 0.969 excursion (vLLM PTX+ptxas13.0) is contradicted by its two sister vLLM arms (driver-JIT 1.007, ptxas13.2 1.013) and by our-PTX+ptxas13.0 (1.004): neither ptxas 13.0 nor vLLM's PTX is systematically faster — it is the low tail of box drift. + +**THE ARBITER VERDICT: NO.** No old-ptxas cubin hits ~157 (none can be built; and the driver-JIT that vLLM actually uses was already our #75 Build C). The flash decode kernel's CODEGEN is at PARITY across every reachable toolchain AND against vLLM's own PTX. **The corrected mechanism:** in ISOLATION every variant is ~144–151 us, whereas in the real engine #75 measured ours 167 / vLLM 157 — so the +10 us/call gap is NOT in the flash kernel's SASS at all; it is ENGINE CONTEXT (the L2/TLB/clock state the neighbour kernels leave for the flash launch — exactly #69's "the flash kernel's cost moves with its NEIGHBOURS' L2 footprint"). This RETIRES #75's "ptxas SASS-scheduling quality" attribution. (Caveat, stated: the microbench is L2-warm — KV re-read back-to-back — so its ~144 us absolute sits below the engine's DRAM-cold 157–167; this does not affect the codegen arbitration because #75 showed the dominant stalls are smem-scoreboard/CTA-barrier, regime-independent, and they tie here across all toolchains.) + +**W2/W3 — none owed.** NO vendor (nothing beat the driver JIT we already use). Binding UNCHANGED: c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969. THE PARITY VERDICT: MXFP4 stays BELOW-FLOOR at c2–c8, TERMINAL, with the corrected residual map — flash-kernel codegen at PARITY (this row); the c8 ~0.969 residual = flash ENGINE-CONTEXT adjacency (glue/marlin-neighbour L2/orchestration, a different lever than the kernel) + portable-fusion glue ~18% (numerics-delicate, sub-parity) + marlin/host remainder. No default flip owed. **hd256 (27B/35B) cross-model projection:** structurally identical — vLLM's fa2 wheel ships the hd256 splitkv kernels as sm_80-only PTX driver-JIT'd on GB10 too (no old-ptxas lineage, same driver JIT), so NO hd256 flash-ptxas vendor is owed either; the 27B/35B flash-decode codegen is at the same parity and their residuals are likewise context/glue, not ptxas. + +Evidence: `dgx:~/mxfp4-ptxas/{exp.sh,exp.log,bench_flash_ptxas.cu,*.cubin,our_c80fm.ptx,vllm.ptx}`; wheel `_vllm_fa2_C.abi3.so` cuobjdump (52×sm_80, PTX 9.0). The CMakeLists NOTE + this entry record the closed levers so they are not re-tried. + ## QUANT-CT-MXFP4-FLASH-OCCUPANCY — the owed ours-vs-vLLM flash decode ncu diff: occupancy is IDENTICAL (8.33%, smem-limited); the flash gap is a COMPILER-CODEGEN difference (vLLM's wheel SASS runs 13% fewer instructions on the byte-identical kernel), NOT occupancy/L2/arch/fast-math — no lever exists on our stack (2026-08-06, `row/QUANT-CT-MXFP4-FLASH-OCCUPANCY`, base `f7a1e322`, GB10 sm_121a, PR #75) #69 closed the compile lens (flash source byte-identical to vLLM's `2c839c33`; `-use_fast_math` REJECTED, +21us/call) and OWED the ours-vs-vLLM ncu diff (vLLM side lost to a box OOM-reboot). This row runs that diff to a MEASURED verdict on an idle box and re-frames the #69 premise. **Vehicle** `Yi30/Qwen3-8B-MXFP4` (dense W4A16 Marlin), oracle arm `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`. diff --git a/.agents/specs/mxfp4-flash-ptxas-2026-08-06.md b/.agents/specs/mxfp4-flash-ptxas-2026-08-06.md new file mode 100644 index 00000000..db3f6ce6 --- /dev/null +++ b/.agents/specs/mxfp4-flash-ptxas-2026-08-06.md @@ -0,0 +1,76 @@ +# QUANT-CT-MXFP4-FLASH-PTXAS — arbitrate the ptxas lineage behind vLLM's faster flash decode SASS + + +Row: `QUANT-CT-MXFP4-FLASH-PTXAS` (helper, `row/QUANT-CT-MXFP4-FLASH-PTXAS`). +Base: `origin/main` `362a3c99`. Vehicle: `Yi30/Qwen3-8B-MXFP4` (dense +`Qwen3ForCausalLM`, W4A16 Marlin keep-quant); oracle arm +`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, vLLM 0.25.0-stage. GB10 sm_121a. + +## Why this row exists + +#75 closed the codegen lens to a MEASURED verdict: same `flash_fwd_splitkv` source +(vendored from vLLM's pinned `2c839c33`), same grid (1×3×64), same 8.33% occupancy +(both smem-limited to 1 CTA/SM by 81.92 KB), same L2 ~1%, and — with +`compute_80 + -use_fast_math` (Build C) — matched vLLM's register (241) AND +instruction (17,008 vs 17,020) counts EXACTLY. Yet Build C is ~167 us/call, vLLM +is ~157 us/call. #75 attributed the residual to "vLLM's wheel `ptxas` +SASS-scheduling quality (older CUDA 12.x lineage)". This row arbitrates that +attribution to a MEASURED yes/no: get the wheel's exact ptxas lineage, re-assemble +our flash PTX with it, and A/B the kernel. + +## W1 — arbitrate the ptxas lineage +(a) Identify the wheel's toolkit lineage: `cuobjdump` the fa2 `.so`. +(b) Obtain that ptxas (venv `nvidia-cuda-nvcc-cu12` wheels or a scratch pip download). +(c) Compile our flash decode TU to PTX (compute_80 + fast-math per #75 Build C), +assemble with the candidate ptxas lineages → cubin, load via `cuModule` in a +microbench with the c8 decode params, and A/B (CUDA-event medians + ncu). +THE ARBITER: does a different-lineage-ptxas cubin hit ~156-157 us? NO ⇒ refutation, +close flash as measured-irreducible-for-us, STOP. + +## W2 — vendor (only on a YES) +Mirror the GDN Triton-AOT precedent: commit the cubin + exact regen recipe under the +vendored-kernels tree; a load path routing the sm_121a decode flash launch through +the vendored cubin, gated `VT_FA2_VENDORED_CUBIN`, default per parity-enablers ONLY +IF the full battery is green (near-tie razor for the fast-math numerics; #44 smoke; +SACRED 0.6B/4B + 32B strict; async; memcheck; eager+graphed; both GQA ratios). +hd256 (27B/35B) cross-model projection measured, not flipped, as a follow-up. + +## W3 — binding + verdict +c1..c8 ×3 production defaults vs 1.020/0.962/0.966/0.969. THE PARITY VERDICT: +≥1.0 every axis ⇒ MXFP4 parity DONE; short ⇒ honest terminal map with the fresh +decomposition (flash-after-arbitration + glue ~18% + marlin/host remainder). + +## Gates +Byte-exact razor: #44 MXFP4-8B smoke 3/3 + coherent. If fast-math shifts the +reduction order: SACRED 0.6B/4B distributional + 32B strict, async, memcheck, +eager+graphed. Box safety: BOTH flock locks, free -g ≥ 90, worker STOPPED, tmux + +done-markers, sequential arms, single-load steady-state, disk floor 15G. + +## CLOSED — verdict (2026-08-06, THE ARBITER = NO) + +The ptxas-lineage hypothesis is REFUTED three ways. **(a) No old lineage exists:** +`cuobjdump` of vLLM 0.25.0's `_vllm_fa2_C.abi3.so` = 52 `sm_80` cubins + 52 +`.target sm_80 .version 9.0` PTX (ISA 9.0 = **CUDA 13.0**, our own major), NO +sm_12x cubin. On GB10 the sm_80 SASS cannot run, so vLLM's flash SASS is +**driver-JIT'd from compute_80 PTX by the box CUDA-13.0 driver** — the SAME +assembler #75's Build C (`code=compute_80`) already used. There is no separate +"wheel ptxas" that baked a fast sm_121a cubin. **(b) Impossible by construction:** +the only sub-13 ptxas on the box (12.8) tops out at sm_120a and cannot target +sm_121a or read PTX 9.0 — an old-CUDA-12.x sm_121a cubin cannot be built at all. +**(c) Measured tie:** a same-params cuModule A/B (harness `bench_flash_ptxas.cu`, +c8 GQA-swap decode, grid (1,3,64), dyn-smem 81 920 B) times the byte-identical +decode kernel from our compute_80+fast-math PTX and from vLLM's PTX #30, each via +driver-JIT / ptxas 13.0 / ptxas 13.2 (all REG=241). Module/native ratios ∈ +[0.969, 1.013] — a ±1.3% tie; the lone 0.969 (vLLM-PTX+ptxas13.0) is contradicted +by both sister vLLM arms (1.007, 1.013) and our-PTX+ptxas13.0 (1.004), i.e. box +drift, not a lever. **Corrected mechanism:** the flash kernel codegen is at PARITY +across every reachable toolchain AND vs vLLM's own PTX (~144–151 us in isolation); +the +10 us/call the engine shows (ours 167 / vLLM 157, #75) is ENGINE CONTEXT +(neighbour-kernel L2/orchestration, per #69), not the kernel's SASS. This RETIRES +#75's "ptxas SASS-quality" attribution. **W2/W3 none owed:** no vendor (nothing +beat the driver JIT we already use); binding UNCHANGED c1 1.020 / c2 0.962 / c4 +0.966 / c8 0.969; MXFP4 terminal below-floor at c2–c8 with the corrected map +(flash codegen at parity; residual = context/glue/marlin-host). hd256 (27B/35B) +projection: structurally identical (sm_80-only PTX driver-JIT'd), NO hd256 vendor +owed. No functional code shipped; CMakeLists NOTE + benchmark-record (#82) capture +the closed levers. No byte-exact razor / SACRED battery owed (nothing shipped). diff --git a/.agents/state.md b/.agents/state.md index c1565604..b61ee57f 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -39742,3 +39742,51 @@ partition/supported_tasks guard mirroring upstream (community files strip the re and the encoder vision tower (W3) for clean image/video-conditioned ref2va/fl2va (ref2va with a synthetic reference + text-only encoder still grids). dgx assets: `~/h3fp4/ckpt/MiniMax-H3- FL2VA-Q3_K_M.gguf`, `~/h3fp4/fl2va_t2va_20/`. Box left clean. + +## 2026-08-07T00:45 - QUANT-CT-MXFP4-FLASH-PTXAS: the ptxas-lineage arbiter = NO; #75's "wheel ptxas SASS-quality" attribution RETIRED - vLLM's flash SASS is driver-JIT'd from CUDA-13.0 PTX, codegen ties across all ptxas AND vs vLLM's own PTX, the +10us gap is ENGINE CONTEXT (row/QUANT-CT-MXFP4-FLASH-PTXAS, helper, PR #82) + + +The last MXFP4 flash lever. #75 owed obtaining vLLM's wheel `ptxas` (hypothesized +"older CUDA 12.x lineage") and A/B'ing it against our flash PTX. Done. **Arbiter = +NO, refuted three ways.** + +- **(a) No old lineage / no runnable GB10 cubin.** `cuobjdump` of vLLM 0.25.0's + `_vllm_fa2_C.abi3.so` = 52 `sm_80` cubins + 52 `.target sm_80 .version 9.0` PTX. + PTX ISA 9.0 = CUDA 13.0 (our own major), NOT 12.x; NO sm_12x cubin. On GB10 the + sm_80 SASS can't run, so vLLM's flash SASS is **driver-JIT'd from compute_80 PTX + by the box CUDA-13.0 driver (580.159.03)** - the SAME assembler #75's Build C + (`code=compute_80`) already used. The "wheel ptxas baked a fast sm_121a cubin" + premise is factually wrong. +- **(b) Impossible by construction.** Box ptxas = 12.8 (triton), 13.0 (toolkit, + build default), 13.2 (nvidia/cu13). ptxas 12.8 tops out at sm_120a - it CANNOT + target sm_121a (first in 13.0) nor read PTX 9.0. So an old-12.x sm_121a cubin + cannot be built at all. +- **(c) Measured tie.** Standalone cuModule harness (`dgx:~/mxfp4-ptxas/`, + `bench_flash_ptxas.cu`) times the byte-identical c8 GQA-swap decode kernel + (grid (1,3,64), dyn-smem 81 920 B, all cubins REG=241) from our compute_80+ + fast-math PTX and vLLM's PTX #30, each via driver-JIT / ptxas 13.0 / ptxas 13.2. + Module/native ratios ∈ [0.969, 1.013] - a ±1.3% tie (native anchors drifted + 138.7-149.4 across the sequential arms; the lone 0.969 is contradicted by both + sister vLLM arms 1.007/1.013 and our-PTX+ptxas13.0 1.004 = box drift, not a + lever). + +**Corrected mechanism:** the flash decode kernel codegen is at PARITY across every +reachable toolchain AND vs vLLM's own PTX (~144-151 us in isolation); the +10 +us/call the engine showed (ours 167 / vLLM 157, #75) is ENGINE CONTEXT +(neighbour-kernel L2/orchestration, exactly #69's finding), NOT the kernel's SASS. +This RETIRES #75's "ptxas SASS-scheduling quality" attribution. (Microbench is +L2-warm so its absolute sits below the engine's DRAM-cold number; irrelevant to the +codegen arbitration since the dominant stalls are smem-scoreboard/CTA-barrier per +#75, regime-independent, and they tie here.) + +**Verdict / follow-up.** W2/W3 none owed: NO vendor (nothing beat the driver JIT we +already use), binding UNCHANGED c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969, MXFP4 +TERMINAL below-floor at c2-c8 with the corrected map (flash codegen at parity; +residual = flash engine-context adjacency + portable-fusion glue ~18% + +marlin/host). No default flip owed. hd256 (27B/35B) cross-model projection: +structurally identical (sm_80-only PTX driver-JIT'd), NO hd256 vendor owed; their +flash-decode residuals are likewise context/glue, not ptxas. No functional code +shipped; CMakeLists NOTE + benchmark-record (#82) + spec CLOSED capture the closed +levers. Box: experiment left artifacts under `~/mxfp4-ptxas/`; box OOM-rebooted +mid-confirmatory-run (another agent's 25 GiB vLLM on the 119 GiB unified pool, not +this <1 GiB microbench) - run-1 data is on disk and decisive. diff --git a/CMakeLists.txt b/CMakeLists.txt index fec00979..4b59576b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1430,9 +1430,14 @@ if(VLLM_CPP_CUDA) # instruction-bound, and the residual is vLLM's wheel-`ptxas` SASS-SCHEDULING # quality, not reachable from our nvcc 13.0. (native+fast-math REGRESSES to # 189.8 us, #68; compute_80 PTX driver-JIT alone is neutral, #75 Build B.) So - # we keep the plain native sm_121a compile and pass no -use_fast_math. The - # residual is an irreducible-for-us ptxas SASS-quality gap on a byte-identical - # kernel; see .agents/benchmark-record.md (#75). + # we keep the plain native sm_121a compile and pass no -use_fast_math. #82 + # (FLASH-PTXAS) then ARBITRATED the "ptxas lineage" claim to a NO and CORRECTED + # it: the wheel ships NO sm_12x cubin — only CUDA-13.0 (PTX ISA 9.0) sm_80 PTX + # driver-JIT'd on GB10 — so there is no old-CUDA-12.x ptxas in vLLM's path (and + # ptxas 12.8 cannot even target sm_121a); a same-params cuModule A/B ties our + # PTX and vLLM's OWN PTX at ~144 us across driver-JIT / ptxas 13.0 / 13.2, so the + # +10 us/call gap is ENGINE CONTEXT (neighbour L2/orchestration), not this + # kernel's codegen. See .agents/benchmark-record.md (#75, #82). set_source_files_properties( src/vt/cuda/cuda_flash_attn_fa2.cu ${_FA2_KERNEL_SRCS} PROPERTIES diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index d6bf145c..a2b7de3c 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -311,7 +311,7 @@ built on it rather than keeping the flattering one. | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. Detail: benchmark-record + spec §8 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920; dequant byte-exact. spec §8.6 | -| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** (+951us). `FLASH-OCCUPANCY` #75: matched-c8 ncu, occupancy IDENTICAL 8.33%; built vLLM's exact flash recipe, matched reg+instr, STILL +10us, gap is ptxas SASS quality, no lever/flip | +| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** (+951us). `FLASH-PTXAS` #82: cuModule A/B ties our+vLLM PTX across ptxas 13.0/13.2/driver-JIT (~144us); +10us is engine CONTEXT not codegen, no ptxas lever/flip (retires #75) | | Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: no number measured, claimed or owed.** Vulkan registers 8 of the CPU backend's 83 ops and runs no model. The 2026-08-06 campaign spec is PLAN ONLY. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-E`: build llama.cpp `-DGGML_VULKAN=ON` at pin `237ad9b96` on dgx, `llama-bench` on the same GGUF, record three columns (ours-Vulkan, llama.cpp-Vulkan, ours-CUDA). Blocked by `VK-B` and by `glslc` on the gate box | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/STATUS.md b/docs/STATUS.md index 17f5ecd6..07a0c586 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -60,7 +60,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror; CPU-only -Werror test-guard fixes x2 | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 bench RAN; FA2 GQA-swap default-ON, c2-c8 <1.0x. `FLASH-OCCUPANCY` #75: occ IDENTICAL, gap=ptxas SASS quality, no lever. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror; CPU-only -Werror test-guard fixes x2 | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 bench RAN; FA2 GQA-swap default-ON, c2-c8 <1.0x. `FLASH-PTXAS` #82: codegen at PARITY (no ptxas lever); gap=engine context. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling |