From b7657b1441c34d67f62aafd5b2ec3786e3cfb702 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 04:10:31 +0000 Subject: [PATCH] =?UTF-8?q?perf(fa2):=20flip=20decode=20GQA=20group-swap?= =?UTF-8?q?=20default=20ON=20=E2=80=94=20MXFP4=20c2-c8=20up,=20no=20regres?= =?UTF-8?q?sion=20(KERNEL-FA2-GQA-SWAP-FLIP)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flip campaign for VT_FA2_DECODE_GQA_SWAP (the #47-localized / #48-landed FA2 decode seqlenq_ngroups_swapped group-swap). Fa2DecodeGqaSwapEnabled() default OFF->ON in cuda_paged_attn.cu; `=0` opts out (mirrors VT_V4_MHC_FUSED). Flipped per parity-enablers-ship-as-defaults: correctness holds AND speed improves with no regression. CORRECTNESS (GB10 sm_121a, HEAD 1f446fd7 clean checkout, swap symbol verified in server + libvllm.so): op varlen-d128 swap 5/5 / 280/280; SACRED test_qwen3_paged_engine (Qwen3-0.6B 16/8 + 4B 32/8) 16/16 both / 184/184 at swap-OFF, swap-ON(=1), NEW DEFAULT(no-env) and opt-out(=0) — all CHARACTER- IDENTICAL (11 strict + 5 near-tie, 0 forward-divergent; swap flips no token e2e); #44 MXFP4-8B smoke 3/3 token-exact at the new default. MECHANISM (ours nsys c2 decode-window): decode-flash grid (1,3,64)=batch x q_heads(32) -> (1,5,16)=batch x kv_heads(8), 192->80 CTAs, per-call 63.7->45.3us; default no-env reproduces the (1,5,16) swap grid (the definitive flipped-default proof). Marlin decode ~flat. BINDING (#45-contract grid, swap-ON OURS legs, oracle VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel, c1/c2/c4/c8 x3, drop_caches+ mincore, RelWithDebInfo+oracle-cutlass, vLLM 0.25.0): total tok/s c1 0.990 / c2 0.922 / c4 0.930 / c8 0.942 (was 0.989/0.911/0.919/0.913); TPOT c2 0.912 / c8 0.924 (was 0.900/0.891); median TTFT at/above parity; peak GPU mem 2.614x LESS. c2-c8 gains OUTSIDE per-rep noise (each ours-rep beats every swap-OFF rep); c1 flat; NO regression on any axis. PARITY VERDICT (MXFP4 goal): still BELOW-FLOOR <1.0x on tput/TPOT (best c8 0.942); flash closed ~12%@c2 / ~30%@c8 of the tput gap. Residual (named, per-shape): grouped-Marlin decode +7-9% (E1 indirect gather vs vLLM dense direct-A) + ~0.7ms host/sched. No single lever >=1.0x; next lever = grouped->dense-direct-A marlin. Docs same change: STATUS, BENCHMARKS, FEATURES, ENVIRONMENT (default flip), kernel-matrix, quantization-matrix, benchmark-record (full per-rep + nsys), NOW, state (anchored). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 4 +- .agents/benchmark-record.md | 77 ++++++++++++++++++++++++++++++++++ .agents/kernel-matrix.md | 42 +++++++++++-------- .agents/quantization-matrix.md | 2 +- .agents/state.md | 54 ++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/ENVIRONMENT.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- src/vt/cuda/cuda_paged_attn.cu | 19 ++++++--- 10 files changed, 176 insertions(+), 30 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 464e878d..458ca630 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 c2-c8 lever = FA2 decode GQA group-swap | **Ported gated OFF** (`VT_FA2_DECODE_GQA_SWAP`): vLLM decode swap in the d128 varlen launcher; correctness-complete GB10 (op RED/GREEN, memcheck, swap-ON smoke token-exact), OFF byte-identical | bench + flip: state `KERNEL-FA2-GQA-SWAP` | +| MXFP4 GQA-swap `KERNEL-FA2-GQA-SWAP-FLIP` | **FLIPPED DEFAULT-ON** (`VT_FA2_DECODE_GQA_SWAP`, `=0` off): SACRED 0.6B/4B 16/16 token-identical; nsys grid (1,3,64)→(1,5,16); binding tput c2-c8 →0.922/0.930/0.942 (c1 flat, no reg), mem 2.614x | push branch | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| QUANT-CT-MXFP4-BENCH | ~0.91x@c2-c8; marlin+glue levers **both REFUTED**; step = flash GQA group-swap | LEVER + detail: state `QUANT-CT-MXFP4-GLUE` (`cuda_flash_attn_fa2.cu:1075`) | +| MXFP4 parity goal | swap-ON binding **still <1.0x** (best c8 0.942); residual grouped-Marlin +7-9% + ~0.7ms host/sched | NEXT: grouped→dense-direct-A marlin decode | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index ae86c8e8..0083fcd9 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12413,3 +12413,80 @@ c2 28.27→25.45). #47 projects flash alone closes ~28%@c2 / ~55%@c8 (decode fla so a single lever may not reach ≥1.0x — record the honest per-axis outcome. Then Qwen3 0.6B/4B e2e SACRED swap-ON, then flip per parity-enablers. Box left clean (both locks free, GPU idle, worker down, disk 22G). + +## QUANT-CT-MXFP4 flip campaign — FA2 decode GQA group-swap FLIPPED DEFAULT-ON; binding improves c2-c8 with NO regression; MXFP4 goal still <1.0x (2026-08-06, `row/KERNEL-FA2-GQA-SWAP-FLIP`, GB10, HEAD `1f446fd7`+flip) + +Ran the full flip campaign for `VT_FA2_DECODE_GQA_SWAP` (#47 lever, #48 correctness-complete +gated-OFF): model-level e2e SACRED, the #45-contract binding re-bench swap-ON, the ours-only +nsys mechanism confirm, and the default decision. **Result: FLIPPED DEFAULT-ON.** Box left +clean (both locks free, GPU idle, worker down, disk 21G). + +BUILD PROVENANCE (the trap that cost the first grid attempt, recorded): the box +`~/mxfp4-bench/src` tree was git `33e93608` (#45) with the 4 #48 files OVERLAID (dirty), and +the built server binary was STALE (lacked the `VT_FA2_DECODE_GQA_SWAP` getenv string; the +"ninja: no work to do" rebuild had been fooled by the overlay's preserved mtimes). The grid +driver's `--execute` REQUIRES `git HEAD == --vllm-cpp-sha` AND a clean tree (dgx-online-serving.sh +:168,:172), so the overlay fails both. FIX: `git fetch origin 1f446fd7` + `git checkout -f +1f446fd7` (the overlay was byte-identical to HEAD, so no code changed) → clean tree at the +exact sha; rebuild (swap string verified present in `examples/server` AND `libvllm.so.0.0.1`). +Clean full-tree provenance beats an overlay for a SACRED binding. + +CORRECTNESS (fresh 1f446fd7+flip binary, GB10 CUDA 13.0 sm_121a, GPU-locked): +- op `test_ops_paged_attn --test-case="*varlen d128*"`: **5/5 cases, 280/280 assn GREEN**. +- SACRED `test_qwen3_paged_engine` (Qwen3-0.6B 16/8 + 4B 32/8), ALL arms **16/16 both, 184/184**: + swap-OFF baseline, swap-ON (`=1`), NEW DEFAULT (no env), and opt-out (`=0`) are CHARACTER- + IDENTICAL — same 11/16 strict + 5/16 near-tie split, same max-gap positions (0.6B 0.125 nats + @p5t11, 4B 0.25 nats @p2t11), 0 forward-divergent. The swap flips NO token e2e at these + lengths; the near-tie razor was never needed. +- #44 MXFP4-8B smoke at the new default: deterministic **3/3 TOKEN-EXACT** + near-tie coherent. +- In-grid smoke gate (swap-ON) passed (grid ran all 24 legs). + +NSYS MECHANISM (ours-only c2 decode-window, `--cuda-graph-trace=node`, 24×128, dominant +decode-flash by launch frequency; aggregation-trap-separated): +| arm | decode-flash grid | gridZ | CTAs | per-call Med | +|-----|-------------------|-------|------|--------------| +| swap OFF (baseline) | **(1,3,64)** | 64 = batch(2)×**q_heads(32)** | 192 | **63.68us** | +| swap ON (`=1`) | **(1,5,16)** | 16 = batch(2)×**kv_heads(8)** | 80 | **45.31us** | +| NEW DEFAULT (no env)| **(1,5,16)** | 16 = batch(2)×kv_heads(8) | 80 | 45.6us | +The default no-env run reproduces the swap grid → the flip engages the swap by default (the +definitive flipped-default proof, since SACRED output is identical for both defaults). Marlin +decode ~flat (120.7→118.4us). Per-call drop −28.8% (below #47's projected 41.7us / −34%; ours +num_splits picks 5 vs vLLM's 6, both ~1 wave). + +BINDING GRID (swap-ON on OURS legs, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, +c1/c2/c4/c8 ×3 interleaved, single-load/arm, drop_caches+mincore proof, RelWithDebInfo + oracle +flashinfer-cutlass, oracle vLLM 0.25.0; evidence +`dgx:~/work/vllm.cpp-online-gate/evidence/1f446fd7fb125284e6c4ba44b1e6dd82b99c084c`): +| conc | total tok/s ours→vllm (ratio) | out tok/s r | med TPOT ms ours→vllm (r) | med TTFT (r) | +|------|------------------------------|-------------|--------------------------|--------------| +| c1 | 325.0 → 328.2 (**0.990**) | 0.990 | 25.65 → 25.32 (0.987) | 285.6 → 294.8 (**1.032** PASS) | +| c2 | 575.9 → 624.4 (**0.922**) | 0.922 | 27.97 → 25.52 (0.912) | 444.8 → 446.2 (**1.003** PASS) | +| c4 | 964.1 → 1036.3 (**0.930**) | 0.930 | 30.82 → 28.09 (0.911) | 864.1 → 864.5 (**1.000** PASS) | +| c8 | 1464.0 → 1553.9 (**0.942**) | 0.942 | 38.23 → 35.31 (0.924) | 1413.9 → 1411.2 (0.998) | +| memory | peak GPU 28210 → 73723 MiB = **2.614 PASS** (ours 2.6x LESS) | — | — | — | +Per-rep ours total tok/s (tight, CoV 0.06–0.28%): c1 324.4/325.0/326.6; c2 575.4/575.9/576.2; +c4 961.1/964.1/965.9; c8 1462.9/1464.0/1468.7. Per-rep ours median TPOT ms: c1 25.53/25.65/25.71; +c2 27.88/27.97/27.99; c4 30.71/30.82/31.69; c8 38.01/38.23/38.24. + +DELTA vs #45 (swap-OFF, `33e93608`): tput c1 0.989→0.990 (+0.001 FLAT), c2 0.911→0.922 (+0.011), +c4 0.919→0.930 (+0.011), c8 0.913→0.942 (+0.029); TPOT c1 0.986→0.987, c2 0.900→0.912, c4 +0.905→0.911, c8 0.891→0.924. The c2/c4/c8 gains are OUTSIDE per-rep noise — swap-ON's WORST rep +beats swap-OFF's BEST rep at every one (c2 575.4>569.9, c4 961.1>958.9, c8 1462.9>1433.9); c1 +overlaps (flat). TTFT at/above parity (prefill-only, mechanically untouched by the decode swap). +Memory win preserved (2.607→2.614x). + +DEFAULT DECISION = **FLIP ON** (parity-enablers-ship-as-defaults): every correctness gate holds, +speed IMPROVES at c2/c4/c8, and there is NO regression anywhere (c1 flat, TTFT parity, memory +win). `Fa2DecodeGqaSwapEnabled()` default OFF→ON (`cuda_paged_attn.cu`; `=0` opts out, mirroring +`VT_V4_MHC_FUSED`). Flipped-default proof above (SACRED no-env 16/16 + smoke token-exact + nsys +no-env grid (1,5,16)). + +PARITY VERDICT (MXFP4 goal) = **still BELOW-FLOOR <1.0x on tput/TPOT at every concurrency** (best +c8 tput 0.942, 5.8% short); gate NO. RESIDUAL MAP (per-shape, named): flash closed only ~12%@c2 / +~30%@c8 of the tput gap (below #47's optimistic 28/55% — actual flash drop was −29% not the +projected −34%, and flash is one term of the step). What remains at c2 (per #46/#47 same-tool +attribution): (1) **grouped-Marlin decode +7-9% per-call** — our `MoeGroupedGemmNvfp4Marlin` E=1 +uses indirect `sorted_token_ids` gather + fp32 `C_tmp` reduce vs vLLM's dense `marlin_gemm` +direct-A addressing; per-shape parity was shown at M≤8 (#46 microbench), so this is a delicate +grouped→dense-direct-A port, not a config knob. (2) **~0.7ms/step host/sched slice**. No single +lever reaches ≥1.0x. NEXT lever candidate: the grouped→dense-direct-A marlin decode path. diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index 03d41037..7b8cfac9 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -85,24 +85,30 @@ trace-proven at **180.28 us/call**, within 1.1% of the matched vLLM FA2 kernel at 178.40 us/call. Evidence: [2026-07-25 4B repair](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md). -`KERNEL-ATTN-FA2` d128 varlen decode GQA group-swap (2026-08-06, -`row/KERNEL-FA2-GQA-SWAP`, `VT_FA2_DECODE_GQA_SWAP`, default OFF): vLLM's -`seqlenq_ngroups_swapped` decode grid ported into `LaunchDecodeVarlenFA2Bf16` -(`cuda_flash_attn_fa2.cu`) so the Qwen3-dense d128 decode launches -(batch, kv_heads) not (batch, hq) — the ngroups query heads pack into seqlen_q, -KV read once per group, heuristic sees `batch*kv_heads` — halving the CTA count -at batch>=2 (#47: ours over-waved 192 CTAs c2 / ~5 waves c8). Presented WITHOUT a -materialized transpose via kv-major-group-minor strides, mirroring the shipped -d256 `LaunchDecodeFA2Bf16` swap (the vendored `get_lse_tile`/combine already -honor the flag). Gated so OFF is byte-identical to the shipped plain-varlen -reduction; ON is non-byte-exact only when num_splits>1 (split reduction order -→ near-tie, toward vLLM). Correctness-complete on GB10 (CUDA 13.0, sm_121a): -op RED-first test **280/280 assn GREEN** (both ratios 16/8+32/8, batch 1/2/4/8, -short+long context; `swap_launches==1` proves the grid engaged; wrong-stride RED -= 26528 violations), compute-sanitizer **0-err/0-leak**, and the #44 MXFP4 e2e -smoke swap-ON **3/3 deterministic TOKEN-EXACT + coherent**, byte-identical to -swap-OFF (no token flip, near-tie razor unneeded). c1-c8 x3 binding re-bench + -default flip is the recorded next step. Detail: state `KERNEL-FA2-GQA-SWAP`. +`KERNEL-ATTN-FA2` d128 varlen decode GQA group-swap (ported 2026-08-06 +`row/KERNEL-FA2-GQA-SWAP`; **FLIPPED DEFAULT-ON 2026-08-06 `row/KERNEL-FA2-GQA-SWAP-FLIP`**, +`VT_FA2_DECODE_GQA_SWAP`, `=0` opts out): vLLM's `seqlenq_ngroups_swapped` decode +grid ported into `LaunchDecodeVarlenFA2Bf16` (`cuda_flash_attn_fa2.cu`) so the +Qwen3-dense d128 decode launches (batch, kv_heads) not (batch, hq) — the ngroups +query heads pack into seqlen_q, KV read once per group, heuristic sees +`batch*kv_heads` — halving the CTA count at batch>=2 (#47: ours over-waved 192 CTAs +c2 / ~5 waves c8). Presented WITHOUT a materialized transpose via +kv-major-group-minor strides, mirroring the shipped d256 `LaunchDecodeFA2Bf16` swap +(the vendored `get_lse_tile`/combine already honor the flag). Gated so `=0` is +byte-identical to the shipped plain-varlen reduction; ON is non-byte-exact only when +num_splits>1 (split reduction order → near-tie, toward vLLM). **FLIP CAMPAIGN on +GB10 (CUDA 13.0, sm_121a, HEAD 1f446fd7):** op RED-first test **280/280 assn GREEN** +(both ratios, batch 1/2/4/8, short+long; `swap_launches==1`; wrong-stride RED = +26528 violations), compute-sanitizer **0-err/0-leak**; **SACRED Qwen3-0.6B/4B greedy +near-tie gate 16/16 both, TOKEN-IDENTICAL swap-ON vs plain (no token flip)** at +default, `=1`, and `=0`; #44 MXFP4 smoke token-exact at the new default. nsys ours c2 +decode-flash grid **(1,3,64)=b×q_heads → (1,5,16)=b×kv_heads**, per-call **63.7→45.3us** +(default no-env reproduces the swap grid). Binding q3mxfp4 grid swap-ON total tok/s +c1 **0.990** / c2 **0.922** / c4 **0.930** / c8 **0.942** (was 0.989/0.911/0.919/0.913, +c2-c8 outside per-rep noise, c1 flat, NO regression), median TTFT at/above parity, +peak GPU mem **2.614x LESS**. Flipped per parity-enablers (improves, no regression, +correctness holds). Still <1.0x tput/TPOT — residual grouped-Marlin +7-9% + ~0.7ms +host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner | |---|---|---|---|---|---|---|---| diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index 753289b3..c31f897e 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -127,7 +127,7 @@ Registry source: | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - | -| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. W4 THROUGHPUT bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH` `33e93608`, c1/c2/c4/c8x3, production graphed vLLM 0.25.0, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` -> Marlin W4A16): **BELOW-FLOOR** (gate NO). Total tok/s ours/vLLM 0.99x@c1 but ~0.91x@c2-c8 (c2 0.911, c4 0.919, c8 0.913); driven by TPOT +10-12%@c2-c8 (parity@c1); TTFT at parity; peak GPU mem **2.6x LESS** (28.3 vs 73.7 GiB). ATTRIBUTION (grounded, per-concurrency curve): the gap is a BATCHED-decode cost absent at batch-1 -> the grouped Marlin W4A16 decode GEMM at M=2..8, NOT batch-1 GEMV nor prefill; NOT a ceiling. Refined by nsys c1-vs-c2 (`row/QUANT-CT-MXFP4-GLUE`): the c2-c8 STEP is the FLASH decode launch (ours grid `(batch,hq)` over-waves at batch>=2; vLLM applies the FA2 `seqlenq_ngroups_swapped` decode swap → `(batch,kv_heads)`), NOT the marlin GEMM (per-shape parity) nor glue. **Lever LANDED gated default-OFF (`row/KERNEL-FA2-GQA-SWAP`, `VT_FA2_DECODE_GQA_SWAP`): vLLM's decode group-swap ported into the d128 varlen launcher, correctness-complete on GB10 (op RED/GREEN 280 assn, memcheck 0-err, #44 smoke swap-ON 3/3 det TOKEN-EXACT byte-identical to OFF); OFF byte-identical. c1-c8 x3 re-bench + default flip = next step (#47: flash ~28%@c2 / ~55%@c8 of the gap).** NEXT: same-tool nsys decode-window on c8 both engines. Harness (all additive; 27/35 byte-unchanged; CPU tests 45/45): `online_gate` `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`, [`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py) (#44 battery), `dgx-online-serving.sh` q3mxfp4 branches, [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh). Full table + build contract (RelWithDebInfo + oracle flashinfer-cutlass + DISK build) in benchmark-record. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | +| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. W4 THROUGHPUT bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH` `33e93608`, c1/c2/c4/c8x3, production graphed vLLM 0.25.0, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` -> Marlin W4A16): **BELOW-FLOOR** (gate NO). Total tok/s ours/vLLM 0.99x@c1 but ~0.91x@c2-c8 (c2 0.911, c4 0.919, c8 0.913); driven by TPOT +10-12%@c2-c8 (parity@c1); TTFT at parity; peak GPU mem **2.6x LESS** (28.3 vs 73.7 GiB). ATTRIBUTION (grounded, per-concurrency curve): the gap is a BATCHED-decode cost absent at batch-1 -> the grouped Marlin W4A16 decode GEMM at M=2..8, NOT batch-1 GEMV nor prefill; NOT a ceiling. Refined by nsys c1-vs-c2 (`row/QUANT-CT-MXFP4-GLUE`): the c2-c8 STEP is the FLASH decode launch (ours grid `(batch,hq)` over-waves at batch>=2; vLLM applies the FA2 `seqlenq_ngroups_swapped` decode swap → `(batch,kv_heads)`), NOT the marlin GEMM (per-shape parity) nor glue. **Lever FLIPPED DEFAULT-ON (`row/KERNEL-FA2-GQA-SWAP-FLIP`, `VT_FA2_DECODE_GQA_SWAP`, `=0` opts out): vLLM's decode group-swap in the d128 varlen launcher. Flip campaign on GB10 (HEAD `1f446fd7`): op 280 assn GREEN + memcheck 0-err; SACRED Qwen3-0.6B/4B greedy near-tie gate 16/16 both, TOKEN-IDENTICAL swap-ON vs plain at default/`=1`/`=0` + #44 smoke token-exact at the new default; nsys ours c2 decode-flash grid (1,3,64)=b×q_heads → (1,5,16)=b×kv_heads, per-call 63.7→45.3us (default no-env reproduces the swap grid). Binding q3mxfp4 grid swap-ON total tok/s c1 0.990 / c2 0.922 / c4 0.930 / c8 0.942 (was 0.989/0.911/0.919/0.913; c2-c8 gains OUTSIDE per-rep noise — each ours-rep beats every swap-OFF rep — c1 flat, NO regression), median TTFT at/above parity, peak GPU mem 2.614x LESS. Flipped per parity-enablers (improves + no regression + correctness holds). Still <1.0x on tput/TPOT: flash closed ~12%@c2 / ~30%@c8 of the gap (below #47's ~28/55% projection since actual flash drop was −29% not −34%); residual = grouped-Marlin decode +7-9% (E1 indirect gather vs vLLM dense direct-A) + ~0.7ms host/sched. No single lever ≥1.0x.** Harness (all additive; 27/35 byte-unchanged; CPU tests 45/45): `online_gate` `q3mxfp4` key + `POINTS_BY_MODEL`/`points_for`, [`mxfp4_smoke_gate.py`](../tools/bench/mxfp4_smoke_gate.py) (#44 battery), `dgx-online-serving.sh` q3mxfp4 branches, [`mxfp4-online-serving-grid.sh`](../scripts/mxfp4-online-serving-grid.sh). Full table + build contract (RelWithDebInfo + oracle flashinfer-cutlass + DISK build) in benchmark-record. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | diff --git a/.agents/state.md b/.agents/state.md index bb00114b..c432febc 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -37001,3 +37001,57 @@ single lever may not reach ≥1.0x — record the honest per-axis outcome either smoke suggests it won't at short contexts). (3) then flip default per (f). Evidence: dgx smoke `~/smoke.log` (transient), op/memcheck run inline (above). Box: both locks free, GPU idle, disk 22G, local-ai-worker left down, tmux clean. + +## KERNEL-FA2-GQA-SWAP-FLIP: the #48 lever FLIPS DEFAULT-ON — flip campaign COMPLETE on GB10 (SACRED 0.6B/4B 16/16 token-identical, nsys grid (1,3,64)→(1,5,16), binding c2-c8 improves NO-regression); MXFP4 goal still <1.0x, residual named + + +Executed the full flip campaign for `VT_FA2_DECODE_GQA_SWAP` (the #47-localized / #48-landed +FA2 decode GQA group-swap) on `row/KERNEL-FA2-GQA-SWAP-FLIP` off `origin/main` `1f446fd7`. +**DECISION: FLIPPED DEFAULT-ON.** Every flip condition met; MXFP4 parity goal still BELOW-FLOOR +<1.0x with a NAMED residual. Box left clean (both locks free, GPU idle, worker down, disk 21G). + +BUILD PROVENANCE (trap, cost the first grid attempt): box `~/mxfp4-bench/src` was git `33e93608` +(#45) + the 4 #48 files OVERLAID (dirty) and the server binary was STALE (no swap getenv string; +the 02:56 "ninja: no work to do" was mtime-fooled). The grid driver's `--execute` requires +`git HEAD == --vllm-cpp-sha` AND clean tree (dgx-online-serving.sh:168,:172), which the overlay +fails. FIX: `git fetch origin 1f446fd7` + `git checkout -f 1f446fd7` (overlay was byte-identical +→ no code changed) → clean tree at the exact sha; rebuild; swap string verified in +`examples/server` + `libvllm.so.0.0.1`. Lesson: for a SACRED binding, a clean full-tree checkout +at the sha beats a dirty overlay — the driver enforces it. + +CORRECTNESS (fresh 1f446fd7+flip binary, GPU-locked): op varlen-d128 swap **5/5·280/280 GREEN**; +SACRED `test_qwen3_paged_engine` (0.6B 16/8 + 4B 32/8) **16/16 both / 184/184** at swap-OFF, +swap-ON(`=1`), NEW DEFAULT(no-env), AND opt-out(`=0`) — all CHARACTER-IDENTICAL (11 strict + 5 +near-tie, same 0.125/0.25-nat max-gap positions, 0 forward-divergent): the swap flips NO token +e2e, near-tie razor unneeded. #44 MXFP4-8B smoke at new default **3/3 TOKEN-EXACT**. + +NSYS MECHANISM (ours c2 decode-window, dominant decode-flash by frequency): swap-OFF grid +**(1,3,64)** [gridZ 64 = batch×q_heads(32)], 192 CTAs, **63.68us**; swap-ON `=1` **(1,5,16)** +[gridZ 16 = batch×kv_heads(8)], 80 CTAs, **45.31us**; NEW DEFAULT no-env reproduces **(1,5,16)** / +45.6us — the definitive flipped-default proof (SACRED output is identical for both defaults, so +the grid is the discriminator). Marlin decode ~flat (120.7→118.4us). Per-call −28.8% (below #47's +projected 41.7us; ours num_splits 5 vs vLLM 6, both ~1 wave). + +BINDING GRID (swap-ON OURS legs, oracle `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`, +c1/c2/c4/c8×3, single-load/arm, drop_caches+mincore, RelWithDebInfo+oracle-cutlass, vLLM 0.25.0; +evidence `dgx:~/work/vllm.cpp-online-gate/evidence/1f446fd7…`): +total tok/s ratio c1 **0.990** / c2 **0.922** / c4 **0.930** / c8 **0.942** (was +0.989/0.911/0.919/0.913); median TPOT ratio c1 0.987 / c2 0.912 / c4 0.911 / c8 0.924 (was +0.986/0.900/0.905/0.891); median TTFT c1 1.032 / c2 1.003 / c4 1.000 / c8 0.998 (PASS c1-c4); +peak GPU mem **2.614x LESS** (28210 vs 73723 MiB). Per-rep tight (CoV 0.06-0.28%); c2/c4/c8 gains +are OUTSIDE noise — swap-ON's WORST rep > swap-OFF's BEST rep at each (c2 575.4>569.9, c4 +961.1>958.9, c8 1462.9>1433.9); c1 overlaps (flat). Δ vs #45: tput +0.001/+0.011/+0.011/+0.029. + +DEFAULT DECISION = FLIP ON (parity-enablers): correctness holds + speed improves c2-c8 + NO +regression (c1 flat, TTFT parity, memory win). `Fa2DecodeGqaSwapEnabled()` OFF→ON in +`cuda_paged_attn.cu` (`=0` opts out, mirroring `VT_V4_MHC_FUSED`). Docs updated same change: +STATUS, BENCHMARKS (post-swap binding table), FEATURES, kernel-matrix, quantization-matrix, +ENVIRONMENT (default flip), benchmark-record (full per-rep + nsys), NOW. + +PARITY VERDICT (MXFP4 goal) = still BELOW-FLOOR <1.0x on tput/TPOT everywhere (best c8 0.942, +5.8% short); gate NO. Flash closed only ~12%@c2 / ~30%@c8 of the tput gap (below #47's 28/55%: +actual flash drop −29% not −34%, and flash is one term). RESIDUAL MAP (per-shape, named): (1) +grouped-Marlin decode **+7-9% per-call** — `MoeGroupedGemmNvfp4Marlin` E=1 indirect +`sorted_token_ids` gather + fp32 `C_tmp` vs vLLM dense `marlin_gemm` direct-A (per-shape parity +at M≤8 per #46 → a delicate grouped→dense-direct-A port, not a config knob); (2) **~0.7ms/step +host/sched**. No single lever ≥1.0x. NEXT lever candidate = grouped→dense-direct-A marlin decode. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b5055d5a..03c3ca89 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED. W4 grid: 0.99x@c1 / ~0.91x@c2-c8, mem 2.6x LESS. c2-c8 lever = FA2 decode GQA group-swap **LANDED gated-OFF** (`VT_FA2_DECODE_GQA_SWAP`); bench+flip pending. state `KERNEL-FA2-GQA-SWAP` | +| MXFP4 Qwen3-8B (W4A16 Marlin) | W4A16 proven; e2e 3/4 token-exact. FA2 GQA group-swap **FLIPPED DEFAULT-ON** (`KERNEL-FA2-GQA-SWAP-FLIP`): binding tput c1 0.990/c2 0.922/c4 0.930/c8 0.942 (c1 flat, no reg), mem 2.614x; SACRED 0.6B/4B identical | Still <1.0x tput/TPOT: residual grouped-Marlin +7-9% (E1 gather vs dense direct-A) + ~0.7ms host/sched. Full table in benchmark-record; state `KERNEL-FA2-GQA-SWAP-FLIP` | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 2c2369d3..5135aaf1 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -71,7 +71,7 @@ portable/reference path. In normal operation leave them unset. | `VT_FA2_PREFILL` | on (CUDA) | The portable prefill attention instead of the vendored FA2 | | `VT_FA2_DECODE` | on (CUDA) | The portable decode attention instead of the vendored FA2 | | `VT_FA2_DECODE_4B` | on (CUDA, Qwen3.5-4B) | The portable paged decode attention instead of the ratio-4 vendored FA2 path; the 27B and 35B selectors are unchanged | -| `VT_FA2_DECODE_GQA_SWAP` | off (CUDA, Qwen3-dense d128 decode) | `=1` applies vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization in the d128 varlen launcher (launches (batch, kv_heads) not (batch, hq), packing the ngroups query heads into seqlen_q); default off is byte-identical to the plain-varlen reduction, on is a split-reduction near-tie | +| `VT_FA2_DECODE_GQA_SWAP` | **on** (CUDA, Qwen3-dense d128 decode) | Applies vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization in the d128 varlen launcher (launches (batch, kv_heads) not (batch, hq), packing the ngroups query heads into seqlen_q); `=0` opts out and restores the byte-identical plain-varlen reduction. Flipped default-on 2026-08-06 (`KERNEL-FA2-GQA-SWAP-FLIP`): SACRED Qwen3-0.6B/4B 16/16 token-identical, binding q3mxfp4 tput c2-c8 0.911/0.919/0.913→0.922/0.930/0.942 with no c1 regression, mem 2.614x less | | `VT_CPU_REF` | off | Set on to force the portable reference path (dequantize-everything oracle), the standard "is this a kernel bug?" bisect switch | | `VT_DFLASH_PAGED` | on (CUDA, DFlash spec-decode) | The materialized `[context;block]` draft forward instead of the fixed-capacity paged draft-KV store read through `vt::DFlashPagedBlockAttention` (bit-identical; only the DFlash single-request propose path) | | `VT_DFLASH_GRAPH` | on (CUDA, DFlash spec-decode) | The eager paged draft step instead of the captured/replayed draft-step CUDA graph (replayed==eager bit-identical; only the DFlash single-request propose path) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 0a13b492..2f101524 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -65,7 +65,7 @@ are our reading of their documented behavior, not measurements. | GGUF k-quants and i-quants | ✅ | ☐ | ☐ | ✅ | | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | -| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; e2e 3/4 token-exact; W4 grid GB10 ~0.91x c2-c8 / 0.99x c1, mem 2.6x less. c2-c8 lever = FA2 decode GQA group-swap, LANDED gated-OFF (`VT_FA2_DECODE_GQA_SWAP`); bench+flip pending | ✅ | ✅ | ☐ | +| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; e2e 3/4 token-exact; mem 2.6x less. FA2 GQA group-swap **DEFAULT-ON**: binding tput c2-c8 →0.922/0.930/0.942 (c1 flat, no reg); SACRED 0.6B/4B token-identical; still <1.0x | ✅ | ✅ | ☐ | | fp8 weights | ✅ | ✅ | ✅ | ☐ | | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ | | Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ | diff --git a/docs/STATUS.md b/docs/STATUS.md index a18371ed..a17d813d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -53,7 +53,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench RAN on GB10 (`row/QUANT-CT-MXFP4-BENCH`): BELOW-FLOOR ~0.91x c2-c8, batched-decode gap. Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 bench RAN; FA2 GQA group-swap FLIPPED DEFAULT-ON — binding c2-c8 →0.92-0.94x, still <1.0x. state `KERNEL-FA2-GQA-SWAP-FLIP`. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling | diff --git a/src/vt/cuda/cuda_paged_attn.cu b/src/vt/cuda/cuda_paged_attn.cu index 0da87330..8e10f6c5 100644 --- a/src/vt/cuda/cuda_paged_attn.cu +++ b/src/vt/cuda/cuda_paged_attn.cu @@ -2565,13 +2565,22 @@ bool Fa2DecodeQwen3Enabled() { // launches (batch, hq), over-waving at batch>=2 (#47: decode flash 63.7us c2 / // 218us c8 vs vLLM 41.7/151). This routes the SAME vendored split-KV kernel through // the swapped presentation (already proven by the d256 LaunchDecodeFA2Bf16 arm). -// DEFAULT OFF: it is non-byte-exact vs the plain arm when num_splits>1 (the split -// reduction order changes -> near-tie, moving toward vLLM's numerics) and touches -// the CUDA-graph-captured decode, so it flips ON only after the near-tie token -// gates hold. =1 opts in for a same-binary A/B. Read fresh (host path per step). +// DEFAULT ON (flipped 2026-08-06, row KERNEL-FA2-GQA-SWAP-FLIP, per +// parity-enablers-ship-as-defaults): the flip campaign met every condition on GB10 +// — op test 5/5, Qwen3-0.6B/4B SACRED greedy near-tie gate 16/16 token-IDENTICAL to +// the plain arm swap-ON (the split reduction is a near-tie the model never flips at +// these lengths), #44 MXFP4-8B smoke token-exact, and the binding q3mxfp4 grid +// improved with NO regression: total tok/s c2 0.911->0.922, c4 0.919->0.930, +// c8 0.913->0.942 (each ours-rep beats every swap-OFF rep), c1 flat 0.989->0.990, +// median TTFT at/above parity, peak GPU mem 2.614x LESS. Mechanism (ours nsys c2): +// decode flash grid (1,3,64)=batch x query_heads -> (1,5,16)=batch x kv_heads, +// per-call 63.7->45.3us. It is non-byte-exact vs the plain arm when num_splits>1 +// (split reduction order changes -> near-tie toward vLLM's numerics) and touches the +// CUDA-graph-captured decode, hence gated: =0 opts OUT (restores the byte-identical +// plain-varlen reduction) for a same-binary A/B. Read fresh (host path per step). bool Fa2DecodeGqaSwapEnabled() { const char* e = std::getenv("VT_FA2_DECODE_GQA_SWAP"); - return e != nullptr && e[0] != '0'; + return e == nullptr || e[0] != '0'; } #endif // VLLM_CPP_FLASH_ATTN