perf(fa2): flip decode GQA group-swap default ON — MXFP4 c2-c8 up, no regression (KERNEL-FA2-GQA-SWAP-FLIP) - #49
Closed
localai-bot wants to merge 1 commit into
Closed
Conversation
… regression (KERNEL-FA2-GQA-SWAP-FLIP) Flip campaign for VT_FA2_DECODE_GQA_SWAP (the #47-localized / #48-landed FA2 decode seqlenq_ngroups_swapped group-swap). Fa2DecodeGqaSwapEnabled() default OFF->ON in cuda_paged_attn.cu; `=0` opts out (mirrors VT_V4_MHC_FUSED). Flipped per parity-enablers-ship-as-defaults: correctness holds AND speed improves with no regression. CORRECTNESS (GB10 sm_121a, HEAD 1f446fd clean checkout, swap symbol verified in server + libvllm.so): op varlen-d128 swap 5/5 / 280/280; SACRED test_qwen3_paged_engine (Qwen3-0.6B 16/8 + 4B 32/8) 16/16 both / 184/184 at swap-OFF, swap-ON(=1), NEW DEFAULT(no-env) and opt-out(=0) — all CHARACTER- IDENTICAL (11 strict + 5 near-tie, 0 forward-divergent; swap flips no token e2e); #44 MXFP4-8B smoke 3/3 token-exact at the new default. MECHANISM (ours nsys c2 decode-window): decode-flash grid (1,3,64)=batch x q_heads(32) -> (1,5,16)=batch x kv_heads(8), 192->80 CTAs, per-call 63.7->45.3us; default no-env reproduces the (1,5,16) swap grid (the definitive flipped-default proof). Marlin decode ~flat. BINDING (#45-contract grid, swap-ON OURS legs, oracle VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel, c1/c2/c4/c8 x3, drop_caches+ mincore, RelWithDebInfo+oracle-cutlass, vLLM 0.25.0): total tok/s c1 0.990 / c2 0.922 / c4 0.930 / c8 0.942 (was 0.989/0.911/0.919/0.913); TPOT c2 0.912 / c8 0.924 (was 0.900/0.891); median TTFT at/above parity; peak GPU mem 2.614x LESS. c2-c8 gains OUTSIDE per-rep noise (each ours-rep beats every swap-OFF rep); c1 flat; NO regression on any axis. PARITY VERDICT (MXFP4 goal): still BELOW-FLOOR <1.0x on tput/TPOT (best c8 0.942); flash closed ~12%@c2 / ~30%@c8 of the tput gap. Residual (named, per-shape): grouped-Marlin decode +7-9% (E1 indirect gather vs vLLM dense direct-A) + ~0.7ms host/sched. No single lever >=1.0x; next lever = grouped->dense-direct-A marlin. Docs same change: STATUS, BENCHMARKS, FEATURES, ENVIRONMENT (default flip), kernel-matrix, quantization-matrix, benchmark-record (full per-rep + nsys), NOW, state (anchored). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
marked this pull request as ready for review
August 6, 2026 04:15
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…side noise (#49) row/KERNEL-FA2-GQA-SWAP-FLIP squash. Fa2DecodeGqaSwapEnabled OFF->ON (VT_FA2_DECODE_GQA_SWAP=0 opts out). Correctness: SACRED 0.6B+4B 16/16 each across FOUR arms (off/on/new-default/opt-out) character-identical; op 5/5-280; #44 smoke 3/3 token-exact at the new default. Mechanism proof at no-env: decode-flash grid (1,5,16)=batch x kv_heads, 80 CTAs 1 wave, per-call 63.7->45.3us (-29%). Binding vs #45: c2 0.911->0.922, c4 0.919->0.930, c8 0.913->0.942 (worst-ON rep > best-OFF rep each), c1 flat, TTFT parity+, memory 2.614x LESS. MXFP4 verdict still BELOW (best 0.942): named residual = grouped-Marlin E=1 +7-9%/call (indirect gather + fp32 C_tmp vs vLLM dense direct-A) + ~0.7ms/step host; next lever = dense-direct-A marlin decode. Ops note: clean checkout beats a file overlay for SACRED bindings (stale-binary trap recorded). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…0.925-0.953 (#51) row/QUANT-CT-MXFP4-CLOSERS squash. DenseAlignFor block=8 at M<=8 (vLLM's m_block_size_8 tile, kernels already vendored) + one-time workspace zeroing (self-reset invariant cited marlin_template.h:2170/:204), both BYTE-EXACT (bitdiff 0/32768, MXFP4+NVFP4) -> default-ON, gate-model twin untouched. Binding at d3b412f: c1 PASSES every axis (1.005/1.002/1.034/ 1.004); c2-c8 0.925/0.939/0.953 tput, up on every axis vs #49, no regression; host-mem 2.18x LESS. VT_LOOP_TRACE attribution: engine-core decode loop interval==step to 0.02ms -> the ~0.7ms host slice is NOT an engine lever (frontend/boundary). Residual map final: only the grouped E=1 indirection remains as a GPU term, per-shape-parity at M<=8 post block-fix. Parity gate NO for batched; c1 + memory WIN recorded. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KERNEL-FA2-GQA-SWAP-FLIP — flip
VT_FA2_DECODE_GQA_SWAPdefault ONThe FA2 decode GQA
seqlenq_ngroups_swappedgroup-swap (ported #47/#48, gated OFF) is flipped default-ON per parity-enablers-ship-as-defaults. One-line default flip incuda_paged_attn.cu(Fa2DecodeGqaSwapEnabled()OFF→ON;=0opts out) plus the same-change record/doc updates.Flip conditions — all met (GB10 sm_121a, clean checkout of
1f446fd7)test_qwen3_paged_engine(Qwen3-0.6B 16/8 + 4B 32/8) 16/16 both / 184/184 at swap-OFF, swap-ON(=1), NEW DEFAULT (no-env), and opt-out(=0) — all character-identical (11 strict + 5 near-tie, 0 forward-divergent; the swap flips no token e2e); row/SERVE-ASYNC-DENSE-MIRROR: classic-dense async device-mirror fix + MXFP4 default-config e2e CLOSED #44 MXFP4-8B smoke 3/3 token-exact at the new default.(1,3,64)=b×q_heads(32)→(1,5,16)=b×kv_heads(8), 192→80 CTAs, per-call 63.7→45.3us; the no-env default reproduces the swap grid (definitive flipped-default proof).VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel, c1/c2/c4/c8 ×3, drop_caches+mincore, RelWithDebInfo + oracle-cutlass, vLLM 0.25.0):c2-c8 gains are outside per-rep noise (each ours-rep beats every swap-OFF rep); c1 flat; no regression on any axis.
Parity verdict (MXFP4 goal)
Still BELOW-FLOOR <1.0x on tput/TPOT (best c8 0.942). Flash closed ~12%@c2 / ~30%@c8 of the tput gap. Residual (named, per-shape): grouped-Marlin decode +7-9% (E1 indirect gather vs vLLM dense direct-A) + ~0.7ms host/sched. No single lever reaches ≥1.0x; next lever = grouped→dense-direct-A marlin decode.
Evidence:
dgx:~/work/vllm.cpp-online-gate/evidence/1f446fd7…+~/mxfp4-nsys. Full per-rep table + nsys in.agents/benchmark-record.md; handoff in.agents/state.md(KERNEL-FA2-GQA-SWAP-FLIP).🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys