row/QUANT-CT-MXFP4-GLUE: glue REFUTED; real c2-c8 lever = FA2 GQA group-swap (ours hard-coded OFF) - #47
Closed
localai-bot wants to merge 1 commit into
Closed
row/QUANT-CT-MXFP4-GLUE: glue REFUTED; real c2-c8 lever = FA2 GQA group-swap (ours hard-coded OFF)#47localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
… = flash decode GQA group-swap Measurement-first localization of the MXFP4 c2-c8 gap (#45/#46 continuation). The binding's prescribed glue-fusion lever is REFUTED by same-tool measurement (vLLM's fused glue ~910us/step >= ours ~776us/step; add+RMSNorm already one kernel). The real c2-c8 step is the FLASH decode launch config: ours iterates 32 query heads (grid 1,3,64 = 2 waves @c2), vLLM applies the FA2 GQA seqlenq_ngroups_swapped so it iterates 8 kv-heads (grid 1,6,16 = 1 wave). Ours c2 flash 63.7us vs vLLM 41.7us (+53%), c8 218 vs 151 (+44%), same vendored flash_fwd_splitkv kernel. Marlin ~parity same-tool (+7-9%). Named lever + exact site (cuda_flash_attn_fa2.cu:1075) recorded; no code shipped (a numerically-near-tie graphed-decode launcher port needs a RED-first TDD cycle, not a rushed change). Evidence dgx:~/mxfp4-nsys/. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ded OFF (#47) row/QUANT-CT-MXFP4-GLUE squash, records-only. c1-vs-c2 per-kernel delta: glue +90us/step only, and vLLM's fused glue is NOT cheaper (~910 vs ~776us/step) - #46's fusion-edge hypothesis measurement-killed. REAL lever CONFIRMED: decode flash 35.5->63.7->218us (c1/c2/c8) because cuda_flash_attn_fa2.cu:1075 hard-codes seqlenq_ngroups_swapped=false; vLLM's grid (batch x kv_heads) proves it swaps - ours over-waves at batch>=2 (192 CTAs/2 waves vs 96/1 on 48 SMs). Marlin flag ~parity. Fix scoped: ngroups-swap port gated VT_FA2_DECODE_GQA_SWAP default-OFF (near-tie class, graphed path), RED-first; expected ~28% c2 / ~55% c8. Honest residual map recorded (marlin ambiguity + ~0.7ms host). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…te (#48) row/KERNEL-FA2-GQA-SWAP squash. LaunchDecodeVarlenFA2Bf16 gains the upstream mha_fwd_kvcache ngroups swap (logical q/o reshape via strides, no transpose; h=kv_heads, seqlen_q=ngroups; num_splits on swapped dims; 1:1 mirror of the shipped d256 LaunchDecodeFA2Bf16). Gate VT_FA2_DECODE_GQA_SWAP default OFF, OFF path untouched. Blast radius: Qwen3-dense d128 family only (0.6B/4B/8B); gate models already swap on d256; Voxtral MHA inert. Gates: op RED-first 5/5 (280 assn; wrong-stride injection fails loudly 26,528 violations), full binary 28/28 (454,679), memcheck 0/0, #44 smoke swap-ON 3/3 token-exact AND byte-identical to OFF (graphed). Flip conditioned on the c1-c8 re-bench + 0.6B/4B e2e (next campaign); #47 projects ~28% c2 / ~55% c8 gap closure. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only localization. c1-vs-c2 per-kernel delta: glue barely moves (+90us/step) and vLLM's Inductor-fused glue is NOT cheaper than ours (~910 vs ~776us/step) — the #46 fusion-edge hypothesis is refuted by measurement. CONFIRMED real lever: decode flash grows 35.5->63.7us (c2) ->218us (c8) because src/vt/cuda/cuda_flash_attn_fa2.cu:1075 hard-codes seqlenq_ngroups_swapped=false while vLLM's grid (1,6,16 = batch x KV-heads) proves it swaps: ours launches per query-head (192 CTAs = 2 waves at c2 on 48 SMs) vs vLLM per-kv-head-group (96 = 1 wave). Marlin cross-tool flag re-checked ~parity. Fix scoped: port the mha_fwd_kvcache ngroups swap, gated VT_FA2_DECODE_GQA_SWAP default-OFF (non-byte-exact near-tie class, graph-captured path) — implementation dispatched. Expected: ~28% of c2 / ~55% of c8 gap.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys