perf(fa2): GB10 num_splits cap (gated-OFF) + MXFP4 parity TERMINAL residual - #60
Closed
localai-bot wants to merge 1 commit into
Closed
perf(fa2): GB10 num_splits cap (gated-OFF) + MXFP4 parity TERMINAL residual#60localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…rity TERMINAL residual
The two last named MXFP4 levers, closed and measured.
LEVER 1 (num_splits cap, default OFF): NumSplitsHeuristic is fed num_sms*2 and
over-splits the batch-1 decode to nnblk (7 at c1, ~41us/layer) because every
eligible split keeps waves<1 on GB10's latency-bound kernel. VT_FA2_NSPLITS_CAP
(off / auto=wave-optimal / explicit-N) clamps it at both decode launch sites.
Grounded in the sanctioned KERNEL-FA2-DECODE-PARAMS A/B (fa2ab_n{1,3}): cap=3 ->
~32us (~17% flash win) but self-corrects to ~3 at c8, so it only moves c1-c2 and
c1 already passes 1.020x. Op test covers both GQA ratios (engages split->no-split,
byte-correct, near-ties). Battery green: op 114/114 + memcheck 0 err; SACRED
0.6B/4B 184/184 and 32B-NVFP4A16 strict 144/144 with cap=auto CHAR-IDENTICAL
(flips no strict token); async 325/325; #44 smoke 3/3. Stays OFF (non-byte-exact,
no failing-axis win).
LEVER 2 (glue): already folds through vt::FusedChain (both add+RMSNorm sites,
default-ON); the residual +198-290us glue is vLLM's Inductor GEMM-pro/epilogue
fusion, a Marlin kernel rewrite the catalog cannot express (#46-declined).
VERDICT: TERMINAL residual statement. Every measured lever exhausted. c1 1.020x
PASS + mem 2.63x WIN, c2-c8 0.962-0.969 GPU-intrinsic. The definitive oracle
binding is projected UNCHANGED vs #57 (cap inert at c4-c8) and not re-run under
box-safety (OOM-risk #52 deferred; ENOSPC hit during build). One unexhausted path
remains: a from-scratch Marlin epilogue-fusion kernel.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). VT_FA2_NSPLITS_CAP ships gated-OFF per parity-enablers (non-byte-exact, no failing-axis win). Terminal MXFP4 verdict recorded. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… closed (#60) row/QUANT-CT-MXFP4-FINAL-STACK squash. Lever 1: VT_FA2_NSPLITS_CAP (off/auto/N) at both FA2 decode launch sites — wave-optimal cap on the real 48 SMs; battery green incl. 32B strict CHARACTER-IDENTICAL at cap=auto, memcheck 0, async 325/325, #44 smoke 3/3. Stays default-OFF per parity-enablers: non-byte-exact split-reduction with no failing-axis win (only touches c1-c2; c1 already 1.020). Lever 2: glue already default-ON through vt::FusedChain at both add+RMSNorm sites + fused gate_up; residual is Inductor GEMM-epilogue fusion the catalog cannot express (a Marlin kernel rewrite, the #46-declined redirect). TERMINAL MXFP4 VERDICT: c1 1.020 PASS + mem 2.63x WIN; c2-c8 0.962-0.969 GPU-intrinsic; every measured lever exhausted. The one unexhausted path = from-scratch Marlin prologue/epilogue fusion kernel. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the MXFP4 parity leg's two last named levers and states the terminal residual.
Lever 1 — num_splits cap
VT_FA2_NSPLITS_CAP(default OFF), LANDEDcuda_flash_attn_fa2.cu:Fa2NsplitsCapConfig/ApplyNsplitsCapat both decode launch sites. The FA2 heuristic is fednum_sms*2and over-splits the batch-1 decode tonnblk(7 at c1, ~41us/layer) because every eligible split keeps waves<1 on GB10's latency-bound kernel. Modes: off /auto(wave-optimal) / explicit-N (the fa2ab A/B knob). Grounded in the sanctioned KERNEL-FA2-DECODE-PARAMS A/B (cap=3 -> ~32us, ~17% flash win) but it self-corrects to ~3 splits at c8, so it only moves c1-c2 and c1 already passes 1.020x.Gate battery (GB10 sm_121a): op
test_ops_paged_attnnew case 114/114 both GQA ratios (cap engages split->no-split, byte-correct, near-ties) + memcheck 0 err; SACRED 0.6B/4B 184/184 and 32B-NVFP4A16 strict 144/144 withcap=autoCHARACTER-IDENTICAL (flips no strict token); async 325/325; #44 smoke 3/3. Stays default-OFF (non-byte-exact reduction order, no failing-axis win to bank).Lever 2 — glue: already landed as scoped
qwen3.cpp::RunLayeralready routes both add+RMSNorm sites throughvt::FusedChain(kFusedAddRmsNormStd)(default-ON). The residual glue is vLLM's Inductor GEMM-pro/epilogue fusion — a Marlin kernel rewrite the catalog cannot express (#46-declined).Verdict — TERMINAL residual
Every measured lever exhausted. c1 1.020x PASS + mem 2.63x WIN; c2-c8 0.962-0.969 GPU-intrinsic. The definitive oracle binding is projected UNCHANGED vs #57 (cap inert at c4-c8) and not re-run under box-safety (OOM-risk deferred by #52; ENOSPC hit during build). One unexhausted path: a from-scratch Marlin epilogue-fusion kernel.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys