row/QUANT-CT-MXFP4-M28-LEVER: M-config lever REFUTED per-shape; real diff = Inductor glue fusion - #46
Closed
localai-bot wants to merge 1 commit into
Closed
row/QUANT-CT-MXFP4-M28-LEVER: M-config lever REFUTED per-shape; real diff = Inductor glue fusion#46localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…LLM per-shape parity (QUANT-CT-MXFP4-M28-LEVER) Chased the #45 binding's named lever ("the c2-c8 TPOT gap is the batched grouped Marlin W4A16 decode GEMM as M grows 2->8; port vLLM's M-dependent thread_m_blocks/tile config"). Profiled BOTH engines on GB10 + ran the per-shape arbiter. The lever is REFUTED. Same-tool per-shape microbench (oracle compiled ops, cuda-event timed): our production MoE-E1 Marlin (moe_wna16_marlin_gemm) == vLLM's dense marlin_gemm at every M (moe/dense per-step 0.99-1.08, deltas <0.6ms noise). Porting vLLM's dense M-config cannot close the ~4.3ms/step (11%) gap — the decode Marlin is already at vLLM per-shape speed. nsys c8 both engines (TPOT reproduced 39.53 vs 33.81ms): decode Marlin dominates our decode (Med 123us) and is at-parity with vLLM's (112.9us); the robust structural divergence is vLLM's Inductor FUSION of the norm/quant/activation glue into the GEMM pro/epilogue (triton_..._add_rms_norm_marlin_gemm, ..._marlin_gemm_mul_silu) vs our separate glue launches. No code shipped (the named fix is refuted). Redirect (no ceiling): portable glue-fusion + a same-tool c1-vs-c2 nsys diff to localize the STEP at c2 (Marlin is M-independent M1->M2, so the step is attention batching / unfused glue / async overlap, not the GEMM). Records: state.md + benchmark-record (nsys + microbench tables) + NOW + STATUS/BENCHMARKS MXFP4 rows. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ductor glue fusion (#46) row/QUANT-CT-MXFP4-M28-LEVER squash, records-only. Per-shape arbiter (oracle-compiled ops, cuda-event timed) kills the #45 M-config hypothesis pre-implementation: our E=1 grouped Marlin matches vLLM's dense marlin at every M/shape (0.99-1.08). Same-tool nsys both engines: GEMMs at parity; vLLM's Inductor fuses the norm/quant/act glue into the Marlin pro/epilogue where we launch ~5 separate kernels/layer (portable- fusion class, 2nd family). Cross-tool per-call flags recorded for same-tool re-check. Next: c1-vs-c2 ours-only step diff, then the fusion port. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ded OFF (#47) row/QUANT-CT-MXFP4-GLUE squash, records-only. c1-vs-c2 per-kernel delta: glue +90us/step only, and vLLM's fused glue is NOT cheaper (~910 vs ~776us/step) - #46's fusion-edge hypothesis measurement-killed. REAL lever CONFIRMED: decode flash 35.5->63.7->218us (c1/c2/c8) because cuda_flash_attn_fa2.cu:1075 hard-codes seqlenq_ngroups_swapped=false; vLLM's grid (batch x kv_heads) proves it swaps - ours over-waves at batch>=2 (192 CTAs/2 waves vs 96/1 on 48 SMs). Marlin flag ~parity. Fix scoped: ngroups-swap port gated VT_FA2_DECODE_GQA_SWAP default-OFF (near-tie class, graphed path), RED-first; expected ~28% c2 / ~55% c8. Honest residual map recorded (marlin ambiguity + ~0.7ms host). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… closed (#60) row/QUANT-CT-MXFP4-FINAL-STACK squash. Lever 1: VT_FA2_NSPLITS_CAP (off/auto/N) at both FA2 decode launch sites — wave-optimal cap on the real 48 SMs; battery green incl. 32B strict CHARACTER-IDENTICAL at cap=auto, memcheck 0, async 325/325, #44 smoke 3/3. Stays default-OFF per parity-enablers: non-byte-exact split-reduction with no failing-axis win (only touches c1-c2; c1 already 1.020). Lever 2: glue already default-ON through vt::FusedChain at both add+RMSNorm sites + fused gate_up; residual is Inductor GEMM-epilogue fusion the catalog cannot express (a Marlin kernel rewrite, the #46-declined redirect). TERMINAL MXFP4 VERDICT: c1 1.020 PASS + mem 2.63x WIN; c2-c8 0.962-0.969 GPU-intrinsic; every measured lever exhausted. The one unexhausted path = from-scratch Marlin prologue/epilogue fusion kernel. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only. The #45 attribution (grouped-Marlin M-config) refuted by the per-shape arbiter BEFORE implementation: ours vs vLLM's dense marlin at M 1/2/4/8 on all Qwen3-8B shapes = 0.99-1.08 (noise); porting the dense M-config cannot close the ~4.3ms/step. Same-tool nsys both engines instead shows vLLM's Inductor fusing add+RMSNorm+quant into the Marlin prologue and silu+mul into the epilogue (triton_red_fused_* / triton_poi_fused_*) where we launch ~5 separate glue kernels/layer — the portable-fusion class, second model family confirmed. Cross-tool per-call flags (marlin 123v113us, attn 218v151us identical traits) recorded for same-tool re-check. Next: c1-vs-c2 ours-only step diff to localize, then the fusion port.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys