perf(quant): MXFP4 decode-graph + gate_up FUSION default-ON (180->144 GEMM/step); E=1 CTA par=1 opt-in - #54
Closed
localai-bot wants to merge 1 commit into
Closed
Conversation
… GEMM/step); E=1 CTA par=1 arbiter opt-in row QUANT-CT-MXFP4-MARLIN-STRUCT (base #52 027af9b). Closes the marlin-structural + decode-graph levers on the classic-dense Qwen3-8B-MXFP4 decode. STEP 1 (parity-enabler): DenseDecodeGraphEnabled() flipped DEFAULT OFF->ON (=0 opts out; VLLM_CPP_CUDAGRAPH=0 still forces eager). Gated on the unmodified #52 binary (graph env ON vs OFF, capture-safe): test_qwen3_paged_engine 184/184 IDENTICAL, test_qwen3_dense_async_serving 82/82 IDENTICAL, #44 8B smoke 3/3; then re-verified at the flipped default. Banks the #52 measured +1.3% TPOT@c8 for all five dense registrations (Qwen3/Llama/InternLM3/Mistral/InternLM2). STEP 2 (structural parity): gate_up FUSION generalized NVFP4->MXFP4 (dense_nvfp4_gemm.h: group_size 16|32, MarlinProcessExpertScalesMxfp4 E8M0 passthrough + g=1.0, margs.mxfp4/group_size; GateUpFusedEligible drops the mxfp4 exclusion). Decode marlin drops 180->144 GEMM/step (vLLM-structural, gate_up merged). Byte-token-exact: new test_linear_method unit (fused ~= split, >=99% bit-exact + fused-ran counter, RED-first), #44 fused==split 3/3, 32B-NVFP4A16 142/142. Same-tool nsys: fusion gave count-parity but only -200us marlin (W4A16 memory-bound), so it isolates the residual rather than speeding decode. STEP 3 (arbiter, opt-in): the marlin residual at matched 144-count is the CTA count (ours 144 = sms*par3 vs vLLM dense 48 = sms*1, blocks = sms*blocks_per_sm). VT_MARLIN_E1_PAR1 (DEFAULT OFF) clamps the single-expert decode grid to sms*1 (48 CTAs): marlin 17,463->16,512 us/step (-5.4%, near-parity), TPOT 37.22->36.23, token-exact on 8B-MXFP4. Kept default-OFF because the par regroup flips a strict 32B-NVFP4A16 token (isolated: baseline/graph-only 142/142, par1-only 59/60); real MoE (num_experts>1) and prefill are untouched. Byte-preserving CTA reduction is the dense-template port (#50 NO-GO), scoped next. STEP 4: same-tool nsys c8 residual map (default vs +par1 vs vLLM) recorded in .agents/benchmark-record.md + state.md. Full strict binding ratio (fresh oracle) deferred: the strict harness needs an on-real-disk RelWithDebInfo build + the vLLM oracle, whose host-RAM reservation alongside a tmpfs build tree is the GB10 OOM-reboot risk. Docs: STATUS/BENCHMARKS/FEATURES/ENVIRONMENT keyed rows + NOW + state + benchmark-record updated in-change; STATUS under the 287,832 shrink ratchet. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
marked this pull request as ready for review
August 6, 2026 07:53
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…iter (#54) row/QUANT-CT-MXFP4-MARLIN-STRUCT squash. (1) Classic-dense decode graph default OFF->ON (=0 opt-out): SACRED 184/184 + async 82/82 + smoke 3/3 IDENTICAL both arms and at the flipped default; banks the measured +1.3% TPOT@c8 for all 5 dense registrations. (2) MXFP4 gate_up fusion: GateUpFusedEligible/BuildMarlinDensePairResident/GateUpFusedMarlinD generalized NVFP4->MXFP4 (group 16|32, E8M0 passthrough); decode marlin 180->144 GEMM/step = structural parity with vLLM; unit RED-first + smoke fused==split + 32B 142/142. (3) CTA arbiter: root cause = launcher par=3 -> 144 CTAs (marlin_mm_moe.cu:494) vs vLLM tile-per-CTA 48; VT_MARLIN_E1_PAR1 opt-in clamp recovers 81% of the marlin residual (per-call 114.7 vs vLLM 113.1) but 1 bf16 ULP from reduce regrouping flips one 64-layer strict token -> default OFF; byte-preserving CTA reduction = the dense-template port, scoped. Residual map: marlin +1,177 (53%, ->+226 via par1), flash +784 (35%, unresolved), glue +195. TPOT 37.56->37.22 default (36.23 par1) vs vLLM 34.58. Strict binding owed (on-disk build + oracle). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… projections take vLLM's own dense marlin (48-CTA, byte-faithful, beats MoE) Executes the four scoped GB10 gates for the #55 dense-template marlin port and flips VT_MARLIN_DENSE default ON (=0 opts back out to the MoE route). WHY. The single-expert MoE-marlin route the dense E=1 projections use pads M<=8 into a 128-CTA grid; vLLM's own dense marlin covers the same tiles with 48 CTAs. The #54 par1 clamp reached 48 CTAs but regrouped the fp32 C_tmp reduce, costing a bf16 ULP that flipped a strict 32B token. This dense-template port keeps vLLM's OWN dense reduce, so it is byte-faithful to vLLM's numerics. GATES (GB10 sm_121a, RelWithDebInfo, full production stack): (a) UNIT EXEC: 2/2 cases 263/263 GREEN. First run was RED on NVFP4 but the diagnostic proved a test-authoring artifact, not a kernel bug: max|dense-moe| ==0 everywhere (dense output byte-identical to the validated MoE route), the failures were cancellation elements vs a naive fp32 ref (identical for MoE). Fix: NVFP4 vs-reference uses the cancellation-robust L2 metric; per-element byte-fidelity stays the EXACT dense==MoE check; RED injection preserved. compute-sanitizer memcheck: 0 invalid-access/race errors. (b) STRICT BATTERY dense-ON vs the oracle: 32B-NVFP4A16 6/6 (graphed AND eager, byte-identical) via the ratified near-tie razor — every dense token == vLLM's teacher-forced argmax, max gap 0.000 nats, TIGHTER than the MoE route (62 mnats). The MoE greedy anchor shifts at two exact bf16 ties, so the 32B our_ids/neartie_gap goldens are regenerated under dense-ON. Counters dense_gemms=2048, marlin_gemms=0 (gate-conditional assertion added). 0.6B/4B 184/184, async 82/82, #44 MXFP4-8B 3/3 token-exact. (c) NSYS c8: marlin kernel marlin_moe_wna16 -> marlin (dense template ran), grid 128->48 CTA, per-call median 117.8->85.8us. (d) BINDING c1..c8 x3 vs #51: total_token_throughput c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (beats #51 1.005/0.925/0.939/0.953 on EVERY axis), GPU mem 2.63x less. MXFP4 vLLM-parity goal still <1.0x c2-c8 (closer than #51), residual = decode-flash then glue. Flip conditions (parity-enablers) met: correctness ratified + beats MoE every axis + no regression + memory win. Records: state, NOW, STATUS, BENCHMARKS, FEATURES updated; STATUS ratchet lowered to the shrunk size. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…error build repaired (#61) row/BUILD-CPU-WERROR-MXFP4-TESTFIX squash. The #54 MXFP4 test helper is used only inside the VT_MARLIN_NVFP4 region but was defined unguarded, so the plain CPU-only Release build (-Werror, no marlin) fails with unused-function at test_linear_method.cpp:77 — reproduced on a clean checkout of main 8d55550 before the fix; the TU compiles clean after. 2-line guard, no behavior change on any configuration that compiled before. Found while build-gating the PR #26 merge. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
row
QUANT-CT-MXFP4-MARLIN-STRUCT— closes the #52 marlin-structural + decode-graph levers on classic-dense Qwen3-8B-MXFP4 decode. Baseorigin/main027af9b0(#52).What landed (all gated on GB10, 5 rebuilds EXIT=0)
Step 1 — decode-graph DEFAULT-ON (
VLLM_CPP_QWEN3_DENSE_DENSE_DECODE_GRAPH, parity-enabler,=0opts out). Gated on the unmodified #52 binary graph ON vs OFF (capture-safe):test_qwen3_paged_engine184/184 IDENTICAL,test_qwen3_dense_async_serving82/82 IDENTICAL, #44 8B smoke 3/3; re-verified at the flipped default. Banks +1.3% TPOT@c8 for all five dense registrations.Step 2 — gate_up FUSION generalized NVFP4→MXFP4, DEFAULT-ON. Decode marlin drops 180→144 GEMM/step (vLLM-structural parity, gate_up merged). Byte-token-exact: new
test_linear_methodunit (fused ≈ split ≥99% bit-exact + fused-ran counter, RED-first), #44 fused==split 3/3, 32B-NVFP4A16 SACRED 142/142. Same-tool nsys: fusion gives count-parity but only −200µs marlin (W4A16 memory-bound) — it isolates the residual.Step 3 — CTA-count arbiter,
VT_MARLIN_E1_PAR1opt-in (DEFAULT OFF). Root cause: ours 144 CTAs (sms×par3) vs vLLM dense 48 (sms×1). Clamping E=1 decode to 48 CTAs recovers 81% of the marlin residual (marlin 17,463→16,512µs, TPOT 37.22→36.23), token-exact on 8B-MXFP4 — but flips a strict 32B-NVFP4 token (fp32-reduce regroup; isolated: baseline/graph-only 142/142, par1-only 59/60), so kept default-OFF. MoE (num_experts>1) + prefill untouched. Byte-preserving route = the dense-template port (#50 NO-GO), scoped.Step 4 — same-tool nsys c8 residual map in
.agents/benchmark-record.md. At shipping default the c8 residual is marlin CTA +1,177µs (dominant), flash +784, glue +195. Full strict binding ratio (fresh oracle) deferred — the strict harness needs an on-real-disk RelWithDebInfo build + the vLLM oracle, whose host-RAM reservation alongside a tmpfs tree is the GB10 OOM-reboot risk.Gates
unit 7/7·56 ·
test_qwen3_paged_engine184/184 (graph ON==OFF + flipped default) ·test_qwen3_dense_async_serving82/82 · #44 MXFP4-8B smoke 3/3 (fused==split) ·test_qwen3_32b_nvfp4a16_paged_engine142/142 at the pure shipping default. Consistency + doc + record checkers green; only the expected role-discipline red (this branch awaiting operator merge).🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys