row/QUANT-CT-MXFP4-CLOSERS: byte-exact slivers ship — c1 FULL PARITY; c2-c8 0.925-0.953 - #51
Closed
localai-bot wants to merge 2 commits into
Closed
row/QUANT-CT-MXFP4-CLOSERS: byte-exact slivers ship — c1 FULL PARITY; c2-c8 0.925-0.953#51localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
…drop per-call ws re-zero, both BYTE-EXACT default-ON The two #50-arbitrated MXFP4 closer slivers, in the shared dense header include/vllm/model_executor/models/dense_nvfp4_gemm.h (the qwen3_5.cpp twin that serves the 27B/35B gate models' dense attn + shared-expert is left byte-untouched): (a) DenseAlignFor forces block=8 at M<=8 (:286): the M=8 dense single-expert GEMM now takes vLLM's 8-row m_block_size_8 tile (marlin.cu:438 m_block_size_8 = prob_m<=8) instead of the padded 16-row tile our MoE-align picked at M=8 (MarlinMoeAlignBlockSizeSelect 8/8 fails the <0.9 test). Recovers the #50-measured ~0.33ms/step (~0.8pp) at c8. The m_block_size_8 kernels are vendored (kernel_selector.h:3-8 nvfp4, :33-38 mxfp4); M>8 unchanged. (b) DenseMarlinWorkspace zeroes the shared reduction workspace ONCE at alloc; the two per-call Memset(ws) in MatmulNvfp4MarlinD/GateUpFusedMarlinD are dropped (mirror vLLM marlin_make_workspace_new torch.zeros, marlin_utils.py:399-407). Invariant: our launch pins use_atomic_add=false (cuda_moe_marlin.cu:141), so only the fp32 barrier reduce is reachable, whose last slice-block release re-zeroes the lock (marlin_template.h:2170 -> :204); slice_count==1 never touches locks (:2162); the atomic-add non-clearing path (:614) is dead. Both are BYTE-EXACT so they ship DEFAULT-ON unconditionally (no VT_* gate). GB10 gates: OP RED-first (test_ops_moe_grouped closers) block8-vs-block16 A/B at M=8 BYTE-EXACT (bitdiff=0/32768) for MXFP4 K=4096/12288 + NVFP4 K=4096, ws all-zero after a GEMM + reuse bit-identical (15/15, 2/2 cases); launch-config DenseAlignFor(d,8).block==8 RED-first (test_qwen3_forward, 7/7); memcheck 0 real errors (leaks are pre-existing static-cache harness artifacts); #44 smoke Yi30/Qwen3-8B-MXFP4 3/3 deterministic token-exact + near-tie coherent. Binding re-bench x3 (clean-checkout grid) + the ~0.7ms/step host/sched slice remain the MXFP4 parity verdict's open terms. Docs updated same change: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…0.925-0.953 (#51) row/QUANT-CT-MXFP4-CLOSERS squash. DenseAlignFor block=8 at M<=8 (vLLM's m_block_size_8 tile, kernels already vendored) + one-time workspace zeroing (self-reset invariant cited marlin_template.h:2170/:204), both BYTE-EXACT (bitdiff 0/32768, MXFP4+NVFP4) -> default-ON, gate-model twin untouched. Binding at d3b412f: c1 PASSES every axis (1.005/1.002/1.034/ 1.004); c2-c8 0.925/0.939/0.953 tput, up on every axis vs #49, no regression; host-mem 2.18x LESS. VT_LOOP_TRACE attribution: engine-core decode loop interval==step to 0.02ms -> the ~0.7ms host slice is NOT an engine lever (frontend/boundary). Residual map final: only the grouped E=1 indirection remains as a GPU term, per-shape-parity at M<=8 post block-fix. Parity gate NO for batched; c1 + memory WIN recorded. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…l 58% (#52) row/QUANT-CT-MXFP4-C8-DIFF squash, records-only. First same-tool nsys both engines at c8 (vLLM offline in-process, identical prompts, M=8). Fair decomposition of the ~3ms TPOT gap: MARLIN +1,377us (58%) = STRUCTURAL - 5 grouped-E1 GEMMs/layer (gate+up unfused, 144 CTAs) vs vLLM's 4 dense fused (48 CTAs), +25% GEMM count/step; settles #50 (the isolated-shape ubench was blind to count+CTA structure). Flash +658us (28%, +11%/call at IDENTICAL grid, unresolved params diff). Glue +290us (12%, portable-fusion class). The #51 binding ran EAGER: classic-dense decode graph is opt-in OFF; graph-ON = byte-coherent, gap 1184->305us (=vLLM), TPOT -1.3%, SACRED gate owed before flip. STATUS ratchet lowered 287838->287832. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… projections take vLLM's own dense marlin (48-CTA, byte-faithful, beats MoE) Executes the four scoped GB10 gates for the #55 dense-template marlin port and flips VT_MARLIN_DENSE default ON (=0 opts back out to the MoE route). WHY. The single-expert MoE-marlin route the dense E=1 projections use pads M<=8 into a 128-CTA grid; vLLM's own dense marlin covers the same tiles with 48 CTAs. The #54 par1 clamp reached 48 CTAs but regrouped the fp32 C_tmp reduce, costing a bf16 ULP that flipped a strict 32B token. This dense-template port keeps vLLM's OWN dense reduce, so it is byte-faithful to vLLM's numerics. GATES (GB10 sm_121a, RelWithDebInfo, full production stack): (a) UNIT EXEC: 2/2 cases 263/263 GREEN. First run was RED on NVFP4 but the diagnostic proved a test-authoring artifact, not a kernel bug: max|dense-moe| ==0 everywhere (dense output byte-identical to the validated MoE route), the failures were cancellation elements vs a naive fp32 ref (identical for MoE). Fix: NVFP4 vs-reference uses the cancellation-robust L2 metric; per-element byte-fidelity stays the EXACT dense==MoE check; RED injection preserved. compute-sanitizer memcheck: 0 invalid-access/race errors. (b) STRICT BATTERY dense-ON vs the oracle: 32B-NVFP4A16 6/6 (graphed AND eager, byte-identical) via the ratified near-tie razor — every dense token == vLLM's teacher-forced argmax, max gap 0.000 nats, TIGHTER than the MoE route (62 mnats). The MoE greedy anchor shifts at two exact bf16 ties, so the 32B our_ids/neartie_gap goldens are regenerated under dense-ON. Counters dense_gemms=2048, marlin_gemms=0 (gate-conditional assertion added). 0.6B/4B 184/184, async 82/82, #44 MXFP4-8B 3/3 token-exact. (c) NSYS c8: marlin kernel marlin_moe_wna16 -> marlin (dense template ran), grid 128->48 CTA, per-call median 117.8->85.8us. (d) BINDING c1..c8 x3 vs #51: total_token_throughput c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (beats #51 1.005/0.925/0.939/0.953 on EVERY axis), GPU mem 2.63x less. MXFP4 vLLM-parity goal still <1.0x c2-c8 (closer than #51), residual = decode-flash then glue. Flip conditions (parity-enablers) met: correctness ratified + beats MoE every axis + no regression + memory win. Records: state, NOW, STATUS, BENCHMARKS, FEATURES updated; STATUS ratchet lowered to the shrunk size. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…very axis (#57) row/KERNEL-MARLIN-DENSE-EXEC squash. All four #55 gates green: unit 263/263 (initial RED was a naive-fp32-reference cancellation artifact, dense output byte-identical to the MoE route; L2 metric fix + stride RED retained); 32B strict decider's two divergences RATIFIED exact bf16 ties via teacher-forcing (0.000000 nats, tighter than MoE) -> goldens regenerated per the razor, 6/6-144 graphed+eager at the flipped default (dense_gemms=2048, marlin_gemms=0); canaries 0.6B/4B/async/smoke green; memcheck 0; nsys: marlin::Marlin 48 CTAs, per-call 117.8->85.8us. Binding vs #51: c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969, TPOT up everywhere, mem 2.63x LESS. MXFP4 parity gate still NO (best c8 0.969); residual = decode-flash +658-784us (next-dominant), glue tail, host. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two #50-arbitrated slivers (block=8 at M<=8 + one-time workspace zeroing), BYTE-EXACT by bit-comparison (0/32768 both MXFP4 and NVFP4), default-ON, RED-first gated, memcheck clean, gate-model twin untouched. Final binding: c1 PASSES every axis (tput 1.005, ttft 1.034); c2-c8 0.925/0.939/0.953 tput (+up on every axis vs #49, no regression); host-mem 2.18x WIN. Host-slice attributed: engine-core loop overhead negligible (interval==step to 0.02ms) — the residual is frontend/measurement-boundary, not an engine lever. Parity gate stays NO for batched; residual map final.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys