row/QUANT-CT-MXFP4-BENCH: binding W4 grid — c1 parity + 2.6x memory WIN; c2-c8 TPOT 0.89-0.91x (lever in flight) - #45
Closed
localai-bot wants to merge 2 commits into
Closed
row/QUANT-CT-MXFP4-BENCH: binding W4 grid — c1 parity + 2.6x memory WIN; c2-c8 TPOT 0.89-0.91x (lever in flight)#45localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
…ator (QUANT-CT-MXFP4-BENCH) Plumb the online-serving harness for the MXFP4 W4A16 keep-quant throughput grid on Yi30/Qwen3-8B-MXFP4. The compute is correctness-DONE on main (#38 + the #44 async-dense-mirror fix: default-config e2e 3/4 token-exact + p2 near-tie); the only owed piece was the binding ours-vs-oracle grid, which online_gate.py could not run because it carried only the "27"/"35" NVFP4 gate-model keys. Additive; existing 27/35 paths byte-unchanged; harness CPU contract tests 45/45. - online_gate.py: q3mxfp4 MODEL_REVISIONS/REPOSITORIES (b3e7ab32), MAX_NUM_BATCHED_TOKENS=2048, MAX_MODEL_LEN=40960, and a POINTS_BY_MODEL reduced c1/c2/c4/c8 sweep surfaced via points_for(model_key) (a strict prefix of POINTS, so prompts_for is unchanged). online_gate_summary threads points_for through its four per-model POINTS loops so a c1-c8 key never flags a missing c16/c32 group. - mxfp4_smoke_gate.py (new): the q3mxfp4 model gate = the #44 e2e smoke battery (vllm-cli greedy vs golden_marlin_w4a16.json; 3 deterministic prompts exact + story coherent). Validated locally against a fake vllm-cli (PASS + two FAILs). - dgx-online-serving.sh: q3mxfp4 branches (validation, dense 2048 batched-tokens, vllm-cli build + smoke model gate, oracle arm VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel with no mamba flag, 1 2 4 8 leg loop). The mamba-pin shell-parse test retargeted to the 27/35 block and strengthened to assert the q3mxfp4 arm's disabled-kernel + no-mamba. - mxfp4-online-serving-grid.sh (new): one-command orchestrator (corpus gen + dry-run manifest + locked --execute grid + summary). No throughput numbers: the grid needs GB10 exclusively (fresh /dev/shm CUDA build, oracle serve, 2 arms x 3 reps x c1-c8 sequential legs, both flock locks, free-g >= 90). Grid GATING on dgx; run scripts/mxfp4-online-serving-grid.sh. Records (STATUS/BENCHMARKS/FEATURES + quantization-matrix + benchmark-record + NOW + state) updated in the same change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ecode gap (QUANT-CT-MXFP4-BENCH) Executed the binding ours-vs-oracle online-serving grid on Yi30/Qwen3-8B-MXFP4 (dense Qwen3, native Marlin W4A16 MXFP4 keep-quant) at 33e9360 on dgx GB10. Production graphed vLLM 0.25.0, oracle forced to Marlin via VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel (sm_121 cute-dsl crashes), c1/c2/c4/c8 x3 interleaved, single load/arm, drop_caches between legs, both flock locks. Smoke model gate reproduced #44 (3/3 deterministic token-exact + coherent near-tie). VERDICT: BELOW-FLOOR (gate NO, 74/84 axes below). Medians (ours->vllm, ratio): - total tok/s: c1 0.989, c2 0.911, c4 0.919, c8 0.913 - median TPOT ms: c1 0.986, c2 0.900, c4 0.905, c8 0.891 - median TTFT: at parity (c1/c4 PASS, c2/c8 within noise) - peak GPU mem: 28.3 vs 73.7 GiB = 2.607 (ours 2.6x LESS) Per-rep spread ~1-3% (tight/reproducible; full per-rep in benchmark-record). FIRST attribution (grounded, no ceiling): the gap is a BATCHED-decode cost absent at batch-1 -- c1 is at parity (0.989 tput / 0.986 TPOT), c2-c8 is ~0.91x driven ENTIRELY by TPOT +10-12% with TTFT/prefill at parity. Divergent hot path = the grouped Marlin W4A16 keep-quant decode GEMM at M=2..8, not batch-1 GEMV nor prefill. Memory a clean 2.6x win. Next: same-tool nsys decode-window on c8 both engines to name the exact kernel/shape. Also: fixed the orchestrator ordering (plan must precede corpus -- online_gate.py `plan` refuses a non-empty evidence root) and recorded the strict online-serving build contract learned this run (DISK build not tmpfs, RelWithDebInfo, oracle-venv ninja, oracle flashinfer cutlass, export-compile-commands, profile-control OFF). Records: BENCHMARKS/STATUS/FEATURES + quantization-matrix + benchmark-record (full table + recipe + build contract) + NOW + state, same change. Box left clean. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship, local commit-tree flow). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…-c8 TPOT 0.89-0.91x (#45) row/QUANT-CT-MXFP4-BENCH squash. Harness q3mxfp4 key (online_gate.py POINTS/REVISIONS/REPOSITORIES, smoke gate vs the #44 battery, dgx orchestrator; 45/45 contract tests). Binding medians of 3 reps ours vs the 0.25.0 oracle (Marlin W4A16 arm via VLLM_DISABLED_KERNELS): c1 0.989x total, c2 0.911x, c4 0.919x, c8 0.913x - deficit entirely TPOT (+10-12% at c2-c8), TTFT at parity, peak GPU memory 2.607x LESS (28.3 vs 73.7 GiB keep-quant). Attribution: the grouped Marlin W4A16 decode GEMM as M grows 2->8 (batch-1 + prefill at parity); kernel-level nsys + lever campaign dispatched, no ceiling declared. Strict record-execution build contract documented (real-disk tree, RelWithDebInfo, oracle cutlass, cache-drop mincore proof). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ductor glue fusion (#46) row/QUANT-CT-MXFP4-M28-LEVER squash, records-only. Per-shape arbiter (oracle-compiled ops, cuda-event timed) kills the #45 M-config hypothesis pre-implementation: our E=1 grouped Marlin matches vLLM's dense marlin at every M/shape (0.99-1.08). Same-tool nsys both engines: GEMMs at parity; vLLM's Inductor fuses the norm/quant/act glue into the Marlin pro/epilogue where we launch ~5 separate kernels/layer (portable- fusion class, 2nd family). Cross-tool per-call flags recorded for same-tool re-check. Next: c1-vs-c2 ours-only step diff, then the fusion port. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…side noise (#49) row/KERNEL-FA2-GQA-SWAP-FLIP squash. Fa2DecodeGqaSwapEnabled OFF->ON (VT_FA2_DECODE_GQA_SWAP=0 opts out). Correctness: SACRED 0.6B+4B 16/16 each across FOUR arms (off/on/new-default/opt-out) character-identical; op 5/5-280; #44 smoke 3/3 token-exact at the new default. Mechanism proof at no-env: decode-flash grid (1,5,16)=batch x kv_heads, 80 CTAs 1 wave, per-call 63.7->45.3us (-29%). Binding vs #45: c2 0.911->0.922, c4 0.919->0.930, c8 0.913->0.942 (worst-ON rep > best-OFF rep each), c1 flat, TTFT parity+, memory 2.614x LESS. MXFP4 verdict still BELOW (best 0.942): named residual = grouped-Marlin E=1 +7-9%/call (indirect gather + fp32 C_tmp vs vLLM dense direct-A) + ~0.7ms/step host; next lever = dense-direct-A marlin decode. Ops note: clean checkout beats a file overlay for SACRED bindings (stale-binary trap recorded). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The MXFP4 W4 binding bench, ours vs the 0.25.0 oracle on Yi30/Qwen3-8B-MXFP4 (harness q3mxfp4 key added: online_gate.py:203-224, smoke gate, orchestrator; 45/45 contract tests). Medians of 3 reps: c1 0.989x total (parity band), c2-c8 ~0.91x driven entirely by TPOT (+10-12%); TTFT at parity (2/4 cells >=1.0); peak GPU memory 2.607x LESS (28.3 vs 73.7 GiB, keep-quant). Correctness precondition reproduced on the bench binary. Attribution grounded at shape level: the deficit is the grouped Marlin W4A16 keep-quant decode GEMM as M grows 2->8; kernel-level nsys + lever dispatched. Build contract for the strict record-execution documented (real-disk build, RelWithDebInfo, oracle cutlass).
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys