row/KERNEL-MARLIN-DENSE-DIRECT: dense-marlin port arbitrated NO-GO (4-run ubench) - #50
Closed
localai-bot wants to merge 1 commit into
Closed
row/KERNEL-MARLIN-DENSE-DIRECT: dense-marlin port arbitrated NO-GO (4-run ubench)#50localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
… measured NO-GO (#50) Arbitrated the named MXFP4 residual-#1 lever (route E=1 dense projections through vLLM's DENSE marlin_gemm direct-A / m_block_size_8 instead of our MoeGroupedGemmNvfp4Marlin E=1) with the same-tool per-shape ubench the lever protocol mandates BEFORE a large port — did NOT vendor the ~2000-line dense kernel. Re-ran dgx:~/mxfp4-nsys/mxfp4_marlin_ubench.py (reference apply_fp4_marlin_linear = the exact kernel a port would vendor, vs ops.moe_wna16_marlin_gemm E=1 = the 1:1 twin of our production kernel) 3x fresh on the idle GB10; with #46 = 4 same-tool runs. Per-STEP moe/dense mean: M=1 0.997, M=2 1.007, M=4 1.010 (all noise, sign of delta flips run-to-run) and M=8 1.039 (the only reproducible signal: m_block_size_8 8-row tile vs our moe_block_size=16, ~+0.33ms/step = ~0.8% of a c8 step). The +7-9% cross-tool figure (#47/#48) is the nsys-vs-torch-profiler tool bias, NOT a recoverable kernel/dispatch cost. VERDICT: NO-GO. The port cannot reach >=1.0x on any axis (recovers ~0.8pp of the 5.8% c8 gap at M=8 only, nothing at c1-c4); vendoring 2000 lines of delicate dense marlin + a NON-byte-exact reroute of the CUDA-graph-captured decode for that would gold-plate a measured near-dead hypothesis (#46 caution). Recoverable slivers named (all cheaper than the port): force moe_block_size=8 at M=8 in DenseAlignFor (engages our already-vendored m_block_size_8 kernels, ~5 lines, no new vendoring); drop the per-call defensive workspace memset; the ~0.7ms/step host/sched slice. Box left clean (both locks free, worker down, disk 21G). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship), with the NOW/state fold included. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…xposed (#50) row/KERNEL-MARLIN-DENSE-DIRECT squash, records-only. 4-run same-tool per-shape ubench (dense marlin_gemm vs our E=1 grouped twin): M=1/2/4 within +-1% with run-to-run sign flips (true zero; the +7-9% per-call was nsys-vs-torch cross-tool bias), M=8 reproducibly +0.33ms/step from m_block_size_8 vs our block-16 padding. NO-GO on the ~2000-line port; the sliver is a ~5-line DenseAlignFor block=8 tweak on already-vendored kernels. Remaining MXFP4 terms: that tweak (~0.8pp at c8), the per-call ws Memset drop (micro), and the ~0.7ms/step host/sched slice. NOW + state folded same-change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
The #50 landing appended to the row without compacting the superseded binding text, pushing STATUS 91 chars over its ratchet (my landing chain gated doc-checkpoint but not public-doc-tables - both now required). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…l 58% (#52) row/QUANT-CT-MXFP4-C8-DIFF squash, records-only. First same-tool nsys both engines at c8 (vLLM offline in-process, identical prompts, M=8). Fair decomposition of the ~3ms TPOT gap: MARLIN +1,377us (58%) = STRUCTURAL - 5 grouped-E1 GEMMs/layer (gate+up unfused, 144 CTAs) vs vLLM's 4 dense fused (48 CTAs), +25% GEMM count/step; settles #50 (the isolated-shape ubench was blind to count+CTA structure). Flash +658us (28%, +11%/call at IDENTICAL grid, unresolved params diff). Glue +290us (12%, portable-fusion class). The #51 binding ran EAGER: classic-dense decode graph is opt-in OFF; graph-ON = byte-coherent, gap 1184->305us (=vLLM), TPOT -1.3%, SACRED gate owed before flip. STATUS ratchet lowered 287838->287832. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only arbiter: the +7-9% was cross-tool bias; M<=4 true-zero (sign flips), M=8 +0.33ms/step = the block-16 padding sliver, capturable by a ~5-line DenseAlignFor tweak on already-vendored kernels. NO-GO on the ~2000-line port. Remaining: block=8 tweak + ws-memset micro + the ~0.7ms host slice.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys