row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97 - #57
Closed
localai-bot wants to merge 1 commit into
Closed
row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97#57localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
… projections take vLLM's own dense marlin (48-CTA, byte-faithful, beats MoE) Executes the four scoped GB10 gates for the #55 dense-template marlin port and flips VT_MARLIN_DENSE default ON (=0 opts back out to the MoE route). WHY. The single-expert MoE-marlin route the dense E=1 projections use pads M<=8 into a 128-CTA grid; vLLM's own dense marlin covers the same tiles with 48 CTAs. The #54 par1 clamp reached 48 CTAs but regrouped the fp32 C_tmp reduce, costing a bf16 ULP that flipped a strict 32B token. This dense-template port keeps vLLM's OWN dense reduce, so it is byte-faithful to vLLM's numerics. GATES (GB10 sm_121a, RelWithDebInfo, full production stack): (a) UNIT EXEC: 2/2 cases 263/263 GREEN. First run was RED on NVFP4 but the diagnostic proved a test-authoring artifact, not a kernel bug: max|dense-moe| ==0 everywhere (dense output byte-identical to the validated MoE route), the failures were cancellation elements vs a naive fp32 ref (identical for MoE). Fix: NVFP4 vs-reference uses the cancellation-robust L2 metric; per-element byte-fidelity stays the EXACT dense==MoE check; RED injection preserved. compute-sanitizer memcheck: 0 invalid-access/race errors. (b) STRICT BATTERY dense-ON vs the oracle: 32B-NVFP4A16 6/6 (graphed AND eager, byte-identical) via the ratified near-tie razor — every dense token == vLLM's teacher-forced argmax, max gap 0.000 nats, TIGHTER than the MoE route (62 mnats). The MoE greedy anchor shifts at two exact bf16 ties, so the 32B our_ids/neartie_gap goldens are regenerated under dense-ON. Counters dense_gemms=2048, marlin_gemms=0 (gate-conditional assertion added). 0.6B/4B 184/184, async 82/82, #44 MXFP4-8B 3/3 token-exact. (c) NSYS c8: marlin kernel marlin_moe_wna16 -> marlin (dense template ran), grid 128->48 CTA, per-call median 117.8->85.8us. (d) BINDING c1..c8 x3 vs #51: total_token_throughput c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (beats #51 1.005/0.925/0.939/0.953 on EVERY axis), GPU mem 2.63x less. MXFP4 vLLM-parity goal still <1.0x c2-c8 (closer than #51), residual = decode-flash then glue. Flip conditions (parity-enablers) met: correctness ratified + beats MoE every axis + no regression + memory win. Records: state, NOW, STATUS, BENCHMARKS, FEATURES updated; STATUS ratchet lowered to the shrunk size. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…very axis (#57) row/KERNEL-MARLIN-DENSE-EXEC squash. All four #55 gates green: unit 263/263 (initial RED was a naive-fp32-reference cancellation artifact, dense output byte-identical to the MoE route; L2 metric fix + stride RED retained); 32B strict decider's two divergences RATIFIED exact bf16 ties via teacher-forcing (0.000000 nats, tighter than MoE) -> goldens regenerated per the razor, 6/6-144 graphed+eager at the flipped default (dense_gemms=2048, marlin_gemms=0); canaries 0.6B/4B/async/smoke green; memcheck 0; nsys: marlin::Marlin 48 CTAs, per-call 117.8->85.8us. Binding vs #51: c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969, TPOT up everywhere, mem 2.63x LESS. MXFP4 parity gate still NO (best c8 0.969); residual = decode-flash +658-784us (next-dominant), glue tail, host. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship) after resolving the ratchet-constant merge conflict; the premature close below was my sequencing error, the content is landed. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…el side CLOSED Direct measurement (params dump + pointer attributes + ncu) on the post-#57 binary: all flash buffers DEVICE-resident, KV layout/params byte-match vLLM, decode kernel latency-bound identically both engines; sole divergence = near-tie num_splits heuristic version skew that self-corrects at c8. Remaining MXFP4 terms: the ~0.7ms/step frontend slice (dominant), the oracle-gated c1-c2 num_splits cap, the glue tail. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…el side CLOSED Direct measurement (params dump + pointer attributes + ncu) on the post-#57 binary: all flash buffers DEVICE-resident, KV layout/params byte-match vLLM, decode kernel latency-bound identically both engines; sole divergence = near-tie num_splits heuristic version skew that self-corrects at c8. Remaining MXFP4 terms: the ~0.7ms/step frontend slice (dominant), the oracle-gated c1-c2 num_splits cap, the glue tail. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This was referenced Aug 6, 2026
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…4 — CPU -Werror repaired (#62) row/BUILD-CPU-WERROR-MOE-GROUPED-TESTFIX squash, companion to #61. The #57 grouped-MoE test helpers are used only inside the VT_MARLIN_NVFP4 region but defined unguarded, failing the plain CPU-only Release -Werror build at test_ops_moe_grouped.cpp:130/:173. A make -k sweep of the full suite confirms these are the LAST instances of the class. TU compiles clean after; no behavior change on any configuration that compiled before. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…l is FLASH-dominant (#67) row/QUANT-CT-MXFP4-FUSED-GLUE squash, records-only W0. The funded fused-glue Marlin kernel mirrors a fusion vLLM does not perform for this model: RMSNormQuantFusionPass/ActivationQuantFusionPass are activation-quant patterns that never fire for W4A16 bf16 activations (rms_quant_fusion.py:629-669, act_quant_fusion.py:296-320, marlin_utils.py:704-717); ops.marlin_gemm is a register_fake'd extern op Inductor cannot decompose (_custom_ops.py:1200-1247); and the same-tool #57 trace shows vLLM runs MORE glue launches than us (299 vs 255). The prior "Inductor GEMM-epilogue fusion" attribution came from a cross-tool read. Honest c8 decomposition under the dense-direct default (+951us marlin already banked by efa6e40): flash +784us (71%) / glue +195us (18%) / marlin residual +226us — even glue->0 leaves ~0.977. Public surfaces corrected (STATUS/BENCHMARKS/NOW). The REAL unexhausted lever: same flash_fwd_splitkv kernel, identical 1x3x64 grid, +22us/call slower — a STRUCTURAL context/residency audit the num_splits refutation never addressed. Surpass-rung fused-Marlin scoped + costed (~150-198us, sub-parity) — not recommended as the parity path. Owed on the box: fresh same-tool c1-c8 bind on current main + the flash audit. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…hanism NAMED (missing -use_fast_math) W1: analyze_decode.py on the #57 same-tool c8 traces confirms flash MAIN ours 178.8 vs vLLM 156.3 us/call (+807us/step), combine tied, totals reconcile — REAL, not an artifact. W2: cuobjdump/ptxas arbiter names the mechanism as a build flag — vLLM's flash-attn is compiled --use_fast_math, ours was not (5448 instrs/REG246 vs 4832/REG255 with it, matching vLLM 4880/REG255; HMMA/LDSM/LDGSTS byte-identical). Fix implemented b3ef458; built-object verified 4832/REG255. Runtime + gate battery in flight (box shared with H3). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…4 flash +807us decode term (#68) QUANT-CT-MXFP4-FLASH-AUDIT closes the #67-OWED flash audit: why the IDENTICAL-grid flash_fwd_splitkv MXFP4-8B decode kernel runs ~+22us/call slower than the vLLM oracle. W1 (fresh same-tool reconciliation, analyze_decode.py on the #57 c8 traces): flash MAIN ours 178.8 vs vLLM 156.3 us/call (+807us/step), combine tied, totals reconcile — REAL, not a reconciliation artifact. W2 (cuobjdump/ptxas arbiter, nvcc/ptxas 13.0.88 both sides): kernel-version REFUTED — vLLM v0.25.0 pins vllm-flash-attn @ 2c839c33, the exact commit we vendored, so flash SOURCE is byte-identical. The real difference is a BUILD FLAG: vLLM's flash-attn is compiled --use_fast_math; our vendored FA2 TUs were not. Decode kernel <128,64,128,4>: ours 5448 instrs/REG246 -> 4832/REG255 with fast-math (= vLLM's 4880/REG255), with HMMA(768)/LDSM(408)/LDGSTS(120) byte-identical. The +616 scalar instrs (precise div/rsqrt) = +11.6% ~ the +14.3% per-call time at the latency-bound decode tile. hd256 (27B/35B) also leans 4928->4560. Ruled out: -maxrregcount=255 (5424/246); our own nvcc-13 compute_80 PTX (5440). FIX: CMakeLists flash TU COMPILE_OPTIONS += -use_fast_math (mirrors flash-attn's own build). Real sm_121a build's flash object VERIFIED = 4832 instrs/REG255. Non-byte-exact (fast-math div/rsqrt) => runtime nsys A/B + SACRED battery OWED (box-contended by the active H3 campaign); NO default flipped. Full forensics in .agents/benchmark-record.md (QUANT-CT-MXFP4-FLASH-AUDIT) + spec; STATUS/BENCHMARKS/NOW/state updated. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ash regression); gap is occupancy/L2, not the SASS QUANT-CT-MXFP4-FLASH-AUDIT runs the #67-OWED flash audit to a MEASURED verdict. W1: fresh same-tool c8 decode-window — the flash gap on CURRENT main is +12.5us/call (+450us/step) ours 168.8 vs vLLM 156.3, SMALLER than #57's +807 (leaner marlin/glue). W2 lens 1 (cuobjdump HYPOTHESIS): vLLM's flash-attn is --use_fast_math, ours was not (same 2c839c33 src; +fast-math -> 4832/REG255 = vLLM's 4880/255). W2 lens 2 (MEASURED, the arbiter): a controlled same-build nsys A/B REFUTES it — -use_fast_math makes flash +21us/call SLOWER (168.8->189.8); the kernel is memory-latency-bound (ncu occ 8.3%, register-limited), so the higher reg count (246->255) lowers occupancy, which dominates. cuobjdump was necessary but NOT sufficient. So the flag is REVERTED (a CMakeLists NOTE records why); no functional code ships. Real residual = occupancy/L2 (ncu: 38% smem-scoreboard + 37% barrier stalls); vLLM-side ncu owed (box OOM-rebooted twice under 3-way contention). No default flip (no throughput win). Records + spec updated. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…idual is occupancy/L2 (#69) row/QUANT-CT-MXFP4-FLASH-AUDIT squash, records + a CMakeLists NOTE (no functional change ships). Fresh same-tool c8 decode-window on current main: flash +12.5us/call (+450us/step) — smaller than #57's +807 (the leaner dense-marlin context already recovered part). The compile-lens hypothesis was MEASURED and REFUTED: vLLM pins vllm-flash-attn at the exact commit we vendored (source byte-identical; the earlier ~5824e6e note was wrong); adding vLLM's -use_fast_math reproduces its SASS (4832 vs 4880 instrs, REG255, HMMA/LDSM/LDGSTS identical) yet runs +21us/call SLOWER — the 246->255 register bump cuts occupancy on a latency-bound kernel (ncu: 8.3% occupancy, L2 53%, smem-scoreboard + CTA-barrier stalls dominate). Rejected + reverted with a NOTE so it is not re-tried; cuobjdump is necessary but not sufficient — runtime arbitrates. vLLM at the same 255 regs is still 156.3us/call, so its edge is L2/scheduling: the next lever is flash occupancy (register pressure / launch bounds) or barrier/smem stalls. The precise ours-vs-vLLM ncu diff is OWED on an idle box — the shared GB10 OOM-rebooted twice under 3-way campaign contention during this run. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dense-template execution campaign: unit 263/263 (initial RED = naive-fp32-reference cancellation artifact, dense byte-identical to the MoE route; L2 metric fix); the 32B strict decider's two divergences RATIFIED as exact bf16 ties by teacher-forcing (0.000000 nats — dense tracks vLLM's own argmax tighter than the MoE route), goldens regenerated per the razor; 6/6-144 graphed+eager at the flipped default with dense_gemms=2048/marlin_gemms=0; memcheck clean; nsys confirms marlin::Marlin at 48 CTAs, per-call 117.8->85.8us. Binding vs #51: c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (every axis up, no regression), memory 2.63x WIN. MXFP4 parity: c1 parity+, c2-c8 best-yet 0.96-0.97, gate still NO; residual = decode-flash (+658-784us, next-dominant) then glue + host.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys