row(KERNEL-MARLIN-DENSE-PORT): byte-preserving dense-template marlin for E=1 W4A16 (gated OFF) - #55
Closed
localai-bot wants to merge 1 commit into
Closed
row(KERNEL-MARLIN-DENSE-PORT): byte-preserving dense-template marlin for E=1 W4A16 (gated OFF)#55localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…GEMM (gated OFF) Row KERNEL-MARLIN-DENSE-PORT. Vendors vLLM's OWN dense marlin GEMM (the direct-A, tile-per-CTA W4A16 kernel it ships for a16 weight-only linears) as a new vt::MarlinDenseGemm op, and routes the E=1 dense NVFP4/MXFP4 projections (dense_nvfp4_gemm.h MatmulNvfp4MarlinD / MatmulMxfp4W4A16D / GateUpFusedMarlinD) through it behind VT_MARLIN_DENSE (default OFF). WHY: #54 proved clamping the single-expert MoE-marlin route to 48 CTAs (VT_MARLIN_E1_PAR1) recovers 81% of the marlin residual (per-call 114.7 vs vLLM 113.1) but the par regrouping of the fp32 C_tmp reduce costs one bf16 ULP that flips a strict token on the 64-layer 32B-NVFP4A16 (test_qwen3_32b_nvfp4a16:344). The dense template is vLLM's OWN dense reduce structure — the byte-preserving fix — so at M<=8 it runs the sms-wide (48-CTA) grid WITHOUT that ULP. Vendored (src/vt/cuda/marlin/libtorch_stable/quantization/marlin/), all cited from vLLM @ 555967922 csrc/libtorch_stable/quantization/marlin/: * kernel.h <- kernel.h (verbatim; namespace marlin, +lda param) * marlin_template.h <- marlin_template.h:1-2081 (verbatim dense kernel; the SHARED marlin.cuh/marlin_dtypes.cuh/dequant.h/ marlin_mma.h it includes are diff-verified byte-identical to our existing vendored copies) * marlin_mm_dense.{h,cu} <- marlin.cu:326-541 marlin::marlin_mm (+ config helpers); torch::stable marlin_gemm wrapper stripped; only the redundant inner is_a_8bit shadow dropped (identical value, avoids -Wshadow); STD_TORCH_CHECK via vt_marlin_check.h * kernel_selector.h, sm80_kernel_...fe2m1f...cu <- generate_kernels.py output. KEY: the dense kernel is a DISTINCT kernel body but the SAME 12-param Marlin<> template as the MoE TUs, so the instantiation set is shared; the dense body + namespace marlin come from the local kernel.h. New op + launcher: * vt::OpId::kMarlinDenseGemm + MarlinDenseArgs + MarlinDenseGemmFn (ops.h, appended before kCount — no id shift); dispatch shim (ops.cpp) * src/vt/cuda/cuda_marlin_dense.cu — vt::Tensor launcher mirroring cuda_moe_marlin.cu (graph-safe c_tmp pool, dense c_tmp sizing marlin.cu:713). Routing reuses the EXISTING resident weights + workspace (same marlin_permute repack for dense and MoE — CONFIRMED, no shim) with rank-2 operand views and NO moe_align gather. dense_gemms execution counter added (the "path RAN" signal). Gates: CPU -fsyntax-only CLEAN on ops.cpp and the VT_MARLIN_NVFP4 routing header. GPU compile: all 3 new dense .cu compile CLEAN on dgx GB10 sm_121a under the EXACT production flags (-Werror=all-warnings, -static-global-template-stub=false, --generate-code=...sm_121a). RED-first unit battery WRITTEN (test_ops_moe_grouped.cpp: NVFP4+MXFP4, M=1..8 x 3 shapes, dense-vs-CPU-ref AND dense-vs-grouped-route, row-shifted stride RED-injection). GPU EXEC gates (unit run + strict token battery dense-ON vs oracle incl. 32B-NVFP4A16:344 + launch-counter + nsys 48-CTA + binding c1..c8 x3) are the scoped dgx follow-up; default stays OFF until the strict battery proves oracle byte-match and the binding beats the MoE route. Records: state.md KERNEL-MARLIN-DENSE-PORT, parity-ledger, porting-inventory §10, kernel-matrix, STATUS/BENCHMARKS/FEATURES. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
marked this pull request as ready for review
August 6, 2026 08:45
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ied on GB10 (#55) row/KERNEL-MARLIN-DENSE-PORT squash. vLLM's dense marlin (direct-A, tile-per-CTA, its own par-split fp32 reduce = vLLM's numerics) vendored from the pin: kernel.h + marlin_template.h verbatim, marlin_mm_dense launcher lift, shared instantiation set (kernel_selector + sm80 TU); shared .cuh/mma/dequant diff-verified byte-identical to our moe copies. New OpId::kMarlinDenseGemm + cuda_marlin_dense.cu (graph-safe c_tmp pool); routed in MatmulNvfp4MarlinD/GateUpFusedMarlinD behind VT_MARLIN_DENSE default OFF, reusing the SAME repacked residents (no layout shim - vLLM shares marlin_permute). RED-first unit battery written (NVFP4+MXFP4, M=1..8, dual-reference + stride RED injection). GB10 sm_121a nvcc compile CLEAN under production flags; full exec gates scoped: unit exec + memcheck -> strict battery vs ORACLE (32B 142/142 is the decider) -> nsys 48-CTA confirm -> binding c1-c8. Flip iff oracle byte-match everywhere + binding beats the MoE route. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…L2 metric The #55 dense NVFP4 unit case compared a bf16 tensor-core marlin result to a naive fp32-accumulate CPU reference at a per-element 3e-2 tolerance. GB10 exec (its first run) showed that fails at cancellation-heavy output elements (true |ref| ~0.04-5, summands ~O(1e3)) where bf16 rounding is O(1) — IDENTICALLY for the validated MoE route (measured max|dense-moe|==0 across all shapes/M). Swap the vs-reference checks (dense and MoE) to L2-relative (the standard cancellation-robust GEMM metric); the per-element byte-preservation claim stays the EXACT dense==MoE check, and the row-shift RED injection still discriminates (uncorrelated => ratio ~1 >> rtol). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… projections take vLLM's own dense marlin (48-CTA, byte-faithful, beats MoE) Executes the four scoped GB10 gates for the #55 dense-template marlin port and flips VT_MARLIN_DENSE default ON (=0 opts back out to the MoE route). WHY. The single-expert MoE-marlin route the dense E=1 projections use pads M<=8 into a 128-CTA grid; vLLM's own dense marlin covers the same tiles with 48 CTAs. The #54 par1 clamp reached 48 CTAs but regrouped the fp32 C_tmp reduce, costing a bf16 ULP that flipped a strict 32B token. This dense-template port keeps vLLM's OWN dense reduce, so it is byte-faithful to vLLM's numerics. GATES (GB10 sm_121a, RelWithDebInfo, full production stack): (a) UNIT EXEC: 2/2 cases 263/263 GREEN. First run was RED on NVFP4 but the diagnostic proved a test-authoring artifact, not a kernel bug: max|dense-moe| ==0 everywhere (dense output byte-identical to the validated MoE route), the failures were cancellation elements vs a naive fp32 ref (identical for MoE). Fix: NVFP4 vs-reference uses the cancellation-robust L2 metric; per-element byte-fidelity stays the EXACT dense==MoE check; RED injection preserved. compute-sanitizer memcheck: 0 invalid-access/race errors. (b) STRICT BATTERY dense-ON vs the oracle: 32B-NVFP4A16 6/6 (graphed AND eager, byte-identical) via the ratified near-tie razor — every dense token == vLLM's teacher-forced argmax, max gap 0.000 nats, TIGHTER than the MoE route (62 mnats). The MoE greedy anchor shifts at two exact bf16 ties, so the 32B our_ids/neartie_gap goldens are regenerated under dense-ON. Counters dense_gemms=2048, marlin_gemms=0 (gate-conditional assertion added). 0.6B/4B 184/184, async 82/82, #44 MXFP4-8B 3/3 token-exact. (c) NSYS c8: marlin kernel marlin_moe_wna16 -> marlin (dense template ran), grid 128->48 CTA, per-call median 117.8->85.8us. (d) BINDING c1..c8 x3 vs #51: total_token_throughput c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (beats #51 1.005/0.925/0.939/0.953 on EVERY axis), GPU mem 2.63x less. MXFP4 vLLM-parity goal still <1.0x c2-c8 (closer than #51), residual = decode-flash then glue. Flip conditions (parity-enablers) met: correctness ratified + beats MoE every axis + no regression + memory win. Records: state, NOW, STATUS, BENCHMARKS, FEATURES updated; STATUS ratchet lowered to the shrunk size. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…very axis (#57) row/KERNEL-MARLIN-DENSE-EXEC squash. All four #55 gates green: unit 263/263 (initial RED was a naive-fp32-reference cancellation artifact, dense output byte-identical to the MoE route; L2 metric fix + stride RED retained); 32B strict decider's two divergences RATIFIED exact bf16 ties via teacher-forcing (0.000000 nats, tighter than MoE) -> goldens regenerated per the razor, 6/6-144 graphed+eager at the flipped default (dense_gemms=2048, marlin_gemms=0); canaries 0.6B/4B/async/smoke green; memcheck 0; nsys: marlin::Marlin 48 CTAs, per-call 117.8->85.8us. Binding vs #51: c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969, TPOT up everywhere, mem 2.63x LESS. MXFP4 parity gate still NO (best c8 0.969); residual = decode-flash +658-784us (next-dominant), glue tail, host. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
KERNEL-MARLIN-DENSE-PORT— byte-preserving dense-template marlin for the E=1 W4A16 path (gated OFF)The byte-preserving completion of the marlin structural lever (#50/#52/#54). Vendors
vLLM's own dense marlin GEMM and routes the E=1 dense NVFP4/MXFP4 projections
through it behind
VT_MARLIN_DENSE(default OFF).Why
#54 proved clamping the single-expert MoE-marlin route to 48 CTAs (
VT_MARLIN_E1_PAR1)recovers 81% of the c8 marlin residual (per-call 114.7 vs vLLM 113.1) but the par
regrouping of the fp32 C_tmp reduce costs one bf16 ULP that flips a strict token on the
64-layer 32B-NVFP4A16 (
test_qwen3_32b_nvfp4a16_paged_engineREQUIRE :344). The densetemplate is vLLM's own dense reduce structure — the byte-preserving fix — so at M<=8 it
runs the sms-wide (48-CTA) grid WITHOUT that ULP.
What
555967922csrc/libtorch_stable/quantization/marlin/):kernel.h,marlin_template.h(verbatim dense kernel, 2081L),marlin_mm_dense.{h,cu}(
marlin.cu:326-541torch-free),kernel_selector.h+sm80_*.cu(generator output).Key: the dense kernel is a DISTINCT body but the SAME 12-param
Marlin<>template as theMoE TUs, so the instantiation set is shared; the shared
.cuh/marlin_mma.h/dequant.hwere diff-verified byte-identical to our existing vendored copies.
vt::MarlinDenseGemm(kMarlinDenseGemm, appended beforekCount) + launchercuda_marlin_dense.cu(mirrorscuda_moe_marlin.cu; graph-safe c_tmp pool).dense_nvfp4_gemm.h, gatedVT_MARLIN_DENSEdefault OFF) reuses the existingmarlin resident + workspace (same
marlin_permuterepack — confirmed, no shim), rank-2views, no moe_align.
dense_gemmsexecution counter added.Gates
-fsyntax-onlyclean (ops.cpp+ theVT_MARLIN_NVFP4routing header)..cucompile CLEAN on dgx GB10 sm_121a under the exact productionflags (
-Werror=all-warnings,-static-global-template-stub=false,...sm_121a).test_ops_moe_grouped.cpp: NVFP4+MXFP4, M=1..8 x 3shapes, dense-vs-CPU-ref AND dense-vs-grouped-route, row-shifted stride RED-injection).
check-doc-checkpoint,check-public-doc-tables,check-now-current,check-state-order,check-env-doc).Scoped follow-up (dgx GPU campaign — default stays OFF until these pass)
Extracted tree is at dgx
~/dense_check/vllm.cpp.0.6B/4B, row/SERVE-ASYNC-DENSE-MIRROR: classic-dense async device-mirror fix + MXFP4 default-config e2e CLOSED #44 MXFP4 smoke, async; + launch-counter assertion. FLIP DEFAULT ON iff oracle
byte-match everywhere; else apply the near-tie razor before any regen.
RelWithDebInfo build; updated parity verdict + residual map (flash +784 expected next).
Draft: opened as the row claim; ready for the operator to drive the GPU battery.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys