W-FP4a: route the MiniMax-H3 device DiT forward through the fp4 (Marlin W4A16) stack - #63
Closed
localai-bot wants to merge 1 commit into
Closed
W-FP4a: route the MiniMax-H3 device DiT forward through the fp4 (Marlin W4A16) stack#63localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…p4 (Marlin W4A16) stack Until now both NVFP4 loaders dequantized the packed FP4 projections to bf16 and the device forward ran vt::MatmulBT, so the sm_121a FP4 tensor-core route had NEVER run for MiniMax-H3. This wires it, adding NO new quant code. The lilcheaty/MiniMax-H3-NVFP4 checkpoint is weight-only W4A16 (no input_activations), so per the landed dispatcher (dense_nvfp4_gemm.h:12-22, 512-525, mirroring vLLM kernels/linear/__init__.py:879-881 forced-Marlin-for-a16) EVERY quantized projection routes UNIFORMLY through dense_nvfp4::MatmulNvfp4W4A16D = Marlin W4A16 single-expert, the SAME kernel the Laguna routed-experts and the dense Qwen3-32B NVFP4 arm use. Not cutlass-fp4/W4A4 (that needs fp4 activations this checkpoint lacks). fc1 is pre-merged [gate;up], so it is one W4A16 GEMM + SiluAndMul (the fused-pair path does not apply). fp32 islands and norms/biases stay bf16/f32, unchanged. Change: Nvfp4Weight carriers on the block/DiT weight structs; a fp4-resident streamer StreamMiniMaxH3Nvfp4ToDeviceFp4 (keeps the compressed-tensors triple host-resident, the shared dispatcher uploads+repacks lazily then frees the fp4 originals, so peak device memory is ~1/4 of the bf16 arm: ~16 GB vs ~66 GB); a LinearDev dispatch that routes a non-Empty fp4 projection through the W4A16 GEMM. Gate (CPU, verified): test_minimax_h3 62/62 cases, 30039 assertions, 0 failed. The synthetic-NVFP4 case streams the fp4 twin, asserts the loader kept the projections PACKED (fp4 slot set / bf16 slot Empty), runs the fp4 and bf16 device forwards on the SAME file, asserts the W4A16 dispatcher executed all 11 quantized GEMMs (the Nvfp4W4A16Stats this-path-ran counter), and bounds fp4-vs-bf16 <= 2e-3. On CPU the dispatcher has no Marlin op so it falls to the bf16 arm's own dequant+matmul (a WIRING gate here); the Marlin kernel numerics are CUDA-gated independently by test_ops_nvfp4_matmul / test_linear_method (2e-3/8e-3). PENDING: GB10 CUDA build + the real fp4-vs-bf16 delta and steady per-step timing (disk/build window); real-checkpoint t2va e2e is DISK-BLOCKED (~41 GB working set). vLLM-Omni serves no quantized H3 (BF16-only) -> HW/loader-forced-indirect. Spec: .agents/specs/minimax-h3.md §8. Row: row/H3-FP4-SPEED (helper, ROAD-V1-H3). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). Residual GPU legs (real Marlin-vs-bf16 delta + t2va e2e) stay tracked in spec §8.4, blocked on ~41 GB dgx disk headroom. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…rlin W4A16 stack (#63) row/H3-FP4-SPEED squash. Before this the H3 NVFP4 loaders dequantized FP4 to bf16 and the device forward ran vt::MatmulBT — the sm_121a fp4 route had never run for H3. Now: Nvfp4Weight carriers on the DiT/block weight structs, StreamMiniMaxH3Nvfp4ToDeviceFp4 keeps the compressed-tensors triple packed (lazy upload+repack then free — peak device ~16 GB vs ~66 GB bf16), LinearDev dispatches to dense_nvfp4::MatmulNvfp4W4A16D. The checkpoint is weight-only W4A16 (no input_activations) so every quantized projection routes uniformly to Marlin W4A16 single-expert per vLLM's own forced-Marlin-for-a16 selection — the same kernel the Laguna and Qwen3-32B NVFP4 arms use; per-shape table in spec §8.1. No new quant code. CPU gate test_minimax_h3 62/62 (30039 asserts): loader keeps projections packed, all 11 quantized GEMMs run through the W4A16 dispatcher, delta bounded. vLLM-Omni comparability (source-pinned a4ea67a2): it cannot serve a quantized H3 (fp32-island guard + hard-bf16 encoder + no GGUF wiring) — BF16-only, so the speed comparison is HW/loader-forced-indirect; supports-audit table in spec §8.2. Two brief premises corrected against source: no "DiT = 88%" figure exists in vllm-omni; the reference config is 768x1344/209f (recipe 4xB300 BF16, FL2VA 209f = 86.964 s). Residuals (spec §8.4): GB10 CUDA run of the NVFP4 case for the real Marlin-vs-bf16 delta + steady per-step, and the real-checkpoint t2va e2e — both blocked on ~41 GB of dgx disk headroom. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claim (row
row/H3-FP4-SPEED,ROAD-V1-H3) — draft PR IS the claimThe MiniMax-H3 NVFP4 arm ran its DiT projections in bf16 (both loaders
dequantized packed FP4 → bf16; the device forward called
vt::MatmulBT), so thesm_121a FP4 tensor-core route had never run for H3. W-FP4a wires it, adding
NO new quant code.
What it does (per-shape routing, grounded)
The
lilcheaty/MiniMax-H3-NVFP4checkpoint is weight-only W4A16 (noinput_activations), so per the landed dispatcherdense_nvfp4_gemm.h:12-22,512-525(mirroring vLLM
kernels/linear/__init__.py:879-881forced-Marlin-for-a16),every quantized projection routes uniformly through
dense_nvfp4::MatmulNvfp4W4A16D= Marlin W4A16 single-expert — the SAME kernelthe Laguna routed-experts and the dense Qwen3-32B NVFP4 arm use. It is not
cutlass-fp4/W4A4 (that needs fp4 activations this checkpoint lacks).
fc1ispre-merged
[gate;up], so it is one W4A16 GEMM +SiluAndMul(the fused-pair pathdoes not apply). fp32 islands + norms/biases stay bf16/f32, unchanged. Full table:
.agents/specs/minimax-h3.md§8.1.Nvfp4Weightcarriers on the block/DiT weight structsStreamMiniMaxH3Nvfp4ToDeviceFp4— keeps the compressed-tensors triplehost-resident; the shared dispatcher uploads+repacks lazily then frees the fp4
originals, so peak device memory is ~1/4 of the bf16 arm (~16 GB vs ~66 GB)
LinearDevdispatch: a non-Empty fp4 projection → the W4A16 GEMMGate (CPU, verified)
test_minimax_h362/62 cases, 30039 assertions, 0 failed. The synthetic-NVFP4case streams the fp4 twin, asserts the loader kept the projections PACKED
(fp4 slot set / bf16 slot Empty), runs fp4 + bf16 device forwards on the SAME file,
asserts the W4A16 dispatcher executed all 11 quantized GEMMs (the this-path-ran
counter), and bounds fp4-vs-bf16 ≤ 2e-3. On CPU the dispatcher has no Marlin op so
it falls back to the bf16 arm's own dequant+matmul (a wiring gate here); the
Marlin kernel numerics are CUDA-gated independently by
test_ops_nvfp4_matmul/test_linear_method(2e-3/8e-3).Pending / honest status
PENDING — build CUDA
test_minimax_h3on dgx, run the NVFP4 case (exercisesMarlin via
marlin_gemms). Blocked on a safe disk/build window.usable; encoder required for a real render).
in practice; source-audited at
a4ea67a2/v0.26.0) → HW/loader-forced-indirect(DeepSeek-GGUF precedent). The "DiT = 88% of latency" figure is not documented
in-source; the real anchor is the recipe's 4×B300 BF16 evidence.
Note on CI/preflight
agent-preflightshows 4 failures (check-env-docVT_DFLASH_*,check-fusion-consistencyminimax_h3_video_vae_device) that are pre-existing onorigin/main— neither file/var is in this diff. W-FP4a adds zero new drift.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys