MiniMax-H3 W-FP4a GB10 leg: CUDA Marlin-W4A16 speed gate + fp4-resident e2e - #64
Closed
localai-bot wants to merge 3 commits into
Closed
MiniMax-H3 W-FP4a GB10 leg: CUDA Marlin-W4A16 speed gate + fp4-resident e2e#64localai-bot wants to merge 3 commits into
localai-bot wants to merge 3 commits into
Conversation
…4-resident driver flag
The CPU wiring gate cannot reach the sm_121a FP4 tensor-core route: the existing
"an NVFP4 checkpoint loads into a runnable DiT" case runs the fp4/bf16 forwards on
a CPU queue, so on a CUDA build the W4A16 dispatcher still lands in
`fallback_gemms`, never `marlin_gemms`. This adds the GB10 leg (spec 8.4a):
- Factor the synthetic compressed-tensors NVFP4 file writer out of the CPU case
into `WriteMiniMaxH3Nvfp4File(params, path)` (behavior-identical; CPU gate still
63/63 · 30039 assertions), generalized to loop num_layers/token_refiner_layers.
- New CUDA case "the NVFP4 fp4 forward runs Marlin W4A16 on CUDA (speed)": builds
the file at REAL H3 geometry (one AdaLN block + one refiner, so exactly 11
quantized GEMMs), streams both the bf16 and the fp4-resident arms to the device
off the SAME file, runs both device forwards, and asserts `marlin_gemms == 11`
(the this-path-ran counter — on CPU these are fallback_gemms). Records the real
fp4-vs-bf16 numeric delta (Marlin bf16-TC accumulate vs the bf16 arm's
dequant+MatmulBT — same fp4 bytes, so the delta is reduction slack, not quant),
steady per-forward times for both arms (median over reps, cold leg discarded),
and a per-GEMM microbench of qkv/out/fc1/fc2 (Marlin fp4 vs the bf16 twin).
Env-tunable (H3_FP4_{LT,LH,LW,AT,AC,TEXT,REPS}) so the one binary sweeps
sequence length and reps on dgx. SKIPs cleanly with no CUDA backend.
- examples/minimax-h3-gen: `--fp4-resident` routes the NVFP4 safetensors CUDA path
through StreamMiniMaxH3Nvfp4ToDeviceFp4 (packed FP4 on device, ~16 GB vs the
bf16 stream's ~66 GB) so the real-checkpoint t2va e2e fits the 119 GiB pool.
GB10 build+run of the CUDA case, and the real-checkpoint render, land in follow-up
commits with the measured numbers. Row row/H3-FP4-GPU-E2E, ROAD-V1-H3.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
First dgx run of the CUDA speed case: on the production default the counter is `dense_gemms==11`, not `marlin_gemms` — VT_MARLIN_DENSE is default-ON, so each W4A16 projection takes vLLM's OWN dense Marlin GEMM (`vt::MarlinDenseGemm`), not the single-expert MoE-grouped Marlin. Both are the Marlin W4A16 path; the "this-path-ran" proof is `marlin_gemms + dense_gemms == 11` AND `fallback_gemms == 0` (no CPU/redundant-dequant fallback). VT_MARLIN_DENSE=0 opts into the grouped route and yields `marlin_gemms==11`. Assertion updated accordingly; the spec's "grouped MoE marlin" routing note (8.1) is corrected in the record. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…emory-win, e2e RUNS but frame is a non-scene GB10 sm_121a results for row/H3-FP4-GPU-E2E (PR #64): Leg 1 (Marlin W4A16 + timing): the fp4 path RAN on GPU — default dense_gemms==11 (VT_MARLIN_DENSE default-ON => vLLM's OWN dense Marlin GEMM, not the grouped route the spec assumed), marlin_gemms==11 under VT_MARLIN_DENSE=0, fallback_gemms==0 both. fp4-vs-bf16 BYTE-EXACT (max|diff|=0). Timing crossover per-forward bf16/fp4: 3.47x @SEQ64 (fp4 faster, memory-bound) -> 0.79-0.83x @seq4224-7040 (fp4 slower, compute-bound). fp4 W4A16 is a WEIGHT-BANDWIDTH win; for H3's large-M diffusion forward it is a MEMORY win (~16 vs ~66 GB), not speed. Leg 2 (real-checkpoint fp4-resident t2va e2e): the real 18.75 GB NVFP4 DiT + both VAEs + the GGUF Qwen3-VL-32B encoder run the WHOLE chain end-to-end -> valid mp4/wav. BUT the decoded frame is a structured non-scene patch-grid at 12/20/50 steps, conditioned or not -> OPEN render-coherence bug (device VAE decode / denoise), separate from the fp4 speed work. DiT s/step 5.45/20.0/209 s @512/768/REF-209f. Leg 3 (indirect): HW/loader-forced-indirect — 4xB300 renders 209f in 86.964 s vs 1xGB10 209 s/forward; vLLM-Omni has no quantized H3 arm. Updates: benchmark-record entry, spec 8.1 routing correction + 8.4 status, STATUS/BENCHMARKS/FEATURES rows, model-matrix + roadmap H3 rows, state + NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The render-coherence bug (structured non-scene patch-grid at real geometry, invariant to steps/conditioning) is now the lane's open front. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…in; first real-weights render, coherence OPEN (#64) row/H3-FP4-GPU-E2E squash. Leg 1: CUDA speed case at real geometry proves the sm_121a path RAN (counter-checked: dense_gemms==11 under the production default, marlin_gemms==11 under VT_MARLIN_DENSE=0, fallback==0 both ways — H3 takes vLLM's own DENSE Marlin, correcting spec §8.1's grouped assumption); real fp4-vs-bf16 delta = 0 BYTE-EXACT at every size. Steady medians: fp4 3.47x faster @SEQ64 but 0.79-0.83x @seq4224-7040 — fp4 W4A16 is a weight-bandwidth/memory win (~16 vs ~66 GB device), a ~1.2x loss in H3's large-M diffusion forwards. Leg 2: the real NVFP4 arm (~39 GB) runs the WHOLE chain e2e on one GB10 (encoder -> fp4-resident DiT -> both VAEs -> valid h264/AAC mp4 + wav; --fp4-resident driver flag). FRAME SANITY caught the real bug: frames are a structured 16-px patch-grid NON-SCENE, invariant to steps/conditioning — render coherence at real geometry is OPEN (device VAE decode / denoise convergence), independent of fp4. DiT s/step: 5.45 @512x512/22f, 20.03 @768x768/61f, 209.09 @768x1344/ 209f (the vllm-omni REF canvas; full 50-step render ~2.85h, not run). Leg 3: HW/loader-forced-indirect anchor — vLLM-Omni is BF16-only on 4xB300 (whole 209f render 86.964s) vs one GB10 fp4-resident. CPU gate 63/63 (30039 asserts). Artifacts cached dgx ~/h3fp4. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Closed
4 tasks
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ender-coherence bisection The #64 real-checkpoint NVFP4 t2va render produces a structured non-scene (a grid of 16-px multicolour blocks) IDENTICALLY at 12/20/50 steps whether conditioned or not. The rectified-flow Euler integration telescopes to (sigma0 - sigmaN) * v = 1.0 * v, so step-count invariance is the exact signature of a velocity that does NOT evolve across the denoise trajectory. This adds, env-gated and byte-identical when unset: - VT_H3_TRACE_MOTION: per-step [h3-motion] velocity + latent-motion stats (v_rms/v_amax/v_mean, per-step drows_rms, running rows_rms) plus INIT/FINAL displacement, to see whether the loop moves the latent and whether the velocity is frozen at real geometry. - VT_H3_DUMP_DIR: dump init/final video latent rows and the exact VAE-input latent as raw f32, so 12-vs-50-step and cond-vs-uncond runs can be byte/stat-compared and the VAE decode replayed on a known latent. Both documented in docs/ENVIRONMENT.md (check-env-doc green). No production path changes; the pre-existing check-fusion-consistency red (minimax_h3_video_vae_device) is untouched. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…e DiT (#70) Records the latent-bisection verdict: the VAE decoder is CORRECT (a real image encode->decode round-trip is coherent), the denoise loop moves the latent step-dependently, and the DiT emits a spatially-WHITE latent at real geometry (adjacent-cell cosine 0.06 vs 0.789 for a real encoded latent) — which the VAE faithfully renders as one independent patch per token = the grid. Not fp4 (bf16 equally white), not the attention kernel (MMA==chunk, chunk==warp==keylane), not the init noise. Overturns the #64 "device VAE decode / denoise convergence" framing. Secondary: driver init noise was uniform, not Gaussian. Updated: state.md (anchored), benchmark-record.md, spec minimax-h3 §8.4, NOW.md, STATUS/BENCHMARKS/FEATURES H3 rows. row/H3-RENDER-COHERENCE. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row row/H3-FP4-GPU-E2E (ROAD-V1-H3). The two GPU legs the disk previously blocked, run on dgx.casa GB10 sm_121a. git-archive transfer (goldens md5 byte-identical), targeted builds only.
Leg 1 — Marlin W4A16 + timing (spec §8.4a) ✅
The existing NVFP4 case runs the fp4/bf16 forwards on a CPU queue, so on CUDA it could never bump the GPU counter. New CUDA case
minimax_h3: the NVFP4 fp4 forward runs Marlin W4A16 on CUDA (speed)builds the synthetic NVFP4 file at REAL geometry (11 quantized projections) and runs both arms on a CUDA queue.dense_gemms==11(VT_MARLIN_DENSE is default-ON → vLLM's OWN dense Marlin GEMM, not the grouped MoE-marlin the spec §8.1 assumed),marlin_gemms==11underVT_MARLIN_DENSE=0,fallback_gemms==0both.Leg 2 — real-checkpoint fp4-resident t2va e2e (spec §8.4b) — RUNS; render coherence OPEN
Real NVFP4 arm downloaded (DiT
ref2va_nvfp4_full18.75 GB unpruned + both VAEs + GGUF Qwen3-VL-32B encoder). Newexamples/minimax-h3-gen --fp4-residentroutes NVFP4 safetensors throughStreamMiniMaxH3Nvfp4ToDeviceFp4(~16 GB device). The WHOLE chain runs — encoder → [16,5120] conditioning → fp4-resident DiT → both VAEs → ffmpeg → valid h264/AAC mp4 + wav.Leg 3 — indirect speed (HW/loader-forced-indirect) ✅
vLLM-Omni has no quantized H3 arm (BF16-only). 4×B300 renders 209f in 86.964 s (whole 50-step render); 1×GB10 fp4-resident is 209 s for ONE forward at the comparable canvas (~116×). Honest same-box number: fp4 0.79–0.83× the bf16 arm per forward, 4× less weight memory.
Records
benchmark-record entry, spec §8.1 routing correction + §8.4 status, STATUS/BENCHMARKS/FEATURES rows, model-matrix + roadmap H3 rows, state + NOW. CPU gate 63/63 · 30039 assertions. Only pre-existing preflight reds (VT_DFLASH_* env-doc, minimax_h3_video_vae_device fusion) remain.
Next: root-cause the H3 render-coherence bug. fp4 speed path CLOSED.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys