row/H3-RENDER-COHERENCE: root-cause the H3 real-geometry render bug (non-scene patch-grid) - #70
row/H3-RENDER-COHERENCE: root-cause the H3 real-geometry render bug (non-scene patch-grid)#70localai-bot wants to merge 6 commits into
Conversation
…ender-coherence bisection The #64 real-checkpoint NVFP4 t2va render produces a structured non-scene (a grid of 16-px multicolour blocks) IDENTICALLY at 12/20/50 steps whether conditioned or not. The rectified-flow Euler integration telescopes to (sigma0 - sigmaN) * v = 1.0 * v, so step-count invariance is the exact signature of a velocity that does NOT evolve across the denoise trajectory. This adds, env-gated and byte-identical when unset: - VT_H3_TRACE_MOTION: per-step [h3-motion] velocity + latent-motion stats (v_rms/v_amax/v_mean, per-step drows_rms, running rows_rms) plus INIT/FINAL displacement, to see whether the loop moves the latent and whether the velocity is frozen at real geometry. - VT_H3_DUMP_DIR: dump init/final video latent rows and the exact VAE-input latent as raw f32, so 12-vs-50-step and cond-vs-uncond runs can be byte/stat-compared and the VAE decode replayed on a known latent. Both documented in docs/ENVIRONMENT.md (check-env-doc green). No production path changes; the pre-existing check-fusion-consistency red (minimax_h3_video_vae_device) is untouched. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
|
Cross-posting findings from an independent NVFP4 investigation on the GGUF/render side, in case they save a bisection step. I hit what is very likely the same wall and could not close it either. What I ruled out on the NVFP4 arm, with evidence:
A gate trap worth knowing about, since it bit me here: our NVFP4 streaming loader gated against the non-streaming one at Two suspects I could not eliminate, both external to our code:
Separately, and this is the one that mattered for render quality on the GGUF path: the artifact there was quantization, not geometry. A controlled A/B (same prompt, seed, code, VAE - only the DiT quant changed) took a murky lattice-covered silhouette to a photoreal close-up going Q3_K_M -> Q4_K_M. ComfyUI PR 15298 explains why: H3's partial split-half RoPE creates channel-wise magnitude outliers that corrupt INT8 quantization, so 3-bit is well past the edge. If the NVFP4 non-scene survives every geometry check, that outlier sensitivity is worth weighing as a cause rather than a coincidence. Your step-count-invariance argument is a much sharper instrument than anything I had - |
…E_PROBE) The render-coherence bisection localized the bug BELOW the denoise loop: final latents differ byte-for-byte at 3/12/50 steps (loop moves the latent; velocity evolves 1.4->5.5 rms), yet the frame is a hard grid of independent 16px blocks at the VAE latent-cell scale. To isolate the video VAE decoder, this perturbs one interior spatial latent cell, re-decodes, and prints the per-16px-block RMS-change map: a single hot block proves the decoder does not mix tokens spatially. Documented; byte-identical when unset. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
…atent (no DiT) The VAE-branch oracle test: decode the SAME real latent (dumped via VT_H3_DUMP_DIR) through the device ViT3D path (cuda) OR the scalar CPU reference (cpu, gated vs upstream at 8.9e-8), with no DiT and no conditioning loaded. Divergence between the two localizes a device-kernel bug at real seq; agreement means the grid is inherent to the latent. Self-contained early branch; unused paths and validation unchanged. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
…+encoder-latent dump) The DiT-independent test that exonerated the VAE: encode a real image through the video VAE encoder, apply post_quant_conv, decode. On the real checkpoint a structured test pattern round-trips to a COHERENT image (same bars, timecode, diagonal) with no grid -- proving the ViT3D decoder, post_quant_conv and the device path are correct, and localizing the render bug to the DiT-produced latent (which is spatially white). VT_H3_DUMP_DIR also dumps the encoder's (in-distribution) latent for spatial-structure comparison against the DiT's. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
…N(0,1), not uniform Found a real secondary fidelity bug while bisecting: the driver seeded the diffusion initial noise from uniform[-1,1] (std 0.577), but a flow-matching model is trained on Gaussian N(0,1) (torch.randn). The old comment claiming the noise distribution "does not matter" is wrong -- only the exact VALUES (RNG identity) don't matter, the DISTRIBUTION does. VT_H3_GAUSSIAN_NOISE=1 draws Box-Muller Gaussians (INIT rms 0.58 -> 1.0). A/B verdict: it is NOT the render bug (latent adjacent-cos 0.057 uniform -> 0.077 Gaussian, still white vs the 0.789 of a real encoded latent) but is a correctness deviation from upstream worth flipping. Left as a toggle pending the operator's default call. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] Row: row/H3-RENDER-COHERENCE (#70)
…e DiT (#70) Records the latent-bisection verdict: the VAE decoder is CORRECT (a real image encode->decode round-trip is coherent), the denoise loop moves the latent step-dependently, and the DiT emits a spatially-WHITE latent at real geometry (adjacent-cell cosine 0.06 vs 0.789 for a real encoded latent) — which the VAE faithfully renders as one independent patch per token = the grid. Not fp4 (bf16 equally white), not the attention kernel (MMA==chunk, chunk==warp==keylane), not the init noise. Overturns the #64 "device VAE decode / denoise convergence" framing. Secondary: driver init noise was uniform, not Gaussian. Updated: state.md (anchored), benchmark-record.md, spec minimax-h3 §8.4, NOW.md, STATUS/BENCHMARKS/FEATURES H3 rows. row/H3-RENDER-COHERENCE. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
5b9d6e4 to
cb0dbde
Compare
|
Landed on main as the squash (mudler authorship), with the Gaussian init-noise default flipped to upstream semantics per the mirror policy (VT_H3_GAUSSIAN_NOISE=0 keeps the legacy uniform draw). Next campaign: the DiT parity gate at real TOKEN geometry (8x8+) vs the upstream CPU oracle at reduced hidden dims — pinning the divergent op does not need real checkpoint scale. 🤖 Generated with Claude Code |
…loop/kernels all ruled out (#70) row/H3-RENDER-COHERENCE squash. Latent bisection on the real NVFP4 arm OVERTURNS the prior framing: the denoise loop MOVES the latent (byte- distinct at 3/12/50 steps, velocity evolves 1.40->5.52 rms); the video VAE decoder is CORRECT (a real test pattern round-trips coherently — the decisive rung); the DiT emits a spatially-WHITE latent at real token grids (adjacent-cell cosine 0.06 vs 0.789 for a real encode) and the VAE faithfully renders one independent patch per cell = the grid. fp4-vs-bf16 and the attention kernels are ruled out (both arms white; MMA==chunk==warp). Gates missed it because the DiT parity gate runs 2x3 spatial tokens; the divergence appears between 2x3 and the real 8x8 — the recurring below-one-tile blind spot. Ships: five env-gated diagnostics + driver modes (VT_H3_TRACE_MOTION, VT_H3_DUMP_DIR, VT_H3_VAE_PROBE, --decode-latent, --roundtrip), byte- identical to production when off, and a REAL secondary bug fixed at upstream semantics per the mirror policy: init noise was uniform[-1,1] (std 0.577) where flow matching needs torch.randn N(0,1) — Gaussian is now the DEFAULT (VT_H3_GAUSSIAN_NOISE=0 keeps the legacy draw for A/B). Not the render fix on its own. Next (spec §8.4): extend the DiT parity gate to real TOKEN geometry (8x8+) at reduced hidden dims vs the upstream CPU oracle — a spatial- mixing bug should reproduce with random weights; real checkpoint scale is not required to pin the divergent op. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ometry ladder (#70 follow-up) Helper claim (DRAFT PR is the claim) for the #70 follow-up: extend the reduced-dim DiT-forward parity gate into a geometry ladder (2x3 -> 8x8 + rectangle + temporal + packed mix) to pin the spatial-mixing divergence at real token geometry. CPU-only, no dgx. Harness verified: the gen script reproduces the checked-in H3 goldens byte-identically against local vllm-omni. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…-mixing-math hypothesis (2x3->8x8+temporal, host+device) The section-5 DiT parity gate only ever ran spatial 2x3. #70 root-caused the render grid to the DiT emitting a spatially-white latent at real token geometry (8x8), so the divergence lives strictly between 2x3 and 8x8. This extends the reduced-dim gate into a GEOMETRY LADDER to pin the divergence at real geometry -- CPU-only, no dgx. Built (scripts/gen-minimax-h3-goldens.py :: emit_dit_ladder): 7 rungs 2x3->4x4->6x6->8x8 + a 4x8 rectangle + an 8x8x3-frame temporal 3D grid + a 6x10x5-frame video+audio packed mix, each emitting the upstream packed-sequence layout, the RefDiT forward logits, and a spatial-mixing probe. Two permanent gate cases in test_minimax_h3.cpp: - "DiT-forward geometry ladder matches upstream (host+device, mixing)" - "DiT device-vs-host forward holds at the REAL head_dim=128 ratio" Goldens regenerate byte-identically against local ~/_git/vllm-omni; suite 65/65 (was 63/63). Result: ours == the RefDiT oracle at EVERY rung, HOST and DEVICE forward, max|diff| <= 3e-7 vs the 2e-5 gate; the mixing probe shows every video-target token couples through the packed bidirectional attention (fraction 1.0). The real-head_dim=128 leg has device==host <= 1.2e-6 across all rungs. The #70 adjacent-cell COSINE does NOT translate here: with random weights the CORRECT oracle is already spatially white by that metric (coherence is a TRAINED-weights property), so the harness's valid discriminators are oracle-logit equality and information flow, both green. VERDICT: the "spatial-mixing bug in the DiT-forward MATH" hypothesis is REFUTED at reduced dims and real token geometry. The #70 white latent is trained-weights / real-scale, not a reduced-dim DiT-forward bug; a GPU re-render is NOT expected to be coherent from this work. Residuals (beyond CPU): a bug shared by our port AND the RefDiT restatement vs true upstream (close on the dgx oracle venv), and the real-scale DiT INPUT wiring (encoder embeds / real position grid / timesteps). Pre-existing check-fusion-consistency red (minimax_h3_video_vae_device) is not this row's. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ken grids; mixing hypothesis REFUTED (#74) row/H3-DIT-SCALE-GATE squash, CPU-only. The #70 redirect ("the spatial- mixing divergence should reproduce at real TOKEN geometry with random weights") is REFUTED the right way: a 7-rung geometry ladder (2x3 -> 8x8, rectangle, 3-frame temporal, 5-frame video+audio packed mix) vs the upstream oracle at reduced hidden dims passes EVERY rung at <=3e-7 host / <=1.2e-6 device (incl. the real head_dim=128 rope ratio), and a perturbation probe shows mixing fraction 1.0 everywhere — the packed bidirectional attention couples all video tokens at real geometry. Load-bearing metric finding: the #70 adjacent-cell-cosine whiteness is MEANINGLESS under random weights — the correct reference is already white by that metric; spatial coherence is a trained-weights property. The ladder ships as two permanent gate cases (suite 65/65) so the below-one-tile blind spot stays closed. The render-bug hunt narrows to two surfaces, both needing the box: (1) a shared blind spot of our port AND the RefDiT restatement vs the TRUE upstream module (arbitrate on the dgx oracle venv); (2) the real-scale DiT INPUT wiring (encoder embeds, fp64 position grid at full canvas, per-token timesteps) — the one surface #70 never isolated. Queued behind the flash-occupancy box campaign. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…(1920) S1 is exonerated: the real-scale DiT INPUTS (packed layout, fp64 position grid, token tags, inverse/combined AdaLN indices, sigmas) diff EXACTLY against upstream pipeline_minimax_h3.py at 512x512/22f (verified on dgx: text_len=8, latent 7x32x32, seq_len 1920); the tokenization matches upstream byte-for-byte; the encoder conditioning is correctly shaped and carries the expected Qwen massive-activation structure; and DequantNvfp4ToBf16 is the shared helper the Laguna and Qwen3-32B NVFP4 arms already prove byte-exact. That leaves ONE untested surface: #74's device-vs-host ladder (incl. the REAL head_dim=128 case) runs on the CPU BACKEND, and the CUDA cases run only at the small fl2va geometry. The CUDA kernels at the REAL render seq (t2va -> latent 7x32x32 -> seq_len 1920, cu_seqlens=[0,1874,1920] non-causal 2-document) at head_dim=128 have never been gated against the trusted host loops. A scale-dependent CUDA-kernel bug (varlen non-causal attention / RoPE cache / AdaLN modulate) would leave every reduced-dim gate green while the render goes white. This case runs the SAME MiniMaxH3DitForwardDevice on the CUDA backend vs the CPU host forward at exactly the render geometry and the step-0 timestep partition. A divergence IS the #70 bug; a match points the hunt at S2 (a shared port<->RefDiT restatement blind spot vs true upstream). Skips without a CUDA backend. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
ROOT CAUSE of the #70 white render FOUND: the render ran task=t2va on the `minimax_h3_ref2va_nvfp4_full` checkpoint, which is the REF2VA partition. Upstream serves t2va/fl2va from the FL2VA partition and ref2va from the Ref2VA partition, and "task must match the served partition" (recipes/MiniMaxAI/MiniMax-H3.md:50,289; _resolve_task raises otherwise). A ref2va-trained DiT fed a t2va sequence (no reference block) is out of distribution -> the spatially-degenerate latent, invariant to the text prompt and step count -- exactly #70's symptom. Every render so far used the wrong task for this checkpoint; the DiT forward, its inputs, the NVFP4 dequant and the CUDA kernels are all correct (verified: S1 inputs byte-exact vs upstream at real 512x512/22f scale; CUDA device forward == CPU host at seq 1920; forward math == upstream source == RefDiT). Running the CORRECT task (ref2va) surfaced a real, previously-unexercised bug in the OUTPUT path: BuildMiniMaxH3PackedSequenceRef2va PREPENDS pinned reference rows (encoded image/video/audio) to the packed layout, the DiT zeroes them in its output (skip_mask_out_condition), and MiniMaxH3GenerateT2va then handed the full (reference + target) row buffer to unpatchify/unpack -- which rejected the non-divisible count ("rows not divisible by t*h*w"). The generated clip is only the TRAILING target rows. Slice to them before unpatchify/unpack. t2va/fl2va have no reference prefix, so the tail is the whole buffer and the change is a no-op there. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Records the H3-RENDER-CLOSE result across the canonical surfaces. The #70/#74 white render was t2va run on the REF2VA-partition checkpoint; upstream serves t2va/fl2va from the FL2VA partition and requires the task to match the partition. t2va on the FL2VA GGUF DiT renders a COHERENT, prompt-matched scene on GB10 (VAE-input latent adj-cell cosine 0.95 vs 0.06 white, no 16px patch grid, valid h264/AAC mp4). Before switching partitions, verified: the t2va DiT inputs diff byte-exact vs upstream at real 512x512/22f scale; the CUDA device forward equals the CPU host at the real render seq (1920, new permanent gate); DequantNvfp4ToBf16 is byte-exact; the forward math equals upstream source. - STATUS/FEATURES/BENCHMARKS: H3 row -> render bug CLOSED (shrink-only respected). - benchmark-record + spec §8.6 + state: full investigation, root cause, the ref2va output-row fix, and the open follow-ups (partition guard + encoder vision tower). - NOW: H3 lane updated, under budget. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…se (#77 follow-up) The #70/#74 white grid cost three campaigns because the H3 driver silently accepted task=t2va on the Ref2VA-partition checkpoint. Upstream `pipeline._resolve_task` RAISES on the mismatch (vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py:374-391, raise at 387-390); the recipe documents the split (recipes/MiniMaxAI/MiniMax-H3.md:50-51,289: one server serves one partition, FL2VA→{t2va,fl2va}, Ref2VA→{ref2va}). This mirrors the raise 1:1 and adds the community-file fallback. Partition detection. `MiniMaxH3PartitionFromModelIndex` mirrors upstream's exact release keys (model_index.json → `_minimax_h3` → {partition,tasks}, pipeline:279-282). Community GGUF/NVFP4 strip that block, and there is NO structural fallback: measured on the two real captured manifests, the Ref2VA NVFP4 (1051 tensors) and FL2VA GGUF (535) carry the IDENTICAL DiT — same 535 base tensor names AND shapes after collapsing the NVFP4 weight/scale/scale_2 split (ref2va prepends reference rows through the SAME video/audio_patch_proj, adding no tensor). So a stripped file must DECLARE the partition (`--partition fl2va|ref2va`; server `--video-partition`); `MiniMaxH3PartitionFromFlag` maps it to the recipe's served-task set. The refuse. `MiniMaxH3CheckTaskPartition` is the raise half of `_resolve_task`; the task is what the request encodes (`MiniMaxH3TaskOfRequest`), and `MiniMaxH3GenerateT2va` calls the pair before denoising. An unknown partition refuses every task as ambiguous and names the recipe lines. A default `declared=false` request leaves the guard inert (pipeline-math tests unaffected). RED-first: new case `test_minimax_h3 :: "the task/partition guard refuses the #77 mismatch"` (38 assertions) — the #77 combo throws, correct pairings pass, stripped refuses + --partition recovers, and the two real manifests are asserted to the same 535-name set. Neutralizing the guard body turned it RED at 10 assertions; restored → GREEN. Suite 67/67 (66 prior +1), 46549 assertions; test_video_api 4/4. Records: spec §8.7 (guard section + behavior table + discriminator finding), STATUS/BENCHMARKS H3 rows, benchmark-record, NOW, anchored state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Row: H3-RENDER-COHERENCE (helper) — render bug ROOT-CAUSED
DRAFT — this PR is the claim. All changes are env-gated diagnostics + driver modes (byte-identical to production when off) plus records; no production path changed.
Problem
The #64 real-checkpoint NVFP4 t2va render RUNS end-to-end but the frame is a rigid grid of 16-px multicolour blocks, identically at 12/20/50 steps, conditioned or not.
Root cause (latent bisection, all MEASURED on the real ~39 GB NVFP4 arm, GB10)
The rectified-flow Euler integral telescopes to
(σ0−σN)·v ≈ v, so step-count invariance would mean a frozen velocity — it is not.--roundtrip(encode→post_quant_conv→decode) returns a coherent frame, no grid. Overturns the #64 "device VAE decode / denoise convergence" framing.Verdict: the bug is in the DiT forward's spatial mixing at real geometry (video tokens not mixed). The forward gate runs spatial 2×3 and matches upstream at 1.6e-7; the divergence appears only at the real 8×8 — the same "below one chunk/tile" blind spot as the temporal-chunking and VAE-tiling misses. Secondary real bug: the driver seeded uniform init noise, not Gaussian N(0,1).
What's here
VT_H3_TRACE_MOTION,VT_H3_DUMP_DIR,VT_H3_VAE_PROBE,VT_H3_GAUSSIAN_NOISE(all documented,check-env-docgreen)--decode-latent(decode a dumped latent, no DiT) and--roundtrip(VAE encode→decode gold-standard test)Status
Pre-existing
check-fusion-consistencyred (minimax_h3_video_vae_device) is not from this row.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys