row/H3-DIT-SCALE-GATE: DiT spatial-mixing geometry ladder (#70 close) - #74
Closed
localai-bot wants to merge 2 commits into
Closed
row/H3-DIT-SCALE-GATE: DiT spatial-mixing geometry ladder (#70 close)#74localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
…ometry ladder (#70 follow-up) Helper claim (DRAFT PR is the claim) for the #70 follow-up: extend the reduced-dim DiT-forward parity gate into a geometry ladder (2x3 -> 8x8 + rectangle + temporal + packed mix) to pin the spatial-mixing divergence at real token geometry. CPU-only, no dgx. Harness verified: the gen script reproduces the checked-in H3 goldens byte-identically against local vllm-omni. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…-mixing-math hypothesis (2x3->8x8+temporal, host+device) The section-5 DiT parity gate only ever ran spatial 2x3. #70 root-caused the render grid to the DiT emitting a spatially-white latent at real token geometry (8x8), so the divergence lives strictly between 2x3 and 8x8. This extends the reduced-dim gate into a GEOMETRY LADDER to pin the divergence at real geometry -- CPU-only, no dgx. Built (scripts/gen-minimax-h3-goldens.py :: emit_dit_ladder): 7 rungs 2x3->4x4->6x6->8x8 + a 4x8 rectangle + an 8x8x3-frame temporal 3D grid + a 6x10x5-frame video+audio packed mix, each emitting the upstream packed-sequence layout, the RefDiT forward logits, and a spatial-mixing probe. Two permanent gate cases in test_minimax_h3.cpp: - "DiT-forward geometry ladder matches upstream (host+device, mixing)" - "DiT device-vs-host forward holds at the REAL head_dim=128 ratio" Goldens regenerate byte-identically against local ~/_git/vllm-omni; suite 65/65 (was 63/63). Result: ours == the RefDiT oracle at EVERY rung, HOST and DEVICE forward, max|diff| <= 3e-7 vs the 2e-5 gate; the mixing probe shows every video-target token couples through the packed bidirectional attention (fraction 1.0). The real-head_dim=128 leg has device==host <= 1.2e-6 across all rungs. The #70 adjacent-cell COSINE does NOT translate here: with random weights the CORRECT oracle is already spatially white by that metric (coherence is a TRAINED-weights property), so the harness's valid discriminators are oracle-logit equality and information flow, both green. VERDICT: the "spatial-mixing bug in the DiT-forward MATH" hypothesis is REFUTED at reduced dims and real token geometry. The #70 white latent is trained-weights / real-scale, not a reduced-dim DiT-forward bug; a GPU re-render is NOT expected to be coherent from this work. Residuals (beyond CPU): a bug shared by our port AND the RefDiT restatement vs true upstream (close on the dgx oracle venv), and the real-scale DiT INPUT wiring (encoder embeds / real position grid / timesteps). Pre-existing check-fusion-consistency red (minimax_h3_video_vae_device) is not this row's. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The two remaining suspects (true-upstream arbitration on the dgx venv + real-scale DiT input wiring) run as the next box campaign once the flash-occupancy one clears. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ken grids; mixing hypothesis REFUTED (#74) row/H3-DIT-SCALE-GATE squash, CPU-only. The #70 redirect ("the spatial- mixing divergence should reproduce at real TOKEN geometry with random weights") is REFUTED the right way: a 7-rung geometry ladder (2x3 -> 8x8, rectangle, 3-frame temporal, 5-frame video+audio packed mix) vs the upstream oracle at reduced hidden dims passes EVERY rung at <=3e-7 host / <=1.2e-6 device (incl. the real head_dim=128 rope ratio), and a perturbation probe shows mixing fraction 1.0 everywhere — the packed bidirectional attention couples all video tokens at real geometry. Load-bearing metric finding: the #70 adjacent-cell-cosine whiteness is MEANINGLESS under random weights — the correct reference is already white by that metric; spatial coherence is a trained-weights property. The ladder ships as two permanent gate cases (suite 65/65) so the below-one-tile blind spot stays closed. The render-bug hunt narrows to two surfaces, both needing the box: (1) a shared blind spot of our port AND the RefDiT restatement vs the TRUE upstream module (arbitrate on the dgx oracle venv); (2) the real-scale DiT INPUT wiring (encoder embeds, fp64 position grid at full canvas, per-token timesteps) — the one surface #70 never isolated. Queued behind the flash-occupancy box campaign. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…(1920) S1 is exonerated: the real-scale DiT INPUTS (packed layout, fp64 position grid, token tags, inverse/combined AdaLN indices, sigmas) diff EXACTLY against upstream pipeline_minimax_h3.py at 512x512/22f (verified on dgx: text_len=8, latent 7x32x32, seq_len 1920); the tokenization matches upstream byte-for-byte; the encoder conditioning is correctly shaped and carries the expected Qwen massive-activation structure; and DequantNvfp4ToBf16 is the shared helper the Laguna and Qwen3-32B NVFP4 arms already prove byte-exact. That leaves ONE untested surface: #74's device-vs-host ladder (incl. the REAL head_dim=128 case) runs on the CPU BACKEND, and the CUDA cases run only at the small fl2va geometry. The CUDA kernels at the REAL render seq (t2va -> latent 7x32x32 -> seq_len 1920, cu_seqlens=[0,1874,1920] non-causal 2-document) at head_dim=128 have never been gated against the trusted host loops. A scale-dependent CUDA-kernel bug (varlen non-causal attention / RoPE cache / AdaLN modulate) would leave every reduced-dim gate green while the render goes white. This case runs the SAME MiniMaxH3DitForwardDevice on the CUDA backend vs the CPU host forward at exactly the render geometry and the step-0 timestep partition. A divergence IS the #70 bug; a match points the hunt at S2 (a shared port<->RefDiT restatement blind spot vs true upstream). Skips without a CUDA backend. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
Records the H3-RENDER-CLOSE result across the canonical surfaces. The #70/#74 white render was t2va run on the REF2VA-partition checkpoint; upstream serves t2va/fl2va from the FL2VA partition and requires the task to match the partition. t2va on the FL2VA GGUF DiT renders a COHERENT, prompt-matched scene on GB10 (VAE-input latent adj-cell cosine 0.95 vs 0.06 white, no 16px patch grid, valid h264/AAC mp4). Before switching partitions, verified: the t2va DiT inputs diff byte-exact vs upstream at real 512x512/22f scale; the CUDA device forward equals the CPU host at the real render seq (1920, new permanent gate); DequantNvfp4ToBf16 is byte-exact; the forward math equals upstream source. - STATUS/FEATURES/BENCHMARKS: H3 row -> render bug CLOSED (shrink-only respected). - benchmark-record + spec §8.6 + state: full investigation, root cause, the ref2va output-row fix, and the open follow-ups (partition guard + encoder vision tower). - NOW: H3 lane updated, under budget. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ent video on GB10 (#77) row/H3-RENDER-CLOSE squash. The white-latent grid was NOT a code bug: MiniMax-H3 ships TWO independently-served DiT partitions and every prior render ran task=t2va on the REF2VA-partition checkpoint — out-of-distribution input, hence white latents invariant to prompt and steps. Upstream's pipeline._resolve_task REFUSES the mismatch (pipeline_minimax_h3.py:387-390, recipe:50,289); ours silently accepted it. On the FL2VA-partition GGUF (Q3_K_M, 15.58 GB) t2va renders a COHERENT prompt-matched scene: an orange cat on a wooden table at 512x512, and a warmly-lit kitchen scene at 768x1344/50-step — the engine's first real video+audio generation on one GB10. VAE-input adj-cell cosine 0.95 (white was 0.06), no patch grid. Exoneration sweep shipped as permanent hardening: every step-0 DiT input byte-exact vs upstream at real scale (packed layout, fp64 grid, token tags, AdaLN indices, sigmas, tokenization); NVFP4 dequant byte-exact; NEW gate CUDA-device-vs-host at real seq 1920 (28/28 — the surface #74 never covered); plus a REAL never-run ref2va output bug fixed (unpatchify was handed the reference+target row buffer; pinned reference rows now stripped). Follow-ups: mirror upstream's task/partition guard in the driver (community quant files strip the release config); encoder vision tower (W3 remnant) for image/video-conditioned ref2va/fl2va. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…se (#77 follow-up) The #70/#74 white grid cost three campaigns because the H3 driver silently accepted task=t2va on the Ref2VA-partition checkpoint. Upstream `pipeline._resolve_task` RAISES on the mismatch (vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py:374-391, raise at 387-390); the recipe documents the split (recipes/MiniMaxAI/MiniMax-H3.md:50-51,289: one server serves one partition, FL2VA→{t2va,fl2va}, Ref2VA→{ref2va}). This mirrors the raise 1:1 and adds the community-file fallback. Partition detection. `MiniMaxH3PartitionFromModelIndex` mirrors upstream's exact release keys (model_index.json → `_minimax_h3` → {partition,tasks}, pipeline:279-282). Community GGUF/NVFP4 strip that block, and there is NO structural fallback: measured on the two real captured manifests, the Ref2VA NVFP4 (1051 tensors) and FL2VA GGUF (535) carry the IDENTICAL DiT — same 535 base tensor names AND shapes after collapsing the NVFP4 weight/scale/scale_2 split (ref2va prepends reference rows through the SAME video/audio_patch_proj, adding no tensor). So a stripped file must DECLARE the partition (`--partition fl2va|ref2va`; server `--video-partition`); `MiniMaxH3PartitionFromFlag` maps it to the recipe's served-task set. The refuse. `MiniMaxH3CheckTaskPartition` is the raise half of `_resolve_task`; the task is what the request encodes (`MiniMaxH3TaskOfRequest`), and `MiniMaxH3GenerateT2va` calls the pair before denoising. An unknown partition refuses every task as ambiguous and names the recipe lines. A default `declared=false` request leaves the guard inert (pipeline-math tests unaffected). RED-first: new case `test_minimax_h3 :: "the task/partition guard refuses the #77 mismatch"` (38 assertions) — the #77 combo throws, correct pairings pass, stripped refuses + --partition recovers, and the two real manifests are asserted to the same 535-name set. Neutralizing the guard body turned it RED at 10 assertions; restored → GREEN. Suite 67/67 (66 prior +1), 46549 assertions; test_video_api 4/4. Records: spec §8.7 (guard section + behavior table + discriminator finding), STATUS/BENCHMARKS H3 rows, benchmark-record, NOW, anchored state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…Stack→device text tower) + ref2va re-attribution Closes the #86 framework residual and re-attributes the ref2va grid with a GB10 render A/B (row/H3-VISION-SCATTER). deepstack_visual_indexes CONFIRMED (was #86-inferred): {8,16,24}, grounded in the release config — MiniMax-H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose vision_config.deepstack_visual_indexes = [8,16,24], depth 27, text num_hidden_layers 64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig default and public Qwen/Qwen3-VL-30B-A3B. No value change; comment updated. Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice now takes the optional visual_pos_mask + per-tap deepstack blocks and ADDS each block into the masked visual rows after each of the first len(deepstack) decoder layers — device mirror of the gated host reference and upstream _deepstack_process (encoder.py:770-800). The merged masked_scatter into inputs_embeds stays the caller's job (upstream _encode, encoder.py:1071). Text-only prompts pass the defaults and are byte-identical. Gate: the device keep-quant encoder test now also runs WITH a visual mask + two DeepStack blocks → device==host max|diff| 3.8e-4 (<=2e-3) AND DeepStack moves the conditioning (scale 1.006->1.062). Driver --cond-image routes a reference image through the encoder vision path (reuse only: Qwen3VLImageProcessor -> Qwen3VLVisionForward -> merged + 3 DeepStack; ExpandImagePlaceholders inserts nm image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex == H3 _get_rope_index for t==1, position math verified). Additive. GB10 render A/B (256x256/22f/12steps): * Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS): frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). No grid. The vision conditioning is SOUND. * Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL): same multicolour patch grid as #86's text-only ref2va. RE-ATTRIBUTION (evidence): the "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. NOT the encoder conditioning — DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. The only difference between coherent fl2va and gridding ref2va: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running reference rows. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward. Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW, state. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Closed
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
row/H3-DIT-SCALE-GATE — pin the DiT spatial-mixing divergence at real TOKEN geometry (#70 follow-up)
CPU-only, no dgx. #70 root-caused the H3 render grid to the DiT emitting a
spatially-WHITE latent at real token geometry (adjacent-cell cosine 0.06 vs 0.789 for a
real encode); the DiT parity gate only ever ran spatial 2x3. This PR extends the
reduced-dim DiT-forward parity gate into a GEOMETRY LADDER to pin the divergence.
What landed
scripts/gen-minimax-h3-goldens.py :: emit_dit_ladder— 7 rungs (2x3, 4x4, 6x6, 8x8,a 4x8 rectangle, an 8x8x3-frame temporal 3D grid, a 6x10x5-frame video+audio packed mix),
each emitting the upstream packed-sequence layout, the RefDiT forward logits, and a
spatial-mixing probe. Goldens regenerate byte-identically against local vllm-omni.
test_minimax_h3.cpp: the geometry ladder (host+device+mixing)and a real-head_dim=128 device-vs-host leg. Suite 65/65 (was 63/63).
Verdict — the #70 spatial-mixing-MATH hypothesis is REFUTED
the 2e-5 gate. The mixing probe: perturbing one video-target token changes EVERY other
target (fraction 1.0) — the packed bidirectional attention couples all video tokens at
real geometry.
already spatially white by that metric (coherence is a TRAINED-weights property). Valid
discriminators are oracle-logit equality + information flow, both green.
Therefore the #70 white latent is NOT a reduced-dim-reproducible DiT-forward bug — it is
trained-weights / real-scale. A GPU re-render is NOT expected to be coherent from this work.
Residuals (both beyond the CPU box; for the follow-up campaign)
minimax_h3_transformer.py(not importable here) is invisible to this ladder — close iton the dgx oracle venv (vllm installed): real
MiniMaxH3DiTModelat TP=1 vs RefDiT.timesteps) is fed with RANDOM data here; a real-weights activation diff of the DiT inputs
is the untested surface row/H3-RENDER-COHERENCE: root-cause the H3 real-geometry render bug (non-scene patch-grid) #70 did not isolate.
Pre-existing
check-fusion-consistencyred (minimax_h3_video_vae_device) is not this row's.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys