row/H3-RENDER-CLOSE: MiniMax-H3 render bug CLOSED (wrong partition; FL2VA t2va renders a coherent scene) - #77
Closed
localai-bot wants to merge 4 commits into
Closed
row/H3-RENDER-CLOSE: MiniMax-H3 render bug CLOSED (wrong partition; FL2VA t2va renders a coherent scene)#77localai-bot wants to merge 4 commits into
localai-bot wants to merge 4 commits into
Conversation
…hunt The #70 white-latent render bug is cornered to WHAT the driver feeds the DiT at real scale (#74 proved the forward MATH exact at every reduced-dim geometry, but the ladder fed RANDOM inputs). Add an env-gated `VT_H3_DUMP_INPUTS=<dir>` that, at denoise step 0, writes every DiT input as raw little-endian binary + a text manifest: the packed layout (input_ids/masks/img_pos/audio_pos/text_pos/ update_mask/cu_seqlens/document_id), the fp64 position grid, the per-token modality tags, the per-token pre-unique timesteps and their unique/inverse/ combined AdaLN selection, both sigma schedules, and the raw prompt_embeds. The `minimax-h3-gen` driver additionally dumps prompt_token_ids so the tokenization can be diffed against upstream `minimax_h3_text_only_ids` (verbatim prompt, add_special_tokens=False). Byte-identical to production when unset (every path is guarded; no file opened). Documented in docs/ENVIRONMENT.md. This is the S1 instrumentation only; the diff against upstream pipeline_minimax_h3.py runs next. Pre-existing preflight red (check-fusion-consistency minimax_h3_video_vae_device) is unrelated to this change and untouched. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…(1920) S1 is exonerated: the real-scale DiT INPUTS (packed layout, fp64 position grid, token tags, inverse/combined AdaLN indices, sigmas) diff EXACTLY against upstream pipeline_minimax_h3.py at 512x512/22f (verified on dgx: text_len=8, latent 7x32x32, seq_len 1920); the tokenization matches upstream byte-for-byte; the encoder conditioning is correctly shaped and carries the expected Qwen massive-activation structure; and DequantNvfp4ToBf16 is the shared helper the Laguna and Qwen3-32B NVFP4 arms already prove byte-exact. That leaves ONE untested surface: #74's device-vs-host ladder (incl. the REAL head_dim=128 case) runs on the CPU BACKEND, and the CUDA cases run only at the small fl2va geometry. The CUDA kernels at the REAL render seq (t2va -> latent 7x32x32 -> seq_len 1920, cu_seqlens=[0,1874,1920] non-causal 2-document) at head_dim=128 have never been gated against the trusted host loops. A scale-dependent CUDA-kernel bug (varlen non-causal attention / RoPE cache / AdaLN modulate) would leave every reduced-dim gate green while the render goes white. This case runs the SAME MiniMaxH3DitForwardDevice on the CUDA backend vs the CPU host forward at exactly the render geometry and the step-0 timestep partition. A divergence IS the #70 bug; a match points the hunt at S2 (a shared port<->RefDiT restatement blind spot vs true upstream). Skips without a CUDA backend. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
ROOT CAUSE of the #70 white render FOUND: the render ran task=t2va on the `minimax_h3_ref2va_nvfp4_full` checkpoint, which is the REF2VA partition. Upstream serves t2va/fl2va from the FL2VA partition and ref2va from the Ref2VA partition, and "task must match the served partition" (recipes/MiniMaxAI/MiniMax-H3.md:50,289; _resolve_task raises otherwise). A ref2va-trained DiT fed a t2va sequence (no reference block) is out of distribution -> the spatially-degenerate latent, invariant to the text prompt and step count -- exactly #70's symptom. Every render so far used the wrong task for this checkpoint; the DiT forward, its inputs, the NVFP4 dequant and the CUDA kernels are all correct (verified: S1 inputs byte-exact vs upstream at real 512x512/22f scale; CUDA device forward == CPU host at seq 1920; forward math == upstream source == RefDiT). Running the CORRECT task (ref2va) surfaced a real, previously-unexercised bug in the OUTPUT path: BuildMiniMaxH3PackedSequenceRef2va PREPENDS pinned reference rows (encoded image/video/audio) to the packed layout, the DiT zeroes them in its output (skip_mask_out_condition), and MiniMaxH3GenerateT2va then handed the full (reference + target) row buffer to unpatchify/unpack -- which rejected the non-divisible count ("rows not divisible by t*h*w"). The generated clip is only the TRAILING target rows. Slice to them before unpatchify/unpack. t2va/fl2va have no reference prefix, so the tail is the whole buffer and the change is a no-op there. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Records the H3-RENDER-CLOSE result across the canonical surfaces. The #70/#74 white render was t2va run on the REF2VA-partition checkpoint; upstream serves t2va/fl2va from the FL2VA partition and requires the task to match the partition. t2va on the FL2VA GGUF DiT renders a COHERENT, prompt-matched scene on GB10 (VAE-input latent adj-cell cosine 0.95 vs 0.06 white, no 16px patch grid, valid h264/AAC mp4). Before switching partitions, verified: the t2va DiT inputs diff byte-exact vs upstream at real 512x512/22f scale; the CUDA device forward equals the CPU host at the real render seq (1920, new permanent gate); DequantNvfp4ToBf16 is byte-exact; the forward math equals upstream source. - STATUS/FEATURES/BENCHMARKS: H3 row -> render bug CLOSED (shrink-only respected). - benchmark-record + spec §8.6 + state: full investigation, root cause, the ref2va output-row fix, and the open follow-ups (partition guard + encoder vision tower). - NOW: H3 lane updated, under budget. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
changed the title
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ent video on GB10 (#77) row/H3-RENDER-CLOSE squash. The white-latent grid was NOT a code bug: MiniMax-H3 ships TWO independently-served DiT partitions and every prior render ran task=t2va on the REF2VA-partition checkpoint — out-of-distribution input, hence white latents invariant to prompt and steps. Upstream's pipeline._resolve_task REFUSES the mismatch (pipeline_minimax_h3.py:387-390, recipe:50,289); ours silently accepted it. On the FL2VA-partition GGUF (Q3_K_M, 15.58 GB) t2va renders a COHERENT prompt-matched scene: an orange cat on a wooden table at 512x512, and a warmly-lit kitchen scene at 768x1344/50-step — the engine's first real video+audio generation on one GB10. VAE-input adj-cell cosine 0.95 (white was 0.06), no patch grid. Exoneration sweep shipped as permanent hardening: every step-0 DiT input byte-exact vs upstream at real scale (packed layout, fp64 grid, token tags, AdaLN indices, sigmas, tokenization); NVFP4 dequant byte-exact; NEW gate CUDA-device-vs-host at real seq 1920 (28/28 — the surface #74 never covered); plus a REAL never-run ref2va output bug fixed (unpatchify was handed the reference+target row buffer; pinned reference rows now stripped). Follow-ups: mirror upstream's task/partition guard in the driver (community quant files strip the release config); encoder vision tower (W3 remnant) for image/video-conditioned ref2va/fl2va. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). First coherent H3 render on the GB10 — milestone recorded. Follow-ups tracked: the task/partition guard + the encoder vision tower. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
The #77 landing accidentally ran in the primary checkout instead of the session worktree (the worktree had been cleaned up mid-turn) and its git add -A swept three personal untracked items (localai.png, rocm-issue-draft.md, the omcheck embedded-repo gitlink) plus an uncommitted local tools/bench/online_gate.py edit into the squash. This removes all four from the tree: the files stay untouched in the owner's working copy, online_gate.py reverts to its pre-squash content, and no #77 feature/record content changes. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…, BENCHMARKS note folded The prior revert commit was pushed with a red gate (my error: the push ran ungated while a mid-script assert left STATUS 103 chars over its shrink-only ratchet and BENCHMARKS untouched). This folds the slip note into existing prose within both budgets. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…o number yet) `startup` has been listed in the online-serving gate protocol as an axis every interleaved repetition must record, but it was never captured: the driver waited for readiness and discarded the duration, and no manifest carried the field. So "is our startup faster than vLLM's?" had no answer, only an incidental ours-only 43 s observation. This lands the measurement; it claims no result. Metric: elapsed wall time from immediately before the server spawn to the first successful GET /health, identical probe on both arms, both through the same start_server so the launched commands stay the timed grid's verbatim. - scripts/dgx-online-serving.sh: new --startup-only mode (3 interleaved ours/vLLM legs under one /tmp/gpu lock, page cache dropped per leg, GPU idle proven before and after, no timed client, `server` target only); wait_ready moves from a 5 s cadence / seq 1 360 to a 0.2 s cadence on a deadline that preserves the identical 1800 s budget; start_server stamps launch and ready immediately around the spawn. - tools/bench/online_gate.py: record-startup + summarize-startup. A record embeds and validates the leg's cache-drop report, so a startup artifact cannot exist for a leg whose page cache was still warm; the summary refuses an incomplete series rather than biasing the median toward whichever arm finished. The 5 s cadence was itself ~12% of a ~40 s startup: it was the resolution floor and the reason this axis was never reportable. Test-first: the suite failed on `cannot import name 'record_startup'` before the implementation existed. test_online_gate_startup 19/19; run with the four neighbouring tool suites 66/66; shellcheck + bash -n clean. The existing --execute purity contract caught a real regression (a second `for repetition in 1 2 3` header broke its split anchor); loop variable renamed. Also repairs main: squash b95543c (#77) swept the half-written test file onto main without its implementation, leaving tests/tools red. Residual, honest: NO ours-vs-vLLM startup number exists. dgx.casa was at 100% disk with no CUDA `server` build; the 27B 3-repetition run is owed. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…se (#77 follow-up) The #70/#74 white grid cost three campaigns because the H3 driver silently accepted task=t2va on the Ref2VA-partition checkpoint. Upstream `pipeline._resolve_task` RAISES on the mismatch (vllm_omni/diffusion/models/minimax_h3/pipeline_minimax_h3.py:374-391, raise at 387-390); the recipe documents the split (recipes/MiniMaxAI/MiniMax-H3.md:50-51,289: one server serves one partition, FL2VA→{t2va,fl2va}, Ref2VA→{ref2va}). This mirrors the raise 1:1 and adds the community-file fallback. Partition detection. `MiniMaxH3PartitionFromModelIndex` mirrors upstream's exact release keys (model_index.json → `_minimax_h3` → {partition,tasks}, pipeline:279-282). Community GGUF/NVFP4 strip that block, and there is NO structural fallback: measured on the two real captured manifests, the Ref2VA NVFP4 (1051 tensors) and FL2VA GGUF (535) carry the IDENTICAL DiT — same 535 base tensor names AND shapes after collapsing the NVFP4 weight/scale/scale_2 split (ref2va prepends reference rows through the SAME video/audio_patch_proj, adding no tensor). So a stripped file must DECLARE the partition (`--partition fl2va|ref2va`; server `--video-partition`); `MiniMaxH3PartitionFromFlag` maps it to the recipe's served-task set. The refuse. `MiniMaxH3CheckTaskPartition` is the raise half of `_resolve_task`; the task is what the request encodes (`MiniMaxH3TaskOfRequest`), and `MiniMaxH3GenerateT2va` calls the pair before denoising. An unknown partition refuses every task as ambiguous and names the recipe lines. A default `declared=false` request leaves the guard inert (pipeline-math tests unaffected). RED-first: new case `test_minimax_h3 :: "the task/partition guard refuses the #77 mismatch"` (38 assertions) — the #77 combo throws, correct pairings pass, stripped refuses + --partition recovers, and the two real manifests are asserted to the same 535-name set. Neutralizing the guard body turned it RED at 10 assertions; restored → GREEN. Suite 67/67 (66 prior +1), 46549 assertions; test_video_api 4/4. Records: spec §8.7 (guard section + behavior table + discriminator finding), STATUS/BENCHMARKS H3 rows, benchmark-record, NOW, anchored state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… loudly (#84) row/H3-TASK-PARTITION-GUARD squash. Mirrors upstream's _resolve_task raise (pipeline_minimax_h3.py:279-282,387-390): FL2VA serves {t2va,fl2va}, Ref2VA serves {ref2va}; any other pairing — including the exact t2va-on-ref2va mismatch that silently burned three render campaigns — refuses with the recipe contract in the message. DEFINITIVE finding: NO structural discriminator exists in principle — the two real captured manifests collapse to IDENTICAL 535 base tensor names AND shapes, so a stripped community file must DECLARE its partition (--partition / --video-partition); MiniMaxH3PartitionFromModelIndex reads the release model_index.json keys when present. RED-first proven (guard body neutralized -> 10 failed assertions); suite 67/67. Merge also repairs an anchorless state entry another session pushed (re-anchored 2026-08-06T23:55) and pays a 5-char STATUS ratchet squeeze. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…ision tower loads + runs (#86) row/H3-CONDITIONED-E2E squash. Record reconciliation: the vision-tower MATH existed and was gated at reduced dims with synthetic weights (minimax_h3_encoder.cpp:572 + surround), but was never wired to real weights — the GGUF encoder loader skipped every visual.* tensor and the device text forward has no deepstack arg; #26 understated the gap, #77 was right in what matters. Wired now: LoadQwen3VLVisionFromGguf dequantizes the on-box encoder's full visual.* tower (27 blocks + patch embed + pos embed + merger + 3 deepstack mergers, shared Qwen3.6-27B vision geometry) into the existing Qwen3VLVisionWeights reuse; CPU loader gate + a driver --prompt-image probe (real 512x512 image -> [256,20480] finite, the Qwen massive-activation signature). Frame sanity on GB10: fl2va with a real first-frame renders a coherent photorealistic scene MATCHING the conditioning frame (pose/table/ background) — the second conditioned modality works. ref2va still grids, now for a PRECISE reason: the loaded tower is not yet scattered into the DiT render-conditioning (the render never used it); fl2va coherent in the same session proves DiT/VAE/partition sound. Residual = vision->prompt_embeds scatter + DeepStack device injection + confirm deepstack_visual_indexes {8,16,24} against upstream. Known flake: full test_minimax_h3 SIGSEGVs at a pre-existing unrelated CUDA case that passes standalone (cross-test resource accumulation). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…Stack→device text tower) + ref2va re-attribution Closes the #86 framework residual and re-attributes the ref2va grid with a GB10 render A/B (row/H3-VISION-SCATTER). deepstack_visual_indexes CONFIRMED (was #86-inferred): {8,16,24}, grounded in the release config — MiniMax-H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose vision_config.deepstack_visual_indexes = [8,16,24], depth 27, text num_hidden_layers 64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig default and public Qwen/Qwen3-VL-30B-A3B. No value change; comment updated. Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice now takes the optional visual_pos_mask + per-tap deepstack blocks and ADDS each block into the masked visual rows after each of the first len(deepstack) decoder layers — device mirror of the gated host reference and upstream _deepstack_process (encoder.py:770-800). The merged masked_scatter into inputs_embeds stays the caller's job (upstream _encode, encoder.py:1071). Text-only prompts pass the defaults and are byte-identical. Gate: the device keep-quant encoder test now also runs WITH a visual mask + two DeepStack blocks → device==host max|diff| 3.8e-4 (<=2e-3) AND DeepStack moves the conditioning (scale 1.006->1.062). Driver --cond-image routes a reference image through the encoder vision path (reuse only: Qwen3VLImageProcessor -> Qwen3VLVisionForward -> merged + 3 DeepStack; ExpandImagePlaceholders inserts nm image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex == H3 _get_rope_index for t==1, position math verified). Additive. GB10 render A/B (256x256/22f/12steps): * Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS): frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). No grid. The vision conditioning is SOUND. * Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL): same multicolour patch grid as #86's text-only ref2va. RE-ATTRIBUTION (evidence): the "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. NOT the encoder conditioning — DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. The only difference between coherent fl2va and gridding ref2va: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running reference rows. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward. Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW, state. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…d re-attributed to the NVFP4 checkpoint/loader (#93) row/H3-REF2VA-ASSEMBLY squash. REAL BUG fixed: EncodeReferenceImages/ Video wrote the RefBlock with PATCHED dims while BuildMiniMaxH3PackedSequenceRef2va divides by the patch again (mirroring upstream packed_sequence.py:328-330, which takes the raw latent visual_shape per pipeline_minimax_h3.py:1141-1145) — the double division under-allocated the reference span by patch_h*patch_w = 4x and the pin-loop silently truncated the encoded reference to its first quarter. Fix: emit raw ls.{t,h,w} (~6 lines). RED-first gate (reintroducing the bug fails 128==512) + the FIRST ref2va-shaped DiT rung (reference rows + ref2va timestep partition + audio update mask; layout byte-exact, logits <=2e-5, mixing 1.0) — the blind-spot class closed permanently. Suite 69/69, 52377 asserts. Timesteps on reference rows, packed layout, tags, #77 bookkeeping all CLEARED vs upstream. Checkpoint-controlled A/B re-attributes the grid: it appears with the ENTIRE reference assembly removed, in fp4 AND bf16, and correlates 1:1 with the ref2va NVFP4 checkpoint — every render loading that file grids; every FL2VA-GGUF render is coherent (control re-verified). §8.6/§8.9 varied task/prompt but never checkpoint — both misattributed. True residual: the NVFP4 DiT load path for THIS file (fp32-island preservation, weight_scale_2, tensor mapping) — the synthetic gates proved the dequant math, never this file against a coherent oracle. Next: independent CPU dequant of the same file as the loader oracle (no new checkpoint needed). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…ssembly driver, byte-gated ARCH-ONE-SURFACE ROW 2, step 1 of the fold order (GROW the library entry point). vllm::multimodal::MiniMaxH3VideoEngine composes what examples/minimax_h3_gen/main.cpp:687-1288 and the /v1/videos twin in examples/server/main.cpp:743-1096 owned privately: the four DiT loader arms (GGUF keep-quant / dequant-bf16 host+streamed, bf16 shard-dir stream, NVFP4 bf16/fp4-resident stream), both VAE decoder halves + lazily-loaded encoder halves, the optional H3-Encoder text tower staged once, the #77 partition/task guard, fl2va keyframe + ref2va reference conditioning, the splitmix64 Box-Muller noise streams (byte-exact; VT_H3_GAUSSIAN_NOISE honored, seeded requests derive the audio stream via splitmix64), artifact writing, and the mux-argv assembly. The library SPAWNS NOTHING: mkdir is std::filesystem (the examples shelled out to mkdir -p), and ffmpeg stays with the caller as mux_argv — the ratified 2026-08-03 boundary. MiniMaxH3VideoGenParamsFromRequest is the library-owned /v1/videos mapping so HTTP and FFI cannot drift. FOLD GATE (test_minimax_h3_video_fold, 4 cases / 131 assertions, GREEN): arm A (seam) == arm B (replicated pre-fold pipeline restated line for line) == arm C (the committed goldens the PRE-fold binary rendered at f63ba07) — frames + WAV byte-identical, mux argv pinned against the pre-fold minimax-h3-mux --print-only capture, second-generation reuse byte-identical, partition-guard refusals + input refusals pinned, VideoRequest mapping field-complete. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…-> 12 ARCH-ONE-SURFACE ROW 2, the C surface over the W1 seam — the ratified shapes (.agents/specs/one-surface-abi.md:92-108) as shipped: vllm_video_engine (an opaque handle SEPARATE from vllm_engine, because H3 loads a checkpoint SET, not a config.json directory), vllm_video_model_params/_default, vllm_video_params/_default, vllm_video_result, vllm_video_engine_load/free, vllm_video_generate, vllm_video_result_free — plus the engine-free vllm_video_mux_argv(+_params_default,+_free) composer the minimax-h3-mux thin client needs (a disclosed append beyond the ratified proposal; the encoding contract stays the library's, the caller execs). Deviations from the proposal, each argued in the spec: + prompt_embeds_path (without an encoder there is NO conditioning path), + partition (the #77 guard refuses every render without it), + fp4_resident and + output_dir; - task/duration (derivable / expressible via num_frames); ref_image stays the ratified SINGLE image (multi-image ref2va remains C++-seam-reachable, named residual). mux_argv is NULL-terminated (execvp-ready). Zero values preserve behaviour (noise_aug <= 0 resolves to the 1.0 pin engine-side). Refuse-both-directions: the seam now refuses a config.json-style directory LOUDLY naming vllm_engine_load; vllm_engine_load on the H3 fixture dir keeps failing byte-for-byte as captured at v11 (status 2, missing config.json). Gates: vllm_capi_c_check (strict-C11 compile) green; test_capi 40/40 (373 assertions) including the v12 section — the FULL golden e2e through the C marshalling (frames+WAV byte-identical to the pre-fold binary's committed goldens, mux argv pinned), the zero-value contract, both refusal directions, and the argument contract; ABI floor pin advanced >= 11 -> >= 12 (the #121 lesson — the == checks move with the macro). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…— vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples ONE-SURFACE fold ROW 2 (row/H3-VIDEO-ABI, task #283; the #121 precedent's shape: three-arm fold gate, guard-ratchet shrink). The library absorbs everything the examples privately owned. GOLDENS FIRST: minimax_h3_video_fold_fixture.h writes a deterministic tiny checkpoint set (857KB F32 ComfyUI-GGUF DiT at the test_minimax_h3.cpp:3786 reduced geometry, on-disk reduced ViT3D/BigVGAN VAEs + config.json, prompt embeds); the PRE-fold minimax-h3-gen binary AT THE BRANCH BASE (fc636c7) rendered it (--partition fl2va --steps 3 --frames 5 --height/width 32, CPU, keep-quant; run-to-run deterministic, diff -r clean) and its 8 frames + WAV + both minimax-h3-mux --print-only argv lines are the committed goldens in fixtures/minimax_h3_video_fold/. W1 seam: vllm::multimodal::MiniMaxH3VideoEngine absorbs the 1293-line example driver AND the server's 354-line /v1/videos twin — 4 DiT loader arms (GGUF keep-quant / dequant-bf16 host+streamed / bf16-shard stream / NVFP4 bf16+fp4-resident), VAE decoder + lazy encoder halves, the H3-Encoder tower staged once (GGUF or bf16 shards), the #77 partition guard, fl2va/ref2va conditioning, the byte-exact splitmix64 Box-Muller noise streams (VT_H3_GAUSSIAN_NOISE honored; seeded requests derive the audio stream via splitmix64), artifact writing and mux-argv assembly. The library SPAWNS NOTHING (mkdir -p became std::filesystem; ffmpeg stays caller-side — the ratified 2026-08-03 boundary). MiniMaxH3VideoGenParamsFromRequest is the ONE /v1/videos mapping so HTTP and FFI cannot drift. W2 ABI: the ratified vllm_video_* slice (one-surface-abi.md:92-108) — vllm_video_engine_load/free, vllm_video_model_params/_default, vllm_video_params/_default, vllm_video_generate, vllm_video_result/_free — plus the engine-free vllm_video_mux_argv(+_params_default,+_free) composer the mux thin client needs. VLLM_ABI_VERSION 11 -> 12; test_capi floor pin >= 12 (the == checks move with the macro). As-shipped deltas vs the proposal argued in the spec (+prompt_embeds_path/+partition/+fp4_resident/+output_dir; -task/-duration; ONE ref_image). Zero values preserve behaviour. Refusals both directions: vllm_video_engine_load on a text dir names vllm_engine_load; vllm_engine_load on the H3 dir fails byte-for-byte as captured at v11. W3 server: the VideoState block is DELETED; /v1/videos = seam Load + a runner lambda (FromRequest -> Generate -> fork/execvp(mux_argv)). Direct MiniMaxH3 refs 54 -> 7, all seam type names. DISCLOSED numeric deltas on the server arm (no goldens existed; the drift WAS the defect): legacy single-stream uniform noise -> the shared ratified recipe; host-f32 GGUF default -> keep-quant; new --video-dequant-bf16 flag. W4 examples: minimax_h3_gen (1293 -> 216 lines) + minimax_h3_mux are vllm.h+vllm::shared thin clients, BOTH byte-identical to the pre-fold binaries on the fixture (frames+WAV cmp clean; mux argv diff clean; the pre-fold diagnostic modes died with the private pipeline — named residual). Ratchet: both example-abi-allowlist rows removed, MAX_INTERNAL_REACHING 11 -> 9 with the equality pin + spec claims moved; the abi-capability video row closes; FEATURES video row -> reachable naming the four symbols. Gates: three-arm fold gate test_minimax_h3_video_fold 4 cases / 131 assertions GREEN (seam == replicated pre-fold pipeline == committed goldens, byte-identical; partition-guard + input refusals; FromRequest field-complete); test_capi 40/373 incl. the v12 golden e2e THROUGH the C marshalling; vllm_capi_c_check strict-C11 green; test_minimax_h3 75/75 unchanged; test_openai_api_server 45/566 incl. the socket-level routes-do-not-exist-without-a-runner 404 pin; test_video_api 14/14; check-surface-coverage green with the SHRINK enforced + its 46-test mutation suite. GB10 real-video re-verification through the v12 ABI is a NAMED RESIDUAL (box on the Kimi campaign); records updated (STATUS/BENCHMARKS/FEATURES/NOW/state/model-matrix + both specs). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…— vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples ONE-SURFACE fold ROW 2 (row/H3-VIDEO-ABI, task #283; the #121 precedent's shape: three-arm fold gate, guard-ratchet shrink). The library absorbs everything the examples privately owned. GOLDENS FIRST: minimax_h3_video_fold_fixture.h writes a deterministic tiny checkpoint set (857KB F32 ComfyUI-GGUF DiT at the test_minimax_h3.cpp:3786 reduced geometry, on-disk reduced ViT3D/BigVGAN VAEs + config.json, prompt embeds); the PRE-fold minimax-h3-gen binary AT THE BRANCH BASE (fc636c7) rendered it (--partition fl2va --steps 3 --frames 5 --height/width 32, CPU, keep-quant; run-to-run deterministic, diff -r clean) and its 8 frames + WAV + both minimax-h3-mux --print-only argv lines are the committed goldens in fixtures/minimax_h3_video_fold/. W1 seam: vllm::multimodal::MiniMaxH3VideoEngine absorbs the 1293-line example driver AND the server's 354-line /v1/videos twin — 4 DiT loader arms (GGUF keep-quant / dequant-bf16 host+streamed / bf16-shard stream / NVFP4 bf16+fp4-resident), VAE decoder + lazy encoder halves, the H3-Encoder tower staged once (GGUF or bf16 shards), the #77 partition guard, fl2va/ref2va conditioning, the byte-exact splitmix64 Box-Muller noise streams (VT_H3_GAUSSIAN_NOISE honored; seeded requests derive the audio stream via splitmix64), artifact writing and mux-argv assembly. The library SPAWNS NOTHING (mkdir -p became std::filesystem; ffmpeg stays caller-side — the ratified 2026-08-03 boundary). MiniMaxH3VideoGenParamsFromRequest is the ONE /v1/videos mapping so HTTP and FFI cannot drift. W2 ABI: the ratified vllm_video_* slice (one-surface-abi.md:92-108) — vllm_video_engine_load/free, vllm_video_model_params/_default, vllm_video_params/_default, vllm_video_generate, vllm_video_result/_free — plus the engine-free vllm_video_mux_argv(+_params_default,+_free) composer the mux thin client needs. VLLM_ABI_VERSION 11 -> 12; test_capi floor pin >= 12 (the == checks move with the macro). As-shipped deltas vs the proposal argued in the spec (+prompt_embeds_path/+partition/+fp4_resident/+output_dir; -task/-duration; ONE ref_image). Zero values preserve behaviour. Refusals both directions: vllm_video_engine_load on a text dir names vllm_engine_load; vllm_engine_load on the H3 dir fails byte-for-byte as captured at v11. W3 server: the VideoState block is DELETED; /v1/videos = seam Load + a runner lambda (FromRequest -> Generate -> fork/execvp(mux_argv)). Direct MiniMaxH3 refs 54 -> 7, all seam type names. DISCLOSED numeric deltas on the server arm (no goldens existed; the drift WAS the defect): legacy single-stream uniform noise -> the shared ratified recipe; host-f32 GGUF default -> keep-quant; new --video-dequant-bf16 flag. W4 examples: minimax_h3_gen (1293 -> 216 lines) + minimax_h3_mux are vllm.h+vllm::shared thin clients, BOTH byte-identical to the pre-fold binaries on the fixture (frames+WAV cmp clean; mux argv diff clean; the pre-fold diagnostic modes died with the private pipeline — named residual). Ratchet: both example-abi-allowlist rows removed, MAX_INTERNAL_REACHING 11 -> 9 with the equality pin + spec claims moved; the abi-capability video row closes; FEATURES video row -> reachable naming the four symbols. Gates: three-arm fold gate test_minimax_h3_video_fold 4 cases / 131 assertions GREEN (seam == replicated pre-fold pipeline == committed goldens, byte-identical; partition-guard + input refusals; FromRequest field-complete); test_capi 40/373 incl. the v12 golden e2e THROUGH the C marshalling; vllm_capi_c_check strict-C11 green; test_minimax_h3 75/75 unchanged; test_openai_api_server 45/566 incl. the socket-level routes-do-not-exist-without-a-runner 404 pin; test_video_api 14/14; check-surface-coverage green with the SHRINK enforced + its 46-test mutation suite. GB10 real-video re-verification through the v12 ABI is a NAMED RESIDUAL (box on the Kimi campaign); records updated (STATUS/BENCHMARKS/FEATURES/NOW/state/model-matrix + both specs). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Result: the MiniMax-H3 render bug is CLOSED
The #70/#74 white-latent render was using the wrong checkpoint PARTITION for the task, not a code bug. MiniMax-H3 ships two independently-served DiT partitions and the task must match (
recipes/MiniMaxAI/MiniMax-H3.md:50,289;pipeline._resolve_taskraises otherwise):MiniMax-H3-FL2VA-Q3_K_M.ggufminimax_h3_ref2va_nvfp4_full(what #70/#74 used)Every render up to #74 ran t2va on the Ref2VA-partition checkpoint — out of distribution → the white latent, invariant to prompt/steps.
Proof: t2va on
MiniMax-H3-FL2VA-Q3_K_M.ggufrenders a COHERENT, prompt-matched scene on GB10 — a photorealistic orange cat sitting on a wooden table. 512×512/22f: VAE-input latent adj-cell cosine 0.9467 (white=0.06, real latent=0.789), frame seam16/interior 1.00 (no 16px patch grid), velocity stable, final latent rms 1.00. 50-step 768×1344 render: adj-cos 0.93, a detailed kitchen scene. Valid h264/AAC mp4s.What was verified BEFORE switching partitions (S1/S2 exoneration)
VT_H3_DUMP_INPUTS, new): the t2va DiT step-0 inputs diff EXACTLY vs upstreampipeline_minimax_h3.pyat real 512×512/22f (packed layout, fp64 grid, token_tags, inverse/combined AdaLN indices, sigmas byte-equal); tokenization byte-equal totokenizer(prompt, add_special_tokens=False).DequantNvfp4ToBf16byte-exact (Laguna/Qwen3 + independent torch dequant).view(m*3,6H), 3D-RoPE, modulate).Commits
spike:VT_H3_DUMP_INPUTSDiT-input dump (env-gated, byte-identical off).test: CUDA-device-vs-host forward at the real render seq (1920).fix: strip prepended pinned reference rows inMiniMaxH3GenerateT2vabefore unpatchify/unpack (a real, never-run ref2va OUTPUT bug; no-op for t2va/fl2va).docs: STATUS/FEATURES/BENCHMARKS/benchmark-record/spec §8.6/state/NOW.Open follow-ups
supported_tasksguard mirroring upstream (community GGUF/NVFP4 files strip the release config).Pre-existing preflight red
check-fusion-consistency minimax_h3_video_vae_deviceis unrelated (0 diff from origin/main). All doc gates green.Generated with Claude Code.