fix(minimax-h3): ref2va block-dim double-division fixed + gated; render grid RE-ATTRIBUTED to the NVFP4 checkpoint - #93
Closed
localai-bot wants to merge 1 commit into
Closed
Conversation
… to the NVFP4 checkpoint (row/H3-REF2VA-ASSEMBLY)
MiniMaxH3EncodeReferenceImages/Video emitted the ref block with PATCHED dims
(ls.h/patch_size_h), but BuildMiniMaxH3PackedSequenceRef2va divides
block.latent_h/kPatchH AGAIN -- it mirrors upstream packed_sequence.py:328-330,
which takes the UNPATCHED latent (upstream feeds the raw latent visual_shape =
(1, height//16, width//16), pipeline_minimax_h3.py:1141-1145). The double
division under-allocated the reference span by patch_h*patch_w (=4); the denoise
pin-loop's >= check silently truncated the oversized encoded reference to its
first quarter -- a coherence-mangling ref2va assembly bug. Fix: emit raw
ls.{t,h,w} in both encode functions.
Permanent gates (the recurring blind-spot class closed):
- goldens section 5c: a ref2va-shaped DiT-forward rung (image + video+audio
reference blocks, 8x8 geometry) forwarded through RefDiT with the ref2va
per-token timestep partition + audio update mask; C++ case "DiT-forward REF2VA
rung matches upstream (reference rows, mixing)" gates layout + host/device
logits (<=2e-5) + reference-row masking + target mixing. No prior DiT-forward
gate ever forwarded a ref2va layout. Needed a 1-line RefDiT extension (honor
audio_update_mask, mirroring minimax_h3_transformer.py:1099-1101).
- RED-first encoded-vs-layout row-count invariant in the ref2va image+video
subcases: reintroducing the bug fails 128==512 (16 encoded vs 4 allocated),
restored -> green. Suite 69/69, 52377 assertions. Goldens regen purely
additive (635 inserts, 0 deletes).
GB10 render A/B + isolation (256x256/22f/12steps, fixed binary) DISPROVES the
prior attribution: ref2va grids in BOTH fp4-resident AND bf16, and t2va with
ZERO reference assembly on the ref2va NVFP4 also grids, while every FL2VA-GGUF
render is coherent. The grid is NOT the assembly and NOT fp4 -- it correlates
1:1 with the ref2va NVFP4 CHECKPOINT/loader. The DiT math, the ref2va assembly
math, and the vision scatter are now all gate-proven vs upstream; the true
residual is the NVFP4 DiT loader for this file (fp32-island preservation /
weight_scale_2 / tensor-name mapping). See spec section 8.10.
Row: row/H3-REF2VA-ASSEMBLY
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/H3-REF2VA-ASSEMBLY
branch
from
August 7, 2026 03:41
c93161f to
b188e82
Compare
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…d re-attributed to the NVFP4 checkpoint/loader (#93) row/H3-REF2VA-ASSEMBLY squash. REAL BUG fixed: EncodeReferenceImages/ Video wrote the RefBlock with PATCHED dims while BuildMiniMaxH3PackedSequenceRef2va divides by the patch again (mirroring upstream packed_sequence.py:328-330, which takes the raw latent visual_shape per pipeline_minimax_h3.py:1141-1145) — the double division under-allocated the reference span by patch_h*patch_w = 4x and the pin-loop silently truncated the encoded reference to its first quarter. Fix: emit raw ls.{t,h,w} (~6 lines). RED-first gate (reintroducing the bug fails 128==512) + the FIRST ref2va-shaped DiT rung (reference rows + ref2va timestep partition + audio update mask; layout byte-exact, logits <=2e-5, mixing 1.0) — the blind-spot class closed permanently. Suite 69/69, 52377 asserts. Timesteps on reference rows, packed layout, tags, #77 bookkeeping all CLEARED vs upstream. Checkpoint-controlled A/B re-attributes the grid: it appears with the ENTIRE reference assembly removed, in fp4 AND bf16, and correlates 1:1 with the ref2va NVFP4 checkpoint — every render loading that file grids; every FL2VA-GGUF render is coherent (control re-verified). §8.6/§8.9 varied task/prompt but never checkpoint — both misattributed. True residual: the NVFP4 DiT load path for THIS file (fp32-island preservation, weight_scale_2, tensor mapping) — the synthetic gates proved the dequant math, never this file against a coherent oracle. Next: independent CPU dequant of the same file as the loader oracle (no new checkpoint needed). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The loader-vs-independent-dequant diff of the ref2va NVFP4 file runs next — no new checkpoint download needed. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…user docs PR #68. A real bug found by RENDERING, not by the suite, plus the user-facing documentation the H3 lane never had. THE BUG. A 124-frame render silently muxed as 61 frames. `audio_t` is the PER-CHANNEL latent length (the planner sets it from 40 Hz * duration) and the packed layout carries `audio_t * audio_channel` ROWS, one per (channel, step). The denormalize step divided by the channel count, so the decoded audio ran half the video's duration; because the muxer passes `-shortest`, that silently truncated the VIDEO to half its frames too. The fix is one line, and it now agrees with the same function's own `target_audio_rows = request.audio_t * request.audio_channel` a few lines above. WHY THE SUITE WAS BLIND, which is the part worth keeping. Every existing gate asserts shape SELF-CONSISTENCY, and a uniformly halved pipeline is perfectly self-consistent: the shapes agreed with each other, they were just half as long as the request asked for. ffprobe on the artifact exposed it. The gate added is the one that is NOT self-consistency: the structural t2va e2e case now decodes `audio_t` latent steps INDEPENDENTLY and requires the pipeline to have produced exactly that many samples. RED-first: restoring the divide turns that case RED (1 failed assertion), removing it goes GREEN. Suite 69/69, 47416 assertions. A SECOND BUG FROM THE SAME SESSION, DROPPED RATHER THAN LANDED TWICE. Conditioning PREPENDS rows to img_pos, so the denoise loop returns condition rows followed by the targets, and unpatchify was handed the whole set. This branch fixed it by carrying the layout's own `update_mask` out of the loop; main fixed the SAME bug independently in row/H3-RENDER-CLOSE by taking the TRAILING target_video_rows with a VT_CHECK on the row accounting. On rebase the update_mask mechanism and its test were DROPPED: landing a second solution to a fixed bug is churn, and the positional form already ships gated. The alternative is recorded in .agents/state.md rather than silently lost. DOCS. The video+audio generation path had no user-facing documentation at all, and the multimodal INPUT interface was documented WRONG: `vllm-cli --image/--video/--audio` does not exist, the CLI is text-only. Multimodal input is served over the OpenAI API as `image_url`, `video_url` and `input_audio`/`audio_url` content parts on /v1/chat/completions (src/vllm/entrypoints/openai/chat_mm.cpp), which was documented NOWHERE, so docs/USAGE.md now states it rather than the wrong claim merely being removed. README gains a News section led by video+audio generation, paid for INSIDE the 30,000-char landing-page budget rather than by raising it. PUBLIC DOCS. The 19-line narrative this branch originally appended to docs/BENCHMARKS.md was a non-canonical H2 section and pushed the page over its prose budget: converted into three keyed ROWS (Thor render speed, render duration, quantization floor), with the forensics moved verbatim into .agents/benchmark-record.md, which is what that page's own checker instructs. ALSO REPAIRS RECORD DAMAGE THAT LEFT check-agent-record RED ON MAIN. #93's squash-merge removed 130 lines from specs/minimax-h3.md: its branch predated #71/#92, so landing it DELETED sections 9 and 10 (the /v1/videos rows) and left two orphaned table fragments with no heading and no separator. The checker reports that as "table has 4 pipes; expected 3" plus "no linked spec names exact stable token SERVE-VIDEOS-REFS", the engine-matrix row pointing at a section that no longer existed. Restored sections 9 and 10 VERBATIM from 548b000 and cut the orphans; #93's own content is preserved byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
localai-bot
added a commit
that referenced
this pull request
Aug 7, 2026
…user docs (#68) PR #68. A real bug found by RENDERING, not by the suite, plus the user-facing documentation the H3 lane never had. THE BUG. A 124-frame render silently muxed as 61 frames. `audio_t` is the PER-CHANNEL latent length (the planner sets it from 40 Hz * duration) and the packed layout carries `audio_t * audio_channel` ROWS, one per (channel, step). The denormalize step divided by the channel count, so the decoded audio ran half the video's duration; because the muxer passes `-shortest`, that silently truncated the VIDEO to half its frames too. The fix is one line, and it now agrees with the same function's own `target_audio_rows = request.audio_t * request.audio_channel` a few lines above. WHY THE SUITE WAS BLIND, which is the part worth keeping. Every existing gate asserts shape SELF-CONSISTENCY, and a uniformly halved pipeline is perfectly self-consistent: the shapes agreed with each other, they were just half as long as the request asked for. ffprobe on the artifact exposed it. The gate added is the one that is NOT self-consistency: the structural t2va e2e case now decodes `audio_t` latent steps INDEPENDENTLY and requires the pipeline to have produced exactly that many samples. RED-first: restoring the divide turns that case RED (1 failed assertion), removing it goes GREEN. Suite 69/69, 47416 assertions. A SECOND BUG FROM THE SAME SESSION, DROPPED RATHER THAN LANDED TWICE. Conditioning PREPENDS rows to img_pos, so the denoise loop returns condition rows followed by the targets, and unpatchify was handed the whole set. This branch fixed it by carrying the layout's own `update_mask` out of the loop; main fixed the SAME bug independently in row/H3-RENDER-CLOSE by taking the TRAILING target_video_rows with a VT_CHECK on the row accounting. On rebase the update_mask mechanism and its test were DROPPED: landing a second solution to a fixed bug is churn, and the positional form already ships gated. The alternative is recorded in .agents/state.md rather than silently lost. DOCS. The video+audio generation path had no user-facing documentation at all, and the multimodal INPUT interface was documented WRONG: `vllm-cli --image/--video/--audio` does not exist, the CLI is text-only. Multimodal input is served over the OpenAI API as `image_url`, `video_url` and `input_audio`/`audio_url` content parts on /v1/chat/completions (src/vllm/entrypoints/openai/chat_mm.cpp), which was documented NOWHERE, so docs/USAGE.md now states it rather than the wrong claim merely being removed. README gains a News section led by video+audio generation, paid for INSIDE the 30,000-char landing-page budget rather than by raising it. PUBLIC DOCS. The 19-line narrative this branch originally appended to docs/BENCHMARKS.md was a non-canonical H2 section and pushed the page over its prose budget: converted into three keyed ROWS (Thor render speed, render duration, quantization floor), with the forensics moved verbatim into .agents/benchmark-record.md, which is what that page's own checker instructs. ALSO REPAIRS RECORD DAMAGE THAT LEFT check-agent-record RED ON MAIN. #93's squash-merge removed 130 lines from specs/minimax-h3.md: its branch predated #71/#92, so landing it DELETED sections 9 and 10 (the /v1/videos rows) and left two orphaned table fragments with no heading and no separator. The checker reports that as "table has 4 pipes; expected 3" plus "no linked spec names exact stable token SERVE-VIDEOS-REFS", the engine-matrix row pointing at a section that no longer existed. Restored sections 9 and 10 VERBATIM from 548b000 and cut the orphans; #93's own content is preserved byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context) Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…ND+FIXED, 2nd defect isolated (#94) Records for row/H3-NVFP4-LOADER-DIFF (fix in the preceding commit): spec §8.11 + §8.2 supports-audit row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record entry, ENVIRONMENT (VT_H3_NVFP4_LOWNIBBLE), NOW, state. Independent-oracle loader diff (no download) root-caused #93's residual: the community ckpt packs fp4 HIGH-first (converter "Star Ultimate Model Converter Pro"); our loader read low-first -> every adjacent pair swapped -> scrambled projections. Vs the coherent FL2VA GGUF (same base model: islands byte-identical, all sampled projections sign-agree 1.000): low-first corr 0.000, high-first sign-agree 1.000 over 115M weights. Fixed by a load-time nibble swap in the 3 H3 NVFP4 loaders; BYTE-VERIFIED on GB10 (binary streamer dump == oracle high-first == GGUF-sign; params identical). #93's three guesses all missed the nibble order. HONEST residual: the render STILL grids (t2va, ref2va, keyframe-pinned all grid) while the FL2VA-GGUF control renders coherent on IDENTICAL weights+params -> a SECOND, independent defect in the NVFP4 render PATH (device stream/forward), NOT the checkpoint content, the nibble order, fp4, or free-generation. fp4-resident Marlin arm grids differently again (3rd, Marlin-specific). The nibble fix is the objectively-correct dequant and lands default-ON (VT_H3_NVFP4_LOWNIBBLE=1 reverts), byte-verified but not yet render-validated. Next: layer activation diff NVFP4 vs GGUF stream. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
… packing unscrambled (#94) row/H3-NVFP4-LOADER-DIFF squash. Independent-oracle loader diff (own fp8-e4m3fn/E2M1/bf16-RNE math, primitives byte-verified vs torch; cross-checked against the coherent FL2VA GGUF): the community ref2va NVFP4 checkpoint ("Star Ultimate Model Converter Pro") packs the two fp4 elements per byte HIGH-first — opposite the modelopt standard our dequant assumes. Read low-first, every adjacent weight pair swaps: corr 0.000 (scrambled) -> sign-agreement 1.000 over 115M+ weights with the swap (corr 0.85-0.94 = pure NVFP4-vs-Q3_K quant noise); fp32/bf16 islands were byte-identical all along (same base model, not nibble-packed). Fix: nibble-swap at load in the three H3 NVFP4 loaders (H3-scoped; the shared DequantNvfp4ToBf16 stays low-first for Laguna/DS4/Qwen3), default ON, VT_H3_NVFP4_LOWNIBBLE=1 reverts. #93's three suspects (islands/weight_scale_2/name-map) all cleared. HONEST: byte-correct weights do NOT un-grid the render — dark grid becomes pale grid; keyframe-pinning does not rescue; the FL2VA-GGUF control stays coherent in the same build. Since checkpoint content now provably matches the coherent GGUF's model, the residual is a SECOND defect in the NVFP4 render PATH (device stream/forward; the fp4-resident Marlin arm grids differently again — a third, wiring-gated-only issue). Next: layer-by-layer activation diff of the NVFP4-bf16 stream vs the GGUF-bf16 stream — identical weights, differing only in dequant source and island read width. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Took §8.9's residual (the ref2va reference-row assembly) and ran the #70 latent-bisection discipline at the assembly boundary: diffed the whole assembly vs upstream on CPU, found + fixed a real bug, extended the permanent gate to a ref2va-shaped rung, then re-rendered on real weights — which disproved §8.9's attribution.
The real assembly bug (fixed)
MiniMaxH3EncodeReferenceImages/Video(minimax_h3_pipeline.cpp) emitted theMiniMaxH3RefBlockwith PATCHED dims (ls.h / patch_size_h), butBuildMiniMaxH3PackedSequenceRef2va(minimax_h3_packing.cpp:453) dividesblock.latent_h / kPatchHagain — mirroring upstreampacked_sequence.py:328-330, which takes the UNPATCHED latent (upstream feeds the raw latent:pipeline_minimax_h3.py:1141-1145setsvisual_shape = (1, height//16, width//16)). The double-division under-allocated the reference span bypatch_h*patch_w(=4); the denoise pin-loop's>=check silently truncated the oversized encoded reference to its first quarter. Fix: emit the rawls.{t,h,w}. The other three suspects (per-token timesteps, packed layout/grid/tags, #77 output bookkeeping) were cleared by the same read — all byte-exact vs upstream.Permanent gates (blind-spot class closed)
RefDiTwith the ref2va per-token timestep partition + audio update mask. C++ case"DiT-forward REF2VA rung matches upstream (reference rows, mixing)"gates the layout, host+device video/audio logits (≤2e-5), reference-row output masking, and target-row spatial mixing. No prior DiT-forward gate ever forwarded a ref2va layout — the §8.9 blind spot. Needed a 1-line RefDiT extension (honoraudio_update_mask, mirroringminimax_h3_transformer.py:1099-1101; our port already did). Goldens regen purely additive (635 inserts, 0 deletes).128 == 512(16 encoded rows vs 4 allocated — the exact 4× patch double-count); restored → green. Suite 69/69, 52377 assertions (CPU Release, standalone).GB10 render A/B + isolation (256×256/22f/12steps, fixed binary)
--partition fl2va)Re-attribution (corrects §8.9 and §8.6)
The grid is NOT the ref2va assembly and NOT fp4 — it appears with the entire reference assembly removed, in both fp4 and bf16. It correlates 1:1 with the ref2va NVFP4 checkpoint (
minimax_h3_ref2va_nvfp4_full): every render loading it grids; every FL2VA-GGUF render is coherent. §8.9's A/B varied only the prompt and §8.6's only the task — neither varied the checkpoint/quant, so both misattributed a checkpoint/loader defect. The true residual is the NVFP4 DiT loader for this file (StreamMiniMaxH3Nvfp4ToDeviceBf16/Fp4): suspect fp32-island preservation (patch/time/output layers,minimax_h3_transformer.py:898-904),weight_scale_2double-dequant, or the 1051-tensor name mapping. Next: a REF2VA GGUF (bf16, known-good loader) as checkpoint oracle — dgx-disk-blocked. See spec §8.10.Note: the one preflight red (
check-fusion-consistency,minimax_h3_video_vae_device) is pre-existing and untouched by this change.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys