Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .agents/specs/minimax-h3.md
Original file line number Diff line number Diff line change
Expand Up @@ -1340,3 +1340,45 @@ and were verified one by one. The defect lived in the INTERPRETATION of a buffer
between two correct functions — invisible to any check that does not compare the two
readers against each other. The fix pins the expected size with a `VT_CHECK` so the
assumption cannot drift again silently.

## 8.20 ALL THREE MODALITIES WORK — the ref2va grid was the QUANTISATION, and §8.9's assembly hypothesis is REFUTED (2026-08-10, `row/H3-REF2VA-WORKS`, Thor sm_110)

Three campaigns fought the ref2va patch grid (#86 text-only, §8.9 vision-enriched, §8.12
load-path). §8.9 re-attributed it to the **ref2va-specific reference-row conditioning
ASSEMBLY** (`MiniMaxH3EncodeReferenceImages` + the ref2va packed-block layout + how the
loop conditions un-pinned target rows), having ruled out the DiT forward and the encoder.
That attribution is **REFUTED**.

**Every prior ref2va attempt ran on NVFP4** (`minimax_h3_ref2va_nvfp4_full`). The one cell
nobody filled in was ref2va on the quantisation this project has verified coherent
everywhere else. `MiniMax-H3-REF2VA-Q4_K_M.gguf` (19,864,208,064 bytes,
`realrebelai/MiniMax-H3_GGUFs`) renders **COHERENTLY** with the identical assembly code:

| arm | period-16 seam | VAE-input latent adj-cell cosine |
|---|---|---|
| ref2va on NVFP4 (§8.9) | grid, every frame | — |
| **ref2va on Q4_K_M (this row)** | **1.13** | **0.8526** |
| t2va on Q4_K_M (§8.16) | 1.19 | 0.8924 |
| fl2va on Q4_K_M (this row) | 1.06-1.44 | — |

512x512/124f/50 steps, `--partition ref2va --ref-image`, seam scale 1.15-1.19 clean /
2.28 broken, cosine scale 0.06 white / 0.789 real encode / 0.89+ coherent. The decoded
frame is a coherent office scene with no trace of the multicolour patch grid. §8.12's
quieter attribution ("100% attributable to the per-weight NVFP4-vs-Q3_K quantization
difference") was the correct one.

**All three H3 modalities are now verified on Q4_K_M**, each with a rendered artifact:
t2va (§8.16, 1344x768 + ASR-verified speech), fl2va (this row: first-frame conditioning
load-bearing at keyframe→frame0 correlation **0.7881**, and the clip evolves away from the
pin at 0.7653), ref2va (this row).

**Why it survived three investigations.** The FL2VA/Q4_K_M cell was verified and the
Ref2VA/NVFP4 cell was verified-broken, but the two variables — PARTITION and QUANTISATION —
were never crossed. Each campaign held quantisation fixed at NVFP4 and searched the ref2va
code path, which is where the bug was not. The lesson generalises: when a mode fails on
exactly one checkpoint, vary the CHECKPOINT before searching the mode's code.

**Residual.** The NVFP4 Ref2VA arm is still broken and unexplained at the weight level
(§8.12 found no discrete load-path defect, so it is the per-weight quantisation error
itself). `docs/USAGE.md` now steers users to the REF2VA GGUF and warns off NVFP4 rather
than leaving that trap live.
30 changes: 24 additions & 6 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,11 +388,11 @@ because a render takes hours before it tells you anything:

| arm | status |
|---|---|
| **Q4_K_M** | **VERIFIED end to end** — every render in this doc. Use this. |
| **Q4_K_M** | **VERIFIED end to end** — every render in this doc, on BOTH partitions (t2va + fl2va on FL2VA, ref2va on REF2VA). Use this. |
| Q3_K_M | verified BAD (the A/B above): murky silhouette under a lattice |
| bf16 (66.3 GB, 13 shards) | loader + device streamer implemented and gated, but **CPU-only** verification — no end-to-end GPU render has been done |
| NVFP4 | exists; pruned variants are NOT drop-in (see above) |
| Q8 | no published GGUF is known and none has been tested |
| Q8 / Q6_K / Q5_0 | published ([unsloth/MiniMax-H3-GGUF](https://huggingface.co/unsloth/MiniMax-H3-GGUF)) but **PRUNED only** — the restructured AdaLN is not drop-in and this loader cannot read it |

### The trap: this checkpoint does not serve every task

Expand All @@ -419,10 +419,28 @@ the resolution you actually want.
Pass `--partition fl2va` explicitly. The driver mirrors upstream's raise, so a
mismatch is rejected at the CLI rather than silently rendered.

For a reference-image render you need the **Ref2VA** partition instead
(`minimax_h3_ref2va_nvfp4_full`, or a REF2VA GGUF). Note that the encoder vision
tower is not yet ported, so image-conditioned renders are not clean even on the
matching partition.
For a reference-image render you need the **Ref2VA** partition instead, and the
one to use is **`MiniMax-H3-REF2VA-Q4_K_M.gguf`** (19.9 GB,
[realrebelai/MiniMax-H3_GGUFs](https://huggingface.co/realrebelai/MiniMax-H3_GGUFs)) —
the same quantisation as the FL2VA file above, and verified coherent:

```sh
build/examples/minimax-h3-gen \
--dit MiniMax-H3-REF2VA-Q4_K_M.gguf --dequant-bf16 --partition ref2va \
--encoder qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf --tokenizer tokenizer.json \
--prompt "..." --ref-image subject.ppm \
--video-vae video_vae.safetensors --video-vae-config video_vae_config.json \
--audio-vae audio_vae.safetensors --audio-vae-config audio_vae_config.json \
--frames 124 --height 512 --width 512 --steps 50 \
--device cuda --out out.mp4 --workdir /tmp/h3
```

**Do NOT use the NVFP4 Ref2VA weights.** `minimax_h3_ref2va_nvfp4_full` renders the
multicolour patch grid, and it took three investigations to establish that this is the
QUANTISATION and not the ref2va path: the identical reference-row assembly, packed-block
layout and denoise loop render coherently on Q4_K_M (period-16 seam **1.13**, VAE-input
latent adjacent-cell cosine **0.8526**). Ref2VA on Q4_K_M is a working mode; Ref2VA on
NVFP4 is not.

### Writing the prompt (read this first)

Expand Down
Loading