row/H3-RENDER-CLOSE: MiniMax-H3 render bug CLOSED (wrong partition; FL2VA t2va renders a coherent scene)

.agents/NOW.md

@@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | DeepSeek-V4-Flash decode | **Closed: beats ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact); phase-2 residency NEG, default-OFF | — |
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + AGENTS.md checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | **#70** DiT-math bug REFUTED (`H3-DIT-SCALE-GATE` PR #74): ladder 2x3->8x8+temporal ours==oracle host+dev <=3e-7; white=TRAINED | dgx: real upstream vs RefDiT |
+| MiniMax-H3 lane | **RENDER BUG CLOSED** (`H3-RENDER-CLOSE` PR #77): #70/#74 white = t2va on the REF2VA ckpt; the FL2VA GGUF t2va renders COHERENT (adj-cos 0.95) | Follow-up: partition guard + vision tower |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |

.agents/benchmark-record.md

@@ -14159,3 +14159,53 @@ noise fix from #70 stands but was already known not to be the render fix.
    full canvas, and the real per-token timestep layout. A real-weights activation diff of
    the DiT INPUTS (encoder output, position grid, condition-noise) at real geometry is
    the untested surface #70 did not isolate.
+
+## MiniMax-H3 render bug CLOSED — the render ran t2va on the ref2va PARTITION checkpoint; t2va on the FL2VA partition renders a COHERENT scene (2026-08-06, `row/H3-RENDER-CLOSE` PR #77, `ROAD-V1-H3`, dgx GB10 sm_121a)
+
+**Verdict.** The #70/#74 white latent was NOT a code bug. Every prior render ran
+**task=t2va on `minimax_h3_ref2va_nvfp4_full`, which is the REF2VA partition.**
+Upstream ships two independently-served partitions and requires the task to match:
+*"Set MODEL to FL2VA for T2VA"*, ref2va runs against the Ref2VA partition, and
+*"task ... must match the served partition"* (`recipes/MiniMaxAI/MiniMax-H3.md:50,222,289`;
+`pipeline._resolve_task` RAISES otherwise, `pipeline_minimax_h3.py:387-390`). A
+ref2va-trained DiT fed a t2va sequence (no reference block) is out of distribution →
+the spatially-white latent, invariant to the text prompt and step count — exactly #70.
+
+**How it was cornered (all NEW, all measured on dgx at real 512x512/22f scale):**
+| Suspect | Test | Result |
+|---|---|---|
+| S1 (a)(b) DiT INPUT wiring at real scale | `VT_H3_DUMP_INPUTS` dumped every step-0 DiT input; diffed vs upstream `pipeline_minimax_h3.py` at t2va 512x512/22f (text_len=8, latent 7x32x32, seq_len 1920) | **EXACT** — packed layout, fp64 grid, token_tags, inverse/combined AdaLN indices, sigmas all byte-equal; tokenization byte-equal to `tokenizer(prompt,add_special_tokens=False)` |
+| encoder conditioning | shape/stat check | correct [8,5120], carries the expected Qwen massive-activation (row0 ch731=15915, others rms~4) — not all-pad, not garbage |
+| NVFP4 dequant | independent torch dequant of `blocks.0.attn.qkv_proj` + Laguna/Qwen3 already prove `DequantNvfp4ToBf16` byte-exact | sane trained weight (rms 0.089, absmax=ws2·6·maxscale=3.61) |
+| CUDA kernels at real seq | NEW gate `test_minimax_h3 :: CUDA device forward tracks the host at the REAL render seq (1920)` (RealRatioParams head_dim=128, seq 1920) | **CUDA device == CPU host** (28/28) — no scale-dependent kernel bug (#74 only ran device-vs-host on the CPU backend) |
+| forward math | RefDiT restatement vs true upstream source, read side by side (block, attention, AdaLN view(m*3,6H), 3D-RoPE, modulate) | identical |
+
+So inputs + forward + kernels + dequant are all correct → the only thing left was
+the checkpoint↔task pairing.
+
+**Proof.** Downloaded the FL2VA-partition DiT `MiniMax-H3-FL2VA-Q3_K_M.gguf` (15.58 GB,
+`realrebelai/MiniMax-H3_GGUFs`, same 50L/5376/head128 geometry) and rendered the SAME
+t2va prompt *"an orange cat sitting on a wooden table"* at 512x512/22f, 20 steps,
+`--dequant-bf16`:
+- VAE-input latent **adj-cell cosine = 0.9467** (vs 0.06 white on the ref2va checkpoint; a real encoded latent is 0.789), latent rms 0.10.
+- frame **seam16/interior = 1.00** (no 16px patch grid), and the decoded frames SHOW a
+  photorealistic orange cat sitting on a wooden table, prompt-matched, temporally
+  evolving across the 22 frames. Valid `h264 512x512 + AAC 32kHz stereo` mp4.
+- healthy denoise signature: velocity STABLE ~1.37 rms, final latent rms **1.00** (the
+  broken ref2va-t2va run blew up to 2.64).
+
+**Secondary bug FOUND+FIXED (this PR).** Running the CORRECT task (ref2va) surfaced a
+real, never-exercised bug in `MiniMaxH3GenerateT2va`: `BuildMiniMaxH3PackedSequenceRef2va`
+PREPENDS pinned reference rows, the DiT zeroes them in its output, and the pipeline handed
+the full (reference+target) buffer to unpatchify → `rows not divisible by t*h*w`. Fixed by
+slicing to the TRAILING target rows (no-op for t2va/fl2va). (ref2va with a SYNTHETIC image+
+tone reference + text-only encoder still gridded — expected: a meaningless reference plus
+the still-unported encoder vision tower is weak conditioning; the clean confirmation is the
+FL2VA t2va render above, which needs neither.)
+
+**Residuals.** (1) The driver takes NO partition/supported_tasks guard (the community GGUF/
+NVFP4 files strip the release config), so picking the right checkpoint per task is on the
+caller — mirror-upstream guard is a follow-up. (2) The encoder vision tower (W3 remnant) is
+still unported, so real image/video-conditioned ref2va/fl2va renders are not yet clean. (3)
+50-step render at the reference canvas (768x1344) is the artifact leg. fp4 speed path
+unchanged.

.agents/specs/minimax-h3.md

@@ -526,3 +526,42 @@ reduced-dim DiT gate into a GEOMETRY LADDER.
   embeddings, real fp64 position grid at full canvas, real per-token timesteps) is fed with
   RANDOM data here; a real-weights activation diff of the DiT inputs is the untested surface.
   Full tables: benchmark record (`row/H3-DIT-SCALE-GATE`).
+
+## 8.6 RENDER BUG CLOSED — wrong checkpoint PARTITION, not a code bug (2026-08-06, `row/H3-RENDER-CLOSE` PR #77)
+
+The #70/#74 white render was **using the wrong checkpoint partition for the task.**
+MiniMax-H3 ships two independently-served DiT partitions and the task MUST match
+(`recipes/MiniMaxAI/MiniMax-H3.md:50,289`; `pipeline._resolve_task` raises otherwise):
+
+| Partition | Serves | Available quantized DiT |
+|---|---|---|
+| **FL2VA** | **t2va + fl2va** | `MiniMax-H3-FL2VA-Q3_K_M.gguf` (GGUF), FL2VA NVFP4 (not downloaded) |
+| **Ref2VA** | ref2va (image/video + audio references) | `minimax_h3_ref2va_nvfp4_full` (the NVFP4 we had), REF2VA GGUF |
+
+Every render up to #74 ran **t2va on `minimax_h3_ref2va_nvfp4_full` (the Ref2VA
+partition)** — an out-of-distribution task/partition combination upstream rejects. That
+is the white latent, invariant to prompt/steps.
+
+**Verified before switching partitions (all NEW, real 512x512/22f scale, dgx):** the
+t2va DiT INPUTS diff EXACTLY vs upstream `pipeline_minimax_h3.py` (`VT_H3_DUMP_INPUTS`:
+packed layout / fp64 grid / token_tags / inverse+combined AdaLN indices / sigmas all
+byte-equal; tokenization byte-equal); the encoder conditioning is correctly shaped and
+carries the expected Qwen massive-activation; `DequantNvfp4ToBf16` is byte-exact
+(Laguna/Qwen3 + independent torch dequant); and the CUDA device forward == the CPU host
+forward at the REAL render seq (1920) at head_dim=128 (new permanent gate
+`test_minimax_h3 :: "CUDA device forward tracks the host at the REAL render seq (1920)"`,
+28/28) — closing the "CUDA kernel at scale" hole #74's CPU-backend device-vs-host left open.
+
+**Proof:** t2va on `MiniMax-H3-FL2VA-Q3_K_M.gguf` (`--dequant-bf16`, 512x512/22f, prompt
+"an orange cat sitting on a wooden table") renders a **COHERENT photorealistic orange cat
+on a wooden table** — VAE-input latent adj-cell cosine **0.9467** (white was 0.06), frame
+seam16/interior **1.00** (no patch grid), velocity stable ~1.37, final latent rms **1.00**.
+Valid h264 512x512 + AAC 32kHz mp4.
+
+**Fixed in this row:** `MiniMaxH3GenerateT2va` now strips the PREPENDED pinned reference
+rows (ref2va) before unpatchify/unpack — they are zeroed in the DiT output and only the
+trailing target rows are the clip; the old code fed unpatchify the full buffer and hit
+"rows not divisible by t*h*w" (no-op for t2va/fl2va). **Open:** a partition/supported_tasks
+guard mirroring upstream (community files strip the release config); the encoder vision
+tower (W3) is still unported, so image/video-conditioned ref2va/fl2va renders are not yet
+clean (ref2va with a synthetic reference + text-only encoder still grids).

.agents/state.md

@@ -39643,3 +39643,38 @@ is an irreducible-for-us ptxas quality gap. NO default flip owed; no functional
 code shipped (CMakeLists NOTE + benchmark-record #75 record the closed levers).
 Box left clean (GPU idle, both locks free, worker down). Evidence:
 `dgx:~/mxfp4-nsys/{ours,vllm,buildB,buildC}_flash_c8_ncu.ncu-rep`; PR #75.
+
+## MiniMax-H3 render bug CLOSED — wrong checkpoint PARTITION, not a code bug (`row/H3-RENDER-CLOSE` PR #77)
+<!-- state: 2026-08-06T23:55 -->
+
+The #70/#74 white render was **using the wrong partition for the task**, not a bug.
+MiniMax-H3 has two independently-served DiT partitions; the task MUST match (upstream
+`recipes/MiniMaxAI/MiniMax-H3.md:50,289` + `_resolve_task` raises): **FL2VA serves
+t2va+fl2va, Ref2VA serves ref2va.** Every render up to #74 ran **t2va on
+`minimax_h3_ref2va_nvfp4_full` (the REF2VA partition)** — out of distribution → the
+white latent, invariant to prompt/steps.
+
+BEFORE switching partitions I exonerated everything else (all NEW, dgx, real 512x512/22f):
+(1) `VT_H3_DUMP_INPUTS` — the t2va DiT step-0 inputs diff EXACTLY vs upstream
+`pipeline_minimax_h3.py` (packed layout, fp64 grid, token_tags, inverse/combined AdaLN
+indices, sigmas byte-equal; tokenization byte-equal). (2) encoder conditioning correctly
+shaped, carries the expected Qwen massive-activation. (3) `DequantNvfp4ToBf16` byte-exact
+(Laguna/Qwen3 + independent torch dequant). (4) NEW permanent gate `test_minimax_h3 ::
+"CUDA device forward tracks the host at the REAL render seq (1920)"` — CUDA device == CPU
+host at head_dim=128, seq 1920 (#74's device-vs-host only ran the CPU backend). (5) forward
+math == upstream source, read side by side.
+
+PROOF: downloaded `MiniMax-H3-FL2VA-Q3_K_M.gguf` (15.58 GB, `realrebelai/MiniMax-H3_GGUFs`,
+FL2VA partition, same geometry) and rendered t2va "an orange cat sitting on a wooden table"
+(`--dequant-bf16`, 512x512/22f, 20 steps) → a COHERENT photorealistic orange cat on a wooden
+table: VAE-input latent adj-cos **0.9467** (white=0.06), frame seam16/interior **1.00** (no
+patch grid), velocity stable ~1.37, final latent rms **1.00**, valid h264+AAC mp4. A 50-step
+768x1344 render was run as the artifact leg.
+
+FIXED (code): `MiniMaxH3GenerateT2va` now strips the PREPENDED pinned reference rows (ref2va)
+before unpatchify/unpack (zeroed in the DiT output; only the trailing target rows are the
+clip) — old code hit "rows not divisible by t*h*w"; no-op for t2va/fl2va. OPEN: a
+partition/supported_tasks guard mirroring upstream (community files strip the release config),
+and the encoder vision tower (W3) for clean image/video-conditioned ref2va/fl2va (ref2va with
+a synthetic reference + text-only encoder still grids). dgx assets: `~/h3fp4/ckpt/MiniMax-H3-
+FL2VA-Q3_K_M.gguf`, `~/h3fp4/fl2va_t2va_20/`. Box left clean.

docs/BENCHMARKS.md

@@ -302,7 +302,8 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) but frame is a non-scene patch-grid | Render coherence ROOT-CAUSED (#70): VAE fine, DiT latent white. Geometry ladder (PR #74) REFUTES a DiT-math bug: ours==oracle to 8x8+temporal; white=trained-wts. fp4 speed CLOSED. Detail: benchmark-record + spec §8 |
+| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. Detail: benchmark-record + spec §8 |
+| MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920; dequant byte-exact. spec §8.6 |
 | MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** (+951us). `FLASH-OCCUPANCY` #75: matched-c8 ncu, occupancy IDENTICAL 8.33%; built vLLM's exact flash recipe, matched reg+instr, STILL +10us, gap is ptxas SASS quality, no lever/flip |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |

docs/ENVIRONMENT.md

@@ -103,6 +103,7 @@ Read-only observability; none change output.
 | `VT_H3_TRACE_MOTION` | unset | `=1` prints one `[h3-motion] step ...` line per MiniMax-H3 denoise step to stderr: the step's velocity stats (`v_rms`/`v_amax`/`v_mean` of the DiT output), the per-step latent motion over the denoise-target rows (`drows_rms`), and the running latent norm (`rows_rms`). Because the rectified-flow Euler integration telescopes to `(sigma0 - sigmaN) * v`, a velocity that does not EVOLVE across steps produces a step-count-invariant result; this trace measures exactly that (added for the render-coherence bisection). Byte-identical when unset — every read is guarded and it only reads buffers the loop already holds |
 | `VT_H3_VAE_PROBE` | unset | `=1` runs a video-VAE receptive-field probe after the normal decode: it perturbs ONE interior spatial latent cell (across all channels and temporal frames), re-decodes, and prints a per-16px-block RMS-change map (`[h3-vae-probe]`) over output frame 0. If only the perturbed cell's block moves, the ViT3D decoder is not mixing tokens spatially. Byte-identical to production when unset (no second decode) |
 | `VT_H3_DUMP_DIR` | unset | Directory into which the MiniMax-H3 denoise loop writes the initial and final video latent rows (`init_video_rows.f32`, `final_video_rows.f32`) and the pipeline writes the exact VAE-input latent (`vae_input_video_latent.f32`), all raw little-endian f32. Lets two runs (e.g. 12 vs 50 steps, conditioned vs not) be byte/stat-compared, and the video VAE decode be replayed on a KNOWN latent, without re-running the denoise. Byte-identical to production when unset (no file is opened) |
+| `VT_H3_DUMP_INPUTS` | unset | Directory into which the MiniMax-H3 denoise loop writes EVERY DiT input at step 0 as raw little-endian binary plus a `manifest.txt` — the packed layout (`input_ids`/`image_mask`/`audio_mask`/`img_pos`/`audio_pos`/`text_pos`/`update_mask`/`cu_seqlens`/`document_id`), the fp64 position grid (`img_position_ids.f64`), the per-token modality tags (`token_tags.i64`), the per-token pre-unique timesteps and their `unique_timesteps`/`inverse_indices`/`combined_indices` AdaLN selection, both sigma schedules, and the raw `prompt_embeds`; the `minimax-h3-gen` driver additionally writes `prompt_token_ids.i32`. Lets the REAL-scale DiT inputs be diffed EXACTLY against upstream `pipeline_minimax_h3.py` (the render-coherence S1 surface the reduced-dim ladder never fed real values into). Byte-identical to production when unset (no file is opened) |
 
 ## Kernel-internal knobs (deferred)
 

docs/FEATURES.md

@@ -131,7 +131,7 @@ they sit outside the gated list above.
 |---|---|---|---|
 | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder TTFT ~17x, pending |
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
-| MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable path 65/65 (DiT geometry ladder 2x3->8x8+temporal, host+device vs oracle); real-weights render coherence OPEN (DiT-math bug REFUTED by the ladder, PR #74) | FP4/Marlin routing landed, GB10 speed pending |
+| MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable path 66/66 (DiT geometry ladder + CUDA-vs-host at the REAL render seq 1920); t2va renders a COHERENT prompt-matched scene on GB10 (render bug CLOSED: #70/#74 was wrong-partition usage, not a code bug) | FP4/Marlin routing landed, GB10 speed pending |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |
 | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending |
@@ -159,7 +159,7 @@ model architecture is wired.
 | Image | ✅ correctness-gated | ✅ | ✅ | ◐ |
 | Video | ✅ correctness-gated | ✅ | ✅ | ☐ |
 | Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
-| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ portable path complete; fp4-resident e2e RUNS on GB10 (real NVFP4 DiT + VAEs + GGUF encoder → mp4/wav); Marlin W4A16 byte-exact; render COHERENCE open, root-caused (#70) to the DiT (not the VAE) | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
+| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ t2va renders a COHERENT prompt-matched scene on GB10 (FL2VA-partition GGUF → h264/AAC mp4); render bug CLOSED (was wrong-partition usage); Marlin W4A16 byte-exact | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
 | Multimodal over the OpenAI server | ☐ | ✅ | ✅ | ◐ |
 
 Image, video and audio are correct through the CLI and library. Serving them

examples/minimax_h3_gen/main.cpp

@@ -487,6 +487,19 @@ int main(int argc, char** argv) {
         const std::vector<int32_t> ids = tokenizer.Encode(prompt);
         VT_CHECK(!ids.empty(), "minimax-h3-gen: the prompt tokenized to nothing");
         std::cerr << "  prompt tokens = " << ids.size() << "\n";
+        // DIAGNOSTIC (env-gated): dump the raw prompt token ids so the tokenization
+        // can be diffed against upstream `minimax_h3_text_only_ids` (verbatim prompt,
+        // add_special_tokens=False). A BOS/template mismatch shifts every text row and
+        // feeds the 32B tower a different string -> different conditioning.
+        if (const char* dd = std::getenv("VT_H3_DUMP_INPUTS")) {
+          if (std::FILE* fp = std::fopen((std::string(dd) + "/prompt_token_ids.i32").c_str(), "wb")) {
+            std::fwrite(ids.data(), sizeof(int32_t), ids.size(), fp);
+            std::fclose(fp);
+            std::cerr << "  [h3-dump-inputs] prompt_token_ids.i32 (" << ids.size() << " ids): ";
+            for (size_t k = 0; k < ids.size() && k < 64; ++k) std::cerr << ids[k] << " ";
+            std::cerr << "\n";
+          }
+        }
         const std::vector<float> embeds = vllm::MiniMaxH3EncoderEmbedTokens(enc, ids);
         // Text-only: all three M-RoPE axes are the token index.
         const int64_t seq = static_cast<int64_t>(ids.size());