row/H3-RENDER-COHERENCE: root-cause the H3 real-geometry render bug (non-scene patch-grid)

.agents/NOW.md

@@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | DeepSeek-V4-Flash decode | **Closed: beats ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact); phase-2 residency NEG, default-OFF | — |
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + AGENTS.md checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | **GB10 leg LANDED** (#64): Marlin W4A16 byte-exact, fp4=MEMORY win; e2e RUNS, frame=non-scene | **OPEN: render bug**; fp4 CLOSED |
+| MiniMax-H3 lane | **Render bug ROOT-CAUSED** (#70): VAE FINE, **DiT latent spatially WHITE** (0.06 vs 0.79) | Pin DiT line; fp4 CLOSED |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |

.agents/benchmark-record.md

@@ -13925,3 +13925,49 @@ and we are now at ~281 on `131,2048,512`, so we are **ahead of ggml's own
 kernel by ~1.3x**; ggml WITH llamafile is ~419, so ~1.5x still separates us and
 that residue is the FMA plus K-vectorised-hsum structure we are deliberately not
 adopting.
+
+## MiniMax-H3 render-coherence ROOT-CAUSED — VAE decoder is CORRECT (encode->decode round-trip is coherent); the DiT emits a spatially-WHITE latent at real geometry (2026-08-06, `row/H3-RENDER-COHERENCE` PR #70, `ROAD-V1-H3`, dgx GB10 sm_121a)
+
+**Setup.** dgx.casa GB10, dual-lock (`$HOME/gpu.lock`+`/tmp/gpu`), worker down,
+`drop_caches` before loads. Real ~39 GB NVFP4 arm cached at `~/h3fp4/ckpt`
+(DiT `minimax_h3_ref2va_nvfp4_full.safetensors` 18.75 GB + both VAEs + GGUF
+encoder). Diagnostics built on `7d05aee9`: env-gated `VT_H3_TRACE_MOTION`
+(per-step velocity + latent-motion), `VT_H3_DUMP_DIR` (raw f32 latents),
+`VT_H3_VAE_PROBE` (receptive field), `VT_H3_GAUSSIAN_NOISE`, and two driver
+modes `--decode-latent` / `--roundtrip`. Spatial coherence metric = mean cosine
+of adjacent latent-cell channel-vectors (16x16 grid) vs random-pair cosine.
+
+**Latent bisection (each rung MEASURED at 256x256/22f):**
+| Rung | Measurement | Verdict |
+|---|---|---|
+| Denoise loop | final latents byte-DIFFER at 3/12/50 steps (md5; s12-s50 corr 0.25); velocity 1.40->5.52 rms; disp-from-init ~1.9 | loop MOVES latent, not frozen -> not upstream-loop |
+| **VAE decoder** | `--roundtrip`: real test pattern encode->post_quant_conv->decode returns a COHERENT image (bars/timecode/diagonal), no grid | **DECODER CORRECT** (overturns #64 "device VAE decode" framing) |
+| DiT latent | adjacent-cos: encoded (coherent) **0.789** vs DiT **0.06**; laplacian 0.93 vs 4.2; 8x8-token adj-cos ~0 (== random) | DiT latent is spatially WHITE -> the grid |
+| fp4 vs bf16 | both white (cos 0.057 / 0.040); NOT byte-exact on real weights (max|diff| 11.2) | not the residency/precision path |
+| DiT attn kernel | MMA vs chunk (`VT_DFLASH_ATTN_MMA=0`): both white (0.057/0.056) | not the attention kernel |
+| VAE attn kernel | f32 chunk/warp/keylane identical (max|diff| 1/255) | not the VAE attention kernel |
+| init noise | Gaussian vs uniform: 0.077 vs 0.057 | marginal, not the fix |
+
+**Why the gates missed it.** DiT-forward gate runs latent 4x6 (spatial 2x3 = 6
+tokens) and matches upstream at **1.6e-7** there, so spatial mixing is correct
+at small scale; the divergence is real-geometry-specific, appearing only between
+2x3 and the real 8x8 (64 tokens) — same shape as the earlier temporal-chunking
+and VAE-tiling misses (every gate sits below the regime that breaks).
+
+**The mechanism of the grid.** The ViT3D decoder's `proj_out` maps EACH latent
+token to its own 16-px pixel patch; cross-patch coherence comes only from the
+latent's spatial structure. Given a real (spatially-coherent) latent it renders
+a coherent image (round-trip proves this); given the DiT's spatially-white
+latent it renders one independent smooth block per cell = the observed grid,
+identically at any step count (the loop only decides WHICH white latent).
+
+**Secondary real bug.** Driver seeded uniform[-1,1] init noise (std 0.577); a
+flow model needs Gaussian N(0,1) (torch.randn). `VT_H3_GAUSSIAN_NOISE=1` fixes
+INIT rms 0.58->1.0; correctness deviation, not the render fix.
+
+**Residual / handoff.** Bug RE-LOCALIZED to the DiT forward's spatial mixing at
+real geometry (video tokens not mixed). Exact line needs an upstream (vllm-omni)
+DiT-activation diff at real geometry — impractical on one GB10 (no quantized
+vllm-omni H3 arm; bf16 is 4xB300). Diagnostics committed on PR #70; latents at
+`dgx:~/h3fp4/diag`. fp4 speed path unchanged/CLOSED. No source/kernel/model/gate
+mark changed (all additions are env-gated diagnostics, byte-identical when off).

.agents/specs/minimax-h3.md

@@ -463,7 +463,20 @@ video flow_shift 12 / audio 3, no CFG; default canvas **768×1344**, default fra
   fp4-resident DiT → both VAEs → ffmpeg, producing a valid `h264 256×256 + AAC 32 kHz`
   mp4 + wav. **But the decoded frame is a structured multicolour patch-grid at the
   latent-cell scale, NOT a coherent scene — identically at 12/20/50 steps, conditioned
-  or not.** So the composed path is proven to RUN e2e on the real checkpoint, but a
+  or not.**
+  - **ROOT-CAUSED (2026-08-06, `row/H3-RENDER-COHERENCE` PR #70) by latent
+    bisection:** the VAE decoder is **CORRECT** — a real image encode→post_quant_conv→
+    decode round-trip (`--roundtrip`) returns a coherent frame — and the denoise loop
+    moves the latent step-dependently (byte-different finals at 3/12/50 steps). The bug
+    is the **DiT forward emitting a spatially-WHITE latent** at real geometry: adjacent
+    latent-cell cosine is **0.06** vs **0.789** for a real encoded latent, so every VAE
+    token decodes an independent patch = the grid. NOT fp4 (bf16 equally white), NOT the
+    attention kernel (MMA≡chunk, VAE chunk≡warp≡keylane), NOT the init noise. The DiT
+    gate runs spatial 2×3 (matches upstream 1.6e-7); the divergence is real-geometry
+    only (2×3→8×8). Secondary: driver used uniform init noise, not Gaussian
+    (`VT_H3_GAUSSIAN_NOISE`). Exact DiT line pends an upstream-oracle diff at real
+    geometry. See the benchmark record + state entry.
+  - So the composed path is proven to RUN e2e on the real checkpoint, but a
   coherent render is an OPEN bug (device video-VAE decode and/or denoise convergence
   at real geometry), independent of the fp4 speed work. **DiT s/step (full 50-layer
   fp4-resident, per forward):** 5.45 s @512×512/22f, 20.03 s @768×768/61f, 209.09 s

.agents/state.md

@@ -39404,3 +39404,59 @@ convergence at real geometry) — the frame-sanity gate caught it (all unit gate
 green + valid mp4, yet a non-scene). fp4 speed path is CLOSED. Box left clean (GPU
 idle, locks free, worker down, disk ≥15 G; checkpoint cached for reruns). Benchmark
 record + spec §8 + STATUS/BENCHMARKS/FEATURES + model-matrix/roadmap updated.
+
+## 2026-08-06T20:30 - H3 render-coherence ROOT-CAUSED by latent bisection: VAE is FINE (round-trip coherent), the DiT emits a spatially-WHITE latent at real geometry
+<!-- state: 2026-08-06T20:30 -->
+
+`row/H3-RENDER-COHERENCE` (helper, DRAFT PR #70, off `7d05aee9`). The #64 grid
+(regular 16-px multicolour blocks, IDENTICAL 12/20/50 steps, cond or not) is
+NOT the VAE and NOT the loop. Bisected the pipeline at the VAE boundary on the
+real ~39 GB NVFP4 arm (dgx GB10, dual-lock, worker down) with env-gated
+instrumentation (`VT_H3_TRACE_MOTION`, `VT_H3_DUMP_DIR`, `VT_H3_VAE_PROBE`) and
+two new driver diagnostics (`--decode-latent`, `--roundtrip`).
+
+**The bisection ladder (each rung a measurement, not an argument):**
+1. **Loop MOVES the latent, step-dependent -> upstream is NOT frozen.** Final
+   latents at 3/12/50 steps are byte-DIFFERENT (distinct md5, s12-vs-s50 corr
+   0.25); velocity evolves 1.40->5.52 rms as sigma 1->0; disp-from-init ~1.9.
+   The Euler integral telescopes to `(sig0-sigN)*v ~= v`, so "identical across
+   steps" would have meant a frozen velocity; it is not frozen.
+2. **The VAE DECODER IS CORRECT (the decisive rung).** `--roundtrip` encodes a
+   real test pattern through the video-VAE encoder, applies `post_quant_conv`,
+   and decodes: the frame comes back COHERENT (same colour bars, timecode,
+   diagonal), no grid. So ViT3D decoder + post_quant_conv + temporal decode +
+   pixel-denormalize all work on an in-distribution latent. This OVERTURNS the
+   #64 "device VAE decode and/or denoise convergence" framing.
+3. **The DiT emits a spatially-WHITE latent.** Adjacent-cell cosine at the 16x16
+   VAE-token scale: a real ENCODED latent (coherent decode) = **0.789**
+   (laplacian 0.93); the DiT-produced latent = **0.06** (laplacian 4.2). At the
+   8x8 DiT-patch scale adjacent tokens are as uncorrelated as random pairs. The
+   VAE faithfully renders that white latent as one independent 16-px patch per
+   token (`proj_out` maps each token to its own patch), which IS the grid.
+4. **Not fp4, not the attention kernel, not the noise.** fp4-resident vs bf16:
+   BOTH white (cos 0.057 vs 0.040), so it is not the residency/precision path
+   (they are NOT byte-exact on real weights, max|diff| 11.2, but both wrong).
+   DiT bf16 MMA kernel vs the CUDA-core chunk kernel (`VT_DFLASH_ATTN_MMA=0`):
+   BOTH white (0.057 vs 0.056). VAE f32 chunk/warp/keylane kernels: identical
+   (max|diff| 1.0/255). Gaussian vs uniform init noise: 0.077 vs 0.057 (marginal).
+5. **The gate blind spot.** The DiT forward gate runs at latent 4x6 (spatial
+   2x3 = 6 tokens) and matches upstream at 1.6e-7 there, so spatial mixing IS
+   exercised at small scale and is correct; the divergence appears only between
+   2x3 and the real 8x8 (64 tokens). A real-geometry-scaling divergence in the
+   DiT forward's spatial mixing -- the same shape as the temporal-chunking miss
+   (every gate below one chunk/tile) and the VAE-tiling miss.
+
+**Secondary real bug found:** the driver seeded uniform[-1,1] init noise
+(std 0.577); a flow model wants Gaussian N(0,1) (torch.randn). `VT_H3_GAUSSIAN_NOISE=1`
+fixes the distribution (INIT rms 0.58->1.0) but not the coherence. The "noise
+distribution doesn't matter" comment was wrong (only the RNG identity doesn't).
+
+**Verdict / handoff.** Render bug RE-LOCALIZED: **VAE exonerated, bug is in the
+DiT forward's spatial mixing at real geometry** (video tokens not spatially
+mixed). Exact line needs an upstream-oracle diff at real geometry (vllm-omni
+DiT activations), which is impractical on one GB10 (the quantized arm does not
+run in vllm-omni; bf16 is 4xB300). Diagnostics + `--decode-latent`/`--roundtrip`
+are committed on the row for the next session. fp4 speed path stays CLOSED.
+Box left clean (GPU idle, both locks free, worker down, ~39 G ckpt cached at
+`dgx:~/h3fp4/ckpt`, diagnostic latents at `dgx:~/h3fp4/diag`). Evidence:
+`dgx:~/h3fp4/{diag,rt_out,out_small,out_cond}`; PR #70.

docs/BENCHMARKS.md

@@ -299,7 +299,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) but frame is a non-scene patch-grid (OPEN bug) | Root-cause H3 render coherence (VAE/denoise); fp4 speed CLOSED. Detail: benchmark-record + spec §8 |
+| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) but frame is a non-scene patch-grid | Render coherence ROOT-CAUSED (#70): VAE fine, DiT latent spatially white. fp4 speed CLOSED. Detail: benchmark-record + spec §8 |
 | MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6; -Werror test-guard fixes x2 | **VT_MARLIN_DENSE default-ON** banks +951us marlin. `FUSED-GLUE` W0: glue-into-marlin REFUTED (vLLM doesn't fuse into extern marlin); c8 residual FLASH-dominant, not glue. Flash same-tool audit OWED; see record |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |

docs/ENVIRONMENT.md

@@ -99,6 +99,10 @@ Read-only observability; none change output.
 | `VT_POOL_BYPASS` | off | `=1` makes every device-scratch pool allocation an exact-size driver `Alloc` and every release a real `Free`, so `compute-sanitizer` can see tensor boundaries and use-after-free that the caching, size-class-rounding pool hides. DEBUGGING ONLY: it reinstates the per-op `cudaMalloc`/`cudaFree` device-sync storm the pool exists to remove, so it is never a timing configuration |
 | `VT_TTFT_DUMP` | unset | `=1` prints one `TTFTSPLIT rid=... intake=.. queued=.. prefill=.. decode=.. e2e=..` line per finished request to stderr, reconstructing the per-request timing split from the event-populated `req_state` timestamps. The async serving frontend otherwise tracks no per-request stats (passes `iteration_stats=nullptr` and never stamps `EngineCoreOutputs.timestamp`); under this flag both are wired so a serving TTFT attribution can read the queue-vs-execution split against vLLM's own `request_{queue,prefill,decode}_time_seconds`. Generation is byte-identical when unset (the default path is instruction-identical to production); the durable replacement is the async `/metrics` stat logger |
 | `VT_LOOP_TRACE` | unset | `=1` prints one `LOOPTRACE ...` line per ~1 s window to stderr from the engine busy loop: the full-iteration cadence (`interval`), `process_engine_step` wall (`step`), per-window admits, input-queue residence (`resid` = enqueue-to-drain, the same endpoints `VT_TTFT_DUMP`'s `intake` measures), per-drain admit max and max backlog depth. Diagnoses whether the admission wait is one busy-loop iteration or the input queue is backing up (it attributed the 35B INTAKE deficit to bursty arrival during long prefill steps). Byte-identical when unset: every trace read is guarded, and the enqueue timestamp is stamped only under the flag |
+| `VT_H3_GAUSSIAN_NOISE` | unset | `minimax-h3-gen` only: `=1` seeds the diffusion initial noise from Box-Muller GAUSSIAN N(0,1) (what a flow-matching model is trained on) instead of the historical uniform[-1,1] draw. A/B knob for the render-coherence investigation; the exact values still do not match torch's RNG (that only selects WHICH sample), but the DISTRIBUTION does |
+| `VT_H3_TRACE_MOTION` | unset | `=1` prints one `[h3-motion] step ...` line per MiniMax-H3 denoise step to stderr: the step's velocity stats (`v_rms`/`v_amax`/`v_mean` of the DiT output), the per-step latent motion over the denoise-target rows (`drows_rms`), and the running latent norm (`rows_rms`). Because the rectified-flow Euler integration telescopes to `(sigma0 - sigmaN) * v`, a velocity that does not EVOLVE across steps produces a step-count-invariant result; this trace measures exactly that (added for the render-coherence bisection). Byte-identical when unset — every read is guarded and it only reads buffers the loop already holds |
+| `VT_H3_VAE_PROBE` | unset | `=1` runs a video-VAE receptive-field probe after the normal decode: it perturbs ONE interior spatial latent cell (across all channels and temporal frames), re-decodes, and prints a per-16px-block RMS-change map (`[h3-vae-probe]`) over output frame 0. If only the perturbed cell's block moves, the ViT3D decoder is not mixing tokens spatially. Byte-identical to production when unset (no second decode) |
+| `VT_H3_DUMP_DIR` | unset | Directory into which the MiniMax-H3 denoise loop writes the initial and final video latent rows (`init_video_rows.f32`, `final_video_rows.f32`) and the pipeline writes the exact VAE-input latent (`vae_input_video_latent.f32`), all raw little-endian f32. Lets two runs (e.g. 12 vs 50 steps, conditioned vs not) be byte/stat-compared, and the video VAE decode be replayed on a KNOWN latent, without re-running the denoise. Byte-identical to production when unset (no file is opened) |
 
 ## Kernel-internal knobs (deferred)
 

docs/FEATURES.md

@@ -115,7 +115,7 @@ oversight.
 | Image | ✅ correctness-gated | ✅ | ✅ | ◐ |
 | Video | ✅ correctness-gated | ✅ | ✅ | ☐ |
 | Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
-| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ portable path complete; fp4-resident e2e RUNS on GB10 (real 18.75 GB NVFP4 DiT + VAEs + GGUF encoder → valid mp4/wav); Marlin W4A16 GB10-verified byte-exact; render COHERENCE open (non-scene patch-grid) | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
+| Video+audio GENERATION (MiniMax-H3 DiT, vLLM-Omni lane) | ◐ portable path complete; fp4-resident e2e RUNS on GB10 (real NVFP4 DiT + VAEs + GGUF encoder → mp4/wav); Marlin W4A16 byte-exact; render COHERENCE open, root-caused (#70) to the DiT (not the VAE) | ✅ (vllm-omni, BF16-only, no quantized H3 arm) | ☐ | ☐ |
 | Multimodal over the OpenAI server | ☐ | ✅ | ✅ | ◐ |
 
 Image, video and audio are correct through the CLI and library. Serving them