row/H3-VISION-SCATTER: scatter vision tower into DiT conditioning (deepstack + merged scatter)

.agents/NOW.md

@@ -1,6 +1,6 @@
 # NOW — the one-Read resume surface
 
-<!-- now-updated: 2026-08-06 -->
+<!-- now-updated: 2026-08-07 -->
 
 Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live,
 the gate being chased, what to do next. Never a log — evidence lives in the
@@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | DeepSeek-V4-Flash decode | **Closed: beats ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact); phase-2 residency NEG, default-OFF | — |
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + AGENTS.md checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | **vision tower RUNS on real weights + fl2va COHERENT** (`H3-CONDITIONED-E2E` PR#86: `visual.*` loader+probe; fl2va matched cat; ref2va grids) | scatter feats→embeds + DeepStack→text tower |
+| MiniMax-H3 lane | **vision→cond scatter WIRED+gated; fl2va COHERENT via encoder path** (`H3-VISION-SCATTER` PR#90; indices {8,16,24} confirmed) | ref2va grids; residual = ref2va ref-row assembly (§8.9) |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |

.agents/benchmark-record.md

@@ -14310,3 +14310,55 @@ UNRELATED CUDA case (line 3503, "an NVFP4 checkpoint loads into a runnable DiT")
 isolation (585 assertions) and runs BEFORE the new case — a pre-existing cross-test CUDA
 resource-accumulation flake, not this change; the new loader gate passes standalone (59
 assertions).
+
+## MiniMax-H3 ENCODER VISION SCATTER + ref2va re-attribution (`row/H3-VISION-SCATTER` PR #90, 2026-08-07, dgx sm_121a)
+
+Closes the #86 residual at the framework level and RE-ATTRIBUTES the ref2va grid with a GB10
+render A/B. Builds on #86 (real `visual.*` tower LOADS + fl2va COHERENT via VAE-keyframe).
+
+**`deepstack_visual_indexes` CONFIRMED (was #86-inferred).** `{8, 16, 24}` grounded in the
+release config: MiniMax-H3's `text_encoder/` IS Qwen3-VL-32B-Instruct
+(HF DeepBeepMeep/MiniMax-H3/Qwen3-VL-32B-Instruct/config.json), whose
+`vision_config.deepstack_visual_indexes = [8, 16, 24]`, depth 27, text `num_hidden_layers = 64`
+(→ min(64,50)=50). Identical to vllm-omni `Qwen3VLMoeVisionConfig` default and public
+`Qwen/Qwen3-VL-30B-A3B-Instruct`. The #86 inference was right; no code value change.
+
+**Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED.** `MiniMaxH3EncoderTextForwardDevice`
+now takes the optional `visual_pos_mask` + per-tap `deepstack` blocks and ADDS each block into the
+masked visual rows after each of the first `len(deepstack)` decoder layers — device mirror of the
+gated host reference and upstream `_deepstack_process` (encoder.py:770-800). The merged masked_scatter
+into inputs_embeds stays the caller's job (upstream `_encode`, encoder.py:1071). Gate
+(`test_minimax_h3 :: "the DEVICE keep-quant encoder matches the host f32 reference"`): device forward
+now also runs WITH a visual mask + 2 DeepStack blocks → device==host max|diff| **3.8e-4** (≤ 2e-3)
+AND DeepStack moves the conditioning (scale 1.006→1.062). PASS, 1562 assertions. Host text tower,
+full vision tower, GGUF visual.* loader, MM processor all green. The full-suite SIGSEGV at line 3503
+(NVFP4 case) is the known cross-test CUDA flake — passes standalone (585 assertions), not this change.
+
+**Driver `--cond-image`** routes a reference image through the encoder vision path (reuse only:
+Qwen3VLImageProcessor → Qwen3VLVisionForward → merged+3 DeepStack; ExpandImagePlaceholders inserts nm
+image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex == H3 `_get_rope_index` for t==1). Additive.
+
+**GB10 render A/B (256×256/22f/12steps).**
+- **Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS).** FL2VA GGUF
+  `--dequant-bf16` + `--first-frame` + `--cond-image` + `--partition fl2va`, prompt "a fluffy orange
+  cat sitting on a windowsill in warm sunlight". conditioning=[82,5120] (16 prompt + 66-token vision
+  block: 64 merged + 2 markers). Frame 0 = coherent orange cat (keyframe); frame 21 = the cat on a
+  WINDOWSILL in warm sunlight — the clip EVOLVED toward the prompt. No grid. Artifact
+  `~/h3fp4/out_vs_fl2va.mp4` (+ out_vs_fl2va/frame_*.ppm). The vision conditioning is SOUND.
+- **Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL).** Ref2VA NVFP4
+  `--fp4-resident` + `--ref-image` (VAE reference rows) + `--cond-image` + `--partition ref2va`, same
+  prompt. conditioning=[82,5120] (64 merged + 3 DeepStack), 1 reference image, latent 7×16×16. Every
+  frame (0/10/21) is the same multicolour PATCH GRID as #86's text-only ref2va. Artifact
+  `~/h3fp4/out_vs_ref2va.mp4`.
+
+**RE-ATTRIBUTION (with evidence).** The "vision-enriched conditioning fixes the grid" hypothesis is
+REFUTED. The ref2va grid is NOT the encoder conditioning: (a) the DiT forward MATH is byte-exact vs
+upstream (geometry ladder green every rung #74; device==host at real seq 1920 #77); (b) the SAME
+vision scatter+DeepStack path renders a COHERENT fl2va-with-`--cond-image`; (c) the grid is INVARIANT
+to text-only (#86) vs vision-enriched prompts. The only difference between coherent fl2va and gridding
+ref2va: fl2va PINS output rows (keyframe cond rows) each step, ref2va PREPENDS free-running reference
+rows. So the residual is the **ref2va reference-row conditioning ASSEMBLY**
+(`MiniMaxH3EncodeReferenceImages` VAE-reference rows + `minimax_h3_packed_sequence_ref2va_blocks`
+noised-anchor layout + how the denoise loop conditions the un-pinned target rows), NOT the prompt and
+NOT the DiT forward. Next diagnostic: dump the ref2va target-row VAE-input adjacency-cosine (like #77's
+0.95 for coherent fl2va) to confirm the target rows are white, and A/B the reference-row condition-noise.

.agents/specs/minimax-h3.md

@@ -400,7 +400,7 @@ vLLM-Omni H3 modules at `vllm_omni/diffusion/models/minimax_h3/`; serving in
 | WebSocket `/v1/video/chat/stream`, `/v1/realtime/video` | `api_server.py:1593,1610` | — | **MISSING** (streaming/realtime) |
 | Request schema (prompt, size/w/h, num_frames, fps, seed, steps, refs) | `protocol/videos.py:97-249` | request contract (W7) | **PARTIAL** (core fields; frame-interp/lora/generate_sound absent) |
 | H3 knobs via `extra_params.{task,duration,flow_shift,audio_flow_shift}` | `pipeline:1034,403,1157-1158` | planner reads task/duration/shift | **DONE** |
-| Modalities in: text/image/video/audio | `pipeline:1036-1104` | t2va (text) done; fl2va COHERENT on real weights (VAE-keyframe); vision tower now LOADS real `visual.*` + runs (probe); ref2va still grids | **PARTIAL** (vision-tower→DiT-conditioning scatter is the residual; §8.8) |
+| Modalities in: text/image/video/audio | `pipeline:1036-1104` | t2va (text) done; vision tower LOADS real `visual.*` + runs; merged→prompt_embeds scatter + DeepStack→device text tower WIRED 1:1 + gated (§8.9); fl2va COHERENT via BOTH the VAE-keyframe AND the encoder vision path; ref2va still grids (residual RE-ATTRIBUTED to the ref2va reference-row assembly, NOT the conditioning) | **PARTIAL** (vision→conditioning scatter DONE §8.9; residual = ref2va reference-row assembly) |
 | Output: joint video+audio, 24 fps, 32 kHz stereo | `pipeline:106-111,1187` | frames + WAV + MP4 mux (W7) | **DONE** |
 | Scheduler: euler-ancestral rectified flow (single) | `scheduling_...euler_ancestral.py`; `time_request.py:34-61` | `MiniMaxH3EulerEta0Step` / `MiniMaxH3TimeShiftSigmas` | **DONE** |
 | CFG: distilled, no CFG (guidance params accepted+ignored; `cfg_parallel_size==1`) | `pipeline:250,275-276` | no CFG branch | **DONE** (matches) |
@@ -709,3 +709,68 @@ extension; its e2e render verdict is recorded honestly in the benchmark record.
   DEVICE text tower) is the tracked residual that would let the vision-enriched-prompt
   hypothesis be tested. The `--ref-video` VAE encode is a slow single-thread CPU 3D-CNN path
   (separate perf limit).
+
+## 8.9 ENCODER VISION SCATTER — merged→prompt_embeds + DeepStack→device text tower (2026-08-07, `row/H3-VISION-SCATTER` PR #90)
+
+Closes the §8.8 residual at the FRAMEWORK level and RE-ATTRIBUTES the ref2va grid with a
+render A/B. Three deliverables.
+
+**`deepstack_visual_indexes` CONFIRMED (was #86-inferred).** The value is `[8, 16, 24]`,
+grounded in the release config: MiniMax-H3's `text_encoder/` IS **Qwen3-VL-32B-Instruct**
+(HF `.../MiniMax-H3/.../Qwen3-VL-32B-Instruct/config.json`), whose
+`vision_config.deepstack_visual_indexes = [8, 16, 24]`, depth 27, text `num_hidden_layers = 64`
+(truncated to 50) — identical to vllm-omni's `Qwen3VLMoeVisionConfig` default and the public
+`Qwen/Qwen3-VL-30B-A3B-Instruct` config. The #86 inference was correct; comment updated in
+`minimax_h3_vision_gguf.cpp:46-52`.
+
+**Deliverable 1 — the DEVICE scatter+inject is WIRED 1:1 + GATED.** `MiniMaxH3EncoderTextForwardDevice`
+(`minimax_h3_encoder_device.cpp:103,216-243`) now takes the optional `visual_pos_mask` + per-tap
+`deepstack` blocks and, after each of the first `len(deepstack)` decoder layers, ADDS each block
+into the masked visual-token rows — the device mirror of the gated host reference and of upstream
+`MiniMaxH3Qwen3VLTextModel._deepstack_process` (`encoder.py:770-800`,
+`hidden_states[visual_pos_masks] += visual_embeds`). The MERGED-feature masked_scatter into
+`inputs_embeds` stays the caller's job (upstream `_encode` scatters it BEFORE the tower runs;
+`encoder.py:1071`), exactly like the host reference. Text-only prompts pass the defaults and are
+byte-identical. **Gate** (`test_minimax_h3.cpp :: "the DEVICE keep-quant encoder matches the host
+f32 reference"`): the device forward now also runs WITH a visual mask + two DeepStack blocks and
+checks device==host-reference (max|diff| **3.8e-4** ≤ 2e-3) AND that DeepStack MOVES the
+conditioning (scale 1.006→1.062) — the surface #86 could not cover. All encoder/vision gates green
+(host text tower + full vision tower + GGUF `visual.*` loader + MM processor).
+
+**Driver wiring — `--cond-image` routes a reference image through the ENCODER vision path**
+(`examples/minimax_h3_gen/main.cpp`, mirroring `_encode`). Reuse-only: `Qwen3VLImageProcessor` →
+`Qwen3VLVisionForward` (real `visual.*` tower) → merged `[nm,5120]` + 3 DeepStack blocks;
+`ExpandImagePlaceholders` inserts `nm` image-pad tokens; merged masked_scatter into the embeds at
+those rows; M-RoPE positions from `Qwen3VLGetRopeIndex` (byte-equivalent to H3's own
+`_get_rope_index` for a single-frame image, t==1 — position math verified: text sequential, image
+block the 3D grid, next-text advances by `max(llm_h,llm_w)`). Additive: without `--cond-image` the
+text-only path is byte-identical.
+
+**GB10 RENDER A/B (2026-08-07, dgx sm_121a, 256×256/22f/12steps).**
+- **Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS).** FL2VA GGUF
+  (`--dequant-bf16`) + `--first-frame` (VAE-keyframe) + **`--cond-image`** (encoder vision) +
+  `--partition fl2va`, prompt "a fluffy orange cat sitting on a windowsill in warm sunlight".
+  Conditioning `[82,5120]` = 16 prompt + a 66-token vision block (64 merged image-pad rows + 2
+  markers). Frame 0 = a coherent photorealistic ORANGE CAT matching the keyframe; frame 21 = the
+  same cat on a **WINDOWSILL in warm sunlight** — the clip EVOLVED toward the text prompt. No grid.
+  The vision-enriched conditioning is SOUND and load-bearing. Artifact `~/h3fp4/out_vs_fl2va.mp4`.
+- **Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL).** Ref2VA NVFP4
+  (`--fp4-resident`) + `--ref-image` (VAE reference rows) + **`--cond-image`** (encoder vision) +
+  `--partition ref2va`, same prompt. Conditioning `[82,5120]` (64 merged + 3 DeepStack), 1 reference
+  image, latent 7×16×16. Every frame (0/10/21) is the same multicolour PATCH GRID as #86's
+  text-only ref2va. Artifact `~/h3fp4/out_vs_ref2va.mp4`.
+
+**RE-ATTRIBUTION (with evidence).** The mission's "vision-enriched conditioning fixes the grid"
+hypothesis is **REFUTED**. The ref2va grid is NOT the encoder conditioning: (a) the DiT forward MATH
+is byte-exact vs upstream (§8.5 geometry ladder green every rung; §8.6 device==host at real seq
+1920); (b) the vision scatter+DeepStack is proven sound by the COHERENT fl2va-with-`--cond-image`
+render — the SAME conditioning path; (c) the ref2va grid is INVARIANT to text-only (#86) vs
+vision-enriched (this row) prompts. The ONLY thing that differs between the coherent fl2va and the
+gridding ref2va is that **fl2va PINS output rows (keyframe cond rows) each denoise step** while
+**ref2va PREPENDS free-running reference rows** — so the residual is the **ref2va-specific
+reference-row conditioning ASSEMBLY** (`MiniMaxH3EncodeReferenceImages` VAE-reference rows +
+`minimax_h3_packed_sequence_ref2va_blocks` noised-anchor layout + how the denoise loop conditions
+the un-pinned target rows on them), NOT the prompt_embeds and NOT the DiT forward. Next diagnostic:
+dump the ref2va target-row VAE-input latent adjacency-cosine (like #77 did for the coherent fl2va,
+0.95) to confirm the target rows are white, and A/B the reference-row condition-noise vs a clean
+anchor.

.agents/state.md

@@ -40259,3 +40259,48 @@ kept). Records: spec §8.8 + §8.2, STATUS/BENCHMARKS/FEATURES, benchmark-record
   Note also that `agent-record` was ALREADY failing on `main` at `4cfeee13`
   before this merge (`check-fusion-consistency`, `minimax_h3_video_vae_device`),
   verified on `origin/main` itself. That one is not from this work.
+
+## 2026-08-07T04:10 - MiniMax-H3 ENCODER VISION SCATTER wired+gated; fl2va COHERENT via encoder vision path; ref2va grid RE-ATTRIBUTED to the reference-row assembly (row/H3-VISION-SCATTER, helper, PR #90)
+<!-- state: 2026-08-07T04:10 -->
+
+Closes the #86 framework residual and re-attributes the ref2va grid with a GB10 render A/B.
+
+**deepstack_visual_indexes CONFIRMED (was #86-inferred).** `{8,16,24}` grounded in the release
+config: H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose vision_config.deepstack_visual_indexes =
+[8,16,24], depth 27, text num_hidden_layers 64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig
+default and public Qwen/Qwen3-VL-30B-A3B. No value change; comment updated (minimax_h3_vision_gguf.cpp).
+
+**Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED.** MiniMaxH3EncoderTextForwardDevice
+(minimax_h3_encoder_device.cpp:103,216-243) now takes visual_pos_mask + per-tap deepstack blocks and
+ADDS each block into the masked visual rows after each of the first N decoder layers — device mirror of
+the gated host reference and upstream _deepstack_process (encoder.py:770-800). Merged masked_scatter
+into inputs_embeds stays the caller's job (upstream _encode). Text-only byte-identical (defaults).
+Gate: device keep-quant encoder test now runs WITH a visual mask + 2 DeepStack blocks → device==host
+max|diff| 3.8e-4 (≤2e-3), DeepStack moves conditioning (scale 1.006→1.062); PASS 1562 assertions.
+Host text tower + full vision tower + GGUF visual.* loader + MM processor all green. NVFP4 line-3503
+SIGSEGV is the known standalone-passing cross-test flake, not mine.
+
+**Driver --cond-image** routes an image through the encoder vision path (reuse: Qwen3VLImageProcessor →
+Qwen3VLVisionForward → merged+3 deepstack; ExpandImagePlaceholders; masked_scatter; Qwen3VLGetRopeIndex
+== H3 _get_rope_index for t==1, position math verified). Additive.
+
+**GB10 render A/B (256x256/22f/12steps).**
+- fl2va + --cond-image = COHERENT + matching (deliverable 3 PASS): FL2VA GGUF --dequant-bf16 +
+  --first-frame + --cond-image + --partition fl2va, conditioning [82,5120] (64 merged + 3 deepstack).
+  Frame 0 = coherent orange cat (keyframe); frame 21 = cat on a WINDOWSILL in warm sunlight (evolved
+  toward the prompt). No grid. Artifact ~/h3fp4/out_vs_fl2va.mp4. Vision conditioning is SOUND.
+- ref2va + --cond-image STILL GRIDS (deliverable 2, honest FAIL): Ref2VA NVFP4 --fp4-resident +
+  --ref-image + --cond-image + --partition ref2va, conditioning [82,5120], latent 7x16x16. Every frame
+  the same multicolour patch grid as #86. Artifact ~/h3fp4/out_vs_ref2va.mp4.
+
+**RE-ATTRIBUTION (evidence).** The "vision-enriched conditioning fixes the grid" hypothesis is REFUTED.
+NOT the encoder conditioning: (a) DiT forward math byte-exact (geometry ladder #74, device==host seq
+1920 #77); (b) SAME vision scatter path renders a COHERENT fl2va; (c) grid invariant to text-only vs
+vision-enriched prompt. Only difference: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running
+reference rows. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages
+VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row
+denoise), NOT prompt_embeds and NOT the DiT forward. Next: dump ref2va target-row VAE-input adjacency
+cosine (vs #77's 0.95 for coherent fl2va) + A/B the reference-row condition-noise.
+
+Box left clean (renders exited, GPU idle, gpu.lock released, worker stays parked, temp PNGs pruned,
+ckpts kept). Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW.