mudler · localai-bot · Aug 7, 2026 · Aug 7, 2026 · Aug 7, 2026 · Aug 7, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -18,7 +18,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | DeepSeek-V4-Flash decode | **Closed: beats ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact); phase-2 residency NEG, default-OFF | — |
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + AGENTS.md checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | **RENDER + task/partition guard CLOSED** (#77 render; `H3-TASK-PARTITION-GUARD`: t2va-on-ref2va now RAISES like `_resolve_task`) | Follow-up: vision tower (W3) |
+| MiniMax-H3 lane | **vision tower RUNS on real weights + fl2va COHERENT** (`H3-CONDITIONED-E2E` PR#86: `visual.*` loader+probe; fl2va matched cat; ref2va grids) | scatter feats→embeds + DeepStack→text tower |
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -14239,3 +14239,74 @@ GREEN. Suite **67/67** (66 prior + this), 46549 assertions; `test_video_api` 4/4
 wiring). Wired at both loading entry points: `MiniMaxH3GenerateT2va` guards every full
 render; the pure pipeline-math tests build `declared=false` requests so the guard is inert
 there. No numbers changed — this is a correctness/refusal gate, not a perf lever.
+
+## MiniMax-H3 ENCODER VISION TOWER — record reconciled + real-weights loader; the encoder GGUF DOES carry the vision tensors (2026-08-07, `row/H3-CONDITIONED-E2E`, `ROAD-V1-H3`, dgx GB10 sm_121a)
+
+**Record reconciliation.** #26/W3 recorded the vision tower "COMPLETE … only the MM
+processor remains"; #77 recorded it "still unported". Reading the code, both are true of
+different halves: the tower MATH exists as a CPU scalar reference
+(`minimax_h3_encoder.cpp` `MiniMaxH3VisionTowerForward:572` + surround) gated ONLY at
+reduced dims with SYNTHETIC weights (`test_minimax_h3.cpp:4041`), but it was NEVER wired to
+real weights — `LoadMiniMaxH3EncoderFromGguf` (`minimax_h3_encoder_gguf.cpp:47`) loads the
+TEXT tower only and skips every `visual.*` tensor; the device forward is text-only; the
+driver/server call only the text path. So: math CPU-gated synthetic, zero real-weights
+wiring.
+
+**Encoder-arm decision (measured, no download).** The on-box encoder
+`~/h3fp4/ckpt/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf` (14 GiB) carries the FULL vision tower —
+`visual.blocks.{0..26}` (Q4_K/Q5_K), `patch_embed.proj` (F16 `[16,16,6,1152]`), `pos_embed`
+(F16 `[2304,1152]`=48²), `merger.*`, and 3 `deepstack_merger_list.*`. Geometry: hidden 1152
+/ 16 heads / depth 27 / intermediate 4304 / out 5120 / patch 16 / temporal 2 / merge 2 /
+pos 2304 (== the Qwen3.6-27B vision config, state.md :23310). ComfyUI quant reshapes
+non-256-aligned rows (in=1152) to ne0=256; dequant of the flat buffer preserves the
+row-major `[out,in]` order the tower reads. `deepstack_visual_indexes` is NOT in the
+weights-only GGUF and must be supplied from the H3 vision_config (residual for a bit-correct
+DeepStack inject).
+
+**Landing this row:** GGUF `visual.*`→`Qwen3VLVisionWeights` loader (`LoadQwen3VLVisionFromGguf`,
+mirrors safetensors `LoadQwen3VLVisionWeights` `qwen3_vl.cpp:417`) + a CPU/real-weights
+vision-tower forward gate + driver `--prompt-image` probe. HONEST residual (recorded, not
+claimed done): the vision-ENRICHED DiT render (DeepStack scatter into the DEVICE text tower
+actually changing the frames) needs the exact deepstack indexes + a device-text
+DeepStack/merge extension + an on-box GPU render; fl2va/ref2va frame-sanity verdicts recorded
+here as they are produced.
+
+### GB10 VERIFICATION (2026-08-07, dgx sm_121a, flock, worker parked)
+
+**Real-weights vision-tower probe RAN (deliverable 1).** `minimax-h3-gen --device cuda
+--encoder qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf --prompt-image <512x512 cat frame>`: the loader
+read the real `visual.*` tower (depth 27 / hidden 1152 / 16 heads / out 5120 / 3 DeepStack
+mergers); the stock Qwen3VLImageProcessor produced grid_thw [1,32,32] = 1024 tokens → 256
+merged; `Qwen3VLVisionForward` on the GGUF-loaded weights returned [256, 20480] = merged +
+3 deepstack concatenated, ALL FINITE and non-degenerate: merged rms 1.45 / maxabs 29.1 (the
+expected Qwen MASSIVE-ACTIVATION signature), deepstack rms 0.54/1.04/0.51. The vision tower now
+runs on real weights.
+
+**fl2va e2e — COHERENT (deliverable 2, frame-sanity PASS).** FL2VA-partition GGUF
+(`MiniMax-H3-FL2VA-Q3_K_M.gguf`, `--dequant-bf16`, ~66 GB device, free 117→23 G, no OOM) +
+`--first-frame` (real coherent orange-cat frame, VAE-keyframe conditioning) + `--partition
+fl2va`, 512x512/22f/12steps. Valid h264 512x512 + AAC mp4, 22 frames. VISUAL sanity (frames
+0/6/12/21 inspected): every frame is a coherent photorealistic ORANGE CAT sitting on a WOODEN
+TABLE, matching the conditioning first-frame's pose/table/gray-background, consistent across
+the whole clip — no patch-grid, no white latent. The render visibly CONTINUES/MATCHES the
+conditioning image. (This uses the VAE-KEYFRAME path, not the vision tower.)
+
+**ref2va e2e — STILL GRIDS (deliverable 2, frame-sanity FAIL, recorded honestly).** Ref2VA
+NVFP4 (`minimax_h3_ref2va_nvfp4_full`, `--fp4-resident`, ~16 GB, free 117→63 G) + `--ref-image`
+(one 256x256 real cat image, VAE-reference conditioning) + `--partition ref2va` + text
+prompt_embeds, 256x256/22f/12steps. Valid mp4, 22 frames, but EVERY frame (0/10/21) is a
+multicolour PATCH GRID — the #70/#74 degenerate render — NOT a coherent scene. First attempt
+(5x512 `--ref-video`) was ABANDONED: the reference-video VAE encode is a single-thread CPU
+3D-CNN scalar path (GPU 0%, 98% CPU) and did not finish in ~15 min — a known perf limit, not a
+bug. **HONEST verdict vs the mission's expectation:** landing the vision-tower LOADER does NOT
+fix the ref2va grid, because the tower is wired as a loader+probe and is NOT yet scattered into
+the DiT render-conditioning path — this render did not use it. fl2va (same partition family,
+same session, VAE-keyframe path) is COHERENT, so the DiT/VAE/partition are sound; the ref2va
+grid is specific to the ref2va conditioning assembly (VAE-reference rows + un-enriched text
+prompt). Testing whether the vision-enriched prompt fixes ref2va needs the render-conditioning
+scatter (merge merged-features into prompt_embeds + DeepStack inject into the DEVICE text
+tower) — the tracked residual. Suite note: the full `test_minimax_h3` run SIGSEGVs at an
+UNRELATED CUDA case (line 3503, "an NVFP4 checkpoint loads into a runnable DiT") that PASSES in
+isolation (585 assertions) and runs BEFORE the new case — a pre-existing cross-test CUDA
+resource-accumulation flake, not this change; the new loader gate passes standalone (59
+assertions).
diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md
@@ -400,7 +400,7 @@ vLLM-Omni H3 modules at `vllm_omni/diffusion/models/minimax_h3/`; serving in
 | WebSocket `/v1/video/chat/stream`, `/v1/realtime/video` | `api_server.py:1593,1610` | — | **MISSING** (streaming/realtime) |
 | Request schema (prompt, size/w/h, num_frames, fps, seed, steps, refs) | `protocol/videos.py:97-249` | request contract (W7) | **PARTIAL** (core fields; frame-interp/lora/generate_sound absent) |
 | H3 knobs via `extra_params.{task,duration,flow_shift,audio_flow_shift}` | `pipeline:1034,403,1157-1158` | planner reads task/duration/shift | **DONE** |
-| Modalities in: text/image/video/audio | `pipeline:1036-1104` | t2va (text) done; fl2va/ref2va image/video/audio WIRED (W6/ref2va) | **PARTIAL** (encoder vision tower still open) |
+| Modalities in: text/image/video/audio | `pipeline:1036-1104` | t2va (text) done; fl2va COHERENT on real weights (VAE-keyframe); vision tower now LOADS real `visual.*` + runs (probe); ref2va still grids | **PARTIAL** (vision-tower→DiT-conditioning scatter is the residual; §8.8) |
 | Output: joint video+audio, 24 fps, 32 kHz stereo | `pipeline:106-111,1187` | frames + WAV + MP4 mux (W7) | **DONE** |
 | Scheduler: euler-ancestral rectified flow (single) | `scheduling_...euler_ancestral.py`; `time_request.py:34-61` | `MiniMaxH3EulerEta0Step` / `MiniMaxH3TimeShiftSigmas` | **DONE** |
 | CFG: distilled, no CFG (guidance params accepted+ignored; `cfg_parallel_size==1`) | `pipeline:250,275-276` | no CFG branch | **DONE** (matches) |
@@ -624,3 +624,88 @@ asserts the two real manifests reduce to the identical 535-name set (proving the
 no-discriminator premise in the harness). Neutralizing the guard body (reviewer mutation)
 turned the case RED at 10 assertions, restoring it turned it GREEN — the test has teeth.
 Suite: 67/67 (66 prior + this), 46549 assertions. `test_video_api` 4/4 (server wiring).
+
+## 8.8 ENCODER VISION TOWER — record reconciliation + real-weights wiring (2026-08-06, `row/H3-CONDITIONED-E2E`)
+
+**The contradictory record, reconciled (file:line).** Two prior lanes disagreed. The
+#26/W3 lane recorded the vision tower as **"W3 COMPLETE … the FULL vision tower at
+6.0e-8 … only the MM processor remains"** (this spec lines 101, 162-163); the #77
+residual recorded **"the encoder vision tower (W3) is still unported"** (lines 565-566).
+Reading the actual code resolves it — **both describe different halves and both are
+literally true of what they describe**:
+
+- The vision-tower **MATH exists** as a CPU scalar f32 reference in
+  `minimax_h3_encoder.cpp`: `MiniMaxH3VisionBlockForward` (:311), the surround
+  `MiniMaxH3VisionPosEmbedInterpolate` (:430) / `MiniMaxH3VisionRotary` (:500) /
+  `PatchMerger` (:545) / `MiniMaxH3VisionTowerForward` (:572). It is gated ONLY in
+  `tests/vllm/models/test_minimax_h3.cpp` (:3942, :4041) at **reduced dims with SYNTHETIC
+  weights** (`MakeParam`), block 6.0e-8 / tower ≤1e-4 vs a self-restated oracle.
+- It is **NEVER wired to real weights.** `LoadMiniMaxH3EncoderFromGguf`
+  (`minimax_h3_encoder_gguf.cpp:47`) loads the **TEXT tower only** — it iterates
+  `model.layers.N.*` + `model.embed_tokens.weight` and **skips every `visual.*` tensor**
+  (the comment at :51-52 even names `visual.*` as present-but-unloaded). The device
+  encoder `MiniMaxH3EncoderTextForwardDevice` (`minimax_h3_encoder_device.cpp:103`) runs
+  text only and takes **no deepstack / no visual-mask** argument (the HOST reference
+  `MiniMaxH3EncoderTextForward` does, :113-118). The driver
+  (`examples/minimax_h3_gen/main.cpp:476-547`) and server
+  (`examples/server/main.cpp:659-716`) call only the text path.
+- **So the reconciled truth:** the tower math is CPU-gated at reduced dims with synthetic
+  weights; there is **zero real-weights wiring** — no GGUF `visual.*` loader, no image→patch
+  MM processor on the H3 path, no device vision forward, no merge/DeepStack injection into
+  the encoded prompt. The #26 "only the MM processor remains" understated the gap (loader,
+  real-weights forward, and the merge/inject were ALSO absent); the #77 "still unported" was
+  right in the sense that matters (nothing real ran through it).
+
+**The encoder ARM already carries the vision weights (no download).** The on-box encoder
+`~/h3fp4/ckpt/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf` (14 GiB, the ComfyUI-format text-tower
+GGUF that already serves text conditioning) **DOES carry the full vision tower**: measured
+`visual.blocks.{0..26}` (27, Q4_K/Q5_K), `visual.patch_embed.proj` (F16 `[16,16,6,1152]` =
+Conv3d as a linear over `patch_elems`=1536), `visual.pos_embed.weight` (F16 `[2304,1152]` =
+48² grid), `visual.merger.*`, and **`visual.deepstack_merger_list.{0,1,2}`** (3 DeepStack
+mergers). Names map 1:1 to `MiniMaxH3VisionTowerForward` / the reuse target
+`multimodal::Qwen3VLVisionWeights`. So the encoder-arm decision is settled: **reuse the
+in-place encoder GGUF; no new download** (disk floor 15 GiB / ~23 GiB free honoured).
+
+**Vision geometry (from the checkpoint + state.md :23310-23318, the Qwen3.6-27B vision
+config which shares this tower):** hidden **1152**, **16 heads** (head_dim 72), depth **27**,
+intermediate **4304**, out_hidden **5120** (== encoder text dim), patch **16**, temporal **2**,
+merge **2**, num_position_embeddings **2304**, gelu-tanh blocks / exact-erf merger. H3 differs
+from the 27B only by having **3 real DeepStack mergers** (the 27B's are empty). The one
+config value NOT recoverable from the ComfyUI GGUF (weights-only, no arch metadata) is
+`deepstack_visual_indexes` — the WHICH-layers taps — needed for a bit-correct DeepStack
+inject; it is inferred + flagged as the residual for a fully-correct conditioned render.
+
+**The reuse path (mission: "stock Qwen3VLProcessor + our existing front end").** The image
+MM processor already exists and is gated: `multimodal::Qwen3VLImageProcessor::ProcessImage`
+(`qwen3vl_processor.h`, patch 16 / temporal 2 / merge 2 / 0.5 normalize → pixel_values +
+grid_thw), `ExpandImagePlaceholders`, and the device tower
+`multimodal::Qwen3VLVisionForward` (`qwen3_vl_vision.cpp`) with `PrepareVisionDeviceWeights`.
+The only genuinely-new code is the **GGUF `visual.*` → `Qwen3VLVisionWeights` loader**
+(`LoadQwen3VLVisionFromGguf`), mirroring the safetensors `LoadQwen3VLVisionWeights`
+(`qwen3_vl.cpp:417`) but dequantizing the Q4_K/Q5_K blocks (ComfyUI reshapes non-256-aligned
+rows to ne0=256; dequant preserves the flat row-major order the tower reads as `[out,in]`)
+and converting the F16 patch/pos tensors.
+
+**This row's status (honest):** loader + real-image processor reuse + the real-weights
+vision-tower forward gate LAND here (see §8.4-style status in STATUS/BENCHMARKS). The full
+vision-ENRICHED DiT render (DeepStack scatter into the DEVICE text tower changing the frames)
+depends additionally on the exact `deepstack_visual_indexes` and a device-text DeepStack/merge
+extension; its e2e render verdict is recorded honestly in the benchmark record.
+
+**GB10 VERIFIED (2026-08-07, dgx sm_121a).**
+- **Vision-tower probe RAN on real weights:** `--prompt-image` loaded the real `visual.*`
+  tower (27 blocks / 3 DeepStack mergers), processed a 512×512 image → grid [1,32,32], and
+  `Qwen3VLVisionForward` returned [256, 20480] all FINITE + non-degenerate (merged rms 1.45 /
+  maxabs 29.1 — the expected Qwen massive-activation). Deliverable-1 core DONE.
+- **fl2va e2e COHERENT:** FL2VA GGUF (`--dequant-bf16`) + a real first-frame (VAE-keyframe) +
+  `--partition fl2va`, 512×512/22f/12steps → all 22 frames a coherent photorealistic orange
+  cat on a wooden table matching the conditioning frame (no grid). Frame-sanity PASS.
+- **ref2va STILL GRIDS (honest):** Ref2VA NVFP4 (`--fp4-resident`) + a real `--ref-image` +
+  `--partition ref2va` → every frame a multicolour patch grid. Landing the tower LOADER does
+  NOT fix it: the tower is a loader+probe, NOT yet scattered into the DiT render-conditioning,
+  so this render never used it. fl2va (same session, VAE-keyframe) is coherent ⇒ DiT/VAE/
+  partition are sound; the ref2va grid is specific to the ref2va conditioning assembly. The
+  render-conditioning scatter (merge features into prompt_embeds + DeepStack inject into the
+  DEVICE text tower) is the tracked residual that would let the vision-enriched-prompt
+  hypothesis be tested. The `--ref-video` VAE encode is a slow single-thread CPU 3D-CNN path
+  (separate perf limit).