H3 image/video-conditioned generation: wire the encoder vision tower to real weights (row/H3-CONDITIONED-E2E) - #86
Closed
localai-bot wants to merge 4 commits into
Closed
H3 image/video-conditioned generation: wire the encoder vision tower to real weights (row/H3-CONDITIONED-E2E)#86localai-bot wants to merge 4 commits into
localai-bot wants to merge 4 commits into
Conversation
…nthetic, never real-wired; encoder GGUF carries visual.* (row/H3-CONDITIONED-E2E) Reconcile the contradictory vision-tower record before re-porting anything. #26/W3 recorded "vision tower COMPLETE, only MM processor remains"; #77 recorded "encoder vision tower still unported". Reading the code, both describe different halves: the tower MATH exists as a CPU scalar reference (minimax_h3_encoder.cpp:311/430/500/545/572) gated ONLY at reduced dims with SYNTHETIC weights (test_minimax_h3.cpp:4041), but it was NEVER wired to real weights — LoadMiniMaxH3EncoderFromGguf (minimax_h3_encoder_gguf.cpp:47) loads the TEXT tower only and skips every visual.* tensor; the device forward is text-only; the driver/server call only the text path. Encoder-arm decision (measured, no download, disk floor honoured): the on-box qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf DOES carry the full vision tower (27 blocks Q4_K/Q5_K, patch_embed F16 [16,16,6,1152], pos_embed F16 [2304,1152]=48², merger, 3 deepstack mergers). Geometry hidden 1152 / 16 heads / depth 27 / intermediate 4304 / out 5120 / patch 16 / temporal 2 / merge 2 / pos 2304 (== the Qwen3.6-27B vision config). Reuse target: multimodal::Qwen3VLVisionForward + Qwen3VLImageProcessor. Records: spec §8.8, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW, state (2026-08-07T01:45). Pre-existing preflight red (check-fusion-consistency minimax_h3_video_vae_device) is not this row. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
… weights + driver probe
The one piece the record reconciliation (spec §8.8) found missing: LoadQwen3VLVisionFromGguf
reads the encoder GGUF's visual.* tower into the shared multimodal::Qwen3VLVisionWeights the
Qwen3-VL front end consumes, DEQUANTIZING each ggml-block projection (Q4_K/Q5_K) + F16
patch_embed/pos_embed to f32. Encoding-agnostic (per-tensor DequantGgufRowToF32), so the
ComfyUI ne0=256 reshape of the non-256-aligned rows is transparent — dequantizing the whole
flat buffer preserves the row-major [out,in] order the tower reads. MiniMaxH3EncoderVisionConfig
returns the measured H3 vision geometry (hidden 1152 / 16 heads / depth 27 / intermediate 4304
/ out 5120 / patch 16 / temporal 2 / merge 2 / pos 2304, 3 DeepStack mergers).
- New TU src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp (+ CMake).
- CPU gate test_minimax_h3 "the encoder GGUF visual.* loader dequantizes the vision tower":
builds a synthetic visual.* GGUF (Q8_0 projections + F16/F32), loads it, asserts every field
is filled at the flat [out*in] size the tower reads, the pre/post-shuffle merger norm widths,
Q8_0 dequant within block tolerance + non-degenerate, and the production config geometry. The
loader is encoding-agnostic so Q8_0 exercises the exact path the shipped Q4_K/Q5_K takes.
- Driver examples/minimax_h3_gen --prompt-image: routes a real image through the shared
Qwen3VLImageProcessor + Qwen3VLVisionForward on the GGUF-loaded weights and reports the
conditioning feature stats (finite / rms / maxabs) — the real-weights proof. Scattering the
merged/deepstack features into the DiT-conditioning path is the tracked residual (spec §8.8).
deepstack_visual_indexes is inferred ({8,16,24}); the weights-only ComfyUI GGUF does not carry
it. Flagged in the header + spec as needing upstream-config confirmation for a bit-correct
DeepStack inject. Pre-existing preflight red (check-fusion-consistency
minimax_h3_video_vae_device) is not this row.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…sage check The probe runs the vision tower only (from --encoder) and returns before the DiT section, like --decode-latent/--roundtrip, so it must not require --dit/--video-vae/ --out. Add it to the usage-check bypass. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
… ref2va still grids Deliverable-1 core VERIFIED on dgx: --prompt-image loaded the real visual.* tower and ran Qwen3VLVisionForward → [256,20480] all finite + non-degenerate (merged rms 1.45 / maxabs 29.1 Qwen massive-activation; 3 deepstack finite). CPU loader gate passes standalone (59 assertions). Deliverable-2 renders (frame-sanity, visually inspected): - fl2va COHERENT: FL2VA GGUF --dequant-bf16 + real --first-frame (VAE-keyframe) + --partition fl2va, 512x512/22f → all 22 frames a coherent orange cat on a wooden table matching the conditioning frame, no grid. PASS. - ref2va STILL GRIDS: Ref2VA NVFP4 --fp4-resident + real --ref-image + --partition ref2va, 256x256/22f → every frame a patch grid. Honest vs the mission's expectation: landing the LOADER does not fix ref2va because the tower is a loader+probe, NOT yet scattered into the DiT render-conditioning. fl2va coherent (same session) ⇒ DiT/VAE/partition sound; the ref2va grid is specific to the ref2va conditioning assembly. Residual: scatter merged features into prompt_embeds + DeepStack inject into the device text tower, then re-test. Records: spec §8.8 (+GB10-verified block) + §8.2 supports-audit row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record (GB10 verification), state (2026-08-07T03:20), NOW. Box left clean. Pre-existing preflight red (check-fusion-consistency minimax_h3_video_vae_device) is not this row. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…ision tower loads + runs (#86) row/H3-CONDITIONED-E2E squash. Record reconciliation: the vision-tower MATH existed and was gated at reduced dims with synthetic weights (minimax_h3_encoder.cpp:572 + surround), but was never wired to real weights — the GGUF encoder loader skipped every visual.* tensor and the device text forward has no deepstack arg; #26 understated the gap, #77 was right in what matters. Wired now: LoadQwen3VLVisionFromGguf dequantizes the on-box encoder's full visual.* tower (27 blocks + patch embed + pos embed + merger + 3 deepstack mergers, shared Qwen3.6-27B vision geometry) into the existing Qwen3VLVisionWeights reuse; CPU loader gate + a driver --prompt-image probe (real 512x512 image -> [256,20480] finite, the Qwen massive-activation signature). Frame sanity on GB10: fl2va with a real first-frame renders a coherent photorealistic scene MATCHING the conditioning frame (pose/table/ background) — the second conditioned modality works. ref2va still grids, now for a PRECISE reason: the loaded tower is not yet scattered into the DiT render-conditioning (the render never used it); fl2va coherent in the same session proves DiT/VAE/partition sound. Residual = vision->prompt_embeds scatter + DeepStack device injection + confirm deepstack_visual_indexes {8,16,24} against upstream. Known flake: full test_minimax_h3 SIGSEGVs at a pre-existing unrelated CUDA case that passes standalone (cross-test resource accumulation). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The vision->DiT-conditioning scatter runs as the next box campaign — the ref2va grid's now-precise residual. 🤖 Generated with Claude Code |
This was referenced Aug 7, 2026
Closed
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…Stack→device text tower) + ref2va re-attribution Closes the #86 framework residual and re-attributes the ref2va grid with a GB10 render A/B (row/H3-VISION-SCATTER). deepstack_visual_indexes CONFIRMED (was #86-inferred): {8,16,24}, grounded in the release config — MiniMax-H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose vision_config.deepstack_visual_indexes = [8,16,24], depth 27, text num_hidden_layers 64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig default and public Qwen/Qwen3-VL-30B-A3B. No value change; comment updated. Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice now takes the optional visual_pos_mask + per-tap deepstack blocks and ADDS each block into the masked visual rows after each of the first len(deepstack) decoder layers — device mirror of the gated host reference and upstream _deepstack_process (encoder.py:770-800). The merged masked_scatter into inputs_embeds stays the caller's job (upstream _encode, encoder.py:1071). Text-only prompts pass the defaults and are byte-identical. Gate: the device keep-quant encoder test now also runs WITH a visual mask + two DeepStack blocks → device==host max|diff| 3.8e-4 (<=2e-3) AND DeepStack moves the conditioning (scale 1.006->1.062). Driver --cond-image routes a reference image through the encoder vision path (reuse only: Qwen3VLImageProcessor -> Qwen3VLVisionForward -> merged + 3 DeepStack; ExpandImagePlaceholders inserts nm image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex == H3 _get_rope_index for t==1, position math verified). Additive. GB10 render A/B (256x256/22f/12steps): * Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS): frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). No grid. The vision conditioning is SOUND. * Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL): same multicolour patch grid as #86's text-only ref2va. RE-ATTRIBUTION (evidence): the "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. NOT the encoder conditioning — DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. The only difference between coherent fl2va and gridding ref2va: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running reference rows. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward. Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW, state. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
jefby
pushed a commit
to jefby/vllm.cpp
that referenced
this pull request
Aug 7, 2026
…was vacuous Self-reported violation and its prevention. PR mudler#80 (Vulkan, merged 5397e91) landed EIGHT commits that touched src/ and tests/ without updating docs/STATUS.md and docs/BENCHMARKS.md in the same commit, reddening documentation-checkpoint for that push range: 9579f94, ba5ea0c, 196ea46, e32c5ed, 3bfa1f1, 34a3efe, 2c86f79, f4738bb. Each deferred its doc update into a following record(...) commit. WHY IT PASSED LOCALLY, which is the part worth fixing. check-doc-checkpoint.py --staged inspects the STAGED paths, so it is VACUOUS when nothing is staged -- and nothing is staged after `git commit`, which is exactly when agent-preflight.sh runs it. The gate reported OK on every one of those commits while checking literally nothing. CI is diff-scoped over the pushed range and checks each commit independently, so the failure only surfaced on main, where a diff-scoped range can never be re-covered by a later run. Preflight now runs --base origin/main --head HEAD whenever the branch is ahead. That check is deliberately OUTSIDE the --staged block: my first attempt nested it inside, which reproduces the identical hole one level up, since --staged is precisely the flag you are not passing when the range is unchecked. Verified two ways -- it prints "ok doc-checkpoint range" here, and pointed at the mudler#80 range it raises 20 errors naming every offending commit. workflow.md carries the same instruction with the reason. NOT REPAIRED BY REWRITING HISTORY: main had already moved (mudler#86 landed on top) and other sessions branch from it, so force-pushing to regroup eight commits' files would cost more than the defect. Substance was never wrong -- STATUS, BENCHMARKS and FEATURES on main all describe the shipped state with the llvmpipe-only and no-speed-number caveats intact. What was violated is the per-COMMIT granularity that keeps a bisect landing on a commit whose docs match its code. Rule restated where it is enforced: a feature commit carries its OWN STATUS/BENCHMARKS update; when the numbers are not yet known the honest line is pending/void with the reason, which is what the gate's own message asks for. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mission
Wire the MiniMax-H3 encoder vision tower into the real encoder path so image/video-conditioned generation (fl2va / ref2va) works on real weights — the lane's last supports gap vs vllm-omni.
Record reconciliation (done first)
Two prior lanes contradicted. Reading the code resolves it — both describe different halves:
minimax_h3_encoder.cpp:572+ surround), gated only at reduced dims with SYNTHETIC weights (test_minimax_h3.cpp:4041).LoadMiniMaxH3EncoderFromGguf(minimax_h3_encoder_gguf.cpp:47) loads the text tower only and skips everyvisual.*tensor; the device forward is text-only; driver/server call only the text path.Encoder-arm decision (measured, no download)
The on-box
qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf(14 GiB) carries the full vision tower (27 blocks Q4_K/Q5_K,patch_embedF16,pos_embedF16 48²,merger, 3 deepstack mergers). Geometry: hidden 1152 / 16 heads / depth 27 / intermediate 4304 / out 5120 / merge 2 / pos 2304.Landing (verified on GB10)
visual.*→Qwen3VLVisionWeightsloader (LoadQwen3VLVisionFromGguf) + CPU gate (59 assertions)--prompt-imageprobe — vision tower RAN on real weights: [256,20480] all finite, merged rms 1.45 / maxabs 29.1 (Qwen massive-activation)Pre-existing preflight red (
check-fusion-consistency minimax_h3_video_vae_device) is not this row.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys