row/H3-VISION-SCATTER: scatter vision tower into DiT conditioning (deepstack + merged scatter) - #90
Closed
localai-bot wants to merge 1 commit into
Closed
row/H3-VISION-SCATTER: scatter vision tower into DiT conditioning (deepstack + merged scatter)#90localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…Stack→device text tower) + ref2va re-attribution Closes the #86 framework residual and re-attributes the ref2va grid with a GB10 render A/B (row/H3-VISION-SCATTER). deepstack_visual_indexes CONFIRMED (was #86-inferred): {8,16,24}, grounded in the release config — MiniMax-H3's text_encoder/ IS Qwen3-VL-32B-Instruct, whose vision_config.deepstack_visual_indexes = [8,16,24], depth 27, text num_hidden_layers 64 (→50). Same as vllm-omni Qwen3VLMoeVisionConfig default and public Qwen/Qwen3-VL-30B-A3B. No value change; comment updated. Deliverable 1 — DEVICE scatter+inject WIRED 1:1 + GATED. MiniMaxH3EncoderTextForwardDevice now takes the optional visual_pos_mask + per-tap deepstack blocks and ADDS each block into the masked visual rows after each of the first len(deepstack) decoder layers — device mirror of the gated host reference and upstream _deepstack_process (encoder.py:770-800). The merged masked_scatter into inputs_embeds stays the caller's job (upstream _encode, encoder.py:1071). Text-only prompts pass the defaults and are byte-identical. Gate: the device keep-quant encoder test now also runs WITH a visual mask + two DeepStack blocks → device==host max|diff| 3.8e-4 (<=2e-3) AND DeepStack moves the conditioning (scale 1.006->1.062). Driver --cond-image routes a reference image through the encoder vision path (reuse only: Qwen3VLImageProcessor -> Qwen3VLVisionForward -> merged + 3 DeepStack; ExpandImagePlaceholders inserts nm image-pad tokens; masked_scatter; Qwen3VLGetRopeIndex == H3 _get_rope_index for t==1, position math verified). Additive. GB10 render A/B (256x256/22f/12steps): * Deliverable 3 — fl2va WITH the encoder vision path = COHERENT + matching (PASS): frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). No grid. The vision conditioning is SOUND. * Deliverable 2 — ref2va WITH the vision-enriched prompt STILL GRIDS (honest FAIL): same multicolour patch grid as #86's text-only ref2va. RE-ATTRIBUTION (evidence): the "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. NOT the encoder conditioning — DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. The only difference between coherent fl2va and gridding ref2va: fl2va PINS output rows (keyframe), ref2va PREPENDS free-running reference rows. Residual = the ref2va reference-row conditioning ASSEMBLY (MiniMaxH3EncodeReferenceImages VAE-reference rows + minimax_h3_packed_sequence_ref2va_blocks noised-anchor layout + un-pinned target-row denoise), NOT prompt_embeds and NOT the DiT forward. Records: spec §8.9 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW, state. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/H3-VISION-SCATTER
branch
from
August 7, 2026 02:17
c7b0271 to
e7a9a59
Compare
Collaborator
Author
|
Landed on main as the squash (mudler authorship). ref2va's reference-row assembly diagnostic runs next. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…a conditions via the encoder (#90) row/H3-VISION-SCATTER squash. deepstack_visual_indexes {8,16,24} CONFIRMED against the release config (the text_encoder IS Qwen3-VL-32B-Instruct) + vllm-omni defaults. Wired 1:1 with upstream: per-layer DeepStack scatter-add into masked rows on the DEVICE text forward (minimax_h3_encoder_device.cpp:103,216-243, mirroring _deepstack_process encoder.py:770-800); merged masked_scatter into inputs_embeds at the caller per upstream _encode; driver --cond-image runs ProcessImage -> vision tower -> placeholder expansion -> scatter -> rope indexes (byte-equivalent to H3's _get_rope_index at t==1). Text-only path byte-identical. Gate: device==host 3.8e-4 with DeepStack proven to move the conditioning (1562 asserts). GB10 renders: fl2va + --cond-image = COHERENT and MATCHING, evolving toward the text prompt (keyframe cat -> windowsill in warm sunlight). ref2va STILL GRIDS — the vision-conditioning hypothesis is REFUTED (same vision path renders coherent fl2va; grid invariant to text-only vs vision-enriched). The re-attributed residual: fl2va PINS output rows while ref2va PREPENDS free-running reference rows — the ref2va reference-row conditioning ASSEMBLY (VAE-reference rows, noised-anchor layout, un-pinned target denoise) is the remaining suspect; next diagnostic = target-row VAE-input adjacency cosine + condition-noise A/B. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mission — scatter the loaded vision tower into the DiT conditioning; fix the ref2va grid (spec §8.9)
After #86 the real
visual.*tower LOADS + RUNS and image-conditioned fl2va is COHERENT via the VAE-keyframe path, but ref2va gridded because vision features never reached the DiT: the encoder's DEVICE text forward had no deepstack arg and nothing scattered merged vision features intoprompt_embeds.Results
deepstack_visual_indexes = {8,16,24}CONFIRMED vs the release config (H3'stext_encoder/IS Qwen3-VL-32B-Instruct:vision_config.deepstack_visual_indexes = [8,16,24], depth 27). H3 image/video-conditioned generation: wire the encoder vision tower to real weights (row/H3-CONDITIONED-E2E) #86 inference was correct.MiniMaxH3EncoderTextForwardDevicenow takesvisual_pos_mask+ per-tapdeepstackand injects into the first N layers (mirrors host reference + upstream_deepstack_process). Merged masked_scatter intoinputs_embedsstays the caller's job (upstream_encode). Gate: device==hostmax|diff| 3.8e-4, DeepStack moves the conditioning. All encoder/vision gates green.--cond-imageroutes an image through the encoder vision path (ExpandImagePlaceholders+Qwen3VLGetRopeIndex== H3_get_rope_indexfor t==1). Additive.--cond-image= COHERENT + matching (PASS). GB10: frame 0 the keyframe cat, frame 21 the cat on a WINDOWSILL in warm sunlight (evolved toward the prompt). Vision conditioning is SOUND.--cond-imageSTILL GRIDS (honest FAIL, recorded).Verdict / re-attribution
The "vision-enriched conditioning fixes the grid" hypothesis is REFUTED. The ref2va grid is NOT the encoder conditioning: DiT math byte-exact (#74/#77), the SAME vision path renders a coherent fl2va, and the grid is invariant to text-only vs vision-enriched prompts. Residual = the ref2va reference-row conditioning ASSEMBLY (
MiniMaxH3EncodeReferenceImagesVAE-reference rows +minimax_h3_packed_sequence_ref2va_blocksnoised-anchor layout + un-pinned target-row denoise), NOTprompt_embedsand NOT the DiT forward. fl2va PINS output rows; ref2va PREPENDS free-running reference rows — the one difference.Records: spec §8.9 + §8.2, STATUS/BENCHMARKS/FEATURES, benchmark-record, NOW, state.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys