fix(h3): route video device selection through the backend seam

.agents/NOW.md

@@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
 | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — |
 | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning |
 | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable |
-| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 2 LANDED (#123)**: H3 video on the surface (`vllm_video_*` v12, `/v1/videos` via the seam, both examples thin clients, ratchet 11→9); ROW 1 (#121) before it | GB10 re-verify residual; next fold row |
+| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 2 LANDED (#123); H3 device seam repair in #134**: ABI 0/1 maps through `DeviceType`; DSR 34→32 without baseline/allowlist change | CI compile/fold gate pending; GB10 re-verify residual; next fold row |
 
 In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna
 bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

.agents/parity-ledger.md

@@ -924,3 +924,4 @@ Columns:
 | 2026-08-06 (`row/H3-BF16-SHARDED-DIT`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; CPU-only, no GPU and no download; lifecycle unchanged) | **MiniMax-H3 — the ORIGINAL bf16 release (13 safetensors shards, 66.3 GB) now LOADS, and it STREAMS.** Every H3 render so far used a QUANTIZED DiT and H3 is unusually quantization-sensitive (Q3_K_M -> Q4_K_M alone turned a murky lattice into a photoreal close-up; ComfyUI PR 15298 blames the partial split-half RoPE's channel-wise magnitude outliers), but the full-precision question was unaskable because every DiT loader took a SINGLE file. Adds (a) `MiniMaxH3ShardedCheckpoint::Open(dir)` (`src/vllm/model_executor/models/minimax_h3_sharded.cpp`), which resolves tensors through the checkpoint's own `model.safetensors.index.json` weight map (never by scanning) with one index over every shard, mirroring the in-tree multi-shard template `LoadMiniMaxH3EncoderWeights(const std::vector<SafetensorsFile>&, ...)`, and throws BY NAME when the index names a tensor its shard does not hold; (b) `StreamMiniMaxH3ShardedToDeviceBf16` (`minimax_h3_device.cpp`, sharing `BindStreamedDitViews` with the GGUF and NVFP4 streamers), which converts+uploads ONE tensor at a time and uploads a BF16 tensor bound for a bf16 device slot DIRECTLY out of the read-only mmap with no host buffer at all, releasing each source range afterwards; (c) `LoadMiniMaxH3DitFromShards`, the host-f32 reference loader; (d) `MiniMaxH3IsFp32IslandTensor`, single-sourcing the upstream fp32-ISLAND split the three existing streamers each hand-rolled; (e) `--dit <dir>` in `examples/minimax_h3_gen` for both `--dump-params` and the run path, every existing `--dit` form unchanged. It MUST stream: the pool is UNIFIED (122 GiB shared host+device) and the non-streaming NVFP4 loader was already OOM-killed at anon-rss 125 GB on half this size. | vLLM-Omni `vllm_omni/diffusion/models/minimax_h3/minimax_h3_transformer.py:85-101` (MINIMAX_H3_FP32_PARAM_NAMES / _BUFFER_NAMES, the island split) and `:906-922` (the parameter set); the shard-index container convention is HF safetensors' own `model.safetensors.index.json` weight_map, already consumed in-tree by `LoadSafetensorsIndex` and the multi-shard encoder/VAE loaders. No vLLM behavior changed; H3 remains BEYOND-PIN (vllm-omni, not the pinned vLLM repo). | **LANDED + CPU-GATED (loader brick; `benchmark_binding=false` — no throughput owed, and NO bf16-vs-quant render or speed number is claimed).** `test_minimax_h3` 68/68 cases / 49300 assertions, clean Release build of `libvllm.a`, `test_minimax_h3` and `minimax-h3-gen`. Four gates: (1) index+name mapping over a synthetic 4-shard set — every tensor resolves to the shard the index named AND to the bytes written there, a tensor missing from its shard throws WITH ITS NAME, and the derived geometry equals the single-file path field for field; (2) streamed == non-streamed — all 46 weight views BIT-EXACT (`memcmp == 0`) vs `StageMiniMaxH3DitWeights(kBF16)`, dtypes included (12 fp32 islands), both device forwards IDENTICAL (max|diff| == 0), `rope.inv_freq` HOST-resident; (3) the loader RAN — `MiniMaxH3ShardStreamStats` (mirroring `Nvfp4W4A16Stats`) proves shards opened, tensors streamed, BOTH upload paths taken, every view owned by this loader, and `host_peak_bytes` bounded by one tensor (< 1/4 of bytes uploaded), i.e. peak cannot scale with the model; (4) a SPARSE 13-shard release declaring the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry 50/5376/56/128/14336/24/32/1x2x2/5120, and `minimax-h3-gen --dit <dir> --dump-params` prints all 20 fields on it. Honest residuals: the real 66.3 GB load and its measured peak RSS, CUDA memcpy from a file-backed mmap, and any bf16-vs-quantized render/speed comparison are all UNVERIFIED here (no GPU, no download, per the operator's instruction). Not pushed. |
 | 2026-08-06 (`row/H3-ENC-BF16-COND-DIFF`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; lifecycle unchanged) | **MiniMax-H3 - the bf16 TEXT ENCODER (14 safetensors shards, 63 GB) now LOADS, it STREAMS, and `--encoder-only` runs the tower alone.** Every H3 render so far conditioned on a Q4_K_M Qwen3-VL-32B encoder and the encoder's contribution had never been measured, but `--encoder` accepted only a GGUF. Adds (a) `MiniMaxH3EncoderConfigFromShards`, deriving the geometry from the shard index's SHAPES alone with the SAME recovery rules AND the same non-shape defaults (`rope_theta`, `mrope_section`, `rms_norm_eps`, `selected_layer`) as the GGUF loader, so an A/B cannot be comparing two RoPEs; (b) `StreamMiniMaxH3EncoderShardsToDevice` (`src/vllm/model_executor/models/minimax_h3_encoder_sharded.cpp`), which fills the SAME `MiniMaxH3EncoderDeviceWeights::views` map the GGUF arm fills - over bf16 instead of ggml blocks - uploading projections DIRECTLY out of the read-only mmap and doing the `[q|k|v]` / `[gate|up]` row fusions ON THE DEVICE into offsets of one allocation, so even the transform costs no host copy; (c) `MiniMaxH3EncoderEmbedTokensFromShards`, a per-row gather out of the `[151936, 5120]` table; (d) a bf16-weight WIDEN step in `MiniMaxH3EncoderTextForwardDevice` (scratch reused across layers) because `vt::MatmulBT` needs one dtype for both operands and these activations are f32 - the 50 layers H3 runs are 48.8 GiB bf16 vs 97.5 GiB f32 on a 122 GiB UNIFIED pool; (e) `--encoder <dir>` and `--encoder-only` in `examples/minimax_h3_gen`, which drops peak from ~96 GiB (DiT loaded first) to ~49 GiB. No new forward: the encoder graph is byte-for-byte the same code for both arms, which is what makes the quantization question measurable. | The name map is the one already gated in-tree for `LoadMiniMaxH3EncoderWeights(const std::vector<SafetensorsFile>&, ...)` (`model.language_model.layers.N.` -> `layers.N.`, q/k/v and gate/up FUSED, final `norm.weight` and `lm_head.weight` deliberately unbound because H3 reads the UNNORMALIZED truncated output); shard resolution reuses `MiniMaxH3ShardedCheckpoint` (§8.13, landed as its own row), i.e. the checkpoint's own HF `model.safetensors.index.json` weight_map. Encoder truncation to `min(num_hidden_layers, 50)` is upstream vLLM-Omni's own. No vLLM behavior changed; H3 remains BEYOND-PIN (vllm-omni). | **LANDED + CPU-GATED (loader brick; `benchmark_binding=false`).** Re-gated AFTER the rebase onto `row/H3-BF16-SHARDED-STREAM`: `test_minimax_h3` 75/75 cases / 55609 assertions, clean Release build of `libvllm.a`, `test_minimax_h3`, `minimax-h3-gen`. Three gates: (1) a synthetic 4-shard encoder at the REAL name spellings resolves, and every fused view is `memcmp`-exact against `q ++ k ++ v` / `gate ++ up` for EVERY layer, unfused projections byte-exact, separate names gone, final norm + lm_head + vision tower NOT bound, truncation honoured, embedding gather exact and out-of-range throwing; (2) the loader RAN and is NOT the GGUF path - `MiniMaxH3EncoderShardStreamStats` asserted on shards/layers/views/fused groups/direct-vs-converted uploads with `host_peak_bytes` equal to ONE norm (peak cannot scale with the model), and the views are `kBF16`, a dtype the GGUF loader can never produce; (3) the WIDENING is exact - the same checkpoint written BF16 and F32 (bf16-rounded values) streams to `kBF16` and `kF32` views respectively and the two full encoder forwards are BIT-IDENTICAL (`memcmp == 0`), so the conditioning A/B cannot be confounded by the widening. The real 63 GB load, its peak RSS, and the Q4_K_M-vs-bf16 conditioning numbers are the GPU follow-up in this row. |
 | 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; MEASUREMENT, lifecycle unchanged) | **MiniMax-H3 - THE NUMBER: what quantizing the TEXT ENCODER to Q4_K_M does to the conditioning.** Same prompt (wuxia, 233 tokens), same tokenizer, same 50-layer truncation, same `MiniMaxH3EncoderTextForwardDevice`, same f32 activations - only the weight bytes differ (Q4_K_M ggml blocks vs the original bf16 14-shard release); both arms self-report IDENTICAL geometry (50/5120/64/8/128/25600), which is what establishes they are the same model. Conditioning `[233, 5120]` f32 via `--encoder-only --save-embeds`. | Not a vLLM-parity change: H3 is BEYOND-PIN (vllm-omni), and vLLM-Omni serves NO quantized H3 at all (BF16-only), so there is no upstream arm to compare against - the bf16 release IS the reference here, and it is the one the loader added in `6d454b00` makes runnable. The quantization-sensitivity premise is ComfyUI PR 15298 (H3's partial split-half RoPE produces channel-wise magnitude outliers that corrupt even INT8), and the measurement CONFIRMS its mechanism concretely: token 0 is an attention sink at norm 15,522 vs a 366 mean (42x), carrying 68% of the total squared error with its DIRECTION intact (cos 0.99962). | **MEASURED on Thor sm_110, build `d1085374` (built and measured as `d1085374`, amended for the row-branch trailer; IDENTICAL tree `dd9283cf`, so the measurement binary IS this commit), GPU idle.** Q4_K_M vs bf16: max|diff| 154.0, RMS 0.5045, rel RMS **0.03403** (0.06849 excluding the sink token), per-token cosine min 0.90916 / mean **0.99745** / median 0.99810, rotation median 3.535 deg / max 24.61 deg, 232 of 233 tokens below cosine 0.999. NOT a scale change: norm ratio mean 0.99010 but the best global rescale only moves 0.03403 -> 0.03280, so it is DIRECTIONAL. CALIBRATION arm (bf16 encoder, ONE-WORD prompt edit `at night`->`at dawn`, also 233 tokens): rel RMS 0.01897 / 0.06666 excl. sink, cosine mean 0.99769 median 0.99963, 172 of 233 tokens above 0.999. So quantization moves the conditioning by the SAME total energy as rewriting a word of the prompt (6.85% vs 6.67%) but with the opposite SHAPE - diffuse over every token instead of concentrated on the words that changed. Cost: Q4 arm 40 s / 18.0 GiB peak; bf16 arm 40 s / 45.41 GiB uploaded / host conversion peak 0.0195 MiB / 51.95 GiB total peak, streamer counters `layers=50 tensors=400 direct=350 converted=200 fused=100` proving the shard path ran. `benchmark_binding=false` (no throughput claim). EXPLICITLY NOT ESTABLISHED: that the RENDER changes - nothing here measures the DiT's sensitivity to a 3.5-degree median rotation; the owed follow-up is a same-DiT/same-seed render A/B over the two saved embeds. |
+| 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 ABI-v12 device dispatch follow-up; lifecycle unchanged) | **MiniMax-H3 video engine device selection is backend-parameterized.** The stable public selector remains 0=CPU / 1=CUDA, but shared code maps it once to `vt::DeviceType`, creates one queue through `GetBackend(device_type)`, and uses that queue's device instead of naming CUDA twice. | vLLM-Omni pipeline ownership remains unchanged; this is a vllm.cpp C-ABI/backend-seam correction with no upstream behavioral delta. | RED: DSR 34 (`kcuda=2`) vs baseline 32. GREEN: DSR 32 with baseline/allowlist unchanged; checker mutations 25/25. New fold unit pins 0/1 and invalid selectors; CPU compile/test NOT RUN locally because the shared filesystem reached 100%, pending GitHub CI. No GPU, download, or performance claim. |