feat(minimax-h3): port the MiniMax-H3 omni-modal video+audio DiT (portable path complete; speed GPU-gated)

[localai-bot]

Ports MiniMax-H3 — a 33.1B CFG-distilled joint video+audio diffusion transformer — into vllm.cpp. It is the project's first DIFFUSION architecture: no KV cache, no sampler, no logits, so it is not a causal-LM registry model. Upstream is vllm-project/vllm-omni, not the vLLM repo, and it sits beyond the parity pin (555967922).

Every portable piece is done. What remains needs a GPU, not more porting.

How this is gated (no weight bytes checked in)

Two techniques carry the whole PR:

1. Upstream-as-oracle. The generators import vLLM-Omni's own Python modules (and, for the VAEs, the checkpoint's own remote code, which ships under trust_remote_code and therefore had to be reimplemented in C++) and execute them at reduced dimensions. Both sides rebuild weights from a shared FNV-1a + splitmix64 PRNG, so a golden is reproducible from source alone.
2. HTTP range requests over checkpoint headers. GGUF and safetensors both put a complete tensor manifest in a header a few tens of KB long. Fetching only that prefix gates the loaders against real multi-GB checkpoints — names, shapes, dtypes — without downloading, or committing, a single weight.

Correctness gates

Gate	Result
fl2va / ref2va packed layout (ids, tags, masks, cu_seqlens, doc ids)	exact
fl2va fp64 position grid	bit-exact
patchify / unpatchify / audio pack / unpack	exact + round-trip identity
euler-ancestral eta0 scheduler + rf_v_to_x0	exact
DiT forward (f32 / bf16 production stream)	1.6e-7 / 2.4e-3
request planning, condition noise, presentation tags, reference-video geometry	exact
AUDIO VAE decoder (DAC/BigVGAN, reimplemented) vs the checkpoint's remote code	4.2e-9
VIDEO VAE full ViT3D decoder (36 blocks) vs the checkpoint's remote code	8.9e-8
whole VAE 3D-CNN encoder + tiling plan + seam blend	exact
ENCODER text tower (layer truncation, unnormalized layer-49 output, DeepStack)	1.2e-7
ENCODER full vision tower (ragged 2-image batch, DeepStack + mergers)	6.0e-8 / <=1e-4
REAL GGUF manifest (535 tensors)	exact match; geometry derived from shapes alone equals the shipped config
REAL NVFP4 manifest (1051 tensors)	exact: textbook compressed-tensors triple, group 16
MP4 mux, end to end	12 PPM frames + WAV through the built example → ffprobe reports h264/yuv420p + stereo AAC 32 kHz
/v1/videos contract, job store, route dispatch	pass
Full CPU suite	333/333, clean build, zero warnings
DEVICE-RESIDENT DiT forward on a REAL GPU (Thor, sm_110)	video 1.49e-7 / audio 8.94e-8 vs upstream — ~134x inside tolerance, on par with the CPU reference's own 1.6e-7

The fp64 grid is gated bit-exact because it feeds RoPE. Matching it required reproducing upstream's arithmetic order, not just its formulas: numpy.linspace(endpoint=False) evaluates i*step + start, and upstream keeps a numpy-pairwise and a Python-sequential span sum deliberately separate because they diverge in the last ulp from n=16 (packed_sequence.py:101-113).

Attention reuses the shared vt::DFlashBlockAttention(causal=false) for packed varlen — no new kernel.

The ffmpeg boundary

/v1/videos returns an MP4, and upstream shells out to ffmpeg. This library has no subprocess precedent, so the split was put to the project owner and ratified: "re: ffmpeg invocation, correct - let's keep in the examples only".

• src/vllm/ builds the artifacts (PPM frames, WAV) and the argv — and spawns nothing.
• examples/minimax_h3_mux/ performs the invocation.
• /v1/videos therefore takes a caller-supplied VideoRunner callback.

Serving

POST /v1/videos (async), POST /v1/videos/sync, GET /v1/videos/{id}, registered on ApiServer via the server's existing additive/opt-in pattern: they appear only when set_video_runner has been called, so a server built without video support is byte-identical to before — no new constructor parameter, no existing caller touched.

Async uses a joinable worker drained in ~ApiServer; a detached thread would outlive this and write into a destroyed job store. A throwing runner fails the job, not the process.

Hardware verdict (corrected mid-PR)

An early reading called this hardware-blocked. That was wrong and the developer corrected it: the bf16 release (~354 GB) does not fit one box, but the quantized arms do — GGUF (DiT 15.6 GB + encoder 14.6 GB + VAEs ~11 GB ≈ 41 GB) and NVFP4 both load. Both quantized loaders are implemented and gated against the real manifests.

The device-resident forward (W2b) — landed and GPU-verified

MiniMaxH3DitForwardDevice runs the whole DiT graph with every activation in device memory, so the block stack — and above it the 50-step denoise loop — never round-trips through the host. Verified on a real GPU, not just compiled: the CUDA case is proven to have run (220 assertions execute; 9673 total on GPU vs 9453 on CPU, where it skips).

Three H3 kernels were needed, and only three, because the port reuses the tuned shared ops (MatmulBT, RmsNorm, QkvSplit, SiluAndMul, Add, IndexSelect/IndexCopy, DFlashBlockAttention). H3's RoPE looks exotic but is plain NeoX rotate_half — only the angles are unusual (three axes off the fp64 position grid), so a per-row cos/sin cache feeds vt::RopeFromCache with no bespoke kernel. That left two indexed AdaLN modulates and an ungated SiLU, in a kMiniMaxH3 glue table mirroring the kLaguna precedent — but registered on both kCPU and kCUDA (Laguna's is CUDA-only), so the whole device path is gated in CPU CI too.

Not bit-identical to the CPU reference, deliberately: vt::RmsNorm reduces in f32 where the reference accumulates in double, and f32 is what upstream torch does. Held to the same goldens instead.

Not yet built — honestly recorded

• The FP4 path, and therefore any speed number vs vLLM-Omni. This needs sm_121a: the GPU these numbers come from is sm_110, which resolves every fp4/cutlass/marlin/fa2 feature DISABLED. No throughput figure is claimed anywhere in this PR.
• bf16 stream policy + vt::FusedChain glue folds on the device forward (the bf16 fold also clears its merged-GEMM allowlist entry, added with a specific reason rather than silently).
• An e2e run on a real checkpoint; VAE/encoder weight loading from real checkpoint files (only the DiT loaders exist today).
• W8 USP multi-GPU.

Open: there is no vllm-omni parity pin — the upstream-sync protocol covers only the vLLM repo, and H3 is both beyond the pin and outside that repository.

Record

.agents/specs/minimax-h3.md, model-matrix row MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit + checklist/rollup, roadmap ROAD-V1-H3, docs/STATUS.md, docs/BENCHMARKS.md, parity-ledger and state entries.

Checkers green on committed HEAD: check-agent-record, check-model-checklist, check-doc-checkpoint, check-readme-structure, check-fusion-consistency, check-runner-routing-consistency, check-device-leakage, check-env-doc, plus the checker mutation suites.

🤖 Generated with Claude Code

[localai-bot]

Update: quantized arms land, and the hardware verdict is corrected

Pushed ad6adc20. The "e2e is impossible on this hardware" verdict in the original PR description is withdrawn — it reasoned from the BF16 release alone (~354 GB, 4× B300). Quantized MiniMax-H3 checkpoints exist and fit one GB10 (119 GiB unified):

Arm	Working set	Fits?
realrebelai/MiniMax-H3_GGUFs	DiT Q3_K_M 15.6 GB + Qwen3-VL encoder Q4_K_M 14.6 GB + VAEs ~11 GB = ~41 GB	yes
lilcheaty/MiniMax-H3-NVFP4	NVFP4 DiT + AWQ encoder + both VAEs	yes

So e2e and a speed comparison are reachable, gated on the remaining bricks (encoder → VAEs → pipeline), not on hardware. NVFP4 is the likely speed path: sm_121 has native FP4 tensor cores and our NVFP4 stack is the most tuned one we own.

What this push adds

1. BF16 production stream. The DiT forward now runs upstream's production dtype policy alongside the f32 parity path — bf16 block stream with the fp32 islands (both patch projections, time embedder, both output heads) preserved. Gated against a bf16 upstream golden at max abs diff 2.4e-3.

2. Request planning (time_request.py + pipeline_minimax_h3.py shape resolution): 17n+5 frame snapping, video/audio latent shapes, the rectified-flow time-shift sigma schedule, canvas resolution, reference-image rescale, and t2va/fl2va/ref2va dispatch. Exact vs upstream. (Python's round() is half-to-even and two of these depend on it.)

3. GGUF arm. The best result here: the name map is the identity. All 535 tensors of MiniMax-H3-FL2VA-Q3_K_M.gguf match the contract EnumerateMiniMaxH3DitTensors derived from upstream source — so the weight contract is now validated against a real checkpoint. Two shape rules:

• GGUF ne is reversed relative to torch;
• comfy.gguf.orig_shape.<name> overrides it where ComfyUI reshaped for quant-block alignment — the 50 AdaLN projections are logical [96768, 2688], and 2688 is not a multiple of the 256-element Q3_K block.

Geometry is derived from shapes alone, since a ComfyUI GGUF ships no transformer config — that's what lets a GGUF load without the original repo.

The gate needed no download: a GGUF header is self-delimiting and at the front of the file, so a 4 MiB HTTP range request yields the whole manifest, which scripts/gen-minimax-h3-gguf-manifest.py freezes into a fixture (names/dims/types/orig_shape, no weight bytes).

Status

test_minimax_h3: 13/13 cases, 3907 assertions, clean CPU build (0 warnings). All checkers + mutation suites green.

Next: download a quantized checkpoint and close the e2e loop (encoder on our existing Qwen3-VL tower → the two VAEs → pipeline), then the NVFP4 arm for speed.

[localai-bot]

Update: the audio VAE is reimplemented (55a9ed4d)

The biggest unknown in this lane is now half-resolved. H3's two VAEs are checkpoint remote code — they ship inside the HF repo (FL2VA/{audio,video}_vae/*.py) and load via get_class_from_dynamic_module under trust_remote_code. vLLM-Omni only adapts them, so a no-Python engine has to reimplement them.

The remote code is now in hand (~130 KB of Python; not vendored here — it ships under the MiniMax H3 Community License), and the audio side is done.

What it is

A DAC-lineage BigVGAN vocoder: dec_in_proj (32→2048, k=1) → conv_pre → 7 upsample stages (ConvTranspose1d, rates 5,5,2,2,2,2,2 / kernels 9,9,4,4,4,4,4), each followed by 3 AMPBlock1 residual stacks (kernels 3,7,11; dilations 1,3,5) whose outputs are averaged → anti-aliased SnakeBeta → conv_post → clamp to [-1, 1]. 32 kHz stereo.

Two things that were easy to get wrong

• Every conv is weight-normalized. The checkpoint stores (g, v) as parametrizations.weight.original0/original1, so the loader materializes g * v / norm(v) with the norm over every dim except dim 0. Note ConvTranspose1d weight is [in, out, k], so its weight-norm dim 0 is the input channel.
• The anti-aliased activation is up-2× → SnakeBeta → down-2× through a kaiser-sinc filter computed at load time, never loaded — needing a Bessel I0, torch's periodic=false Kaiser window, and replicate padding. It's gated separately so a filter bug can't masquerade as a decoder bug.

Gate

scripts/gen-minimax-h3-audio-vae-goldens.py imports the checkpoint's own modules and runs them at reduced dimensions as the oracle, with weights from the shared H3Rand stream (no weight bytes checked in):

	max abs diff
waveform	4.2e-9 (f32 round-off)
kaiser-sinc filter	3.0e-8

One methodology note worth flagging: the first golden had 18 of 32 samples pinned at the final clamp, which would have hidden errors. The weight scale is tuned so the output is fully unsaturated, and the test now asserts non-saturation explicitly.

test_minimax_h3: 14/14 cases, 3983 assertions, clean build, all checkers green.

Remaining

Video VAE (W4) is the largest brick left — klvae.py alone is ~48 KB, plus a CNN/ViT hybrid, tiling, and a parallel path. Then the encoder (W3, mostly reuse of our Qwen3-VL tower) and the pipeline (W6). After those, an e2e run on a quantized checkpoint is reachable — and only then is a speed number meaningful.

[localai-bot]

Update: NVFP4 arm grounded, video VAE re-scoped (f46c9e87)

Two de-risking results on the critical path, both from real checkpoints without downloading any payload. Generalized the GGUF header trick: a safetensors header is also front-loaded (8-byte length + JSON) and only tens of KB even for a 10 GB file, so one range request captures the whole manifest.

1. The NVFP4 checkpoint is our layout, exactly

lilcheaty/MiniMax-H3-NVFP4 (1051 tensors) is the textbook compressed-tensors triple:

tensor	dtype	shape (fused qkv, block 0)
weight	U8	[21504, 2688] — FP4 packed 2-per-byte, logical [21504, 5376]
weight_scale	F8_E4M3	[21504, 336] — one per group of 16 along K
weight_scale_2	F32	scalar global

258 quantized projections, each carrying all three. The fp32/bf16 islands (patch projections, time embedder, output heads, norms, rope) are left unquantized, so the DiT's dtype policy survives. Names identical to the contract derived from source.

So W10 is loader wiring onto the NVFP4 stack we already tuned for Laguna — not a new quant scheme. That matters because this is the speed path: sm_121 has native FP4 tensor cores.

2. The video VAE decoder is a ViT, not a CNN

I had scoped W4 as "port a 48 KB klvae.py" and called it the scary brick. The real 560-tensor manifest says otherwise:

• Encoder = the 3D CNN (116 tensors, rank-5 Conv3d down blocks)
• Decoder = a 36-block transformer (440 tensors: attn.to_qkv/to_out, ff.w1/w2, two norms + two learned residual scales per block, plus x_embedder, mask_token, register_tokens, norm_out, proj_out), fp32 throughout

Generation only needs the decoder. We have every primitive for a ViT, so W4 is materially smaller than previously recorded.

test_minimax_h3: 16/16 cases, 5908 assertions, clean build, all checkers green.

Honest status

Still no e2e result and no speed number. What changed is that the two biggest unknowns left on the critical path are now measured instead of guessed — and both came back smaller than feared. Remaining: W4 video-VAE decoder (ViT port), W3 encoder (Qwen3-VL reuse), W6 pipeline, then W10 loader wiring. Only after those does a speed measurement mean anything.

[localai-bot]

Update: video-VAE decoder block ported (06953d38)

W4 is underway. The repeated unit of the 36-block ViT decoder now matches the checkpoint's own remote code at 6.0e-8.

Per block, all fp32:

h += scale1 * Attention(RMSNorm(h))
h += scale2 * GatedSiLU_FeedForward(RMSNorm(h))

with scale1/scale2 learned per-channel vectors (not scalars) and per-head RMS q/k norm carrying no affine weight.

The trap this caught

This ViT's to_qkv output is per-head interleaved — upstream does qkv.view(B, S, -1, 3*dim_head) then chunk(3, dim=-1), giving [head0_q | head0_k | head0_v | head1_q | ...]. That is not the [q_all | k_all | v_all] the H3 DiT's fused qkv uses.

Both layouts are the same size, so reading it the DiT way raises no error and no shape mismatch — it just produces a plausible-but-wrong image. Only a numeric gate against the real module finds it. Worth flagging for anyone else porting this.

Tooling note

The bundle imports a few diffusers symbols (a logger, two mixin bases, two no-op decorators). The generator stubs them rather than pulling the whole diffusers dependency in just to run an oracle.

test_minimax_h3: 17/17 cases, 5911 assertions, clean build, all checkers green.

Still open

W4 is partial: the 36-block stack surround remains (x_embedder, mask_token/register_tokens, 3D RoPE, norm_out/proj_out, unpatchify, tiling), plus the 3D-CNN encoder — which is only needed for image/video conditioning, not for generation output. Then W3 (encoder), W6 (pipeline), W10 (NVFP4 loader wiring).

No e2e run and no speed number yet, and neither is reachable from this box (no GPU, 90 GB free vs a ~41 GB checkpoint plus build space). The remaining work is a run on the DGX once the pipeline closes.

[localai-bot]

Update: video-VAE ViT3D decoder complete (24fb5231)

Both VAE decoders are now done. The full ViT3D video decoder reproduces the checkpoint's own ViT3DDecoder at 8.9e-8.

Ported surround around the already-gated block:
_pack_tensors_3d → x_embedder → register tokens + a zero cls token → 3D RoPE → 36-block stack → norm_out (Layer norm, while the blocks use RMS) → proj_out → _unpack_tensors_3d → [C, T·pt, H·ps, W·ps].

Run at the real hyperparameters from source/config.json::vit_decoder_kwargs: 36 layers, 32 heads × 64, rms_norm affine, qk rms_norm without affine, gated SiLU, rope_theta 100.0, rope_dim_ratio 0.75.

3D RoPE detail worth recording: RotaryEmbeddingND builds angles from length-normalized token ids ((i+0.5)/n mapped into [-1,1)), scales by 2π (use_angle=True), and concatenates the three per-axis frequency blocks before tiling twice to fill rot_dim. Suffix tokens carry id 0, so their cos/sin are 1/0 — an identity rotation.

VAE gates, all vs the checkpoint's own remote code

	max abs diff
video VAE full ViT3D decoder	8.9e-8
video VAE decoder block	6.0e-8
audio VAE decoder	4.2e-9

test_minimax_h3: 18/18 cases, 5918 assertions, clean build, all checkers green.

What's left

On the VAE side: video tiling (vae_tile_size 256 / overlap 64) and the 3D-CNN encoder — and the encoder is only needed for image/video conditioning (fl2va/ref2va), not for producing output frames, so a t2va path doesn't need it.

On the critical path to e2e: W3 (H3-Encoder on our existing Qwen3-VL tower) and W6 (pipeline). Then W10 NVFP4 loader wiring.

Still no e2e run and no speed number — and neither is reachable from this machine (no GPU). That's a run on the DGX once W3+W6 close.

[localai-bot]

Update: encoder text tower ported (778629b6)

W3 is underway. The H3-Encoder's text tower — which produces the [seq, 5120] prompt_embeds the DiT consumes — matches upstream at 1.2e-7 on both the plain and DeepStack paths.

Its architecture is a Qwen3-VL (which we already port), so the value here is pinning and gating the three H3-specific deltas:

1. Layer truncation — num_layers = min(config.num_hidden_layers, 50). The gate's config deliberately declares more layers than are selected so truncation is actually exercised, plus an assertion it never extends a shallower model.
2. Unnormalized output — the load-bearing one. H3 consumes the state straight out of layer 49 with no final RMSNorm, unlike a stock Qwen3-VL text model. Applying one raises no error and changes no shape — it just silently shifts every conditioning vector.
3. DeepStack — visual features added at visual token positions after each of the first N layers. The test asserts DeepStack actually changes the result, so a no-op injection can't pass.

Oracle note: encoder.py imports exactly one vllm symbol (vllm.logger), so a one-line stub runs it without vllm or any dependency — same pattern as the diffusers stub for the video VAE.

test_minimax_h3: 19/19 cases, 5930 assertions, clean build, all checkers + mutation suites green.

---

Branch status (7 commits)

Component	Gate
packed layout (fl2va + ref2va)	exact, fp64 grid bit-exact
latent packing	exact + round-trip
flow-matching scheduler	exact
DiT forward (f32)	1.6e-7
DiT forward (bf16 production stream)	2.4e-3
request planning	exact
GGUF manifest (535 real tensors)	exact
NVFP4 manifest (1051 real tensors)	exact, layout = ours
audio VAE decoder	4.2e-9
video VAE ViT3D decoder	8.9e-8
encoder text tower	1.2e-7

Remaining: encoder vision tower + MM processor, W6 pipeline assembly, W7 serving/MP4, W10 NVFP4 loader wiring, W2b device-resident forward, video tiling.

Still no e2e run and no speed number — and neither is reachable from this machine (no GPU, 90 GB free vs a ~41 GB checkpoint). W6 is now the main thing between here and a t2va run on the DGX.

[localai-bot]

Milestone: the whole t2va path composes end to end (b549a4ba)

MiniMaxH3GenerateT2va now wires the separately-gated stages into one path:

prompt_embeds → packed layout → sigma schedules (video shift 12, audio 3)
              → denoise loop (one DiT forward per step, euler-eta0 update)
              → unpatchify / audio unpack → per-channel denormalize
              → video ViT3D decoder + audio BigVGAN decoder
              → frames [3, T·pt, H·ps, W·ps] + stereo waveform @ 32 kHz

A structural end-to-end test runs the whole thing at reduced dimensions with random weights. Explicitly not a quality result — it's proof the stages compose: shapes are right, every value finite, the waveform inside [-1, 1], and the denoise loop demonstrably moves the latents rather than passing noise through.

Assembling it caught a real gap

Which is exactly why doing this before the checkpoint arrives was worth it: the audio decode was missing the checkpoint's dec_in_proj (Conv1d k=1, vae_latent_channels → num_mels) ahead of BigVGAN. The DiT emits a 32-wide audio latent while BigVGAN expects 2048 mels — without that projection the two never meet. Fixed, with the standalone BigVGAN gate untouched.

Design note: noise is an input, not generated internally. Upstream seeds a torch CPU generator; matching torch's RNG bit-exactly decides which sample you get, not whether the pipeline is correct — recorded as an open item rather than guessed at.

---

Branch summary — 8 commits

Component	Gate
packed layout (fl2va + ref2va)	exact, fp64 grid bit-exact
latent packing / scheduler / request planning	exact
DiT forward (f32 / bf16 production stream)	1.6e-7 / 2.4e-3
GGUF manifest (535 real tensors)	exact, identity name map
NVFP4 manifest (1051 real tensors)	exact, layout = our stack
audio VAE decoder	4.2e-9
video VAE ViT3D decoder	8.9e-8
encoder text tower	1.2e-7
whole t2va path	composes; correct shapes, finite, in range

test_minimax_h3: 20/20 cases, 6370 assertions, clean build, all checkers + mutation suites green. No weight bytes checked in anywhere.

Remaining

Encoder vision tower (reuse of qwen3_vl_vision.cpp) + MM processor · fl2va/ref2va conditioning · quantized loader wiring (GGUF dequant / NVFP4) · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident forward (W2b) · video tiling.

No run on a real checkpoint and no speed number. Those need a GPU with the ~41 GB checkpoint on it — a DGX operation, not something reachable from this box.

[localai-bot]

GGUF arm complete — load into a runnable DiT (c46b7b8b)

LoadMiniMaxH3DitFromGguf closes W9. It resolves the manifest (identity name map, ne reversal, the comfy.gguf.orig_shape reshape rule), derives the geometry from shapes alone — a ComfyUI GGUF ships no transformer config — and dequantizes every tensor through the shared DequantGgufRowToF32, so the Q2_K/Q3_K/Q4_K families the H3 GGUFs use need no new quant code. Missing tensors throw by name rather than yielding a null view the forward would silently read as zeros.

Gated by a synthetic-file load-and-run test rather than a shape check: geometry recovered exactly, loaded weights carry the logical (torch) shape not the reversed ne, and a real DiT forward executes off them with finite, correctly-sized outputs.

test_minimax_h3: 21/21 cases, 6668 assertions. Full scripts suite: 128 passed. Clean build, all checkers green.

---

Branch complete for this session — 9 commits

Component	Gate
packed layout (fl2va + ref2va)	exact, fp64 grid bit-exact
latent packing / scheduler / request planning	exact
DiT forward (f32 / bf16 production stream)	1.6e-7 / 2.4e-3
audio VAE decoder	4.2e-9
video VAE ViT3D decoder	8.9e-8
encoder text tower	1.2e-7
GGUF manifest (535 real tensors)	exact, identity map
NVFP4 manifest (1051 real tensors)	exact, layout = our stack
GGUF load → runnable DiT	geometry from shapes; forward runs
whole t2va path	composes; correct, finite, in range

No weight bytes checked in anywhere — every gate runs against upstream code, the checkpoint's own remote code, or real checkpoint headers fetched by range request.

What remains

Encoder vision tower (reuse of qwen3_vl_vision.cpp) + MM processor · fl2va/ref2va conditioning · NVFP4 loader wiring (W10 — layout already proven identical to ours) · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident forward (W2b) · video tiling.

Environment

dgx.casa is currently unreachable ("no route to host") and this workstation has no GPU. So a run on a real checkpoint — and therefore any speed number — is not possible from here regardless of how much of the port is finished. That's the gating constraint on the remaining goal, not the code.

[localai-bot]

Encoder vision block ported — both tower cores done (9d8ad9b4)

The repeated unit of the encoder's Qwen3-VL vision tower matches upstream at 6.0e-8.

The two ViTs in this model disagree on qkv layout

Worth stating plainly now that I've ported both:

ViT	qkv layout
video VAE decoder	per-head interleaved — [head][q,k,v]
encoder vision tower	[q_all, k_all, v_all]

Same tensor size, no error either way — just a wrong result. Never assume a qkv layout carries across ViTs inside one checkpoint; read the reshape/permute in the source each time.

Other deltas from the text tower, all exercised: LayerNorm with bias (not RMSNorm), fp32 rotary, tanh-approximate GELU (not exact erf), and cu_seqlens-segmented non-causal attention.

The test proves the segmentation rather than assuming it: perturbing a token in the second packed image leaves every output of the first bit-identical, while the second's do change. A segmentation bug would slip past a plain tolerance check.

test_minimax_h3: 23/23 cases, 7040 assertions; scripts suite 128 passed; clean build, all checkers green.

---

Branch: 11 commits

Component	Gate
packed layout (fl2va + ref2va)	exact, fp64 grid bit-exact
latent packing / scheduler / request planning	exact
DiT forward (f32 / bf16)	1.6e-7 / 2.4e-3
audio VAE decoder	4.2e-9
video VAE ViT3D decoder	8.9e-8
encoder text tower	1.2e-7
encoder vision block	6.0e-8
GGUF + NVFP4 manifests (real checkpoints)	exact, identity map
GGUF load → runnable DiT	geometry from shapes; forward runs
NVFP4 load → runnable DiT	triple dequantized; forward runs
whole t2va path	composes; correct, finite, in range

Remaining

Vision surround (Conv3d patch embed, pos-embed interpolation, 2D rotary table, patch + DeepStack mergers) · MM processor · fl2va/ref2va conditioning · video tiling · /v1/videos + MP4 muxing (needs a dependency decision) · device-resident FP4 forward (where speed actually comes from).

No speed number. dgx.casa is unreachable and this workstation has no GPU.

[localai-bot]

Encoder complete — full vision tower (6d167a7e)

Both encoder towers are now ported end to end: Conv3d patch embed → bilinear-interpolated position embedding → 2D rotary → block stack over per-frame cu_seqlens → DeepStack mergers + final patch merger.

Gated over a ragged two-image batch (different h/w) — which is what actually exercises the position-embedding interpolation and the per-frame segmentation. Merged output and DeepStack features both within 1e-4 of upstream.

Details that were load-bearing

• the patch embed's Conv3d has kernel == stride, so it's a plain linear over the flattened patch — no sliding window
• torch.linspace(0, n-1, 1) returns the start, not the end (single-row grids)
• .int() truncates when picking the bilinear corners — it does not round
• the two merger flavours differ: the final merger norms the pre-shuffle width, the DeepStack mergers norm the post-shuffle width — and both use exact-erf GELU, unlike the block MLP's tanh approximation

Three different GELU/norm conventions inside one encoder. Each of those would have been a silent wrong-result, not an error.

test_minimax_h3: 24/24 cases, 7050 assertions, clean build, all checkers green.

---

Branch: 12 commits

Component	Gate
packed layout (fl2va + ref2va)	exact, fp64 grid bit-exact
latent packing / scheduler / request planning	exact
DiT forward (f32 / bf16)	1.6e-7 / 2.4e-3
audio VAE decoder	4.2e-9
video VAE ViT3D decoder	8.9e-8
encoder text tower	1.2e-7
encoder vision tower (full)	6.0e-8 / ≤1e-4
GGUF + NVFP4 manifests (real checkpoints)	exact, identity map
GGUF → runnable DiT · NVFP4 → runnable DiT	geometry from shapes; forward runs
whole t2va path	composes; correct, finite, in range

No weight bytes checked in anywhere — every gate runs against upstream code, the checkpoint's own remote code, or real checkpoint headers fetched by range request.

Remaining

MM processor (input plumbing) · fl2va/ref2va conditioning · video tiling · /v1/videos + MP4 muxing (blocked on a dependency decision — the tree has no muxer or A/V encoder) · device-resident FP4 forward.

No speed number. The speed work is the device-resident forward; the reference forward is deliberately CPU and unoptimized. dgx.casa is unreachable and this workstation has no GPU, so there is nothing to measure on.

[localai-bot]

Landed on main as 574399c (mudler-authored squash of the 84-commit lane, rebased across today's six landings). Merge gate: all doc/record gates + a clean full CPU -Werror build + full ctest on the merged tree (the two pre-existing plain-CPU -Werror breaks it surfaced landed separately as #61/#62; test_async_llm is the documented parallel-starve flake, passes serially). Next per the lane: the FP4 speed path on GB10 (sm_121).

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys