LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam

[localai-bot]

What

Port LTX-2.5 (Lightricks) — a 21.00B joint video+audio flow-matching DiT — and
generalize the video seam so it, MiniMax-H3, and every future video model share one
surface instead of forking.

Why now

LTX-2.5 released with weights that fit one GB10 comfortably (~29 GB for the whole
NVFP4 arm vs H3's ~41 GB GGUF arm), and its text encoder is Gemma-4 12B, which this
project has already ported (gemma4.cpp, gemma4_weights.cpp). The two largest costs of
a new video model — hardware headroom and a text tower — are already paid.

Geometry (measured from the checkpoint header, not inferred)

Read by range request from vonkaiser/LTX-2.5-FP8-NVFP4
transformer/ltx-2.5-22b-distilled-fp8.safetensors (6124 tensors, 881 KB header):

Field	Value
Params	21.00B (blocks 18.560B + audio connector 2.016B + global 0.427B)
Blocks	48 @ 386.7M
Video stream	hidden 4096, 32 heads x 128
Audio stream	hidden 2048, 32 heads x 64
Per block	attn1, attn2 (cross 4096), audio_attn1, audio_attn2 (cross 2048), audio_to_video_attn, video_to_audio_attn, ff 4096->16384->4096 (no bias), audio_ff 2048->8192->2048 (bias)
Gated attention	to_gate_logits [32, dim] on EVERY attention — one gate per head
AdaLN	scale_shift_table [9, dim]/block; adaln_single.linear [36864, 4096]; audio tower [18432, 2048]; scale_shift_table_a2v_ca_{video,audio} [5, dim]
Prompt AdaLN	prompt_scale_shift_table [2, dim] static pair, no timestep MLP
Latents	patchify_proj [4096,128] / proj_out [128,4096]; audio [2048,128] / [128,2048]
Audio connector	8 x 1-D transformer blocks + learnable_registers [128, 2048]
Quant	F8_E4M3 + per-tensor F32 weight_scale; biases/norms BF16; 1775 FP8 tensors

The ff-no-bias / audio_ff-bias split confirms ff_bias=false + audio_ff_bias=true
in ltx-core/model/transformer/model_configurator.py:78-80 — checkpoint and source agree.

The one free win

LTX-2.5 sets use_prompt_adaln_single=false (model_configurator.py:74-76: "KV-cacheable
checkpoints set use_prompt_adaln_single=false, dropping the timestep-dependence of the
cross-attention K/V so they can be computed once per prompt and reused across steps"), and
the checkpoint carries only the static prompt_scale_shift_table [2, dim] — no prompt-side
timestep MLP. So cross-attention K/V is computed once per request, not per step. Across a
30-step loop that is a property of the checkpoint, not an optimization we must justify.

How it differs from MiniMax-H3

H3 packs every modality into ONE sequence with per-row token tags. LTX-2.5 runs two
separate streams joined by explicit audio<->video cross-attention. Also: gelu-approximate
FFN (H3: SwiGLU), per-head gated attention (H3: none), Gemma-4 TE (H3: Qwen3-VL 32B).

Oracle

vLLM-Omni does not support 2.5 — its recipes key on ("one_stage","2"), ("one_stage","2.3"),
("distilled_two_stage","2") (ltx2_recipes.py:162); upstream issues
vllm-omni#6066 (filed 2026-08-11) and
#4985 are open. Its
DiffusersAdapterPipeline is generic (DiffusionPipeline.from_pretrained), so vLLM-Omni CAN
run 2.5 via --load-format diffusers once Lightricks/LTX-2.5-Diffusers access is granted.

Binding oracle: vLLM-Omni + diffusers adapter. Immediate cross-check: Lightricks
ltx-pipelines, which is available now and keeps work unblocked.

Recorded as owed up front

• Speed gate lands PENDING. The diffusers adapter is a black box
  (supports_step_execution=False, supports_request_batch=False), so a throughput number
  through it is NOT vLLM's production configuration, which AGENTS.md requires as the denominator.
• DiffVAE refused, not silently downgraded. Its neighborhood attention is a new kernel; until
  its row lands, requesting it fails with a message naming the missing piece rather than falling
  back to the Conv VAE.
• No token-exact gate. Diffusion; mirrors H3's ratified reduced-dimension CPU parity approach.

Checkpoint access

Repo	Status (verified via HF API)
Lightricks/LTX-2.5	gated: auto — accepting the license opens it; holds the first-party NVFP4 DiT (18.72 GB)
Lightricks/LTX-2.5-Diffusers	restricted, manual approval — needed for the binding oracle
vonkaiser/LTX-2.5-FP8-NVFP4	ungated — FP8 DiT + NVFP4 Gemma-4 TE; unblocks work today

Checkpoints land in $CHECKPOINT_ROOT (/mnt/nas_share/checkpoints) so dgx and the cluster
nodes share one copy.

Rows

Row	Scope
L0	Spec, checkpoint inventory, oracle stand-up
L1	vllm::multimodal::VideoEngine interface + registry; H3 moved behind it unchanged; ABI v13 additive
L2	DiT layout + forward (dual stream, gated attn, AV cross-attn, split/interleaved RoPE); reduced-dim CPU parity
L3	Gemma-4 12B TE + dual caption projections (4096 video / 2048 audio)
L4	Conv video VAE + audio VAE + vocoder (DiffVAE = own row)
L5	Pipeline: flow-matching recipe, distilled two-stage, latent spatial x2 upsampler, duration head
L6	NVFP4 DiT + NVFP4 TE arms, GB10 load-time residency
L7	e2e on dgx.casa under flock $HOME/gpu.lock, /v1/videos route