feat(ltx-2.5): port the prompt-side AdaLN, and stop the opt-in from clearing it

.agents/roadmap_v1.md

@@ -147,6 +147,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature |
 | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug |
 | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug |
+| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature |
 | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug |
 | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature |
 | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug |

docs/FEATURES.md

@@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending |
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
 | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams |
-| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWED | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8, torchao and first-party NVFP4 all load. DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING |
+| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |
 | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending |

docs/USAGE.md

@@ -2073,11 +2073,14 @@ other marker-less NVFP4 checkpoint as unsupported until it is. See
 `.agents/specs/nvfp4-nibble-order.md`.
 
 Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the
-shipped DiT by default, because that file carries **three** module families phase
-L2 does not port (`prompt_adaln_single`, `audio_prompt_adaln_single` and
-`keyframes_abs_pos_embedding`); pass `Ltx2DitLoadOptions::allow_unported_modules`
+shipped DiT by default, because that file carries **one** module family this port
+does not carry (`keyframes_abs_pos_embedding`); pass
+`Ltx2DitLoadOptions::allow_unported_modules`
 to load the ported subset, which still reports every one of them in
-`Ltx2DitCheckpoint::unported`. The two `*_embeddings_connector` towers are
+`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and
+`audio_prompt_adaln_single` were on that list until 2026-08-13 and are now
+PORTED, so a checkpoint carrying them needs no opt-in on their account, and the
+opt-in no longer disables them. The two `*_embeddings_connector` towers are
 **not** among them and never will be:
 `UnportedFamilies` filters them out at `ltx2_loader.cpp:439` (`LoadedElsewhere`),
 `RefuseUnported`'s own message says so in capitals at `ltx2_loader.cpp:461-464`,

include/vllm/model_executor/models/ltx2.h

@@ -51,8 +51,14 @@
 //     checkpoint: `caption_proj_before_connector=true` puts them in the TEXT
 //     ENCODER, so the DiT has none (model_configurator.py:199-219). They are
 //     phase L3.
-//   - `prompt_adaln_single` (model.py:223-227). LTX-2.5 sets
-//     `use_prompt_adaln_single=false`; see Ltx2PromptKvCache.
+//
+// PORTED 2026-08-13 — `prompt_adaln_single` / `audio_prompt_adaln_single`
+// (model.py:222-227, :252-257), which this list previously carried as unported on
+// the strength of "LTX-2.5 sets use_prompt_adaln_single=false". It does not: the
+// flag defaults TRUE in both references (model.py:77,
+// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185) and the
+// shipped DiT carries the module's tensors. See
+// .agents/specs/ltx25-prompt-adaln.md and issue #644.
 #pragma once
 
 #include <cstdint>
@@ -112,8 +118,12 @@ struct Ltx2DitParams {
   bool double_precision_rope = false;
   bool apply_gated_attention = false;
   bool cross_attention_adaln = false;
-  // model_configurator.py:74-76. FALSE on LTX-2.5, which is what makes the
-  // cross-attention K/V timestep-independent — see Ltx2PromptKvCache.
+  // model_configurator.py:74-76 (`config.get("use_prompt_adaln_single", True)`),
+  // model.py:77, diffusers transformer_ltx2.py:1185 — TRUE by default in every
+  // reference, and TRUE for the shipped LTX-2.5 DiT, which carries the module's
+  // tensors. When true a prompt-side AdaLN MLP adds a timestep term to the
+  // cross-attention K/V modulation (transformer.py:441-443), which is what makes
+  // those K/V timestep-DEPENDENT and so uncacheable — see Ltx2PromptKvCache.
   bool use_prompt_adaln_single = true;
   // model_configurator.py:77-80. LTX-2.5 (gemma4) sets ff_bias=false and leaves
   // audio_ff_bias at its true default; the checkpoint's shapes agree.
@@ -214,6 +224,11 @@ struct Ltx2BlockWeights {
 struct Ltx2DitWeights {
   Ltx2LinearWeight patchify_proj, proj_out;
   Ltx2AdaLayerNormSingleWeights adaln_single;
+  // model.py:222-227 / :252-257 — built only when `cross_attention_adaln AND
+  // use_prompt_adaln_single`, with embedding_coefficient 2 (shift + scale for the
+  // prompt K/V), NOT `adaln_embedding_coefficient()`. Left unbound otherwise.
+  Ltx2AdaLayerNormSingleWeights prompt_adaln_single;
+  Ltx2AdaLayerNormSingleWeights audio_prompt_adaln_single;
   vt::Tensor scale_shift_table;  // [2, dim] — the OUTPUT table (:230), not the block's [9, dim]
   Ltx2LinearWeight audio_patchify_proj, audio_proj_out;
   Ltx2AdaLayerNormSingleWeights audio_adaln_single;
@@ -295,13 +310,19 @@ Ltx2AdalnOut Ltx2AdaLayerNormSingle(vt::Device device, const Ltx2AdaLayerNormSin
 std::vector<float> Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeights& w,
                                    const float* x, int64_t rows, int64_t dim, int64_t inner);
 
-// The K/V half of Attention.forward, split out because LTX-2.5 can CACHE it:
-// with `use_prompt_adaln_single=false` the prompt modulation carries no timestep
-// term (transformer.py:441), so `to_k`/`to_v` over the modulated context — and
-// their k_norm, and the absence of RoPE on the text path — depend only on the
-// prompt. The denoise loop computes them ONCE PER REQUEST and reuses them for
-// every step. Layout: k/v are [batch * context_tokens, heads * dim_head], held
-// exactly as the attention op consumes them (post-norm, post-RoPE).
+// The K/V half of Attention.forward, split out because a checkpoint that sets
+// `use_prompt_adaln_single=false` can CACHE it: the prompt modulation then
+// carries no timestep term (transformer.py:441-443), so `to_k`/`to_v` over the
+// modulated context — and their k_norm, and the absence of RoPE on the text path
+// — depend only on the prompt. The denoise loop computes them ONCE PER REQUEST
+// and reuses them for every step. Layout: k/v are
+// [batch * context_tokens, heads * dim_head], held exactly as the attention op
+// consumes them (post-norm, post-RoPE).
+//
+// THIS DOES NOT APPLY TO THE SHIPPED LTX-2.5 DiT, which sets the flag TRUE
+// (.agents/specs/ltx-2-5.md §1.2, and .agents/specs/ltx25-prompt-adaln.md). The
+// mechanism stays here, gated bit-identical, for a checkpoint that does set it
+// false; `Ltx2DitForward` refuses a cache when the flag is on.
 struct Ltx2CrossKv {
   std::vector<float> k, v;
 };
@@ -463,6 +484,14 @@ struct Ltx2BlockArgs {
   // Per-token AdaLN modulation, [batch, tokens, coefficient * dim].
   const float* video_timestep_modulation = nullptr;
   const float* audio_timestep_modulation = nullptr;
+  // The PROMPT-side AdaLN modulation, [batch, 1, 2 * dim] — shift then scale, one
+  // row per batch element broadcast over the prompt tokens (transformer.py:443,
+  // whose `prompt_timestep` has token dimension 1 because `_prepare_timestep` ran
+  // on the modality's per-sample `sigma`). `nullptr` is upstream's
+  // `prompt_timestep is None`, i.e. `use_prompt_adaln_single=false`, in which case
+  // only the static per-block table applies (:441).
+  const float* video_prompt_modulation = nullptr;
+  const float* audio_prompt_modulation = nullptr;
   // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411).
   const float* video_cross_scale_shift = nullptr;  // [batch, video tokens, 4 * dim]
   const float* video_cross_gate = nullptr;         // [batch, 1, dim]

include/vllm/model_executor/models/ltx2_loader.h

@@ -98,27 +98,38 @@
 // holds both copies at once, which is what wedged the box during MiniMax-H3's
 // port (minimax_h3.h:1598-1606). Same shape, same reason.
 //
-// ─── WHAT THE SHIPPED DiT CARRIES THAT PHASE L2 DOES NOT PORT ────────────────
+// ─── WHAT THE SHIPPED DiT CARRIES THAT THIS PORT DOES NOT CARRY ──────────────
 //
 // MEASURED 2026-08-12 from the FP8 checkpoint's own header, and reported rather
-// than absorbed. The file carries four families outside the L2 contract, and
-// they fall into TWO groups that this comment used to conflate — corrected
-// 2026-08-13, because the conflation is what made a downstream refusal state
-// something untrue about the tree for a whole phase.
+// than absorbed. The file carries four families outside the ORIGINAL L2 contract,
+// and they fall into THREE groups. Two of them this comment once conflated
+// (corrected 2026-08-13, because the conflation made a downstream refusal state
+// something untrue about the tree for a whole phase); the third is a family that
+// has since been ported.
 //
 // UNPORTED. `Ltx2LoadDitFromSafetensors` REFUSES the load by naming these, and
 // only an explicit `allow_unported_modules` — which exists so the ported subset
 // stays gateable — proceeds, still reporting every one of them in `unported`:
 //
-//   prompt_adaln_single.*, audio_prompt_adaln_single.*
-//       Upstream builds these only when `cross_attention_adaln AND
-//       use_prompt_adaln_single` (model.py:222-226, :253-257). Their presence
-//       means the shipped LTX-2.5 sets `use_prompt_adaln_single = TRUE`, which
-//       contradicts .agents/specs/ltx-2-5.md §1.2 and ltx2.h:115-117 — and with
-//       it the prompt-K/V "free win", whose whole premise is that the prompt
-//       modulation carries no timestep term.
 //   keyframes_abs_pos_embedding  [1, 4096]
 //       So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49.
+//       This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its
+//       config copy, and it must stay that way: a flag cleared there is invisible
+//       to the contract-equality check, so clearing a PORTED one silently drops
+//       its tensors.
+//
+// PORTED 2026-08-13 — no longer named in that refusal:
+//
+//   prompt_adaln_single.*, audio_prompt_adaln_single.*
+//       Upstream builds these exactly when `cross_attention_adaln AND
+//       use_prompt_adaln_single` (model.py:222-226, :252-256), and the flag
+//       defaults TRUE in every reference (model.py:77,
+//       model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185). Their
+//       presence means the shipped LTX-2.5 sets it TRUE — which is what
+//       .agents/specs/ltx-2-5.md §1.2 established and what row LTX25-PROMPT-ADALN
+//       (.agents/specs/ltx25-prompt-adaln.md, issue #644) implemented. The
+//       prompt-K/V cache stays correct-and-inapplicable for this checkpoint, and
+//       is refused by name rather than served.
 //
 // LOADED ELSEWHERE — NOT UNPORTED, and never named in that refusal:
 //
@@ -515,11 +526,17 @@ nlohmann::json Ltx2ReadCheckpointConfig(const SafetensorsFile& file);
 // `Ltx2StreamDitToDevice` directly and therefore owes the same adoption.
 //
 // `allow_unported_modules` clears `use_keyframes_abs_pos_embedding` IN A COPY of
-// the config before parsing, mirroring what the loader does for
-// `use_prompt_adaln_single`: the flag is cleared for the CONTRACT, the module
+// the config before parsing: the flag is cleared for the CONTRACT, the module
 // stays unported, and the checkpoint's `unported` list still names it. Without
 // the opt-in `ParseLtx2DitParams` throws, which is the refusal.
 //
+// IT CLEARS EXACTLY THAT ONE FLAG, and the scoping is the rule, not an accident.
+// It also used to clear `use_prompt_adaln_single`, whose module has been ported
+// since 2026-08-13 — so the opt-in a real render REQUIRES was silently turning a
+// correctness setting off, and the contract-equality check below could not see it
+// because both sides had been forced to the same cleared value. A ported module's
+// flag belongs in the contract; only a module nothing applies may be cleared here.
+//
 // `source` names the config in every refusal, so a reader knows whether the
 // checkpoint declared it or a caller supplied it.
 Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config,

include/vllm/multimodal/ltx2_video.h

@@ -170,11 +170,20 @@ inline constexpr char kLtx2ModelVersionExtra[] = "model_version";
 // belonging to another checkpoint is refused rather than bound.
 inline constexpr char kLtx2DitConfigPathExtra[] = "dit_config_path";
 
-// Proceed past the module families the L2 contract does not carry —
-// `prompt_adaln_single` / `audio_prompt_adaln_single` and
+// Proceed past the module families this port does not carry —
 // `keyframes_abs_pos_embedding` (ltx2_loader.h). "1" opts in; anything else
-// leaves the loader's refusal in place. The shipped DiTs all carry at least one
-// of them, so this is the flag that says "gate the ported subset knowingly".
+// leaves the loader's refusal in place. The shipped DiTs carry it, so this is the
+// flag that says "gate the ported subset knowingly".
+//
+// IT MUST NEVER DISABLE A PORTED FEATURE, and until 2026-08-13 it did:
+// `prompt_adaln_single` / `audio_prompt_adaln_single` were on this list, and
+// setting the extra reached three loader assignments that cleared
+// `use_prompt_adaln_single`, so every real render dropped the timestep half of
+// the prompt K/V modulation — finite, same-shaped, and invisible to every gate.
+// Those families are ported now (.agents/specs/ltx25-prompt-adaln.md, issue
+// #644), the loader asserts the flag against the file rather than clearing it,
+// and this extra is scoped to the one module nothing applies. Adding a family
+// here only ever means "the forward genuinely has no code for this".
 //
 // The two `*_embeddings_connector` families are NOT in that set and this extra
 // has nothing to do with them: they are outside the DiT contract by design and