From baa92ccf7be05012a307489a42895c0b4400ac6e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:05:09 +0000 Subject: [PATCH 1/2] spec(ltx-2.5): the prompt-side AdaLN path is DROPPED on every render MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185` and LTX-2 `model.py:77` / `model_configurator.py:76,138` — and the shipped FP8 DiT carries the 18 tensors the flag builds. `ltx2_loader.cpp:988` (and `:573`, `:626`) cleared it unconditionally, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Shapes, finiteness and every committed golden agree with the defect, because the goldens were generated with the flag OFF. Spec only; no code. Records the upstream anchors for the module (`model.py:222-227`, `:252-257`), the producer (`transformer_args.py:274-277`, driven by `modality.sigma` and NOT `timesteps`), the consumer (`transformer.py:441-446`), and what replaces the cleared flag so `allow_unported_modules` can no longer silently disable a ported feature. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 259 ++++++++++++++++++++++++++++ 2 files changed, 260 insertions(+) create mode 100644 .agents/specs/ltx25-prompt-adaln.md diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 3cc1d0234..df766a903 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -147,6 +147,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature | | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | +| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md new file mode 100644 index 000000000..70cb486b5 --- /dev/null +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -0,0 +1,259 @@ +# LTX-2.5 — the prompt-side AdaLN path (`use_prompt_adaln_single`) + +Row: `LTX25-PROMPT-ADALN`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; not +edited by this row). Issue: +[#644](https://github.com/mudler/vllm.cpp/issues/644), row 0. + +Upstream pins: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `fd4ded7f` | +| huggingface/diffusers | `3a2f35d4` | + +Both are read from local checkouts at those revisions, and the golden generator +IMPORTS and EXECUTES the LTX-2 modules rather than restating them. + +## 0. What is wrong today + +`src/vllm/model_executor/models/ltx2_loader.cpp:988` sets, unconditionally: + +```cpp +declared.use_prompt_adaln_single = false; +``` + +and `:573` / `:626` do the same on the two manifest paths. The flag defaults +**TRUE** in both references: + +- diffusers `src/diffusers/models/transformers/transformer_ltx2.py:1185` — + `use_prompt_adaln_single: bool = True` +- LTX-2 `packages/ltx-core/src/ltx_core/model/transformer/model.py:77` — same + default, and `model_configurator.py:76` / `:138` read it as + `config.get("use_prompt_adaln_single", True)` + +The shipped FP8 DiT carries the 18 tensors the flag builds (12 +`prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`; see +`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is +TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those +tensors by name, so a real render needs `allow_unported_modules=1` +(`src/vllm/multimodal/ltx2_video.cpp:570`) — which reaches the loader lines above +and **silently clears the flag**. + +Net effect: every render drops the timestep-conditioned half of the prompt K/V +modulation, keeping only the static `prompt_scale_shift_table`. Nothing observes +it: shapes are unchanged, values stay finite, and the goldens were generated with +`use_prompt_adaln_single=False` (`scripts/gen-ltx2-goldens.py:149`), so the gate +agrees with the defect. + +Campaign history this row does **not** re-derive: `ltx-2-5.md` §1.2 already +RETRACTED the "prompt K/V carry no timestep term" claim and recorded that the +shipped checkpoint carries a `[4096, 256]` prompt timestep embedder. What was +never closed is *using* the tensors. + +## 1. What upstream does, with anchors + +### 1.1 The module + +`model.py:222-227` (video) and `:252-257` (audio): + +```python +self.prompt_adaln_single = ( + AdaLayerNormSingle(self.inner_dim, embedding_coefficient=2) + if self.cross_attention_adaln and self.use_prompt_adaln_single + else None +) +``` + +`AdaLayerNormSingle` (`adaln.py:19-45`) is the same brick the port already has +(`Ltx2AdaLayerNormSingle`): `emb.timestep_embedder.linear_1 [dim, 256]`, +`linear_2 [dim, dim]`, `linear [coefficient * dim, dim]`. Coefficient **2** here, +not `adaln_embedding_coefficient()` — shift and scale for the K/V only. + +diffusers twin: `transformer_ltx2.py:1255-1259`, `num_mod_params=2`. + +Registration order inside `_init_video` puts it between `adaln_single` and +`proj_out`, which is where `EnumerateLtx2DitTensors` already reserves its slot +(the `VT_CHECK` at `ltx2.cpp:274-276`). + +### 1.2 The producer + +`transformer_args.py:274-277`, inside `TransformerArgsPreprocessor.prepare`: + +```python +prompt_timestep = None +if self.prompt_adaln is not None: + prompt_timestep, _ = self._prepare_timestep( + modality.sigma, self.prompt_adaln, batch_size, modality.latent.dtype + ) +``` + +Three things this fixes in one line, each of which a shape check cannot see: + +1. The input is **`modality.sigma`**, `(B,)` (`modality.py:54`) — the per-sample + scalar noise level — **not** `modality.timesteps`, which is per-token `(B, T)`. +2. `_prepare_timestep` (`transformer_args.py:173-186`) multiplies by + `timestep_scale_multiplier` before the embedder, exactly as the port's + `PrepareTimestep` already does for the main AdaLN. +3. The result is viewed to `(B, -1, 2 * dim)`, i.e. `(B, 1, 2 * dim)` — one row + broadcast over the prompt tokens. + +Wired into both preprocessor kinds at `model.py:313`, `:333` (multimodal) and +`:348`, `:364` (single-modality). + +diffusers twin: `transformer_ltx2.py:1536-1547`. diffusers passes `sigma` already +scaled from the pipeline (`pipeline_ltx2_image2video.py:1481` — `sigma=timestep`, +and `timestep` is the scheduler's 0..1000 value), so the two references agree on +the value reaching the embedder; only the place the x1000 happens differs. This +port mirrors LTX-2, so the multiply happens here. + +### 1.3 The consumer + +`transformer.py:427-447` (`apply_cross_attention_adaln`): + +```python +kv_modulation = prompt_scale_shift_table[None, None].to(...) # :441 +if prompt_timestep is not None: # :442 + kv_modulation = kv_modulation + prompt_timestep.reshape( + batch_size, prompt_timestep.shape[1], 2, -1) # :443 +shift_kv, scale_kv = kv_modulation.unbind(dim=2) # :444 +... +encoder_hidden_states = context * (1 + scale_kv) + shift_kv # :446 +``` + +Reached from `_apply_text_cross_attention` (`:223-251`), which is called for the +video stream at `:288-296` and the audio stream at `:317-325`, passing +`video.prompt_timestep` / `audio.prompt_timestep`. Every block, both streams. + +diffusers twin: `transformer_ltx2.py:677-693` (`get_mod_params` over +`prompt_scale_shift_table`), threaded at `:1648-1649`. + +Layout consequence: the flat `[B, 1, 2 * dim]` row is read as `[2, dim]` with +**shift first, scale second** — the same order the static table already uses in +`ModulateContext` (`ltx2_dit.cpp:118-129`). + +## 2. Scope + +**In.** + +1. `Ltx2DitParams::use_prompt_adaln_single` is honoured end to end: contract, + binding, host forward, device forward. +2. The 18 tensors enter `EnumerateLtx2DitTensors` / `BindLtx2DitWeights` when + `cross_attention_adaln && use_prompt_adaln_single`. +3. `temb_prompt` / `temb_prompt_audio` computed from each stream's own `sigma` + and threaded into every block's text cross-attention on both the host + (`ltx2_dit.cpp`) and device (`ltx2_device.cpp`) paths. +4. The three loader `= false` assignments are deleted, and replaced by a guard + (§3.2) that makes a future silent clear impossible. +5. The `ltx2.cpp:274-276` refusal is deleted for these two families. +6. Goldens executed from upstream at reduced dims, with a mutation proving the + new term is load-bearing, and a measured magnitude. + +**Out.** Anything the campaign already records as owed: keyframe absolute +position embedding (still genuinely unported, still what +`allow_unported_modules` is for), the caption projections, guidance +perturbations, and the bf16/FP8/NVFP4 stream dtypes on the host forward. + +## 3. Design + +### 3.1 The seam + +`Ltx2DitWeights` gains two `Ltx2AdaLayerNormSingleWeights` members; +`Ltx2BlockArgs` / `BlockArgsDev` gain a `[batch, 1, 2 * width]` prompt-modulation +pointer per stream, `nullptr` when the flag is off. `ModulateContext` and its +device twin take that pointer and add `prompt_mod[b, {0,1} * width + c]` to the +table row before applying `context * (1 + scale) + shift`. `nullptr` gives the +existing static-only behaviour byte-for-byte, which is what keeps every current +golden valid. + +The **order of the two additions** is upstream's: the table and the timestep row +are summed FIRST (`:443`), and only then does `(1 + scale)` apply. Folding it the +other way would round differently — the same trap `ProcessOutput` +(`ltx2_dit.cpp:530-540`) already documents. + +### 3.2 The prompt-K/V cache, and what replaces the cleared flag + +`Ltx2DitForward` already refuses a cache when the flag is on +(`ltx2_dit.cpp:672-676`); with the flag no longer cleared, that refusal becomes +*reachable* rather than dead, and it is correct: the K/V now carry a timestep +term. Nothing in the shipped pipeline passes a cache (`grep` over +`ltx2_pipeline.cpp` finds none), so no caller regresses. + +**What replaced the `= false`.** The loader clearing existed so +`EnumerateLtx2DitTensors` would not throw. With the tensors ported the contract +simply includes them, so the assignment has no job left. In its place the loader +asserts the invariant the clearing used to violate: + +> the resolved `use_prompt_adaln_single` must equal whether the FILE carries +> `prompt_adaln_single.linear.weight` + +A future edit that re-clears the flag then hits a named refusal instead of +quietly dropping 18 tensors. This is deliberately an *equality*, not a one-sided +check: clearing the flag with the tensors present is the defect this row fixes, +and setting it with the tensors absent would bind missing weights. + +### 3.3 `allow_unported_modules` + +After this row it is scoped to genuinely-unported modules only: the sole flag it +still clears in a config copy is `use_keyframes_abs_pos_embedding` +(`ltx2_loader.cpp:979-984`), whose module really is unported. The guard in §3.2 +is what makes that scoping structural rather than a comment — the extra `= false` +cannot come back without going red. + +`Ltx2AdoptDeclaredDitParams`'s contract-equality check becomes load-bearing in a +second way: a checkpoint whose config declares `use_prompt_adaln_single=false` +while its shapes carry the tensors now produces two DIFFERENT contracts and is +refused, instead of both sides being forced to the same cleared value. + +## 4. Memory format + +Mirrors the existing L2 parity forward exactly: f32 host, f32/bf16 device stream +with the `prompt_scale_shift_table` read at F32 (`ltx2_device.cpp:508-511`). The +prompt modulation is one `[batch, 1, 2 * width]` buffer per stream per forward — +`2 * 4096 * batch` floats for the video stream at full size, computed once +outside the block loop, not per block. No new per-token buffer, so no per-token +byte cost. + +## 5. Tests + +1. `EnumerateLtx2DitTensors` with the flag ON reproduces upstream + `named_parameters()` verbatim — names, order, ranks, dims — for a model built + with `use_prompt_adaln_single=True`. This is the 18-tensor contract. +2. Full dual-stream DiT forward, flag ON, against upstream's executed output. +3. **The mutation**: the same forward with the prompt-AdaLN contribution zeroed + must go RED against that golden. A term that is present but inert is not a + port. +4. The flag-OFF goldens stay byte-identical, proving the new path is off when + upstream's is off. +5. The prompt-K/V cache stays refused with the flag on (existing case). + +## 6. Measured magnitude + +Recorded in §Outcome: the relative change in the modulated prompt context and in +the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer +to "does this matter"; a number below round-off would mean the term is inert and +the whole row is decoration. + +## 7. Risks + +- **The goldens agree with the defect.** Every existing LTX golden was generated + with the flag off, so no existing case can fail whatever this row does. The new + flag-ON case is the only instrument, which is why §5.3 mutates it rather than + asserting it. +- **`sigma` vs `timesteps`.** Using the per-token `timesteps` instead of the + per-sample `sigma` produces a same-shaped, finite, wrong result at batch 1 with + uniform timesteps. The golden runs a batch of 2 with per-token timesteps that + differ from sigma, so the two are distinguishable. +- **Order within the `[2, dim]` row.** Swapping shift and scale is finite and + same-shaped. The golden's random weights make it observable. + +## 8. Stop conditions + +- The flag-ON forward cannot be made to match upstream to the existing + `kRoundOff` bound: stop and report rather than widening the bound. +- The mutation in §5.3 stays green: the path is not reached, and the row is not + done. Escalate the mutation's magnitude before concluding anything about + reachability (issue #604). + +## Now + +`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. From 02038167677dcd8d0c0bece21d11f976130193df Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:54:16 +0000 Subject: [PATCH 2/2] feat(ltx-2.5): port the prompt-side AdaLN, and stop the opt-in from clearing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185`, LTX-2 `model.py:77` and `model_configurator.py:76`/`:138` — and the shipped DiT carries the module's tensors. `ltx2_loader.cpp` cleared it in three places, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Nothing saw it: shapes unchanged, values finite, and every committed golden was generated with the flag OFF. MEASURED, flag ON vs OFF over the same shared weights: the timestep term is 51.7% the magnitude of the static table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and moves the DiT output by 1.46e-4 — 73x the gate's f32 floor. Ported, citing what each piece came from: the module <- model.py:222-227 (video), :252-257 (audio); AdaLayerNormSingle(dim, embedding_coefficient=2), adaln.py:19-45 the producer <- transformer_args.py:274-277 driven by `modality.sigma` [batch] and NOT the per-token `timesteps`, scaled by timestep_scale_multiplier at :173-186 the consumer <- transformer.py:441-446, summed into the table BEFORE `(1 + scale)` applies, threaded per stream at :288-296/:317-325 diffusers twin <- transformer_ltx2.py:1536-1547, :1648-1649, :677-693 Both arms: `Ltx2DitForward` and `Ltx2DitForwardDevice` (f32 and the bf16 production stream). The three `= false` assignments are replaced by an invariant: the resolved flag must equal whether the FILE carries `prompt_adaln_single`. And `Ltx2AdoptDeclaredDitParams` now clears exactly ONE flag in its config copy — `use_keyframes_abs_pos_embedding`, whose module really is unported — so `allow_unported_modules=1`, which a real render still needs, can no longer switch a ported feature off. A config that disagrees with the shapes about the flag is refused rather than forced into agreement. Both shipped DiTs were re-read off the NAS to check this refuses neither: the FP8 file carries no `__metadata__` at all and the NVFP4 file's config OMITS the key, so upstream's TRUE default decides on both, matching their tensors. Evidence. Goldens executed from upstream LTX-2 @fd4ded7f at reduced dims (`gen-ltx2-goldens.py` section 6); every pre-existing golden VALUE is byte-identical. Five mutations, each RED and each restored byte-for-byte: dropping the term host-side (3/35 cases), dropping it device-side (1/15), re-adding the cleared flag (the guard throws; assertion count drops 4826->4815), driving the MLP from `timesteps` instead of `sigma` (2/35), and swapping the shift and scale rows (2/35). A permanent in-tree case asserts the flag-ON forward MISSES the flag-OFF golden, so a bound-but-unused module cannot pass. Suites vs `cefacd2d0`: test_ltx2 30/1627 -> 35/2435, test_ltx2_loader 24/4817 -> 26/4826, test_ltx2_device 13/498 -> 15/523, test_ltx2_video 30/502 -> 30/502 (its fixture now carries the module, which is the shipped shape). Full `ctest` 423 registered, 422 pass under -j8 and `test_serve_low_tools` passes on its serial re-run. Spec .agents/specs/ltx25-prompt-adaln.md @baa92ccf7. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-prompt-adaln.md | 104 ++++- docs/FEATURES.md | 2 +- docs/USAGE.md | 11 +- include/vllm/model_executor/models/ltx2.h | 51 ++- .../vllm/model_executor/models/ltx2_loader.h | 45 +- include/vllm/multimodal/ltx2_video.h | 17 +- scripts/gen-ltx2-goldens.py | 151 +++++- src/vllm/model_executor/models/ltx2.cpp | 16 +- .../model_executor/models/ltx2_device.cpp | 79 +++- src/vllm/model_executor/models/ltx2_dit.cpp | 90 +++- .../model_executor/models/ltx2_loader.cpp | 67 ++- tests/vllm/models/ltx2_goldens.inc | 432 +++++++++++++++++- tests/vllm/models/test_ltx2.cpp | 141 ++++++ tests/vllm/models/test_ltx2_device.cpp | 88 +++- tests/vllm/models/test_ltx2_loader.cpp | 184 ++++++-- tests/vllm/multimodal/ltx2_video_fixture.h | 9 +- tests/vllm/multimodal/test_ltx2_video.cpp | 18 +- 17 files changed, 1381 insertions(+), 124 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 70cb486b5..a24f127ce 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -254,6 +254,108 @@ the whole row is decoration. done. Escalate the mutation's magnitude before concluding anything about reachability (issue #604). +## Outcome + +### What was measured + +The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints +them on stderr, from the SAME shared weight stream on both arms (keyed by +parameter name, so every common weight is bit-identical and the difference is the +term and nothing else): + +| Quantity | Flag ON vs OFF | +|---|---| +| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | +| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | +| DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | + +**The answer to "does this matter" is the first two rows.** Roughly half the +magnitude of the prompt K/V modulation is the timestep-conditioned term, and +including it moves the modulated prompt context by ~6%. Every render before this +row discarded that. + +The two output rows are the GATE's floor, not a claim about the trained +checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and +by a 2-block stack rather than 48. They are reported because a mutation must be +shown to move something, and 73x/37x above round-off is what makes the mutation +below meaningful. + +### The mutations + +All five run on the committed head, restored byte-for-byte afterwards (source +md5s re-checked). Exit status is the authority; assertion COUNTS are recorded +because doctest's summary and the exit code disagree in both directions. + +| # | Mutation | Result | +|---|---|---| +| M1 | host `ModulateContext` ignores `prompt_mod` | RED — 3/35 cases, 6/2435 assertions, exit 1 | +| M2 | device `TextCrossAttentionDev` takes the static-only branch always | RED — 1/15 cases, 6/523 assertions, exit 1 | +| M3 | re-add `use_prompt_adaln_single = false` before the loader guard | RED — the guard throws by name; assertion count DROPS 4826 → 4815, exit 1 | +| M4 | prompt AdaLN driven by `m.timesteps` instead of `m.sigma` | RED — 2/35 cases, 4/2435 assertions, exit 1 | +| M5 | shift and scale rows swapped within the `[2, width]` row | RED — 2/35 cases, 4/2435 assertions, exit 1 | + +M3 is the one that cannot be reached by any INPUT, and that is stated rather than +papered over: `ParseLtx2DitParamsFromManifest` derives the flag from the same +manifest the guard reads, so they agree by construction unless an assignment +intervenes — which is exactly the edit the guard exists to catch. The +input-driven half of the same rule lives in `Ltx2AdoptDeclaredDitParams`, where a +config that disagrees with the shapes now produces two different contracts and is +refused; that one is gated by a test with real inputs in both directions. + +### The gate + +`BUILD_EXIT=0` on every build; build logs grepped for `No space left|BFD assertion` +(0 hits) and `df -h /` logged (88% used, 52G free at the end). Case AND assertion +counts against the `cefacd2d0` baseline, measured by reverting the working tree to +HEAD, rebuilding the four targets and running them, then re-applying the diff and +re-checking its md5 (`03324d42…`, identical before and after): + +| Suite | HEAD `cefacd2d0` | this row | delta | +|---|---|---|---| +| `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | +| `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | +| `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | + +The `test_ltx2_video` fixture had to move: it declared a config that omits +`use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said +false, so the config/shape equality check refused it — correctly. It now carries +the module, which is the shipped shape and puts the whole video engine on the new +path. + +### What was rejected + +- **Widening `modulate`'s kernel contract** with a `rows_per_src_row` divisor, to + express "one row per batch element broadcast over that element's tokens" in one + launch. Rejected: it changes a kernel's semantics for a dimension that is 1 or 2 + in every shipped call, and would owe its own red-before evidence. The device + path loops over batch and offsets the pointers instead. +- **Narrowing the static prompt table to the stream dtype** on the flag-ON path, + which is literally what upstream's `.to(dtype=x_normed.dtype)` does. Rejected as + out of scope: the existing static-only path deliberately keeps the table at F32 + (`ltx2_device.cpp`, "a narrowed table would be the dtype rule applied + backwards"), and changing that polarity is a separate decision. The flag-ON path + routes the sum through `ada_value`, which stores at the stream dtype — the same + rounding every other table+modulation sum in that file already has. + +### Why the defaults are what they are + +`Ltx2DitParams::use_prompt_adaln_single` keeps its `true` default, which is now +honoured rather than overwritten. It matches `model.py:77`, +`model_configurator.py:76`/`:138` and diffusers `transformer_ltx2.py:1185`, and +it matches both shipped DiTs: the FP8 file carries no config at all (so the +default decides), and the NVFP4 file's config OMITS the key — verified by reading +both headers off the NAS. So neither shipped checkpoint is refused by the new +config/shape equality check. + +`allow_unported_modules` keeps existing, because +`keyframes_abs_pos_embedding` is genuinely unported and a real render still needs +the opt-in for it. What changed is that it can no longer switch a ported feature +off: the loader asserts the flag against the file instead of clearing it, and +`Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing +applies. + ## Now -`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. +`DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7586e32ac..3c7ba3e68 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWED | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8, torchao and first-party NVFP4 all load. DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 142cd6a17..4f80a43ef 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2073,11 +2073,14 @@ other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **three** module families phase -L2 does not port (`prompt_adaln_single`, `audio_prompt_adaln_single` and -`keyframes_abs_pos_embedding`); pass `Ltx2DitLoadOptions::allow_unported_modules` +shipped DiT by default, because that file carries **one** module family this port +does not carry (`keyframes_abs_pos_embedding`); pass +`Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. The two `*_embeddings_connector` towers are +`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`audio_prompt_adaln_single` were on that list until 2026-08-13 and are now +PORTED, so a checkpoint carrying them needs no opt-in on their account, and the +opt-in no longer disables them. The two `*_embeddings_connector` towers are **not** among them and never will be: `UnportedFamilies` filters them out at `ltx2_loader.cpp:439` (`LoadedElsewhere`), `RefuseUnported`'s own message says so in capitals at `ltx2_loader.cpp:461-464`, diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index f71a1a404..99898406b 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -51,8 +51,14 @@ // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are // phase L3. -// - `prompt_adaln_single` (model.py:223-227). LTX-2.5 sets -// `use_prompt_adaln_single=false`; see Ltx2PromptKvCache. +// +// PORTED 2026-08-13 — `prompt_adaln_single` / `audio_prompt_adaln_single` +// (model.py:222-227, :252-257), which this list previously carried as unported on +// the strength of "LTX-2.5 sets use_prompt_adaln_single=false". It does not: the +// flag defaults TRUE in both references (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185) and the +// shipped DiT carries the module's tensors. See +// .agents/specs/ltx25-prompt-adaln.md and issue #644. #pragma once #include @@ -112,8 +118,12 @@ struct Ltx2DitParams { bool double_precision_rope = false; bool apply_gated_attention = false; bool cross_attention_adaln = false; - // model_configurator.py:74-76. FALSE on LTX-2.5, which is what makes the - // cross-attention K/V timestep-independent — see Ltx2PromptKvCache. + // model_configurator.py:74-76 (`config.get("use_prompt_adaln_single", True)`), + // model.py:77, diffusers transformer_ltx2.py:1185 — TRUE by default in every + // reference, and TRUE for the shipped LTX-2.5 DiT, which carries the module's + // tensors. When true a prompt-side AdaLN MLP adds a timestep term to the + // cross-attention K/V modulation (transformer.py:441-443), which is what makes + // those K/V timestep-DEPENDENT and so uncacheable — see Ltx2PromptKvCache. bool use_prompt_adaln_single = true; // model_configurator.py:77-80. LTX-2.5 (gemma4) sets ff_bias=false and leaves // audio_ff_bias at its true default; the checkpoint's shapes agree. @@ -214,6 +224,11 @@ struct Ltx2BlockWeights { struct Ltx2DitWeights { Ltx2LinearWeight patchify_proj, proj_out; Ltx2AdaLayerNormSingleWeights adaln_single; + // model.py:222-227 / :252-257 — built only when `cross_attention_adaln AND + // use_prompt_adaln_single`, with embedding_coefficient 2 (shift + scale for the + // prompt K/V), NOT `adaln_embedding_coefficient()`. Left unbound otherwise. + Ltx2AdaLayerNormSingleWeights prompt_adaln_single; + Ltx2AdaLayerNormSingleWeights audio_prompt_adaln_single; vt::Tensor scale_shift_table; // [2, dim] — the OUTPUT table (:230), not the block's [9, dim] Ltx2LinearWeight audio_patchify_proj, audio_proj_out; Ltx2AdaLayerNormSingleWeights audio_adaln_single; @@ -295,13 +310,19 @@ Ltx2AdalnOut Ltx2AdaLayerNormSingle(vt::Device device, const Ltx2AdaLayerNormSin std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeights& w, const float* x, int64_t rows, int64_t dim, int64_t inner); -// The K/V half of Attention.forward, split out because LTX-2.5 can CACHE it: -// with `use_prompt_adaln_single=false` the prompt modulation carries no timestep -// term (transformer.py:441), so `to_k`/`to_v` over the modulated context — and -// their k_norm, and the absence of RoPE on the text path — depend only on the -// prompt. The denoise loop computes them ONCE PER REQUEST and reuses them for -// every step. Layout: k/v are [batch * context_tokens, heads * dim_head], held -// exactly as the attention op consumes them (post-norm, post-RoPE). +// The K/V half of Attention.forward, split out because a checkpoint that sets +// `use_prompt_adaln_single=false` can CACHE it: the prompt modulation then +// carries no timestep term (transformer.py:441-443), so `to_k`/`to_v` over the +// modulated context — and their k_norm, and the absence of RoPE on the text path +// — depend only on the prompt. The denoise loop computes them ONCE PER REQUEST +// and reuses them for every step. Layout: k/v are +// [batch * context_tokens, heads * dim_head], held exactly as the attention op +// consumes them (post-norm, post-RoPE). +// +// THIS DOES NOT APPLY TO THE SHIPPED LTX-2.5 DiT, which sets the flag TRUE +// (.agents/specs/ltx-2-5.md §1.2, and .agents/specs/ltx25-prompt-adaln.md). The +// mechanism stays here, gated bit-identical, for a checkpoint that does set it +// false; `Ltx2DitForward` refuses a cache when the flag is on. struct Ltx2CrossKv { std::vector k, v; }; @@ -463,6 +484,14 @@ struct Ltx2BlockArgs { // Per-token AdaLN modulation, [batch, tokens, coefficient * dim]. const float* video_timestep_modulation = nullptr; const float* audio_timestep_modulation = nullptr; + // The PROMPT-side AdaLN modulation, [batch, 1, 2 * dim] — shift then scale, one + // row per batch element broadcast over the prompt tokens (transformer.py:443, + // whose `prompt_timestep` has token dimension 1 because `_prepare_timestep` ran + // on the modality's per-sample `sigma`). `nullptr` is upstream's + // `prompt_timestep is None`, i.e. `use_prompt_adaln_single=false`, in which case + // only the static per-block table applies (:441). + const float* video_prompt_modulation = nullptr; + const float* audio_prompt_modulation = nullptr; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 1c2da38a5..46aa57faa 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -98,27 +98,38 @@ // holds both copies at once, which is what wedged the box during MiniMax-H3's // port (minimax_h3.h:1598-1606). Same shape, same reason. // -// ─── WHAT THE SHIPPED DiT CARRIES THAT PHASE L2 DOES NOT PORT ──────────────── +// ─── WHAT THE SHIPPED DiT CARRIES THAT THIS PORT DOES NOT CARRY ────────────── // // MEASURED 2026-08-12 from the FP8 checkpoint's own header, and reported rather -// than absorbed. The file carries four families outside the L2 contract, and -// they fall into TWO groups that this comment used to conflate — corrected -// 2026-08-13, because the conflation is what made a downstream refusal state -// something untrue about the tree for a whole phase. +// than absorbed. The file carries four families outside the ORIGINAL L2 contract, +// and they fall into THREE groups. Two of them this comment once conflated +// (corrected 2026-08-13, because the conflation made a downstream refusal state +// something untrue about the tree for a whole phase); the third is a family that +// has since been ported. // // UNPORTED. `Ltx2LoadDitFromSafetensors` REFUSES the load by naming these, and // only an explicit `allow_unported_modules` — which exists so the ported subset // stays gateable — proceeds, still reporting every one of them in `unported`: // -// prompt_adaln_single.*, audio_prompt_adaln_single.* -// Upstream builds these only when `cross_attention_adaln AND -// use_prompt_adaln_single` (model.py:222-226, :253-257). Their presence -// means the shipped LTX-2.5 sets `use_prompt_adaln_single = TRUE`, which -// contradicts .agents/specs/ltx-2-5.md §1.2 and ltx2.h:115-117 — and with -// it the prompt-K/V "free win", whose whole premise is that the prompt -// modulation carries no timestep term. // keyframes_abs_pos_embedding [1, 4096] // So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its +// config copy, and it must stay that way: a flag cleared there is invisible +// to the contract-equality check, so clearing a PORTED one silently drops +// its tensors. +// +// PORTED 2026-08-13 — no longer named in that refusal: +// +// prompt_adaln_single.*, audio_prompt_adaln_single.* +// Upstream builds these exactly when `cross_attention_adaln AND +// use_prompt_adaln_single` (model.py:222-226, :252-256), and the flag +// defaults TRUE in every reference (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185). Their +// presence means the shipped LTX-2.5 sets it TRUE — which is what +// .agents/specs/ltx-2-5.md §1.2 established and what row LTX25-PROMPT-ADALN +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) implemented. The +// prompt-K/V cache stays correct-and-inapplicable for this checkpoint, and +// is refused by name rather than served. // // LOADED ELSEWHERE — NOT UNPORTED, and never named in that refusal: // @@ -515,11 +526,17 @@ nlohmann::json Ltx2ReadCheckpointConfig(const SafetensorsFile& file); // `Ltx2StreamDitToDevice` directly and therefore owes the same adoption. // // `allow_unported_modules` clears `use_keyframes_abs_pos_embedding` IN A COPY of -// the config before parsing, mirroring what the loader does for -// `use_prompt_adaln_single`: the flag is cleared for the CONTRACT, the module +// the config before parsing: the flag is cleared for the CONTRACT, the module // stays unported, and the checkpoint's `unported` list still names it. Without // the opt-in `ParseLtx2DitParams` throws, which is the refusal. // +// IT CLEARS EXACTLY THAT ONE FLAG, and the scoping is the rule, not an accident. +// It also used to clear `use_prompt_adaln_single`, whose module has been ported +// since 2026-08-13 — so the opt-in a real render REQUIRES was silently turning a +// correctness setting off, and the contract-equality check below could not see it +// because both sides had been forced to the same cleared value. A ported module's +// flag belongs in the contract; only a module nothing applies may be cleared here. +// // `source` names the config in every refusal, so a reader knows whether the // checkpoint declared it or a caller supplied it. Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index fb683ce54..0953aebff 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -170,11 +170,20 @@ inline constexpr char kLtx2ModelVersionExtra[] = "model_version"; // belonging to another checkpoint is refused rather than bound. inline constexpr char kLtx2DitConfigPathExtra[] = "dit_config_path"; -// Proceed past the module families the L2 contract does not carry — -// `prompt_adaln_single` / `audio_prompt_adaln_single` and +// Proceed past the module families this port does not carry — // `keyframes_abs_pos_embedding` (ltx2_loader.h). "1" opts in; anything else -// leaves the loader's refusal in place. The shipped DiTs all carry at least one -// of them, so this is the flag that says "gate the ported subset knowingly". +// leaves the loader's refusal in place. The shipped DiTs carry it, so this is the +// flag that says "gate the ported subset knowingly". +// +// IT MUST NEVER DISABLE A PORTED FEATURE, and until 2026-08-13 it did: +// `prompt_adaln_single` / `audio_prompt_adaln_single` were on this list, and +// setting the extra reached three loader assignments that cleared +// `use_prompt_adaln_single`, so every real render dropped the timestep half of +// the prompt K/V modulation — finite, same-shaped, and invisible to every gate. +// Those families are ported now (.agents/specs/ltx25-prompt-adaln.md, issue +// #644), the loader asserts the flag against the file rather than clearing it, +// and this extra is scoped to the one module nothing applies. Adding a family +// here only ever means "the forward genuinely has no code for this". // // The two `*_embeddings_connector` families are NOT in that set and this extra // has nothing to do with them: they are outside the DiT contract by design and diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 367cc4ddb..4c5d3e54b 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -316,10 +316,19 @@ def build_modalities(masked: bool, audio_enabled: bool = True, dense_self_mask: return video, audio -def build_model(rope_type_name: str, double_rope: bool): +def build_model(rope_type_name: str, double_rope: bool, prompt_adaln: bool | None = None): + """`prompt_adaln` overrides ARCH's `use_prompt_adaln_single` for the flag-ON arm. + + Every OTHER parameter keeps its name, and the deterministic stream is keyed by + NAME alone, so the two arms share their common weights bit-for-bit. That is + what makes the flag-ON and flag-OFF forwards directly subtractable, which is + how the magnitude in section 6 is measured. + """ from ltx_core.model.transformer.model import LTXModel, LTXModelType # noqa: PLC0415 from ltx_core.model.transformer.rope import LTXRopeType # noqa: PLC0415 + if prompt_adaln is None: + prompt_adaln = ARCH["use_prompt_adaln_single"] model = LTXModel( model_type=LTXModelType.AudioVideo, num_attention_heads=ARCH["num_attention_heads"], @@ -344,7 +353,7 @@ def build_model(rope_type_name: str, double_rope: bool): double_precision_rope=double_rope, apply_gated_attention=ARCH["apply_gated_attention"], cross_attention_adaln=ARCH["cross_attention_adaln"], - use_prompt_adaln_single=ARCH["use_prompt_adaln_single"], + use_prompt_adaln_single=prompt_adaln, ff_bias=ARCH["ff_bias"], audio_ff_bias=ARCH["audio_ff_bias"], ) @@ -617,12 +626,15 @@ def emit_bricks(out, model) -> None: def emit_forward( out, tag: str, rope_type_name: str, double_rope: bool, masked: bool, audio_enabled: bool = True, dense_self_mask: bool = False, + prompt_adaln: bool | None = None, ) -> None: out.write( f"// --- forward case {tag}: rope={rope_type_name} float64_freqs={double_rope} " - f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} ---\n" + f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} " + f"prompt_adaln={ARCH['use_prompt_adaln_single'] if prompt_adaln is None else prompt_adaln}" + " ---\n" ) - model = build_model(rope_type_name, double_rope) + model = build_model(rope_type_name, double_rope, prompt_adaln) video, audio = build_modalities(masked, audio_enabled, dense_self_mask) with torch.no_grad(): vx, ax = model(video=video, audio=audio, perturbations=None) @@ -631,6 +643,129 @@ def emit_forward( return model +def emit_prompt_adaln(out) -> None: + """Section 6 — the PROMPT-SIDE AdaLN arm (`use_prompt_adaln_single=True`). + + This is upstream's DEFAULT (model.py:77, model_configurator.py:76 and :138; + diffusers transformer_ltx2.py:1185) and what the shipped LTX-2.5 DiT carries. + Everything above this point runs the flag OFF, so nothing above can observe a + port that drops the term — which is exactly what happened. + + Emitted here: + * the flag-ON parameter list, i.e. the 18-tensor contract the flag adds + (12 video + 6 audio), in upstream's own registration order; + * the prompt AdaLN MLP run STANDALONE on `sigma * timestep_scale_multiplier` + (transformer_args.py:274-277 -> :173-186), so a failure localizes to the + MLP rather than to the threading; + * two full forwards, unmasked and masked; + * the MAGNITUDE: flag-ON minus flag-OFF over the same shared weights. + """ + out.write("// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) ---\n") + model = build_model("split", False, prompt_adaln=True) + + names, ranks, dims = [], [], [] + for name, param in model.named_parameters(): + names.append(name) + ranks.append(len(param.shape)) + dims.extend(int(d) for d in param.shape) + out.write("inline constexpr const char* kLtx2PromptAdalnParamNames[] = {\n") + for name in names: + out.write(f' "{name}",\n') + out.write("};\n\n") + emit_i64(out, "kLtx2PromptAdalnParamRanks", ranks) + emit_i64(out, "kLtx2PromptAdalnParamDims", dims) + emit_scalar(out, "kLtx2PromptAdalnParamCount", len(names)) + out.write("\n") + + # The MLP alone. `_prepare_timestep` scales by timestep_scale_multiplier and + # feeds the modality's SIGMA -- (B,), one scalar per sample -- not its + # per-token `timesteps`. Emitting both halves keeps that distinction gateable. + scale = float(ARCH["timestep_scale_multiplier"]) + vsigma = rand_input("input.video.sigma", (BATCH,), 0.25, 0.5) * scale + asigma = rand_input("input.audio.sigma", (BATCH,), 0.25, 0.5) * scale + vmod, _ = model.prompt_adaln_single(vsigma.flatten(), hidden_dtype=torch.float32) + amod, _ = model.audio_prompt_adaln_single(asigma.flatten(), hidden_dtype=torch.float32) + emit_f32(out, "kLtx2PromptAdalnVideoTimesteps", tensor(vsigma)) + emit_f32(out, "kLtx2PromptAdalnAudioTimesteps", tensor(asigma)) + emit_f32(out, "kLtx2PromptAdalnVideoModulation", tensor(vmod)) + emit_f32(out, "kLtx2PromptAdalnAudioModulation", tensor(amod)) + + emit_forward(out, "PromptAdaln", "split", False, False, prompt_adaln=True) + emit_forward(out, "PromptAdalnMasked", "split", False, True, prompt_adaln=True) + + +def measure_prompt_adaln_magnitude() -> str: + """How far the conditioning MOVES when the term is included, vs when it is not. + + Both arms share every common parameter bit-for-bit (the stream is keyed by + parameter NAME), so the difference below is the term itself and nothing else. + Reported as a comment in the generated header AND on stderr, because "does + this matter" is a number, not an argument. + """ + lines = [] + off = build_model("split", False, prompt_adaln=False) + on = build_model("split", False, prompt_adaln=True) + video, audio = build_modalities(False) + with torch.no_grad(): + vx_off, ax_off = off(video=video, audio=audio, perturbations=None) + vx_on, ax_on = on(video=video, audio=audio, perturbations=None) + + def rel(a, b): + a32 = a.to(torch.float32) + b32 = b.to(torch.float32) + denom = float(a32.abs().max()) + return float((b32 - a32).abs().max()), ( + float((b32 - a32).abs().max()) / denom if denom > 0 else float("nan") + ) + + vabs, vrel = rel(vx_off, vx_on) + aabs, arel = rel(ax_off, ax_on) + lines.append(f"// DiT video output: max|on-off| = {vabs:.6g} ({vrel * 100:.2f}% of max|off|)") + lines.append(f"// DiT audio output: max|on-off| = {aabs:.6g} ({arel * 100:.2f}% of max|off|)") + + # The modulated prompt context of block 0, which is where the term enters. + scale = float(ARCH["timestep_scale_multiplier"]) + with torch.no_grad(): + vmod, _ = on.prompt_adaln_single( + (video.sigma * scale).flatten(), hidden_dtype=torch.float32 + ) + vmod = vmod.view(BATCH, -1, vmod.size(-1)) + table = on.transformer_blocks[0].prompt_scale_shift_table[None, None] + static = table.expand(BATCH, 1, 2, video_dim()) + full = static + vmod.reshape(BATCH, vmod.shape[1], 2, -1) + shift_off, scale_off = static.unbind(dim=2) + shift_on, scale_on = full.unbind(dim=2) + ctx = video.context + kv_off = ctx * (1 + scale_off) + shift_off + kv_on = ctx * (1 + scale_on) + shift_on + kabs, krel = rel(kv_off, kv_on) + lines.append( + f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"({krel * 100:.2f}% of max|off|)" + ) + # How much of the K/V modulation is timestep-conditioned at all: the MLP row + # against the static per-block table it is added to (transformer.py:441-443). + static_max = float(table.abs().max()) + term_max = float(vmod.abs().max()) + lines.append( + f"// timestep term vs static table: max|term| = {term_max:.6g} vs " + f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + ) + lines.append( + "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + ) + lines.append( + "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + ) + lines.append( + "// needs, not a claim about the trained checkpoint. The K/V row is where the" + ) + lines.append("// term actually enters and is the number that answers 'does this matter'.") + text = "\n".join(lines) + print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) + return text + + def emit_masks(out) -> None: out.write("// --- section 5: the prompt and self-attention masks the masked case runs ---\n") emit_i64(out, "kLtx2VideoContextMask", video_context_mask()) @@ -680,6 +815,14 @@ def main() -> int: # only this case — separates a kernel that indexes the bias by query from # one that reads bias row 0 for every query. emit_forward(out, "DenseMask", "split", False, True, dense_self_mask=True) + # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. + emit_prompt_adaln(out) + out.write( + "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// Same shared weights, same inputs, flag ON vs OFF:\n" + + measure_prompt_adaln_magnitude() + + "\n" + ) out.write("} // namespace vllm_test\n") print(f"wrote {args.out}", file=sys.stderr) return 0 diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 7e5a4cffc..3d9af69ce 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -268,17 +268,21 @@ std::vector EnumerateLtx2DitTensors(const Ltx2DitParams& p) { out.push_back({"scale_shift_table", {2, dim}}); out.push_back({"audio_scale_shift_table", {2, adim}}); + // model.py:222-226 / :252-256 — built only when BOTH flags hold, and always + // with embedding_coefficient 2 (shift + scale for the prompt K/V), never + // `adaln_embedding_coefficient()`. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + // _init_video (model.py:202-232), in child-registration order. PushLinear(out, "patchify_proj", dim, p.in_channels, true); PushAdaLayerNormSingle(out, "adaln_single", dim, coefficient); - VT_CHECK(!p.use_prompt_adaln_single, - "ltx2: use_prompt_adaln_single=true adds a prompt AdaLN MLP (model.py:223-227) whose " - "timestep term makes the cross-attention K/V uncacheable; not ported in phase L2"); + if (prompt_adaln) PushAdaLayerNormSingle(out, "prompt_adaln_single", dim, 2); PushLinear(out, "proj_out", p.out_channels, dim, true); // _init_audio (model.py:234-262). PushLinear(out, "audio_patchify_proj", adim, p.audio_in_channels, true); PushAdaLayerNormSingle(out, "audio_adaln_single", adim, coefficient); + if (prompt_adaln) PushAdaLayerNormSingle(out, "audio_prompt_adaln_single", adim, 2); PushLinear(out, "audio_proj_out", p.audio_out_channels, adim, true); // _init_audio_video (model.py:264-287); num_scale_shift_values is 4 (:133). @@ -482,9 +486,15 @@ Ltx2DitWeights BindLtx2DitWeights(const Ltx2DitParams& p, w.audio_scale_shift_table = Lookup(t, "audio_scale_shift_table"); w.patchify_proj = BindLinear(t, "patchify_proj", true); w.adaln_single = BindAdaln(t, "adaln_single"); + // model.py:222-226 / :252-256 — bound only when the module exists, exactly as + // the optional block tables above are. Left default-constructed otherwise, and + // the forward reads it only when `use_prompt_adaln_single` says it is there. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + if (prompt_adaln) w.prompt_adaln_single = BindAdaln(t, "prompt_adaln_single"); w.proj_out = BindLinear(t, "proj_out", true); w.audio_patchify_proj = BindLinear(t, "audio_patchify_proj", true); w.audio_adaln_single = BindAdaln(t, "audio_adaln_single"); + if (prompt_adaln) w.audio_prompt_adaln_single = BindAdaln(t, "audio_prompt_adaln_single"); w.audio_proj_out = BindLinear(t, "audio_proj_out", true); w.av_ca_video_scale_shift = BindAdaln(t, "av_ca_video_scale_shift_adaln_single"); w.av_ca_audio_scale_shift = BindAdaln(t, "av_ca_audio_scale_shift_adaln_single"); diff --git a/src/vllm/model_executor/models/ltx2_device.cpp b/src/vllm/model_executor/models/ltx2_device.cpp index 3408007ab..e54436e4e 100644 --- a/src/vllm/model_executor/models/ltx2_device.cpp +++ b/src/vllm/model_executor/models/ltx2_device.cpp @@ -458,6 +458,10 @@ struct BlockArgsDev { bool video_enabled = true, audio_enabled = true; const Tensor* video_timestep_modulation = nullptr; const Tensor* audio_timestep_modulation = nullptr; + // The prompt-side AdaLN modulation, [batch, 2 * width] — nullptr is upstream's + // `prompt_timestep is None` (transformer.py:442), i.e. the flag is off. + const Tensor* video_prompt_modulation = nullptr; + const Tensor* audio_prompt_modulation = nullptr; const Tensor* video_cross_scale_shift = nullptr; const Tensor* video_cross_gate = nullptr; const Tensor* audio_cross_scale_shift = nullptr; @@ -480,6 +484,7 @@ struct BlockArgsDev { // device twin of ltx2_dit.cpp's TextCrossAttention. void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tensor& sst, const Tensor& prompt_table, const Tensor& modulation, + const Tensor* prompt_modulation, const Tensor& x_normed, const Tensor& context, const Tensor* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, @@ -501,18 +506,44 @@ void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tenso c.s, c.s); // apply_cross_attention_adaln (transformer.py:420-447): the STATIC [2, dim] - // per-block table, with no timestep term at all when use_prompt_adaln_single is - // false (:441-443). `src_row_stride = 0` is that broadcast, and the table is - // read at F32 while the stream stays at c.s — which is the whole reason - // `modulate` carries a separate src_dtype. + // per-block table (:441), plus the prompt-side AdaLN row when the flag is on + // (:442-443). VT_CHECK(prompt_table.dtype == DType::kF32, "ltx2 device: the prompt scale-shift table is F32 in the checkpoint and is read as " "F32 here; a narrowed table would be the dtype rule applied backwards"); DBuf encoder(c.d, c.s, {batch * context_tokens, width}); c.d.b.Copy(c.d.q, encoder.ptr(), context.data, encoder.bytes()); - auto* table = prompt_table.Ptr(); - c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, - c.s, DType::kF32); + if (prompt_modulation == nullptr) { + // No timestep term at all. `src_row_stride = 0` is the broadcast of the + // table's single row over every token, and the table is read at F32 while the + // stream stays at c.s — which is the whole reason `modulate` carries a + // separate src_dtype. + auto* table = prompt_table.Ptr(); + c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, + c.s, DType::kF32); + } else { + // `kv_modulation = table[None, None] + prompt_timestep.reshape(B, 1, 2, -1)` + // (:441-443) is EXACTLY `ada_value`'s `table[row] + modulation[r, row]` over + // a 2-parameter modulation with one row per BATCH element, so the sum is + // formed here the way every other table+modulation sum in this file is — + // before `(1 + scale)` applies, which is the order upstream rounds in. + DBuf shift_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/0); + DBuf scale_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/1); + // `modulate`'s `src_row_stride` is a single stride, so it can broadcast ONE + // row over every token (stride 0) or give every row its own (stride width) — + // but not "row b for this batch element's context_tokens rows". The batch + // loop supplies that offset rather than widening the kernel's contract for a + // dimension that is 1 or 2 in every shipped call. + const int64_t elem = static_cast(vt::SizeOf(c.s)); + for (int64_t b = 0; b < batch; ++b) { + void* dst = static_cast(encoder.ptr()) + b * context_tokens * width * elem; + const void* sc = static_cast(scale_kv.t().data) + b * width * elem; + const void* sh = static_cast(shift_kv.t().data) + b * width * elem; + c.k->modulate(c.d.q, dst, sc, sh, context_tokens, width, 0, c.s, c.s); + } + } AttnArgsDev a; a.batch = batch; @@ -575,7 +606,8 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args RmsNormNoWeight(c, vx_normed->t(), *video_x, rows, dim); TextCrossAttentionDev(c, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - *args.video_timestep_modulation, vx_normed->t(), *args.video_context, + *args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed->t(), *args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, c.p->num_attention_heads, c.p->attention_head_dim, *video_x); } @@ -609,6 +641,7 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args TextCrossAttentionDev(c, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, *args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed->t(), *args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, c.p->audio_num_attention_heads, c.p->audio_attention_head_dim, *audio_x); @@ -740,6 +773,8 @@ struct PreparedStreamDev { DevFreqs pe, cross_pe; std::optional cross_scale_shift; std::optional cross_gate; + // [batch, 2 * width] — empty when use_prompt_adaln_single is false. + std::optional prompt_modulation; }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -757,6 +792,7 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross, @@ -774,6 +810,19 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, out.modulation = std::move(ada.modulation); out.embedded = std::move(ada.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + AdalnOutDev prompt = PrepareTimestepDev(c, *prompt_adaln, m.sigma, m.batch, width, + c.p->timestep_scale_multiplier); + out.prompt_modulation = std::move(prompt.modulation); + } + if (m.context != nullptr && m.context_tokens > 0) { out.context = UploadStream(c.d, c.s, m.context, {m.batch * m.context_tokens, context_dim}); } @@ -892,6 +941,10 @@ void CheckWeightsResident(const Ltx2DitWeights& w, vt::Device dev) { CheckLinearResident(w.audio_proj_out, dev, "audio_proj_out"); CheckAdalnResident(w.adaln_single, dev, "adaln_single"); CheckAdalnResident(w.audio_adaln_single, dev, "audio_adaln_single"); + // Bound only when use_prompt_adaln_single is on; `CheckResident` no-ops on an + // unbound view, so this needs no flag and cannot go stale against one. + CheckAdalnResident(w.prompt_adaln_single, dev, "prompt_adaln_single"); + CheckAdalnResident(w.audio_prompt_adaln_single, dev, "audio_prompt_adaln_single"); CheckAdalnResident(w.av_ca_video_scale_shift, dev, "av_ca_video_scale_shift"); CheckAdalnResident(w.av_ca_audio_scale_shift, dev, "av_ca_audio_scale_shift"); CheckAdalnResident(w.av_ca_a2v_gate, dev, "av_ca_a2v_gate"); @@ -1044,14 +1097,18 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param OnesCache ones(d); Ctx c{d, compute_dtype, Glue(d), ¶ms, &ones}; + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; PreparedStreamDev vs = PrepareStreamDev( c, weights.patchify_proj, weights.adaln_single, weights.av_ca_video_scale_shift, - weights.av_ca_a2v_gate, *video, dim, params.in_channels, 3, + weights.av_ca_a2v_gate, prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, + params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, audio, params.cross_attention_dim); PreparedStreamDev as = PrepareStreamDev( c, weights.audio_patchify_proj, weights.audio_adaln_single, weights.av_ca_audio_scale_shift, - weights.av_ca_v2a_gate, *audio, adim, params.audio_in_channels, 1, + weights.av_ca_v2a_gate, prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, + adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, video, params.audio_cross_attention_dim); @@ -1066,6 +1123,8 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param a.audio_enabled = audio->enabled; a.video_timestep_modulation = &vs.modulation->t(); a.audio_timestep_modulation = &as.modulation->t(); + a.video_prompt_modulation = vs.prompt_modulation ? &vs.prompt_modulation->t() : nullptr; + a.audio_prompt_modulation = as.prompt_modulation ? &as.prompt_modulation->t() : nullptr; a.video_cross_scale_shift = vs.cross_scale_shift ? &vs.cross_scale_shift->t() : nullptr; a.video_cross_gate = vs.cross_gate ? &vs.cross_gate->t() : nullptr; a.audio_cross_scale_shift = as.cross_scale_shift ? &as.cross_scale_shift->t() : nullptr; diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index 0abd15fe3..b21655c64 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -114,18 +114,45 @@ void PostSelfAttention(float* x, const float* y, const std::vector& gate, } // apply_cross_attention_adaln (transformer.py:420-447). `prompt_table` is the -// STATIC [2, dim] per-block table; with use_prompt_adaln_single=false there is no -// timestep term at all (:441-443), which is exactly what makes the resulting K/V -// cacheable across denoise steps. +// STATIC [2, width] per-block table (:441). `prompt_mod` is the prompt-side AdaLN +// MLP's output for this stream, [batch, 2 * width] — shift row then scale row, +// one row per BATCH element because `_prepare_timestep` ran on the modality's +// per-sample `sigma` (transformer_args.py:274-277). It is nullptr exactly when +// upstream's `prompt_timestep is None` (:442), i.e. use_prompt_adaln_single=false +// — which is what makes the resulting K/V cacheable across denoise steps. +// +// ORDER. Upstream sums the table and the timestep row FIRST and only then +// applies `(1 + scale)` (:441-446). Folding the two additions the other way round +// would round differently, so the sum is materialized here as upstream forms it. std::vector ModulateContext(const float* context, const vt::Tensor& prompt_table, - int64_t rows, int64_t width) { - const float* shift_kv = prompt_table.Ptr(); - const float* scale_kv = prompt_table.Ptr() + width; - std::vector out(static_cast(rows * width)); - for (int64_t r = 0; r < rows; ++r) { - const float* src = context + r * width; - float* dst = out.data() + r * width; - for (int64_t c = 0; c < width; ++c) dst[c] = src[c] * (1.0f + scale_kv[c]) + shift_kv[c]; + const float* prompt_mod, int64_t batch, + int64_t context_tokens, int64_t width) { + const float* table_shift = prompt_table.Ptr(); + const float* table_scale = prompt_table.Ptr() + width; + std::vector out(static_cast(batch * context_tokens * width)); + std::vector shift_kv(static_cast(width)); + std::vector scale_kv(static_cast(width)); + for (int64_t b = 0; b < batch; ++b) { + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] = table_shift[c]; + scale_kv[static_cast(c)] = table_scale[c]; + } + if (prompt_mod != nullptr) { + const float* m = prompt_mod + b * 2 * width; + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] += m[c]; + scale_kv[static_cast(c)] += m[width + c]; + } + } + for (int64_t s = 0; s < context_tokens; ++s) { + const int64_t r = b * context_tokens + s; + const float* src = context + r * width; + float* dst = out.data() + r * width; + for (int64_t c = 0; c < width; ++c) { + dst[c] = src[c] * (1.0f + scale_kv[static_cast(c)]) + + shift_kv[static_cast(c)]; + } + } } return out; } @@ -148,7 +175,8 @@ void AddGatedBroadcast(float* x, const std::vector& y, const std::vector< void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, const Ltx2AttentionWeights& attn, const vt::Tensor& sst, const vt::Tensor& prompt_table, const float* modulation, - const float* x_normed, const float* context, const float* context_bias, + const float* prompt_modulation, const float* x_normed, + const float* context, const float* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, int64_t dim_head, const Ltx2CrossKv* kv_in, Ltx2CrossKv* kv_out, float* x) { @@ -174,7 +202,8 @@ void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, // The modulated context is only needed when the K/V are actually recomputed. std::vector encoder; if (kv_in == nullptr) { - encoder = ModulateContext(context, prompt_table, batch * context_tokens, width); + encoder = ModulateContext(context, prompt_table, prompt_modulation, batch, context_tokens, + width); } Ltx2AttentionArgs a; @@ -250,7 +279,8 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, PostSelfAttention(video_x, msa.data(), gate, batch * tv, dim, eps, &vx_normed); TextCrossAttention(device, params, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - args.video_timestep_modulation, vx_normed.data(), args.video_context, + args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed.data(), args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, params.num_attention_heads, params.attention_head_dim, args.prompt_kv_filled ? args.video_prompt_kv : nullptr, @@ -287,6 +317,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, TextCrossAttention(device, params, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed.data(), args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, params.audio_num_attention_heads, params.audio_attention_head_dim, @@ -432,6 +463,7 @@ struct PreparedStream { Ltx2FreqsCis cross_pe; std::vector cross_scale_shift; // [batch, tokens, 4 * width] std::vector cross_gate; // [batch, 1, width] + std::vector prompt_modulation; // [batch, 1, 2 * width], empty when the flag is off }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -451,6 +483,7 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross) { @@ -478,6 +511,21 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, PrepareTimestep(device, adaln, m.timesteps, rows, width, params.timestep_scale_multiplier, &out.modulation, &out.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. `_prepare_timestep` + // applies the same timestep_scale_multiplier, and the result views to + // [batch, 1, 2 * width]: one row per sample, broadcast over the prompt tokens. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + std::vector unused; + PrepareTimestep(device, *prompt_adaln, m.sigma, m.batch, width, + params.timestep_scale_multiplier, &out.prompt_modulation, &unused); + } + if (m.context_mask != nullptr) { out.context_bias = Ltx2PrepareContextMask(m.context_mask, m.batch, m.context_tokens); } @@ -675,19 +723,25 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, "timestep term and caching them would be wrong"); } + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold, and + // `prompt_adaln=getattr(self, "prompt_adaln_single", None)` (:313, :333) is how + // upstream turns its absence into `prompt_timestep is None`. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; const bool have_both = video != nullptr && audio != nullptr; PreparedStream vs, as; if (video != nullptr) { VT_CHECK(video->context_tokens == 0 || video->context != nullptr, "ltx2: the video stream needs a context when context_tokens > 0"); vs = PrepareStream(device, params, weights.patchify_proj, weights.adaln_single, - weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, *video, dim, + weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, + prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, have_both ? audio : nullptr); } if (audio != nullptr) { as = PrepareStream(device, params, weights.audio_patchify_proj, weights.audio_adaln_single, - weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, *audio, adim, + weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, + prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, have_both ? video : nullptr); } @@ -719,6 +773,10 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, a.audio_enabled = audio != nullptr && audio->enabled; a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); + a.video_prompt_modulation = + vs.prompt_modulation.empty() ? nullptr : vs.prompt_modulation.data(); + a.audio_prompt_modulation = + as.prompt_modulation.empty() ? nullptr : as.prompt_modulation.data(); a.video_cross_scale_shift = vs.cross_scale_shift.empty() ? nullptr : vs.cross_scale_shift.data(); a.video_cross_gate = vs.cross_gate.empty() ? nullptr : vs.cross_gate.data(); a.audio_cross_scale_shift = as.cross_scale_shift.empty() ? nullptr : as.cross_scale_shift.data(); diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index 946771019..bfc85075c 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -540,18 +540,53 @@ std::vector ContractOf(const Ltx2DitParams& params) { list += families[i]; } Fail( - "the checkpoint carries modules phase L2 does NOT port: " + list + - ". They are not dropped silently: prompt_adaln_single / " - "audio_prompt_adaln_single mean use_prompt_adaln_single is TRUE, which " - "contradicts .agents/specs/ltx-2-5.md section 1.2 and voids the prompt-K/V " - "cache's premise; keyframes_abs_pos_embedding contradicts ltx2.h:47-49. The " - "two *_embeddings_connector families are NOT in this list and never will be " - "— they are outside the DiT contract by design and are loaded by " + "the checkpoint carries modules this port does NOT carry: " + list + + ". They are not dropped silently: keyframes_abs_pos_embedding means " + "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " + "— they were ported by row LTX25-PROMPT-ADALN " + "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " + "contract whenever the checkpoint carries them. The two " + "*_embeddings_connector families are not in this list either and never will " + "be — they are outside the DiT contract by design and are loaded by " "Ltx2LoadConnectorWeights, which is what the video engine calls. Pass " "Ltx2DitLoadOptions::allow_unported_modules to load the ported SUBSET, which " "still reports every one of them."); } +// THE GUARD THAT REPLACED THREE `use_prompt_adaln_single = false` ASSIGNMENTS. +// +// Those assignments existed only so `EnumerateLtx2DitTensors` would not throw on +// a module this port did not carry. The module is carried now, so the contract +// simply includes it — and the assignments would have become a silent DROP of the +// module's 12 parameters (18 entries in the shipped FP8 manifest, which carries a +// `weight_scale` per quantized weight), reachable through +// `allow_unported_modules=1`, which is exactly the defect issue #644 row 0 fixes. +// +// So the invariant they violated is asserted instead: the resolved flag must say +// what the FILE says. Deliberately an EQUALITY. Clearing it with the tensors +// present is the old defect; setting it with them absent would bind weights that +// are not there. Either way this refuses by name rather than rendering. +void CheckPromptAdalnAgreesWithFile(const DitPlan& plan, const Ltx2DitParams& params, + const char* where) { + bool file_has = false; + for (const Ltx2TensorSpec& spec : plan.manifest) { + if (spec.name == "prompt_adaln_single.linear.weight") { + file_has = true; + break; + } + } + if (file_has == params.use_prompt_adaln_single) return; + Fail(std::string(where) + ": use_prompt_adaln_single resolved to " + + (params.use_prompt_adaln_single ? "TRUE" : "FALSE") + " while the file " + + (file_has ? "DOES" : "does NOT") + + " carry prompt_adaln_single. Upstream builds that module exactly when the flag is set " + "(model.py:222-226), so the two cannot disagree. A FALSE flag over a file that carries " + "the module would drop the timestep term from every cross-attention K/V modulation " + "(transformer.py:441-443) and render with only the static table — finite, same-shaped, " + "and invisible to every gate, which is why this is checked rather than assumed."); +} + } // namespace Ltx2DitParams Ltx2ParseDitParamsFromCheckpoint(const SafetensorsFile& file, @@ -568,9 +603,7 @@ Ltx2DitCheckpoint Ltx2LoadDitFromSafetensors(const SafetensorsFile& file, out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - // The one flag whose module this port does not carry. Cleared for the CONTRACT - // only; `checkpoint_params` keeps what the file actually says. - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2LoadDitFromSafetensors"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -623,7 +656,7 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2StreamDitToDevice"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -976,16 +1009,20 @@ Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, // first-party LTX-2.5 DiT declares it, so reading the declared config verbatim // would refuse a real checkpoint the loader has just accepted under // `allow_unported_modules`. + // + // EXACTLY ONE FLAG, and that is now structural rather than a comment. This block + // also cleared `use_prompt_adaln_single`, whose module IS ported + // (.agents/specs/ltx25-prompt-adaln.md, issue #644) — so `allow_unported=1`, + // which a real render needs, silently turned off a correctness setting. Whatever + // is cleared here must be a module nothing below applies; a ported one belongs + // in the contract, where the equality check further down can see it. if (allow_unported_modules && copy.contains("transformer") && copy["transformer"].is_object()) { copy["transformer"]["use_keyframes_abs_pos_embedding"] = false; } nlohmann::json wrapper; wrapper["config"] = copy; - Ltx2DitParams declared = ParseLtx2DitParams(wrapper); - // The one flag the L2 contract clears, mirroring the manifest path above - // ("cleared for the CONTRACT only"). - declared.use_prompt_adaln_single = false; + const Ltx2DitParams declared = ParseLtx2DitParams(wrapper); const std::vector a = EnumerateLtx2DitTensors(from_shapes); const std::vector b = EnumerateLtx2DitTensors(declared); diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index ff1c964eb..964047788 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -2583,7 +2583,7 @@ inline constexpr float kLtx2AudioSelfMaskDense[] = { }; // --- section 4: the full DiT forward --- -// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardSplitVideo[] = { -0.112128824f, -0.0224671941f, -0.0198019743f, -0.0801180974f, -0.00492008496f, -0.0997069329f, -0.141085327f, -0.110280119f, 0.242049828f, 0.0161605272f, 0.0691268668f, -0.128027037f, @@ -2622,7 +2622,7 @@ inline constexpr float kLtx2ForwardSplitAudio[] = { -0.0946981385f, 0.0834118277f, 0.000482047704f, -0.244372234f, 0.0668168291f, 0.076439999f, }; -// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardInterleavedVideo[] = { -0.111679666f, -0.0226908159f, -0.019702062f, -0.0793902054f, -0.00504480582f, -0.0992663726f, -0.141819194f, -0.110411108f, 0.240116999f, 0.0159209836f, 0.0681923628f, -0.130362436f, @@ -2661,7 +2661,7 @@ inline constexpr float kLtx2ForwardInterleavedAudio[] = { -0.0936588645f, 0.0829930827f, -0.000500017079f, -0.244381592f, 0.0684131756f, 0.0764773339f, }; -// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardDoubleVideo[] = { -0.112128653f, -0.0224669855f, -0.0198022574f, -0.0801180974f, -0.00491989311f, -0.0997067019f, -0.141085491f, -0.110280029f, 0.242049828f, 0.0161606614f, 0.0691268519f, -0.128027067f, @@ -2700,7 +2700,7 @@ inline constexpr float kLtx2ForwardDoubleAudio[] = { -0.0946978033f, 0.0834115744f, 0.000482207892f, -0.244371802f, 0.0668166652f, 0.0764404312f, }; -// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False --- +// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardMaskedVideo[] = { -0.110146172f, -0.0195347276f, -0.0187677145f, -0.0824092105f, -0.00312865246f, -0.102393486f, -0.140672266f, -0.111012168f, 0.238716885f, 0.0195573885f, 0.0744788125f, -0.129781172f, @@ -2739,7 +2739,7 @@ inline constexpr float kLtx2ForwardMaskedAudio[] = { -0.0945852846f, 0.0829883665f, 0.000665711937f, -0.24394846f, 0.0671078935f, 0.0765393525f, }; -// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False --- +// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardAudioOffVideo[] = { -0.11211022f, -0.0225408655f, -0.0197366029f, -0.0800089315f, -0.00495309103f, -0.099537693f, -0.141150832f, -0.110280804f, 0.242115393f, 0.0160462279f, 0.0691971257f, -0.127955094f, @@ -2778,7 +2778,7 @@ inline constexpr float kLtx2ForwardAudioOffAudio[] = { -0.0867422372f, 0.0840105489f, -0.000306644681f, -0.244602948f, 0.0686045066f, 0.0746730566f, }; -// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True --- +// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True prompt_adaln=False --- inline constexpr float kLtx2ForwardDenseMaskVideo[] = { -0.112229049f, -0.0196566191f, -0.0203980505f, -0.08254648f, -0.00351214409f, -0.102313384f, -0.139395878f, -0.10990762f, 0.244468585f, 0.0156435464f, 0.0681985766f, -0.126972139f, @@ -2817,4 +2817,424 @@ inline constexpr float kLtx2ForwardDenseMaskAudio[] = { -0.0944952741f, 0.0835703015f, 6.12687072e-05f, -0.244034111f, 0.0671091005f, 0.0762160644f, }; +// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) --- +inline constexpr const char* kLtx2PromptAdalnParamNames[] = { + "scale_shift_table", + "audio_scale_shift_table", + "patchify_proj.weight", + "patchify_proj.bias", + "adaln_single.emb.timestep_embedder.linear_1.weight", + "adaln_single.emb.timestep_embedder.linear_1.bias", + "adaln_single.emb.timestep_embedder.linear_2.weight", + "adaln_single.emb.timestep_embedder.linear_2.bias", + "adaln_single.linear.weight", + "adaln_single.linear.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "prompt_adaln_single.linear.weight", + "prompt_adaln_single.linear.bias", + "proj_out.weight", + "proj_out.bias", + "audio_patchify_proj.weight", + "audio_patchify_proj.bias", + "audio_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_adaln_single.linear.weight", + "audio_adaln_single.linear.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_prompt_adaln_single.linear.weight", + "audio_prompt_adaln_single.linear.bias", + "audio_proj_out.weight", + "audio_proj_out.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_video_scale_shift_adaln_single.linear.weight", + "av_ca_video_scale_shift_adaln_single.linear.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_audio_scale_shift_adaln_single.linear.weight", + "av_ca_audio_scale_shift_adaln_single.linear.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_a2v_gate_adaln_single.linear.weight", + "av_ca_a2v_gate_adaln_single.linear.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_v2a_gate_adaln_single.linear.weight", + "av_ca_v2a_gate_adaln_single.linear.bias", + "transformer_blocks.0.scale_shift_table", + "transformer_blocks.0.audio_scale_shift_table", + "transformer_blocks.0.scale_shift_table_a2v_ca_audio", + "transformer_blocks.0.scale_shift_table_a2v_ca_video", + "transformer_blocks.0.prompt_scale_shift_table", + "transformer_blocks.0.audio_prompt_scale_shift_table", + "transformer_blocks.0.attn1.q_norm.weight", + "transformer_blocks.0.attn1.k_norm.weight", + "transformer_blocks.0.attn1.to_q.weight", + "transformer_blocks.0.attn1.to_q.bias", + "transformer_blocks.0.attn1.to_k.weight", + "transformer_blocks.0.attn1.to_k.bias", + "transformer_blocks.0.attn1.to_v.weight", + "transformer_blocks.0.attn1.to_v.bias", + "transformer_blocks.0.attn1.to_gate_logits.weight", + "transformer_blocks.0.attn1.to_gate_logits.bias", + "transformer_blocks.0.attn1.to_out.0.weight", + "transformer_blocks.0.attn1.to_out.0.bias", + "transformer_blocks.0.attn2.q_norm.weight", + "transformer_blocks.0.attn2.k_norm.weight", + "transformer_blocks.0.attn2.to_q.weight", + "transformer_blocks.0.attn2.to_q.bias", + "transformer_blocks.0.attn2.to_k.weight", + "transformer_blocks.0.attn2.to_k.bias", + "transformer_blocks.0.attn2.to_v.weight", + "transformer_blocks.0.attn2.to_v.bias", + "transformer_blocks.0.attn2.to_gate_logits.weight", + "transformer_blocks.0.attn2.to_gate_logits.bias", + "transformer_blocks.0.attn2.to_out.0.weight", + "transformer_blocks.0.attn2.to_out.0.bias", + "transformer_blocks.0.ff.net.0.proj.weight", + "transformer_blocks.0.ff.net.2.weight", + "transformer_blocks.0.audio_attn1.q_norm.weight", + "transformer_blocks.0.audio_attn1.k_norm.weight", + "transformer_blocks.0.audio_attn1.to_q.weight", + "transformer_blocks.0.audio_attn1.to_q.bias", + "transformer_blocks.0.audio_attn1.to_k.weight", + "transformer_blocks.0.audio_attn1.to_k.bias", + "transformer_blocks.0.audio_attn1.to_v.weight", + "transformer_blocks.0.audio_attn1.to_v.bias", + "transformer_blocks.0.audio_attn1.to_gate_logits.weight", + "transformer_blocks.0.audio_attn1.to_gate_logits.bias", + "transformer_blocks.0.audio_attn1.to_out.0.weight", + "transformer_blocks.0.audio_attn1.to_out.0.bias", + "transformer_blocks.0.audio_attn2.q_norm.weight", + "transformer_blocks.0.audio_attn2.k_norm.weight", + "transformer_blocks.0.audio_attn2.to_q.weight", + "transformer_blocks.0.audio_attn2.to_q.bias", + "transformer_blocks.0.audio_attn2.to_k.weight", + "transformer_blocks.0.audio_attn2.to_k.bias", + "transformer_blocks.0.audio_attn2.to_v.weight", + "transformer_blocks.0.audio_attn2.to_v.bias", + "transformer_blocks.0.audio_attn2.to_gate_logits.weight", + "transformer_blocks.0.audio_attn2.to_gate_logits.bias", + "transformer_blocks.0.audio_attn2.to_out.0.weight", + "transformer_blocks.0.audio_attn2.to_out.0.bias", + "transformer_blocks.0.audio_ff.net.0.proj.weight", + "transformer_blocks.0.audio_ff.net.0.proj.bias", + "transformer_blocks.0.audio_ff.net.2.weight", + "transformer_blocks.0.audio_ff.net.2.bias", + "transformer_blocks.0.audio_to_video_attn.q_norm.weight", + "transformer_blocks.0.audio_to_video_attn.k_norm.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.bias", + "transformer_blocks.0.audio_to_video_attn.to_k.weight", + "transformer_blocks.0.audio_to_video_attn.to_k.bias", + "transformer_blocks.0.audio_to_video_attn.to_v.weight", + "transformer_blocks.0.audio_to_video_attn.to_v.bias", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.0.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.0.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.0.video_to_audio_attn.q_norm.weight", + "transformer_blocks.0.video_to_audio_attn.k_norm.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.bias", + "transformer_blocks.0.video_to_audio_attn.to_k.weight", + "transformer_blocks.0.video_to_audio_attn.to_k.bias", + "transformer_blocks.0.video_to_audio_attn.to_v.weight", + "transformer_blocks.0.video_to_audio_attn.to_v.bias", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.0.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.0.video_to_audio_attn.to_out.0.bias", + "transformer_blocks.1.scale_shift_table", + "transformer_blocks.1.audio_scale_shift_table", + "transformer_blocks.1.scale_shift_table_a2v_ca_audio", + "transformer_blocks.1.scale_shift_table_a2v_ca_video", + "transformer_blocks.1.prompt_scale_shift_table", + "transformer_blocks.1.audio_prompt_scale_shift_table", + "transformer_blocks.1.attn1.q_norm.weight", + "transformer_blocks.1.attn1.k_norm.weight", + "transformer_blocks.1.attn1.to_q.weight", + "transformer_blocks.1.attn1.to_q.bias", + "transformer_blocks.1.attn1.to_k.weight", + "transformer_blocks.1.attn1.to_k.bias", + "transformer_blocks.1.attn1.to_v.weight", + "transformer_blocks.1.attn1.to_v.bias", + "transformer_blocks.1.attn1.to_gate_logits.weight", + "transformer_blocks.1.attn1.to_gate_logits.bias", + "transformer_blocks.1.attn1.to_out.0.weight", + "transformer_blocks.1.attn1.to_out.0.bias", + "transformer_blocks.1.attn2.q_norm.weight", + "transformer_blocks.1.attn2.k_norm.weight", + "transformer_blocks.1.attn2.to_q.weight", + "transformer_blocks.1.attn2.to_q.bias", + "transformer_blocks.1.attn2.to_k.weight", + "transformer_blocks.1.attn2.to_k.bias", + "transformer_blocks.1.attn2.to_v.weight", + "transformer_blocks.1.attn2.to_v.bias", + "transformer_blocks.1.attn2.to_gate_logits.weight", + "transformer_blocks.1.attn2.to_gate_logits.bias", + "transformer_blocks.1.attn2.to_out.0.weight", + "transformer_blocks.1.attn2.to_out.0.bias", + "transformer_blocks.1.ff.net.0.proj.weight", + "transformer_blocks.1.ff.net.2.weight", + "transformer_blocks.1.audio_attn1.q_norm.weight", + "transformer_blocks.1.audio_attn1.k_norm.weight", + "transformer_blocks.1.audio_attn1.to_q.weight", + "transformer_blocks.1.audio_attn1.to_q.bias", + "transformer_blocks.1.audio_attn1.to_k.weight", + "transformer_blocks.1.audio_attn1.to_k.bias", + "transformer_blocks.1.audio_attn1.to_v.weight", + "transformer_blocks.1.audio_attn1.to_v.bias", + "transformer_blocks.1.audio_attn1.to_gate_logits.weight", + "transformer_blocks.1.audio_attn1.to_gate_logits.bias", + "transformer_blocks.1.audio_attn1.to_out.0.weight", + "transformer_blocks.1.audio_attn1.to_out.0.bias", + "transformer_blocks.1.audio_attn2.q_norm.weight", + "transformer_blocks.1.audio_attn2.k_norm.weight", + "transformer_blocks.1.audio_attn2.to_q.weight", + "transformer_blocks.1.audio_attn2.to_q.bias", + "transformer_blocks.1.audio_attn2.to_k.weight", + "transformer_blocks.1.audio_attn2.to_k.bias", + "transformer_blocks.1.audio_attn2.to_v.weight", + "transformer_blocks.1.audio_attn2.to_v.bias", + "transformer_blocks.1.audio_attn2.to_gate_logits.weight", + "transformer_blocks.1.audio_attn2.to_gate_logits.bias", + "transformer_blocks.1.audio_attn2.to_out.0.weight", + "transformer_blocks.1.audio_attn2.to_out.0.bias", + "transformer_blocks.1.audio_ff.net.0.proj.weight", + "transformer_blocks.1.audio_ff.net.0.proj.bias", + "transformer_blocks.1.audio_ff.net.2.weight", + "transformer_blocks.1.audio_ff.net.2.bias", + "transformer_blocks.1.audio_to_video_attn.q_norm.weight", + "transformer_blocks.1.audio_to_video_attn.k_norm.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.bias", + "transformer_blocks.1.audio_to_video_attn.to_k.weight", + "transformer_blocks.1.audio_to_video_attn.to_k.bias", + "transformer_blocks.1.audio_to_video_attn.to_v.weight", + "transformer_blocks.1.audio_to_video_attn.to_v.bias", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.1.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.1.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.1.video_to_audio_attn.q_norm.weight", + "transformer_blocks.1.video_to_audio_attn.k_norm.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.bias", + "transformer_blocks.1.video_to_audio_attn.to_k.weight", + "transformer_blocks.1.video_to_audio_attn.to_k.bias", + "transformer_blocks.1.video_to_audio_attn.to_v.weight", + "transformer_blocks.1.video_to_audio_attn.to_v.bias", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.1.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.1.video_to_audio_attn.to_out.0.bias", +}; + +inline constexpr int64_t kLtx2PromptAdalnParamRanks[] = { + 2, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamDims[] = { + 2, 32, 2, 16, 32, 8, 32, 32, 256, 32, 32, 32, + 32, 288, 32, 288, 32, 256, 32, 32, 32, 32, 64, 32, + 64, 8, 32, 8, 16, 6, 16, 16, 256, 16, 16, 16, + 16, 144, 16, 144, 16, 256, 16, 16, 16, 16, 32, 16, + 32, 6, 16, 6, 32, 256, 32, 32, 32, 32, 128, 32, + 128, 16, 256, 16, 16, 16, 16, 64, 16, 64, 32, 256, + 32, 32, 32, 32, 32, 32, 32, 16, 256, 16, 16, 16, + 16, 16, 16, 16, 9, 32, 9, 16, 5, 16, 5, 32, + 2, 32, 2, 16, 32, 32, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 4, 32, 4, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, + 32, 32, 128, 32, 32, 128, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 4, 16, 4, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, + 4, 16, 16, 16, 64, 16, 64, 16, 64, 16, 16, 16, + 16, 32, 16, 16, 16, 16, 16, 16, 16, 4, 32, 4, + 32, 16, 32, 16, 16, 16, 16, 16, 16, 32, 16, 16, + 32, 16, 4, 16, 4, 16, 16, 16, 9, 32, 9, 16, + 5, 16, 5, 32, 2, 32, 2, 16, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, + 4, 32, 4, 32, 32, 32, 128, 32, 32, 128, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, 4, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 4, 16, 4, 16, 16, 16, 64, 16, 64, 16, + 64, 16, 16, 16, 16, 32, 16, 16, 16, 16, 16, 16, + 16, 4, 32, 4, 32, 16, 32, 16, 16, 16, 16, 16, + 16, 32, 16, 16, 32, 16, 4, 16, 4, 16, 16, 16, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamCount = 226; + +inline constexpr float kLtx2PromptAdalnVideoTimesteps[] = { + 619.314514f, 424.717133f, +}; + +inline constexpr float kLtx2PromptAdalnAudioTimesteps[] = { + 473.940033f, 471.390289f, +}; + +inline constexpr float kLtx2PromptAdalnVideoModulation[] = { + 0.00287261885f, 0.0172509346f, 0.0149077997f, 0.00213927543f, 0.016918283f, 0.00749109872f, + 0.00878333207f, -0.00887233205f, -0.0209352672f, 0.0127304699f, 0.0109749483f, -0.0130382832f, + -0.00827212445f, -0.00580324279f, -0.0134638241f, 0.00964590162f, -0.016003741f, -0.0158182271f, + -0.00942324009f, -0.0159717295f, -0.00885281898f, -0.00412264559f, -0.00142920704f, -0.0168404803f, + 0.0133440131f, -0.0136787146f, 0.0163368266f, 0.00325167179f, 0.0156327151f, 0.000542653725f, + -0.0113121318f, -0.00703316741f, -0.0195021238f, 0.0151473908f, 0.0180853736f, 0.0211514644f, + 0.000123613572f, -0.00695834123f, -0.00497911917f, -0.0171213523f, 0.0199449155f, -0.0122921225f, + 0.0164053645f, 0.000113433227f, 0.00943795405f, -0.0121485656f, 0.00261268346f, 0.00783557631f, + -0.00460897852f, 0.0153093971f, 0.0194506291f, 0.014570266f, -0.0162125602f, -0.00318641192f, + -0.013586767f, 0.00904892385f, -0.0157831647f, -0.0182319973f, 0.0170140117f, 0.00303793233f, + -0.0181853306f, -0.00718653016f, 0.0158703439f, 0.00020046602f, 0.00623062998f, 0.0252011884f, + 0.0163403824f, 0.000290016062f, 0.014817901f, 0.00552449375f, 0.00866025314f, -0.00905592274f, + -0.0175954159f, 0.0110378982f, 0.0159901753f, -0.0115343379f, -0.0125738885f, -0.00668510282f, + -0.0144314552f, 0.0128096947f, -0.0149616096f, -0.0173609052f, -0.0133896163f, -0.0159855876f, + -0.00701024011f, -0.00505556632f, -0.0011741718f, -0.0174259786f, 0.00958774798f, -0.0130143911f, + 0.0113152023f, 0.00158250146f, 0.0137644717f, 0.00236273953f, -0.0104373507f, -0.000455798348f, + -0.0210349094f, 0.0129983025f, 0.0220505167f, 0.0202870686f, 0.000495404762f, -0.00149141857f, + -0.00326095428f, -0.0143762864f, 0.0244831145f, -0.01158146f, 0.00986448769f, 0.000591582619f, + 0.013319497f, -0.0125091095f, 0.00394352386f, 0.0113571035f, -0.00364819146f, 0.013945261f, + 0.0238441341f, 0.0148018515f, -0.0194255728f, -0.00321829435f, -0.0185660385f, 0.0112335477f, + -0.0152186789f, -0.0156262573f, 0.0185506362f, 0.00343540264f, -0.0173967108f, -0.00660126004f, + 0.0198770091f, -0.000762573676f, +}; + +inline constexpr float kLtx2PromptAdalnAudioModulation[] = { + 0.0187722091f, -0.00105935254f, -0.00853663217f, 0.00841550902f, -0.0137612084f, -0.0173807591f, + -0.0139704188f, -0.0109588308f, 0.00186792947f, -0.00266267685f, -0.00698805135f, -0.0129980808f, + 0.00670515466f, -0.0131825786f, 0.0064191781f, 0.00632017246f, -0.0122615304f, 0.00804066937f, + 0.0122477906f, 0.0157948378f, 0.0128126917f, -0.00316316052f, -0.00883637834f, -0.0100117959f, + -0.0133155594f, 0.00456165429f, 0.0183124971f, -0.0130120357f, 0.018200513f, 0.00201772549f, + 0.0119188903f, -0.0103679141f, 0.0201762542f, -0.000246715499f, -0.00900617056f, 0.00820883084f, + -0.0132663827f, -0.0180532821f, -0.0145739987f, -0.0113234669f, 0.00138104265f, -0.00278934441f, + -0.00686988048f, -0.013545204f, 0.00675359881f, -0.0138096726f, 0.00569557305f, 0.00596735114f, + -0.0114674894f, 0.00765935751f, 0.0126606431f, 0.0161463916f, 0.0141690942f, -0.00382941356f, + -0.0089492118f, -0.00970011577f, -0.0132083837f, 0.00381002505f, 0.0185851343f, -0.0123645896f, + 0.0179123916f, 0.00177870644f, 0.0127155371f, -0.0096385479f, +}; + +// --- forward case PromptAdaln: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnVideo[] = { + -0.112144604f, -0.0224400591f, -0.0198658258f, -0.0801219419f, -0.00492565054f, -0.0996806622f, + -0.141095862f, -0.110312767f, 0.242095038f, 0.0162332151f, 0.0690327138f, -0.127996832f, + -0.327992439f, -0.067813158f, -0.0181498341f, 0.0133159151f, -0.285277247f, 0.0288095567f, + -0.103010967f, 0.0120137185f, 0.175781652f, -0.133603424f, -0.20695813f, -0.182875797f, + -0.275936812f, -0.0216972958f, 0.194740638f, -0.250197232f, -0.196145147f, -0.0676094294f, + -0.115069151f, -0.0885243863f, -0.0418822765f, 0.120427184f, -0.0467535444f, -0.166882023f, + -0.145086944f, -0.127097756f, -0.125268698f, -0.0937946662f, -0.10143815f, -0.0346935503f, + 0.347262383f, -0.0154392701f, 0.0253714621f, -0.151571572f, 0.0893032998f, -0.217105299f, + -0.113787532f, 0.107526913f, 0.111182854f, -0.0174661856f, 0.0259091742f, -0.199445173f, + -0.0387741439f, -0.157782555f, 0.159147635f, 0.0321634449f, 0.232984737f, -0.122400187f, + -0.27637139f, -0.0666981712f, 0.0339653119f, -0.140525416f, 0.0675620735f, -0.120439962f, + 0.0212921053f, 0.112628169f, 0.190132841f, -0.144298837f, -0.0116734616f, -0.159263715f, + 0.0784623772f, -0.179399416f, -0.328205913f, -0.0860405788f, -0.0919508785f, -0.0483957306f, + -0.30280292f, 0.158935979f, -0.127778456f, -0.174414024f, 0.315983474f, -0.0398868211f, + -0.11850293f, 0.0652811453f, -0.0993542522f, -0.234619945f, -0.284468323f, -0.183528736f, + 0.225024208f, -0.049902156f, -0.0650495142f, -0.0445851982f, -0.193018377f, -0.149498582f, + -0.0479458123f, -0.0666453466f, -0.0906915069f, 0.0227309261f, 0.0560047664f, -0.146587655f, + -0.172089651f, -0.0502583086f, 0.0590079799f, 0.19477278f, 0.218966454f, -0.112773709f, + -0.16504398f, -0.111266837f, 0.15254502f, -0.120294176f, 0.114031032f, -0.0505410358f, + -0.150275424f, -0.024936026f, -0.139517337f, -0.00708480086f, -0.160885945f, 0.0312023032f, + 0.27590549f, 0.0162312202f, -0.389412105f, -0.066493772f, -0.005505112f, -0.114737555f, + -0.136138007f, 0.0470205061f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnAudio[] = { + -0.00688289292f, 0.0557257049f, -0.169716612f, 0.136525676f, -0.0794674978f, -0.106710114f, + -0.109623641f, 0.0357984938f, -0.0222658273f, -0.0561756417f, 0.145198658f, 0.0478727296f, + 0.0297143031f, 0.0544307008f, -0.205607757f, 0.122264318f, -0.105997331f, -0.191682264f, + -0.0355675966f, 0.0435401984f, -0.107154176f, 0.192362309f, -0.0537107848f, -0.080251269f, + -0.0339412056f, -0.0676042289f, -0.161326066f, -0.0567146167f, 0.166392133f, -0.0363745391f, + -0.00992792659f, 0.0993538871f, -0.196374968f, 0.146272391f, -0.0314919427f, -0.147439599f, + 0.00373091619f, 0.119020537f, -0.16169925f, 0.149572372f, -0.0535437092f, -0.17913793f, + -0.0729758665f, 0.0560201257f, -0.103747681f, 0.0615522824f, 0.147424534f, -0.0767256021f, + -0.0455891043f, -0.074104555f, -0.0908109695f, 0.14269349f, 0.00909528229f, 0.000857369974f, + -0.094678238f, 0.0833884105f, 0.000516908942f, -0.244370729f, 0.0668335408f, 0.0764729604f, +}; + +// --- forward case PromptAdalnMasked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnMaskedVideo[] = { + -0.110144161f, -0.0195061471f, -0.0188482404f, -0.0824035481f, -0.00313981529f, -0.102331489f, + -0.140647486f, -0.111037448f, 0.238786951f, 0.019598389f, 0.0744047314f, -0.129760861f, + -0.326935619f, -0.0698587522f, -0.0180070139f, 0.0129257301f, -0.282629013f, 0.0306178499f, + -0.100754961f, 0.00948145799f, 0.175529778f, -0.135220498f, -0.206774727f, -0.182601333f, + -0.275893211f, -0.0184815805f, 0.198321715f, -0.251868665f, -0.194811657f, -0.0697973669f, + -0.114349291f, -0.089783825f, -0.0404667407f, 0.121526636f, -0.0449657179f, -0.170914561f, + -0.144608453f, -0.129623294f, -0.126025274f, -0.0928756446f, -0.0987707227f, -0.0303842705f, + 0.349234581f, -0.0163635593f, 0.0282549523f, -0.152388036f, 0.0889601111f, -0.216030046f, + -0.112322196f, 0.108959578f, 0.112902433f, -0.0188521128f, 0.0273712836f, -0.20055984f, + -0.0387175046f, -0.156802967f, 0.156722441f, 0.0369930752f, 0.239927039f, -0.124359809f, + -0.273358971f, -0.0692257583f, 0.0352873057f, -0.141093776f, 0.0673692524f, -0.120450936f, + 0.0237387791f, 0.111967109f, 0.187116802f, -0.139871866f, -0.011316698f, -0.153532162f, + 0.0786181018f, -0.177252695f, -0.331035018f, -0.0842280909f, -0.0938364938f, -0.0409911275f, + -0.301013172f, 0.167324454f, -0.124323212f, -0.174904421f, 0.316507936f, -0.0429520756f, + -0.120540164f, 0.067215845f, -0.101051688f, -0.225887731f, -0.286542743f, -0.182181612f, + 0.224588186f, -0.049729839f, -0.0685416833f, -0.0383053645f, -0.189322844f, -0.138665244f, + -0.0496713445f, -0.068905741f, -0.093425855f, 0.0234508831f, 0.0515494831f, -0.140059441f, + -0.166952908f, -0.041921705f, 0.0591960102f, 0.194034085f, 0.216313154f, -0.111525752f, + -0.165100992f, -0.10801211f, 0.152435049f, -0.115520097f, 0.110954985f, -0.0485403165f, + -0.152800724f, -0.0228928309f, -0.138780624f, -0.00368307461f, -0.157897815f, 0.0359253138f, + 0.277324349f, 0.0178819038f, -0.393882304f, -0.0665938482f, -0.00776502956f, -0.109243833f, + -0.134122461f, 0.0536325052f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { + -0.00623655878f, 0.0550939925f, -0.169768706f, 0.134756476f, -0.0800285041f, -0.106801011f, + -0.109707221f, 0.0349297225f, -0.0223909412f, -0.0580404438f, 0.145440862f, 0.0479248241f, + 0.0300626066f, 0.0543725975f, -0.205051288f, 0.121049382f, -0.106699832f, -0.19155553f, + -0.0357390642f, 0.043708574f, -0.106640562f, 0.191323146f, -0.0545797087f, -0.0801633745f, + -0.0338453911f, -0.067704387f, -0.161020443f, -0.0588866882f, 0.16616787f, -0.0362463593f, + -0.00972275995f, 0.0983785167f, -0.197862387f, 0.146195069f, -0.0301159211f, -0.147994176f, + 0.00437853113f, 0.118179567f, -0.162635818f, 0.14993386f, -0.0523924604f, -0.179551288f, + -0.0724292099f, 0.0543062873f, -0.103864521f, 0.0613403916f, 0.14883852f, -0.0763024017f, + -0.0452010371f, -0.0744607076f, -0.0918518007f, 0.142673746f, 0.0106131276f, 5.35193831e-05f, + -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, +}; + +// --- the MEASURED magnitude of the prompt-AdaLN term --- +// Same shared weights, same inputs, flag ON vs OFF: +// DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) +// DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) +// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// The two output rows are bounded by this generator's SYNTHETIC weight scale +// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate +// needs, not a claim about the trained checkpoint. The K/V row is where the +// term actually enters and is the number that answers 'does this matter'. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index dd6a8e077..f00771f38 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -135,6 +135,16 @@ Ltx2DitParams ReducedParams(Ltx2RopeType rope_type, bool double_precision) { return p; } +// Upstream's DEFAULT arm (model.py:77, model_configurator.py:76/:138, diffusers +// transformer_ltx2.py:1185) and the one the shipped LTX-2.5 DiT carries: the +// prompt-side AdaLN MLP is built, so the cross-attention K/V modulation carries a +// timestep term on top of the static per-block table (transformer.py:441-443). +Ltx2DitParams ReducedParamsPromptAdaln(Ltx2RopeType rope_type, bool double_precision) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = true; + return p; +} + // A materialized weight set: owned f32 storage plus the views the forward takes. struct WeightSet { std::map> storage; @@ -831,6 +841,137 @@ TEST_CASE("ltx2 forward: a disabled audio stream still feeds audio->video") { /*audio_enabled=*/false); } +// --------------------------------------------------------------------------- +// The prompt-side AdaLN arm — upstream's DEFAULT +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) +// --------------------------------------------------------------------------- + +namespace { + +// The same forward, run with `use_prompt_adaln_single = true`. Kept separate from +// CheckForward rather than folded into it, because the two arms have DIFFERENT +// weight contracts (12 extra parameters) and sharing one helper would hide which +// contract a case ran under. +vllm::Ltx2DitOutputs RunPromptAdalnForward(const Ltx2DitParams& p, WeightSet& set, + Modalities* m, bool masked) { + BuildModalities(m, masked); + return Ltx2DitForward(Cpu(), p, set.weights, &m->video, &m->audio, vt::DType::kF32); +} + +void CheckPromptAdalnForward(bool masked, const float* want_video, const float* want_audio, + const char* label) { + INFO("prompt-AdaLN forward case: " << std::string(label)); + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, masked); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, want_video, vcount); + const double adiff = MaxAbsDiff(out.audio, want_audio, acount); + MESSAGE("max|diff| video=" << vdiff << " audio=" << adiff); + CHECK(vdiff < kRoundOff); + CHECK(adiff < kRoundOff); +} + +} // namespace + +// The 12 parameters the flag adds (6 per stream: three linears x weight+bias), +// which the shipped FP8 checkpoint carries as 18 manifest entries because each +// quantized weight also has a `weight_scale`. Order matters as much as presence: +// upstream registers `prompt_adaln_single` between `adaln_single` and `proj_out` +// inside `_init_video` (model.py:222-232), and the audio twin in `_init_audio` +// (:252-262). +TEST_CASE("ltx2 layout: the flag-ON contract matches upstream named_parameters()") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + const std::vector manifest = EnumerateLtx2DitTensors(p); + REQUIRE(static_cast(manifest.size()) == vllm_test::kLtx2PromptAdalnParamCount); + // The flag is the ONLY difference, so the count delta is exactly the module. + CHECK(vllm_test::kLtx2PromptAdalnParamCount - vllm_test::kLtx2ParamCount == 12); + size_t dim_cursor = 0; + for (size_t i = 0; i < manifest.size(); ++i) { + CAPTURE(i); + CAPTURE(manifest[i].name); + CHECK(manifest[i].name == std::string(vllm_test::kLtx2PromptAdalnParamNames[i])); + const int64_t rank = vllm_test::kLtx2PromptAdalnParamRanks[i]; + REQUIRE(static_cast(manifest[i].shape.size()) == rank); + for (int64_t d = 0; d < rank; ++d) { + CHECK(manifest[i].shape[static_cast(d)] == + vllm_test::kLtx2PromptAdalnParamDims[dim_cursor]); + ++dim_cursor; + } + } +} + +// The MLP on its own, so a failure localizes here rather than in the threading. +// The input is the modality's SIGMA scaled by timestep_scale_multiplier +// (transformer_args.py:274-277 -> :173-186) — one scalar per batch element, NOT +// the per-token `timesteps` the main AdaLN consumes. +TEST_CASE("ltx2 brick: the prompt AdaLN MLP runs on sigma, not on the per-token timesteps") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + const int64_t b = vllm_test::kLtx2Batch; + const int64_t dim = p.inner_dim(); + const int64_t adim = p.audio_inner_dim(); + + std::vector vts(vllm_test::kLtx2PromptAdalnVideoTimesteps, + vllm_test::kLtx2PromptAdalnVideoTimesteps + b); + const vllm::Ltx2AdalnOut vout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.prompt_adaln_single, vts.data(), b, dim); + REQUIRE(vout.modulation.size() == static_cast(b * 2 * dim)); + CHECK(MaxAbsDiff(vout.modulation, vllm_test::kLtx2PromptAdalnVideoModulation, + static_cast(b * 2 * dim)) < kRoundOff); + + std::vector ats(vllm_test::kLtx2PromptAdalnAudioTimesteps, + vllm_test::kLtx2PromptAdalnAudioTimesteps + b); + const vllm::Ltx2AdalnOut aout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.audio_prompt_adaln_single, ats.data(), b, adim); + REQUIRE(aout.modulation.size() == static_cast(b * 2 * adim)); + CHECK(MaxAbsDiff(aout.modulation, vllm_test::kLtx2PromptAdalnAudioModulation, + static_cast(b * 2 * adim)) < kRoundOff); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm") { + CheckPromptAdalnForward(false, vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, "prompt AdaLN"); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { + CheckPromptAdalnForward(true, vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, + "prompt AdaLN + masks"); +} + +// THE INSTRUMENT THAT MAKES THE ARM ABOVE MEAN SOMETHING. +// +// Both goldens come from the same deterministic weight stream, keyed by parameter +// NAME, so every weight the two arms share is bit-identical and the ONLY thing +// separating `kLtx2ForwardPromptAdaln*` from `kLtx2ForwardSplit*` is the timestep +// term. A port that accepted the flag, bound the 12 tensors and then never added +// their output would reproduce the flag-OFF numbers exactly and pass nothing here. +// +// Measured on this fixture (generator stderr, and the comment block at the end of +// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block +// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and +// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound +// below is set at 20x kRoundOff so it is comfortably inside the measured signal +// and comfortably outside f32 noise. +TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, false); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, vllm_test::kLtx2ForwardSplitVideo, vcount); + const double adiff = MaxAbsDiff(out.audio, vllm_test::kLtx2ForwardSplitAudio, acount); + MESSAGE("flag-ON vs flag-OFF: video=" << vdiff << " audio=" << adiff); + CHECK(vdiff > 20.0 * kRoundOff); + CHECK(adiff > 20.0 * kRoundOff); +} + TEST_CASE("ltx2 forward: a single-stream model type is REFUSED") { const Ltx2DitParams p = ReducedParams(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/models/test_ltx2_device.cpp b/tests/vllm/models/test_ltx2_device.cpp index 46bea0b78..cafef9ceb 100644 --- a/tests/vllm/models/test_ltx2_device.cpp +++ b/tests/vllm/models/test_ltx2_device.cpp @@ -304,8 +304,9 @@ struct CaseResult { CaseResult RunDeviceCase(vt::Queue& q, vt::DType stream, Ltx2RopeType rope_type, bool double_precision, bool masked, const float* want_video, const float* want_audio, bool audio_enabled = true, - bool dense_self_mask = false) { - const Ltx2DitParams p = ReducedParams(rope_type, double_precision); + bool dense_self_mask = false, bool prompt_adaln = false) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = prompt_adaln; WeightSet set = BuildWeights(p); // Staged at the SAME dtype the stream computes in. Staging f32 weights under a // bf16 stream would compare a DIFFERENT MODEL, not a different dtype policy. @@ -406,6 +407,74 @@ void CheckBf16Stream(vt::Queue& q, const char* label) { CHECK(bf16.video > kDeviceRoundOff); } +// The prompt-side AdaLN arm — upstream's DEFAULT and what the shipped DiT runs +// (.agents/specs/ltx25-prompt-adaln.md, issue #644). The device path forms +// `kv_modulation = table + prompt_timestep` (transformer.py:441-443) through +// `ada_value` and then broadcasts one row per BATCH element over that element's +// prompt tokens, which is a DIFFERENT `modulate` call shape from the static +// arm — so it needs its own case rather than riding on the loop above. +void CheckPromptAdalnCases(vt::Queue& q, const char* label) { + { + INFO(label << " / prompt AdaLN, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // The masked case: the prompt mask and the prompt modulation act on the SAME + // context tensor, and a per-batch broadcast written as a per-token one would + // survive the unmasked case at batch 1. + INFO(label << " / prompt AdaLN + masks, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, true, + vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN masked f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // AND IT IS LOAD-BEARING. The flag-ON and flag-OFF goldens share every common + // weight bit-for-bit, so a device path that bound the module and never added + // its output would reproduce the flag-OFF numbers exactly. + // + // The bound is `kDeviceRoundOff` itself, and that is the precise statement: + // the flag-ON run must miss the flag-OFF golden by MORE than the tolerance + // the flag-OFF case is held to, or a dropped term would pass that gate. + // Measured here: video 1.46e-4 (7.3x the bound), audio 7.37e-5 (3.7x). + INFO(label << " / prompt AdaLN is load-bearing"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardSplitVideo, vllm_test::kLtx2ForwardSplitAudio, + /*audio_enabled=*/true, /*dense_self_mask=*/false, + /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN vs flag-OFF golden: video=" << r.video << " audio=" << r.audio); + CHECK(r.video > kDeviceRoundOff); + CHECK(r.audio > kDeviceRoundOff); + } + { + // The bf16 PRODUCTION stream on the same arm: `ada_value` stores the + // table+timestep sum at the stream dtype, so this is where a bf16 store of the + // new sum is exercised at all. + INFO(label << " / prompt AdaLN, bf16"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kBF16, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN bf16: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kBf16RoundOff); + CHECK(r.audio < kBf16RoundOff); + CHECK(r.video > kDeviceRoundOff); + } +} + vt::Backend* TryCuda() { return vt::TryGetBackend(vt::DeviceType::kCUDA); } } // namespace @@ -527,6 +596,21 @@ TEST_CASE("ltx2 device: the DEVICE-RESIDENT forward matches upstream on CUDA") { CheckAllForwardCases(q, "cuda"); } +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream (CPU backend)") { + vt::Queue q{Cpu(), nullptr}; + CheckPromptAdalnCases(q, "cpu-backend"); +} + +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream on CUDA") { + vt::Backend* cuda = TryCuda(); + if (cuda == nullptr) { + MESSAGE("SKIP: no CUDA backend registered"); + return; + } + vt::Queue q = cuda->CreateQueue(); + CheckPromptAdalnCases(q, "cuda"); +} + TEST_CASE("ltx2 device: the bf16 PRODUCTION stream matches upstream (CPU backend)") { vt::Queue q{Cpu(), nullptr}; CheckBf16Stream(q, "cpu-backend bf16"); diff --git a/tests/vllm/models/test_ltx2_loader.cpp b/tests/vllm/models/test_ltx2_loader.cpp index 654ecd134..e191d958b 100644 --- a/tests/vllm/models/test_ltx2_loader.cpp +++ b/tests/vllm/models/test_ltx2_loader.cpp @@ -827,8 +827,10 @@ TEST_CASE("ltx2 loader: the shipped FP8 DiT manifest is fully accounted for") { CHECK(bf16 == 2284); CHECK(f32 == 290); // the tables; the 1775 scalar scales were counted above - // The four families outside the phase-L2 contract, named so their absence - // from the port cannot be discovered later. + // The five families the file carries beyond the ORIGINAL L2 contract, named so + // their status cannot be discovered later. Two are ported (the prompt AdaLN + // pair), two are loaded elsewhere (the connectors), one is genuinely unported + // (keyframes). CHECK(families.count("prompt_adaln_single") == 1); CHECK(families.count("audio_prompt_adaln_single") == 1); CHECK(families.count("keyframes_abs_pos_embedding") == 1); @@ -886,18 +888,19 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D CHECK(p.cross_attention_adaln); CHECK_FALSE(p.ff_bias); CHECK(p.audio_ff_bias); - // MEASURED, and it contradicts .agents/specs/ltx-2-5.md section 1.2 and - // ltx2.h:115-117: the SHIPPED checkpoint carries prompt_adaln_single, which - // upstream builds only when use_prompt_adaln_single is TRUE (model.py:222-226). - // The prompt-K/V cache's premise does not hold for this checkpoint. + // MEASURED: the SHIPPED checkpoint carries prompt_adaln_single, which upstream + // builds only when use_prompt_adaln_single is TRUE (model.py:222-226), so the + // prompt-K/V cache's premise does not hold for this checkpoint + // (.agents/specs/ltx-2-5.md §1.2) and the module is PORTED + // (.agents/specs/ltx25-prompt-adaln.md, issue #644). CHECK(p.use_prompt_adaln_single); - // Enumerate the contract for the subset this port DOES carry, and require - // every one of its names in the file. - Ltx2DitParams contract = p; - contract.use_prompt_adaln_single = false; - const std::vector want = vllm::EnumerateLtx2DitTensors(contract); - CHECK(want.size() == 4078); + // Enumerate the contract AS THE FILE DESCRIBES IT — no flag is forced here any + // more — and require every one of its names in the file. This line used to read + // `contract.use_prompt_adaln_single = false`, which is the shape of the defect: + // the gate agreed with the port about a module they were both dropping. + const std::vector want = vllm::EnumerateLtx2DitTensors(p); + CHECK(want.size() == 4078 + 12); int64_t missing = 0; std::string first_missing; for (const Ltx2TensorSpec& spec : want) { @@ -913,15 +916,16 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D // ... and account for every name the file has that the contract does not, so // "the rest is fine" is a counted claim rather than a hope. // 258 = 2 connectors x (8 blocks x 16 + 1 learnable_registers) - // 12 = prompt_adaln_single + audio_prompt_adaln_single, 6 tensors each // 1 = keyframes_abs_pos_embedding + // The 12 prompt-AdaLN tensors are no longer here: they moved INTO `want` when + // the module was ported, which is the whole delta this row landed. std::set want_set; for (const Ltx2TensorSpec& spec : want) want_set.insert(spec.name); int64_t extra = 0; for (const std::string& name : present) { if (want_set.count(name) == 0) ++extra; } - CHECK(extra == 258 + 12 + 1); + CHECK(extra == 258 + 1); } // =========================================================================== @@ -1542,8 +1546,7 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit( p, Ltx2DitQuant::kFp8, - {"prompt_adaln_single.linear.weight", "audio_prompt_adaln_single.linear.weight", - "keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", + {"keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", "audio_embeddings_connector.learnable_registers"}); const std::string path = TmpPath("unported"); WriteSafetensors(syn.entries, path); @@ -1557,12 +1560,18 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" } const std::string what_msg = "what: " + what; INFO(what_msg); - // THREE families now, not five. `audio_prompt_adaln_single` is here because it - // goes through the identical generic FamilyOf path as its video twin and - // covering it literally is one line. - CHECK(what.find("prompt_adaln_single") != std::string::npos); - CHECK(what.find("audio_prompt_adaln_single") != std::string::npos); - CHECK(what.find("keyframes_abs_pos_embedding") != std::string::npos); + // ONE family in the LIST now, not five. `prompt_adaln_single` / + // `audio_prompt_adaln_single` left it when they were PORTED (issue #644); the + // case below proves a checkpoint carrying them needs no opt-in at all. + // + // The list is asserted as a whole rather than by substring, because the message + // deliberately goes on to NAME the families that are not in it — a + // `find(...) == npos` over the whole message would only test the prose. + const std::string head = "does NOT carry: "; + const size_t at = what.find(head); + REQUIRE(at != std::string::npos); + const std::string list = what.substr(at + head.size(), what.find('.', at) - at - head.size()); + CHECK(list == "keyframes_abs_pos_embedding"); // THE TWO CONNECTOR FAMILIES ARE NOT UNPORTED AS OF PHASE L9c. They are // outside the DiT's contract by design — upstream loads them into the text @@ -1574,17 +1583,15 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" CHECK(what.find("video_embeddings_connector") == std::string::npos); CHECK(what.find("audio_embeddings_connector") == std::string::npos); - // The opt-in still REPORTS every one of the three; it does not make them vanish. + // The opt-in still REPORTS it; it does not make it vanish. Ltx2DitLoadOptions options; options.allow_unported_modules = true; const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file, options); - CHECK(ck.unported.size() == 3); + CHECK(ck.unported.size() == 1); for (const std::string& family : ck.unported) { CHECK(family != "video_embeddings_connector"); CHECK(family != "audio_embeddings_connector"); } - CHECK(ck.checkpoint_params.use_prompt_adaln_single); - CHECK_FALSE(ck.params.use_prompt_adaln_single); std::remove(path.c_str()); // AND THE CONNECTOR-ONLY CHECKPOINT LOADS WITH NO OPT-IN AT ALL, which is the @@ -1604,6 +1611,131 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" std::remove(conn_path.c_str()); } +// THE REGRESSION GATE FOR ISSUE #644 ROW 0. +// +// A checkpoint that carries `prompt_adaln_single` — which the shipped LTX-2.5 DiT +// does — used to be refused without `allow_unported_modules=1`, and setting that +// extra reached three assignments that CLEARED `use_prompt_adaln_single`. So the +// only way to load a real DiT was also the way to silently drop the timestep half +// of every cross-attention K/V modulation. +// +// Both halves are asserted here, and neither is true by construction: the load +// with no opt-in exercises the contract, and the resolved flag is read back off +// the checkpoint the loader actually bound. +TEST_CASE("ltx2 loader: a DiT carrying prompt_adaln_single loads with NO opt-in") { + Ltx2DitParams p = TinyParams(); + p.use_prompt_adaln_single = true; + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string path = TmpPath("prompt_adaln"); + WriteSafetensors(syn.entries, path); + const SafetensorsFile file = SafetensorsFile::Open(path); + + // No options at all: the families are in the contract, so nothing is unported. + const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file); + CHECK(ck.unported.empty()); + CHECK(ck.checkpoint_params.use_prompt_adaln_single); + // The flag the FORWARD reads. This is the assertion the defect broke: it was + // false while `checkpoint_params` said true. + CHECK(ck.params.use_prompt_adaln_single); + // And the weights are bound, not merely enumerated — a contract that listed the + // tensors while `BindLtx2DitWeights` left the views null would render the same + // wrong picture with a different failure mode. + CHECK(ck.weights.prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.prompt_adaln_single.linear.weight.shape[0] == 2 * p.inner_dim()); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.shape[0] == 2 * p.audio_inner_dim()); + // The 12 tensors, counted rather than assumed. + Ltx2DitParams off = p; + off.use_prompt_adaln_single = false; + CHECK(vllm::EnumerateLtx2DitTensors(p).size() - + vllm::EnumerateLtx2DitTensors(off).size() == + 12); + std::remove(path.c_str()); + + // AND THE OPT-IN CANNOT UNDO IT. `allow_unported_modules` is what a real render + // still passes for `keyframes_abs_pos_embedding`; it must leave a ported feature + // alone. Same bytes, opt-in set, same resolved flag. + const std::string path2 = TmpPath("prompt_adaln_optin"); + WriteSafetensors(syn.entries, path2); + const SafetensorsFile file2 = SafetensorsFile::Open(path2); + Ltx2DitLoadOptions options; + options.allow_unported_modules = true; + const vllm::Ltx2DitCheckpoint ck2 = vllm::Ltx2LoadDitFromSafetensors(file2, options); + CHECK(ck2.params.use_prompt_adaln_single); + CHECK(ck2.weights.prompt_adaln_single.linear.weight.data != nullptr); + std::remove(path2.c_str()); +} + +// The DECLARED config path. `Ltx2AdoptDeclaredDitParams` no longer forces both +// sides of its comparison to a cleared flag, so a config that disagrees with the +// file's shapes about `use_prompt_adaln_single` now produces two DIFFERENT +// contracts and is refused — which is an INPUT-driven gate, not a mutation-only +// one. Both directions are checked, because the equality is the point. +TEST_CASE("ltx2 loader: a config that disagrees about use_prompt_adaln_single is REFUSED") { + Ltx2DitParams shapes = TinyParams(); + shapes.use_prompt_adaln_single = true; + + nlohmann::json t; + t["num_attention_heads"] = shapes.num_attention_heads; + t["attention_head_dim"] = shapes.attention_head_dim; + t["in_channels"] = shapes.in_channels; + t["out_channels"] = shapes.out_channels; + t["num_layers"] = shapes.num_layers; + t["cross_attention_dim"] = shapes.cross_attention_dim; + t["audio_num_attention_heads"] = shapes.audio_num_attention_heads; + t["audio_attention_head_dim"] = shapes.audio_attention_head_dim; + t["audio_in_channels"] = shapes.audio_in_channels; + t["audio_out_channels"] = shapes.audio_out_channels; + t["audio_cross_attention_dim"] = shapes.audio_cross_attention_dim; + t["apply_gated_attention"] = true; + t["cross_attention_adaln"] = true; + t["ff_bias"] = false; + t["audio_ff_bias"] = true; + t["rope_type"] = "split"; + // The checks ParseLtx2DitParams asserts verbatim (model_configurator.py:26-44). + t["dropout"] = 0.0; + t["attention_bias"] = true; + t["num_vector_embeds"] = nullptr; + t["activation_fn"] = "gelu-approximate"; + t["num_embeds_ada_norm"] = 1000; + t["use_linear_projection"] = false; + t["only_cross_attention"] = false; + t["cross_attention_norm"] = true; + t["double_self_attention"] = false; + t["upcast_attention"] = false; + t["standardization_norm"] = "rms_norm"; + t["norm_elementwise_affine"] = false; + t["qk_norm"] = "rms_norm"; + t["positional_embedding_type"] = "rope"; + t["use_audio_video_cross_attention"] = true; + t["share_ff"] = false; + t["av_cross_ada_norm"] = true; + t["use_middle_indices_grid"] = true; + t["caption_proj_before_connector"] = true; + + nlohmann::json agreeing; + agreeing["transformer"] = t; + agreeing["transformer"]["use_prompt_adaln_single"] = true; + // Agreement is adopted, and carries the flag through. + const Ltx2DitParams adopted = + vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes, false, "the test config"); + CHECK(adopted.use_prompt_adaln_single); + + nlohmann::json disagreeing; + disagreeing["transformer"] = t; + disagreeing["transformer"]["use_prompt_adaln_single"] = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, false, "the test config")); + // The opt-in must not rescue it: clearing a PORTED flag in the config copy is + // exactly what made this comparison blind before. + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, true, "the test config")); + + // And the other direction: shapes WITHOUT the module against a config that + // declares it. + Ltx2DitParams shapes_off = shapes; + shapes_off.use_prompt_adaln_single = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes_off, false, "the test config")); +} + TEST_CASE("ltx2 loader: the f32 widening is OPT-IN and bit-exact over bf16") { const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index e2191ed9b..ec3fca7af 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -248,7 +248,14 @@ inline vllm::Ltx2DitParams ReducedDitParams() { p.use_middle_indices_grid = true; p.apply_gated_attention = true; // every LTX-2.5 attention is gated p.cross_attention_adaln = true; // the shipped config sets it - p.use_prompt_adaln_single = false; + // TRUE, exactly as both shipped DiTs resolve it: the FP8 file carries no + // config at all and the NVFP4 file's config OMITS the key, so upstream's + // default (model.py:77, model_configurator.py:76) decides — and both files + // carry `prompt_adaln_single`'s tensors. `ReducedDitTransformerConfig` below + // likewise omits the key, so this fixture reproduces the shipped shape rather + // than a configuration nothing ships, and the whole video engine renders + // through the prompt-side AdaLN path (.agents/specs/ltx25-prompt-adaln.md). + p.use_prompt_adaln_single = true; p.ff_bias = false; // LTX-2.5 (gemma4) sets ff_bias=false p.audio_ff_bias = true; return p; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index e8eaf1220..85b47025a 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -927,10 +927,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const vllm::SafetensorsFile file = vllm::SafetensorsFile::Open(path); vllm::Ltx2DitQuant quant = vllm::Ltx2DitQuant::kFp8; vllm::Ltx2DitParams from_shapes = vllm::Ltx2ParseDitParamsFromCheckpoint(file, &quant); - // The manifest parser leaves `use_prompt_adaln_single` at its default; the - // LOADER clears it for the contract (ltx2_loader.cpp), and so does the - // engine. Mirror that here so the two contracts are compared like for like. - from_shapes.use_prompt_adaln_single = false; + // MEASURED from this file's own header: it carries `prompt_adaln_single`, so + // the manifest parser resolves `use_prompt_adaln_single = TRUE` — and nothing + // clears it any more (.agents/specs/ltx25-prompt-adaln.md, issue #644). This + // line used to force it false on BOTH sides of the comparison below, which is + // what made a config/shape disagreement about it unobservable. + CHECK(from_shapes.use_prompt_adaln_single); CHECK(quant == vllm::Ltx2DitQuant::kNvfp4); CHECK(from_shapes.num_layers == 48); CHECK(from_shapes.inner_dim() == 4096); @@ -950,8 +952,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { config["transformer"]["use_keyframes_abs_pos_embedding"] = false; nlohmann::json wrapper; wrapper["config"] = config; - vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); - declared.use_prompt_adaln_single = false; + const vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); + // The shipped config OMITS `use_prompt_adaln_single`, so it resolves to + // upstream's TRUE default (model_configurator.py:76) — which is what the + // file's own tensors say. Asserted rather than forced: the two sides of the + // contract comparison below must AGREE about it, not be made to. + CHECK(declared.use_prompt_adaln_single); CHECK(declared.double_precision_rope); // frequencies_precision float64 CHECK(declared.av_ca_timestep_scale_multiplier == 1000); CHECK(declared.apply_gated_attention);