From baa92ccf7be05012a307489a42895c0b4400ac6e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:05:09 +0000 Subject: [PATCH 1/4] spec(ltx-2.5): the prompt-side AdaLN path is DROPPED on every render MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185` and LTX-2 `model.py:77` / `model_configurator.py:76,138` — and the shipped FP8 DiT carries the 18 tensors the flag builds. `ltx2_loader.cpp:988` (and `:573`, `:626`) cleared it unconditionally, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Shapes, finiteness and every committed golden agree with the defect, because the goldens were generated with the flag OFF. Spec only; no code. Records the upstream anchors for the module (`model.py:222-227`, `:252-257`), the producer (`transformer_args.py:274-277`, driven by `modality.sigma` and NOT `timesteps`), the consumer (`transformer.py:441-446`), and what replaces the cleared flag so `allow_unported_modules` can no longer silently disable a ported feature. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 259 ++++++++++++++++++++++++++++ 2 files changed, 260 insertions(+) create mode 100644 .agents/specs/ltx25-prompt-adaln.md diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 3cc1d0234..df766a903 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -147,6 +147,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature | | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | +| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md new file mode 100644 index 000000000..70cb486b5 --- /dev/null +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -0,0 +1,259 @@ +# LTX-2.5 — the prompt-side AdaLN path (`use_prompt_adaln_single`) + +Row: `LTX25-PROMPT-ADALN`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; not +edited by this row). Issue: +[#644](https://github.com/mudler/vllm.cpp/issues/644), row 0. + +Upstream pins: + +| Reference | Revision | +|---|---| +| Lightricks/LTX-2 (`packages/ltx-core`) | `fd4ded7f` | +| huggingface/diffusers | `3a2f35d4` | + +Both are read from local checkouts at those revisions, and the golden generator +IMPORTS and EXECUTES the LTX-2 modules rather than restating them. + +## 0. What is wrong today + +`src/vllm/model_executor/models/ltx2_loader.cpp:988` sets, unconditionally: + +```cpp +declared.use_prompt_adaln_single = false; +``` + +and `:573` / `:626` do the same on the two manifest paths. The flag defaults +**TRUE** in both references: + +- diffusers `src/diffusers/models/transformers/transformer_ltx2.py:1185` — + `use_prompt_adaln_single: bool = True` +- LTX-2 `packages/ltx-core/src/ltx_core/model/transformer/model.py:77` — same + default, and `model_configurator.py:76` / `:138` read it as + `config.get("use_prompt_adaln_single", True)` + +The shipped FP8 DiT carries the 18 tensors the flag builds (12 +`prompt_adaln_single.*`, 6 `audio_prompt_adaln_single.*`; see +`tests/vllm/models/ltx2_fp8_dit_manifest.inc:232-240,286-294`), so the flag is +TRUE for the checkpoint this campaign renders. `ltx2.cpp:274-276` refuses those +tensors by name, so a real render needs `allow_unported_modules=1` +(`src/vllm/multimodal/ltx2_video.cpp:570`) — which reaches the loader lines above +and **silently clears the flag**. + +Net effect: every render drops the timestep-conditioned half of the prompt K/V +modulation, keeping only the static `prompt_scale_shift_table`. Nothing observes +it: shapes are unchanged, values stay finite, and the goldens were generated with +`use_prompt_adaln_single=False` (`scripts/gen-ltx2-goldens.py:149`), so the gate +agrees with the defect. + +Campaign history this row does **not** re-derive: `ltx-2-5.md` §1.2 already +RETRACTED the "prompt K/V carry no timestep term" claim and recorded that the +shipped checkpoint carries a `[4096, 256]` prompt timestep embedder. What was +never closed is *using* the tensors. + +## 1. What upstream does, with anchors + +### 1.1 The module + +`model.py:222-227` (video) and `:252-257` (audio): + +```python +self.prompt_adaln_single = ( + AdaLayerNormSingle(self.inner_dim, embedding_coefficient=2) + if self.cross_attention_adaln and self.use_prompt_adaln_single + else None +) +``` + +`AdaLayerNormSingle` (`adaln.py:19-45`) is the same brick the port already has +(`Ltx2AdaLayerNormSingle`): `emb.timestep_embedder.linear_1 [dim, 256]`, +`linear_2 [dim, dim]`, `linear [coefficient * dim, dim]`. Coefficient **2** here, +not `adaln_embedding_coefficient()` — shift and scale for the K/V only. + +diffusers twin: `transformer_ltx2.py:1255-1259`, `num_mod_params=2`. + +Registration order inside `_init_video` puts it between `adaln_single` and +`proj_out`, which is where `EnumerateLtx2DitTensors` already reserves its slot +(the `VT_CHECK` at `ltx2.cpp:274-276`). + +### 1.2 The producer + +`transformer_args.py:274-277`, inside `TransformerArgsPreprocessor.prepare`: + +```python +prompt_timestep = None +if self.prompt_adaln is not None: + prompt_timestep, _ = self._prepare_timestep( + modality.sigma, self.prompt_adaln, batch_size, modality.latent.dtype + ) +``` + +Three things this fixes in one line, each of which a shape check cannot see: + +1. The input is **`modality.sigma`**, `(B,)` (`modality.py:54`) — the per-sample + scalar noise level — **not** `modality.timesteps`, which is per-token `(B, T)`. +2. `_prepare_timestep` (`transformer_args.py:173-186`) multiplies by + `timestep_scale_multiplier` before the embedder, exactly as the port's + `PrepareTimestep` already does for the main AdaLN. +3. The result is viewed to `(B, -1, 2 * dim)`, i.e. `(B, 1, 2 * dim)` — one row + broadcast over the prompt tokens. + +Wired into both preprocessor kinds at `model.py:313`, `:333` (multimodal) and +`:348`, `:364` (single-modality). + +diffusers twin: `transformer_ltx2.py:1536-1547`. diffusers passes `sigma` already +scaled from the pipeline (`pipeline_ltx2_image2video.py:1481` — `sigma=timestep`, +and `timestep` is the scheduler's 0..1000 value), so the two references agree on +the value reaching the embedder; only the place the x1000 happens differs. This +port mirrors LTX-2, so the multiply happens here. + +### 1.3 The consumer + +`transformer.py:427-447` (`apply_cross_attention_adaln`): + +```python +kv_modulation = prompt_scale_shift_table[None, None].to(...) # :441 +if prompt_timestep is not None: # :442 + kv_modulation = kv_modulation + prompt_timestep.reshape( + batch_size, prompt_timestep.shape[1], 2, -1) # :443 +shift_kv, scale_kv = kv_modulation.unbind(dim=2) # :444 +... +encoder_hidden_states = context * (1 + scale_kv) + shift_kv # :446 +``` + +Reached from `_apply_text_cross_attention` (`:223-251`), which is called for the +video stream at `:288-296` and the audio stream at `:317-325`, passing +`video.prompt_timestep` / `audio.prompt_timestep`. Every block, both streams. + +diffusers twin: `transformer_ltx2.py:677-693` (`get_mod_params` over +`prompt_scale_shift_table`), threaded at `:1648-1649`. + +Layout consequence: the flat `[B, 1, 2 * dim]` row is read as `[2, dim]` with +**shift first, scale second** — the same order the static table already uses in +`ModulateContext` (`ltx2_dit.cpp:118-129`). + +## 2. Scope + +**In.** + +1. `Ltx2DitParams::use_prompt_adaln_single` is honoured end to end: contract, + binding, host forward, device forward. +2. The 18 tensors enter `EnumerateLtx2DitTensors` / `BindLtx2DitWeights` when + `cross_attention_adaln && use_prompt_adaln_single`. +3. `temb_prompt` / `temb_prompt_audio` computed from each stream's own `sigma` + and threaded into every block's text cross-attention on both the host + (`ltx2_dit.cpp`) and device (`ltx2_device.cpp`) paths. +4. The three loader `= false` assignments are deleted, and replaced by a guard + (§3.2) that makes a future silent clear impossible. +5. The `ltx2.cpp:274-276` refusal is deleted for these two families. +6. Goldens executed from upstream at reduced dims, with a mutation proving the + new term is load-bearing, and a measured magnitude. + +**Out.** Anything the campaign already records as owed: keyframe absolute +position embedding (still genuinely unported, still what +`allow_unported_modules` is for), the caption projections, guidance +perturbations, and the bf16/FP8/NVFP4 stream dtypes on the host forward. + +## 3. Design + +### 3.1 The seam + +`Ltx2DitWeights` gains two `Ltx2AdaLayerNormSingleWeights` members; +`Ltx2BlockArgs` / `BlockArgsDev` gain a `[batch, 1, 2 * width]` prompt-modulation +pointer per stream, `nullptr` when the flag is off. `ModulateContext` and its +device twin take that pointer and add `prompt_mod[b, {0,1} * width + c]` to the +table row before applying `context * (1 + scale) + shift`. `nullptr` gives the +existing static-only behaviour byte-for-byte, which is what keeps every current +golden valid. + +The **order of the two additions** is upstream's: the table and the timestep row +are summed FIRST (`:443`), and only then does `(1 + scale)` apply. Folding it the +other way would round differently — the same trap `ProcessOutput` +(`ltx2_dit.cpp:530-540`) already documents. + +### 3.2 The prompt-K/V cache, and what replaces the cleared flag + +`Ltx2DitForward` already refuses a cache when the flag is on +(`ltx2_dit.cpp:672-676`); with the flag no longer cleared, that refusal becomes +*reachable* rather than dead, and it is correct: the K/V now carry a timestep +term. Nothing in the shipped pipeline passes a cache (`grep` over +`ltx2_pipeline.cpp` finds none), so no caller regresses. + +**What replaced the `= false`.** The loader clearing existed so +`EnumerateLtx2DitTensors` would not throw. With the tensors ported the contract +simply includes them, so the assignment has no job left. In its place the loader +asserts the invariant the clearing used to violate: + +> the resolved `use_prompt_adaln_single` must equal whether the FILE carries +> `prompt_adaln_single.linear.weight` + +A future edit that re-clears the flag then hits a named refusal instead of +quietly dropping 18 tensors. This is deliberately an *equality*, not a one-sided +check: clearing the flag with the tensors present is the defect this row fixes, +and setting it with the tensors absent would bind missing weights. + +### 3.3 `allow_unported_modules` + +After this row it is scoped to genuinely-unported modules only: the sole flag it +still clears in a config copy is `use_keyframes_abs_pos_embedding` +(`ltx2_loader.cpp:979-984`), whose module really is unported. The guard in §3.2 +is what makes that scoping structural rather than a comment — the extra `= false` +cannot come back without going red. + +`Ltx2AdoptDeclaredDitParams`'s contract-equality check becomes load-bearing in a +second way: a checkpoint whose config declares `use_prompt_adaln_single=false` +while its shapes carry the tensors now produces two DIFFERENT contracts and is +refused, instead of both sides being forced to the same cleared value. + +## 4. Memory format + +Mirrors the existing L2 parity forward exactly: f32 host, f32/bf16 device stream +with the `prompt_scale_shift_table` read at F32 (`ltx2_device.cpp:508-511`). The +prompt modulation is one `[batch, 1, 2 * width]` buffer per stream per forward — +`2 * 4096 * batch` floats for the video stream at full size, computed once +outside the block loop, not per block. No new per-token buffer, so no per-token +byte cost. + +## 5. Tests + +1. `EnumerateLtx2DitTensors` with the flag ON reproduces upstream + `named_parameters()` verbatim — names, order, ranks, dims — for a model built + with `use_prompt_adaln_single=True`. This is the 18-tensor contract. +2. Full dual-stream DiT forward, flag ON, against upstream's executed output. +3. **The mutation**: the same forward with the prompt-AdaLN contribution zeroed + must go RED against that golden. A term that is present but inert is not a + port. +4. The flag-OFF goldens stay byte-identical, proving the new path is off when + upstream's is off. +5. The prompt-K/V cache stays refused with the flag on (existing case). + +## 6. Measured magnitude + +Recorded in §Outcome: the relative change in the modulated prompt context and in +the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer +to "does this matter"; a number below round-off would mean the term is inert and +the whole row is decoration. + +## 7. Risks + +- **The goldens agree with the defect.** Every existing LTX golden was generated + with the flag off, so no existing case can fail whatever this row does. The new + flag-ON case is the only instrument, which is why §5.3 mutates it rather than + asserting it. +- **`sigma` vs `timesteps`.** Using the per-token `timesteps` instead of the + per-sample `sigma` produces a same-shaped, finite, wrong result at batch 1 with + uniform timesteps. The golden runs a batch of 2 with per-token timesteps that + differ from sigma, so the two are distinguishable. +- **Order within the `[2, dim]` row.** Swapping shift and scale is finite and + same-shaped. The golden's random weights make it observable. + +## 8. Stop conditions + +- The flag-ON forward cannot be made to match upstream to the existing + `kRoundOff` bound: stop and report rather than widening the bound. +- The mutation in §5.3 stays green: the path is not reached, and the row is not + done. Escalate the mutation's magnitude before concluding anything about + reachability (issue #604). + +## Now + +`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. From 02038167677dcd8d0c0bece21d11f976130193df Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 17:54:16 +0000 Subject: [PATCH 2/4] feat(ltx-2.5): port the prompt-side AdaLN, and stop the opt-in from clearing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `use_prompt_adaln_single` defaults TRUE in both references — diffusers `transformer_ltx2.py:1185`, LTX-2 `model.py:77` and `model_configurator.py:76`/`:138` — and the shipped DiT carries the module's tensors. `ltx2_loader.cpp` cleared it in three places, so every render kept only the static `prompt_scale_shift_table` and dropped the timestep-conditioned half of the prompt K/V modulation. Nothing saw it: shapes unchanged, values finite, and every committed golden was generated with the flag OFF. MEASURED, flag ON vs OFF over the same shared weights: the timestep term is 51.7% the magnitude of the static table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and moves the DiT output by 1.46e-4 — 73x the gate's f32 floor. Ported, citing what each piece came from: the module <- model.py:222-227 (video), :252-257 (audio); AdaLayerNormSingle(dim, embedding_coefficient=2), adaln.py:19-45 the producer <- transformer_args.py:274-277 driven by `modality.sigma` [batch] and NOT the per-token `timesteps`, scaled by timestep_scale_multiplier at :173-186 the consumer <- transformer.py:441-446, summed into the table BEFORE `(1 + scale)` applies, threaded per stream at :288-296/:317-325 diffusers twin <- transformer_ltx2.py:1536-1547, :1648-1649, :677-693 Both arms: `Ltx2DitForward` and `Ltx2DitForwardDevice` (f32 and the bf16 production stream). The three `= false` assignments are replaced by an invariant: the resolved flag must equal whether the FILE carries `prompt_adaln_single`. And `Ltx2AdoptDeclaredDitParams` now clears exactly ONE flag in its config copy — `use_keyframes_abs_pos_embedding`, whose module really is unported — so `allow_unported_modules=1`, which a real render still needs, can no longer switch a ported feature off. A config that disagrees with the shapes about the flag is refused rather than forced into agreement. Both shipped DiTs were re-read off the NAS to check this refuses neither: the FP8 file carries no `__metadata__` at all and the NVFP4 file's config OMITS the key, so upstream's TRUE default decides on both, matching their tensors. Evidence. Goldens executed from upstream LTX-2 @fd4ded7f at reduced dims (`gen-ltx2-goldens.py` section 6); every pre-existing golden VALUE is byte-identical. Five mutations, each RED and each restored byte-for-byte: dropping the term host-side (3/35 cases), dropping it device-side (1/15), re-adding the cleared flag (the guard throws; assertion count drops 4826->4815), driving the MLP from `timesteps` instead of `sigma` (2/35), and swapping the shift and scale rows (2/35). A permanent in-tree case asserts the flag-ON forward MISSES the flag-OFF golden, so a bound-but-unused module cannot pass. Suites vs `cefacd2d0`: test_ltx2 30/1627 -> 35/2435, test_ltx2_loader 24/4817 -> 26/4826, test_ltx2_device 13/498 -> 15/523, test_ltx2_video 30/502 -> 30/502 (its fixture now carries the module, which is the shipped shape). Full `ctest` 423 registered, 422 pass under -j8 and `test_serve_low_tools` passes on its serial re-run. Spec .agents/specs/ltx25-prompt-adaln.md @baa92ccf7. Issue #644, row 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-prompt-adaln.md | 104 ++++- docs/FEATURES.md | 2 +- docs/USAGE.md | 11 +- include/vllm/model_executor/models/ltx2.h | 51 ++- .../vllm/model_executor/models/ltx2_loader.h | 45 +- include/vllm/multimodal/ltx2_video.h | 17 +- scripts/gen-ltx2-goldens.py | 151 +++++- src/vllm/model_executor/models/ltx2.cpp | 16 +- .../model_executor/models/ltx2_device.cpp | 79 +++- src/vllm/model_executor/models/ltx2_dit.cpp | 90 +++- .../model_executor/models/ltx2_loader.cpp | 67 ++- tests/vllm/models/ltx2_goldens.inc | 432 +++++++++++++++++- tests/vllm/models/test_ltx2.cpp | 141 ++++++ tests/vllm/models/test_ltx2_device.cpp | 88 +++- tests/vllm/models/test_ltx2_loader.cpp | 184 ++++++-- tests/vllm/multimodal/ltx2_video_fixture.h | 9 +- tests/vllm/multimodal/test_ltx2_video.cpp | 18 +- 17 files changed, 1381 insertions(+), 124 deletions(-) diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 70cb486b5..a24f127ce 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -254,6 +254,108 @@ the whole row is decoration. done. Escalate the mutation's magnitude before concluding anything about reachability (issue #604). +## Outcome + +### What was measured + +The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints +them on stderr, from the SAME shared weight stream on both arms (keyed by +parameter name, so every common weight is bit-identical and the difference is the +term and nothing else): + +| Quantity | Flag ON vs OFF | +|---|---| +| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | +| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | +| DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | + +**The answer to "does this matter" is the first two rows.** Roughly half the +magnitude of the prompt K/V modulation is the timestep-conditioned term, and +including it moves the modulated prompt context by ~6%. Every render before this +row discarded that. + +The two output rows are the GATE's floor, not a claim about the trained +checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and +by a 2-block stack rather than 48. They are reported because a mutation must be +shown to move something, and 73x/37x above round-off is what makes the mutation +below meaningful. + +### The mutations + +All five run on the committed head, restored byte-for-byte afterwards (source +md5s re-checked). Exit status is the authority; assertion COUNTS are recorded +because doctest's summary and the exit code disagree in both directions. + +| # | Mutation | Result | +|---|---|---| +| M1 | host `ModulateContext` ignores `prompt_mod` | RED — 3/35 cases, 6/2435 assertions, exit 1 | +| M2 | device `TextCrossAttentionDev` takes the static-only branch always | RED — 1/15 cases, 6/523 assertions, exit 1 | +| M3 | re-add `use_prompt_adaln_single = false` before the loader guard | RED — the guard throws by name; assertion count DROPS 4826 → 4815, exit 1 | +| M4 | prompt AdaLN driven by `m.timesteps` instead of `m.sigma` | RED — 2/35 cases, 4/2435 assertions, exit 1 | +| M5 | shift and scale rows swapped within the `[2, width]` row | RED — 2/35 cases, 4/2435 assertions, exit 1 | + +M3 is the one that cannot be reached by any INPUT, and that is stated rather than +papered over: `ParseLtx2DitParamsFromManifest` derives the flag from the same +manifest the guard reads, so they agree by construction unless an assignment +intervenes — which is exactly the edit the guard exists to catch. The +input-driven half of the same rule lives in `Ltx2AdoptDeclaredDitParams`, where a +config that disagrees with the shapes now produces two different contracts and is +refused; that one is gated by a test with real inputs in both directions. + +### The gate + +`BUILD_EXIT=0` on every build; build logs grepped for `No space left|BFD assertion` +(0 hits) and `df -h /` logged (88% used, 52G free at the end). Case AND assertion +counts against the `cefacd2d0` baseline, measured by reverting the working tree to +HEAD, rebuilding the four targets and running them, then re-applying the diff and +re-checking its md5 (`03324d42…`, identical before and after): + +| Suite | HEAD `cefacd2d0` | this row | delta | +|---|---|---|---| +| `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | +| `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | +| `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | + +The `test_ltx2_video` fixture had to move: it declared a config that omits +`use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said +false, so the config/shape equality check refused it — correctly. It now carries +the module, which is the shipped shape and puts the whole video engine on the new +path. + +### What was rejected + +- **Widening `modulate`'s kernel contract** with a `rows_per_src_row` divisor, to + express "one row per batch element broadcast over that element's tokens" in one + launch. Rejected: it changes a kernel's semantics for a dimension that is 1 or 2 + in every shipped call, and would owe its own red-before evidence. The device + path loops over batch and offsets the pointers instead. +- **Narrowing the static prompt table to the stream dtype** on the flag-ON path, + which is literally what upstream's `.to(dtype=x_normed.dtype)` does. Rejected as + out of scope: the existing static-only path deliberately keeps the table at F32 + (`ltx2_device.cpp`, "a narrowed table would be the dtype rule applied + backwards"), and changing that polarity is a separate decision. The flag-ON path + routes the sum through `ada_value`, which stores at the stream dtype — the same + rounding every other table+modulation sum in that file already has. + +### Why the defaults are what they are + +`Ltx2DitParams::use_prompt_adaln_single` keeps its `true` default, which is now +honoured rather than overwritten. It matches `model.py:77`, +`model_configurator.py:76`/`:138` and diffusers `transformer_ltx2.py:1185`, and +it matches both shipped DiTs: the FP8 file carries no config at all (so the +default decides), and the NVFP4 file's config OMITS the key — verified by reading +both headers off the NAS. So neither shipped checkpoint is refused by the new +config/shape equality check. + +`allow_unported_modules` keeps existing, because +`keyframes_abs_pos_embedding` is genuinely unported and a real render still needs +the opt-in for it. What changed is that it can no longer switch a ported feature +off: the loader asserts the flag against the file instead of clearing it, and +`Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing +applies. + ## Now -`ACTIVE` — implementation on `row/LTX25-PROMPT-ADALN`. +`DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7586e32ac..3c7ba3e68 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWED | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8, torchao and first-party NVFP4 all load. DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/USAGE.md b/docs/USAGE.md index 142cd6a17..4f80a43ef 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2073,11 +2073,14 @@ other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **three** module families phase -L2 does not port (`prompt_adaln_single`, `audio_prompt_adaln_single` and -`keyframes_abs_pos_embedding`); pass `Ltx2DitLoadOptions::allow_unported_modules` +shipped DiT by default, because that file carries **one** module family this port +does not carry (`keyframes_abs_pos_embedding`); pass +`Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. The two `*_embeddings_connector` towers are +`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`audio_prompt_adaln_single` were on that list until 2026-08-13 and are now +PORTED, so a checkpoint carrying them needs no opt-in on their account, and the +opt-in no longer disables them. The two `*_embeddings_connector` towers are **not** among them and never will be: `UnportedFamilies` filters them out at `ltx2_loader.cpp:439` (`LoadedElsewhere`), `RefuseUnported`'s own message says so in capitals at `ltx2_loader.cpp:461-464`, diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index f71a1a404..99898406b 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -51,8 +51,14 @@ // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are // phase L3. -// - `prompt_adaln_single` (model.py:223-227). LTX-2.5 sets -// `use_prompt_adaln_single=false`; see Ltx2PromptKvCache. +// +// PORTED 2026-08-13 — `prompt_adaln_single` / `audio_prompt_adaln_single` +// (model.py:222-227, :252-257), which this list previously carried as unported on +// the strength of "LTX-2.5 sets use_prompt_adaln_single=false". It does not: the +// flag defaults TRUE in both references (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185) and the +// shipped DiT carries the module's tensors. See +// .agents/specs/ltx25-prompt-adaln.md and issue #644. #pragma once #include @@ -112,8 +118,12 @@ struct Ltx2DitParams { bool double_precision_rope = false; bool apply_gated_attention = false; bool cross_attention_adaln = false; - // model_configurator.py:74-76. FALSE on LTX-2.5, which is what makes the - // cross-attention K/V timestep-independent — see Ltx2PromptKvCache. + // model_configurator.py:74-76 (`config.get("use_prompt_adaln_single", True)`), + // model.py:77, diffusers transformer_ltx2.py:1185 — TRUE by default in every + // reference, and TRUE for the shipped LTX-2.5 DiT, which carries the module's + // tensors. When true a prompt-side AdaLN MLP adds a timestep term to the + // cross-attention K/V modulation (transformer.py:441-443), which is what makes + // those K/V timestep-DEPENDENT and so uncacheable — see Ltx2PromptKvCache. bool use_prompt_adaln_single = true; // model_configurator.py:77-80. LTX-2.5 (gemma4) sets ff_bias=false and leaves // audio_ff_bias at its true default; the checkpoint's shapes agree. @@ -214,6 +224,11 @@ struct Ltx2BlockWeights { struct Ltx2DitWeights { Ltx2LinearWeight patchify_proj, proj_out; Ltx2AdaLayerNormSingleWeights adaln_single; + // model.py:222-227 / :252-257 — built only when `cross_attention_adaln AND + // use_prompt_adaln_single`, with embedding_coefficient 2 (shift + scale for the + // prompt K/V), NOT `adaln_embedding_coefficient()`. Left unbound otherwise. + Ltx2AdaLayerNormSingleWeights prompt_adaln_single; + Ltx2AdaLayerNormSingleWeights audio_prompt_adaln_single; vt::Tensor scale_shift_table; // [2, dim] — the OUTPUT table (:230), not the block's [9, dim] Ltx2LinearWeight audio_patchify_proj, audio_proj_out; Ltx2AdaLayerNormSingleWeights audio_adaln_single; @@ -295,13 +310,19 @@ Ltx2AdalnOut Ltx2AdaLayerNormSingle(vt::Device device, const Ltx2AdaLayerNormSin std::vector Ltx2FeedForward(vt::Device device, const Ltx2FeedForwardWeights& w, const float* x, int64_t rows, int64_t dim, int64_t inner); -// The K/V half of Attention.forward, split out because LTX-2.5 can CACHE it: -// with `use_prompt_adaln_single=false` the prompt modulation carries no timestep -// term (transformer.py:441), so `to_k`/`to_v` over the modulated context — and -// their k_norm, and the absence of RoPE on the text path — depend only on the -// prompt. The denoise loop computes them ONCE PER REQUEST and reuses them for -// every step. Layout: k/v are [batch * context_tokens, heads * dim_head], held -// exactly as the attention op consumes them (post-norm, post-RoPE). +// The K/V half of Attention.forward, split out because a checkpoint that sets +// `use_prompt_adaln_single=false` can CACHE it: the prompt modulation then +// carries no timestep term (transformer.py:441-443), so `to_k`/`to_v` over the +// modulated context — and their k_norm, and the absence of RoPE on the text path +// — depend only on the prompt. The denoise loop computes them ONCE PER REQUEST +// and reuses them for every step. Layout: k/v are +// [batch * context_tokens, heads * dim_head], held exactly as the attention op +// consumes them (post-norm, post-RoPE). +// +// THIS DOES NOT APPLY TO THE SHIPPED LTX-2.5 DiT, which sets the flag TRUE +// (.agents/specs/ltx-2-5.md §1.2, and .agents/specs/ltx25-prompt-adaln.md). The +// mechanism stays here, gated bit-identical, for a checkpoint that does set it +// false; `Ltx2DitForward` refuses a cache when the flag is on. struct Ltx2CrossKv { std::vector k, v; }; @@ -463,6 +484,14 @@ struct Ltx2BlockArgs { // Per-token AdaLN modulation, [batch, tokens, coefficient * dim]. const float* video_timestep_modulation = nullptr; const float* audio_timestep_modulation = nullptr; + // The PROMPT-side AdaLN modulation, [batch, 1, 2 * dim] — shift then scale, one + // row per batch element broadcast over the prompt tokens (transformer.py:443, + // whose `prompt_timestep` has token dimension 1 because `_prepare_timestep` ran + // on the modality's per-sample `sigma`). `nullptr` is upstream's + // `prompt_timestep is None`, i.e. `use_prompt_adaln_single=false`, in which case + // only the static per-block table applies (:441). + const float* video_prompt_modulation = nullptr; + const float* audio_prompt_modulation = nullptr; // Audio<->video cross-attention AdaLN inputs (transformer_args.py:388-411). const float* video_cross_scale_shift = nullptr; // [batch, video tokens, 4 * dim] const float* video_cross_gate = nullptr; // [batch, 1, dim] diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 1c2da38a5..46aa57faa 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -98,27 +98,38 @@ // holds both copies at once, which is what wedged the box during MiniMax-H3's // port (minimax_h3.h:1598-1606). Same shape, same reason. // -// ─── WHAT THE SHIPPED DiT CARRIES THAT PHASE L2 DOES NOT PORT ──────────────── +// ─── WHAT THE SHIPPED DiT CARRIES THAT THIS PORT DOES NOT CARRY ────────────── // // MEASURED 2026-08-12 from the FP8 checkpoint's own header, and reported rather -// than absorbed. The file carries four families outside the L2 contract, and -// they fall into TWO groups that this comment used to conflate — corrected -// 2026-08-13, because the conflation is what made a downstream refusal state -// something untrue about the tree for a whole phase. +// than absorbed. The file carries four families outside the ORIGINAL L2 contract, +// and they fall into THREE groups. Two of them this comment once conflated +// (corrected 2026-08-13, because the conflation made a downstream refusal state +// something untrue about the tree for a whole phase); the third is a family that +// has since been ported. // // UNPORTED. `Ltx2LoadDitFromSafetensors` REFUSES the load by naming these, and // only an explicit `allow_unported_modules` — which exists so the ported subset // stays gateable — proceeds, still reporting every one of them in `unported`: // -// prompt_adaln_single.*, audio_prompt_adaln_single.* -// Upstream builds these only when `cross_attention_adaln AND -// use_prompt_adaln_single` (model.py:222-226, :253-257). Their presence -// means the shipped LTX-2.5 sets `use_prompt_adaln_single = TRUE`, which -// contradicts .agents/specs/ltx-2-5.md §1.2 and ltx2.h:115-117 — and with -// it the prompt-K/V "free win", whose whole premise is that the prompt -// modulation carries no timestep term. // keyframes_abs_pos_embedding [1, 4096] // So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its +// config copy, and it must stay that way: a flag cleared there is invisible +// to the contract-equality check, so clearing a PORTED one silently drops +// its tensors. +// +// PORTED 2026-08-13 — no longer named in that refusal: +// +// prompt_adaln_single.*, audio_prompt_adaln_single.* +// Upstream builds these exactly when `cross_attention_adaln AND +// use_prompt_adaln_single` (model.py:222-226, :252-256), and the flag +// defaults TRUE in every reference (model.py:77, +// model_configurator.py:76/:138, diffusers transformer_ltx2.py:1185). Their +// presence means the shipped LTX-2.5 sets it TRUE — which is what +// .agents/specs/ltx-2-5.md §1.2 established and what row LTX25-PROMPT-ADALN +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) implemented. The +// prompt-K/V cache stays correct-and-inapplicable for this checkpoint, and +// is refused by name rather than served. // // LOADED ELSEWHERE — NOT UNPORTED, and never named in that refusal: // @@ -515,11 +526,17 @@ nlohmann::json Ltx2ReadCheckpointConfig(const SafetensorsFile& file); // `Ltx2StreamDitToDevice` directly and therefore owes the same adoption. // // `allow_unported_modules` clears `use_keyframes_abs_pos_embedding` IN A COPY of -// the config before parsing, mirroring what the loader does for -// `use_prompt_adaln_single`: the flag is cleared for the CONTRACT, the module +// the config before parsing: the flag is cleared for the CONTRACT, the module // stays unported, and the checkpoint's `unported` list still names it. Without // the opt-in `ParseLtx2DitParams` throws, which is the refusal. // +// IT CLEARS EXACTLY THAT ONE FLAG, and the scoping is the rule, not an accident. +// It also used to clear `use_prompt_adaln_single`, whose module has been ported +// since 2026-08-13 — so the opt-in a real render REQUIRES was silently turning a +// correctness setting off, and the contract-equality check below could not see it +// because both sides had been forced to the same cleared value. A ported module's +// flag belongs in the contract; only a module nothing applies may be cleared here. +// // `source` names the config in every refusal, so a reader knows whether the // checkpoint declared it or a caller supplied it. Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index fb683ce54..0953aebff 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -170,11 +170,20 @@ inline constexpr char kLtx2ModelVersionExtra[] = "model_version"; // belonging to another checkpoint is refused rather than bound. inline constexpr char kLtx2DitConfigPathExtra[] = "dit_config_path"; -// Proceed past the module families the L2 contract does not carry — -// `prompt_adaln_single` / `audio_prompt_adaln_single` and +// Proceed past the module families this port does not carry — // `keyframes_abs_pos_embedding` (ltx2_loader.h). "1" opts in; anything else -// leaves the loader's refusal in place. The shipped DiTs all carry at least one -// of them, so this is the flag that says "gate the ported subset knowingly". +// leaves the loader's refusal in place. The shipped DiTs carry it, so this is the +// flag that says "gate the ported subset knowingly". +// +// IT MUST NEVER DISABLE A PORTED FEATURE, and until 2026-08-13 it did: +// `prompt_adaln_single` / `audio_prompt_adaln_single` were on this list, and +// setting the extra reached three loader assignments that cleared +// `use_prompt_adaln_single`, so every real render dropped the timestep half of +// the prompt K/V modulation — finite, same-shaped, and invisible to every gate. +// Those families are ported now (.agents/specs/ltx25-prompt-adaln.md, issue +// #644), the loader asserts the flag against the file rather than clearing it, +// and this extra is scoped to the one module nothing applies. Adding a family +// here only ever means "the forward genuinely has no code for this". // // The two `*_embeddings_connector` families are NOT in that set and this extra // has nothing to do with them: they are outside the DiT contract by design and diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 367cc4ddb..4c5d3e54b 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -316,10 +316,19 @@ def build_modalities(masked: bool, audio_enabled: bool = True, dense_self_mask: return video, audio -def build_model(rope_type_name: str, double_rope: bool): +def build_model(rope_type_name: str, double_rope: bool, prompt_adaln: bool | None = None): + """`prompt_adaln` overrides ARCH's `use_prompt_adaln_single` for the flag-ON arm. + + Every OTHER parameter keeps its name, and the deterministic stream is keyed by + NAME alone, so the two arms share their common weights bit-for-bit. That is + what makes the flag-ON and flag-OFF forwards directly subtractable, which is + how the magnitude in section 6 is measured. + """ from ltx_core.model.transformer.model import LTXModel, LTXModelType # noqa: PLC0415 from ltx_core.model.transformer.rope import LTXRopeType # noqa: PLC0415 + if prompt_adaln is None: + prompt_adaln = ARCH["use_prompt_adaln_single"] model = LTXModel( model_type=LTXModelType.AudioVideo, num_attention_heads=ARCH["num_attention_heads"], @@ -344,7 +353,7 @@ def build_model(rope_type_name: str, double_rope: bool): double_precision_rope=double_rope, apply_gated_attention=ARCH["apply_gated_attention"], cross_attention_adaln=ARCH["cross_attention_adaln"], - use_prompt_adaln_single=ARCH["use_prompt_adaln_single"], + use_prompt_adaln_single=prompt_adaln, ff_bias=ARCH["ff_bias"], audio_ff_bias=ARCH["audio_ff_bias"], ) @@ -617,12 +626,15 @@ def emit_bricks(out, model) -> None: def emit_forward( out, tag: str, rope_type_name: str, double_rope: bool, masked: bool, audio_enabled: bool = True, dense_self_mask: bool = False, + prompt_adaln: bool | None = None, ) -> None: out.write( f"// --- forward case {tag}: rope={rope_type_name} float64_freqs={double_rope} " - f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} ---\n" + f"masked={masked} audio_enabled={audio_enabled} dense_self_mask={dense_self_mask} " + f"prompt_adaln={ARCH['use_prompt_adaln_single'] if prompt_adaln is None else prompt_adaln}" + " ---\n" ) - model = build_model(rope_type_name, double_rope) + model = build_model(rope_type_name, double_rope, prompt_adaln) video, audio = build_modalities(masked, audio_enabled, dense_self_mask) with torch.no_grad(): vx, ax = model(video=video, audio=audio, perturbations=None) @@ -631,6 +643,129 @@ def emit_forward( return model +def emit_prompt_adaln(out) -> None: + """Section 6 — the PROMPT-SIDE AdaLN arm (`use_prompt_adaln_single=True`). + + This is upstream's DEFAULT (model.py:77, model_configurator.py:76 and :138; + diffusers transformer_ltx2.py:1185) and what the shipped LTX-2.5 DiT carries. + Everything above this point runs the flag OFF, so nothing above can observe a + port that drops the term — which is exactly what happened. + + Emitted here: + * the flag-ON parameter list, i.e. the 18-tensor contract the flag adds + (12 video + 6 audio), in upstream's own registration order; + * the prompt AdaLN MLP run STANDALONE on `sigma * timestep_scale_multiplier` + (transformer_args.py:274-277 -> :173-186), so a failure localizes to the + MLP rather than to the threading; + * two full forwards, unmasked and masked; + * the MAGNITUDE: flag-ON minus flag-OFF over the same shared weights. + """ + out.write("// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) ---\n") + model = build_model("split", False, prompt_adaln=True) + + names, ranks, dims = [], [], [] + for name, param in model.named_parameters(): + names.append(name) + ranks.append(len(param.shape)) + dims.extend(int(d) for d in param.shape) + out.write("inline constexpr const char* kLtx2PromptAdalnParamNames[] = {\n") + for name in names: + out.write(f' "{name}",\n') + out.write("};\n\n") + emit_i64(out, "kLtx2PromptAdalnParamRanks", ranks) + emit_i64(out, "kLtx2PromptAdalnParamDims", dims) + emit_scalar(out, "kLtx2PromptAdalnParamCount", len(names)) + out.write("\n") + + # The MLP alone. `_prepare_timestep` scales by timestep_scale_multiplier and + # feeds the modality's SIGMA -- (B,), one scalar per sample -- not its + # per-token `timesteps`. Emitting both halves keeps that distinction gateable. + scale = float(ARCH["timestep_scale_multiplier"]) + vsigma = rand_input("input.video.sigma", (BATCH,), 0.25, 0.5) * scale + asigma = rand_input("input.audio.sigma", (BATCH,), 0.25, 0.5) * scale + vmod, _ = model.prompt_adaln_single(vsigma.flatten(), hidden_dtype=torch.float32) + amod, _ = model.audio_prompt_adaln_single(asigma.flatten(), hidden_dtype=torch.float32) + emit_f32(out, "kLtx2PromptAdalnVideoTimesteps", tensor(vsigma)) + emit_f32(out, "kLtx2PromptAdalnAudioTimesteps", tensor(asigma)) + emit_f32(out, "kLtx2PromptAdalnVideoModulation", tensor(vmod)) + emit_f32(out, "kLtx2PromptAdalnAudioModulation", tensor(amod)) + + emit_forward(out, "PromptAdaln", "split", False, False, prompt_adaln=True) + emit_forward(out, "PromptAdalnMasked", "split", False, True, prompt_adaln=True) + + +def measure_prompt_adaln_magnitude() -> str: + """How far the conditioning MOVES when the term is included, vs when it is not. + + Both arms share every common parameter bit-for-bit (the stream is keyed by + parameter NAME), so the difference below is the term itself and nothing else. + Reported as a comment in the generated header AND on stderr, because "does + this matter" is a number, not an argument. + """ + lines = [] + off = build_model("split", False, prompt_adaln=False) + on = build_model("split", False, prompt_adaln=True) + video, audio = build_modalities(False) + with torch.no_grad(): + vx_off, ax_off = off(video=video, audio=audio, perturbations=None) + vx_on, ax_on = on(video=video, audio=audio, perturbations=None) + + def rel(a, b): + a32 = a.to(torch.float32) + b32 = b.to(torch.float32) + denom = float(a32.abs().max()) + return float((b32 - a32).abs().max()), ( + float((b32 - a32).abs().max()) / denom if denom > 0 else float("nan") + ) + + vabs, vrel = rel(vx_off, vx_on) + aabs, arel = rel(ax_off, ax_on) + lines.append(f"// DiT video output: max|on-off| = {vabs:.6g} ({vrel * 100:.2f}% of max|off|)") + lines.append(f"// DiT audio output: max|on-off| = {aabs:.6g} ({arel * 100:.2f}% of max|off|)") + + # The modulated prompt context of block 0, which is where the term enters. + scale = float(ARCH["timestep_scale_multiplier"]) + with torch.no_grad(): + vmod, _ = on.prompt_adaln_single( + (video.sigma * scale).flatten(), hidden_dtype=torch.float32 + ) + vmod = vmod.view(BATCH, -1, vmod.size(-1)) + table = on.transformer_blocks[0].prompt_scale_shift_table[None, None] + static = table.expand(BATCH, 1, 2, video_dim()) + full = static + vmod.reshape(BATCH, vmod.shape[1], 2, -1) + shift_off, scale_off = static.unbind(dim=2) + shift_on, scale_on = full.unbind(dim=2) + ctx = video.context + kv_off = ctx * (1 + scale_off) + shift_off + kv_on = ctx * (1 + scale_on) + shift_on + kabs, krel = rel(kv_off, kv_on) + lines.append( + f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"({krel * 100:.2f}% of max|off|)" + ) + # How much of the K/V modulation is timestep-conditioned at all: the MLP row + # against the static per-block table it is added to (transformer.py:441-443). + static_max = float(table.abs().max()) + term_max = float(vmod.abs().max()) + lines.append( + f"// timestep term vs static table: max|term| = {term_max:.6g} vs " + f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + ) + lines.append( + "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + ) + lines.append( + "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + ) + lines.append( + "// needs, not a claim about the trained checkpoint. The K/V row is where the" + ) + lines.append("// term actually enters and is the number that answers 'does this matter'.") + text = "\n".join(lines) + print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) + return text + + def emit_masks(out) -> None: out.write("// --- section 5: the prompt and self-attention masks the masked case runs ---\n") emit_i64(out, "kLtx2VideoContextMask", video_context_mask()) @@ -680,6 +815,14 @@ def main() -> int: # only this case — separates a kernel that indexes the bias by query from # one that reads bias row 0 for every query. emit_forward(out, "DenseMask", "split", False, True, dense_self_mask=True) + # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. + emit_prompt_adaln(out) + out.write( + "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// Same shared weights, same inputs, flag ON vs OFF:\n" + + measure_prompt_adaln_magnitude() + + "\n" + ) out.write("} // namespace vllm_test\n") print(f"wrote {args.out}", file=sys.stderr) return 0 diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 7e5a4cffc..3d9af69ce 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -268,17 +268,21 @@ std::vector EnumerateLtx2DitTensors(const Ltx2DitParams& p) { out.push_back({"scale_shift_table", {2, dim}}); out.push_back({"audio_scale_shift_table", {2, adim}}); + // model.py:222-226 / :252-256 — built only when BOTH flags hold, and always + // with embedding_coefficient 2 (shift + scale for the prompt K/V), never + // `adaln_embedding_coefficient()`. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + // _init_video (model.py:202-232), in child-registration order. PushLinear(out, "patchify_proj", dim, p.in_channels, true); PushAdaLayerNormSingle(out, "adaln_single", dim, coefficient); - VT_CHECK(!p.use_prompt_adaln_single, - "ltx2: use_prompt_adaln_single=true adds a prompt AdaLN MLP (model.py:223-227) whose " - "timestep term makes the cross-attention K/V uncacheable; not ported in phase L2"); + if (prompt_adaln) PushAdaLayerNormSingle(out, "prompt_adaln_single", dim, 2); PushLinear(out, "proj_out", p.out_channels, dim, true); // _init_audio (model.py:234-262). PushLinear(out, "audio_patchify_proj", adim, p.audio_in_channels, true); PushAdaLayerNormSingle(out, "audio_adaln_single", adim, coefficient); + if (prompt_adaln) PushAdaLayerNormSingle(out, "audio_prompt_adaln_single", adim, 2); PushLinear(out, "audio_proj_out", p.audio_out_channels, adim, true); // _init_audio_video (model.py:264-287); num_scale_shift_values is 4 (:133). @@ -482,9 +486,15 @@ Ltx2DitWeights BindLtx2DitWeights(const Ltx2DitParams& p, w.audio_scale_shift_table = Lookup(t, "audio_scale_shift_table"); w.patchify_proj = BindLinear(t, "patchify_proj", true); w.adaln_single = BindAdaln(t, "adaln_single"); + // model.py:222-226 / :252-256 — bound only when the module exists, exactly as + // the optional block tables above are. Left default-constructed otherwise, and + // the forward reads it only when `use_prompt_adaln_single` says it is there. + const bool prompt_adaln = p.cross_attention_adaln && p.use_prompt_adaln_single; + if (prompt_adaln) w.prompt_adaln_single = BindAdaln(t, "prompt_adaln_single"); w.proj_out = BindLinear(t, "proj_out", true); w.audio_patchify_proj = BindLinear(t, "audio_patchify_proj", true); w.audio_adaln_single = BindAdaln(t, "audio_adaln_single"); + if (prompt_adaln) w.audio_prompt_adaln_single = BindAdaln(t, "audio_prompt_adaln_single"); w.audio_proj_out = BindLinear(t, "audio_proj_out", true); w.av_ca_video_scale_shift = BindAdaln(t, "av_ca_video_scale_shift_adaln_single"); w.av_ca_audio_scale_shift = BindAdaln(t, "av_ca_audio_scale_shift_adaln_single"); diff --git a/src/vllm/model_executor/models/ltx2_device.cpp b/src/vllm/model_executor/models/ltx2_device.cpp index 3408007ab..e54436e4e 100644 --- a/src/vllm/model_executor/models/ltx2_device.cpp +++ b/src/vllm/model_executor/models/ltx2_device.cpp @@ -458,6 +458,10 @@ struct BlockArgsDev { bool video_enabled = true, audio_enabled = true; const Tensor* video_timestep_modulation = nullptr; const Tensor* audio_timestep_modulation = nullptr; + // The prompt-side AdaLN modulation, [batch, 2 * width] — nullptr is upstream's + // `prompt_timestep is None` (transformer.py:442), i.e. the flag is off. + const Tensor* video_prompt_modulation = nullptr; + const Tensor* audio_prompt_modulation = nullptr; const Tensor* video_cross_scale_shift = nullptr; const Tensor* video_cross_gate = nullptr; const Tensor* audio_cross_scale_shift = nullptr; @@ -480,6 +484,7 @@ struct BlockArgsDev { // device twin of ltx2_dit.cpp's TextCrossAttention. void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tensor& sst, const Tensor& prompt_table, const Tensor& modulation, + const Tensor* prompt_modulation, const Tensor& x_normed, const Tensor& context, const Tensor* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, @@ -501,18 +506,44 @@ void TextCrossAttentionDev(Ctx& c, const Ltx2AttentionWeights& attn, const Tenso c.s, c.s); // apply_cross_attention_adaln (transformer.py:420-447): the STATIC [2, dim] - // per-block table, with no timestep term at all when use_prompt_adaln_single is - // false (:441-443). `src_row_stride = 0` is that broadcast, and the table is - // read at F32 while the stream stays at c.s — which is the whole reason - // `modulate` carries a separate src_dtype. + // per-block table (:441), plus the prompt-side AdaLN row when the flag is on + // (:442-443). VT_CHECK(prompt_table.dtype == DType::kF32, "ltx2 device: the prompt scale-shift table is F32 in the checkpoint and is read as " "F32 here; a narrowed table would be the dtype rule applied backwards"); DBuf encoder(c.d, c.s, {batch * context_tokens, width}); c.d.b.Copy(c.d.q, encoder.ptr(), context.data, encoder.bytes()); - auto* table = prompt_table.Ptr(); - c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, - c.s, DType::kF32); + if (prompt_modulation == nullptr) { + // No timestep term at all. `src_row_stride = 0` is the broadcast of the + // table's single row over every token, and the table is read at F32 while the + // stream stays at c.s — which is the whole reason `modulate` carries a + // separate src_dtype. + auto* table = prompt_table.Ptr(); + c.k->modulate(c.d.q, encoder.t().data, table + width, table, batch * context_tokens, width, 0, + c.s, DType::kF32); + } else { + // `kv_modulation = table[None, None] + prompt_timestep.reshape(B, 1, 2, -1)` + // (:441-443) is EXACTLY `ada_value`'s `table[row] + modulation[r, row]` over + // a 2-parameter modulation with one row per BATCH element, so the sum is + // formed here the way every other table+modulation sum in this file is — + // before `(1 + scale)` applies, which is the order upstream rounds in. + DBuf shift_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/0); + DBuf scale_kv = AdaValueDev(c, prompt_table, *prompt_modulation, batch, width, + /*num_params=*/2, /*index=*/1); + // `modulate`'s `src_row_stride` is a single stride, so it can broadcast ONE + // row over every token (stride 0) or give every row its own (stride width) — + // but not "row b for this batch element's context_tokens rows". The batch + // loop supplies that offset rather than widening the kernel's contract for a + // dimension that is 1 or 2 in every shipped call. + const int64_t elem = static_cast(vt::SizeOf(c.s)); + for (int64_t b = 0; b < batch; ++b) { + void* dst = static_cast(encoder.ptr()) + b * context_tokens * width * elem; + const void* sc = static_cast(scale_kv.t().data) + b * width * elem; + const void* sh = static_cast(shift_kv.t().data) + b * width * elem; + c.k->modulate(c.d.q, dst, sc, sh, context_tokens, width, 0, c.s, c.s); + } + } AttnArgsDev a; a.batch = batch; @@ -575,7 +606,8 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args RmsNormNoWeight(c, vx_normed->t(), *video_x, rows, dim); TextCrossAttentionDev(c, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - *args.video_timestep_modulation, vx_normed->t(), *args.video_context, + *args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed->t(), *args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, c.p->num_attention_heads, c.p->attention_head_dim, *video_x); } @@ -609,6 +641,7 @@ void BlockForwardDev(Ctx& c, const Ltx2BlockWeights& w, const BlockArgsDev& args TextCrossAttentionDev(c, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, *args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed->t(), *args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, c.p->audio_num_attention_heads, c.p->audio_attention_head_dim, *audio_x); @@ -740,6 +773,8 @@ struct PreparedStreamDev { DevFreqs pe, cross_pe; std::optional cross_scale_shift; std::optional cross_gate; + // [batch, 2 * width] — empty when use_prompt_adaln_single is false. + std::optional prompt_modulation; }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -757,6 +792,7 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross, @@ -774,6 +810,19 @@ PreparedStreamDev PrepareStreamDev(Ctx& c, const Ltx2LinearWeight& patchify, out.modulation = std::move(ada.modulation); out.embedded = std::move(ada.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + AdalnOutDev prompt = PrepareTimestepDev(c, *prompt_adaln, m.sigma, m.batch, width, + c.p->timestep_scale_multiplier); + out.prompt_modulation = std::move(prompt.modulation); + } + if (m.context != nullptr && m.context_tokens > 0) { out.context = UploadStream(c.d, c.s, m.context, {m.batch * m.context_tokens, context_dim}); } @@ -892,6 +941,10 @@ void CheckWeightsResident(const Ltx2DitWeights& w, vt::Device dev) { CheckLinearResident(w.audio_proj_out, dev, "audio_proj_out"); CheckAdalnResident(w.adaln_single, dev, "adaln_single"); CheckAdalnResident(w.audio_adaln_single, dev, "audio_adaln_single"); + // Bound only when use_prompt_adaln_single is on; `CheckResident` no-ops on an + // unbound view, so this needs no flag and cannot go stale against one. + CheckAdalnResident(w.prompt_adaln_single, dev, "prompt_adaln_single"); + CheckAdalnResident(w.audio_prompt_adaln_single, dev, "audio_prompt_adaln_single"); CheckAdalnResident(w.av_ca_video_scale_shift, dev, "av_ca_video_scale_shift"); CheckAdalnResident(w.av_ca_audio_scale_shift, dev, "av_ca_audio_scale_shift"); CheckAdalnResident(w.av_ca_a2v_gate, dev, "av_ca_a2v_gate"); @@ -1044,14 +1097,18 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param OnesCache ones(d); Ctx c{d, compute_dtype, Glue(d), ¶ms, &ones}; + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; PreparedStreamDev vs = PrepareStreamDev( c, weights.patchify_proj, weights.adaln_single, weights.av_ca_video_scale_shift, - weights.av_ca_a2v_gate, *video, dim, params.in_channels, 3, + weights.av_ca_a2v_gate, prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, + params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, audio, params.cross_attention_dim); PreparedStreamDev as = PrepareStreamDev( c, weights.audio_patchify_proj, weights.audio_adaln_single, weights.av_ca_audio_scale_shift, - weights.av_ca_v2a_gate, *audio, adim, params.audio_in_channels, 1, + weights.av_ca_v2a_gate, prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, + adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, video, params.audio_cross_attention_dim); @@ -1066,6 +1123,8 @@ Ltx2DitOutputs Ltx2DitForwardDevice(vt::Queue& queue, const Ltx2DitParams& param a.audio_enabled = audio->enabled; a.video_timestep_modulation = &vs.modulation->t(); a.audio_timestep_modulation = &as.modulation->t(); + a.video_prompt_modulation = vs.prompt_modulation ? &vs.prompt_modulation->t() : nullptr; + a.audio_prompt_modulation = as.prompt_modulation ? &as.prompt_modulation->t() : nullptr; a.video_cross_scale_shift = vs.cross_scale_shift ? &vs.cross_scale_shift->t() : nullptr; a.video_cross_gate = vs.cross_gate ? &vs.cross_gate->t() : nullptr; a.audio_cross_scale_shift = as.cross_scale_shift ? &as.cross_scale_shift->t() : nullptr; diff --git a/src/vllm/model_executor/models/ltx2_dit.cpp b/src/vllm/model_executor/models/ltx2_dit.cpp index 0abd15fe3..b21655c64 100644 --- a/src/vllm/model_executor/models/ltx2_dit.cpp +++ b/src/vllm/model_executor/models/ltx2_dit.cpp @@ -114,18 +114,45 @@ void PostSelfAttention(float* x, const float* y, const std::vector& gate, } // apply_cross_attention_adaln (transformer.py:420-447). `prompt_table` is the -// STATIC [2, dim] per-block table; with use_prompt_adaln_single=false there is no -// timestep term at all (:441-443), which is exactly what makes the resulting K/V -// cacheable across denoise steps. +// STATIC [2, width] per-block table (:441). `prompt_mod` is the prompt-side AdaLN +// MLP's output for this stream, [batch, 2 * width] — shift row then scale row, +// one row per BATCH element because `_prepare_timestep` ran on the modality's +// per-sample `sigma` (transformer_args.py:274-277). It is nullptr exactly when +// upstream's `prompt_timestep is None` (:442), i.e. use_prompt_adaln_single=false +// — which is what makes the resulting K/V cacheable across denoise steps. +// +// ORDER. Upstream sums the table and the timestep row FIRST and only then +// applies `(1 + scale)` (:441-446). Folding the two additions the other way round +// would round differently, so the sum is materialized here as upstream forms it. std::vector ModulateContext(const float* context, const vt::Tensor& prompt_table, - int64_t rows, int64_t width) { - const float* shift_kv = prompt_table.Ptr(); - const float* scale_kv = prompt_table.Ptr() + width; - std::vector out(static_cast(rows * width)); - for (int64_t r = 0; r < rows; ++r) { - const float* src = context + r * width; - float* dst = out.data() + r * width; - for (int64_t c = 0; c < width; ++c) dst[c] = src[c] * (1.0f + scale_kv[c]) + shift_kv[c]; + const float* prompt_mod, int64_t batch, + int64_t context_tokens, int64_t width) { + const float* table_shift = prompt_table.Ptr(); + const float* table_scale = prompt_table.Ptr() + width; + std::vector out(static_cast(batch * context_tokens * width)); + std::vector shift_kv(static_cast(width)); + std::vector scale_kv(static_cast(width)); + for (int64_t b = 0; b < batch; ++b) { + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] = table_shift[c]; + scale_kv[static_cast(c)] = table_scale[c]; + } + if (prompt_mod != nullptr) { + const float* m = prompt_mod + b * 2 * width; + for (int64_t c = 0; c < width; ++c) { + shift_kv[static_cast(c)] += m[c]; + scale_kv[static_cast(c)] += m[width + c]; + } + } + for (int64_t s = 0; s < context_tokens; ++s) { + const int64_t r = b * context_tokens + s; + const float* src = context + r * width; + float* dst = out.data() + r * width; + for (int64_t c = 0; c < width; ++c) { + dst[c] = src[c] * (1.0f + scale_kv[static_cast(c)]) + + shift_kv[static_cast(c)]; + } + } } return out; } @@ -148,7 +175,8 @@ void AddGatedBroadcast(float* x, const std::vector& y, const std::vector< void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, const Ltx2AttentionWeights& attn, const vt::Tensor& sst, const vt::Tensor& prompt_table, const float* modulation, - const float* x_normed, const float* context, const float* context_bias, + const float* prompt_modulation, const float* x_normed, + const float* context, const float* context_bias, int64_t batch, int64_t tokens, int64_t context_tokens, int64_t width, int64_t heads, int64_t dim_head, const Ltx2CrossKv* kv_in, Ltx2CrossKv* kv_out, float* x) { @@ -174,7 +202,8 @@ void TextCrossAttention(vt::Device device, const Ltx2DitParams& params, // The modulated context is only needed when the K/V are actually recomputed. std::vector encoder; if (kv_in == nullptr) { - encoder = ModulateContext(context, prompt_table, batch * context_tokens, width); + encoder = ModulateContext(context, prompt_table, prompt_modulation, batch, context_tokens, + width); } Ltx2AttentionArgs a; @@ -250,7 +279,8 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, PostSelfAttention(video_x, msa.data(), gate, batch * tv, dim, eps, &vx_normed); TextCrossAttention(device, params, w.attn2, w.scale_shift_table, w.prompt_scale_shift_table, - args.video_timestep_modulation, vx_normed.data(), args.video_context, + args.video_timestep_modulation, args.video_prompt_modulation, + vx_normed.data(), args.video_context, args.video_context_bias, batch, tv, args.video_context_tokens, dim, params.num_attention_heads, params.attention_head_dim, args.prompt_kv_filled ? args.video_prompt_kv : nullptr, @@ -287,6 +317,7 @@ void Ltx2TransformerBlockForward(vt::Device device, const Ltx2DitParams& params, TextCrossAttention(device, params, w.audio_attn2, w.audio_scale_shift_table, w.audio_prompt_scale_shift_table, args.audio_timestep_modulation, + args.audio_prompt_modulation, ax_normed.data(), args.audio_context, args.audio_context_bias, batch, ta, args.audio_context_tokens, adim, params.audio_num_attention_heads, params.audio_attention_head_dim, @@ -432,6 +463,7 @@ struct PreparedStream { Ltx2FreqsCis cross_pe; std::vector cross_scale_shift; // [batch, tokens, 4 * width] std::vector cross_gate; // [batch, 1, width] + std::vector prompt_modulation; // [batch, 1, 2 * width], empty when the flag is off }; // _prepare_timestep (transformer_args.py:173-186) + AdaLayerNormSingle. @@ -451,6 +483,7 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, const Ltx2AdaLayerNormSingleWeights& adaln, const Ltx2AdaLayerNormSingleWeights& cross_scale_shift_adaln, const Ltx2AdaLayerNormSingleWeights& cross_gate_adaln, + const Ltx2AdaLayerNormSingleWeights* prompt_adaln, const Ltx2ModalityInput& m, int64_t width, int64_t in_channels, int64_t n_pos_dims, const std::vector& max_pos, int64_t heads, const Ltx2ModalityInput* cross) { @@ -478,6 +511,21 @@ PreparedStream PrepareStream(vt::Device device, const Ltx2DitParams& params, PrepareTimestep(device, adaln, m.timesteps, rows, width, params.timestep_scale_multiplier, &out.modulation, &out.embedded); + // transformer_args.py:274-277 — the PROMPT-side AdaLN runs on this modality's + // own SIGMA, [batch], not on its per-token `timesteps`. `_prepare_timestep` + // applies the same timestep_scale_multiplier, and the result views to + // [batch, 1, 2 * width]: one row per sample, broadcast over the prompt tokens. + if (prompt_adaln != nullptr) { + VT_CHECK(m.sigma != nullptr, + "ltx2: use_prompt_adaln_single=true needs this modality's sigma " + "(transformer_args.py:274-277); it drives the prompt-side AdaLN MLP whose output is " + "added to the cross-attention K/V modulation, and a missing sigma would silently " + "fall back to the static table"); + std::vector unused; + PrepareTimestep(device, *prompt_adaln, m.sigma, m.batch, width, + params.timestep_scale_multiplier, &out.prompt_modulation, &unused); + } + if (m.context_mask != nullptr) { out.context_bias = Ltx2PrepareContextMask(m.context_mask, m.batch, m.context_tokens); } @@ -675,19 +723,25 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, "timestep term and caching them would be wrong"); } + // model.py:222-226 / :252-256 — the module exists only when BOTH flags hold, and + // `prompt_adaln=getattr(self, "prompt_adaln_single", None)` (:313, :333) is how + // upstream turns its absence into `prompt_timestep is None`. + const bool prompt_adaln = params.cross_attention_adaln && params.use_prompt_adaln_single; const bool have_both = video != nullptr && audio != nullptr; PreparedStream vs, as; if (video != nullptr) { VT_CHECK(video->context_tokens == 0 || video->context != nullptr, "ltx2: the video stream needs a context when context_tokens > 0"); vs = PrepareStream(device, params, weights.patchify_proj, weights.adaln_single, - weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, *video, dim, + weights.av_ca_video_scale_shift, weights.av_ca_a2v_gate, + prompt_adaln ? &weights.prompt_adaln_single : nullptr, *video, dim, params.in_channels, 3, params.positional_embedding_max_pos, params.num_attention_heads, have_both ? audio : nullptr); } if (audio != nullptr) { as = PrepareStream(device, params, weights.audio_patchify_proj, weights.audio_adaln_single, - weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, *audio, adim, + weights.av_ca_audio_scale_shift, weights.av_ca_v2a_gate, + prompt_adaln ? &weights.audio_prompt_adaln_single : nullptr, *audio, adim, params.audio_in_channels, 1, params.audio_positional_embedding_max_pos, params.audio_num_attention_heads, have_both ? video : nullptr); } @@ -719,6 +773,10 @@ Ltx2DitOutputs Ltx2DitForward(vt::Device device, const Ltx2DitParams& params, a.audio_enabled = audio != nullptr && audio->enabled; a.video_timestep_modulation = vs.modulation.empty() ? nullptr : vs.modulation.data(); a.audio_timestep_modulation = as.modulation.empty() ? nullptr : as.modulation.data(); + a.video_prompt_modulation = + vs.prompt_modulation.empty() ? nullptr : vs.prompt_modulation.data(); + a.audio_prompt_modulation = + as.prompt_modulation.empty() ? nullptr : as.prompt_modulation.data(); a.video_cross_scale_shift = vs.cross_scale_shift.empty() ? nullptr : vs.cross_scale_shift.data(); a.video_cross_gate = vs.cross_gate.empty() ? nullptr : vs.cross_gate.data(); a.audio_cross_scale_shift = as.cross_scale_shift.empty() ? nullptr : as.cross_scale_shift.data(); diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index 946771019..bfc85075c 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -540,18 +540,53 @@ std::vector ContractOf(const Ltx2DitParams& params) { list += families[i]; } Fail( - "the checkpoint carries modules phase L2 does NOT port: " + list + - ". They are not dropped silently: prompt_adaln_single / " - "audio_prompt_adaln_single mean use_prompt_adaln_single is TRUE, which " - "contradicts .agents/specs/ltx-2-5.md section 1.2 and voids the prompt-K/V " - "cache's premise; keyframes_abs_pos_embedding contradicts ltx2.h:47-49. The " - "two *_embeddings_connector families are NOT in this list and never will be " - "— they are outside the DiT contract by design and are loaded by " + "the checkpoint carries modules this port does NOT carry: " + list + + ". They are not dropped silently: keyframes_abs_pos_embedding means " + "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " + "— they were ported by row LTX25-PROMPT-ADALN " + "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " + "contract whenever the checkpoint carries them. The two " + "*_embeddings_connector families are not in this list either and never will " + "be — they are outside the DiT contract by design and are loaded by " "Ltx2LoadConnectorWeights, which is what the video engine calls. Pass " "Ltx2DitLoadOptions::allow_unported_modules to load the ported SUBSET, which " "still reports every one of them."); } +// THE GUARD THAT REPLACED THREE `use_prompt_adaln_single = false` ASSIGNMENTS. +// +// Those assignments existed only so `EnumerateLtx2DitTensors` would not throw on +// a module this port did not carry. The module is carried now, so the contract +// simply includes it — and the assignments would have become a silent DROP of the +// module's 12 parameters (18 entries in the shipped FP8 manifest, which carries a +// `weight_scale` per quantized weight), reachable through +// `allow_unported_modules=1`, which is exactly the defect issue #644 row 0 fixes. +// +// So the invariant they violated is asserted instead: the resolved flag must say +// what the FILE says. Deliberately an EQUALITY. Clearing it with the tensors +// present is the old defect; setting it with them absent would bind weights that +// are not there. Either way this refuses by name rather than rendering. +void CheckPromptAdalnAgreesWithFile(const DitPlan& plan, const Ltx2DitParams& params, + const char* where) { + bool file_has = false; + for (const Ltx2TensorSpec& spec : plan.manifest) { + if (spec.name == "prompt_adaln_single.linear.weight") { + file_has = true; + break; + } + } + if (file_has == params.use_prompt_adaln_single) return; + Fail(std::string(where) + ": use_prompt_adaln_single resolved to " + + (params.use_prompt_adaln_single ? "TRUE" : "FALSE") + " while the file " + + (file_has ? "DOES" : "does NOT") + + " carry prompt_adaln_single. Upstream builds that module exactly when the flag is set " + "(model.py:222-226), so the two cannot disagree. A FALSE flag over a file that carries " + "the module would drop the timestep term from every cross-attention K/V modulation " + "(transformer.py:441-443) and render with only the static table — finite, same-shaped, " + "and invisible to every gate, which is why this is checked rather than assumed."); +} + } // namespace Ltx2DitParams Ltx2ParseDitParamsFromCheckpoint(const SafetensorsFile& file, @@ -568,9 +603,7 @@ Ltx2DitCheckpoint Ltx2LoadDitFromSafetensors(const SafetensorsFile& file, out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - // The one flag whose module this port does not carry. Cleared for the CONTRACT - // only; `checkpoint_params` keeps what the file actually says. - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2LoadDitFromSafetensors"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -623,7 +656,7 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& out.quant = plan.quant; out.checkpoint_params = ParseLtx2DitParamsFromManifest(plan.manifest); out.params = out.checkpoint_params; - out.params.use_prompt_adaln_single = false; + CheckPromptAdalnAgreesWithFile(plan, out.params, "Ltx2StreamDitToDevice"); const std::vector contract = ContractOf(out.params); out.unported = UnportedFamilies(plan, contract); @@ -976,16 +1009,20 @@ Ltx2DitParams Ltx2AdoptDeclaredDitParams(const nlohmann::json& config, // first-party LTX-2.5 DiT declares it, so reading the declared config verbatim // would refuse a real checkpoint the loader has just accepted under // `allow_unported_modules`. + // + // EXACTLY ONE FLAG, and that is now structural rather than a comment. This block + // also cleared `use_prompt_adaln_single`, whose module IS ported + // (.agents/specs/ltx25-prompt-adaln.md, issue #644) — so `allow_unported=1`, + // which a real render needs, silently turned off a correctness setting. Whatever + // is cleared here must be a module nothing below applies; a ported one belongs + // in the contract, where the equality check further down can see it. if (allow_unported_modules && copy.contains("transformer") && copy["transformer"].is_object()) { copy["transformer"]["use_keyframes_abs_pos_embedding"] = false; } nlohmann::json wrapper; wrapper["config"] = copy; - Ltx2DitParams declared = ParseLtx2DitParams(wrapper); - // The one flag the L2 contract clears, mirroring the manifest path above - // ("cleared for the CONTRACT only"). - declared.use_prompt_adaln_single = false; + const Ltx2DitParams declared = ParseLtx2DitParams(wrapper); const std::vector a = EnumerateLtx2DitTensors(from_shapes); const std::vector b = EnumerateLtx2DitTensors(declared); diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index ff1c964eb..964047788 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -2583,7 +2583,7 @@ inline constexpr float kLtx2AudioSelfMaskDense[] = { }; // --- section 4: the full DiT forward --- -// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Split: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardSplitVideo[] = { -0.112128824f, -0.0224671941f, -0.0198019743f, -0.0801180974f, -0.00492008496f, -0.0997069329f, -0.141085327f, -0.110280119f, 0.242049828f, 0.0161605272f, 0.0691268668f, -0.128027037f, @@ -2622,7 +2622,7 @@ inline constexpr float kLtx2ForwardSplitAudio[] = { -0.0946981385f, 0.0834118277f, 0.000482047704f, -0.244372234f, 0.0668168291f, 0.076439999f, }; -// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Interleaved: rope=interleaved float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardInterleavedVideo[] = { -0.111679666f, -0.0226908159f, -0.019702062f, -0.0793902054f, -0.00504480582f, -0.0992663726f, -0.141819194f, -0.110411108f, 0.240116999f, 0.0159209836f, 0.0681923628f, -0.130362436f, @@ -2661,7 +2661,7 @@ inline constexpr float kLtx2ForwardInterleavedAudio[] = { -0.0936588645f, 0.0829930827f, -0.000500017079f, -0.244381592f, 0.0684131756f, 0.0764773339f, }; -// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False --- +// --- forward case Double: rope=split float64_freqs=True masked=False audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardDoubleVideo[] = { -0.112128653f, -0.0224669855f, -0.0198022574f, -0.0801180974f, -0.00491989311f, -0.0997067019f, -0.141085491f, -0.110280029f, 0.242049828f, 0.0161606614f, 0.0691268519f, -0.128027067f, @@ -2700,7 +2700,7 @@ inline constexpr float kLtx2ForwardDoubleAudio[] = { -0.0946978033f, 0.0834115744f, 0.000482207892f, -0.244371802f, 0.0668166652f, 0.0764404312f, }; -// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False --- +// --- forward case Masked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardMaskedVideo[] = { -0.110146172f, -0.0195347276f, -0.0187677145f, -0.0824092105f, -0.00312865246f, -0.102393486f, -0.140672266f, -0.111012168f, 0.238716885f, 0.0195573885f, 0.0744788125f, -0.129781172f, @@ -2739,7 +2739,7 @@ inline constexpr float kLtx2ForwardMaskedAudio[] = { -0.0945852846f, 0.0829883665f, 0.000665711937f, -0.24394846f, 0.0671078935f, 0.0765393525f, }; -// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False --- +// --- forward case AudioOff: rope=split float64_freqs=False masked=False audio_enabled=False dense_self_mask=False prompt_adaln=False --- inline constexpr float kLtx2ForwardAudioOffVideo[] = { -0.11211022f, -0.0225408655f, -0.0197366029f, -0.0800089315f, -0.00495309103f, -0.099537693f, -0.141150832f, -0.110280804f, 0.242115393f, 0.0160462279f, 0.0691971257f, -0.127955094f, @@ -2778,7 +2778,7 @@ inline constexpr float kLtx2ForwardAudioOffAudio[] = { -0.0867422372f, 0.0840105489f, -0.000306644681f, -0.244602948f, 0.0686045066f, 0.0746730566f, }; -// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True --- +// --- forward case DenseMask: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=True prompt_adaln=False --- inline constexpr float kLtx2ForwardDenseMaskVideo[] = { -0.112229049f, -0.0196566191f, -0.0203980505f, -0.08254648f, -0.00351214409f, -0.102313384f, -0.139395878f, -0.10990762f, 0.244468585f, 0.0156435464f, 0.0681985766f, -0.126972139f, @@ -2817,4 +2817,424 @@ inline constexpr float kLtx2ForwardDenseMaskAudio[] = { -0.0944952741f, 0.0835703015f, 6.12687072e-05f, -0.244034111f, 0.0671091005f, 0.0762160644f, }; +// --- section 6: the prompt-side AdaLN arm (use_prompt_adaln_single=TRUE) --- +inline constexpr const char* kLtx2PromptAdalnParamNames[] = { + "scale_shift_table", + "audio_scale_shift_table", + "patchify_proj.weight", + "patchify_proj.bias", + "adaln_single.emb.timestep_embedder.linear_1.weight", + "adaln_single.emb.timestep_embedder.linear_1.bias", + "adaln_single.emb.timestep_embedder.linear_2.weight", + "adaln_single.emb.timestep_embedder.linear_2.bias", + "adaln_single.linear.weight", + "adaln_single.linear.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "prompt_adaln_single.linear.weight", + "prompt_adaln_single.linear.bias", + "proj_out.weight", + "proj_out.bias", + "audio_patchify_proj.weight", + "audio_patchify_proj.bias", + "audio_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_adaln_single.linear.weight", + "audio_adaln_single.linear.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_1.bias", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.weight", + "audio_prompt_adaln_single.emb.timestep_embedder.linear_2.bias", + "audio_prompt_adaln_single.linear.weight", + "audio_prompt_adaln_single.linear.bias", + "audio_proj_out.weight", + "audio_proj_out.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_video_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_video_scale_shift_adaln_single.linear.weight", + "av_ca_video_scale_shift_adaln_single.linear.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_audio_scale_shift_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_audio_scale_shift_adaln_single.linear.weight", + "av_ca_audio_scale_shift_adaln_single.linear.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_a2v_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_a2v_gate_adaln_single.linear.weight", + "av_ca_a2v_gate_adaln_single.linear.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_1.bias", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.weight", + "av_ca_v2a_gate_adaln_single.emb.timestep_embedder.linear_2.bias", + "av_ca_v2a_gate_adaln_single.linear.weight", + "av_ca_v2a_gate_adaln_single.linear.bias", + "transformer_blocks.0.scale_shift_table", + "transformer_blocks.0.audio_scale_shift_table", + "transformer_blocks.0.scale_shift_table_a2v_ca_audio", + "transformer_blocks.0.scale_shift_table_a2v_ca_video", + "transformer_blocks.0.prompt_scale_shift_table", + "transformer_blocks.0.audio_prompt_scale_shift_table", + "transformer_blocks.0.attn1.q_norm.weight", + "transformer_blocks.0.attn1.k_norm.weight", + "transformer_blocks.0.attn1.to_q.weight", + "transformer_blocks.0.attn1.to_q.bias", + "transformer_blocks.0.attn1.to_k.weight", + "transformer_blocks.0.attn1.to_k.bias", + "transformer_blocks.0.attn1.to_v.weight", + "transformer_blocks.0.attn1.to_v.bias", + "transformer_blocks.0.attn1.to_gate_logits.weight", + "transformer_blocks.0.attn1.to_gate_logits.bias", + "transformer_blocks.0.attn1.to_out.0.weight", + "transformer_blocks.0.attn1.to_out.0.bias", + "transformer_blocks.0.attn2.q_norm.weight", + "transformer_blocks.0.attn2.k_norm.weight", + "transformer_blocks.0.attn2.to_q.weight", + "transformer_blocks.0.attn2.to_q.bias", + "transformer_blocks.0.attn2.to_k.weight", + "transformer_blocks.0.attn2.to_k.bias", + "transformer_blocks.0.attn2.to_v.weight", + "transformer_blocks.0.attn2.to_v.bias", + "transformer_blocks.0.attn2.to_gate_logits.weight", + "transformer_blocks.0.attn2.to_gate_logits.bias", + "transformer_blocks.0.attn2.to_out.0.weight", + "transformer_blocks.0.attn2.to_out.0.bias", + "transformer_blocks.0.ff.net.0.proj.weight", + "transformer_blocks.0.ff.net.2.weight", + "transformer_blocks.0.audio_attn1.q_norm.weight", + "transformer_blocks.0.audio_attn1.k_norm.weight", + "transformer_blocks.0.audio_attn1.to_q.weight", + "transformer_blocks.0.audio_attn1.to_q.bias", + "transformer_blocks.0.audio_attn1.to_k.weight", + "transformer_blocks.0.audio_attn1.to_k.bias", + "transformer_blocks.0.audio_attn1.to_v.weight", + "transformer_blocks.0.audio_attn1.to_v.bias", + "transformer_blocks.0.audio_attn1.to_gate_logits.weight", + "transformer_blocks.0.audio_attn1.to_gate_logits.bias", + "transformer_blocks.0.audio_attn1.to_out.0.weight", + "transformer_blocks.0.audio_attn1.to_out.0.bias", + "transformer_blocks.0.audio_attn2.q_norm.weight", + "transformer_blocks.0.audio_attn2.k_norm.weight", + "transformer_blocks.0.audio_attn2.to_q.weight", + "transformer_blocks.0.audio_attn2.to_q.bias", + "transformer_blocks.0.audio_attn2.to_k.weight", + "transformer_blocks.0.audio_attn2.to_k.bias", + "transformer_blocks.0.audio_attn2.to_v.weight", + "transformer_blocks.0.audio_attn2.to_v.bias", + "transformer_blocks.0.audio_attn2.to_gate_logits.weight", + "transformer_blocks.0.audio_attn2.to_gate_logits.bias", + "transformer_blocks.0.audio_attn2.to_out.0.weight", + "transformer_blocks.0.audio_attn2.to_out.0.bias", + "transformer_blocks.0.audio_ff.net.0.proj.weight", + "transformer_blocks.0.audio_ff.net.0.proj.bias", + "transformer_blocks.0.audio_ff.net.2.weight", + "transformer_blocks.0.audio_ff.net.2.bias", + "transformer_blocks.0.audio_to_video_attn.q_norm.weight", + "transformer_blocks.0.audio_to_video_attn.k_norm.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.weight", + "transformer_blocks.0.audio_to_video_attn.to_q.bias", + "transformer_blocks.0.audio_to_video_attn.to_k.weight", + "transformer_blocks.0.audio_to_video_attn.to_k.bias", + "transformer_blocks.0.audio_to_video_attn.to_v.weight", + "transformer_blocks.0.audio_to_video_attn.to_v.bias", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.0.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.0.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.0.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.0.video_to_audio_attn.q_norm.weight", + "transformer_blocks.0.video_to_audio_attn.k_norm.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.weight", + "transformer_blocks.0.video_to_audio_attn.to_q.bias", + "transformer_blocks.0.video_to_audio_attn.to_k.weight", + "transformer_blocks.0.video_to_audio_attn.to_k.bias", + "transformer_blocks.0.video_to_audio_attn.to_v.weight", + "transformer_blocks.0.video_to_audio_attn.to_v.bias", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.0.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.0.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.0.video_to_audio_attn.to_out.0.bias", + "transformer_blocks.1.scale_shift_table", + "transformer_blocks.1.audio_scale_shift_table", + "transformer_blocks.1.scale_shift_table_a2v_ca_audio", + "transformer_blocks.1.scale_shift_table_a2v_ca_video", + "transformer_blocks.1.prompt_scale_shift_table", + "transformer_blocks.1.audio_prompt_scale_shift_table", + "transformer_blocks.1.attn1.q_norm.weight", + "transformer_blocks.1.attn1.k_norm.weight", + "transformer_blocks.1.attn1.to_q.weight", + "transformer_blocks.1.attn1.to_q.bias", + "transformer_blocks.1.attn1.to_k.weight", + "transformer_blocks.1.attn1.to_k.bias", + "transformer_blocks.1.attn1.to_v.weight", + "transformer_blocks.1.attn1.to_v.bias", + "transformer_blocks.1.attn1.to_gate_logits.weight", + "transformer_blocks.1.attn1.to_gate_logits.bias", + "transformer_blocks.1.attn1.to_out.0.weight", + "transformer_blocks.1.attn1.to_out.0.bias", + "transformer_blocks.1.attn2.q_norm.weight", + "transformer_blocks.1.attn2.k_norm.weight", + "transformer_blocks.1.attn2.to_q.weight", + "transformer_blocks.1.attn2.to_q.bias", + "transformer_blocks.1.attn2.to_k.weight", + "transformer_blocks.1.attn2.to_k.bias", + "transformer_blocks.1.attn2.to_v.weight", + "transformer_blocks.1.attn2.to_v.bias", + "transformer_blocks.1.attn2.to_gate_logits.weight", + "transformer_blocks.1.attn2.to_gate_logits.bias", + "transformer_blocks.1.attn2.to_out.0.weight", + "transformer_blocks.1.attn2.to_out.0.bias", + "transformer_blocks.1.ff.net.0.proj.weight", + "transformer_blocks.1.ff.net.2.weight", + "transformer_blocks.1.audio_attn1.q_norm.weight", + "transformer_blocks.1.audio_attn1.k_norm.weight", + "transformer_blocks.1.audio_attn1.to_q.weight", + "transformer_blocks.1.audio_attn1.to_q.bias", + "transformer_blocks.1.audio_attn1.to_k.weight", + "transformer_blocks.1.audio_attn1.to_k.bias", + "transformer_blocks.1.audio_attn1.to_v.weight", + "transformer_blocks.1.audio_attn1.to_v.bias", + "transformer_blocks.1.audio_attn1.to_gate_logits.weight", + "transformer_blocks.1.audio_attn1.to_gate_logits.bias", + "transformer_blocks.1.audio_attn1.to_out.0.weight", + "transformer_blocks.1.audio_attn1.to_out.0.bias", + "transformer_blocks.1.audio_attn2.q_norm.weight", + "transformer_blocks.1.audio_attn2.k_norm.weight", + "transformer_blocks.1.audio_attn2.to_q.weight", + "transformer_blocks.1.audio_attn2.to_q.bias", + "transformer_blocks.1.audio_attn2.to_k.weight", + "transformer_blocks.1.audio_attn2.to_k.bias", + "transformer_blocks.1.audio_attn2.to_v.weight", + "transformer_blocks.1.audio_attn2.to_v.bias", + "transformer_blocks.1.audio_attn2.to_gate_logits.weight", + "transformer_blocks.1.audio_attn2.to_gate_logits.bias", + "transformer_blocks.1.audio_attn2.to_out.0.weight", + "transformer_blocks.1.audio_attn2.to_out.0.bias", + "transformer_blocks.1.audio_ff.net.0.proj.weight", + "transformer_blocks.1.audio_ff.net.0.proj.bias", + "transformer_blocks.1.audio_ff.net.2.weight", + "transformer_blocks.1.audio_ff.net.2.bias", + "transformer_blocks.1.audio_to_video_attn.q_norm.weight", + "transformer_blocks.1.audio_to_video_attn.k_norm.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.weight", + "transformer_blocks.1.audio_to_video_attn.to_q.bias", + "transformer_blocks.1.audio_to_video_attn.to_k.weight", + "transformer_blocks.1.audio_to_video_attn.to_k.bias", + "transformer_blocks.1.audio_to_video_attn.to_v.weight", + "transformer_blocks.1.audio_to_video_attn.to_v.bias", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.weight", + "transformer_blocks.1.audio_to_video_attn.to_gate_logits.bias", + "transformer_blocks.1.audio_to_video_attn.to_out.0.weight", + "transformer_blocks.1.audio_to_video_attn.to_out.0.bias", + "transformer_blocks.1.video_to_audio_attn.q_norm.weight", + "transformer_blocks.1.video_to_audio_attn.k_norm.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.weight", + "transformer_blocks.1.video_to_audio_attn.to_q.bias", + "transformer_blocks.1.video_to_audio_attn.to_k.weight", + "transformer_blocks.1.video_to_audio_attn.to_k.bias", + "transformer_blocks.1.video_to_audio_attn.to_v.weight", + "transformer_blocks.1.video_to_audio_attn.to_v.bias", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.weight", + "transformer_blocks.1.video_to_audio_attn.to_gate_logits.bias", + "transformer_blocks.1.video_to_audio_attn.to_out.0.weight", + "transformer_blocks.1.video_to_audio_attn.to_out.0.bias", +}; + +inline constexpr int64_t kLtx2PromptAdalnParamRanks[] = { + 2, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, + 2, 2, 2, 2, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 2, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 1, 1, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamDims[] = { + 2, 32, 2, 16, 32, 8, 32, 32, 256, 32, 32, 32, + 32, 288, 32, 288, 32, 256, 32, 32, 32, 32, 64, 32, + 64, 8, 32, 8, 16, 6, 16, 16, 256, 16, 16, 16, + 16, 144, 16, 144, 16, 256, 16, 16, 16, 16, 32, 16, + 32, 6, 16, 6, 32, 256, 32, 32, 32, 32, 128, 32, + 128, 16, 256, 16, 16, 16, 16, 64, 16, 64, 32, 256, + 32, 32, 32, 32, 32, 32, 32, 16, 256, 16, 16, 16, + 16, 16, 16, 16, 9, 32, 9, 16, 5, 16, 5, 32, + 2, 32, 2, 16, 32, 32, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 4, 32, 4, 32, 32, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, + 32, 32, 128, 32, 32, 128, 16, 16, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 4, 16, 4, 16, 16, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, + 4, 16, 16, 16, 64, 16, 64, 16, 64, 16, 16, 16, + 16, 32, 16, 16, 16, 16, 16, 16, 16, 4, 32, 4, + 32, 16, 32, 16, 16, 16, 16, 16, 16, 32, 16, 16, + 32, 16, 4, 16, 4, 16, 16, 16, 9, 32, 9, 16, + 5, 16, 5, 32, 2, 32, 2, 16, 32, 32, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 4, 32, 4, 32, 32, + 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, + 4, 32, 4, 32, 32, 32, 128, 32, 32, 128, 16, 16, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 4, 16, 4, + 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, + 16, 16, 4, 16, 4, 16, 16, 16, 64, 16, 64, 16, + 64, 16, 16, 16, 16, 32, 16, 16, 16, 16, 16, 16, + 16, 4, 32, 4, 32, 16, 32, 16, 16, 16, 16, 16, + 16, 32, 16, 16, 32, 16, 4, 16, 4, 16, 16, 16, +}; + +inline constexpr int64_t kLtx2PromptAdalnParamCount = 226; + +inline constexpr float kLtx2PromptAdalnVideoTimesteps[] = { + 619.314514f, 424.717133f, +}; + +inline constexpr float kLtx2PromptAdalnAudioTimesteps[] = { + 473.940033f, 471.390289f, +}; + +inline constexpr float kLtx2PromptAdalnVideoModulation[] = { + 0.00287261885f, 0.0172509346f, 0.0149077997f, 0.00213927543f, 0.016918283f, 0.00749109872f, + 0.00878333207f, -0.00887233205f, -0.0209352672f, 0.0127304699f, 0.0109749483f, -0.0130382832f, + -0.00827212445f, -0.00580324279f, -0.0134638241f, 0.00964590162f, -0.016003741f, -0.0158182271f, + -0.00942324009f, -0.0159717295f, -0.00885281898f, -0.00412264559f, -0.00142920704f, -0.0168404803f, + 0.0133440131f, -0.0136787146f, 0.0163368266f, 0.00325167179f, 0.0156327151f, 0.000542653725f, + -0.0113121318f, -0.00703316741f, -0.0195021238f, 0.0151473908f, 0.0180853736f, 0.0211514644f, + 0.000123613572f, -0.00695834123f, -0.00497911917f, -0.0171213523f, 0.0199449155f, -0.0122921225f, + 0.0164053645f, 0.000113433227f, 0.00943795405f, -0.0121485656f, 0.00261268346f, 0.00783557631f, + -0.00460897852f, 0.0153093971f, 0.0194506291f, 0.014570266f, -0.0162125602f, -0.00318641192f, + -0.013586767f, 0.00904892385f, -0.0157831647f, -0.0182319973f, 0.0170140117f, 0.00303793233f, + -0.0181853306f, -0.00718653016f, 0.0158703439f, 0.00020046602f, 0.00623062998f, 0.0252011884f, + 0.0163403824f, 0.000290016062f, 0.014817901f, 0.00552449375f, 0.00866025314f, -0.00905592274f, + -0.0175954159f, 0.0110378982f, 0.0159901753f, -0.0115343379f, -0.0125738885f, -0.00668510282f, + -0.0144314552f, 0.0128096947f, -0.0149616096f, -0.0173609052f, -0.0133896163f, -0.0159855876f, + -0.00701024011f, -0.00505556632f, -0.0011741718f, -0.0174259786f, 0.00958774798f, -0.0130143911f, + 0.0113152023f, 0.00158250146f, 0.0137644717f, 0.00236273953f, -0.0104373507f, -0.000455798348f, + -0.0210349094f, 0.0129983025f, 0.0220505167f, 0.0202870686f, 0.000495404762f, -0.00149141857f, + -0.00326095428f, -0.0143762864f, 0.0244831145f, -0.01158146f, 0.00986448769f, 0.000591582619f, + 0.013319497f, -0.0125091095f, 0.00394352386f, 0.0113571035f, -0.00364819146f, 0.013945261f, + 0.0238441341f, 0.0148018515f, -0.0194255728f, -0.00321829435f, -0.0185660385f, 0.0112335477f, + -0.0152186789f, -0.0156262573f, 0.0185506362f, 0.00343540264f, -0.0173967108f, -0.00660126004f, + 0.0198770091f, -0.000762573676f, +}; + +inline constexpr float kLtx2PromptAdalnAudioModulation[] = { + 0.0187722091f, -0.00105935254f, -0.00853663217f, 0.00841550902f, -0.0137612084f, -0.0173807591f, + -0.0139704188f, -0.0109588308f, 0.00186792947f, -0.00266267685f, -0.00698805135f, -0.0129980808f, + 0.00670515466f, -0.0131825786f, 0.0064191781f, 0.00632017246f, -0.0122615304f, 0.00804066937f, + 0.0122477906f, 0.0157948378f, 0.0128126917f, -0.00316316052f, -0.00883637834f, -0.0100117959f, + -0.0133155594f, 0.00456165429f, 0.0183124971f, -0.0130120357f, 0.018200513f, 0.00201772549f, + 0.0119188903f, -0.0103679141f, 0.0201762542f, -0.000246715499f, -0.00900617056f, 0.00820883084f, + -0.0132663827f, -0.0180532821f, -0.0145739987f, -0.0113234669f, 0.00138104265f, -0.00278934441f, + -0.00686988048f, -0.013545204f, 0.00675359881f, -0.0138096726f, 0.00569557305f, 0.00596735114f, + -0.0114674894f, 0.00765935751f, 0.0126606431f, 0.0161463916f, 0.0141690942f, -0.00382941356f, + -0.0089492118f, -0.00970011577f, -0.0132083837f, 0.00381002505f, 0.0185851343f, -0.0123645896f, + 0.0179123916f, 0.00177870644f, 0.0127155371f, -0.0096385479f, +}; + +// --- forward case PromptAdaln: rope=split float64_freqs=False masked=False audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnVideo[] = { + -0.112144604f, -0.0224400591f, -0.0198658258f, -0.0801219419f, -0.00492565054f, -0.0996806622f, + -0.141095862f, -0.110312767f, 0.242095038f, 0.0162332151f, 0.0690327138f, -0.127996832f, + -0.327992439f, -0.067813158f, -0.0181498341f, 0.0133159151f, -0.285277247f, 0.0288095567f, + -0.103010967f, 0.0120137185f, 0.175781652f, -0.133603424f, -0.20695813f, -0.182875797f, + -0.275936812f, -0.0216972958f, 0.194740638f, -0.250197232f, -0.196145147f, -0.0676094294f, + -0.115069151f, -0.0885243863f, -0.0418822765f, 0.120427184f, -0.0467535444f, -0.166882023f, + -0.145086944f, -0.127097756f, -0.125268698f, -0.0937946662f, -0.10143815f, -0.0346935503f, + 0.347262383f, -0.0154392701f, 0.0253714621f, -0.151571572f, 0.0893032998f, -0.217105299f, + -0.113787532f, 0.107526913f, 0.111182854f, -0.0174661856f, 0.0259091742f, -0.199445173f, + -0.0387741439f, -0.157782555f, 0.159147635f, 0.0321634449f, 0.232984737f, -0.122400187f, + -0.27637139f, -0.0666981712f, 0.0339653119f, -0.140525416f, 0.0675620735f, -0.120439962f, + 0.0212921053f, 0.112628169f, 0.190132841f, -0.144298837f, -0.0116734616f, -0.159263715f, + 0.0784623772f, -0.179399416f, -0.328205913f, -0.0860405788f, -0.0919508785f, -0.0483957306f, + -0.30280292f, 0.158935979f, -0.127778456f, -0.174414024f, 0.315983474f, -0.0398868211f, + -0.11850293f, 0.0652811453f, -0.0993542522f, -0.234619945f, -0.284468323f, -0.183528736f, + 0.225024208f, -0.049902156f, -0.0650495142f, -0.0445851982f, -0.193018377f, -0.149498582f, + -0.0479458123f, -0.0666453466f, -0.0906915069f, 0.0227309261f, 0.0560047664f, -0.146587655f, + -0.172089651f, -0.0502583086f, 0.0590079799f, 0.19477278f, 0.218966454f, -0.112773709f, + -0.16504398f, -0.111266837f, 0.15254502f, -0.120294176f, 0.114031032f, -0.0505410358f, + -0.150275424f, -0.024936026f, -0.139517337f, -0.00708480086f, -0.160885945f, 0.0312023032f, + 0.27590549f, 0.0162312202f, -0.389412105f, -0.066493772f, -0.005505112f, -0.114737555f, + -0.136138007f, 0.0470205061f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnAudio[] = { + -0.00688289292f, 0.0557257049f, -0.169716612f, 0.136525676f, -0.0794674978f, -0.106710114f, + -0.109623641f, 0.0357984938f, -0.0222658273f, -0.0561756417f, 0.145198658f, 0.0478727296f, + 0.0297143031f, 0.0544307008f, -0.205607757f, 0.122264318f, -0.105997331f, -0.191682264f, + -0.0355675966f, 0.0435401984f, -0.107154176f, 0.192362309f, -0.0537107848f, -0.080251269f, + -0.0339412056f, -0.0676042289f, -0.161326066f, -0.0567146167f, 0.166392133f, -0.0363745391f, + -0.00992792659f, 0.0993538871f, -0.196374968f, 0.146272391f, -0.0314919427f, -0.147439599f, + 0.00373091619f, 0.119020537f, -0.16169925f, 0.149572372f, -0.0535437092f, -0.17913793f, + -0.0729758665f, 0.0560201257f, -0.103747681f, 0.0615522824f, 0.147424534f, -0.0767256021f, + -0.0455891043f, -0.074104555f, -0.0908109695f, 0.14269349f, 0.00909528229f, 0.000857369974f, + -0.094678238f, 0.0833884105f, 0.000516908942f, -0.244370729f, 0.0668335408f, 0.0764729604f, +}; + +// --- forward case PromptAdalnMasked: rope=split float64_freqs=False masked=True audio_enabled=True dense_self_mask=False prompt_adaln=True --- +inline constexpr float kLtx2ForwardPromptAdalnMaskedVideo[] = { + -0.110144161f, -0.0195061471f, -0.0188482404f, -0.0824035481f, -0.00313981529f, -0.102331489f, + -0.140647486f, -0.111037448f, 0.238786951f, 0.019598389f, 0.0744047314f, -0.129760861f, + -0.326935619f, -0.0698587522f, -0.0180070139f, 0.0129257301f, -0.282629013f, 0.0306178499f, + -0.100754961f, 0.00948145799f, 0.175529778f, -0.135220498f, -0.206774727f, -0.182601333f, + -0.275893211f, -0.0184815805f, 0.198321715f, -0.251868665f, -0.194811657f, -0.0697973669f, + -0.114349291f, -0.089783825f, -0.0404667407f, 0.121526636f, -0.0449657179f, -0.170914561f, + -0.144608453f, -0.129623294f, -0.126025274f, -0.0928756446f, -0.0987707227f, -0.0303842705f, + 0.349234581f, -0.0163635593f, 0.0282549523f, -0.152388036f, 0.0889601111f, -0.216030046f, + -0.112322196f, 0.108959578f, 0.112902433f, -0.0188521128f, 0.0273712836f, -0.20055984f, + -0.0387175046f, -0.156802967f, 0.156722441f, 0.0369930752f, 0.239927039f, -0.124359809f, + -0.273358971f, -0.0692257583f, 0.0352873057f, -0.141093776f, 0.0673692524f, -0.120450936f, + 0.0237387791f, 0.111967109f, 0.187116802f, -0.139871866f, -0.011316698f, -0.153532162f, + 0.0786181018f, -0.177252695f, -0.331035018f, -0.0842280909f, -0.0938364938f, -0.0409911275f, + -0.301013172f, 0.167324454f, -0.124323212f, -0.174904421f, 0.316507936f, -0.0429520756f, + -0.120540164f, 0.067215845f, -0.101051688f, -0.225887731f, -0.286542743f, -0.182181612f, + 0.224588186f, -0.049729839f, -0.0685416833f, -0.0383053645f, -0.189322844f, -0.138665244f, + -0.0496713445f, -0.068905741f, -0.093425855f, 0.0234508831f, 0.0515494831f, -0.140059441f, + -0.166952908f, -0.041921705f, 0.0591960102f, 0.194034085f, 0.216313154f, -0.111525752f, + -0.165100992f, -0.10801211f, 0.152435049f, -0.115520097f, 0.110954985f, -0.0485403165f, + -0.152800724f, -0.0228928309f, -0.138780624f, -0.00368307461f, -0.157897815f, 0.0359253138f, + 0.277324349f, 0.0178819038f, -0.393882304f, -0.0665938482f, -0.00776502956f, -0.109243833f, + -0.134122461f, 0.0536325052f, +}; + +inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { + -0.00623655878f, 0.0550939925f, -0.169768706f, 0.134756476f, -0.0800285041f, -0.106801011f, + -0.109707221f, 0.0349297225f, -0.0223909412f, -0.0580404438f, 0.145440862f, 0.0479248241f, + 0.0300626066f, 0.0543725975f, -0.205051288f, 0.121049382f, -0.106699832f, -0.19155553f, + -0.0357390642f, 0.043708574f, -0.106640562f, 0.191323146f, -0.0545797087f, -0.0801633745f, + -0.0338453911f, -0.067704387f, -0.161020443f, -0.0588866882f, 0.16616787f, -0.0362463593f, + -0.00972275995f, 0.0983785167f, -0.197862387f, 0.146195069f, -0.0301159211f, -0.147994176f, + 0.00437853113f, 0.118179567f, -0.162635818f, 0.14993386f, -0.0523924604f, -0.179551288f, + -0.0724292099f, 0.0543062873f, -0.103864521f, 0.0613403916f, 0.14883852f, -0.0763024017f, + -0.0452010371f, -0.0744607076f, -0.0918518007f, 0.142673746f, 0.0106131276f, 5.35193831e-05f, + -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, +}; + +// --- the MEASURED magnitude of the prompt-AdaLN term --- +// Same shared weights, same inputs, flag ON vs OFF: +// DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) +// DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) +// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// The two output rows are bounded by this generator's SYNTHETIC weight scale +// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate +// needs, not a claim about the trained checkpoint. The K/V row is where the +// term actually enters and is the number that answers 'does this matter'. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index dd6a8e077..f00771f38 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -135,6 +135,16 @@ Ltx2DitParams ReducedParams(Ltx2RopeType rope_type, bool double_precision) { return p; } +// Upstream's DEFAULT arm (model.py:77, model_configurator.py:76/:138, diffusers +// transformer_ltx2.py:1185) and the one the shipped LTX-2.5 DiT carries: the +// prompt-side AdaLN MLP is built, so the cross-attention K/V modulation carries a +// timestep term on top of the static per-block table (transformer.py:441-443). +Ltx2DitParams ReducedParamsPromptAdaln(Ltx2RopeType rope_type, bool double_precision) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = true; + return p; +} + // A materialized weight set: owned f32 storage plus the views the forward takes. struct WeightSet { std::map> storage; @@ -831,6 +841,137 @@ TEST_CASE("ltx2 forward: a disabled audio stream still feeds audio->video") { /*audio_enabled=*/false); } +// --------------------------------------------------------------------------- +// The prompt-side AdaLN arm — upstream's DEFAULT +// (.agents/specs/ltx25-prompt-adaln.md, issue #644) +// --------------------------------------------------------------------------- + +namespace { + +// The same forward, run with `use_prompt_adaln_single = true`. Kept separate from +// CheckForward rather than folded into it, because the two arms have DIFFERENT +// weight contracts (12 extra parameters) and sharing one helper would hide which +// contract a case ran under. +vllm::Ltx2DitOutputs RunPromptAdalnForward(const Ltx2DitParams& p, WeightSet& set, + Modalities* m, bool masked) { + BuildModalities(m, masked); + return Ltx2DitForward(Cpu(), p, set.weights, &m->video, &m->audio, vt::DType::kF32); +} + +void CheckPromptAdalnForward(bool masked, const float* want_video, const float* want_audio, + const char* label) { + INFO("prompt-AdaLN forward case: " << std::string(label)); + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, masked); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, want_video, vcount); + const double adiff = MaxAbsDiff(out.audio, want_audio, acount); + MESSAGE("max|diff| video=" << vdiff << " audio=" << adiff); + CHECK(vdiff < kRoundOff); + CHECK(adiff < kRoundOff); +} + +} // namespace + +// The 12 parameters the flag adds (6 per stream: three linears x weight+bias), +// which the shipped FP8 checkpoint carries as 18 manifest entries because each +// quantized weight also has a `weight_scale`. Order matters as much as presence: +// upstream registers `prompt_adaln_single` between `adaln_single` and `proj_out` +// inside `_init_video` (model.py:222-232), and the audio twin in `_init_audio` +// (:252-262). +TEST_CASE("ltx2 layout: the flag-ON contract matches upstream named_parameters()") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + const std::vector manifest = EnumerateLtx2DitTensors(p); + REQUIRE(static_cast(manifest.size()) == vllm_test::kLtx2PromptAdalnParamCount); + // The flag is the ONLY difference, so the count delta is exactly the module. + CHECK(vllm_test::kLtx2PromptAdalnParamCount - vllm_test::kLtx2ParamCount == 12); + size_t dim_cursor = 0; + for (size_t i = 0; i < manifest.size(); ++i) { + CAPTURE(i); + CAPTURE(manifest[i].name); + CHECK(manifest[i].name == std::string(vllm_test::kLtx2PromptAdalnParamNames[i])); + const int64_t rank = vllm_test::kLtx2PromptAdalnParamRanks[i]; + REQUIRE(static_cast(manifest[i].shape.size()) == rank); + for (int64_t d = 0; d < rank; ++d) { + CHECK(manifest[i].shape[static_cast(d)] == + vllm_test::kLtx2PromptAdalnParamDims[dim_cursor]); + ++dim_cursor; + } + } +} + +// The MLP on its own, so a failure localizes here rather than in the threading. +// The input is the modality's SIGMA scaled by timestep_scale_multiplier +// (transformer_args.py:274-277 -> :173-186) — one scalar per batch element, NOT +// the per-token `timesteps` the main AdaLN consumes. +TEST_CASE("ltx2 brick: the prompt AdaLN MLP runs on sigma, not on the per-token timesteps") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + const int64_t b = vllm_test::kLtx2Batch; + const int64_t dim = p.inner_dim(); + const int64_t adim = p.audio_inner_dim(); + + std::vector vts(vllm_test::kLtx2PromptAdalnVideoTimesteps, + vllm_test::kLtx2PromptAdalnVideoTimesteps + b); + const vllm::Ltx2AdalnOut vout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.prompt_adaln_single, vts.data(), b, dim); + REQUIRE(vout.modulation.size() == static_cast(b * 2 * dim)); + CHECK(MaxAbsDiff(vout.modulation, vllm_test::kLtx2PromptAdalnVideoModulation, + static_cast(b * 2 * dim)) < kRoundOff); + + std::vector ats(vllm_test::kLtx2PromptAdalnAudioTimesteps, + vllm_test::kLtx2PromptAdalnAudioTimesteps + b); + const vllm::Ltx2AdalnOut aout = vllm::Ltx2AdaLayerNormSingle( + Cpu(), set.weights.audio_prompt_adaln_single, ats.data(), b, adim); + REQUIRE(aout.modulation.size() == static_cast(b * 2 * adim)); + CHECK(MaxAbsDiff(aout.modulation, vllm_test::kLtx2PromptAdalnAudioModulation, + static_cast(b * 2 * adim)) < kRoundOff); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm") { + CheckPromptAdalnForward(false, vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, "prompt AdaLN"); +} + +TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { + CheckPromptAdalnForward(true, vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, + "prompt AdaLN + masks"); +} + +// THE INSTRUMENT THAT MAKES THE ARM ABOVE MEAN SOMETHING. +// +// Both goldens come from the same deterministic weight stream, keyed by parameter +// NAME, so every weight the two arms share is bit-identical and the ONLY thing +// separating `kLtx2ForwardPromptAdaln*` from `kLtx2ForwardSplit*` is the timestep +// term. A port that accepted the flag, bound the 12 tensors and then never added +// their output would reproduce the flag-OFF numbers exactly and pass nothing here. +// +// Measured on this fixture (generator stderr, and the comment block at the end of +// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block +// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and +// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound +// below is set at 20x kRoundOff so it is comfortably inside the measured signal +// and comfortably outside f32 noise. +TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { + const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); + WeightSet set = BuildWeights(p); + Modalities m; + const vllm::Ltx2DitOutputs out = RunPromptAdalnForward(p, set, &m, false); + const size_t vcount = static_cast(m.video.batch * m.video.tokens * p.out_channels); + const size_t acount = + static_cast(m.audio.batch * m.audio.tokens * p.audio_out_channels); + const double vdiff = MaxAbsDiff(out.video, vllm_test::kLtx2ForwardSplitVideo, vcount); + const double adiff = MaxAbsDiff(out.audio, vllm_test::kLtx2ForwardSplitAudio, acount); + MESSAGE("flag-ON vs flag-OFF: video=" << vdiff << " audio=" << adiff); + CHECK(vdiff > 20.0 * kRoundOff); + CHECK(adiff > 20.0 * kRoundOff); +} + TEST_CASE("ltx2 forward: a single-stream model type is REFUSED") { const Ltx2DitParams p = ReducedParams(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/models/test_ltx2_device.cpp b/tests/vllm/models/test_ltx2_device.cpp index 46bea0b78..cafef9ceb 100644 --- a/tests/vllm/models/test_ltx2_device.cpp +++ b/tests/vllm/models/test_ltx2_device.cpp @@ -304,8 +304,9 @@ struct CaseResult { CaseResult RunDeviceCase(vt::Queue& q, vt::DType stream, Ltx2RopeType rope_type, bool double_precision, bool masked, const float* want_video, const float* want_audio, bool audio_enabled = true, - bool dense_self_mask = false) { - const Ltx2DitParams p = ReducedParams(rope_type, double_precision); + bool dense_self_mask = false, bool prompt_adaln = false) { + Ltx2DitParams p = ReducedParams(rope_type, double_precision); + p.use_prompt_adaln_single = prompt_adaln; WeightSet set = BuildWeights(p); // Staged at the SAME dtype the stream computes in. Staging f32 weights under a // bf16 stream would compare a DIFFERENT MODEL, not a different dtype policy. @@ -406,6 +407,74 @@ void CheckBf16Stream(vt::Queue& q, const char* label) { CHECK(bf16.video > kDeviceRoundOff); } +// The prompt-side AdaLN arm — upstream's DEFAULT and what the shipped DiT runs +// (.agents/specs/ltx25-prompt-adaln.md, issue #644). The device path forms +// `kv_modulation = table + prompt_timestep` (transformer.py:441-443) through +// `ada_value` and then broadcasts one row per BATCH element over that element's +// prompt tokens, which is a DIFFERENT `modulate` call shape from the static +// arm — so it needs its own case rather than riding on the loop above. +void CheckPromptAdalnCases(vt::Queue& q, const char* label) { + { + INFO(label << " / prompt AdaLN, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // The masked case: the prompt mask and the prompt modulation act on the SAME + // context tensor, and a per-batch broadcast written as a per-token one would + // survive the unmasked case at batch 1. + INFO(label << " / prompt AdaLN + masks, f32"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, true, + vllm_test::kLtx2ForwardPromptAdalnMaskedVideo, + vllm_test::kLtx2ForwardPromptAdalnMaskedAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN masked f32: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kDeviceRoundOff); + CHECK(r.audio < kDeviceRoundOff); + } + { + // AND IT IS LOAD-BEARING. The flag-ON and flag-OFF goldens share every common + // weight bit-for-bit, so a device path that bound the module and never added + // its output would reproduce the flag-OFF numbers exactly. + // + // The bound is `kDeviceRoundOff` itself, and that is the precise statement: + // the flag-ON run must miss the flag-OFF golden by MORE than the tolerance + // the flag-OFF case is held to, or a dropped term would pass that gate. + // Measured here: video 1.46e-4 (7.3x the bound), audio 7.37e-5 (3.7x). + INFO(label << " / prompt AdaLN is load-bearing"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kF32, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardSplitVideo, vllm_test::kLtx2ForwardSplitAudio, + /*audio_enabled=*/true, /*dense_self_mask=*/false, + /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN vs flag-OFF golden: video=" << r.video << " audio=" << r.audio); + CHECK(r.video > kDeviceRoundOff); + CHECK(r.audio > kDeviceRoundOff); + } + { + // The bf16 PRODUCTION stream on the same arm: `ada_value` stores the + // table+timestep sum at the stream dtype, so this is where a bf16 store of the + // new sum is exercised at all. + INFO(label << " / prompt AdaLN, bf16"); + const CaseResult r = + RunDeviceCase(q, vt::DType::kBF16, Ltx2RopeType::kSplit, false, false, + vllm_test::kLtx2ForwardPromptAdalnVideo, + vllm_test::kLtx2ForwardPromptAdalnAudio, /*audio_enabled=*/true, + /*dense_self_mask=*/false, /*prompt_adaln=*/true); + MESSAGE("prompt AdaLN bf16: max|diff| video=" << r.video << " audio=" << r.audio); + CHECK(r.video < kBf16RoundOff); + CHECK(r.audio < kBf16RoundOff); + CHECK(r.video > kDeviceRoundOff); + } +} + vt::Backend* TryCuda() { return vt::TryGetBackend(vt::DeviceType::kCUDA); } } // namespace @@ -527,6 +596,21 @@ TEST_CASE("ltx2 device: the DEVICE-RESIDENT forward matches upstream on CUDA") { CheckAllForwardCases(q, "cuda"); } +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream (CPU backend)") { + vt::Queue q{Cpu(), nullptr}; + CheckPromptAdalnCases(q, "cpu-backend"); +} + +TEST_CASE("ltx2 device: the prompt-side AdaLN arm matches upstream on CUDA") { + vt::Backend* cuda = TryCuda(); + if (cuda == nullptr) { + MESSAGE("SKIP: no CUDA backend registered"); + return; + } + vt::Queue q = cuda->CreateQueue(); + CheckPromptAdalnCases(q, "cuda"); +} + TEST_CASE("ltx2 device: the bf16 PRODUCTION stream matches upstream (CPU backend)") { vt::Queue q{Cpu(), nullptr}; CheckBf16Stream(q, "cpu-backend bf16"); diff --git a/tests/vllm/models/test_ltx2_loader.cpp b/tests/vllm/models/test_ltx2_loader.cpp index 654ecd134..e191d958b 100644 --- a/tests/vllm/models/test_ltx2_loader.cpp +++ b/tests/vllm/models/test_ltx2_loader.cpp @@ -827,8 +827,10 @@ TEST_CASE("ltx2 loader: the shipped FP8 DiT manifest is fully accounted for") { CHECK(bf16 == 2284); CHECK(f32 == 290); // the tables; the 1775 scalar scales were counted above - // The four families outside the phase-L2 contract, named so their absence - // from the port cannot be discovered later. + // The five families the file carries beyond the ORIGINAL L2 contract, named so + // their status cannot be discovered later. Two are ported (the prompt AdaLN + // pair), two are loaded elsewhere (the connectors), one is genuinely unported + // (keyframes). CHECK(families.count("prompt_adaln_single") == 1); CHECK(families.count("audio_prompt_adaln_single") == 1); CHECK(families.count("keyframes_abs_pos_embedding") == 1); @@ -886,18 +888,19 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D CHECK(p.cross_attention_adaln); CHECK_FALSE(p.ff_bias); CHECK(p.audio_ff_bias); - // MEASURED, and it contradicts .agents/specs/ltx-2-5.md section 1.2 and - // ltx2.h:115-117: the SHIPPED checkpoint carries prompt_adaln_single, which - // upstream builds only when use_prompt_adaln_single is TRUE (model.py:222-226). - // The prompt-K/V cache's premise does not hold for this checkpoint. + // MEASURED: the SHIPPED checkpoint carries prompt_adaln_single, which upstream + // builds only when use_prompt_adaln_single is TRUE (model.py:222-226), so the + // prompt-K/V cache's premise does not hold for this checkpoint + // (.agents/specs/ltx-2-5.md §1.2) and the module is PORTED + // (.agents/specs/ltx25-prompt-adaln.md, issue #644). CHECK(p.use_prompt_adaln_single); - // Enumerate the contract for the subset this port DOES carry, and require - // every one of its names in the file. - Ltx2DitParams contract = p; - contract.use_prompt_adaln_single = false; - const std::vector want = vllm::EnumerateLtx2DitTensors(contract); - CHECK(want.size() == 4078); + // Enumerate the contract AS THE FILE DESCRIBES IT — no flag is forced here any + // more — and require every one of its names in the file. This line used to read + // `contract.use_prompt_adaln_single = false`, which is the shape of the defect: + // the gate agreed with the port about a module they were both dropping. + const std::vector want = vllm::EnumerateLtx2DitTensors(p); + CHECK(want.size() == 4078 + 12); int64_t missing = 0; std::string first_missing; for (const Ltx2TensorSpec& spec : want) { @@ -913,15 +916,16 @@ TEST_CASE("ltx2 loader: the L2 contract's every name is present in the shipped D // ... and account for every name the file has that the contract does not, so // "the rest is fine" is a counted claim rather than a hope. // 258 = 2 connectors x (8 blocks x 16 + 1 learnable_registers) - // 12 = prompt_adaln_single + audio_prompt_adaln_single, 6 tensors each // 1 = keyframes_abs_pos_embedding + // The 12 prompt-AdaLN tensors are no longer here: they moved INTO `want` when + // the module was ported, which is the whole delta this row landed. std::set want_set; for (const Ltx2TensorSpec& spec : want) want_set.insert(spec.name); int64_t extra = 0; for (const std::string& name : present) { if (want_set.count(name) == 0) ++extra; } - CHECK(extra == 258 + 12 + 1); + CHECK(extra == 258 + 1); } // =========================================================================== @@ -1542,8 +1546,7 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit( p, Ltx2DitQuant::kFp8, - {"prompt_adaln_single.linear.weight", "audio_prompt_adaln_single.linear.weight", - "keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", + {"keyframes_abs_pos_embedding", "video_embeddings_connector.learnable_registers", "audio_embeddings_connector.learnable_registers"}); const std::string path = TmpPath("unported"); WriteSafetensors(syn.entries, path); @@ -1557,12 +1560,18 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" } const std::string what_msg = "what: " + what; INFO(what_msg); - // THREE families now, not five. `audio_prompt_adaln_single` is here because it - // goes through the identical generic FamilyOf path as its video twin and - // covering it literally is one line. - CHECK(what.find("prompt_adaln_single") != std::string::npos); - CHECK(what.find("audio_prompt_adaln_single") != std::string::npos); - CHECK(what.find("keyframes_abs_pos_embedding") != std::string::npos); + // ONE family in the LIST now, not five. `prompt_adaln_single` / + // `audio_prompt_adaln_single` left it when they were PORTED (issue #644); the + // case below proves a checkpoint carrying them needs no opt-in at all. + // + // The list is asserted as a whole rather than by substring, because the message + // deliberately goes on to NAME the families that are not in it — a + // `find(...) == npos` over the whole message would only test the prose. + const std::string head = "does NOT carry: "; + const size_t at = what.find(head); + REQUIRE(at != std::string::npos); + const std::string list = what.substr(at + head.size(), what.find('.', at) - at - head.size()); + CHECK(list == "keyframes_abs_pos_embedding"); // THE TWO CONNECTOR FAMILIES ARE NOT UNPORTED AS OF PHASE L9c. They are // outside the DiT's contract by design — upstream loads them into the text @@ -1574,17 +1583,15 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" CHECK(what.find("video_embeddings_connector") == std::string::npos); CHECK(what.find("audio_embeddings_connector") == std::string::npos); - // The opt-in still REPORTS every one of the three; it does not make them vanish. + // The opt-in still REPORTS it; it does not make it vanish. Ltx2DitLoadOptions options; options.allow_unported_modules = true; const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file, options); - CHECK(ck.unported.size() == 3); + CHECK(ck.unported.size() == 1); for (const std::string& family : ck.unported) { CHECK(family != "video_embeddings_connector"); CHECK(family != "audio_embeddings_connector"); } - CHECK(ck.checkpoint_params.use_prompt_adaln_single); - CHECK_FALSE(ck.params.use_prompt_adaln_single); std::remove(path.c_str()); // AND THE CONNECTOR-ONLY CHECKPOINT LOADS WITH NO OPT-IN AT ALL, which is the @@ -1604,6 +1611,131 @@ TEST_CASE("ltx2 loader: the unported families are refused by name, not absorbed" std::remove(conn_path.c_str()); } +// THE REGRESSION GATE FOR ISSUE #644 ROW 0. +// +// A checkpoint that carries `prompt_adaln_single` — which the shipped LTX-2.5 DiT +// does — used to be refused without `allow_unported_modules=1`, and setting that +// extra reached three assignments that CLEARED `use_prompt_adaln_single`. So the +// only way to load a real DiT was also the way to silently drop the timestep half +// of every cross-attention K/V modulation. +// +// Both halves are asserted here, and neither is true by construction: the load +// with no opt-in exercises the contract, and the resolved flag is read back off +// the checkpoint the loader actually bound. +TEST_CASE("ltx2 loader: a DiT carrying prompt_adaln_single loads with NO opt-in") { + Ltx2DitParams p = TinyParams(); + p.use_prompt_adaln_single = true; + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string path = TmpPath("prompt_adaln"); + WriteSafetensors(syn.entries, path); + const SafetensorsFile file = SafetensorsFile::Open(path); + + // No options at all: the families are in the contract, so nothing is unported. + const vllm::Ltx2DitCheckpoint ck = vllm::Ltx2LoadDitFromSafetensors(file); + CHECK(ck.unported.empty()); + CHECK(ck.checkpoint_params.use_prompt_adaln_single); + // The flag the FORWARD reads. This is the assertion the defect broke: it was + // false while `checkpoint_params` said true. + CHECK(ck.params.use_prompt_adaln_single); + // And the weights are bound, not merely enumerated — a contract that listed the + // tensors while `BindLtx2DitWeights` left the views null would render the same + // wrong picture with a different failure mode. + CHECK(ck.weights.prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.data != nullptr); + CHECK(ck.weights.prompt_adaln_single.linear.weight.shape[0] == 2 * p.inner_dim()); + CHECK(ck.weights.audio_prompt_adaln_single.linear.weight.shape[0] == 2 * p.audio_inner_dim()); + // The 12 tensors, counted rather than assumed. + Ltx2DitParams off = p; + off.use_prompt_adaln_single = false; + CHECK(vllm::EnumerateLtx2DitTensors(p).size() - + vllm::EnumerateLtx2DitTensors(off).size() == + 12); + std::remove(path.c_str()); + + // AND THE OPT-IN CANNOT UNDO IT. `allow_unported_modules` is what a real render + // still passes for `keyframes_abs_pos_embedding`; it must leave a ported feature + // alone. Same bytes, opt-in set, same resolved flag. + const std::string path2 = TmpPath("prompt_adaln_optin"); + WriteSafetensors(syn.entries, path2); + const SafetensorsFile file2 = SafetensorsFile::Open(path2); + Ltx2DitLoadOptions options; + options.allow_unported_modules = true; + const vllm::Ltx2DitCheckpoint ck2 = vllm::Ltx2LoadDitFromSafetensors(file2, options); + CHECK(ck2.params.use_prompt_adaln_single); + CHECK(ck2.weights.prompt_adaln_single.linear.weight.data != nullptr); + std::remove(path2.c_str()); +} + +// The DECLARED config path. `Ltx2AdoptDeclaredDitParams` no longer forces both +// sides of its comparison to a cleared flag, so a config that disagrees with the +// file's shapes about `use_prompt_adaln_single` now produces two DIFFERENT +// contracts and is refused — which is an INPUT-driven gate, not a mutation-only +// one. Both directions are checked, because the equality is the point. +TEST_CASE("ltx2 loader: a config that disagrees about use_prompt_adaln_single is REFUSED") { + Ltx2DitParams shapes = TinyParams(); + shapes.use_prompt_adaln_single = true; + + nlohmann::json t; + t["num_attention_heads"] = shapes.num_attention_heads; + t["attention_head_dim"] = shapes.attention_head_dim; + t["in_channels"] = shapes.in_channels; + t["out_channels"] = shapes.out_channels; + t["num_layers"] = shapes.num_layers; + t["cross_attention_dim"] = shapes.cross_attention_dim; + t["audio_num_attention_heads"] = shapes.audio_num_attention_heads; + t["audio_attention_head_dim"] = shapes.audio_attention_head_dim; + t["audio_in_channels"] = shapes.audio_in_channels; + t["audio_out_channels"] = shapes.audio_out_channels; + t["audio_cross_attention_dim"] = shapes.audio_cross_attention_dim; + t["apply_gated_attention"] = true; + t["cross_attention_adaln"] = true; + t["ff_bias"] = false; + t["audio_ff_bias"] = true; + t["rope_type"] = "split"; + // The checks ParseLtx2DitParams asserts verbatim (model_configurator.py:26-44). + t["dropout"] = 0.0; + t["attention_bias"] = true; + t["num_vector_embeds"] = nullptr; + t["activation_fn"] = "gelu-approximate"; + t["num_embeds_ada_norm"] = 1000; + t["use_linear_projection"] = false; + t["only_cross_attention"] = false; + t["cross_attention_norm"] = true; + t["double_self_attention"] = false; + t["upcast_attention"] = false; + t["standardization_norm"] = "rms_norm"; + t["norm_elementwise_affine"] = false; + t["qk_norm"] = "rms_norm"; + t["positional_embedding_type"] = "rope"; + t["use_audio_video_cross_attention"] = true; + t["share_ff"] = false; + t["av_cross_ada_norm"] = true; + t["use_middle_indices_grid"] = true; + t["caption_proj_before_connector"] = true; + + nlohmann::json agreeing; + agreeing["transformer"] = t; + agreeing["transformer"]["use_prompt_adaln_single"] = true; + // Agreement is adopted, and carries the flag through. + const Ltx2DitParams adopted = + vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes, false, "the test config"); + CHECK(adopted.use_prompt_adaln_single); + + nlohmann::json disagreeing; + disagreeing["transformer"] = t; + disagreeing["transformer"]["use_prompt_adaln_single"] = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, false, "the test config")); + // The opt-in must not rescue it: clearing a PORTED flag in the config copy is + // exactly what made this comparison blind before. + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(disagreeing, shapes, true, "the test config")); + + // And the other direction: shapes WITHOUT the module against a config that + // declares it. + Ltx2DitParams shapes_off = shapes; + shapes_off.use_prompt_adaln_single = false; + CHECK_THROWS(vllm::Ltx2AdoptDeclaredDitParams(agreeing, shapes_off, false, "the test config")); +} + TEST_CASE("ltx2 loader: the f32 widening is OPT-IN and bit-exact over bf16") { const Ltx2DitParams p = TinyParams(); const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index e2191ed9b..ec3fca7af 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -248,7 +248,14 @@ inline vllm::Ltx2DitParams ReducedDitParams() { p.use_middle_indices_grid = true; p.apply_gated_attention = true; // every LTX-2.5 attention is gated p.cross_attention_adaln = true; // the shipped config sets it - p.use_prompt_adaln_single = false; + // TRUE, exactly as both shipped DiTs resolve it: the FP8 file carries no + // config at all and the NVFP4 file's config OMITS the key, so upstream's + // default (model.py:77, model_configurator.py:76) decides — and both files + // carry `prompt_adaln_single`'s tensors. `ReducedDitTransformerConfig` below + // likewise omits the key, so this fixture reproduces the shipped shape rather + // than a configuration nothing ships, and the whole video engine renders + // through the prompt-side AdaLN path (.agents/specs/ltx25-prompt-adaln.md). + p.use_prompt_adaln_single = true; p.ff_bias = false; // LTX-2.5 (gemma4) sets ff_bias=false p.audio_ff_bias = true; return p; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index e8eaf1220..85b47025a 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -927,10 +927,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const vllm::SafetensorsFile file = vllm::SafetensorsFile::Open(path); vllm::Ltx2DitQuant quant = vllm::Ltx2DitQuant::kFp8; vllm::Ltx2DitParams from_shapes = vllm::Ltx2ParseDitParamsFromCheckpoint(file, &quant); - // The manifest parser leaves `use_prompt_adaln_single` at its default; the - // LOADER clears it for the contract (ltx2_loader.cpp), and so does the - // engine. Mirror that here so the two contracts are compared like for like. - from_shapes.use_prompt_adaln_single = false; + // MEASURED from this file's own header: it carries `prompt_adaln_single`, so + // the manifest parser resolves `use_prompt_adaln_single = TRUE` — and nothing + // clears it any more (.agents/specs/ltx25-prompt-adaln.md, issue #644). This + // line used to force it false on BOTH sides of the comparison below, which is + // what made a config/shape disagreement about it unobservable. + CHECK(from_shapes.use_prompt_adaln_single); CHECK(quant == vllm::Ltx2DitQuant::kNvfp4); CHECK(from_shapes.num_layers == 48); CHECK(from_shapes.inner_dim() == 4096); @@ -950,8 +952,12 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { config["transformer"]["use_keyframes_abs_pos_embedding"] = false; nlohmann::json wrapper; wrapper["config"] = config; - vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); - declared.use_prompt_adaln_single = false; + const vllm::Ltx2DitParams declared = vllm::ParseLtx2DitParams(wrapper); + // The shipped config OMITS `use_prompt_adaln_single`, so it resolves to + // upstream's TRUE default (model_configurator.py:76) — which is what the + // file's own tensors say. Asserted rather than forced: the two sides of the + // contract comparison below must AGREE about it, not be made to. + CHECK(declared.use_prompt_adaln_single); CHECK(declared.double_precision_rope); // frequencies_precision float64 CHECK(declared.av_ca_timestep_scale_multiplier == 1000); CHECK(declared.apply_gated_attention); From 2990ed81b7b9075c5bace4ef0b93210f554c1b43 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 20:00:50 +0000 Subject: [PATCH 3/4] record(LTX25-PROMPT-ADALN): the prompt-AdaLN defect was understated ~138x, and the keyframes claim next door was false (#644) The port landed and reviewed PASS. What we WROTE about it did not. ## The headline numbers were fixture artifacts The spec Outcome, the PR body and test_ltx2.cpp all quoted 51.7% ("term vs table") and 5.82% (block-0 K/V) as the answer to "does this matter". Both come from the generator's synthetic init: `prompt_scale_shift_table` AND every prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at scale=0.05 (gen-ltx2-goldens.py:100-106), so the ratio between them is a property of the FIXTURE. It moves with the init scale: 0.005 gives 4.1%, 0.2 gives 1450%. The Outcome disclaimed only the two OUTPUT rows as synthetic-bounded and then called the other two the answer, though all four have identical provenance. Measured on the SHIPPED weights instead, through upstream's own AdaLayerNormSingle(dim, embedding_coefficient=2) evaluated on sigma*1000 exactly as transformer_args.py:274-278/:177 do, against all 48 prompt_scale_shift_table tensors of `ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`: video audio rms|table| (48 blocks) 0.017553 0.021925 rms|term| (uniform s) 0.236446 0.347171 term/table RMS 1347% 1583% max|term|/max|table| 7119% 2817% Like for like -- 51.7% is a max-vs-max ratio, and the same ratio on the real weights is 7119% / 2817% -- the fixture UNDERSTATED the defect by 138x (video) and 54x (audio). On the shipped model the timestep term DOMINATES the static table; the table is the perturbation. Pre-row renders applied `context*(1 + ~0.018 rms) + ~0.018` where upstream applies `context*(1 + ~0.32 rms) + ~0.10`; on a unit-rms context the modulated context moves +8.8% (video) / +31.3% (audio). Both dispositions the finding offered are taken: the four fixture rows are now labelled GATE-FLOOR numbers from synthetic weights in the spec, the goldens' comment block and test_ltx2.cpp, AND the real-checkpoint measurement is stated as the answer. `scripts/measure-ltx2-prompt-adaln.py` is committed so the number is re-runnable rather than transcribed; it asserts `ltx_core.__file__` under the named checkout before reading anything. ## A false claim about a checkpoint, the same class this row just fixed ltx2.h:47-49 -- a line surviving inside the paragraph this row rewrote -- said "LTX-2.5's checkpoint does not carry the parameter" about keyframes_abs_pos_embedding. Both shipped DiTs contradict it, each on a different half, read off their headers and run through upstream's own configurator: FP8 (vonkaiser): CARRIES keyframes_abs_pos_embedding F8_E4M3 [1,4096] + scale, and declares NO __metadata__ at all -- so LTXModelConfigurator.from_metadata cannot configure it: KeyError: 'caption_channels', raised by _build_caption_projections on the empty dict, BEFORE the flag at model_configurator.py:82 is ever read. NVFP4 (1st-party): DECLARES the flag true and does NOT carry the tensor, so the parameter stays torch.zeros(1, inner_dim) (model.py:217-219) through load_state_dict(..., strict=False) (single_gpu_model_builder.py:98) -- a genuine no-op there. It is also not keyframe-only: transformer_args.py:269 applies it on every prepare whose keyframes_mask is set, and tools.py:186-196 sets that mask unconditionally. REFUSAL KEYING: unchanged, deliberately. It fires on the TENSORS the file carries. Keying it on the resolved flag would, on the FP8 DiT, read a DEFAULT rather than the file -- that file declares nothing -- and load it while silently discarding a trained [1,4096] parameter. No behaviour changed, so no new gate is owed; three refusal MESSAGES changed, because they asserted an implication that is false in both directions. ## Records the lifecycle change did not move model-matrix.md's inventory row still listed "timestep-INDEPENDENT prompt K/V (cacheable once per request)" -- precisely what this row disproved. docs/STATUS.md's LTX-2.5 row did not mention the prompt-side AdaLN at all; FEATURES.md and USAGE.md described the keyframes opt-in as if only one file were involved. ## Two things recorded honestly rather than fixed - test_ltx2_video's shipped-checkpoint case is env-gated on LTX2_CHECKPOINT_ROOT and SKIPS by default, so the row's "unchanged 30/502" meant the whole real-header case did not run. With the variable set the same binary measures 30/8734 -- verified here, exit 0 both ways. Noted in the spec and at the skip site. - Upstream loads with strict=False, so a config declaring the prompt-AdaLN flag false over a file that carries the module would silently run flag-OFF upstream where we now refuse. Ours is better; it is a DOCUMENTED DIVERGENCE, not a mirror, and it is named as one. ## Gate BUILD_EXIT=0; build logs grepped for `No space left|BFD assertion` (0 hits); df -h / 92% used, 37G free. ctest -N = 423; full ctest -j8 422/423 with test_serve_low_tools starved under -j (known parallel flake), 1/1 PASS serially, exit 0; 2 skipped as on the baseline. Suite counts unchanged from the row: test_ltx2 35/2435, test_ltx2_loader 26/4826, test_ltx2_device 15/523, test_ltx2_video 30/502 (skipped default) and 30/8734 with the checkpoint root. ltx2_goldens.inc REGENERATED from gen-ltx2-goldens.py against ltx_core fd4ded7f: every golden VALUE byte-identical, the diff is the comment block alone. An earlier full run was VOIDED rather than reported: another session's disk-pressure cleanup deleted build/ while ctest was at 421/423 and the last two tests recorded "Not Run -- Failed to change working directory". A run whose tree vanished under it is not a result; it was rebuilt and re-run from scratch. One finding detail did not hold on inspection: it cited `.agents/model-matrix.md:98` as already saying the two DiTs "differ in a TRAINED keyframes_abs_pos_embedding". That file contains no occurrence of "keyframes" at all on this branch; the corroborating anchors are `.agents/specs/ltx-2-5.md` section 3.1 and `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/model-matrix.md | 2 +- .agents/specs/ltx25-prompt-adaln.md | 187 ++++++++++++++++-- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 17 +- include/vllm/model_executor/models/ltx2.h | 27 ++- .../vllm/model_executor/models/ltx2_loader.h | 6 +- scripts/gen-ltx2-goldens.py | 25 ++- scripts/measure-ltx2-prompt-adaln.py | 164 +++++++++++++++ src/vllm/model_executor/models/ltx2.cpp | 4 +- .../model_executor/models/ltx2_loader.cpp | 7 +- tests/vllm/models/ltx2_goldens.inc | 15 +- tests/vllm/models/test_ltx2.cpp | 21 +- tests/vllm/multimodal/test_ltx2_video.cpp | 5 + 14 files changed, 435 insertions(+), 49 deletions(-) create mode 100755 scripts/measure-ltx2-prompt-adaln.py diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index e63964f10..ec14bf4bd 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -429,7 +429,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | +| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-CONDITIONED prompt K/V via `prompt_adaln_single` (model.py:223-227, transformer.py:441-446), so the K/V cache is inapplicable on the shipped DiT -- the row's first claim of cacheability was RETRACTED 2026-08-12 and the module PORTED 2026-08-13 (#644, [spec](specs/ltx25-prompt-adaln.md)); on the real weights the timestep term DOMINATES the static table, RMS ratio 1347% video / 1583% audio; split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index a24f127ce..724566583 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -228,10 +228,18 @@ byte cost. ## 6. Measured magnitude -Recorded in §Outcome: the relative change in the modulated prompt context and in -the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer -to "does this matter"; a number below round-off would mean the term is inert and -the whole row is decoration. +Recorded in §Outcome, on two fixtures that answer two different questions. + +The reduced-dimension generator gives the relative change in the modulated prompt +context and in the DiT's outputs, flag ON vs OFF. That is a GATE FLOOR: a number +below round-off would mean the term is inert and the mutation in §5.3 could not +bite. It is **not** the answer to "does this matter", because both the static +table and the prompt-AdaLN MLP are drawn from the same synthetic init scale, so +every ratio it produces is a property of the fixture. + +"Does this matter" is answered on the SHIPPED checkpoint's own weights, run +through upstream's `AdaLayerNormSingle`. That measurement is required before the +row's Outcome may state a magnitude. ## 7. Risks @@ -256,7 +264,7 @@ the whole row is decoration. ## Outcome -### What was measured +### What was measured — (a) the gate floor, on SYNTHETIC weights The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints them on stderr, from the SAME shared weight stream on both arms (keyed by @@ -265,21 +273,71 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | -| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| timestep term vs the static table it is added to | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | -**The answer to "does this matter" is the first two rows.** Roughly half the -magnitude of the prompt K/V modulation is the timestep-conditioned term, and -including it moves the modulated prompt context by ~6%. Every render before this -row discarded that. - -The two output rows are the GATE's floor, not a claim about the trained -checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and -by a 2-block stack rather than 48. They are reported because a mutation must be -shown to move something, and 73x/37x above round-off is what makes the mutation -below meaningful. +**ALL FOUR ROWS ARE GATE-FLOOR NUMBERS, AND NONE OF THEM ANSWERS "DOES THIS +MATTER".** Corrected 2026-08-13 (issue #644) — this section originally billed the +first two as the answer and disclaimed only the last two as synthetic-bounded. +They have identical provenance: `prompt_scale_shift_table` and every +prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at +`scale=0.05` (`scripts/gen-ltx2-goldens.py:100-106`), so the ratio between them +is a property of the FIXTURE, not of the conditioning. Vary only the MLP init and +it moves with it: 0.005 → 4.1% / 0.49%, 0.05 (committed) → 51.7% / 5.82%, +0.2 → 1450% / 142%. What these rows are FOR is the mutation below: 73x and 37x +above round-off is what makes a zeroed term detectable. + +### What was measured — (b) the SHIPPED checkpoint, which is the answer + +Measured 2026-08-13 by loading the real tensors into upstream's own +`AdaLayerNormSingle(inner_dim, embedding_coefficient=2)` (`adaln.py:19-45`, built +by `model.py:223-227` / `:253-257`) and evaluating it on `sigma * +timestep_scale_multiplier` — the file's own config gives 1000 — exactly as +`transformer_args.py:274-278` and `:177` do, then comparing against all 48 +`prompt_scale_shift_table` / `audio_prompt_scale_shift_table` tensors it is +summed with at `transformer.py:441-443`. + +- File: `/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` + (7876 tensors, 1,179,408-byte header; the prompt-AdaLN tensors are BF16 there, + so they are read directly with no dequantisation step of ours in the path). +- Upstream: `ltx_core` at `fd4ded7f`, imported BY PATH from + `/home/mudler/_git/LTX-2` with `ltx_core.__file__` asserted under that checkout. +- Sigmas: a uniform grid over the whole range, `linspace(0, 1, 101)`, and + separately the sampler the file's own scheduler config names — + `LinearQuadraticScheduler().execute(8)` (`components/schedulers.py:60-88`). + +| | video (dim 4096) | audio (dim 2048) | +|---|---|---| +| `rms\|table\|`, 48 blocks | 0.017553 | 0.021925 | +| `rms\|term\|`, uniform σ | 0.236446 | 0.347171 | +| **term/table, RMS** | **1347%** | **1583%** | +| `max\|term\|` / `max\|table\|` | **7119%** | **2817%** | +| term/table RMS, LinearQuadratic 8-step | 1275% | 1492% | + +Split by row, on the uniform grid: shift 605% (video) / 461% (audio), scale 1732% +/ 2044%. The scale row is where it concentrates, which is the row that multiplies. + +**On the shipped model the timestep term DOMINATES the static table; the table is +the perturbation.** What the pre-row renders applied was +`context * (1 + ~0.018 rms) + ~0.018` where upstream applies +`context * (1 + ~0.32 rms) + ~0.10`. Stated as the quantity actually consumed at +`transformer.py:446`, for a context of unit rms: the modulated context has rms +1.0035 static-only against 1.0915 upstream (video, **+8.8%**) and 1.0033 against +1.3170 (audio, **+31.3%**). + +**The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** +Comparing like with like — the fixture's 51.7% is a `max\|term\|` / `max\|table\|` +ratio, and that same ratio on the shipped weights is 7119% (video) and 2817% +(audio): **138x and 54x** larger. Recorded because the original Outcome quoted +"roughly half the magnitude" from the fixture as if it described the checkpoint. + +Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 +--checkpoint `, committed by this repair so the number is +re-runnable rather than transcribed. It asserts `ltx_core.__file__` under the +named checkout before it reads anything, and nothing of ours is in its numeric +path — the only vllm.cpp input is which tensors to read. ### The mutations @@ -316,7 +374,16 @@ re-checking its md5 (`03324d42…`, identical before and after): | `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | | `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | | `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | -| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — see below; 502 is the SKIPPED default | + +**What `30 / 502` does and does not say (corrected 2026-08-13).** The +shipped-checkpoint case is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints +`SKIPPED` and returns at `test_ltx2_video.cpp:917-921`, so `30 / 502` means the +whole real-header case DID NOT RUN — not "it ran and no assertion counted the +module". With the variable pointing at the Lightricks tree the same binary +measures **30 cases / 8734 assertions**, both before and after this repair +(re-measured on this branch, exit 0 in both configurations). Any future quote of +this suite's count owes the configuration alongside it. The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said @@ -356,6 +423,90 @@ off: the loader asserts the flag against the file instead of clearing it, and `Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing applies. +### The keyframes claim next door, corrected 2026-08-13 + +`ltx2.h` carried, in the same paragraph this row rewrote, *"LTX-2.5's checkpoint +does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — +the same class of claim as the `use_prompt_adaln_single=false` assertion this row +exists to remove — and the tree already contradicted it twice +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`). +Read straight off both files' headers, and run through upstream's own loader and +configurator: + +| | FP8 (`vonkaiser`) | NVFP4 (first-party) | +|---|---|---| +| carries `keyframes_abs_pos_embedding` | YES — `F8_E4M3 [1, 4096]` + F32 scale | NO | +| declares the flag in `__metadata__` | **no `__metadata__` AT ALL** | `true` | +| `LTXModelConfigurator.from_metadata` | **RAISES** `KeyError: 'caption_channels'` | builds it, `[1, 4096]` | + +So the two files each contradict one half of the retired claim, and neither +supports it. Two corrections to the reasoning that came with the finding, both +measured rather than read: + +- The FP8 file does not "resolve the flag `False` at `model_configurator.py:82`". + Upstream never reaches line 82 on it: `_build_caption_projections` indexes + `caption_channels` on the empty config first and raises. That file ships no + config, so what its flag resolves to is decided entirely out of band — and the + tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). +- On the NVFP4 file the flag IS on and the module IS built, but the tensor is + absent, so it keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` + (`loader/single_gpu_model_builder.py:98`) — a genuine no-op there. + +It is also not a keyframe-only feature: `transformer_args.py:269` applies it on +every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that +mask unconditionally on the target's first latent frame. (Diffusers' own pipeline +does not consume it — `.agents/specs/ltx-2-5.md` §3.1 records that — but `ltx_core` +is what this campaign ports, and `ltx_core` does.) + +**The refusal keying does NOT change, and that is the decision, not an omission.** +`ltx2_loader.cpp`'s `RefuseUnported` fires on the TENSORS the file carries. Keying +it on the resolved flag instead would, on the FP8 DiT, read a DEFAULT rather than +the file — because that file declares nothing — and would therefore load it +silently while discarding a trained `[1, 4096]` parameter. Tensor presence is the +only signal that file actually carries, and refusing loudly with an opt-in is +strictly safer than resolving quietly. No behaviour changed, so no new gate is +owed; the refusal MESSAGE changed, because it asserted the implication that is +false in both directions. + +### The claims repair's own gate (2026-08-13) + +Nothing executable changed except three refusal MESSAGES, so the numbers are +expected to be identical and the point is that they are: + +- `BUILD_EXIT=0`; build logs grepped for `No space left|BFD assertion` — 0 hits; + `df -h /` 92% used, 37G free at the end. +- `ctest -N` = **423**. Full `ctest -j8` = 422/423 with `test_serve_low_tools` + starved under `-j` (a known parallel flake); serially **1/1 PASS, exit 0**. + 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) as + on the baseline. +- Suite counts, unchanged from the row above: `test_ltx2` 35/2435, + `test_ltx2_loader` 26/4826, `test_ltx2_device` 15/523, `test_ltx2_video` + 30/502 skipped-default and **30/8734** with `LTX2_CHECKPOINT_ROOT` set. Exit 0 + on all five runs. +- `tests/vllm/models/ltx2_goldens.inc` REGENERATED from + `scripts/gen-ltx2-goldens.py` against `ltx_core` `fd4ded7f`: every golden VALUE + byte-identical, the diff is the comment block alone. That re-proves provenance + as well as the wording. + +An earlier full run was voided rather than reported: another session's +disk-pressure cleanup deleted `build/` while ctest was at 421/423, and the last +two tests recorded `Not Run — Failed to change working directory`. A run whose +tree vanished under it is not a result; it was rebuilt and re-run from scratch. + +### Two divergences from upstream, recorded rather than fixed + +- **We are stricter than upstream about a config that disagrees with its file.** + Upstream loads with `load_state_dict(..., strict=False)` + (`loader/single_gpu_model_builder.py:98`), so a config declaring + `use_prompt_adaln_single=false` over a file that carries the module would build + no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's + equality check refuses that. Ours is better; it is still a DIVERGENCE, not a + mirror, and it is named here so it is not later mistaken for ported behaviour. +- **We refuse `keyframes_abs_pos_embedding` by tensor presence** where upstream + would take an out-of-band config's word for it (above). Same shape of + divergence, same reason it stands. + ## Now `DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3c7ba3e68..cd6498160 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4, ~44 GB FP8, +~24 GB tower. FP8/torchao/NVFP4 load; `keyframes_abs_pos_embedding` needs `allow_unported` on BOTH DiTs. DiffVAE/LoRA/image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/STATUS.md b/docs/STATUS.md index 2acbe72f1..70b830b60 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -139,7 +139,7 @@ token-for-token correctness against the pinned oracle. | Laguna-S-2.1 MoE (`LagunaForCausalLM`, 118B/8B) | **BINDING 2026-08-04: was 87% of vLLM (37.55 vs 43.10, same-tool nsys)**; root cause was bf16 projections on UNIFIED/ATS host memory, and device-resident staging (byte-exact) gives 44.6, parity+ vs 43.1, default-ON | 48 layers (12 global + 36 SWA-512), 256 routed top-10 + 1 shared expert, per-head softplus attn out-gate, sigmoid `noaux_tc` router, dual per-layer RoPE, GQA 8 KV / 128 head-dim, 1M ctx. History: benchmark-record | | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | -| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, the embeddings connector, pipeline, NVFP4/FP8 arms, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | +| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435); prompt-side AdaLN #644.** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (f32 host, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..6cf9f4f0f 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2072,12 +2072,19 @@ Only the LTX-2.5 DiT is gated against an independent oracle here, so treat any other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. -Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **one** module family this port -does not carry (`keyframes_abs_pos_embedding`); pass -`Ltx2DitLoadOptions::allow_unported_modules` +Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES a +shipped DiT by default over **one** module family this port does not carry, +`keyframes_abs_pos_embedding`; pass `Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`Ltx2DitCheckpoint::unported`. Both shipped DiTs need the opt-in, for DIFFERENT +reasons, and the distinction matters if you are reading the refusal text +(corrected 2026-08-13): the **FP8** file CARRIES the tensor +(`F8_E4M3 [1, 4096]`) and declares no `__metadata__` at all, so the refusal is on +its TENSORS; the first-party **NVFP4** file does NOT carry the tensor but its +config DECLARES `use_keyframes_abs_pos_embedding: true`, so it is refused by the +FLAG, in `ParseLtx2DitParams`, and `allow_unported_modules` clears that flag in a +config copy. Neither file supports the retired claim that "LTX-2.5's checkpoint +does not carry the parameter". `prompt_adaln_single` and `audio_prompt_adaln_single` were on that list until 2026-08-13 and are now PORTED, so a checkpoint carrying them needs no opt-in on their account, and the opt-in no longer disables them. The two `*_embeddings_connector` towers are diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 99898406b..87f4ddebb 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -44,9 +44,30 @@ // attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation // configuration, whose masks are all-ones and whose flags are all false — // upstream's own `perturbations=None` path (model.py:509-511). -// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). LTX-2.5's -// checkpoint does not carry the parameter; the enumeration refuses a config -// that asks for it. +// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). The +// enumeration refuses a config that asks for it. +// +// CORRECTED 2026-08-13 (issue #644). This line read "LTX-2.5's checkpoint +// does not carry the parameter", which is FALSE, and it is the same class of +// claim the prompt-AdaLN repair below exists to remove. The two shipped DiTs +// disagree with each other and each contradicts one half of it, read straight +// off their safetensors headers: +// * FP8 (`vonkaiser`, ltx-2.5-22b-distilled-fp8): CARRIES +// `keyframes_abs_pos_embedding` `F8_E4M3 [1, 4096]` plus its F32 scale, +// and declares NO `__metadata__` at all — so it ships no config, and +// upstream's own `LTXModelConfigurator.from_metadata` cannot configure it +// (run 2026-08-13: `KeyError: 'caption_channels'`, raised by +// `_build_caption_projections` on the empty dict, BEFORE the flag at +// model_configurator.py:82 is ever read). What that file's flag resolves +// to therefore depends on a config it does not ship. +// * NVFP4 (first-party Lightricks): DECLARES the flag `true` and does NOT +// carry the tensor, so upstream's parameter stays its +// `torch.zeros(1, inner_dim)` initialiser (model.py:217-219) and +// `load_state_dict(..., strict=False)` +// (loader/single_gpu_model_builder.py:98) leaves it zero — a no-op. +// It is also NOT a keyframe-only feature: `transformer_args.py:269` applies it +// on EVERY prepare whenever `keyframes_mask` is non-None, and `tools.py:186-196` +// marks the target's first latent frame unconditionally. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 46aa57faa..8d472913d 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -112,7 +112,11 @@ // stays gateable — proceeds, still reporting every one of them in `unported`: // // keyframes_abs_pos_embedding [1, 4096] -// So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// Present in the FP8 DiT and ABSENT from the NVFP4 one, whose config +// nonetheless DECLARES the flag `true` — so tensors and flag do not imply +// each other in either direction, and the refusal is keyed on the tensors. +// Corrected 2026-08-13 (#644); the measured behaviour of BOTH files, and +// why the keying stays on tensors, is in ltx2.h. // This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its // config copy, and it must stay that way: a flag cleared there is invisible // to the contract-equality check, so clearing a PORTED one silently drops diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 4c5d3e54b..119374ae9 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -752,15 +752,30 @@ def rel(a, b): f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( - "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" ) lines.append( - "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + "// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every" ) lines.append( - "// needs, not a claim about the trained checkpoint. The K/V row is where the" + "// ratio here is a property of THIS FIXTURE and moves with that scale; the output" ) - lines.append("// term actually enters and is the number that answers 'does this matter'.") + lines.append( + "// rows are bounded by it AND by a 2-block stack. They are reported because a" + ) + lines.append( + "// mutation has to be shown to move something -- NOT as a claim about the trained" + ) + lines.append( + "// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to:" + ) + lines.append( + "// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's" + ) + lines.append( + "// own AdaLayerNormSingle on the real weights. See" + ) + lines.append("// .agents/specs/ltx25-prompt-adaln.md section Outcome.") text = "\n".join(lines) print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) return text @@ -818,7 +833,7 @@ def main() -> int: # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. emit_prompt_adaln(out) out.write( - "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE ---\n" "// Same shared weights, same inputs, flag ON vs OFF:\n" + measure_prompt_adaln_magnitude() + "\n" diff --git a/scripts/measure-ltx2-prompt-adaln.py b/scripts/measure-ltx2-prompt-adaln.py new file mode 100755 index 000000000..8961f845c --- /dev/null +++ b/scripts/measure-ltx2-prompt-adaln.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Measure the LTX-2.5 prompt-AdaLN timestep term against the static per-block +table, ON THE SHIPPED CHECKPOINT'S OWN WEIGHTS. + +Why this exists. `scripts/gen-ltx2-goldens.py` reports the same ratio at reduced +dimensions, but there the table AND the prompt-AdaLN MLP are both drawn from +`param_spec`'s `scale=0.05`, so the ratio it prints is a property of that fixture +and moves with the init scale. It is a gate FLOOR, not a statement about the +trained model. This script answers the other question — how much of the prompt +K/V modulation is timestep-conditioned on the model people actually run — and its +output is what `.agents/specs/ltx25-prompt-adaln.md` §Outcome quotes (issue #644). + +Nothing of ours is in the numeric path. It reads the safetensors header, pulls +six tensors per stream plus the 96 per-block tables, and runs UPSTREAM's own +module: + + ltx_core/model/transformer/adaln.py:19-45 AdaLayerNormSingle + ltx_core/model/transformer/model.py:223-227,253-257 built at embedding_coefficient=2 + ltx_core/model/transformer/transformer_args.py:274-278 driven by SIGMA, not timesteps + ltx_core/model/transformer/transformer_args.py:177 scaled by timestep_scale_multiplier + ltx_core/model/transformer/transformer.py:441-446 table + term, then context*(1+scale)+shift + ltx_core/components/schedulers.py:60-88 the sampler the file's config names + +Usage: + python3 scripts/measure-ltx2-prompt-adaln.py \ + --ltx2 ~/_git/LTX-2 \ + --checkpoint $CHECKPOINT_ROOT/ltx-2.5/lightricks-ltx-2.5/diffusion_models/\ +ltx-2.5-22b-distilled-transformer-nvfp4.safetensors +""" + +from __future__ import annotations + +import argparse +import json +import struct +import sys +from pathlib import Path + +PREFIX = "model.diffusion_model." +ADALN_KEYS = ( + "emb.timestep_embedder.linear_1.weight", + "emb.timestep_embedder.linear_1.bias", + "emb.timestep_embedder.linear_2.weight", + "emb.timestep_embedder.linear_2.bias", + "linear.weight", + "linear.bias", +) + + +def load_upstream(root: Path): + """Import `ltx_core` BY PATH from `root`, and PROVE that is what resolved. + + A pip-installed or otherwise-shadowing `ltx_core` would import silently and + every number below would describe an upstream nobody pinned — the failure + mode `.agents/specs/ltx-2-5.md` records at its "(b) byte-identical goldens" + entry. Identity is asserted, not assumed. + """ + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core").is_dir(): + raise SystemExit(f"no ltx_core under {src}; point --ltx2 at a Lightricks/LTX-2 checkout") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + resolved = Path(ltx_core.__file__).resolve() + if resolved.parent != (src / "ltx_core").resolve(): + raise SystemExit(f"ltx_core resolved to {resolved}, NOT to the checkout at {src}") + return resolved + + +def read_header(path: Path) -> dict: + """The safetensors header alone: an 8-byte length prefix and its JSON. No payload.""" + with path.open("rb") as fh: + length = struct.unpack(" float: + return float(t.pow(2).mean().sqrt()) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--ltx2", required=True, type=Path, help="path to a Lightricks/LTX-2 checkout") + ap.add_argument("--checkpoint", required=True, type=Path, help="a shipped LTX-2.5 DiT .safetensors") + ap.add_argument("--steps", type=int, default=8, help="step count for the shipped sampler arm") + args = ap.parse_args() + + resolved = load_upstream(args.ltx2.expanduser()) + import torch # noqa: PLC0415 + from safetensors import safe_open # noqa: PLC0415 + + from ltx_core.components.schedulers import LinearQuadraticScheduler # noqa: PLC0415 + from ltx_core.model.transformer.adaln import AdaLayerNormSingle # noqa: PLC0415 + + ckpt = args.checkpoint.expanduser() + print(f"ltx_core: {resolved}") + print(f"checkpoint: {ckpt}") + + header = read_header(ckpt) + meta = header.get("__metadata__") + if not meta or "config" not in meta: + raise SystemExit(f"{ckpt} declares no __metadata__ config; this script needs the file's own geometry") + config = json.loads(meta["config"]) + tcfg = config["transformer"] + mult = tcfg.get("timestep_scale_multiplier", 1000) + blocks = tcfg["num_layers"] + dims = { + "video": (tcfg["num_attention_heads"] * tcfg["attention_head_dim"], + "prompt_adaln_single.", "prompt_scale_shift_table"), + "audio": (tcfg["audio_num_attention_heads"] * tcfg["audio_attention_head_dim"], + "audio_prompt_adaln_single.", "audio_prompt_scale_shift_table"), + } + print(f"config: num_layers={blocks} timestep_scale_multiplier={mult} " + f"cross_attention_adaln={tcfg.get('cross_attention_adaln')} " + f"use_prompt_adaln_single={tcfg.get('use_prompt_adaln_single', ' upstream default True>')} " + f"sampler={config.get('scheduler', {}).get('sampler')}") + + arms = { + "uniform sigma [0,1] x101": torch.linspace(0.0, 1.0, 101, dtype=torch.float32), + f"shipped LinearQuadratic, {args.steps} steps": LinearQuadraticScheduler().execute(args.steps), + } + + torch.set_grad_enabled(False) + with safe_open(str(ckpt), framework="pt") as fh: + for stream, (dim, sub, table_name) in dims.items(): + module = AdaLayerNormSingle(dim, embedding_coefficient=2) + module.load_state_dict( + {k: fh.get_tensor(PREFIX + sub + k).float() for k in ADALN_KEYS}, strict=True + ) + module.eval() + tables = torch.stack([ + fh.get_tensor(f"{PREFIX}transformer_blocks.{i}.{table_name}").float() + for i in range(blocks) + ]) # [blocks, 2, dim]; row 0 shift, row 1 scale (transformer.py:444) + + t_rms, t_max = rms(tables), float(tables.abs().max()) + print(f"\n=== {stream} dim={dim} blocks={blocks}") + print(f" rms|table|={t_rms:.6f} max|table|={t_max:.6f}") + for arm, sigmas in arms.items(): + term = module(sigmas * mult, hidden_dtype=torch.float32)[0] + term = term.reshape(sigmas.numel(), 2, dim) # transformer.py:443 + e_rms, e_max = rms(term), float(term.abs().max()) + print(f" [{arm}] rms|term|={e_rms:.6f} -> {100.0 * e_rms / t_rms:.0f}% of rms|table|; " + f"max|term|={e_max:.6f} -> {100.0 * e_max / t_max:.0f}% of max|table|") + for row, name in ((0, "shift"), (1, "scale")): + print(f" {name}: rms|table|={rms(tables[:, row]):.6f} " + f"rms|term|={rms(term[:, row]):.6f} " + f"-> {100.0 * rms(term[:, row]) / rms(tables[:, row]):.0f}%") + # What is actually consumed (transformer.py:446): context*(1+scale)+shift. + # For a context of unit rms with independent entries the modulated + # context has rms sqrt(mean((1+scale)^2) + mean(shift^2)). + def modulated(scale, shift) -> float: + return float(((1.0 + scale).pow(2).mean() + shift.pow(2).mean()).sqrt()) + + static = modulated(tables[:, 1], tables[:, 0]) + full = modulated(tables[:, 1].unsqueeze(0) + term[:, 1].unsqueeze(1), + tables[:, 0].unsqueeze(0) + term[:, 0].unsqueeze(1)) + print(f" modulated context on a UNIT-rms context: static-only={static:.4f} " + f"upstream={full:.4f} ({100.0 * (full - static) / static:+.1f}%)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 3d9af69ce..344155af1 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -193,7 +193,9 @@ Ltx2DitParams ParseLtx2DitParams(const nlohmann::json& metadata) { !cfg.at("use_keyframes_abs_pos_embedding").is_boolean() || !cfg.at("use_keyframes_abs_pos_embedding").get(), "ltx2: use_keyframes_abs_pos_embedding is not ported (transformer_args.py:23-43); " - "the LTX-2.5 checkpoint does not carry keyframes_abs_pos_embedding"); + "upstream applies it on every prepare whose keyframes_mask is set, which " + "tools.py:186-196 populates unconditionally, so this is refused rather than " + "ignored (see ltx2.h for what each shipped DiT actually declares and carries)"); VT_CHECK(p.num_attention_heads > 0 && p.attention_head_dim > 0 && p.audio_num_attention_heads > 0 && p.audio_attention_head_dim > 0 && diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index bfc85075c..398d578a9 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -541,8 +541,11 @@ std::vector ContractOf(const Ltx2DitParams& params) { } Fail( "the checkpoint carries modules this port does NOT carry: " + list + - ". They are not dropped silently: keyframes_abs_pos_embedding means " - "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + ". They are not dropped silently. This refusal is keyed on the TENSORS the " + "file carries, NOT on a declared flag, and that is deliberate: the FP8 DiT " + "carries a trained keyframes_abs_pos_embedding while declaring no " + "__metadata__ at all, so a flag-keyed refusal would read a DEFAULT rather " + "than the file and would silently discard it (see ltx2.h). " "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " "— they were ported by row LTX25-PROMPT-ADALN " "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index 964047788..d7f73a4ce 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3227,14 +3227,19 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, }; -// --- the MEASURED magnitude of the prompt-AdaLN term --- +// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE --- // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) // block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) // timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) -// The two output rows are bounded by this generator's SYNTHETIC weight scale -// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate -// needs, not a claim about the trained checkpoint. The K/V row is where the -// term actually enters and is the number that answers 'does this matter'. +// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and +// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every +// ratio here is a property of THIS FIXTURE and moves with that scale; the output +// rows are bounded by it AND by a 2-block stack. They are reported because a +// mutation has to be shown to move something -- NOT as a claim about the trained +// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to: +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's +// own AdaLayerNormSingle on the real weights. See +// .agents/specs/ltx25-prompt-adaln.md section Outcome. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index f00771f38..b4896462f 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -951,12 +951,21 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // term. A port that accepted the flag, bound the 12 tensors and then never added // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // -// Measured on this fixture (generator stderr, and the comment block at the end of -// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block -// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and -// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound -// below is set at 20x kRoundOff so it is comfortably inside the measured signal -// and comfortably outside f32 noise. +// WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the +// comment at the end of ltx2_goldens.inc report the term at 51.7% of the static +// per-block table, the block-0 prompt K/V moving 5.82%, and the DiT output moving +// 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, +// not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are +// both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every +// ratio is a property of THIS FIXTURE and moves with the init scale. +// +// On the SHIPPED DiT the term DOMINATES the table rather than halving it — +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's own +// AdaLayerNormSingle on the real weights (.agents/specs/ltx25-prompt-adaln.md +// §Outcome). So this fixture UNDERSTATES the defect; it does not bound it. +// +// The bound below is set at 20x kRoundOff: comfortably inside the signal this +// fixture does produce, and comfortably outside f32 noise. TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 85b47025a..ad1e39e6d 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -913,6 +913,11 @@ TEST_CASE("ltx2 video: an ABI client loads, detects and generates through vllm.h // It is NOT interchangeable with the `vonkaiser` FP8 copy — they differ in a // TRAINED `keyframes_abs_pos_embedding` (spec section 3.1) — so the file this // case reads is named here and in every report of its result. +// WHEN QUOTING THIS SUITE'S ASSERTION COUNT, QUOTE THE CONFIGURATION WITH IT. +// This case skips by default, and it is most of the suite: unset, the binary +// measures 30 cases / 502 assertions; with LTX2_CHECKPOINT_ROOT set it measures +// 30 / 8734. An "unchanged 30 / 502" across a change therefore says the real +// headers were never read, not that they were read and nothing counted them. TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const char* root_env = std::getenv("LTX2_CHECKPOINT_ROOT"); if (root_env == nullptr) { From 8531fd0eb2610cea4eec00b4ce84ed8535219c5e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:28:03 +0000 Subject: [PATCH 4/4] record(LTX25-ADALN-CLAIMS): the keyframes parameter is left on META, not zeroed, and the audio factor had the wrong denominator (#644, #673) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings F-1 and F-2 on PR #663. The reviewer re-ran every measurement in that PR and every published digit reproduced; this repairs the two claims that did not. ## F-1 (blocking) — a claim about upstream that upstream's own code contradicts Three places said that on the NVFP4 DiT the absent `keyframes_abs_pos_embedding` means the parameter "keeps `torch.zeros(1, inner_dim)` (model.py:217-219) through `load_state_dict(..., strict=False)` (single_gpu_model_builder.py:98) — a genuine no-op there": `include/vllm/model_executor/models/ltx2.h` (THE PRODUCT TREE), the spec, and the PR body. IT IS NOT A NO-OP. The claim quoted `single_gpu_model_builder.py:98` while dropping the `assign=True` that is on that same line, and upstream builds on the META DEVICE -- `loader/helpers.py:84-95`, `create_meta_model`: `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key absent from the state dict is therefore never materialised at all. RUN HERE, not transcribed, through upstream's own `create_meta_model` on the shipped file's real `__metadata__` (read with upstream's own `read_model_metadata` / `SafetensorsModelStateDictLoader`, sft_loader.py:58-74, which JSON-decodes each value -- the flag lives at `config.transformer`, not at the top level, so a raw `__metadata__` read returns None for it): keys matching 'keyframes_abs_pos' in the file : [] (0 of 7876 entries) config.transformer['use_keyframes_abs_pos_embedding'] = True keyframes_abs_pos_embedding: shape=(1, 4096) f32 device=meta is_meta=True supports_keyframes_abs_pos_embedding (BEFORE load) : False after load_state_dict(sd, strict=False, assign=True): neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised keyframes_abs_pos_embedding : device=meta is_meta=True in missing_keys : True reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors supports_keyframes_abs_pos_embedding (AFTER load) : False Upstream states this itself and then never asks. `supports_keyframes_abs_pos_embedding` (model.py:166-173) returns False for "a model whose config set the flag but whose checkpoint carried no weight for it (the parameter would still be on ``meta``)" (:170), and `enable_keyframes_abs_pos_embedding` (model.py:175-200) exists because such a parameter "would fail at the first forward" (:182). BOTH ARE DEFINED AND NEVER CALLED: `grep -rn` over the whole Lightricks/LTX-2 checkout at fd4ded7f returns exactly 1 hit each, the definition, re-confirmed here against a positive control rather than asserted from a null grep. Polarity is not what fails: `apply_keyframes_absolute_embedding` is `hidden_states + mask * embedding` (transformer_args.py:23-43, the sum at :43), so REAL zeros would be inert. The mechanism claim fails on `meta`, not on additivity. This STRENGTHENS the row's conclusion and changes nothing downstream: the refusal stays keyed on TENSOR PRESENCE for the FP8 file, and the NVFP4 file is refused by FLAG in `ParseLtx2DitParams` (ltx2.cpp:192-198). NOTHING RUNTIME CHANGED IN THIS COMMIT. `ltx2.h` and `test_ltx2.cpp` move comment text only; the one script edit (`gen-ltx2-goldens.py`) alters only the COMMENT the generator emits; `ltx2_goldens.inc` moves two comment lines with every value byte-identical. No `src/` file is touched at all. Suite counts are therefore expected to be identical, and the point of the gate below is that they are. The same anchor is now quoted in full at the OTHER place it appears (the documented-divergence bullet), because dropping the `assign=True` is precisely what made the claim above wrong. ## F-2 — the audio factor's denominator The Outcome read "138x (video) and 54x (audio)" as if both were per-stream. The video figure is right and like-for-like. The audio one divided the shipped AUDIO ratio by the VIDEO fixture denominator, because `gen-ltx2-goldens.py` emits exactly ONE fixture ratio and it is the video stream's. 54x is literally true about that single published figure and errs CONSERVATIVE. Recomputed by IMPORTING the generator, so the weight stream is bit-identical and no weight is re-drawn: video: max|term| = 0.0252012 max|table| = 0.0487142 -> 51.7327% (7119/51.73 = 137.6x) audio: max|term| = 0.0201763 max|table| = 0.0496868 -> 40.6068% (2817/40.61 = 69.4x) FIXED AT THE SOURCE, not only in prose. The generator emitted BOTH video-only rows unlabelled -- "timestep term vs static table" and "block 0 modulated prompt K/V" -- which is exactly how a video denominator came to be applied to an audio numerator. `gen-ltx2-goldens.py` now names the stream on both, and `ltx2_goldens.inc` was REGENERATED against ltx_core fd4ded7f to carry them (REGEN_EXIT=0). The regeneration diff is those two comment lines and NOTHING else: every golden VALUE byte-identical, and the "Regenerate with:" header unchanged because the committed command was re-run verbatim -- so this re-proves provenance as well as the labels. The Outcome now carries a per-stream table with each row against its own denominator, and `test_ltx2.cpp` and the spec's gate-floor table say which stream they mean. No fifth golden was added: the audio figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS ARE GATE-FLOOR NUMBERS" still holds and the three files that label those four are otherwise untouched. ## Recorded, not fixed — issue #673 `LTX2_CHECKPOINT_ROOT` is set by NO workflow: `grep -rn CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in `tests/` and `.agents/` (positive control run in the same command). CI therefore executes 502 of 8734 assertions -- 5.7% -- of `test_ltx2_video`, at an UNCHANGED case count of 30, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. This row's checkpoint-derived evidence is manual and host-local. Filed as visible debt and linked in the roadmap issue table and the spec; wiring checkpoints into CI is explicitly not in this row. ## No test asserts this text, and none was invented Searched in the subject's own vocabulary: the corrected claim lives only in a `ltx2.h` comment, the spec and the PR body. The runtime refusal MESSAGES (`ltx2.cpp:195-198`, `ltx2_loader.cpp` `RefuseUnported`) never carried the "zeros"/"no-op" implication, and no checker reads the header comment. Writing a test here would be writing one that cannot fail, so this change adds none. The tensor-keyed refusal it defends is already pinned by a mutation the reviewer ran: making `LoadedElsewhere` swallow the tensor takes `test_ltx2_loader` to 25/26 cases, 4817/4818 assertions, exit 1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-prompt-adaln.md | 125 ++++++++++++++++++++-- docs/USAGE.md | 19 +++- include/vllm/model_executor/models/ltx2.h | 40 ++++++- scripts/gen-ltx2-goldens.py | 11 +- tests/vllm/models/ltx2_goldens.inc | 4 +- tests/vllm/models/test_ltx2.cpp | 7 +- 7 files changed, 183 insertions(+), 24 deletions(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 29ab578da..f0fd89fa6 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -151,6 +151,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#673](https://github.com/mudler/vllm.cpp/issues/673) | `ROAD-V1-LTX25` | #644's checkpoint-derived evidence is MANUAL and host-local: `LTX2_CHECKPOINT_ROOT` is set by no workflow (`grep -rn CHECKPOINT_ROOT .github/` exits 1, positive control in `tests/`), so CI runs **502 of 8734 assertions — 5.7%** of `test_ltx2_video` at an unchanged case count of 30, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Recorded as visible debt; the skip itself is already announced in four places and that disposition stands | bug | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index 724566583..4ffcf7335 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -273,8 +273,8 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| timestep term vs the static table it is added to | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | -| block-0 modulated prompt K/V | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | +| timestep term vs the static table it is added to (**VIDEO stream**) | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V (**VIDEO stream**) | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | @@ -328,10 +328,53 @@ the perturbation.** What the pre-row renders applied was 1.3170 (audio, **+31.3%**). **The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** -Comparing like with like — the fixture's 51.7% is a `max\|term\|` / `max\|table\|` -ratio, and that same ratio on the shipped weights is 7119% (video) and 2817% -(audio): **138x and 54x** larger. Recorded because the original Outcome quoted -"roughly half the magnitude" from the fixture as if it described the checkpoint. +Comparing like with like — the fixture ratio is `max\|term\|` / `max\|table\|`, +and so is the shipped-weights ratio — **per stream, each against its own fixture +denominator**: + +| | fixture `max\|term\|`/`max\|table\|` | shipped | understatement | +|---|---|---|---| +| video (`prompt_adaln_single` vs `prompt_scale_shift_table`) | 51.7% | 7119% | **138x** | +| audio (`audio_prompt_adaln_single` vs `audio_prompt_scale_shift_table`) | 40.6% | 2817% | **69x** | + +**CORRECTED 2026-08-13 (issue #644): this read "138x and 54x", which is not a +per-stream pair.** The generator emits exactly ONE fixture ratio and it is the +VIDEO stream's (`gen-ltx2-goldens.py`, `measure_prompt_adaln_magnitude`), so 54x +divided the shipped AUDIO number by the VIDEO denominator. It is literally true +about the single published figure and it errs CONSERVATIVE — the real audio +understatement is larger. The audio stream's own fixture ratio, recomputed with +the identical statistic on the identical fixture (the generator is imported, so +the weight stream is bit-identical), is **40.6%** — `max|term|` 0.0201763 vs +`max|table|` 0.0496868 — giving **2817 / 40.6 = 69x**. The video row is +unchanged and was always like-for-like: `max|term|` 0.0252012 vs `max|table|` +0.0487142 = 51.7327%, and 7119 / 51.73 = 137.6. + +Reproduce the audio denominator by importing the generator and asking it the same +question the video row is built from (`measure_prompt_adaln_magnitude`), so no +weight is re-drawn: + +```python +gen.load_upstream(LTX2); torch.set_grad_enabled(False) +on = gen.build_model("split", False, prompt_adaln=True) +video, audio = gen.build_modalities(False) +scale = float(gen.ARCH["timestep_scale_multiplier"]) +amod, _ = on.audio_prompt_adaln_single((audio.sigma * scale).flatten(), hidden_dtype=torch.float32) +amod.abs().max() / on.transformer_blocks[0].audio_prompt_scale_shift_table.abs().max() +``` + +**Fixed at the source, not only in prose.** The generator emitted both video-only +rows unlabelled, which is how a video denominator came to be applied to an audio +numerator; `gen-ltx2-goldens.py` now names the stream on both, and +`ltx2_goldens.inc` was REGENERATED against `ltx_core` `fd4ded7f` to carry them. +The regeneration diff is those two comment lines and nothing else — every golden +VALUE byte-identical, and the `Regenerate with:` header unchanged because the +committed command was re-run verbatim, so this re-proves provenance as well as +the labels. `test_ltx2.cpp` says the same. No fifth golden was added: the audio +figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS" +still holds. + +Recorded because the original Outcome quoted "roughly half the magnitude" from +the fixture as if it described the checkpoint. Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 --checkpoint `, committed by this repair so the number is @@ -385,6 +428,19 @@ measures **30 cases / 8734 assertions**, both before and after this repair (re-measured on this branch, exit 0 in both configurations). Any future quote of this suite's count owes the configuration alongside it. +**And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), +filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn +CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in +`tests/` and `.agents/` (positive control run in the same command, so this is not +an assertion from a failed grep). CI therefore executes **502 of 8734 assertions — +5.7%** of this suite, at an unchanged case count of 30, and +`scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights +number in this Outcome — is a manual tool no gate invokes (`grep -rn +measure-ltx2-prompt-adaln` hits only its own usage string and this file). So this +row's checkpoint-derived evidence is **manual and host-local**: reproducible only +on a box carrying the 18.72 GB NVFP4 DiT and the 23 GB FP8 DiT under +`$CHECKPOINT_ROOT`. Wiring checkpoints into CI is explicitly NOT in this row. + The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said false, so the config/shape equality check refused it — correctly. It now carries @@ -449,9 +505,54 @@ measured rather than read: config, so what its flag resolves to is decided entirely out of band — and the tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). - On the NVFP4 file the flag IS on and the module IS built, but the tensor is - absent, so it keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through - `load_state_dict(..., strict=False)` - (`loader/single_gpu_model_builder.py:98`) — a genuine no-op there. + absent — 0 of its 7876 entries match `keyframes_abs_pos`. + + **CORRECTED AGAIN 2026-08-13, same issue.** This bullet said the parameter + "keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` (`loader/single_gpu_model_builder.py:98`) + — a genuine no-op there". **It is not a no-op.** The claim quoted that line + while dropping the `assign=True` that is ON THE SAME LINE, and upstream builds + on the **meta device** — `loader/helpers.py:84-95`, `create_meta_model`: + `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key + absent from the state dict is therefore never materialised at all; it is not a + zero, it is an **unmaterialised meta parameter**. + + Run 2026-08-13 through upstream's own `create_meta_model` on this file's real + `__metadata__` (read with upstream's own `read_model_metadata` / + `SafetensorsModelStateDictLoader`, which JSON-decodes each value — + `sft_loader.py:58-74`; the flag lives at `config.transformer`, not at the top + level): + + ```text + config.transformer['use_keyframes_abs_pos_embedding'] = True + keyframes_abs_pos_embedding: shape=(1, 4096) dtype=torch.float32 device=meta is_meta=True + supports_keyframes_abs_pos_embedding (BEFORE load) : False + after load_state_dict(sd, strict=False, assign=True): + neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised + keyframes_abs_pos_embedding : device=meta is_meta=True + in missing_keys : True + reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors + supports_keyframes_abs_pos_embedding (AFTER load) : False + ``` + + **Upstream says exactly this itself**, and then never asks. + `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) returns `False` for + "a model whose config set the flag but whose checkpoint carried no weight for it + (the parameter would still be on `meta`)", and + `enable_keyframes_abs_pos_embedding` (`model.py:175-200`) exists because such a + parameter "would fail at the first forward". **Both are defined and never + called** — one `grep -rn` hit each across the whole `Lightricks/LTX-2` checkout + at `fd4ded7f`, the definition itself, re-confirmed here rather than transcribed. + + Polarity is not what fails. `apply_keyframes_absolute_embedding` is + `hidden_states + mask * embedding` (`transformer_args.py:23-43`, the sum at + `:43`), so **real** zeros would be inert — the mechanism claim fails on `meta`, + not on additivity. + + This **strengthens** the row's conclusion and changes nothing downstream: the + refusal stays keyed on tensor presence for the FP8 file, and the NVFP4 file is + refused by flag in `ParseLtx2DitParams` (`ltx2.cpp:192-198`) — where upstream, + had it loaded, would carry a parameter its own guard reports as unsupported. It is also not a keyframe-only feature: `transformer_args.py:269` applies it on every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that @@ -497,8 +598,10 @@ tree vanished under it is not a result; it was rebuilt and re-run from scratch. ### Two divergences from upstream, recorded rather than fixed - **We are stricter than upstream about a config that disagrees with its file.** - Upstream loads with `load_state_dict(..., strict=False)` - (`loader/single_gpu_model_builder.py:98`), so a config declaring + Upstream loads with `load_state_dict(..., strict=False, assign=True)` + (`loader/single_gpu_model_builder.py:98` — quoted in full, because dropping the + `assign=True` is exactly what made the keyframes claim above wrong), so a config + declaring `use_prompt_adaln_single=false` over a file that carries the module would build no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's equality check refuses that. Ours is better; it is still a DIVERGENCE, not a diff --git a/docs/USAGE.md b/docs/USAGE.md index 6cf9f4f0f..2249df95d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2084,7 +2084,24 @@ its TENSORS; the first-party **NVFP4** file does NOT carry the tensor but its config DECLARES `use_keyframes_abs_pos_embedding: true`, so it is refused by the FLAG, in `ParseLtx2DitParams`, and `allow_unported_modules` clears that flag in a config copy. Neither file supports the retired claim that "LTX-2.5's checkpoint -does not carry the parameter". `prompt_adaln_single` and +does not carry the parameter". + +**What the opt-in costs you on the NVFP4 file: nothing upstream has either.** +Corrected 2026-08-13 — an earlier note said upstream keeps a zero-initialised +parameter there, "a no-op". It does not. Upstream builds on the **meta device** +(`create_meta_model`, `loader/helpers.py:90-91`) and loads with +`load_state_dict(..., strict=False, assign=True)` +(`loader/single_gpu_model_builder.py:98`), so a tensor the file does not carry is +never materialised: the parameter stays on `meta`, lands in `missing_keys`, and +reading it raises `RuntimeError: Tensor.item() cannot be called on meta tensors`. +Upstream's own `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) reports +such a model as NOT supporting the feature — and nothing in the LTX-2 checkout +ever calls that guard, or the `enable_…` repair next to it. So passing +`allow_unported_modules` on the NVFP4 DiT does not drop a capability upstream +would have given you; it declines a marker upstream cannot use on that file +either. On the **FP8** file the opt-in does drop something real — a trained +`[1, 4096]` tensor — which is why the refusal there is keyed on tensor presence +and is worth taking seriously. `prompt_adaln_single` and `audio_prompt_adaln_single` were on that list until 2026-08-13 and are now PORTED, so a checkpoint carrying them needs no opt-in on their account, and the opt-in no longer disables them. The two `*_embeddings_connector` towers are diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 87f4ddebb..5429f0555 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -60,11 +60,41 @@ // `_build_caption_projections` on the empty dict, BEFORE the flag at // model_configurator.py:82 is ever read). What that file's flag resolves // to therefore depends on a config it does not ship. -// * NVFP4 (first-party Lightricks): DECLARES the flag `true` and does NOT -// carry the tensor, so upstream's parameter stays its -// `torch.zeros(1, inner_dim)` initialiser (model.py:217-219) and -// `load_state_dict(..., strict=False)` -// (loader/single_gpu_model_builder.py:98) leaves it zero — a no-op. +// * NVFP4 (first-party Lightricks): DECLARES the flag `true` (at +// `config.transformer`, not at the top level) and does NOT carry the +// tensor — 0 of its 7876 entries match `keyframes_abs_pos`. +// +// CORRECTED AGAIN 2026-08-13, same issue. This bullet said the parameter +// "stays its `torch.zeros(1, inner_dim)` initialiser (model.py:217-219)" +// and that `load_state_dict(..., strict=False)` +// (loader/single_gpu_model_builder.py:98) "leaves it zero — a no-op". +// IT IS NOT A NO-OP. That quote dropped the `assign=True` that is ON THE +// SAME LINE, and upstream builds on the META DEVICE — helpers.py:84-95, +// `create_meta_model`, `with torch.device("meta"): +// configurator.from_metadata(...)` at :90-91. A key absent from the state +// dict is never materialised at all. What this path leaves is an +// UNMATERIALISED META PARAMETER, not a silent zero. +// RUN 2026-08-13 through upstream's own `create_meta_model` on this +// file's real `__metadata__`: `[1, 4096]` f32 `device=meta`; after +// `load_state_dict(sd, strict=False, assign=True)` still `is_meta=True`, +// present in `missing_keys`, and reading it raises `RuntimeError: +// Tensor.item() cannot be called on meta tensors`. +// Upstream states this itself and then never asks: +// `supports_keyframes_abs_pos_embedding` (model.py:166-173) returns FALSE +// for "a model whose config set the flag but whose checkpoint carried no +// weight for it (the parameter would still be on `meta`)" (:170), and +// `enable_keyframes_abs_pos_embedding` (model.py:175-200) exists because +// such a parameter "would fail at the first forward" (:182). BOTH ARE +// DEFINED AND NEVER CALLED — one `grep -rn` hit each across the whole +// Lightricks/LTX-2 checkout at fd4ded7f, the definition itself. +// Polarity is not what fails: `apply_keyframes_absolute_embedding` is +// `hidden_states + mask * embedding` (transformer_args.py:23-43, the sum +// at :43), so REAL zeros would be inert. The claim fails on `meta`, not +// on additivity. +// This STRENGTHENS the refusals below rather than changing them: this +// file is refused by FLAG in `ParseLtx2DitParams` (ltx2.cpp:192-198) and +// the FP8 file by TENSOR PRESENCE, and upstream's own guard would report +// a model loaded this way as not supporting the feature at all. // It is also NOT a keyframe-only feature: `transformer_args.py:269` applies it // on EVERY prepare whenever `keyframes_mask` is non-None, and `tools.py:186-196` // marks the target's first latent frame unconditionally. diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 119374ae9..d09108634 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -739,8 +739,13 @@ def rel(a, b): kv_off = ctx * (1 + scale_off) + shift_off kv_on = ctx * (1 + scale_on) + shift_on kabs, krel = rel(kv_off, kv_on) + # NAME THE STREAM. Both rows below are computed from `vmod` and the VIDEO + # `prompt_scale_shift_table`, and neither said so, which is how the Outcome came + # to divide a shipped AUDIO ratio by this VIDEO denominator (issue #644). The + # audio stream's own value differs -- 40.6% against this row's 51.7% -- so an + # unlabelled ratio here is a denominator waiting to be misapplied. lines.append( - f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = {kabs:.6g} " f"({krel * 100:.2f}% of max|off|)" ) # How much of the K/V modulation is timestep-conditioned at all: the MLP row @@ -748,8 +753,8 @@ def rel(a, b): static_max = float(table.abs().max()) term_max = float(vmod.abs().max()) lines.append( - f"// timestep term vs static table: max|term| = {term_max:.6g} vs " - f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + f"// timestep term vs static table (VIDEO stream): max|term| = {term_max:.6g} " + f"vs max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index d7f73a4ce..b56693b48 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3231,8 +3231,8 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) -// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) -// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table (VIDEO stream): max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) // ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and // the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every // ratio here is a property of THIS FIXTURE and moves with that scale; the output diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index b4896462f..4afa40777 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -952,8 +952,11 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // // WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the -// comment at the end of ltx2_goldens.inc report the term at 51.7% of the static -// per-block table, the block-0 prompt K/V moving 5.82%, and the DiT output moving +// comment at the end of ltx2_goldens.inc report the VIDEO stream's term at 51.7% +// of its static per-block table — that ratio is emitted for the video stream ONLY, +// and the audio stream's own value on the same fixture is 40.6%, so the 51.7% is +// not a denominator for anything audio — the block-0 prompt K/V moving 5.82%, and +// the DiT output moving // 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, // not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are // both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every