diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index cd53f75c3..1d54a595f 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -445,7 +445,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-higgs-audio-v3-higgs-audio-v3-talker-for-conditional-generation` | `HiggsMultimodalQwen3ForConditionalGeneration`, `HiggsAudioV3TalkerForConditionalGeneration` | **OUT-OF-REPO** — not in `555967922`, and absent from `registry.py` on vLLM `main`; registered by `vllm-project/vllm-omni` at `vllm_omni/model_executor/models/registry.py:213-217` (and the explicit `HiggsAudioV3TalkerForConditionalGeneration` spelling at `:218`) @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` → module `vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py`, class `HiggsAudioV3TalkerForConditionalGeneration` (separate module and class fields rather than the compact colon-pair form, per the out-of-repo convention), with code2wav at `registry.py:223-227` → module `higgs_audio_v3/higgs_audio_v3_code2wav.py`, class `HiggsAudioV3Code2WavForConditionalGeneration`. Documented at `vllm-omni` `docs/models/supported_models.md:81`. Official recipe: `bosonai/higgs-audio-v3-tts-4b` (published at `recipes.vllm.ai`), whose `config.json` declares `architectures: ["HiggsMultimodalQwen3ForConditionalGeneration"]`. The distinct v2 lineage (`HiggsAudioV2ForConditionalGeneration`, `higgs_audio_v2/`) is a DIFFERENT target and has no recipe in the audited set, so it gets no row here | conditional generation / text+audio in, AUDIO OUT (TTS) | MM processor; audio detokenizer / code2wav; audio tokenizer; Qwen3-derived multimodal decoder; streaming audio output | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` — a DIFFERENT string — is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audio→TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) — modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING — the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` → module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` — note the `2` — commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) — modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | +| `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** — not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-CONDITIONED prompt K/V via `prompt_adaln_single` (model.py:223-227, transformer.py:441-446), so the K/V cache is inapplicable on the shipped DiT -- the row's first claim of cacheability was RETRACTED 2026-08-12 and the module PORTED 2026-08-13 (#644, [spec](specs/ltx25-prompt-adaln.md)); on the real weights the timestep term DOMINATES the static table, RMS ratio 1347% video / 1583% audio; split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request — the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN — `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 29ab578da..f0fd89fa6 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -151,6 +151,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#673](https://github.com/mudler/vllm.cpp/issues/673) | `ROAD-V1-LTX25` | #644's checkpoint-derived evidence is MANUAL and host-local: `LTX2_CHECKPOINT_ROOT` is set by no workflow (`grep -rn CHECKPOINT_ROOT .github/` exits 1, positive control in `tests/`), so CI runs **502 of 8734 assertions — 5.7%** of `test_ltx2_video` at an unchanged case count of 30, and `scripts/measure-ltx2-prompt-adaln.py` is a manual tool no gate invokes. Recorded as visible debt; the skip itself is already announced in four places and that disposition stands | bug | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | diff --git a/.agents/specs/ltx25-prompt-adaln.md b/.agents/specs/ltx25-prompt-adaln.md index a24f127ce..4ffcf7335 100644 --- a/.agents/specs/ltx25-prompt-adaln.md +++ b/.agents/specs/ltx25-prompt-adaln.md @@ -228,10 +228,18 @@ byte cost. ## 6. Measured magnitude -Recorded in §Outcome: the relative change in the modulated prompt context and in -the DiT's outputs, flag ON vs OFF, at the reduced dimensions. This is the answer -to "does this matter"; a number below round-off would mean the term is inert and -the whole row is decoration. +Recorded in §Outcome, on two fixtures that answer two different questions. + +The reduced-dimension generator gives the relative change in the modulated prompt +context and in the DiT's outputs, flag ON vs OFF. That is a GATE FLOOR: a number +below round-off would mean the term is inert and the mutation in §5.3 could not +bite. It is **not** the answer to "does this matter", because both the static +table and the prompt-AdaLN MLP are drawn from the same synthetic init scale, so +every ratio it produces is a property of the fixture. + +"Does this matter" is answered on the SHIPPED checkpoint's own weights, run +through upstream's `AdaLayerNormSingle`. That measurement is required before the +row's Outcome may state a magnitude. ## 7. Risks @@ -256,7 +264,7 @@ the whole row is decoration. ## Outcome -### What was measured +### What was measured — (a) the gate floor, on SYNTHETIC weights The generator emits these into `tests/vllm/models/ltx2_goldens.inc` and prints them on stderr, from the SAME shared weight stream on both arms (keyed by @@ -265,21 +273,114 @@ term and nothing else): | Quantity | Flag ON vs OFF | |---|---| -| **timestep term vs the static table it is added to** | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — **51.7%** | -| **block-0 modulated prompt K/V** | `max\|on-off\|` 0.0310 — **5.82%** of `max\|off\|` | +| timestep term vs the static table it is added to (**VIDEO stream**) | `max\|term\|` 0.0252 vs `max\|table\|` 0.0487 — 51.7% | +| block-0 modulated prompt K/V (**VIDEO stream**) | `max\|on-off\|` 0.0310 — 5.82% of `max\|off\|` | | DiT video output (2 blocks) | 1.4567e-4 — 0.04% of `max\|off\|`, **73x** the gate's 2e-6 floor | | DiT audio output (2 blocks) | 7.367e-5 — 0.03%, **37x** the floor | -**The answer to "does this matter" is the first two rows.** Roughly half the -magnitude of the prompt K/V modulation is the timestep-conditioned term, and -including it moves the modulated prompt context by ~6%. Every render before this -row discarded that. +**ALL FOUR ROWS ARE GATE-FLOOR NUMBERS, AND NONE OF THEM ANSWERS "DOES THIS +MATTER".** Corrected 2026-08-13 (issue #644) — this section originally billed the +first two as the answer and disclaimed only the last two as synthetic-bounded. +They have identical provenance: `prompt_scale_shift_table` and every +prompt-AdaLN MLP parameter are drawn from the same `param_spec` rule at +`scale=0.05` (`scripts/gen-ltx2-goldens.py:100-106`), so the ratio between them +is a property of the FIXTURE, not of the conditioning. Vary only the MLP init and +it moves with it: 0.005 → 4.1% / 0.49%, 0.05 (committed) → 51.7% / 5.82%, +0.2 → 1450% / 142%. What these rows are FOR is the mutation below: 73x and 37x +above round-off is what makes a zeroed term detectable. + +### What was measured — (b) the SHIPPED checkpoint, which is the answer + +Measured 2026-08-13 by loading the real tensors into upstream's own +`AdaLayerNormSingle(inner_dim, embedding_coefficient=2)` (`adaln.py:19-45`, built +by `model.py:223-227` / `:253-257`) and evaluating it on `sigma * +timestep_scale_multiplier` — the file's own config gives 1000 — exactly as +`transformer_args.py:274-278` and `:177` do, then comparing against all 48 +`prompt_scale_shift_table` / `audio_prompt_scale_shift_table` tensors it is +summed with at `transformer.py:441-443`. + +- File: `/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors` + (7876 tensors, 1,179,408-byte header; the prompt-AdaLN tensors are BF16 there, + so they are read directly with no dequantisation step of ours in the path). +- Upstream: `ltx_core` at `fd4ded7f`, imported BY PATH from + `/home/mudler/_git/LTX-2` with `ltx_core.__file__` asserted under that checkout. +- Sigmas: a uniform grid over the whole range, `linspace(0, 1, 101)`, and + separately the sampler the file's own scheduler config names — + `LinearQuadraticScheduler().execute(8)` (`components/schedulers.py:60-88`). + +| | video (dim 4096) | audio (dim 2048) | +|---|---|---| +| `rms\|table\|`, 48 blocks | 0.017553 | 0.021925 | +| `rms\|term\|`, uniform σ | 0.236446 | 0.347171 | +| **term/table, RMS** | **1347%** | **1583%** | +| `max\|term\|` / `max\|table\|` | **7119%** | **2817%** | +| term/table RMS, LinearQuadratic 8-step | 1275% | 1492% | + +Split by row, on the uniform grid: shift 605% (video) / 461% (audio), scale 1732% +/ 2044%. The scale row is where it concentrates, which is the row that multiplies. + +**On the shipped model the timestep term DOMINATES the static table; the table is +the perturbation.** What the pre-row renders applied was +`context * (1 + ~0.018 rms) + ~0.018` where upstream applies +`context * (1 + ~0.32 rms) + ~0.10`. Stated as the quantity actually consumed at +`transformer.py:446`, for a context of unit rms: the modulated context has rms +1.0035 static-only against 1.0915 upstream (video, **+8.8%**) and 1.0033 against +1.3170 (audio, **+31.3%**). + +**The synthetic fixture UNDERSTATES the real defect by two orders of magnitude.** +Comparing like with like — the fixture ratio is `max\|term\|` / `max\|table\|`, +and so is the shipped-weights ratio — **per stream, each against its own fixture +denominator**: + +| | fixture `max\|term\|`/`max\|table\|` | shipped | understatement | +|---|---|---|---| +| video (`prompt_adaln_single` vs `prompt_scale_shift_table`) | 51.7% | 7119% | **138x** | +| audio (`audio_prompt_adaln_single` vs `audio_prompt_scale_shift_table`) | 40.6% | 2817% | **69x** | + +**CORRECTED 2026-08-13 (issue #644): this read "138x and 54x", which is not a +per-stream pair.** The generator emits exactly ONE fixture ratio and it is the +VIDEO stream's (`gen-ltx2-goldens.py`, `measure_prompt_adaln_magnitude`), so 54x +divided the shipped AUDIO number by the VIDEO denominator. It is literally true +about the single published figure and it errs CONSERVATIVE — the real audio +understatement is larger. The audio stream's own fixture ratio, recomputed with +the identical statistic on the identical fixture (the generator is imported, so +the weight stream is bit-identical), is **40.6%** — `max|term|` 0.0201763 vs +`max|table|` 0.0496868 — giving **2817 / 40.6 = 69x**. The video row is +unchanged and was always like-for-like: `max|term|` 0.0252012 vs `max|table|` +0.0487142 = 51.7327%, and 7119 / 51.73 = 137.6. + +Reproduce the audio denominator by importing the generator and asking it the same +question the video row is built from (`measure_prompt_adaln_magnitude`), so no +weight is re-drawn: -The two output rows are the GATE's floor, not a claim about the trained -checkpoint: they are bounded by the generator's synthetic weight scale (0.05) and -by a 2-block stack rather than 48. They are reported because a mutation must be -shown to move something, and 73x/37x above round-off is what makes the mutation -below meaningful. +```python +gen.load_upstream(LTX2); torch.set_grad_enabled(False) +on = gen.build_model("split", False, prompt_adaln=True) +video, audio = gen.build_modalities(False) +scale = float(gen.ARCH["timestep_scale_multiplier"]) +amod, _ = on.audio_prompt_adaln_single((audio.sigma * scale).flatten(), hidden_dtype=torch.float32) +amod.abs().max() / on.transformer_blocks[0].audio_prompt_scale_shift_table.abs().max() +``` + +**Fixed at the source, not only in prose.** The generator emitted both video-only +rows unlabelled, which is how a video denominator came to be applied to an audio +numerator; `gen-ltx2-goldens.py` now names the stream on both, and +`ltx2_goldens.inc` was REGENERATED against `ltx_core` `fd4ded7f` to carry them. +The regeneration diff is those two comment lines and nothing else — every golden +VALUE byte-identical, and the `Regenerate with:` header unchanged because the +committed command was re-run verbatim, so this re-proves provenance as well as +the labels. `test_ltx2.cpp` says the same. No fifth golden was added: the audio +figure is a denominator for this record, not a gate floor, so "ALL FOUR ROWS" +still holds. + +Recorded because the original Outcome quoted "roughly half the magnitude" from +the fixture as if it described the checkpoint. + +Reproduce with `scripts/measure-ltx2-prompt-adaln.py --ltx2 +--checkpoint `, committed by this repair so the number is +re-runnable rather than transcribed. It asserts `ltx_core.__file__` under the +named checkout before it reads anything, and nothing of ours is in its numeric +path — the only vllm.cpp input is which tensors to read. ### The mutations @@ -316,7 +417,29 @@ re-checking its md5 (`03324d42…`, identical before and after): | `test_ltx2` | 30 cases / 1627 assertions | 35 / 2435 | +5 cases, +808 assertions | | `test_ltx2_loader` | 24 / 4817 | 26 / 4826 | +2 cases, +9 (new cases minus the assertions the retired unported-family claims took with them) | | `test_ltx2_device` | 13 / 498 | 15 / 523 | +2 cases, +25 assertions | -| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — the fixture now carries the module, and no assertion counted it | +| `test_ltx2_video` | 30 / 502 | 30 / 502 | unchanged — see below; 502 is the SKIPPED default | + +**What `30 / 502` does and does not say (corrected 2026-08-13).** The +shipped-checkpoint case is env-gated: with `LTX2_CHECKPOINT_ROOT` unset it prints +`SKIPPED` and returns at `test_ltx2_video.cpp:917-921`, so `30 / 502` means the +whole real-header case DID NOT RUN — not "it ran and no assertion counted the +module". With the variable pointing at the Lightricks tree the same binary +measures **30 cases / 8734 assertions**, both before and after this repair +(re-measured on this branch, exit 0 in both configurations). Any future quote of +this suite's count owes the configuration alongside it. + +**And CI never sets it — [#673](https://github.com/mudler/vllm.cpp/issues/673), +filed 2026-08-13 as visible debt rather than repaired here.** `grep -rn +CHECKPOINT_ROOT .github/` exits 1 with zero hits while the same pattern matches in +`tests/` and `.agents/` (positive control run in the same command, so this is not +an assertion from a failed grep). CI therefore executes **502 of 8734 assertions — +5.7%** of this suite, at an unchanged case count of 30, and +`scripts/measure-ltx2-prompt-adaln.py` — which produces every shipped-weights +number in this Outcome — is a manual tool no gate invokes (`grep -rn +measure-ltx2-prompt-adaln` hits only its own usage string and this file). So this +row's checkpoint-derived evidence is **manual and host-local**: reproducible only +on a box carrying the 18.72 GB NVFP4 DiT and the 23 GB FP8 DiT under +`$CHECKPOINT_ROOT`. Wiring checkpoints into CI is explicitly NOT in this row. The `test_ltx2_video` fixture had to move: it declared a config that omits `use_prompt_adaln_single` (mirroring the shipped NVFP4 DiT) while its SHAPES said @@ -356,6 +479,137 @@ off: the loader asserts the flag against the file instead of clearing it, and `Ltx2AdoptDeclaredDitParams` clears exactly one flag, for a module nothing applies. +### The keyframes claim next door, corrected 2026-08-13 + +`ltx2.h` carried, in the same paragraph this row rewrote, *"LTX-2.5's checkpoint +does not carry the parameter"* about `keyframes_abs_pos_embedding`. It is FALSE — +the same class of claim as the `use_prompt_adaln_single=false` assertion this row +exists to remove — and the tree already contradicted it twice +(`.agents/model-matrix.md`, `tests/vllm/multimodal/test_ltx2_video.cpp:913-914`). +Read straight off both files' headers, and run through upstream's own loader and +configurator: + +| | FP8 (`vonkaiser`) | NVFP4 (first-party) | +|---|---|---| +| carries `keyframes_abs_pos_embedding` | YES — `F8_E4M3 [1, 4096]` + F32 scale | NO | +| declares the flag in `__metadata__` | **no `__metadata__` AT ALL** | `true` | +| `LTXModelConfigurator.from_metadata` | **RAISES** `KeyError: 'caption_channels'` | builds it, `[1, 4096]` | + +So the two files each contradict one half of the retired claim, and neither +supports it. Two corrections to the reasoning that came with the finding, both +measured rather than read: + +- The FP8 file does not "resolve the flag `False` at `model_configurator.py:82`". + Upstream never reaches line 82 on it: `_build_caption_projections` indexes + `caption_channels` on the empty config first and raises. That file ships no + config, so what its flag resolves to is decided entirely out of band — and the + tensor it carries is trained (`.agents/specs/ltx-2-5.md` §3.1 reads its bytes). +- On the NVFP4 file the flag IS on and the module IS built, but the tensor is + absent — 0 of its 7876 entries match `keyframes_abs_pos`. + + **CORRECTED AGAIN 2026-08-13, same issue.** This bullet said the parameter + "keeps `torch.zeros(1, inner_dim)` (`model.py:217-219`) through + `load_state_dict(..., strict=False)` (`loader/single_gpu_model_builder.py:98`) + — a genuine no-op there". **It is not a no-op.** The claim quoted that line + while dropping the `assign=True` that is ON THE SAME LINE, and upstream builds + on the **meta device** — `loader/helpers.py:84-95`, `create_meta_model`: + `with torch.device("meta"): configurator.from_metadata(...)` at `:90-91`. A key + absent from the state dict is therefore never materialised at all; it is not a + zero, it is an **unmaterialised meta parameter**. + + Run 2026-08-13 through upstream's own `create_meta_model` on this file's real + `__metadata__` (read with upstream's own `read_model_metadata` / + `SafetensorsModelStateDictLoader`, which JSON-decodes each value — + `sft_loader.py:58-74`; the flag lives at `config.transformer`, not at the top + level): + + ```text + config.transformer['use_keyframes_abs_pos_embedding'] = True + keyframes_abs_pos_embedding: shape=(1, 4096) dtype=torch.float32 device=meta is_meta=True + supports_keyframes_abs_pos_embedding (BEFORE load) : False + after load_state_dict(sd, strict=False, assign=True): + neighbour patchify_proj.weight : device=cpu is_meta=False <- materialised + keyframes_abs_pos_embedding : device=meta is_meta=True + in missing_keys : True + reading the value RAISES : RuntimeError: Tensor.item() cannot be called on meta tensors + supports_keyframes_abs_pos_embedding (AFTER load) : False + ``` + + **Upstream says exactly this itself**, and then never asks. + `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) returns `False` for + "a model whose config set the flag but whose checkpoint carried no weight for it + (the parameter would still be on `meta`)", and + `enable_keyframes_abs_pos_embedding` (`model.py:175-200`) exists because such a + parameter "would fail at the first forward". **Both are defined and never + called** — one `grep -rn` hit each across the whole `Lightricks/LTX-2` checkout + at `fd4ded7f`, the definition itself, re-confirmed here rather than transcribed. + + Polarity is not what fails. `apply_keyframes_absolute_embedding` is + `hidden_states + mask * embedding` (`transformer_args.py:23-43`, the sum at + `:43`), so **real** zeros would be inert — the mechanism claim fails on `meta`, + not on additivity. + + This **strengthens** the row's conclusion and changes nothing downstream: the + refusal stays keyed on tensor presence for the FP8 file, and the NVFP4 file is + refused by flag in `ParseLtx2DitParams` (`ltx2.cpp:192-198`) — where upstream, + had it loaded, would carry a parameter its own guard reports as unsupported. + +It is also not a keyframe-only feature: `transformer_args.py:269` applies it on +every `prepare` whose `keyframes_mask` is set, and `tools.py:186-196` sets that +mask unconditionally on the target's first latent frame. (Diffusers' own pipeline +does not consume it — `.agents/specs/ltx-2-5.md` §3.1 records that — but `ltx_core` +is what this campaign ports, and `ltx_core` does.) + +**The refusal keying does NOT change, and that is the decision, not an omission.** +`ltx2_loader.cpp`'s `RefuseUnported` fires on the TENSORS the file carries. Keying +it on the resolved flag instead would, on the FP8 DiT, read a DEFAULT rather than +the file — because that file declares nothing — and would therefore load it +silently while discarding a trained `[1, 4096]` parameter. Tensor presence is the +only signal that file actually carries, and refusing loudly with an opt-in is +strictly safer than resolving quietly. No behaviour changed, so no new gate is +owed; the refusal MESSAGE changed, because it asserted the implication that is +false in both directions. + +### The claims repair's own gate (2026-08-13) + +Nothing executable changed except three refusal MESSAGES, so the numbers are +expected to be identical and the point is that they are: + +- `BUILD_EXIT=0`; build logs grepped for `No space left|BFD assertion` — 0 hits; + `df -h /` 92% used, 37G free at the end. +- `ctest -N` = **423**. Full `ctest -j8` = 422/423 with `test_serve_low_tools` + starved under `-j` (a known parallel flake); serially **1/1 PASS, exit 0**. + 2 skipped (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`) as + on the baseline. +- Suite counts, unchanged from the row above: `test_ltx2` 35/2435, + `test_ltx2_loader` 26/4826, `test_ltx2_device` 15/523, `test_ltx2_video` + 30/502 skipped-default and **30/8734** with `LTX2_CHECKPOINT_ROOT` set. Exit 0 + on all five runs. +- `tests/vllm/models/ltx2_goldens.inc` REGENERATED from + `scripts/gen-ltx2-goldens.py` against `ltx_core` `fd4ded7f`: every golden VALUE + byte-identical, the diff is the comment block alone. That re-proves provenance + as well as the wording. + +An earlier full run was voided rather than reported: another session's +disk-pressure cleanup deleted `build/` while ctest was at 421/423, and the last +two tests recorded `Not Run — Failed to change working directory`. A run whose +tree vanished under it is not a result; it was rebuilt and re-run from scratch. + +### Two divergences from upstream, recorded rather than fixed + +- **We are stricter than upstream about a config that disagrees with its file.** + Upstream loads with `load_state_dict(..., strict=False, assign=True)` + (`loader/single_gpu_model_builder.py:98` — quoted in full, because dropping the + `assign=True` is exactly what made the keyframes claim above wrong), so a config + declaring + `use_prompt_adaln_single=false` over a file that carries the module would build + no module, drop 18 tensors on the floor and run flag-OFF without a word. §3.2's + equality check refuses that. Ours is better; it is still a DIVERGENCE, not a + mirror, and it is named here so it is not later mistaken for ported behaviour. +- **We refuse `keyframes_abs_pos_embedding` by tensor presence** where upstream + would take an out-of-band config's word for it (above). Same shape of + divergence, same reason it stands. + ## Now `DONE` — landed on `row/LTX25-PROMPT-ADALN`. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d835bbd30..70ba51b39 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -161,7 +161,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4, ~44 GB FP8, +~24 GB tower. FP8/torchao/NVFP4 load; `keyframes_abs_pos_embedding` needs `allow_unported` on BOTH DiTs. DiffVAE/LoRA/image cond refused. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/STATUS.md b/docs/STATUS.md index c93061da2..d8e4c9a73 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -139,7 +139,7 @@ token-for-token correctness against the pinned oracle. | Laguna-S-2.1 MoE (`LagunaForCausalLM`, 118B/8B) | **BINDING 2026-08-04: was 87% of vLLM (37.55 vs 43.10, same-tool nsys)**; root cause was bf16 projections on UNIFIED/ATS host memory, and device-resident staging (byte-exact) gives 44.6, parity+ vs 43.1, default-ON | 48 layers (12 global + 36 SWA-512), 256 routed top-10 + 1 shared expert, per-head softplus attn out-gate, sigmoid `noaux_tc` router, dual per-layer RoPE, GQA 8 KV / 128 head-dim, 1M ctx. History: benchmark-record | | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | -| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, the embeddings connector, pipeline, NVFP4/FP8 arms, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | +| LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435); prompt-side AdaLN #644.** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (f32 host, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..2249df95d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2072,12 +2072,36 @@ Only the LTX-2.5 DiT is gated against an independent oracle here, so treat any other marker-less NVFP4 checkpoint as unsupported until it is. See `.agents/specs/nvfp4-nibble-order.md`. -Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES the -shipped DiT by default, because that file carries **one** module family this port -does not carry (`keyframes_abs_pos_embedding`); pass -`Ltx2DitLoadOptions::allow_unported_modules` +Two behaviours a caller has to know. `Ltx2LoadDitFromSafetensors` REFUSES a +shipped DiT by default over **one** module family this port does not carry, +`keyframes_abs_pos_embedding`; pass `Ltx2DitLoadOptions::allow_unported_modules` to load the ported subset, which still reports every one of them in -`Ltx2DitCheckpoint::unported`. `prompt_adaln_single` and +`Ltx2DitCheckpoint::unported`. Both shipped DiTs need the opt-in, for DIFFERENT +reasons, and the distinction matters if you are reading the refusal text +(corrected 2026-08-13): the **FP8** file CARRIES the tensor +(`F8_E4M3 [1, 4096]`) and declares no `__metadata__` at all, so the refusal is on +its TENSORS; the first-party **NVFP4** file does NOT carry the tensor but its +config DECLARES `use_keyframes_abs_pos_embedding: true`, so it is refused by the +FLAG, in `ParseLtx2DitParams`, and `allow_unported_modules` clears that flag in a +config copy. Neither file supports the retired claim that "LTX-2.5's checkpoint +does not carry the parameter". + +**What the opt-in costs you on the NVFP4 file: nothing upstream has either.** +Corrected 2026-08-13 — an earlier note said upstream keeps a zero-initialised +parameter there, "a no-op". It does not. Upstream builds on the **meta device** +(`create_meta_model`, `loader/helpers.py:90-91`) and loads with +`load_state_dict(..., strict=False, assign=True)` +(`loader/single_gpu_model_builder.py:98`), so a tensor the file does not carry is +never materialised: the parameter stays on `meta`, lands in `missing_keys`, and +reading it raises `RuntimeError: Tensor.item() cannot be called on meta tensors`. +Upstream's own `supports_keyframes_abs_pos_embedding` (`model.py:166-173`) reports +such a model as NOT supporting the feature — and nothing in the LTX-2 checkout +ever calls that guard, or the `enable_…` repair next to it. So passing +`allow_unported_modules` on the NVFP4 DiT does not drop a capability upstream +would have given you; it declines a marker upstream cannot use on that file +either. On the **FP8** file the opt-in does drop something real — a trained +`[1, 4096]` tensor — which is why the refusal there is keyed on tensor presence +and is worth taking seriously. `prompt_adaln_single` and `audio_prompt_adaln_single` were on that list until 2026-08-13 and are now PORTED, so a checkpoint carrying them needs no opt-in on their account, and the opt-in no longer disables them. The two `*_embeddings_connector` towers are diff --git a/include/vllm/model_executor/models/ltx2.h b/include/vllm/model_executor/models/ltx2.h index 99898406b..5429f0555 100644 --- a/include/vllm/model_executor/models/ltx2.h +++ b/include/vllm/model_executor/models/ltx2.h @@ -44,9 +44,60 @@ // attention.py:545-552 `perturbation_mask`). L2 runs the no-perturbation // configuration, whose masks are all-ones and whose flags are all false — // upstream's own `perturbations=None` path (model.py:509-511). -// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). LTX-2.5's -// checkpoint does not carry the parameter; the enumeration refuses a config -// that asks for it. +// - `use_keyframes_abs_pos_embedding` (transformer_args.py:23-43). The +// enumeration refuses a config that asks for it. +// +// CORRECTED 2026-08-13 (issue #644). This line read "LTX-2.5's checkpoint +// does not carry the parameter", which is FALSE, and it is the same class of +// claim the prompt-AdaLN repair below exists to remove. The two shipped DiTs +// disagree with each other and each contradicts one half of it, read straight +// off their safetensors headers: +// * FP8 (`vonkaiser`, ltx-2.5-22b-distilled-fp8): CARRIES +// `keyframes_abs_pos_embedding` `F8_E4M3 [1, 4096]` plus its F32 scale, +// and declares NO `__metadata__` at all — so it ships no config, and +// upstream's own `LTXModelConfigurator.from_metadata` cannot configure it +// (run 2026-08-13: `KeyError: 'caption_channels'`, raised by +// `_build_caption_projections` on the empty dict, BEFORE the flag at +// model_configurator.py:82 is ever read). What that file's flag resolves +// to therefore depends on a config it does not ship. +// * NVFP4 (first-party Lightricks): DECLARES the flag `true` (at +// `config.transformer`, not at the top level) and does NOT carry the +// tensor — 0 of its 7876 entries match `keyframes_abs_pos`. +// +// CORRECTED AGAIN 2026-08-13, same issue. This bullet said the parameter +// "stays its `torch.zeros(1, inner_dim)` initialiser (model.py:217-219)" +// and that `load_state_dict(..., strict=False)` +// (loader/single_gpu_model_builder.py:98) "leaves it zero — a no-op". +// IT IS NOT A NO-OP. That quote dropped the `assign=True` that is ON THE +// SAME LINE, and upstream builds on the META DEVICE — helpers.py:84-95, +// `create_meta_model`, `with torch.device("meta"): +// configurator.from_metadata(...)` at :90-91. A key absent from the state +// dict is never materialised at all. What this path leaves is an +// UNMATERIALISED META PARAMETER, not a silent zero. +// RUN 2026-08-13 through upstream's own `create_meta_model` on this +// file's real `__metadata__`: `[1, 4096]` f32 `device=meta`; after +// `load_state_dict(sd, strict=False, assign=True)` still `is_meta=True`, +// present in `missing_keys`, and reading it raises `RuntimeError: +// Tensor.item() cannot be called on meta tensors`. +// Upstream states this itself and then never asks: +// `supports_keyframes_abs_pos_embedding` (model.py:166-173) returns FALSE +// for "a model whose config set the flag but whose checkpoint carried no +// weight for it (the parameter would still be on `meta`)" (:170), and +// `enable_keyframes_abs_pos_embedding` (model.py:175-200) exists because +// such a parameter "would fail at the first forward" (:182). BOTH ARE +// DEFINED AND NEVER CALLED — one `grep -rn` hit each across the whole +// Lightricks/LTX-2 checkout at fd4ded7f, the definition itself. +// Polarity is not what fails: `apply_keyframes_absolute_embedding` is +// `hidden_states + mask * embedding` (transformer_args.py:23-43, the sum +// at :43), so REAL zeros would be inert. The claim fails on `meta`, not +// on additivity. +// This STRENGTHENS the refusals below rather than changing them: this +// file is refused by FLAG in `ParseLtx2DitParams` (ltx2.cpp:192-198) and +// the FP8 file by TENSOR PRESENCE, and upstream's own guard would report +// a model loaded this way as not supporting the feature at all. +// It is also NOT a keyframe-only feature: `transformer_args.py:269` applies it +// on EVERY prepare whenever `keyframes_mask` is non-None, and `tools.py:186-196` +// marks the target's first latent frame unconditionally. // - The caption projections (text_projection.py:31-38). LTX-2.5 is a 22B-form // checkpoint: `caption_proj_before_connector=true` puts them in the TEXT // ENCODER, so the DiT has none (model_configurator.py:199-219). They are diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 46aa57faa..8d472913d 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -112,7 +112,11 @@ // stays gateable — proceeds, still reporting every one of them in `unported`: // // keyframes_abs_pos_embedding [1, 4096] -// So `use_keyframes_abs_pos_embedding = TRUE`, contradicting ltx2.h:47-49. +// Present in the FP8 DiT and ABSENT from the NVFP4 one, whose config +// nonetheless DECLARES the flag `true` — so tensors and flag do not imply +// each other in either direction, and the refusal is keyed on the tensors. +// Corrected 2026-08-13 (#644); the measured behaviour of BOTH files, and +// why the keying stays on tensors, is in ltx2.h. // This is now the ONLY flag `Ltx2AdoptDeclaredDitParams` clears in its // config copy, and it must stay that way: a flag cleared there is invisible // to the contract-equality check, so clearing a PORTED one silently drops diff --git a/scripts/gen-ltx2-goldens.py b/scripts/gen-ltx2-goldens.py index 4c5d3e54b..d09108634 100644 --- a/scripts/gen-ltx2-goldens.py +++ b/scripts/gen-ltx2-goldens.py @@ -739,8 +739,13 @@ def rel(a, b): kv_off = ctx * (1 + scale_off) + shift_off kv_on = ctx * (1 + scale_on) + shift_on kabs, krel = rel(kv_off, kv_on) + # NAME THE STREAM. Both rows below are computed from `vmod` and the VIDEO + # `prompt_scale_shift_table`, and neither said so, which is how the Outcome came + # to divide a shipped AUDIO ratio by this VIDEO denominator (issue #644). The + # audio stream's own value differs -- 40.6% against this row's 51.7% -- so an + # unlabelled ratio here is a denominator waiting to be misapplied. lines.append( - f"// block 0 modulated prompt K/V: max|on-off| = {kabs:.6g} " + f"// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = {kabs:.6g} " f"({krel * 100:.2f}% of max|off|)" ) # How much of the K/V modulation is timestep-conditioned at all: the MLP row @@ -748,19 +753,34 @@ def rel(a, b): static_max = float(table.abs().max()) term_max = float(vmod.abs().max()) lines.append( - f"// timestep term vs static table: max|term| = {term_max:.6g} vs " - f"max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" + f"// timestep term vs static table (VIDEO stream): max|term| = {term_max:.6g} " + f"vs max|table| = {static_max:.6g} ({term_max / static_max * 100:.1f}%)" ) lines.append( - "// The two output rows are bounded by this generator's SYNTHETIC weight scale" + "// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and" ) lines.append( - "// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate" + "// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every" ) lines.append( - "// needs, not a claim about the trained checkpoint. The K/V row is where the" + "// ratio here is a property of THIS FIXTURE and moves with that scale; the output" ) - lines.append("// term actually enters and is the number that answers 'does this matter'.") + lines.append( + "// rows are bounded by it AND by a 2-block stack. They are reported because a" + ) + lines.append( + "// mutation has to be shown to move something -- NOT as a claim about the trained" + ) + lines.append( + "// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to:" + ) + lines.append( + "// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's" + ) + lines.append( + "// own AdaLayerNormSingle on the real weights. See" + ) + lines.append("// .agents/specs/ltx25-prompt-adaln.md section Outcome.") text = "\n".join(lines) print("prompt-AdaLN magnitude:\n" + text, file=sys.stderr) return text @@ -818,7 +838,7 @@ def main() -> int: # Upstream's DEFAULT arm, and the one the shipped checkpoint runs. emit_prompt_adaln(out) out.write( - "// --- the MEASURED magnitude of the prompt-AdaLN term ---\n" + "// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE ---\n" "// Same shared weights, same inputs, flag ON vs OFF:\n" + measure_prompt_adaln_magnitude() + "\n" diff --git a/scripts/measure-ltx2-prompt-adaln.py b/scripts/measure-ltx2-prompt-adaln.py new file mode 100755 index 000000000..8961f845c --- /dev/null +++ b/scripts/measure-ltx2-prompt-adaln.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Measure the LTX-2.5 prompt-AdaLN timestep term against the static per-block +table, ON THE SHIPPED CHECKPOINT'S OWN WEIGHTS. + +Why this exists. `scripts/gen-ltx2-goldens.py` reports the same ratio at reduced +dimensions, but there the table AND the prompt-AdaLN MLP are both drawn from +`param_spec`'s `scale=0.05`, so the ratio it prints is a property of that fixture +and moves with the init scale. It is a gate FLOOR, not a statement about the +trained model. This script answers the other question — how much of the prompt +K/V modulation is timestep-conditioned on the model people actually run — and its +output is what `.agents/specs/ltx25-prompt-adaln.md` §Outcome quotes (issue #644). + +Nothing of ours is in the numeric path. It reads the safetensors header, pulls +six tensors per stream plus the 96 per-block tables, and runs UPSTREAM's own +module: + + ltx_core/model/transformer/adaln.py:19-45 AdaLayerNormSingle + ltx_core/model/transformer/model.py:223-227,253-257 built at embedding_coefficient=2 + ltx_core/model/transformer/transformer_args.py:274-278 driven by SIGMA, not timesteps + ltx_core/model/transformer/transformer_args.py:177 scaled by timestep_scale_multiplier + ltx_core/model/transformer/transformer.py:441-446 table + term, then context*(1+scale)+shift + ltx_core/components/schedulers.py:60-88 the sampler the file's config names + +Usage: + python3 scripts/measure-ltx2-prompt-adaln.py \ + --ltx2 ~/_git/LTX-2 \ + --checkpoint $CHECKPOINT_ROOT/ltx-2.5/lightricks-ltx-2.5/diffusion_models/\ +ltx-2.5-22b-distilled-transformer-nvfp4.safetensors +""" + +from __future__ import annotations + +import argparse +import json +import struct +import sys +from pathlib import Path + +PREFIX = "model.diffusion_model." +ADALN_KEYS = ( + "emb.timestep_embedder.linear_1.weight", + "emb.timestep_embedder.linear_1.bias", + "emb.timestep_embedder.linear_2.weight", + "emb.timestep_embedder.linear_2.bias", + "linear.weight", + "linear.bias", +) + + +def load_upstream(root: Path): + """Import `ltx_core` BY PATH from `root`, and PROVE that is what resolved. + + A pip-installed or otherwise-shadowing `ltx_core` would import silently and + every number below would describe an upstream nobody pinned — the failure + mode `.agents/specs/ltx-2-5.md` records at its "(b) byte-identical goldens" + entry. Identity is asserted, not assumed. + """ + src = root / "packages" / "ltx-core" / "src" + if not (src / "ltx_core").is_dir(): + raise SystemExit(f"no ltx_core under {src}; point --ltx2 at a Lightricks/LTX-2 checkout") + sys.path.insert(0, str(src)) + import ltx_core # noqa: PLC0415 + + resolved = Path(ltx_core.__file__).resolve() + if resolved.parent != (src / "ltx_core").resolve(): + raise SystemExit(f"ltx_core resolved to {resolved}, NOT to the checkout at {src}") + return resolved + + +def read_header(path: Path) -> dict: + """The safetensors header alone: an 8-byte length prefix and its JSON. No payload.""" + with path.open("rb") as fh: + length = struct.unpack(" float: + return float(t.pow(2).mean().sqrt()) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--ltx2", required=True, type=Path, help="path to a Lightricks/LTX-2 checkout") + ap.add_argument("--checkpoint", required=True, type=Path, help="a shipped LTX-2.5 DiT .safetensors") + ap.add_argument("--steps", type=int, default=8, help="step count for the shipped sampler arm") + args = ap.parse_args() + + resolved = load_upstream(args.ltx2.expanduser()) + import torch # noqa: PLC0415 + from safetensors import safe_open # noqa: PLC0415 + + from ltx_core.components.schedulers import LinearQuadraticScheduler # noqa: PLC0415 + from ltx_core.model.transformer.adaln import AdaLayerNormSingle # noqa: PLC0415 + + ckpt = args.checkpoint.expanduser() + print(f"ltx_core: {resolved}") + print(f"checkpoint: {ckpt}") + + header = read_header(ckpt) + meta = header.get("__metadata__") + if not meta or "config" not in meta: + raise SystemExit(f"{ckpt} declares no __metadata__ config; this script needs the file's own geometry") + config = json.loads(meta["config"]) + tcfg = config["transformer"] + mult = tcfg.get("timestep_scale_multiplier", 1000) + blocks = tcfg["num_layers"] + dims = { + "video": (tcfg["num_attention_heads"] * tcfg["attention_head_dim"], + "prompt_adaln_single.", "prompt_scale_shift_table"), + "audio": (tcfg["audio_num_attention_heads"] * tcfg["audio_attention_head_dim"], + "audio_prompt_adaln_single.", "audio_prompt_scale_shift_table"), + } + print(f"config: num_layers={blocks} timestep_scale_multiplier={mult} " + f"cross_attention_adaln={tcfg.get('cross_attention_adaln')} " + f"use_prompt_adaln_single={tcfg.get('use_prompt_adaln_single', ' upstream default True>')} " + f"sampler={config.get('scheduler', {}).get('sampler')}") + + arms = { + "uniform sigma [0,1] x101": torch.linspace(0.0, 1.0, 101, dtype=torch.float32), + f"shipped LinearQuadratic, {args.steps} steps": LinearQuadraticScheduler().execute(args.steps), + } + + torch.set_grad_enabled(False) + with safe_open(str(ckpt), framework="pt") as fh: + for stream, (dim, sub, table_name) in dims.items(): + module = AdaLayerNormSingle(dim, embedding_coefficient=2) + module.load_state_dict( + {k: fh.get_tensor(PREFIX + sub + k).float() for k in ADALN_KEYS}, strict=True + ) + module.eval() + tables = torch.stack([ + fh.get_tensor(f"{PREFIX}transformer_blocks.{i}.{table_name}").float() + for i in range(blocks) + ]) # [blocks, 2, dim]; row 0 shift, row 1 scale (transformer.py:444) + + t_rms, t_max = rms(tables), float(tables.abs().max()) + print(f"\n=== {stream} dim={dim} blocks={blocks}") + print(f" rms|table|={t_rms:.6f} max|table|={t_max:.6f}") + for arm, sigmas in arms.items(): + term = module(sigmas * mult, hidden_dtype=torch.float32)[0] + term = term.reshape(sigmas.numel(), 2, dim) # transformer.py:443 + e_rms, e_max = rms(term), float(term.abs().max()) + print(f" [{arm}] rms|term|={e_rms:.6f} -> {100.0 * e_rms / t_rms:.0f}% of rms|table|; " + f"max|term|={e_max:.6f} -> {100.0 * e_max / t_max:.0f}% of max|table|") + for row, name in ((0, "shift"), (1, "scale")): + print(f" {name}: rms|table|={rms(tables[:, row]):.6f} " + f"rms|term|={rms(term[:, row]):.6f} " + f"-> {100.0 * rms(term[:, row]) / rms(tables[:, row]):.0f}%") + # What is actually consumed (transformer.py:446): context*(1+scale)+shift. + # For a context of unit rms with independent entries the modulated + # context has rms sqrt(mean((1+scale)^2) + mean(shift^2)). + def modulated(scale, shift) -> float: + return float(((1.0 + scale).pow(2).mean() + shift.pow(2).mean()).sqrt()) + + static = modulated(tables[:, 1], tables[:, 0]) + full = modulated(tables[:, 1].unsqueeze(0) + term[:, 1].unsqueeze(1), + tables[:, 0].unsqueeze(0) + term[:, 0].unsqueeze(1)) + print(f" modulated context on a UNIT-rms context: static-only={static:.4f} " + f"upstream={full:.4f} ({100.0 * (full - static) / static:+.1f}%)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/model_executor/models/ltx2.cpp b/src/vllm/model_executor/models/ltx2.cpp index 3d9af69ce..344155af1 100644 --- a/src/vllm/model_executor/models/ltx2.cpp +++ b/src/vllm/model_executor/models/ltx2.cpp @@ -193,7 +193,9 @@ Ltx2DitParams ParseLtx2DitParams(const nlohmann::json& metadata) { !cfg.at("use_keyframes_abs_pos_embedding").is_boolean() || !cfg.at("use_keyframes_abs_pos_embedding").get(), "ltx2: use_keyframes_abs_pos_embedding is not ported (transformer_args.py:23-43); " - "the LTX-2.5 checkpoint does not carry keyframes_abs_pos_embedding"); + "upstream applies it on every prepare whose keyframes_mask is set, which " + "tools.py:186-196 populates unconditionally, so this is refused rather than " + "ignored (see ltx2.h for what each shipped DiT actually declares and carries)"); VT_CHECK(p.num_attention_heads > 0 && p.attention_head_dim > 0 && p.audio_num_attention_heads > 0 && p.audio_attention_head_dim > 0 && diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index bfc85075c..398d578a9 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -541,8 +541,11 @@ std::vector ContractOf(const Ltx2DitParams& params) { } Fail( "the checkpoint carries modules this port does NOT carry: " + list + - ". They are not dropped silently: keyframes_abs_pos_embedding means " - "use_keyframes_abs_pos_embedding is TRUE, and nothing here applies it. " + ". They are not dropped silently. This refusal is keyed on the TENSORS the " + "file carries, NOT on a declared flag, and that is deliberate: the FP8 DiT " + "carries a trained keyframes_abs_pos_embedding while declaring no " + "__metadata__ at all, so a flag-keyed refusal would read a DEFAULT rather " + "than the file and would silently discard it (see ltx2.h). " "prompt_adaln_single / audio_prompt_adaln_single are NO LONGER in this list " "— they were ported by row LTX25-PROMPT-ADALN " "(.agents/specs/ltx25-prompt-adaln.md, issue #644) and are now part of the " diff --git a/tests/vllm/models/ltx2_goldens.inc b/tests/vllm/models/ltx2_goldens.inc index 964047788..b56693b48 100644 --- a/tests/vllm/models/ltx2_goldens.inc +++ b/tests/vllm/models/ltx2_goldens.inc @@ -3227,14 +3227,19 @@ inline constexpr float kLtx2ForwardPromptAdalnMaskedAudio[] = { -0.09456487f, 0.0829611197f, 0.000701400219f, -0.243942767f, 0.0671276823f, 0.0765738413f, }; -// --- the MEASURED magnitude of the prompt-AdaLN term --- +// --- the prompt-AdaLN term's magnitude ON THIS SYNTHETIC FIXTURE --- // Same shared weights, same inputs, flag ON vs OFF: // DiT video output: max|on-off| = 0.000145674 (0.04% of max|off|) // DiT audio output: max|on-off| = 7.36713e-05 (0.03% of max|off|) -// block 0 modulated prompt K/V: max|on-off| = 0.0309528 (5.82% of max|off|) -// timestep term vs static table: max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) -// The two output rows are bounded by this generator's SYNTHETIC weight scale -// (0.05, param_spec above) and by a 2-block stack; they are the FLOOR the gate -// needs, not a claim about the trained checkpoint. The K/V row is where the -// term actually enters and is the number that answers 'does this matter'. +// block 0 modulated prompt K/V (VIDEO stream): max|on-off| = 0.0309528 (5.82% of max|off|) +// timestep term vs static table (VIDEO stream): max|term| = 0.0252012 vs max|table| = 0.0487142 (51.7%) +// ALL FOUR ROWS ARE GATE-FLOOR NUMBERS FROM SYNTHETIC WEIGHTS. The table and +// the prompt-AdaLN MLP are BOTH drawn at param_spec's scale=0.05 above, so every +// ratio here is a property of THIS FIXTURE and moves with that scale; the output +// rows are bounded by it AND by a 2-block stack. They are reported because a +// mutation has to be shown to move something -- NOT as a claim about the trained +// checkpoint. On the SHIPPED DiT the term DOMINATES the table it is added to: +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's +// own AdaLayerNormSingle on the real weights. See +// .agents/specs/ltx25-prompt-adaln.md section Outcome. } // namespace vllm_test diff --git a/tests/vllm/models/test_ltx2.cpp b/tests/vllm/models/test_ltx2.cpp index f00771f38..4afa40777 100644 --- a/tests/vllm/models/test_ltx2.cpp +++ b/tests/vllm/models/test_ltx2.cpp @@ -951,12 +951,24 @@ TEST_CASE("ltx2 forward: the prompt-side AdaLN arm, with both masks") { // term. A port that accepted the flag, bound the 12 tensors and then never added // their output would reproduce the flag-OFF numbers exactly and pass nothing here. // -// Measured on this fixture (generator stderr, and the comment block at the end of -// ltx2_goldens.inc): the term is 51.7% the magnitude of the static per-block -// table it is added to, moves the block-0 modulated prompt K/V by 5.82%, and -// moves the DiT's own output by 1.46e-4 — 73x the kRoundOff floor. The bound -// below is set at 20x kRoundOff so it is comfortably inside the measured signal -// and comfortably outside f32 noise. +// WHAT THIS FIXTURE'S NUMBERS ARE, AND ARE NOT. The generator's stderr and the +// comment at the end of ltx2_goldens.inc report the VIDEO stream's term at 51.7% +// of its static per-block table — that ratio is emitted for the video stream ONLY, +// and the audio stream's own value on the same fixture is 40.6%, so the 51.7% is +// not a denominator for anything audio — the block-0 prompt K/V moving 5.82%, and +// the DiT output moving +// 1.46e-4 (73x kRoundOff). ALL FOUR are GATE-FLOOR numbers from SYNTHETIC weights, +// not a claim about the trained checkpoint: the table and the prompt-AdaLN MLP are +// both drawn at `param_spec`'s scale=0.05 (gen-ltx2-goldens.py:100-106), so every +// ratio is a property of THIS FIXTURE and moves with the init scale. +// +// On the SHIPPED DiT the term DOMINATES the table rather than halving it — +// rms|term|/rms|table| = 1347% video, 1583% audio, measured through upstream's own +// AdaLayerNormSingle on the real weights (.agents/specs/ltx25-prompt-adaln.md +// §Outcome). So this fixture UNDERSTATES the defect; it does not bound it. +// +// The bound below is set at 20x kRoundOff: comfortably inside the signal this +// fixture does produce, and comfortably outside f32 noise. TEST_CASE("ltx2 forward: the prompt-AdaLN term is LOAD-BEARING, not decoration") { const Ltx2DitParams p = ReducedParamsPromptAdaln(Ltx2RopeType::kSplit, false); WeightSet set = BuildWeights(p); diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 76c8f8e31..d220957cc 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -925,6 +925,11 @@ TEST_CASE("ltx2 video: an ABI client loads, detects and generates through vllm.h // It is NOT interchangeable with the `vonkaiser` FP8 copy — they differ in a // TRAINED `keyframes_abs_pos_embedding` (spec section 3.1) — so the file this // case reads is named here and in every report of its result. +// WHEN QUOTING THIS SUITE'S ASSERTION COUNT, QUOTE THE CONFIGURATION WITH IT. +// This case skips by default, and it is most of the suite: unset, the binary +// measures 30 cases / 502 assertions; with LTX2_CHECKPOINT_ROOT set it measures +// 30 / 8734. An "unchanged 30 / 502" across a change therefore says the real +// headers were never read, not that they were read and nothing counted them. TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { const char* root_env = std::getenv("LTX2_CHECKPOINT_ROOT"); if (root_env == nullptr) {