From 38f8a7b760d49744342ab2d229be98c175a6478f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 19:11:40 +0000 Subject: [PATCH 1/6] spec(LTX25-RETIRE-DEAD-ARMS): the ledger claims arms upstream does not have (#644, #611) Spec before code for row LTX25-RETIRE-DEAD-ARMS. Grounds every entry of `Ltx2UnportedPipelineFeature` against LTX-2 fd4ded7f and diffusers 3a2f35d4, both re-derived by this row rather than taken from a prior grounding report. Four findings, three dispositions: - `kMultishot` is FABRICATED. Its comment cites "ltx-pipelines multishot entry points"; there is no such entry point, symbol or string in either reference. Searched as a SUBJECT, not as our own phrasing: the only upstream sense of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration", README:136 "a cinematographer describing a shot list"), and the one `scene` hit is PySceneDetect in the TRAINER. Retire it -- a defect in our record is not a gap in our port, so recording it as owed is the error. - `kCfgParallelism` names something upstream does not do: 0 `cfg` hits in either multigpu tree. The three real forms are sequence-parallel, tiled data parallel and distributed VAE decode, and upstream's own README calls MGPU "a latency tool, not a memory tool". CFG is not in our path at all -- the distilled recipe denoises with SimpleDenoiser, "single transformer call, no guidance". - `kVideoEngineWiring` says the wiring "is phase L7, not L5". L7 landed in cefacd2d0. Retire it. - `kInt8ConvRot` is correct: the four inference quantization kinds are fp8-cast, fp8-scaled-mm, nvfp4-cast, nvfp4-prequant, and every upstream int8 is either training-only or a uint8 buffer. Kept and re-anchored so nobody re-audits it. Five of the seven enumerators have NO product call site, so "refused by name" overstated them; they are declared-out-of-scope markers and the code will say so. Separately (#611): `duration_head_path` is accepted and read by nothing. The spec carries the full ten-key `kKnownLoadExtras` audit -- nine READ, that one ignored -- so the sweep #611 asks for is closed, not sampled. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 1 + .agents/specs/ltx25-retire-dead-arms.md | 354 ++++++++++++++++++++++++ 2 files changed, 355 insertions(+) create mode 100644 .agents/specs/ltx25-retire-dead-arms.md diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 759da0349..c1e2fc2ab 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -148,6 +148,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#611](https://github.com/mudler/vllm.cpp/issues/611) | `ROAD-V1-LTX25` | `duration_head_path` is accepted by `kKnownLoadExtras` and read by NOTHING, so a caller who points at a duration head silently gets the recipe default. Row `LTX25-RETIRE-DEAD-ARMS` (spec [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md)) refuses it by name and records the full accepted/read inventory of all ten extras | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | | [#365](https://github.com/mudler/vllm.cpp/issues/365) | `PERF-27B-DENSE-MARLIN-GATEUP` | 27B gap decomposed vs vLLM's own decode profile; dense W4A16 MLP bypassed the fused gate_up seam (spec `specs/perf-27b-dense-marlin-gateup.md`) | bug | diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md new file mode 100644 index 000000000..3c4879fda --- /dev/null +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -0,0 +1,354 @@ +# LTX-2.5 — retire the arms that do not exist, and refuse the extra that is ignored + +Row: `LTX25-RETIRE-DEAD-ARMS`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +not edited by this row). Issues: +[#644](https://github.com/mudler/vllm.cpp/issues/644) items D–I and N, +[#611](https://github.com/mudler/vllm.cpp/issues/611). +Pattern this row is an instance of: [#604](https://github.com/mudler/vllm.cpp/issues/604). + +Upstream pins: + +| Reference | Revision | Local checkout verified at | +|---|---|---| +| Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | `git rev-parse HEAD`, 2026-08-13 | +| huggingface/diffusers | `3a2f35d4efa4c059c8bfb3bc0d6c906264895c81` | `git rev-parse HEAD`, 2026-08-13 | + +Every anchor below was re-derived from those two checkouts by this row. A prior +grounding pass reached the same conclusions; it is an input, not a result. + +## 0. What is wrong today + +`Ltx2UnportedPipelineFeature` (`include/vllm/model_executor/models/ltx2_pipeline.h:580-587`) +is a ledger of seven arms this port refuses by name. Read as a whole it claims +more than is true, in four separate ways: + +1. **One arm is fabricated.** `kMultishot` refuses "multishot generation" and its + enum comment cites *"ltx-pipelines multishot entry points"*. No such entry + point exists. Neither does the symbol, the string, or the concept. +2. **One arm's anchor names something upstream does not do.** `kCfgParallelism` + is anchored to `ltx-pipelines/multigpu`, which contains no CFG batching at + all — the three real parallelisms are orthogonal to guidance, and the recipe + this port runs uses no guidance in the first place. +3. **One arm is stale.** `kVideoEngineWiring` says the end-to-end wiring "is + phase L7, not L5". L7 landed in `cefacd2d0`. +4. **Five of the seven have no product call site.** Recording them as "refused by + name" overstates what exists: nothing a caller can send reaches them. They are + declared-out-of-scope markers, and the code did not say so. + +Separately, and user-visible: `duration_head_path` is accepted by +`kKnownLoadExtras` (`src/vllm/multimodal/ltx2_video.cpp:257`) and **read by +nothing**. A caller who points at a duration head gets the recipe default and is +told nothing. + +## 1. What the references actually contain + +### 1.1 `multishot` — FABRICATED + +Searched as a **subject**, not as our own phrasing (the self-confirming-grep trap, +#604): + +| Query | LTX-2 `fd4ded7f` | diffusers `3a2f35d4` | +|---|---|---| +| `multishot` / `multi_shot` / `multi-shot` (case-insensitive) | 0 hits | 0 hits | +| `\bshots?\b` in `*.py` `*.md` `*.json` `*.yaml` | 5 files, none a generation mode | — | +| `multi[_ -]?(gpu\|scale\|stage\|shot\|clip\|prompt\|segment)` | `multi-GPU` / `multi_gpu` only | — | +| `scene` / `storyboard` | `ltx-trainer/scripts/split_scenes.py` (PySceneDetect, a **training-data** preprocessor) | — | + +The five `shot` files, with what the word means in each: + +- `ltx-core/duration_head/duration_head.py:1,5` — "predicts **shot** duration", + i.e. the natural length of ONE camera take. +- `ltx-core/duration_head/__init__.py:1` — same. +- `ltx-pipelines/utils/blocks.py:804` — `DurationPredictor`, "Predicts **shot** + duration (in frames)". +- `ltx-trainer/src/ltx_trainer/captioning.py` — captioning prose. +- `LTX-2/README.md:59,136` — an example prompt ("a medium close-up **shot**") and + the prompting guide ("think like a cinematographer describing a **shot** list"). + +Upstream's "shot" is a single continuous take. There is no multi-shot generation +mode, and nothing that composes several takes into one output. + +`ltx-pipelines`' actual entry points, from `docs/pipelines.md` ("Full reference +for all 11 pipelines") and the module list: +`ti2vid_one_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, +`ti2vid_two_stages_mgpu`, `ti2vid_two_stages_hq_mgpu`, `distilled`, +`distilled_mgpu`, `ic_lora`, `hdr_ic_lora`, `a2vid_two_stage`, `t2a_one_stage`, +`dubit`, `retake`, `dfr_pipeline`, `keyframe_interpolation`. None is multi-shot. + +**Disposition: RETIRE.** This is a defect in our record, not a gap in our port. +There is nothing to owe, so recording it as owed is the error. The enumerator is +removed and the retirement recorded in the header, in this spec, and in the +commit message — which is where an exception's reason lives (AGENTS.md, "there is +no waiver registry"). + +### 1.2 `int8-convrot` — real absence, deliberately out of scope + +The inference quantization kinds LTX-2 defines, exhaustively +(`ltx-pipelines/utils/quantization_factory.py:23-27`, a `str`-valued enum with an +`assert_never` on the match): + +```python +class QuantizationKind(str, Enum): + FP8_CAST = "fp8-cast" + FP8_SCALED_MM = "fp8-scaled-mm" + NVFP4_CAST = "nvfp4-cast" + NVFP4_PREQUANT = "nvfp4-prequant" +``` + +No int8 arm, and no rotation of any kind: `convrot` / `conv_rot` / `hadamard` / +`quarot` / `spinquant` are 0 hits across the repository. `rotation` hits only EXIF +image orientation (`media_io/decode.py:32`) and the VAE's per-slab spatial +rotation (`video_vae/transformer/det_attn_rope.py:61`) — neither a quantization +transform. + +`int8` upstream is **training-only**, in two places, and neither is an inference +weight format: + +- `ltx-trainer/src/ltx_trainer/gemma_8bit.py:33-36` — bitsandbytes `LLM.int8()` + for the Gemma backbone during LoRA training. +- `ltx-trainer/src/ltx_trainer/quantization.py:11-15` — optimum-quanto precisions + (`int8-quanto`, `int4-quanto`, …) for the trainer. + +Every other `int8` match in the repository is `uint8`: pixel buffers, packed NVFP4 +nibbles, block-streaming staging. + +**Disposition: KEEP, re-anchored.** The refusal message is honest — it is a +ComfyUI-ecosystem quantization, not an LTX-2 arm — but the enum comment did not +say it had been checked. It now records the absence at these pins so nobody +re-audits it. Its *kind* changes: it is a declared-out-of-scope marker, not a +reachable refusal. + +### 1.3 CFG parallelism — the name describes something upstream does not do + +`cfg` (case-insensitive, word) is **0 hits** in both multi-GPU trees: +`ltx-pipelines/src/ltx_pipelines/multigpu/` and +`ltx-core/src/ltx_core/multigpu/`. + +The three parallelisms upstream actually implements: + +| Form | Anchor | What it splits | +|---|---|---| +| Sequence parallel | `multigpu/sp_builder.py:25` (`SequenceParallelBuilder`), `ltx-core multigpu/transformer/sequence_parallel.py`, all-to-all attention | the token axis of one denoise step | +| Tiled data parallel | `multigpu/tdp_builder.py:25` (`TiledDataParallelBuilder`) | spatial tiles, **upscale stage only** | +| Distributed VAE decode | `ltx-core multigpu/vae/distributed_decoder.py:204-256` (`DistributedVideoDecoder.decode_video`) | latent tiles across ranks, driver blends | + +Upstream states the purpose in its own words +(`ltx-pipelines/docs/multigpu/README.md:7`, inside the ⚠️ block at `:5-16`): + +> **Multi-GPU (MGPU) is a latency tool, not a memory tool.** + +and adds that the mutable transformer is a **full replica on every GPU**, so it +cannot make a checkpoint fit. + +And CFG is not in the path this port runs at all. The distilled recipe denoises +with `SimpleDenoiser` at both stages (`ltx-pipelines/distilled.py:266`, `:295`), +documented as "**single transformer call, no guidance**" +(`utils/denoisers.py:3`); the guider it degenerates to is +`MultiModalGuiderParams(cfg_scale=1.0, stg_scale=0.0, modality_scale=1.0)` +(`utils/denoisers.py:25-26`), "only runs the conditioned pass and returns cond +unchanged". A `cfg_scale` of 1.0 is one pass, so there is no second pass to place +on a second GPU. + +**Disposition: RENAME + re-anchor.** `kCfgParallelism` → `kMultiGpuParallelism`, +anchored to the three real forms, with the reason it is out of scope stated as +what it is: a single-node multi-GPU **latency** feature, on a port whose target is +one GB10. + +### 1.4 `kLoraFusion` — real upstream, correctly refused + +Verified present: `ltx-core/loader/primitives.py:160` +(`class LoraPathStrengthAndSDOps(NamedTuple)`), exported at +`loader/__init__.py:14,46`, consumed by `loader/single_gpu_model_builder.py:21` +and `block_streaming/builder.py:34,90`, fused by `loader/fuse_loras.py`. The +anchor stands; only its *kind* is corrected (marker, not reachable refusal). + +### 1.5 `kVideoEngineWiring` — LANDED + +`cefacd2d0` ("feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that +is no longer MiniMax-only (#435) (#641)", 2026-08-13) shipped exactly this: the +composition through `vllm::multimodal::VideoEngine`, reachable through the C ABI +as video family `ltx-2.5`. `include/vllm/multimodal/ltx2_video.h:12` already +speaks of the refusal in the past tense. + +**Disposition: RETIRE.** A refusal whose subject shipped is a false statement, not +a record of debt. + +### 1.6 The five with no product call site + +`grep` over `src/`, `include/`, `examples/`, `tests/` for every enumerator: + +| Enumerator | Product call site | +|---|---| +| `kTemporalUpsampler` | `src/vllm/model_executor/models/ltx2_upsampler.cpp:395` | +| `kBetaScheduler` | `src/vllm/model_executor/models/ltx2_pipeline.cpp:199` | +| `kLoraFusion` | **none** | +| `kMultishot` | **none** | +| `kInt8ConvRot` | **none** | +| `kCfgParallelism` | **none** | +| `kVideoEngineWiring` | **none** | + +The five are enumerated only by `tests/vllm/models/test_ltx2_pipeline.cpp:1237-1244`. +There is no request field, load extra, or CLI flag that asks for a LoRA, an +int8-convrot checkpoint, or a second GPU, so no caller can trip them. + +That is not a defect on its own — a declared boundary is worth having. The defect +is calling it a refusal. The header and the messages now distinguish: + +- **reachable refusal** — a product path constructs the condition and throws; +- **declared-out-of-scope marker** — a record of what upstream has and this port + does not, reached only by the ledger test. + +## 2. `duration_head_path` (#611) + +### 2.1 The full `kKnownLoadExtras` audit + +Every key the family accepts, and whether any code reads it. Reader anchors are in +`src/vllm/multimodal/ltx2_video.cpp` unless noted. + +| Key | Constant | Reader | Status | +|---|---|---|---| +| `audio_prompt_embeds_path` | `kLtx2AudioPromptEmbedsExtra` | `:901`, `:914` | READ | +| `pipeline_kind` | `kLtx2PipelineKindExtra` | `:737` | READ | +| `model_version` | `kLtx2ModelVersionExtra` | `:721` | READ | +| `allow_unported_modules` | `kLtx2AllowUnportedExtra` | `:570` | READ | +| `max_phase` | `kLtx2MaxPhaseExtra` | `:739` | READ | +| `dit_config_path` | `kLtx2DitConfigPathExtra` | `:625` | READ | +| `prompt_embeds_valid_rows` | `kLtx2PromptValidRowsExtra` | `:942` | READ | +| `encoder_config_path` | `kLtx2EncoderConfigPathExtra` | `:796` | READ | +| `upsampler_path` | (literal) | `:771` | READ | +| `duration_head_path` | (literal) | **none** | **ACCEPTED AND IGNORED** | + +Nine of ten are wired. `duration_head_path` is the only defect, so the sweep this +row owes is complete and closes the "the sweep that found this one did not cover +them all" clause of #611. + +One documentation gap found by the same sweep and fixed here: +`docs/USAGE.md:1650-1654` lists the LTX-2.5 extras and omits `encoder_config_path` +entirely, though it is defined and read. + +### 2.2 Why it is inert + +`ltx2_duration_head.h` / `ltx2_duration_head.cpp` port `DurationHead` and +`AttentionPooler` and gate them as bricks. Nothing in `ltx2_video.cpp` constructs +one. The AUTO-duration path (`resolve_num_frames`, `ltx-pipelines/utils/blocks.py`) +therefore cannot run, and `Generate` computes `frames` from +`duration_seconds * fps` directly (`ltx2_video.cpp:1152`). + +The stated reason for this moved twice already (#604's pattern, and the finding +that produced #611). The reason recorded here is the current one: **no head is +constructed**, so a supplied path is a file the engine never opens. + +### 2.3 Fix + +Refuse `duration_head_path` **by name** when it is supplied and non-empty, with a +message that says the head is unported, that the recipe default would otherwise be +substituted silently, and what to use instead (`num_frames`, or `duration`, which +is exact arithmetic against the recipe frame rate). + +Rejected alternative: dropping the key from `kKnownLoadExtras`. That produces +"unknown load extra", which is *wrong* — the key is defined by this family and its +meaning is understood; what is missing is the implementation. AGENTS.md requires +"a message naming the missing piece", and "unknown key" does not name it. + +Rejected alternative: reading it and constructing the head. That is the real fix +and it stays owed — it needs the connector-output plumbing the head consumes +(`duration_head.py:89-118` takes audio and/or video connector token states), which +is a different row. Refusing is the cheap correct answer until then. + +## 3. Scope + +**In.** + +1. Retire `kMultishot` and `kVideoEngineWiring`; record both retirements in the + header, this spec, and the commit message. +2. Rename `kCfgParallelism` → `kMultiGpuParallelism` and re-anchor its message to + the three real parallelisms plus the reason CFG is not in our path. +3. Re-anchor `kInt8ConvRot`'s comment and message to record the verified absence + at these pins. +4. Split the ledger into reachable refusals and declared-out-of-scope markers, in + the header comment, in the messages, and in the test. +5. Refuse `duration_head_path` by name (#611). +6. `docs/USAGE.md`: the extras paragraph, corrected on both counts. + +**Out.** + +- `.agents/specs/ltx-2-5.md` — operator-owned. This row does not edit it. Its §2 + "Out" list still names `multishot`; correcting that is the operator's edit, and + this spec is the record it would cite. +- Constructing a duration head, and the AUTO-duration path. Stays owed (#611 + remains open after this row, retitled by the fix rather than closed by it — + see §7). +- `kTemporalUpsampler` (row `LTX25-TEMPORAL-UPSAMPLER`), image conditioning (row + `LTX25-IMAGE-COND-FIX`), tiled decode (row `LTX25-TILED-DECODE`), AdaLN claims + (row `LTX25-ADALN-CLAIMS`). No file in this diff is theirs. + +## 4. Tests + +RED first for the behavioural change; the records changes are gated by the ledger +test's own assertions. + +1. **`ltx2 duration_head_path is REFUSED by name` (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`).** Builds a valid load, adds + `extras["duration_head_path"]`, and requires a throw whose message names the + key, names the duration head as the missing piece, and names the alternative. + **RED before the fix**: the load succeeds, because nothing reads the key. That + is the defect stated as a test. +2. **`ltx2 an accepted load extra is READ by something` (new).** Asserts the + inventory of §2.1 does not silently grow: every key in `kKnownLoadExtras` + either round-trips through a reader or is refused by name. Implemented as the + two known-inert keys being refused and the rest being accepted, so adding a + tenth decorative key fails. +3. **`ltx2 every L5 out-of-scope feature is refused BY NAME`** (existing, updated). + The list drops from 7 to 5 entries — a CHANGED COUNT, reported as such — and + splits into `reachable` and `markers`, with the marker messages required to say + they are not requestable. Adds a guard that no refusal message mentions + `multishot` again. + +## 5. Risks + +| Risk | Mitigation | +|---|---| +| Removing an enumerator breaks an out-of-tree caller | `Ltx2UnportedPipelineFeature` is internal (`include/vllm/model_executor/`), not part of `include/vllm.h`. Grep shows 3 call sites, all in-tree. | +| Refusing `duration_head_path` breaks a working caller | It cannot: no code reads it, so no caller was getting anything from it. A caller who passes it today is being silently ignored, which is the bug. | +| The ledger test's case count changes and reads as a regression | Stated up front in the gate report: 7 → 5 entries in one case, plus 2 new cases. The exit code is the authority. | +| A concurrent LTX row edits the same header | Four live rows named; none owns `ltx2_pipeline.h:575-590`, `ltx2_video.cpp:250-300`, or the ledger test case. `LTX25-TEMPORAL-UPSAMPLER` owns `kTemporalUpsampler`, which this row does not touch. | + +## 6. Stop conditions + +- Return `NEEDS_DECISION` if `multishot` turns out to exist under vocabulary not + searched here — the retirement is then wrong and the arm must be re-anchored + instead. +- Return `NEEDS_DECISION` on a collision with a live LTX row. +- Do not close #611 if the refusal cannot be made to fail RED first. + +## 7. Outcome + +**Every claim in §1 was re-derived by this row against the two pinned checkouts; +none was taken from the prior grounding pass.** Results: + +- `multishot`: **fabricated, confirmed.** 0 hits for the term in either reference; + the only upstream sense of "shot" is one camera take (§1.1). Retired. +- `int8-convrot`: **absent upstream, confirmed.** Four inference quantization kinds + exist and none is int8; int8 is training-only (§1.2). Kept, re-anchored, so the + absence does not have to be re-audited. +- CFG parallelism: **the name was wrong, the exclusion is right.** 0 `cfg` hits in + either multi-GPU tree; upstream's own README calls MGPU a latency tool; the + distilled recipe runs `SimpleDenoiser` with no guidance at all, so there is no + CFG pass to parallelize (§1.3). Renamed and re-anchored. +- `kVideoEngineWiring`: **stale, confirmed.** L7 landed in `cefacd2d0`. Retired. +- Five enumerators had no product call site (§1.6). The ledger now says which + kind each is, because "refused by name" overstated a marker. +- `duration_head_path` was the **only** unread key of ten (§2.1). Now refused by + name. The full inventory is the durable half of this row: it means the next + person asking "which extras are decorative?" reads a table instead of grepping. + +What this row deliberately did **not** do: construct a duration head. #611 stays +open for that, with its user-visible half — silent substitution of the recipe +default — closed. + +## Now + +Row `LTX25-RETIRE-DEAD-ARMS` is `DONE`. The ledger carries five entries, split by +kind; `duration_head_path` is refused by name with a RED-first test; the +`kKnownLoadExtras` inventory is recorded in §2.1. `.agents/specs/ltx-2-5.md` §2 +"Out" still lists `multishot` and is the operator's to correct, citing §1.1 here. From 1b43819470a30224d3bd0d4adca33d48a8989c24 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 19:39:21 +0000 Subject: [PATCH 2/6] fix(LTX25-RETIRE-DEAD-ARMS): retire the invented refusals, refuse the ignored extra (#644, #611) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LTX-2.5 out-of-scope ledger claimed more than was true, in four ways, and one load extra was accepted and read by nothing. Every claim below was re-derived by this change against Lightricks/LTX-2 fd4ded7f and huggingface/diffusers 3a2f35d4; a prior grounding report was an input, never the result. RETIRED, because they are defects in our record rather than gaps in our port: - `kMultishot` refused "multishot generation" and cited "ltx-pipelines multishot entry points". There is no such entry point, symbol or string in either reference. Searched as a SUBJECT, not from our own phrasing: the only upstream sense of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration"; README.md:136 "a cinematographer describing a shot list"), and the only `scene` hit is PySceneDetect in the TRAINER. There was nothing to owe. - `kVideoEngineWiring` said the wiring through vllm::multimodal::VideoEngine "is phase L7, not L5". L7 landed in cefacd2d0. A refusal whose subject shipped is a false statement, not a record of debt. RE-ANCHORED: - `kCfgParallelism` -> `kMultiGpuParallelism`. Zero `cfg` hits in either multigpu tree; the three real forms are sequence-parallel (sp_builder.py:25), tiled data parallel (tdp_builder.py:25, upscale only) and distributed VAE decode (distributed_decoder.py:204-256). Upstream calls MGPU "a latency tool, not a memory tool" (docs/multigpu/README.md:5-16), and there is no CFG pass here to split anyway: the distilled recipe runs SimpleDenoiser at both stages (distilled.py:266,295), "single transformer call, no guidance" (denoisers.py:3). - `kInt8ConvRot` was already honest and is now recorded as VERIFIED ABSENT so it is not re-audited: upstream defines exactly four inference quantization kinds (quantization_factory.py:23-27), none int8; `convrot` is 0 hits; every other upstream int8 is the TRAINER's bitsandbytes/quanto path or a uint8 buffer. CLASSIFIED. Five of the seven enumerators had NO product call site, so "refused by name" overstated them. The header, the messages and the test now separate a REACHABLE REFUSAL (kTemporalUpsampler, kBetaScheduler -- a product path builds the condition) from a DECLARED-OUT-OF-SCOPE MARKER (the rest -- nothing a caller can send reaches it). The ledger case drops 7 -> 5 entries: a CHANGED COUNT that is the retirement, and it gains assertions rather than losing them. REFUSED (#611). `duration_head_path` was in `kKnownLoadExtras` and read by nothing, so a caller who pointed at a duration head silently got the recipe default. AGENTS.md requires an unimplemented arm to be refused with a message naming the missing piece. `CheckUnservedExtras` now does that, naming the head, why it cannot load, and `num_frames` / `duration` as the alternatives. NOT by dropping the key: "unknown load extra" would claim the family does not define it, which is false and sends the reader hunting a typo. The full audit is in the spec §2.1 and closes #611's "audit the rest the same way": nine of the ten keys reach a reader (ltx2_video.cpp:570, 625, 721, 737, 739, 771, 796, 901, 942); `duration_head_path` was the only one with none. RED first. `ltx2 video: duration_head_path is REFUSED by name` and `ltx2 video: every accepted load extra is READ by something` both FAILED before the fix (2 failed / 12 of 14 assertions passing) for exactly the intended reason -- the load succeeded. The inventory case parses the accepted list out of the unknown-extra refusal, which is built from `kKnownLoadExtras` itself, so an eleventh decorative key fails it; asserting on a copy of the list kept in the test would have gated nothing. Gate: test_ltx2_video 32 cases / 520 assertions SUCCESS (was 30 cases), and test_ltx2_pipeline 37 cases / 2388 assertions SUCCESS. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- docs/FEATURES.md | 2 + docs/USAGE.md | 16 +-- examples/ltx2_gen/main.cpp | 3 + .../model_executor/models/ltx2_pipeline.h | 54 +++++++-- include/vllm/multimodal/ltx2_video.h | 7 +- .../model_executor/models/ltx2_pipeline.cpp | 48 +++++--- src/vllm/multimodal/ltx2_video.cpp | 51 +++++++-- tests/vllm/models/test_ltx2_pipeline.cpp | 49 ++++++-- tests/vllm/multimodal/test_ltx2_video.cpp | 105 ++++++++++++++++++ 9 files changed, 287 insertions(+), 48 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3c7ba3e68..a8a6beca6 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -324,6 +324,8 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the |---|---|---| | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | +| LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | +| LTX-2.5 out-of-scope arms | Declared, not requestable | Temporal x2 upsampler, `BetaScheduler`, LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU. `multishot` was RETIRED: no such entry point exists in either reference | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started | diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..fe0dd1378 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1650,14 +1650,16 @@ knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` (default `distilled_two_stage`), `model_version` (only for a checkpoint that -declares none), `dit_config_path`, `allow_unported_modules`, `max_phase`, -`prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a -family does not define is refused, never ignored. One caveat inside that set: -`duration_head_path` is accepted but INERT — the duration head is ported and gated -as a brick, nothing in the video engine constructs one, and no code reads that -key, so supplying it neither loads a head nor enables an AUTO duration. Give +declares none), `dit_config_path`, `encoder_config_path`, +`allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, +`upsampler_path` and `duration_head_path`. An extra a family does not define is +refused, never ignored. One caveat inside that set: `duration_head_path` is +defined but UNSERVED — the duration head is ported and gated as a brick, and +nothing in the video engine constructs one — so supplying it is **refused by +name** at load rather than accepted. It used to be accepted and read by nothing, +which silently substituted the recipe default for the file you named. Give `num_frames` (or `duration`, which is exact arithmetic against the recipe's frame -rate) instead. +rate) instead. Every other key in that list reaches a reader. `prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real tokens; absent, every row is. It matters because the embeddings connector diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index e17bb6953..3ec787a3e 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -168,6 +168,9 @@ int main(int argc, char** argv) { else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + // Kept although the library REFUSES this extra by name (#611): the duration + // head is unported, and forwarding the flag gets the caller that named + // refusal instead of "unknown option", which says nothing about why. else if (f == "--duration-head") SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str())); else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str())); diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 264668416..6eb51b670 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -571,20 +571,56 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, const std::string& model_version); // --------------------------------------------------------------------------- -// Out of scope for L5, refused by name (spec section 2, "Out") +// Out of scope, refused by name (spec section 2 "Out"; the 2026-08-13 grounding +// pass is .agents/specs/ltx25-retire-dead-arms.md, row LTX25-RETIRE-DEAD-ARMS) // --------------------------------------------------------------------------- // Each of these renders something plausible if it is silently downgraded, which // is why none of them falls back. `Ltx2RefuseUnportedPipelineFeature` throws with -// a message naming the missing piece and the phase or row that owes it. +// a message naming the missing piece and the row that owes it. +// +// TWO KINDS live here, and conflating them overstated what this port refuses: +// +// REACHABLE REFUSAL — a product path constructs the condition and throws, so a +// caller CAN trip it. `kTemporalUpsampler` (ltx2_upsampler.cpp:395) and +// `kBetaScheduler` (ltx2_pipeline.cpp:199) are the two. +// +// DECLARED-OUT-OF-SCOPE MARKER — no request field, load extra or CLI flag asks +// for it, so nothing outside the ledger test reaches it. It is a record of what +// upstream HAS and this port does NOT, which is worth keeping; calling it a +// refusal is what was wrong. `kLoraFusion`, `kInt8ConvRot` and +// `kMultiGpuParallelism` are markers, and their messages say so. +// +// TWO ENUMERATORS WERE RETIRED on 2026-08-13, recorded here because the +// retirement IS the record — a reader who finds them in git history needs to know +// they did not simply move: +// +// `kMultishot` — FABRICATED. It refused "multishot generation" and cited +// "ltx-pipelines multishot entry points". No such entry point, symbol or string +// exists in Lightricks/LTX-2 @ fd4ded7f or huggingface/diffusers @ 3a2f35d4. +// Searched as a SUBJECT rather than by our own phrasing: upstream's only sense +// of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration"; +// README.md:136 "a cinematographer describing a shot list"), and the only +// `scene` hit is PySceneDetect in the TRAINER. A defect in our record is not a +// gap in our port, so there was nothing to owe. +// +// `kVideoEngineWiring` — LANDED. It said the end-to-end composition through +// `vllm::multimodal::VideoEngine` "is phase L7, not L5"; L7 shipped in +// `cefacd2d0`. A refusal whose subject shipped is a false statement. enum class Ltx2UnportedPipelineFeature { - kTemporalUpsampler, // model/upsampler with temporal_upsample=True - kLoraFusion, // loader/LoraPathStrengthAndSDOps - kMultishot, // ltx-pipelines multishot entry points - kInt8ConvRot, // ComfyUI-only quantization - kCfgParallelism, // ltx-pipelines/multigpu - kVideoEngineWiring, // end-to-end through vllm::multimodal::VideoEngine (L7) - kBetaScheduler, // components/schedulers.py:91-120 + // Reachable refusals. + kTemporalUpsampler, // model/upsampler with temporal_upsample=True + kBetaScheduler, // components/schedulers.py:91-120 + // Declared-out-of-scope markers. + kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps), + // fused by loader/fuse_loras.py + kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four + // kinds upstream defines are fp8-cast / fp8-scaled-mm / + // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-27), + // int8 appears only in the TRAINER, and convrot nowhere at all + kMultiGpuParallelism, // ltx-pipelines/multigpu — sequence-parallel, tiled data + // parallel and distributed VAE decode. NOT CFG batching: + // zero `cfg` hits in either multigpu tree }; [[noreturn]] void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature); diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 0953aebff..11c4045b9 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -9,10 +9,13 @@ // Phases L2-L6 shipped a DiT forward, a text feature extractor, two VAEs, a // vocoder, an upsampler, a duration head, and a pipeline COMPONENT library — // schedules, noisers, steppers, guiders, patchifiers, recipes. Nothing drove -// them: `Ltx2RefuseUnportedPipelineFeature(kVideoEngineWiring)` refused the +// them: an `Ltx2UnportedPipelineFeature::kVideoEngineWiring` refusal named the // composition BY NAME and named this phase as its owner. This TU is that // composition and nothing else. It adds no numerics; every line either resolves -// a parameter, moves a buffer, or calls a brick that already has a golden. +// a parameter, moves a buffer, or calls a brick that already has a golden. (That +// enumerator was RETIRED in row LTX25-RETIRE-DEAD-ARMS once L7 landed in +// `cefacd2d0` — a refusal whose subject shipped is a false statement, not a +// record of debt. It is named here in the past tense on purpose.) // // ─── WHAT IT IS A PORT OF (file:line on BOTH sides) ────────────────────────── // Upstream: Lightricks/LTX-2 @ fd4ded7, packages/ltx-pipelines/src/ltx_pipelines/ diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index e8d61fa1d..4a28173a4 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1138,30 +1138,50 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { const std::string owed = - " Not ported by phase L5; recorded as owed in .agents/specs/ltx-2-5.md."; + " Recorded as owed in .agents/specs/ltx-2-5.md; grounded against Lightricks/LTX-2 " + "fd4ded7f in .agents/specs/ltx25-retire-dead-arms.md."; + // A marker is not a refusal a caller can trip, and saying so is the point: this + // enum used to read as six live refusals when only two had a product call site. + const std::string marker = + " DECLARED, NOT REQUESTABLE: no request field or load extra asks for this, so nothing " + "but the out-of-scope ledger reaches this message."; switch (feature) { case Ltx2UnportedPipelineFeature::kTemporalUpsampler: Refuse("ltx2: the temporal x2 latent upsampler (model/upsampler/model.py:55-72, " "temporal_upsample=True) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kLoraFusion: - Refuse("ltx2: LoRA fusion (loader/LoraPathStrengthAndSDOps) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kMultishot: - Refuse("ltx2: multishot generation is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kInt8ConvRot: - Refuse("ltx2: the int8-convrot quantization (ComfyUI-only) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kCfgParallelism: - Refuse("ltx2: CFG / multi-GPU parallelism (ltx-pipelines/multigpu) is out of scope." + - owed); - case Ltx2UnportedPipelineFeature::kVideoEngineWiring: - Refuse("ltx2: end-to-end wiring through vllm::multimodal::VideoEngine is phase L7, not " - "L5." + - owed); case Ltx2UnportedPipelineFeature::kBetaScheduler: Refuse("ltx2: BetaScheduler (components/schedulers.py:91-120) is not ported. It inverts " "a Beta CDF through scipy.stats.beta.ppf, and no ltx-pipelines entry point " "constructs it." + owed); + case Ltx2UnportedPipelineFeature::kLoraFusion: + Refuse("ltx2: LoRA fusion (ltx-core loader/primitives.py:160 LoraPathStrengthAndSDOps, " + "fused by loader/fuse_loras.py) is out of scope." + + marker + owed); + case Ltx2UnportedPipelineFeature::kInt8ConvRot: + // VERIFIED ABSENT so nobody re-audits it: LTX-2 @ fd4ded7f defines exactly four + // inference quantization kinds and none is int8, `convrot` is 0 hits, and every + // upstream `int8` is either the TRAINER's bitsandbytes / quanto path or a uint8 + // buffer. This is a ComfyUI-ecosystem format, deliberately not ported. + Refuse("ltx2: the int8-convrot quantization is out of scope. It is a ComfyUI-ecosystem " + "format, not an LTX-2 arm: upstream's own kinds are fp8-cast, fp8-scaled-mm, " + "nvfp4-cast and nvfp4-prequant (ltx-pipelines/utils/quantization_factory.py:" + "23-27), and int8 appears upstream only in the trainer." + + marker + owed); + case Ltx2UnportedPipelineFeature::kMultiGpuParallelism: + // The old spelling was `kCfgParallelism`, which named something upstream does + // not do. There is no CFG pass to split here in the first place: the distilled + // recipe denoises with SimpleDenoiser at both stages (distilled.py:266,295), + // "single transformer call, no guidance" (utils/denoisers.py:3). + Refuse("ltx2: single-node multi-GPU parallelism (ltx-pipelines/multigpu) is out of " + "scope. Upstream has three forms and none of them is CFG batching: " + "sequence-parallel (multigpu/sp_builder.py:25), tiled data parallel " + "(multigpu/tdp_builder.py:25, upscale stage only) and distributed VAE decode " + "(ltx-core multigpu/vae/distributed_decoder.py:204-256). It is a LATENCY tool, " + "not a memory tool (docs/multigpu/README.md:5-16), and this port targets one " + "GB10." + + marker + owed); } Refuse("ltx2: unknown unported pipeline feature." + owed); } diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index a50f184c0..54221f546 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -246,15 +246,30 @@ int64_t ExtraInt(const std::map& extras, const std::st } } +// The one key this family DEFINES and does not SERVE. `Ltx2DurationPredict` is +// ported and gated as a brick (`ltx2_duration_head.h`), but nothing here +// constructs one, so a supplied path names a file the engine never opens. +constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; + // Every extra key this family DEFINES. An extra outside this set is refused // rather than ignored, for the same reason H3 refuses one // (minimax_h3_video.cpp): a mistyped knob that is silently dropped renders the // DEFAULT and looks like the feature not working. +// +// DEFINED IS NOT THE SAME AS SERVED, and conflating the two was #611: nine of +// these ten reach a reader (`:570`, `:625`, `:721`, `:737`, `:739`, `:771`, +// `:796`, `:901`, `:942`), and `duration_head_path` reached none, so supplying a +// duration head substituted the recipe default in silence — the failure mode this +// very list exists to prevent, one level in. It stays in the list because the +// family DOES define the key and DOES know what it means; `CheckUnservedExtras` +// refuses it by name instead, which is a different and truer message than +// "unknown load extra". The full audit is in +// .agents/specs/ltx25-retire-dead-arms.md §2.1. const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, - "upsampler_path", "duration_head_path", + "upsampler_path", kLtx2DurationHeadPathExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -293,6 +308,27 @@ void CheckKnownExtras(const std::map& extras) { } } +// A key this family DEFINES but does not SERVE, refused BY NAME when supplied +// (#611). The alternative — accepting it — is the worst of the three options: +// worse than refusing, and worse than not defining the key, because the caller +// pointed at a specific file and got the recipe default with no diagnostic. +// +// Deliberately NOT the "unknown load extra" path above. That message says the +// family does not define the key, which is false here and would send the reader +// looking for a typo instead of for the unported head. +void CheckUnservedExtras(const std::map& extras) { + const std::string duration_head = VideoExtra(extras, kLtx2DurationHeadPathExtra); + if (!duration_head.empty()) { + Fail("the '" + std::string(kLtx2DurationHeadPathExtra) + "' extra names '" + duration_head + + "', but the duration head is NOT WIRED into this engine: `Ltx2DurationPredict` is ported " + "and gated as a brick (ltx2_duration_head.h, upstream duration_head.py:89-118) and " + "nothing here constructs one, so that file would never be opened and an AUTO duration " + "would fall back to the recipe default. Give 'num_frames', or 'duration' (exact " + "arithmetic against the recipe frame rate), instead. Refused rather than ignored; " + "recorded as owed in .agents/specs/ltx25-retire-dead-arms.md (#611)."); + } +} + // `detect_model_version` normalizes the separator before parsing // (utils/constants.py:161), and the recipe table is keyed on the two-component // spelling ("2.5"), not on the checkpoint's three-component "2.5.0". Reduce it @@ -509,6 +545,7 @@ Ltx2ConditioningTrace Ltx2VideoEngine::last_conditioning() const { std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& params) { if (params.dit_path.empty()) Fail("dit_path is required"); CheckKnownExtras(params.extras); + CheckUnservedExtras(params.extras); auto engine = std::unique_ptr(new Ltx2VideoEngine()); engine->impl_ = std::make_unique(); @@ -1146,12 +1183,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // needs an encoded prompt this engine cannot produce", and since `has_encoder` // above the engine produces exactly that. What is missing now is the head // itself — `ltx2_duration_head.h` is ported and gated as a brick, but nothing - // here constructs one, and `duration_head_path` is accepted in - // `kKnownLoadExtras` while NO code reads it (grep: it appears at that one - // site). So the extra is inert rather than wired, and that is recorded as owed - // rather than left to be discovered by someone who supplies it and gets the - // recipe default. An explicit duration is exact arithmetic, so it is served; - // the AUTO path is what is missing, and `num_frames` is how to avoid it. + // here constructs one. `duration_head_path` used to be ACCEPTED while no code + // read it, so a caller who supplied a head silently landed on this line + // instead; `CheckUnservedExtras` now refuses that key by name at load (#611, + // .agents/specs/ltx25-retire-dead-arms.md §2). What remains owed is the head + // itself. An explicit duration is exact arithmetic, so it is served; the AUTO + // path is what is missing, and `num_frames` is how to avoid it. frames = static_cast(std::llround(gen.duration_seconds * fps)); } if (frames < 1) Fail("num_frames resolved to " + std::to_string(frames)); diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 11996d926..274c4fd76 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1231,25 +1231,56 @@ TEST_CASE("ltx2 the two references disagree on the default negative prompt, and // Out-of-scope refusals // =========================================================================== -TEST_CASE("ltx2 every L5 out-of-scope feature is refused BY NAME") { - // Spec section 2 "Out", plus the L7 boundary. A silent downgrade of any of - // these produces a video, which is exactly why none of them may fall back. - const std::vector> owed = { +// THE LIST SHRANK FROM SEVEN TO FIVE on 2026-08-13, and the shrink is the point. +// Two enumerators were retired by row LTX25-RETIRE-DEAD-ARMS (#644): +// `kMultishot`, which refused a feature that exists in NEITHER reference, and +// `kVideoEngineWiring`, whose subject shipped in `cefacd2d0`. See +// .agents/specs/ltx25-retire-dead-arms.md §1.1 and §1.5. A CHANGED CASE COUNT +// here is that retirement, not a lost assertion. +TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { + // Spec section 2 "Out". A silent downgrade of any of these produces a video, + // which is exactly why none of them may fall back. + // + // REACHABLE REFUSALS: a product path constructs the condition, so a caller can + // trip these. The call sites are `ltx2_upsampler.cpp:395` and + // `ltx2_pipeline.cpp:199`. + const std::vector> reachable = { {vllm::Ltx2UnportedPipelineFeature::kTemporalUpsampler, "temporal"}, + {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, + }; + // DECLARED-OUT-OF-SCOPE MARKERS: nothing a caller can send reaches these, so + // the message must not claim otherwise. Recording them as refusals overstated + // what this port has, which is the defect this row closes. + const std::vector> markers = { {vllm::Ltx2UnportedPipelineFeature::kLoraFusion, "LoRA"}, - {vllm::Ltx2UnportedPipelineFeature::kMultishot, "multishot"}, {vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot, "int8-convrot"}, - {vllm::Ltx2UnportedPipelineFeature::kCfgParallelism, "parallelism"}, - {vllm::Ltx2UnportedPipelineFeature::kVideoEngineWiring, "VideoEngine"}, - {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, + {vllm::Ltx2UnportedPipelineFeature::kMultiGpuParallelism, "multi-GPU"}, }; - for (const auto& item : owed) { + + std::vector> all = reachable; + all.insert(all.end(), markers.begin(), markers.end()); + for (const auto& item : all) { const std::string message = RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); INFO("feature = ", item.second, " refusal = ", message); CHECK(Mentions(message, item.second)); // Naming WHERE the work is owed is what keeps it from being rediscovered. CHECK(Mentions(message, "ltx-2-5.md")); + // The RETIRED arm must not come back: a refusal that cites a feature neither + // reference has sends the next reader looking upstream for it. + CHECK_FALSE(Mentions(message, "multishot")); + } + for (const auto& item : reachable) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("reachable = ", item.second, " refusal = ", message); + CHECK_FALSE(Mentions(message, "DECLARED, NOT REQUESTABLE")); + } + for (const auto& item : markers) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("marker = ", item.second, " refusal = ", message); + CHECK(Mentions(message, "DECLARED, NOT REQUESTABLE")); } } diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 85b47025a..c81fe5b55 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -476,6 +476,111 @@ TEST_CASE("ltx2 video: an unknown extra is refused, not ignored") { } } +// A key this family DEFINES but does not serve is the worse half of the same +// defect, and the one an "unknown extra" check cannot see. `duration_head_path` +// was in `kKnownLoadExtras` and read by NOTHING (#611): supplying a duration head +// loaded no head, opened no file, and handed back the recipe default with no +// diagnostic. AGENTS.md requires an unimplemented arm to be refused with a +// message naming the missing piece, so it is refused rather than accepted. +// +// Dropping the key from `kKnownLoadExtras` instead would produce "unknown load +// extra", which is a DIFFERENT and wrong claim — the family defines the key and +// understands what it means; what is missing is the head. Hence the assertion on +// the missing piece and the alternative, not only on the key. +TEST_CASE("ltx2 video: duration_head_path is REFUSED by name, not silently ignored") { + Workspace ws; + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + // Any path at all: the point is that NOTHING opens it. Naming a file that does + // exist keeps a not-found error from standing in for the refusal. + mp.extras["duration_head_path"] = ws.paths.dit; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("duration_head_path is served by no code; accepting it substitutes the recipe default"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("duration_head_path") != std::string::npos); + // The MISSING PIECE, which is what separates this from "unknown key". + CHECK(msg.find("duration head") != std::string::npos); + // And what to use instead, so the refusal is actionable. + CHECK(msg.find("num_frames") != std::string::npos); + // Not the unknown-key message: that one would say the family does not define + // it, and this family does. + CHECK(msg.find("unknown load extra") == std::string::npos); + } +} + +// The INVENTORY, so the defect above cannot come back as a different key. Every +// extra this family accepts is either read by something or refused by name; a +// tenth decorative key fails this case rather than waiting to be discovered by +// the caller who supplies it. +// +// The audit behind it is in .agents/specs/ltx25-retire-dead-arms.md §2.1: nine of +// the ten keys have a reader (`ltx2_video.cpp:570,625,721,737,739,771,796,901,942`), +// and `duration_head_path` was the only one with none. +TEST_CASE("ltx2 video: every accepted load extra is READ by something") { + Workspace ws; + // The keys with a reader. + const std::vector served = { + vllm::multimodal::kLtx2AudioPromptEmbedsExtra, vllm::multimodal::kLtx2PipelineKindExtra, + vllm::multimodal::kLtx2ModelVersionExtra, vllm::multimodal::kLtx2AllowUnportedExtra, + vllm::multimodal::kLtx2MaxPhaseExtra, vllm::multimodal::kLtx2DitConfigPathExtra, + vllm::multimodal::kLtx2PromptValidRowsExtra, vllm::multimodal::kLtx2EncoderConfigPathExtra, + "upsampler_path", + }; + // The keys the family defines and does NOT serve. Growing this list is a + // deliberate act; growing it silently is the defect #611 records. + const std::vector refused = {"duration_head_path"}; + + // THE HANDLE ON THE REAL ARRAY. The unknown-extra refusal builds its listing + // from `kKnownLoadExtras` itself, so parsing that listing gates the ACTUAL + // accepted set rather than a copy of it maintained here. Without this the two + // vectors above would be true by construction and would gate nothing. + std::string listing; + { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["definitely_not_a_key"] = "1"; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an unknown extra must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + const size_t at = msg.find("This family defines: "); + REQUIRE(at != std::string::npos); + listing = msg.substr(at + std::string("This family defines: ").size()); + } + } + INFO("listing = " << listing); + // Every name this row inventoried is still accepted... + for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); + for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); + // ...and there is no ELEVENTH name that this inventory has never seen. The + // separator is ", ", so the count is one more than the separators. + size_t names = 1; + for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { + ++names; + } + CHECK_MESSAGE(names == served.size() + refused.size(), + "kKnownLoadExtras grew; add the key to `served` (with its reader) or to " + "`refused` (with a by-name refusal), per .agents/specs/ltx25-retire-dead-arms.md"); + + // And the unserved half is refused rather than accepted. + for (const std::string& key : refused) { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[key] = ws.paths.dit; + INFO("key = " << key); + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an accepted-but-unread extra must be refused by name"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find(key) != std::string::npos); + CHECK(msg.find("unknown load extra") == std::string::npos); + } + } +} + // ─── the config the SHAPES cannot see ─────────────────────────────────────── // // WHY THESE ASSERT ON THE ENGINE AND NOT ON A LOCAL. The L7 repair is one line in From 4e5be2585853687a6efbf7c69f4fda99e7836b74 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 19:45:29 +0000 Subject: [PATCH 3/6] spec(LTX25-RETIRE-DEAD-ARMS): the diffusers half of the multishot check (#644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §1.1 argued the retirement mostly from Lightricks/LTX-2, where "shot" is rare. diffusers @ 3a2f35d4 is the stronger check because the word is COMMON there and still never means a generation mode: every `shot` under src/diffusers/pipelines/ltx2/ is a camera shot TYPE in the prompt-enhancement guidance (utils.py:217, "extreme wide shot / wide shot / medium shot / ...") or the duration head's own docstring (duration_head.py:83, "the natural duration of the shot implied by a caption"). Its sixteen ltx2 modules are enumerated; none is multi-shot. Records only; no code, no gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/ltx25-retire-dead-arms.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md index 3c4879fda..27fc34d4d 100644 --- a/.agents/specs/ltx25-retire-dead-arms.md +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -68,6 +68,16 @@ The five `shot` files, with what the word means in each: Upstream's "shot" is a single continuous take. There is no multi-shot generation mode, and nothing that composes several takes into one output. +diffusers says the same thing in its own vocabulary, which is the stronger check +because the word is far more common there: every `shot` in +`src/diffusers/pipelines/ltx2/` is a **camera shot type** in the prompt-enhancement +guidance — `utils.py:217`, "Shot type (exactly one: extreme wide shot / wide shot +/ medium shot / medium close-up / close-up / extreme close-up)" — or the duration +head's own docstring, `duration_head.py:83`, "Predicts the natural duration of the +**shot** implied by a caption". Its sixteen `ltx2/` modules are +`pipeline_ltx2`, `_condition`, `_diffusion_decode`, `_hdr_lora`, `_ic_lora`, +`_image2video`, `_latent_upsample`, plus the components. None is multi-shot. + `ltx-pipelines`' actual entry points, from `docs/pipelines.md` ("Full reference for all 11 pipelines") and the module list: `ti2vid_one_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, From 2d1d59d9feb94f324e1de12d0df90a66cbeac71b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:42:08 +0000 Subject: [PATCH 4/6] fix(LTX25-RETIRE-DEAD-ARMS): repair the evidence, including the false absence this row exists to retire (#644, #611) Review of `4e5be2585` returned FAIL with six findings, every one of them evidence accuracy rather than logic. No disposition moved. The two that matter most are instances of #604 -- an absence asserted from a grep of our own vocabulary -- committed inside the row whose whole purpose is to close them. F1 -- a FALSE ABSENCE shipped in a user-visible refusal. The `kInt8ConvRot` message ended "int8 appears upstream only in the trainer". Re-derived at Lightricks/LTX-2 `fd4ded7f`: `ltx-kernels`, an INFERENCE package, carries a per-row int8 quantize kernel with fp32 scales -- `blockwise/triton_ops.py:25-50`, output dtype `torch.int8` at `:43`, aliased `rowwise_int_quantize_triton` at `:436`. It is dead (that alias is its only reference; `blockwise/functional.py: 12-18` re-exports five other names and not this one), which is why the disposition stands and only the sentence was wrong. Context: the package is a fork of Lightricks' int8 kernel library retargeted to fp8/fp6/nvfp4 -- its custom-op namespace is still literally `q8_kernels_ops` (`functional.py:25`). The message now says int8 is UNREACHABLE, names the trainer paths AND the dead kernel, and corrects `quantization_factory.py:23-27` to `:23-26`. F2 -- the spec asserted an absence that was false, on the one search term that mattered. Section 1.2 listed `hadamard` among "0 hits" and concluded "no rotation of any kind". LTX-2 vendors Tri Dao's fast-Hadamard-transform family in `ltx-kernels/csrc/ops/include/`, explicitly quantization-coupled: `DequantHadamardParamsBase:54`, `QuantHadamardParamsBase:71`, `QuantMax` at `fast_hadamard_transform_common.h:14`. A Hadamard transform IS the "rot" of QuaRot/SpinQuant. The conclusion survives on different evidence, re-derived here: NOTHING calls it -- `hadamard_mult_thread`/`_warp` have no call site in `csrc/`, `fast_hadamard_transform_cuda` is commented out (`rms_norm_rope_cuda.cu:225-228`), and the only live struct, `NormRopeHadamardParamsBase`, drives `norm_rope_cvt_kernel` (`rms_norm_rope_cuda.cu:51-170`), which is RMS-norm -> RoPE -> store with no rotation at all; its instantiations are bf16 and `Float8_e4m3fn` only (`:219-223`). `convrot`/`conv_rot`/`quarot`/`spinquant` are genuinely 0 hits. F3 -- nine reader anchors were stale, cited in the file they were wrong about: `ltx2_video.cpp:259-261` pointed at `:570` etc., where `:570` was `if (backend == nullptr)`. Correcting them, the merge of `origin/main` below moved the real ones AGAIN. So they are no longer written by hand. The anchors live in one `READER ANCHORS` comment and `test_ltx2_video` derives them from the source on every run, printing the replacement list when they drift. It hard-codes no line number itself, so it cannot rot, and the only obligation it creates is on whoever moves a reader, in the file they are already editing. F4 -- `docs/FEATURES.md` re-merged the split this row had just made, labelling all five arms "Declared, not requestable" including `kTemporalUpsampler` and `kBetaScheduler`, the two a caller CAN trip. Split into two rows, gated by a new case that also requires exactly one such row to exist so it cannot pass vacuously. F5 and F6 are filed, not fixed here: the reachable/marker split gates message TEXT and not reachability, and `.agents/specs/ltx-2-5.md:291` still owes `multishot` but is operator-owned. Also merged `origin/main` (`11cc1d589` .. `7965f12bf`), which clears the inherited `device-leakage` red on this PR -- the two `kcuda` hits were in `ltx2_video.cpp:532,549` and were fixed on main after this branch was cut. `.agents/roadmap_v1.md` conflicted add/add and was resolved by taking the target version and reapplying the `#611` row by key; no duplicate row was produced. RED before GREEN, per finding: F1 `test_ltx2_pipeline.cpp:1303,1306` failed against the shipped message. F3 `Recorded: []. Actual: [638 693 789 805 807 839 864 969 1010]`. F4 both reachable arms found on `FEATURES.md:328`. test_ltx2_video 32/520 -> 33/541 SUCCESS (exit 0) test_ltx2_pipeline 37/2388 -> 38/2395 SUCCESS (exit 0) F2 is prose in a spec about an upstream checkout; no test in this tree can carry it, and it is stated as re-derived evidence instead. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code] --- .agents/specs/ltx25-retire-dead-arms.md | 160 ++++++++++++++--- docs/FEATURES.md | 3 +- docs/USAGE.md | 10 ++ .../model_executor/models/ltx2_pipeline.h | 8 +- .../model_executor/models/ltx2_pipeline.cpp | 27 ++- src/vllm/multimodal/ltx2_video.cpp | 10 +- tests/CMakeLists.txt | 13 ++ tests/vllm/models/test_ltx2_pipeline.cpp | 79 +++++++++ tests/vllm/multimodal/test_ltx2_video.cpp | 161 +++++++++++++++++- 9 files changed, 428 insertions(+), 43 deletions(-) diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md index 27fc34d4d..315f87e3a 100644 --- a/.agents/specs/ltx25-retire-dead-arms.md +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -105,28 +105,72 @@ class QuantizationKind(str, Enum): NVFP4_PREQUANT = "nvfp4-prequant" ``` -No int8 arm, and no rotation of any kind: `convrot` / `conv_rot` / `hadamard` / -`quarot` / `spinquant` are 0 hits across the repository. `rotation` hits only EXIF -image orientation (`media_io/decode.py:32`) and the VAE's per-slab spatial -rotation (`video_vae/transformer/det_attn_rope.py:61`) — neither a quantization -transform. - -`int8` upstream is **training-only**, in two places, and neither is an inference -weight format: +No int8 arm. `convrot` / `conv_rot` / `quarot` / `spinquant` are 0 hits across the +repository, re-run at these pins. + +**Correction, 2026-08-13 (review finding F2).** An earlier revision of this +section put `hadamard` in that 0-hit list and concluded "no rotation of any kind". +That was itself the #604 pattern this row exists to close — an absence asserted +from a grep of our own vocabulary — and it is false. LTX-2 vendors Tri Dao's +fast-Hadamard-transform family in `ltx-kernels` +(`csrc/ops/include/fast_hadamard_transform.h`, `_common.h`, `_special.h`), and it +is explicitly **quantization-coupled**: `DequantHadamardParamsBase:54`, +`QuantHadamardParamsBase:71`, and `QuantMax = 127.0` +(`fast_hadamard_transform_common.h:14`). A fast Hadamard transform *is* the "rot" +of rotation-based quantization, so the word we should have searched for was there +all along. + +What the conclusion rests on instead, re-derived here: + +- **Nothing calls it.** `hadamard_mult_thread` / `hadamard_mult_warp` are defined + at `fast_hadamard_transform_common.h:102,123` and `_special.h:12,31,58,93` and + invoked at **no site in `csrc/`**. The `fast_hadamard_transform_cuda` + instantiations are commented out (`rms_norm_rope_cuda.cu:225-228`), and + `DequantHadamardParamsBase` / `QuantHadamardParamsBase` / `QuantMax` are + declared and referenced nowhere. +- **The live kernel that carries the Hadamard NAME does not rotate.** The only + params struct in use is `NormRopeHadamardParamsBase` + (`rms_norm_rope.cpp:12,19,95-98`, `rms_norm_rope_cuda.cu:188`), and + `norm_rope_cvt_kernel` (`rms_norm_rope_cuda.cu:51-170`) is RMS-norm → RoPE → + store, with no Hadamard step. The name is vestigial, inherited from the fork's + ancestry. +- **And where it is instantiated it is fp8 or bf16, never int8**: + `` and `` + (`rms_norm_rope_cuda.cu:219-223`), matching `rms_norm_split_rope_cuda.cu:196,200`. + +Corrected statement, same conclusion: **upstream vendors a +quantization-coupled Hadamard family, wires none of it, and never exposes a +rotation as a `QuantizationKind`.** So `int8-convrot` is still unreachable +upstream — but by dead code, not by absence. + +`int8` upstream is **unreachable, not absent**. Everything *wired* is trainer-only: - `ltx-trainer/src/ltx_trainer/gemma_8bit.py:33-36` — bitsandbytes `LLM.int8()` for the Gemma backbone during LoRA training. - `ltx-trainer/src/ltx_trainer/quantization.py:11-15` — optimum-quanto precisions (`int8-quanto`, `int4-quanto`, …) for the trainer. +**But one int8 kernel lives in an INFERENCE package** (review finding F1, and the +same false-absence pattern shipped inside the user-visible refusal message): +`ltx-kernels/src/ltx_kernels/blockwise/triton_ops.py` defines `_kernel` at +`:25-37` and `run_quantize_kernel` at `:40-50` — a per-row int8 quantize with fp32 +scales, output dtype `torch.int8` at `:43`, not a `uint8` buffer — aliased +`rowwise_int_quantize_triton` at `:436`. It is **dead**: that alias is its only +reference anywhere in the repository, and `blockwise/functional.py:12-18` +re-exports five other names and not this one. Context for why it is there at all: +the package is a fork of Lightricks' int8 kernel library retargeted to +fp8/fp6/nvfp4, and its custom-op namespace is still literally `q8_kernels_ops` +(`functional.py:25`). + Every other `int8` match in the repository is `uint8`: pixel buffers, packed NVFP4 nibbles, block-streaming staging. -**Disposition: KEEP, re-anchored.** The refusal message is honest — it is a -ComfyUI-ecosystem quantization, not an LTX-2 arm — but the enum comment did not -say it had been checked. It now records the absence at these pins so nobody -re-audits it. Its *kind* changes: it is a declared-out-of-scope marker, not a -reachable refusal. +**Disposition: KEEP, re-anchored.** Unchanged by the two corrections above — +nothing wired upstream reaches int8, and no rotation is exposed as a quantization +kind. What changes is the sentence: the refusal now says int8 is UNREACHABLE +(trainer-only for anything wired, plus one dead kernel in `ltx-kernels`) instead +of claiming it appears "only in the trainer". Its *kind* also changes: it is a +declared-out-of-scope marker, not a reachable refusal. ### 1.3 CFG parallelism — the name describes something upstream does not do @@ -159,6 +203,14 @@ documented as "**single transformer call, no guidance**" unchanged". A `cfg_scale` of 1.0 is one pass, so there is no second pass to place on a second GPU. +Stronger still, and true even for the recipes that *do* guide: upstream's guided +path is not distributed either. `utils/denoisers.py:7-8` records that +`GuidedDenoiser` and `FactoryGuidedDenoiser` share `_guided_denoise`, "which +batches all guidance passes into a single transformer call". So upstream's CFG is +a batch dimension, never a second device — which is why no amount of `multigpu` +code was ever going to be CFG parallelism, and why the old enumerator name could +not have been right for any recipe. + **Disposition: RENAME + re-anchor.** `kCfgParallelism` → `kMultiGpuParallelism`, anchored to the three real forms, with the reason it is out of scope stated as what it is: a single-node multi-GPU **latency** feature, on a port whose target is @@ -213,25 +265,37 @@ is calling it a refusal. The header and the messages now distinguish: ### 2.1 The full `kKnownLoadExtras` audit Every key the family accepts, and whether any code reads it. Reader anchors are in -`src/vllm/multimodal/ltx2_video.cpp` unless noted. +`src/vllm/multimodal/ltx2_video.cpp` unless noted, **as of `3c6706cd0`** — this +table is a dated record, not a live one; see the note below it. | Key | Constant | Reader | Status | |---|---|---|---| -| `audio_prompt_embeds_path` | `kLtx2AudioPromptEmbedsExtra` | `:901`, `:914` | READ | -| `pipeline_kind` | `kLtx2PipelineKindExtra` | `:737` | READ | -| `model_version` | `kLtx2ModelVersionExtra` | `:721` | READ | -| `allow_unported_modules` | `kLtx2AllowUnportedExtra` | `:570` | READ | -| `max_phase` | `kLtx2MaxPhaseExtra` | `:739` | READ | -| `dit_config_path` | `kLtx2DitConfigPathExtra` | `:625` | READ | -| `prompt_embeds_valid_rows` | `kLtx2PromptValidRowsExtra` | `:942` | READ | -| `encoder_config_path` | `kLtx2EncoderConfigPathExtra` | `:796` | READ | -| `upsampler_path` | (literal) | `:771` | READ | -| `duration_head_path` | (literal) | **none** | **ACCEPTED AND IGNORED** | +| `audio_prompt_embeds_path` | `kLtx2AudioPromptEmbedsExtra` | `:969`, `:982` | READ | +| `pipeline_kind` | `kLtx2PipelineKindExtra` | `:805` | READ | +| `model_version` | `kLtx2ModelVersionExtra` | `:789` | READ | +| `allow_unported_modules` | `kLtx2AllowUnportedExtra` | `:638` | READ | +| `max_phase` | `kLtx2MaxPhaseExtra` | `:807` | READ | +| `dit_config_path` | `kLtx2DitConfigPathExtra` | `:693` | READ | +| `prompt_embeds_valid_rows` | `kLtx2PromptValidRowsExtra` | `:1010` | READ | +| `encoder_config_path` | `kLtx2EncoderConfigPathExtra` | `:864` | READ | +| `upsampler_path` | (literal) | `:839` | READ | +| `duration_head_path` | `kLtx2DurationHeadPathExtra` | **none** | **was ACCEPTED AND IGNORED; now refused by name** | Nine of ten are wired. `duration_head_path` is the only defect, so the sweep this row owes is complete and closes the "the sweep that found this one did not cover them all" clause of #611. +**These numbers rotted twice inside this one PR (review finding F3).** The first +set shipped off by 37 lines and named nine lines that read nothing — cited, worst +of all, *in the file they were wrong about*. Correcting them, a merge of +`origin/main` moved the real ones by another 25. A hand-written `file:line` is +stale by the next commit, so the anchors are no longer maintained by hand: the +`READER ANCHORS` comment in `ltx2_video.cpp` carries the one live copy, and +`test_ltx2_video`'s "the recorded reader anchors are the ones in the source" case +derives them from the source on every run and fails with the replacement list +printed. The table above stays because a spec is a dated record and the SHA is +named; the code comment is the live one. + One documentation gap found by the same sweep and fixed here: `docs/USAGE.md:1650-1654` lists the LTX-2.5 extras and omits `encoder_config_path` entirely, though it is defined and read. @@ -279,6 +343,11 @@ is a different row. Refusing is the cheap correct answer until then. the header comment, in the messages, and in the test. 5. Refuse `duration_head_path` by name (#611). 6. `docs/USAGE.md`: the extras paragraph, corrected on both counts. +7. Added by the review repair: correct the int8 sentence in the shipped refusal + (§1.2), correct this spec's own false `hadamard` absence (§1.2), derive the + reader anchors instead of writing them by hand (§2.1), and split the + `docs/FEATURES.md` row so the public surface stops calling the two REACHABLE + arms unrequestable (§1.6). No disposition changes. **Out.** @@ -308,12 +377,33 @@ test's own assertions. either round-trips through a reader or is refused by name. Implemented as the two known-inert keys being refused and the rest being accepted, so adding a tenth decorative key fails. -3. **`ltx2 every L5 out-of-scope feature is refused BY NAME`** (existing, updated). +3. **`ltx2 every out-of-scope feature is refused BY NAME`** (existing, updated). The list drops from 7 to 5 entries — a CHANGED COUNT, reported as such — and splits into `reachable` and `markers`, with the marker messages required to say they are not requestable. Adds a guard that no refusal message mentions `multishot` again. +Three more added by the review repair, each RED before its fix: + +4. **The int8 marker's own evidence** (an addition to case 3). The message shipped + the sentence "int8 appears upstream only in the trainer", which §1.2 shows is + false. The case now requires the message NOT to contain it and to name + `ltx-kernels`. **RED before**: both assertions failed against the shipped text. +5. **`ltx2 video: the recorded reader anchors are the ones in the source`** (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`). Derives each served key's reader + line from `ltx2_video.cpp` itself and compares it with the `READER ANCHORS` + comment, and separately requires every `kLtx2DurationHeadPathExtra` reference + after the array to sit inside `CheckUnservedExtras`. It hard-codes no line + number of its own, so it cannot go stale; the obligation it creates falls on + whoever moves a reader, in the file they are already editing. **RED before**: + `Recorded: []. Actual: [638 693 789 805 807 839 864 969 1010]`. +6. **`ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable`** (new, + `tests/vllm/models/test_ltx2_pipeline.cpp`). The public surface had re-merged + the reachable/marker split into one "Declared, not requestable" row naming the + temporal upsampler and `BetaScheduler`. Anti-vacuous: it also requires exactly + one such LTX-2.5 row to exist, so renaming the row away fails it. **RED + before**: both reachable arms found on `FEATURES.md:328`. + ## 5. Risks | Risk | Mitigation | @@ -338,9 +428,12 @@ none was taken from the prior grounding pass.** Results: - `multishot`: **fabricated, confirmed.** 0 hits for the term in either reference; the only upstream sense of "shot" is one camera take (§1.1). Retired. -- `int8-convrot`: **absent upstream, confirmed.** Four inference quantization kinds - exist and none is int8; int8 is training-only (§1.2). Kept, re-anchored, so the - absence does not have to be re-audited. +- `int8-convrot`: **unreachable upstream, and the first wording of that was wrong.** + Four inference quantization kinds exist and none is int8, and no rotation is + exposed as one — but int8 is not trainer-only (a dead per-row int8 kernel lives + in the `ltx-kernels` inference package) and `hadamard` is not 0 hits (a whole + quantization-coupled Hadamard family is vendored, and wired to nothing). Both + corrections are in §1.2; neither moves the disposition. Kept, re-anchored. - CFG parallelism: **the name was wrong, the exclusion is right.** 0 `cfg` hits in either multi-GPU tree; upstream's own README calls MGPU a latency tool; the distilled recipe runs `SimpleDenoiser` with no guidance at all, so there is no @@ -356,6 +449,17 @@ What this row deliberately did **not** do: construct a duration head. #611 stays open for that, with its user-visible half — silent substitution of the recipe default — closed. +**The review's own finding, recorded because it is the point of the row.** Six +findings came back and every one was evidence accuracy, not logic: a false absence +inside the shipped refusal (§1.2), a false absence in this spec's §1.2 — the one +search term that mattered — nine stale reader anchors cited in the file they were +wrong about (§2.1), and a public-doc row that re-merged the split the ledger had +just made. A row whose subject is #604 committed three instances of #604. That is +not irony to note and move past; it is why the anchors are now derived rather than +written, why the int8 absence is stated as UNREACHABLE with its dead kernel named, +and why the `hadamard` search this spec got wrong is spelled out in full rather +than quietly corrected. + ## Now Row `LTX25-RETIRE-DEAD-ARMS` is `DONE`. The ledger carries five entries, split by diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 45a93a711..1a65b0572 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -325,7 +325,8 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | | LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | -| LTX-2.5 out-of-scope arms | Declared, not requestable | Temporal x2 upsampler, `BetaScheduler`, LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU. `multishot` was RETIRED: no such entry point exists in either reference | +| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The temporal x2 latent upsampler and `BetaScheduler`. A product path constructs each condition and throws, naming the missing piece | +| LTX-2.5 arms nothing can request | Declared, not requestable | LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU. No request field, load extra or CLI flag asks for these. `multishot` was RETIRED: no such entry point exists in either reference | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started | diff --git a/docs/USAGE.md b/docs/USAGE.md index fe0dd1378..9934eefd9 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1661,6 +1661,16 @@ which silently substituted the recipe default for the file you named. Give `num_frames` (or `duration`, which is exact arithmetic against the recipe's frame rate) instead. Every other key in that list reaches a reader. +Two LTX-2.5 arms are refused where a render would otherwise silently downgrade: +the temporal x2 latent upsampler and `BetaScheduler`. Both are reachable — a +render asking for either gets a refusal naming the missing piece. Three more are +recorded as out of scope but are **not requestable**, so no flag or extra can +reach them: LoRA fusion, `int8-convrot`, and single-node multi-GPU. Their messages +say `DECLARED, NOT REQUESTABLE` so the two kinds are not confused. `int8-convrot` +in particular is a ComfyUI-ecosystem format: upstream LTX-2's own inference +quantization kinds are `fp8-cast`, `fp8-scaled-mm`, `nvfp4-cast` and +`nvfp4-prequant`, and nothing wired upstream reaches int8 at all. + `prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real tokens; absent, every row is. It matters because the embeddings connector substitutes its learnable register table at PADDED positions, so padding decides diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 6eb51b670..f4520cc89 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -615,9 +615,11 @@ enum class Ltx2UnportedPipelineFeature { kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps), // fused by loader/fuse_loras.py kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four - // kinds upstream defines are fp8-cast / fp8-scaled-mm / - // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-27), - // int8 appears only in the TRAINER, and convrot nowhere at all + // inference kinds upstream defines are fp8-cast / fp8-scaled-mm / + // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-26). + // `convrot` is nowhere at all; int8 is UNREACHABLE rather than + // absent — trainer-only for anything wired, plus one DEAD kernel + // in ltx-kernels (triton_ops.py:35,43). §1.2 of the row spec kMultiGpuParallelism, // ltx-pipelines/multigpu — sequence-parallel, tiled data // parallel and distributed VAE decode. NOT CFG batching: // zero `cfg` hits in either multigpu tree diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index 4a28173a4..bea3ea40c 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1160,14 +1160,27 @@ void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { "fused by loader/fuse_loras.py) is out of scope." + marker + owed); case Ltx2UnportedPipelineFeature::kInt8ConvRot: - // VERIFIED ABSENT so nobody re-audits it: LTX-2 @ fd4ded7f defines exactly four - // inference quantization kinds and none is int8, `convrot` is 0 hits, and every - // upstream `int8` is either the TRAINER's bitsandbytes / quanto path or a uint8 - // buffer. This is a ComfyUI-ecosystem format, deliberately not ported. + // VERIFIED UNREACHABLE so nobody re-audits it, and stated as UNREACHABLE rather + // than ABSENT because absent is what the first version of this message claimed + // and it was false. At LTX-2 @ fd4ded7f: `convrot` / `conv_rot` / `quarot` / + // `spinquant` really are 0 hits, and the four inference quantization kinds are + // exhaustive (quantization_factory.py:23-26, `assert_never` at :50). But int8 + // is NOT trainer-only. `ltx-kernels` — an inference package — carries a per-row + // int8 quantize kernel with fp32 scales (blockwise/triton_ops.py:25-50, out + // dtype `torch.int8` at :43), aliased `rowwise_int_quantize_triton` at :436. + // That alias is its ONLY reference: blockwise/functional.py:12-18 re-exports + // five names and not this one, so nothing constructs it. The package is a fork + // of Lightricks' int8 kernel library retargeted to fp8/fp6/nvfp4 — its custom-op + // namespace is still literally `q8_kernels_ops` (functional.py:25) — and the + // int8 half is what was left behind. Nothing wired reaches int8, which is why + // the disposition is unchanged; only the sentence was wrong. Refuse("ltx2: the int8-convrot quantization is out of scope. It is a ComfyUI-ecosystem " - "format, not an LTX-2 arm: upstream's own kinds are fp8-cast, fp8-scaled-mm, " - "nvfp4-cast and nvfp4-prequant (ltx-pipelines/utils/quantization_factory.py:" - "23-27), and int8 appears upstream only in the trainer." + + "format, not an LTX-2 arm: upstream's own inference kinds are fp8-cast, " + "fp8-scaled-mm, nvfp4-cast and nvfp4-prequant (ltx-pipelines/utils/" + "quantization_factory.py:23-26), and int8 is UNREACHABLE upstream — trainer-only " + "for anything wired (ltx-trainer gemma_8bit.py:33-36, quantization.py:11-15), plus " + "one DEAD per-row int8 quantize kernel in the ltx-kernels inference package " + "(blockwise/triton_ops.py:35,43, reached only by its own alias at :436)." + marker + owed); case Ltx2UnportedPipelineFeature::kMultiGpuParallelism: // The old spelling was `kCfgParallelism`, which named something upstream does diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index c3cf04bd5..d134855cc 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -258,14 +258,20 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // DEFAULT and looks like the feature not working. // // DEFINED IS NOT THE SAME AS SERVED, and conflating the two was #611: nine of -// these ten reach a reader (`:570`, `:625`, `:721`, `:737`, `:739`, `:771`, -// `:796`, `:901`, `:942`), and `duration_head_path` reached none, so supplying a +// these ten reach a reader, and `duration_head_path` reached none, so supplying a // duration head substituted the recipe default in silence — the failure mode this // very list exists to prevent, one level in. It stays in the list because the // family DOES define the key and DOES know what it means; `CheckUnservedExtras` // refuses it by name instead, which is a different and truer message than // "unknown load extra". The full audit is in // .agents/specs/ltx25-retire-dead-arms.md §2.1. +// +// The first hand-written set of these anchors named nine lines that were readers +// of NOTHING, in this very file, and a later merge moved the real ones again. So +// they are no longer trusted: the list below is derived from this file on every +// run and compared, and the failure prints the replacement to paste in. +// READER ANCHORS (derived and gated by test_ltx2_video): +// 638 693 789 805 807 839 864 969 1010 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index ddeaa7484..4c631c698 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -127,6 +127,13 @@ target_compile_definitions(test_video_engine PRIVATE # LTX-2.5 L7: the family behind the VideoEngine seam and the driving loop, run # over a reduced-dimension checkpoint set written in the SHIPPED file format. vllm_cpp_add_test(test_ltx2_video vllm/multimodal/test_ltx2_video.cpp) +# The reader-anchor claim this suite gates is a claim ABOUT a source file, so it +# reads that file. Recorded `file:line` anchors go stale within the same PR here +# (a merge of origin/main moved all nine of these by 25 lines mid-review), which +# is why the anchors are derived from the source and compared with the recorded +# ones rather than trusted. +target_compile_definitions(test_ltx2_video PRIVATE + LTX2_VIDEO_SOURCE_PATH="${CMAKE_SOURCE_DIR}/src/vllm/multimodal/ltx2_video.cpp") # LTX-2.5 DiT parity gate (row MODEL-DIFFUSION-LTX25, spec phase L2). The goldens # live next to the test source, like every other generated .inc. vllm_cpp_add_test(test_ltx2 vllm/models/test_ltx2.cpp) @@ -149,6 +156,12 @@ target_include_directories(test_ltx2_text_encoder PRIVATE ${CMAKE_CURRENT_SOURCE # next to the test source. vllm_cpp_add_test(test_ltx2_pipeline vllm/models/test_ltx2_pipeline.cpp) target_include_directories(test_ltx2_pipeline PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) +# The out-of-scope ledger separates a REACHABLE refusal from a declared marker, and +# docs/FEATURES.md is where that distinction reaches a user. It had re-merged the two +# into one "Declared, not requestable" row, so the public surface contradicted the +# ledger it projects; the case below reads the doc and holds it to the split. +target_compile_definitions(test_ltx2_pipeline PRIVATE + VLLM_CPP_FEATURES_DOC_PATH="${CMAKE_SOURCE_DIR}/docs/FEATURES.md") # LTX-2.5 L6: the quantized loaders, gated three ways — the SHIPPED checkpoints' # own manifests, a few hundred of their own bytes against torch, and synthetic # files for whole-model materialization. All three .inc files live beside the diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 274c4fd76..47a611d03 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -30,8 +30,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -1282,6 +1284,83 @@ TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { INFO("marker = ", item.second, " refusal = ", message); CHECK(Mentions(message, "DECLARED, NOT REQUESTABLE")); } + + // THE ABSENCE THIS MESSAGE STATES IS ITSELF EVIDENCE, so it is gated. The first + // version of this marker ended "int8 appears upstream only in the trainer" — a + // false-absence claim of exactly the kind row LTX25-RETIRE-DEAD-ARMS exists to + // retire (#604), shipped inside a user-visible refusal. LTX-2 @ fd4ded7f carries + // a per-row int8 quantize kernel with fp32 scales in `ltx-kernels`, which is an + // INFERENCE package, not the trainer: `blockwise/triton_ops.py:35,43`, aliased + // `rowwise_int_quantize_triton` at `:436`. It is dead — that alias is its only + // reference and `blockwise/functional.py:12-18` does not re-export it — so the + // disposition is unchanged and only the sentence was wrong. See + // .agents/specs/ltx25-retire-dead-arms.md §1.2. + { + const std::string message = RefusalMessage([] { + vllm::Ltx2RefuseUnportedPipelineFeature(vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot); + }); + INFO("int8 marker = ", message); + CHECK_FALSE(Mentions(message, "only in the trainer")); + // The true statement names where the one inference-side int8 lives, so a + // reader who greps upstream and finds it is not left thinking we missed it. + CHECK(Mentions(message, "ltx-kernels")); + } +} + +// THE PUBLIC SURFACE MUST NOT RE-MERGE WHAT THE LEDGER SPLIT. `docs/FEATURES.md` +// is where the reachable/marker distinction reaches a user, and it carried ONE +// "Declared, not requestable" row listing all five arms — including the two a +// caller CAN trip. That is the same conflation the ledger above stopped making, +// reintroduced one surface out. Gated here rather than left to review because a +// doc row and an enum drift silently. +// +// Deliberately narrow: it asserts only that no row calling something +// unrequestable names a REACHABLE arm. It does not police the wording of the doc. +TEST_CASE("ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable") { + std::ifstream in(VLLM_CPP_FEATURES_DOC_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << VLLM_CPP_FEATURES_DOC_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string doc = buf.str(); + REQUIRE(doc.size() > 1000); + + // The two arms with a product call site: `ltx2_upsampler.cpp` constructs the + // temporal-upsampler condition and `ltx2_pipeline.cpp` the BetaScheduler one. + // Named by the words the doc uses for them, since that is what a reader sees. + const std::vector reachable_words = {"upsampler", "BetaScheduler"}; + + size_t rows_examined = 0; + size_t line_no = 0; + size_t at = 0; + while (at <= doc.size()) { + const size_t end = doc.find('\n', at); + const std::string line = doc.substr(at, end == std::string::npos ? std::string::npos : end - at); + ++line_no; + // A table row that makes the not-requestable claim, in either casing the doc + // uses for it. + const bool is_row = !line.empty() && line[0] == '|'; + const bool claims_unrequestable = + line.find("not requestable") != std::string::npos || + line.find("NOT REQUESTABLE") != std::string::npos; + if (is_row && claims_unrequestable && line.find("LTX-2.5") != std::string::npos) { + ++rows_examined; + for (const std::string& word : reachable_words) { + INFO("FEATURES.md:" << line_no << " = " << line); + CHECK_MESSAGE(line.find(word) == std::string::npos, + "a row claiming 'not requestable' names the REACHABLE arm '" + << word + << "'; split the reachable refusals out, per " + ".agents/specs/ltx25-retire-dead-arms.md §1.6"); + } + } + if (end == std::string::npos) break; + at = end + 1; + } + // ANTI-VACUOUS. Without this the case passes when the row is renamed away and + // proves nothing — the failure mode this whole row is about. + CHECK_MESSAGE(rows_examined == 1, + "expected exactly ONE LTX-2.5 'not requestable' row in docs/FEATURES.md, found " + << rows_examined); } // =========================================================================== diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 05c4fe0c5..1e1aa9de1 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -25,8 +25,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -528,8 +530,10 @@ TEST_CASE("ltx2 video: duration_head_path is REFUSED by name, not silently ignor // the caller who supplies it. // // The audit behind it is in .agents/specs/ltx25-retire-dead-arms.md §2.1: nine of -// the ten keys have a reader (`ltx2_video.cpp:570,625,721,737,739,771,796,901,942`), -// and `duration_head_path` was the only one with none. +// the ten keys have a reader and `duration_head_path` was the only one with none. +// The reader LINES are deliberately not repeated here — they moved twice while +// this row was in review. They live in one place, the READER ANCHORS comment in +// `ltx2_video.cpp`, and the case below derives them and holds that comment to it. TEST_CASE("ltx2 video: every accepted load extra is READ by something") { Workspace ws; // The keys with a reader. @@ -593,6 +597,159 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { } } +// THE ANCHORS FOR THAT INVENTORY, DERIVED RATHER THAN TRUSTED. +// +// The case above proves each key is accepted or refused; it cannot prove WHERE a +// key is read, and "nine of ten reach a reader at these lines" is the claim this +// row rests on. That claim shipped wrong: the recorded anchors were nine lines +// that named no reader at all, in the very file they were recorded in. Then, in +// review, a merge of `origin/main` moved all nine again. A `file:line` written by +// hand is stale by the next commit, so this derives them from the source and +// holds the recorded list to what it finds. When it fails it prints the answer. +// +// It does NOT assert absolute line numbers of its own — nothing here to rot. The +// only obligation it creates is on whoever moves a reader: update the one comment +// block in the same file they are already editing. +namespace { + +std::string ReadSourceFile(const char* path) { + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open " << path); + std::stringstream buf; + buf << in.rdbuf(); + return buf.str(); +} + +std::vector SplitLines(const std::string& text) { + std::vector lines; + size_t at = 0; + while (at <= text.size()) { + const size_t end = text.find('\n', at); + lines.push_back(text.substr(at, end == std::string::npos ? std::string::npos : end - at)); + if (end == std::string::npos) break; + at = end + 1; + } + return lines; +} + +// 1-based index of the ONLY line containing `needle`, or 0. Uniqueness is the +// point: an anchor that matches twice anchors nothing, and existence alone is +// what let the stale numbers survive. +size_t UniqueLineWith(const std::vector& lines, const std::string& needle) { + size_t found = 0; + size_t count = 0; + for (size_t i = 0; i < lines.size(); ++i) { + if (lines[i].find(needle) != std::string::npos) { + ++count; + found = i + 1; + } + } + return count == 1 ? found : 0; +} + +std::string JoinNumbers(const std::vector& v) { + std::string s; + for (size_t n : v) s += (s.empty() ? "" : " ") + std::to_string(n); + return s; +} + +} // namespace + +TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") { + const std::string source = ReadSourceFile(LTX2_VIDEO_SOURCE_PATH); + const std::vector lines = SplitLines(source); + REQUIRE(lines.size() > 500); + + // Everything is measured relative to the accepted-keys array, so a stray + // mention in the file header cannot be mistaken for a reader. + const size_t array_line = UniqueLineWith(lines, "const char* const kKnownLoadExtras[] = {"); + REQUIRE_MESSAGE(array_line != 0, "kKnownLoadExtras[] declaration is not unique in the source"); + size_t array_end = 0; + for (size_t i = array_line; i < lines.size(); ++i) { + if (lines[i] == "};") { + array_end = i + 1; + break; + } + } + REQUIRE(array_end > array_line); + + // The nine SERVED keys, by the token each is spelled with in the source. Order + // is irrelevant — the comparison is on the sorted set — so this list is not a + // second place the anchors live. + const std::vector served_tokens = { + "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", + "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", + "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", + }; + std::vector derived; + for (const std::string& token : served_tokens) { + size_t first = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find(token) != std::string::npos) { + first = i + 1; + break; + } + } + INFO("token = " << token); + CHECK_MESSAGE(first != 0, "no reader found after kKnownLoadExtras for " << token); + if (first != 0) derived.push_back(first); + } + REQUIRE(derived.size() == served_tokens.size()); + std::sort(derived.begin(), derived.end()); + + // The RECORDED list, parsed out of the one comment line that carries it. + const size_t marker = UniqueLineWith(lines, "READER ANCHORS (derived and gated by"); + REQUIRE_MESSAGE(marker != 0, + "the READER ANCHORS comment marker is missing or not unique in the source"); + const std::string recorded_line = lines[marker]; // the line AFTER the marker (1-based) + std::vector recorded; + for (size_t i = 0; i < recorded_line.size();) { + if (std::isdigit(static_cast(recorded_line[i]))) { + size_t j = i; + while (j < recorded_line.size() && std::isdigit(static_cast(recorded_line[j]))) { + ++j; + } + recorded.push_back(static_cast(std::stoul(recorded_line.substr(i, j - i)))); + i = j; + } else { + ++i; + } + } + std::sort(recorded.begin(), recorded.end()); + CHECK_MESSAGE(recorded == derived, "the reader anchors recorded in ltx2_video.cpp are STALE. " + "Recorded: [" + << JoinNumbers(recorded) << "]. Actual: [" + << JoinNumbers(derived) + << "]. Paste the actual list into the READER ANCHORS " + "comment; the spec's §2.1 table is dated and stays."); + + // And the UNSERVED key is touched only by the refusal, never by a reader. This + // is the half a "has a reader" sweep cannot express. + const size_t refuse_line = UniqueLineWith(lines, "void CheckUnservedExtras("); + REQUIRE(refuse_line != 0); + size_t refuse_end = 0; + for (size_t i = refuse_line; i < lines.size(); ++i) { + if (lines[i] == "}") { + refuse_end = i + 1; + break; + } + } + REQUIRE(refuse_end > refuse_line); + size_t duration_hits = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find("kLtx2DurationHeadPathExtra") == std::string::npos) continue; + ++duration_hits; + const size_t at = i + 1; + const bool inside_refusal = (at >= refuse_line) && (at <= refuse_end); + CHECK_MESSAGE(inside_refusal, + "ltx2_video.cpp:" << at + << " touches the duration-head extra OUTSIDE " + "CheckUnservedExtras; if it now has a real reader, move " + "it to the served list and drop the refusal (#611)"); + } + CHECK(duration_hits > 0); +} + // ─── the config the SHAPES cannot see ─────────────────────────────────────── // // WHY THESE ASSERT ON THE ENGINE AND NOT ON A LOCAL. The L7 repair is one line in From 24897ad96a624e3c86353255c20a1929b7a18c7d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 22:24:18 +0000 Subject: [PATCH 5/6] record(LTX25-RETIRE-DEAD-ARMS): file the two review findings this row does not own (#691, #692) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings F5 and F6 on #661. Neither is repaired here, and the reason is recorded so neither becomes debt nobody holds. F5 (#691) — the reachable/marker split gates the MESSAGE TEXT, not reachability. The ledger test is not tautological in the direction it tests; both directions were mutation-proven. But nothing asserts that a marker still has no product call site, so adding one for `kLoraFusion` and changing nothing else leaves the test green while the shipped refusal tells a caller "no request field or load extra asks for this" immediately after they asked for it through a load extra. The issue names the anti-tautological shape to copy: the extras inventory gate parses the accepted set out of the product's OWN unknown-extra refusal rather than from a list kept in the test, which is why adding an eleventh key REDs it. F6 (#692) — `.agents/specs/ltx-2-5.md:291` still records the fabricated `multishot` arm as owed. That file is the operator-owned campaign spec and this row is scoped out of editing it, so `main` was carrying the claim with nobody holding it. The issue carries the full re-derivation and two smaller corrections on the same line. Both placed in the roadmap issue table under `ROAD-V1-LTX25` and named in this row's spec section 3 "Out", so the three surfaces agree. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code] --- .agents/roadmap_v1.md | 2 ++ .agents/specs/ltx25-retire-dead-arms.md | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index ce8f604c5..50cb95099 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -150,6 +150,8 @@ issue is not yet placed. Keyed record: update in place, never append. | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | | [#611](https://github.com/mudler/vllm.cpp/issues/611) | `ROAD-V1-LTX25` | `duration_head_path` is accepted by `kKnownLoadExtras` and read by NOTHING, so a caller who points at a duration head silently gets the recipe default. Row `LTX25-RETIRE-DEAD-ARMS` (spec [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md)) refuses it by name and records the full accepted/read inventory of all ten extras | bug | +| [#691](https://github.com/mudler/vllm.cpp/issues/691) | `ROAD-V1-LTX25` | The LTX-2.5 out-of-scope ledger's reachable/marker split gates the MESSAGE TEXT, not reachability: give `kLoraFusion` a product call site and the ledger test still passes while the shipped refusal tells the caller `DECLARED, NOT REQUESTABLE` right after they requested it. Copy the anti-tautological shape of the extras inventory gate | bug | +| [#692](https://github.com/mudler/vllm.cpp/issues/692) | `ROAD-V1-LTX25` | Campaign spec `ltx-2-5.md:291` still records the FABRICATED `multishot` arm as owed after row `LTX25-RETIRE-DEAD-ARMS` retired it; same line needs `CFG parallelism` renamed to multi-GPU. Operator-owned file, so filed rather than edited by that row | bug | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md index 315f87e3a..39bb38446 100644 --- a/.agents/specs/ltx25-retire-dead-arms.md +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -353,7 +353,16 @@ is a different row. Refusing is the cheap correct answer until then. - `.agents/specs/ltx-2-5.md` — operator-owned. This row does not edit it. Its §2 "Out" list still names `multishot`; correcting that is the operator's edit, and - this spec is the record it would cite. + this spec is the record it would cite. Filed as + [#692](https://github.com/mudler/vllm.cpp/issues/692) by the review repair so + `main` stops carrying the claim unowned. +- Making the reachable/marker split gate REACHABILITY rather than message text. + The ledger test asserts what each message says; nothing asserts that a marker + still has no product call site, so giving `kLoraFusion` one would leave the test + green while the shipped refusal told a caller it is not requestable immediately + after they requested it. Filed as + [#691](https://github.com/mudler/vllm.cpp/issues/691), with the extras inventory + gate named as the anti-tautological shape to copy. - Constructing a duration head, and the AUTO-duration path. Stays owed (#611 remains open after this row, retitled by the fix rather than closed by it — see §7). From 18ac7a6a056edd81e3126c6a274de216082633c7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 13:06:26 +0000 Subject: [PATCH 6/6] fix(LTX25-RETIRE-DEAD-ARMS): the third #604 instance, in the shipped header (#644, #611) The row whose stated purpose is retiring instances of #604 shipped three of them. This is the third, and it is the one that reached a header a porter reads. `include/vllm/model_executor/models/ltx2_pipeline.h`, the spec's section 1.1 table and the PR body all said "the only `scene` hit is PySceneDetect in the TRAINER". False at Lightricks/LTX-2 `fd4ded7f`. Re-derived with a positive control in the same command -- `git grep -n -i -E 'multi[_ -]?shot|scene cuts|single continuous take|DurationPredictor'` returns 0 lines for the absence and 22 control lines, so the command demonstrably ran -- `scene` has three senses upstream: * `scene-linear` HDR colour, in ltx-core color/hlg.py and hdr.py; * PySceneDetect in ltx-trainer, the only CODE sense and a training-data preprocessor; * prompt-writing guidance, which ships at INFERENCE inside `ltx-core` as text_encoders/gemma/encoders/prompts/gemma{3,4}_{i2v,t2v}_system_prompt.txt. The third is the one that was denied, and `README.md:136` -- "chronological descriptions of actions and scenes" -- is the same line the header already cited two clauses earlier for "a cinematographer describing a shot list". The disposition does not move; it is STRENGTHENED. Those shipped prompts tell the enhancer NOT to describe scene cuts (gemma3_i2v:6,18, gemma3_t2v:24) and to keep a "Single continuous take -- no hard cuts" (gemma4_i2v:3). An upstream whose own prompt enhancer is instructed toward one continuous take is affirmative evidence that no multi-shot generation mode exists, which is what `kMultishot` claimed to refuse. The failure this leaves behind if unfixed is a porter grepping `scene`, finding cuts prohibited in a shipped inference prompt, and concluding we missed a multi-shot path our own header told them did not exist. Two more evidence defects from the same round: * The section 1.1 `shot` enumeration was offered as exhaustive at five files. It is seven. The cause is recorded because it is reusable: the query was `\bshots?\b` restricted to `*.py *.md *.json *.yaml`, and upstream's prompt files are `.txt`. Re-running the old filter reproduces exactly the five recorded files, so the filter -- not the pattern -- hid the answer, and it hid the two that ship at inference. Those two carry VERBATIM the "Shot type (exactly one: extreme wide shot / ...)" sentence the spec presented as diffusers' independent corroboration; diffusers vendored Lightricks' gemma4 prompts, so `utils.py:217` is one source quoted twice, not two sources agreeing. Recorded as such. * `quantization_factory.py:23-27` was wrong at both ends: the quoted block starts at `:22` (`class QuantizationKind(str, Enum):`) and the four members are `:23-26`, which is what the header and the shipped refusal already say. The file is 50 lines, so `:27` is a blank line. And section 7's "Every claim in section 1 was re-derived" is replaced by what actually happened -- three rounds, three false-absence claims, two of them shipped -- because the claim was the mechanism: writing that every claim was re-derived is what made the next reader stop checking. RED first. `test_ltx2_pipeline` gains "the kMultishot retirement note states the scene evidence correctly", which reads the shipped header through a new LTX2_PIPELINE_HEADER_PATH compile definition, flattens the comment block so a reflow cannot make it vacuous, and requires the false sentence absent and the replacement evidence present. 4 of its 8 assertions failed at b718f580f. It is anti-vacuous: it first requires exactly ONE ``kMultishot` -- FABRICATED` note, and renaming that marker in a staged copy gives `REQUIRE( 0 == 1 )`. What that case cannot do is stated in its own comment rather than papered over: it holds the header's TEXT, so the false sentence cannot come back, but it cannot verify the upstream claim, because no upstream checkout exists in this tree. The verification is the spec's commands with their controls, run against the two pins. docs/USAGE.md gains the user-facing half: multi-shot generation is not on the refusal list because upstream has no such mode, with the three senses of `scene` named so a user does not read the absence as a missing feature. It is also what `doc-checkpoint --staged` requires of any `include/vllm/` edit, and it is recorded here that both reasons applied. test_ltx2_pipeline 38 cases / 2395 assertions -> 39 / 2403, exit 0. Nothing removed. test_ltx2_video unchanged at 33 / 541. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code] --- .agents/specs/ltx25-retire-dead-arms.md | 262 +++++++++++++++--- docs/USAGE.md | 11 + .../model_executor/models/ltx2_pipeline.h | 20 +- tests/CMakeLists.txt | 6 +- tests/vllm/models/test_ltx2_pipeline.cpp | 81 ++++++ 5 files changed, 336 insertions(+), 44 deletions(-) diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md index 39bb38446..d9ebc217b 100644 --- a/.agents/specs/ltx25-retire-dead-arms.md +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -10,11 +10,14 @@ Upstream pins: | Reference | Revision | Local checkout verified at | |---|---|---| -| Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | `git rev-parse HEAD`, 2026-08-13 | -| huggingface/diffusers | `3a2f35d4efa4c059c8bfb3bc0d6c906264895c81` | `git rev-parse HEAD`, 2026-08-13 | +| Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | +| huggingface/diffusers | `3a2f35d4efa4c059c8bfb3bc0d6c906264895c81` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | -Every anchor below was re-derived from those two checkouts by this row. A prior -grounding pass reached the same conclusions; it is an input, not a result. +Anchors here were derived from those two checkouts by this row. **Three of them +were wrong and were found by review, not by the row** — see §7, which records +what was not re-derived rather than claiming everything was. Where an ABSENCE is +asserted, §1.1 carries the command and the control hits that prove the command +ran; treat any absence in this file without that pairing as unverified. ## 0. What is wrong today @@ -45,38 +48,93 @@ told nothing. ### 1.1 `multishot` — FABRICATED Searched as a **subject**, not as our own phrasing (the self-confirming-grep trap, -#604): +#604) — and **every absence below was re-derived on 2026-08-14 with a POSITIVE +CONTROL in the same command**, after two of the claims in this section shipped +false and a third was offered as exhaustive when it was not. A +grep that returns nothing proves nothing on its own — the same command has to +return the hits you know are there. The two commands, verbatim: + +```sh +# LTX-2 @ fd4ded7f — `multishot` ABSENCE, controls `scene cuts` / +# `single continuous take` / `DurationPredictor` in the SAME command. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|single continuous take|DurationPredictor' +# -> 0 lines matching multi[_ -]?shot, 22 control lines. +# LTX-2 @ fd4ded7f — every `shot` FILE, control `DurationPredictor`. +git grep -l -E '\bshots?\b' # -> 7 files (see below) +# LTX-2 @ fd4ded7f — `storyboard` ABSENCE, control `shot list` same command. +git grep -n -i -E 'storyboard|shot list' # -> 1 line, README.md:136, the control. +# LTX-2 @ fd4ded7f — what the `multi` prefix actually attaches to. +git grep -h -o -i -E 'multi[_ -]?(gpu|scale|stage|shot|clip|prompt|segment)' \ + | sort | uniq -c | sort -rn +# -> 94 multigpu, 44 multi-GPU, 25 Multi-GPU, 4 MULTI_GPU, 2 multi-prompt. +# diffusers @ 3a2f35d4 — same absence in the ltx2 pipelines package. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|hard cuts|single continuous take' \ + -- src/diffusers/pipelines/ltx2/ +# -> 0 lines matching multi[_ -]?shot, 4 control lines. +``` | Query | LTX-2 `fd4ded7f` | diffusers `3a2f35d4` | |---|---|---| -| `multishot` / `multi_shot` / `multi-shot` (case-insensitive) | 0 hits | 0 hits | -| `\bshots?\b` in `*.py` `*.md` `*.json` `*.yaml` | 5 files, none a generation mode | — | -| `multi[_ -]?(gpu\|scale\|stage\|shot\|clip\|prompt\|segment)` | `multi-GPU` / `multi_gpu` only | — | -| `scene` / `storyboard` | `ltx-trainer/scripts/split_scenes.py` (PySceneDetect, a **training-data** preprocessor) | — | - -The five `shot` files, with what the word means in each: +| `multi[_ -]?shot` (case-insensitive) | 0 hits, alongside 22 control hits | 0 hits, alongside 4 control hits | +| `\bshots?\b`, whole repository | **7 files**, none a generation mode | — | +| `multi[_ -]?(gpu\|scale\|stage\|shot\|clip\|prompt\|segment)` | `multi[-_ ]?gpu` **167**, `multi-prompt` **2** — and nothing else | — | +| `scene`, whole repository | **three senses, none a generation mode** — see §1.1a | only the prompt-guidance sense inside `pipelines/ltx2/` (`scene cuts` ×3, `scene opens` ×4) | +| `storyboard` | 0 hits, alongside its control | 0 hits | + +An earlier revision of this table said the `multi` prefix attaches to +`multi-GPU`/`multi_gpu` **only**. It does not: `multi-prompt` appears twice +(`ltx-pipelines/utils/args.py:618`, "Helps multi-prompt enhance"; and +`utils/denoisers.py:174`, "multi-prompt benchmark panels"). Both are *batching +several captions through one run* — the enhancer's KV cache and a benchmark +panel — not composing several takes into one output, so the conclusion is +unaffected and the "only" was still wrong. Recorded rather than quietly +corrected, for the same reason as everything else in this section. + +The **seven** `shot` files, with what the word means in each. An earlier revision +of this list said five and presented that as exhaustive (review finding M1). The +cause is recorded because it is reusable: that query was +`\bshots?\b` **restricted to `*.py *.md *.json *.yaml`**, and upstream's prompt +files are `.txt`. The filter, not the pattern, hid the answer — and it hid +exactly the two files that matter most here, because they are the ones that ship +at inference. Re-run without a path filter it is seven: - `ltx-core/duration_head/duration_head.py:1,5` — "predicts **shot** duration", i.e. the natural length of ONE camera take. - `ltx-core/duration_head/__init__.py:1` — same. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_i2v_system_prompt.txt:13,14,17` + — camera **shot type** in the prompt-enhancer's instructions. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_t2v_system_prompt.txt:11,12,15` + — same. - `ltx-pipelines/utils/blocks.py:804` — `DurationPredictor`, "Predicts **shot** duration (in frames)". -- `ltx-trainer/src/ltx_trainer/captioning.py` — captioning prose. +- `ltx-trainer/src/ltx_trainer/captioning.py:41,42,54,57` — captioning prose, for + describing an EXISTING video as training data. - `LTX-2/README.md:59,136` — an example prompt ("a medium close-up **shot**") and the prompting guide ("think like a cinematographer describing a **shot** list"). Upstream's "shot" is a single continuous take. There is no multi-shot generation mode, and nothing that composes several takes into one output. -diffusers says the same thing in its own vocabulary, which is the stronger check -because the word is far more common there: every `shot` in -`src/diffusers/pipelines/ltx2/` is a **camera shot type** in the prompt-enhancement -guidance — `utils.py:217`, "Shot type (exactly one: extreme wide shot / wide shot -/ medium shot / medium close-up / close-up / extreme close-up)" — or the duration -head's own docstring, `duration_head.py:83`, "Predicts the natural duration of the -**shot** implied by a caption". Its sixteen `ltx2/` modules are -`pipeline_ltx2`, `_condition`, `_diffusion_decode`, `_hdr_lora`, `_ic_lora`, -`_image2video`, `_latent_upsample`, plus the components. None is multi-shot. +The nearest thing in either tree runs the other direction: +`ltx-trainer/captioning.py:54` tells the CAPTIONER "the video contains multiple +shots, describe each one in turn" — an instruction for describing an input video +during dataset preparation, not for generating one. Named here so the next porter +who greps `shots` does not have to re-derive that it is the trainer. + +**The `shot` claim, corrected on the diffusers side too.** This section used to +call `utils.py:217` — "Shot type (exactly one: extreme wide shot / wide shot / +medium shot / medium close-up / close-up / extreme close-up)" — the "stronger +check" because it came from a second, independent reference. It is not +independent: that sentence is **verbatim** LTX-2's own +`gemma4_t2v_system_prompt.txt:12`, and `utils.py:251` is verbatim +`gemma4_i2v_system_prompt.txt:14`. diffusers vendored Lightricks' gemma4 system +prompts wholesale. It corroborates that upstream's `shot` is a camera shot type, +and it does so at a second pin, but it is one source quoted twice, not two +sources agreeing. diffusers' own contribution is the duration head's docstring, +`duration_head.py:83`, "Predicts the natural duration of the **shot** implied by +a caption", and its module list: `pipeline_ltx2`, `_condition`, +`_diffusion_decode`, `_hdr_lora`, `_ic_lora`, `_image2video`, `_latent_upsample`, +plus the components. None is multi-shot. `ltx-pipelines`' actual entry points, from `docs/pipelines.md` ("Full reference for all 11 pipelines") and the module list: @@ -85,6 +143,47 @@ for all 11 pipelines") and the module list: `distilled_mgpu`, `ic_lora`, `hdr_ic_lora`, `a2vid_two_stage`, `t2a_one_stage`, `dubit`, `retake`, `dfr_pipeline`, `keyframe_interpolation`. None is multi-shot. +### 1.1a `scene` — three senses, and the one that was missed twice + +**This is the third #604 instance this row shipped, and the one that took three +review rounds to find** (review finding B1). The header, this table and the PR +body all said "the only `scene` hit is PySceneDetect in the TRAINER". That is +false at `fd4ded7f`, and it was false in a SHIPPED header — the failure scenario +being a porter who greps `scene`, finds "scene cuts" prohibited in an inference +prompt, and concludes we missed a multi-shot path our own header denied. + +`scene` has three senses upstream, established by reading **every** hit of +`git grep -n -i scene` — no path filter, since a path filter is what hid the +`shot` answer above — rather than by grepping our own vocabulary. The anchors +below are representative of each sense, not a transcript of every line; the +command above is the transcript: + +| Sense | Where | Is it a generation mode? | +|---|---|---| +| `scene-linear` HDR colour (scene-referred light) | `ltx-core/color/hlg.py:1,5,44,74,93,256`, `ltx-core/hdr.py:29,132,146,162,171`, `ltx-pipelines/docs/hdr.md:15,16,34`, `utils/media_io/{color_config,encode,exr}.py`, `utils/args.py:813,861`, `ic_lora.py:422`, `retake.py:178`, `hdr_ic_lora.py:18`, `CHANGELOG.md:24` | no — colour science | +| PySceneDetect splitting | `ltx-trainer/scripts/split_scenes.py`, `ltx-trainer/docs/*`, `ltx-trainer/pyproject.toml:28` (`scenedetect>=0.6.5.2`), `ltx-trainer/AGENTS.md:85`, `.claude/skills/train-model/**` | no — a **training-data** preprocessor, and the only CODE sense | +| narrative prose, i.e. prompt-writing guidance | **`ltx-core`, which ships at INFERENCE**: `text_encoders/gemma/encoders/prompts/gemma3_i2v_system_prompt.txt:5,6,18,20`, `gemma3_t2v_system_prompt.txt:1,5,24,25`, `gemma4_i2v_system_prompt.txt:7,15,23,27`, `gemma4_t2v_system_prompt.txt:5,13,21,25`; plus `README.md:136` and `ltx-trainer/captioning.py:39` | no — instructions to the prompt enhancer | + +The third sense is the one the old sentence denied, and `README.md:136` is +**the same line** the header cites two clauses earlier for "a cinematographer +describing a shot list" — it reads "chronological descriptions of actions and +scenes". + +**The disposition does not move; it is STRENGTHENED.** Read as a subject rather +than as a keyword, the shipped enhancer prompts instruct the model *against* +multi-shot output: + +- `gemma3_i2v_system_prompt.txt:6` — "Inaccurate descriptions may cause scene cuts." +- `gemma3_i2v_system_prompt.txt:18`, `gemma3_t2v_system_prompt.txt:24` — "No + timestamps or cuts: DO NOT use timestamps or describe scene cuts unless + explicitly requested." +- `gemma4_i2v_system_prompt.txt:3` — "**Single continuous take — no hard cuts.**" + +diffusers carries the same four lines (`utils.py:121,161,182,240`), because it +vendored the same prompts. An upstream whose own prompt enhancer is told to +produce one continuous take is affirmative evidence that no multi-shot generation +mode exists — which is what `kMultishot` claimed to refuse. + **Disposition: RETIRE.** This is a defect in our record, not a gap in our port. There is nothing to owe, so recording it as owed is the error. The enumerator is removed and the retirement recorded in the header, in this spec, and in the @@ -94,8 +193,12 @@ no waiver registry"). ### 1.2 `int8-convrot` — real absence, deliberately out of scope The inference quantization kinds LTX-2 defines, exhaustively -(`ltx-pipelines/utils/quantization_factory.py:23-27`, a `str`-valued enum with an -`assert_never` on the match): +(`ltx-pipelines/utils/quantization_factory.py:22-26`, a `str`-valued enum with an +`assert_never` at `:50`, the file's last line — re-derived 2026-08-14, review +finding L1. The block quoted below starts at the `class` statement on `:22` and +its four members are `:23-26`, which is the range the header comment and the +shipped refusal message cite; `:23-27` was wrong at both ends and this spec was +the last place still carrying it): ```python class QuantizationKind(str, Enum): @@ -348,6 +451,14 @@ is a different row. Refusing is the cheap correct answer until then. reader anchors instead of writing them by hand (§2.1), and split the `docs/FEATURES.md` row so the public surface stops calling the two REACHABLE arms unrequestable (§1.6). No disposition changes. +8. Added by the THIRD review round: correct the false `scene` absence in the + shipped header, in §1.1's table and in the PR body (§1.1a); correct the `shot` + enumeration from five files to seven and record the path filter that caused it + (§1.1); correct `quantization_factory.py:23-27` to `:22-26` (§1.2); replace + §7's "every claim in §1 was re-derived" with what actually happened; and gate + the header's retirement note with a RED-first case so the sentence cannot come + back. No disposition changes — the `scene` evidence strengthens the + `kMultishot` retirement rather than weakening it. **Out.** @@ -413,6 +524,30 @@ Three more added by the review repair, each RED before its fix: one such LTX-2.5 row to exist, so renaming the row away fails it. **RED before**: both reachable arms found on `FEATURES.md:328`. +One more added by the third review round: + +7. **`ltx2 the kMultishot retirement note states the scene evidence correctly`** + (new, `tests/vllm/models/test_ltx2_pipeline.cpp`). Reads + `include/vllm/model_executor/models/ltx2_pipeline.h` through a new + `LTX2_PIPELINE_HEADER_PATH` compile definition, flattens the comment block so a + reflow cannot make it vacuous, and requires the retired false sentence to be + absent and the replacement evidence — `scene cuts`, `ltx-core`, + `system_prompt` — to be present. Anti-vacuous: it first requires **exactly one** + ``kMultishot` — FABRICATED` note in the file, so deleting or renaming the + paragraph fails rather than passes. **RED before**: 4 of its 8 assertions + failed on `b718f580f` — `flat.find("the only \`scene\` hit is PySceneDetect")` + returned 29407 where npos was required, and `scene cuts` / `system_prompt` + both returned npos. **Anti-vacuity proven by mutation**: renaming the note's + marker in a staged copy gives `REQUIRE( 0 == 1 )`; the header was restored and + verified by sha256. + + What it can and cannot prove is stated in the case's own comment: it holds the + header's TEXT, so the false sentence cannot return, but it cannot verify the + upstream claim — no upstream checkout exists in this tree. That verification is + §1.1's commands with their positive controls, run against the two pins, and it + is not automatable here. Saying so is better than a test that appears to gate + it and does not. + ## 5. Risks | Risk | Mitigation | @@ -432,11 +567,43 @@ Three more added by the review repair, each RED before its fix: ## 7. Outcome -**Every claim in §1 was re-derived by this row against the two pinned checkouts; -none was taken from the prior grounding pass.** Results: - -- `multishot`: **fabricated, confirmed.** 0 hits for the term in either reference; - the only upstream sense of "shot" is one camera take (§1.1). Retired. +**Three of the claims in §1 were NOT re-derived, and three rounds of review found +them one at a time.** The sentence that stood here said "every claim in §1 was +re-derived"; it was written before the second and third rounds, and it was not +true when it was written either. What actually happened: + +- Round 1 found the `hadamard` absence in §1.2 false, and the "int8 appears only + in the trainer" sentence false **in the shipped refusal message**. +- Round 2 found nine reader anchors in §2.1 stale, cited in the file they were + wrong about. +- Round 3 found `scene` asserted trainer-only **in the shipped header**, the + `shot` file list presented as exhaustive at five when it is seven, and a + `quantization_factory.py` range wrong at both ends. Re-deriving the rest of + §1.1's table in the same pass turned up a fourth: the `multi` prefix was + recorded as `multi-GPU`/`multi_gpu` **only**, and `multi-prompt` is there twice. + Nobody had asked about that row; it was wrong because the row had never been + re-run either. + +Every one of the three is the same defect: an absence asserted from our own +vocabulary, with no positive control, and in two cases behind a path filter that +excluded the files carrying the answer. **This row's stated purpose is retiring +instances of [#604](https://github.com/mudler/vllm.cpp/issues/604), and it +shipped three of them** — two of those into files a user or a porter reads. That +self-record is worth more than the claim it replaces, because the claim was the +mechanism: writing "every claim was re-derived" is what made the next reader stop +checking. §1.1 now carries the commands with their positive controls inline, so +the claim is reproducible instead of asserted, and +`test_ltx2_pipeline`'s "the kMultishot retirement note states the scene evidence +correctly" holds the header's text so the false sentence cannot come back. + +Results, as they now stand: + +- `multishot`: **fabricated, confirmed** — and the confirmation is now stronger + than the original. 0 hits for the term in either reference against 22 and 4 + control hits in the same commands; the only upstream sense of "shot" is one + camera take across all seven files; and the shipped prompt-enhancer prompts + instruct the model to keep a "Single continuous take — no hard cuts" + (§1.1, §1.1a). Retired. - `int8-convrot`: **unreachable upstream, and the first wording of that was wrong.** Four inference quantization kinds exist and none is int8, and no rotation is exposed as one — but int8 is not trainer-only (a dead per-row int8 kernel lives @@ -458,20 +625,35 @@ What this row deliberately did **not** do: construct a duration head. #611 stays open for that, with its user-visible half — silent substitution of the recipe default — closed. -**The review's own finding, recorded because it is the point of the row.** Six -findings came back and every one was evidence accuracy, not logic: a false absence -inside the shipped refusal (§1.2), a false absence in this spec's §1.2 — the one -search term that mattered — nine stale reader anchors cited in the file they were -wrong about (§2.1), and a public-doc row that re-merged the split the ledger had -just made. A row whose subject is #604 committed three instances of #604. That is -not irony to note and move past; it is why the anchors are now derived rather than -written, why the int8 absence is stated as UNREACHABLE with its dead kernel named, -and why the `hadamard` search this spec got wrong is spelled out in full rather -than quietly corrected. +**The review's own findings, recorded because they are the point of the row.** +Across three rounds every finding was evidence accuracy, not logic: a false +absence inside the shipped refusal (§1.2), a false absence in this spec's §1.2 — +the one search term that mattered — nine stale reader anchors cited in the file +they were wrong about (§2.1), a public-doc row that re-merged the split the ledger +had just made, a false absence inside the shipped HEADER (§1.1a), a five-item list +offered as exhaustive when it is seven (§1.1), and a line range wrong at both ends +(§1.2). A row whose subject is #604 committed three instances of #604, and each +one survived at least one review that had already been told to look for exactly +this. That is not irony to note and move past; it is why the anchors are now +derived rather than written, why the int8 absence is stated as UNREACHABLE with +its dead kernel named, why the `hadamard` search this spec got wrong is spelled +out in full rather than quietly corrected, and why every absence in §1.1 now ships +with the command that proves it AND the control hits that prove the command ran. + +The transferable lesson, in one line: **a path filter is an absence claim too.** +The `shot` enumeration is the case where this is provable rather than inferred — +the pattern `\bshots?\b` was right and the filter `*.py *.md *.json *.yaml` was +wrong, over a tree whose prompt files are `.txt`, and it returns exactly the five +files that were recorded. (The `scene` claim's original command was never written +down, so why *it* missed `ltx-core` is not recoverable; that it was never +reproduced is the point.) A grep that returns nothing has to be re-run without its +narrowing, and with a control, before it is written down as an absence. ## Now Row `LTX25-RETIRE-DEAD-ARMS` is `DONE`. The ledger carries five entries, split by kind; `duration_head_path` is refused by name with a RED-first test; the -`kKnownLoadExtras` inventory is recorded in §2.1. `.agents/specs/ltx-2-5.md` §2 -"Out" still lists `multishot` and is the operator's to correct, citing §1.1 here. +`kKnownLoadExtras` inventory is recorded in §2.1; the header's `kMultishot` +retirement note is gated by `test_ltx2_pipeline` against the false `scene` +absence it shipped (§1.1a). `.agents/specs/ltx-2-5.md` §2 "Out" still lists +`multishot` and is the operator's to correct, citing §1.1 here. diff --git a/docs/USAGE.md b/docs/USAGE.md index 070b2a2a9..b1175a589 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1698,6 +1698,17 @@ in particular is a ComfyUI-ecosystem format: upstream LTX-2's own inference quantization kinds are `fp8-cast`, `fp8-scaled-mm`, `nvfp4-cast` and `nvfp4-prequant`, and nothing wired upstream reaches int8 at all. +What is **not** on that list, and why: **multi-shot or multi-scene generation.** +A request that composes several camera takes into one output has no flag here +because upstream LTX-2 has no such mode to mirror — its `shot` is one continuous +take, and its own prompt-enhancement prompts instruct the model to keep a "single +continuous take" and not to describe scene cuts. `scene` does appear across the +upstream tree, in three unrelated senses (`scene-linear` HDR colour, PySceneDetect +in the trainer's dataset preprocessor, and that prompt-writing guidance); none of +them is a generation mode. This port carried a `multishot` refusal until +2026-08-13, which was a defect in our own record rather than a gap, and it was +retired. Generate one take per request. + `prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real tokens; absent, every row is. It matters because the embeddings connector substitutes its learnable register table at PADDED positions, so padding decides diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index f4520cc89..d33ff281b 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -600,9 +600,23 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // exists in Lightricks/LTX-2 @ fd4ded7f or huggingface/diffusers @ 3a2f35d4. // Searched as a SUBJECT rather than by our own phrasing: upstream's only sense // of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration"; -// README.md:136 "a cinematographer describing a shot list"), and the only -// `scene` hit is PySceneDetect in the TRAINER. A defect in our record is not a -// gap in our port, so there was nothing to owe. +// README.md:136 "a cinematographer describing a shot list"). `scene` has THREE +// senses upstream and none is a generation mode: `scene-linear` HDR colour +// (ltx-core color/hlg.py, hdr.py), PySceneDetect in the TRAINER — the only CODE +// sense — and prompt-writing guidance, which ships at INFERENCE inside +// `ltx-core`, in text_encoders/gemma/encoders/prompts/ as +// gemma{3,4}_{i2v,t2v}_system_prompt.txt. That third sense is why the +// retirement HOLDS rather than being undermined: those prompts tell the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:6,18, gemma3_t2v:24, gemma4_i2v:3). A defect in our record is not +// a gap in our port, so there was nothing to owe. +// +// Recorded because it is the row's own subject: this paragraph used to claim +// that `scene` appeared upstream ONLY as PySceneDetect in the trainer. It was +// an absence asserted from our own vocabulary with no positive control — #604 — +// shipped in the header of the row that exists to retire #604 instances, and it +// took a third review round to find. The derivation, with its positive control +// in the same command, is .agents/specs/ltx25-retire-dead-arms.md §1.1. // // `kVideoEngineWiring` — LANDED. It said the end-to-end composition through // `vllm::multimodal::VideoEngine` "is phase L7, not L5"; L7 shipped in diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index eb1dbc723..2f69fed83 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -189,8 +189,12 @@ target_include_directories(test_ltx2_pipeline PRIVATE ${CMAKE_CURRENT_SOURCE_DIR # docs/FEATURES.md is where that distinction reaches a user. It had re-merged the two # into one "Declared, not requestable" row, so the public surface contradicted the # ledger it projects; the case below reads the doc and holds it to the split. +# The ledger's RETIREMENT note carries an absence claim about upstream, and it +# shipped false twice in the same header. The case that guards it reads the header +# itself rather than a copy of the sentence kept in the test. target_compile_definitions(test_ltx2_pipeline PRIVATE - VLLM_CPP_FEATURES_DOC_PATH="${CMAKE_SOURCE_DIR}/docs/FEATURES.md") + VLLM_CPP_FEATURES_DOC_PATH="${CMAKE_SOURCE_DIR}/docs/FEATURES.md" + LTX2_PIPELINE_HEADER_PATH="${CMAKE_SOURCE_DIR}/include/vllm/model_executor/models/ltx2_pipeline.h") # LTX-2.5 L6: the quantized loaders, gated three ways — the SHIPPED checkpoints' # own manifests, a few hundred of their own bytes against torch, and synthetic # files for whole-model materialization. All three .inc files live beside the diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 47a611d03..024917a36 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -27,6 +27,7 @@ #include #include +#include #include #include #include @@ -1363,6 +1364,86 @@ TEST_CASE("ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable") << rows_examined); } +// THE RETIREMENT NOTE'S OWN EVIDENCE, held to what upstream actually says. +// +// Retiring `kMultishot` rests on an ABSENCE claim about upstream, and the header +// above the enum is where a porter reads it. That sentence shipped wrong: it said +// the only `scene` hit upstream was PySceneDetect in the TRAINER. At +// Lightricks/LTX-2 @ fd4ded7f `scene` has THREE senses, and the third — +// prompt-writing guidance — lives in `ltx-core`, which ships at INFERENCE. So a +// porter greps `scene`, finds "scene cuts" in a shipped prompt-enhancer prompt, +// and concludes we missed a multi-shot path that our own header told them did not +// exist. Third instance of #604 inside the row whose subject is retiring #604. +// +// The disposition did not move — it got STRONGER. Those prompts instruct the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:18, gemma3_t2v:24, gemma4_i2v:3), which is affirmative evidence +// that no multi-shot generation mode exists. +// +// WHAT THIS CASE CAN AND CANNOT PROVE. It reads the shipped header and holds its +// text, so the false sentence cannot come back and the true evidence cannot be +// dropped. It CANNOT verify the upstream claim — no upstream checkout exists in +// this tree — so the derivation, with the positive control in the same command, +// lives in .agents/specs/ltx25-retire-dead-arms.md §1.1. +TEST_CASE("ltx2 the kMultishot retirement note states the scene evidence correctly") { + std::ifstream in(LTX2_PIPELINE_HEADER_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << LTX2_PIPELINE_HEADER_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string header = buf.str(); + REQUIRE(header.size() > 1000); + + // The note is prose wrapped across comment lines, so every claim below is + // matched against a flattened copy: comment markers dropped, runs of whitespace + // collapsed to one space. Without this a reflow of the paragraph would silently + // turn every assertion vacuous. + std::string flat; + flat.reserve(header.size()); + bool pending_space = false; + for (size_t i = 0; i < header.size(); ++i) { + const char c = header[i]; + if (c == '/' && i + 1 < header.size() && header[i + 1] == '/') { + i += 1; + pending_space = true; + continue; + } + if (std::isspace(static_cast(c)) != 0) { + pending_space = true; + continue; + } + if (pending_space && !flat.empty()) flat += ' '; + pending_space = false; + flat += c; + } + + // ANTI-VACUOUS. Every assertion below is about ONE paragraph; if that paragraph + // is renamed or removed they all pass while proving nothing. + size_t notes = 0; + for (size_t at = flat.find("`kMultishot` — FABRICATED"); at != std::string::npos; + at = flat.find("`kMultishot` — FABRICATED", at + 1)) { + ++notes; + } + REQUIRE_MESSAGE(notes == 1, + "expected exactly ONE `kMultishot` — FABRICATED retirement note in " + << LTX2_PIPELINE_HEADER_PATH << ", found " << notes); + + // The sentence that was false. It asserted an upstream absence from our own + // vocabulary, with no positive control, in a SHIPPED header. + CHECK_MESSAGE(flat.find("the only `scene` hit is PySceneDetect") == std::string::npos, + "the retired false claim is back: `scene` is not trainer-only upstream — " + "ltx-core's shipped gemma prompt files carry it, see " + ".agents/specs/ltx25-retire-dead-arms.md §1.1"); + CHECK(flat.find("only `scene` hit") == std::string::npos); + + // And the evidence that replaced it, by the three things a porter needs: that + // the narrative sense is PROMPT GUIDANCE, that it ships in `ltx-core` rather + // than only the trainer, and that the guidance FORBIDS scene cuts. + CHECK_MESSAGE(flat.find("scene cuts") != std::string::npos, + "the retirement note must name the prompt guidance it now rests on"); + CHECK(flat.find("ltx-core") != std::string::npos); + CHECK(flat.find("system_prompt") != std::string::npos); +} + // =========================================================================== // Section 8 — the latent spatial upsampler // ===========================================================================