diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index d1347f1e4..c11178219 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -154,6 +154,9 @@ issue is not yet placed. Keyed record: update in place, never append. | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | | [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#611](https://github.com/mudler/vllm.cpp/issues/611) | `ROAD-V1-LTX25` | `duration_head_path` is accepted by `kKnownLoadExtras` and read by NOTHING, so a caller who points at a duration head silently gets the recipe default. Row `LTX25-RETIRE-DEAD-ARMS` (spec [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md)) refuses it by name and records the full accepted/read inventory of all ten extras | bug | +| [#691](https://github.com/mudler/vllm.cpp/issues/691) | `ROAD-V1-LTX25` | The LTX-2.5 out-of-scope ledger's reachable/marker split gates the MESSAGE TEXT, not reachability: give `kLoraFusion` a product call site and the ledger test still passes while the shipped refusal tells the caller `DECLARED, NOT REQUESTABLE` right after they requested it. Copy the anti-tautological shape of the extras inventory gate | bug | +| [#692](https://github.com/mudler/vllm.cpp/issues/692) | `ROAD-V1-LTX25` | Campaign spec `ltx-2-5.md:291` still records the FABRICATED `multishot` arm as owed after row `LTX25-RETIRE-DEAD-ARMS` retired it; same line needs `CFG parallelism` renamed to multi-GPU. Operator-owned file, so filed rather than edited by that row | bug | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | | [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature | diff --git a/.agents/specs/ltx25-retire-dead-arms.md b/.agents/specs/ltx25-retire-dead-arms.md new file mode 100644 index 000000000..d9ebc217b --- /dev/null +++ b/.agents/specs/ltx25-retire-dead-arms.md @@ -0,0 +1,659 @@ +# LTX-2.5 — retire the arms that do not exist, and refuse the extra that is ignored + +Row: `LTX25-RETIRE-DEAD-ARMS`. Campaign: [`ltx-2-5.md`](ltx-2-5.md) (operator-owned; +not edited by this row). Issues: +[#644](https://github.com/mudler/vllm.cpp/issues/644) items D–I and N, +[#611](https://github.com/mudler/vllm.cpp/issues/611). +Pattern this row is an instance of: [#604](https://github.com/mudler/vllm.cpp/issues/604). + +Upstream pins: + +| Reference | Revision | Local checkout verified at | +|---|---|---| +| Lightricks/LTX-2 | `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | +| huggingface/diffusers | `3a2f35d4efa4c059c8bfb3bc0d6c906264895c81` | `git rev-parse HEAD`, 2026-08-13; re-verified 2026-08-14 | + +Anchors here were derived from those two checkouts by this row. **Three of them +were wrong and were found by review, not by the row** — see §7, which records +what was not re-derived rather than claiming everything was. Where an ABSENCE is +asserted, §1.1 carries the command and the control hits that prove the command +ran; treat any absence in this file without that pairing as unverified. + +## 0. What is wrong today + +`Ltx2UnportedPipelineFeature` (`include/vllm/model_executor/models/ltx2_pipeline.h:580-587`) +is a ledger of seven arms this port refuses by name. Read as a whole it claims +more than is true, in four separate ways: + +1. **One arm is fabricated.** `kMultishot` refuses "multishot generation" and its + enum comment cites *"ltx-pipelines multishot entry points"*. No such entry + point exists. Neither does the symbol, the string, or the concept. +2. **One arm's anchor names something upstream does not do.** `kCfgParallelism` + is anchored to `ltx-pipelines/multigpu`, which contains no CFG batching at + all — the three real parallelisms are orthogonal to guidance, and the recipe + this port runs uses no guidance in the first place. +3. **One arm is stale.** `kVideoEngineWiring` says the end-to-end wiring "is + phase L7, not L5". L7 landed in `cefacd2d0`. +4. **Five of the seven have no product call site.** Recording them as "refused by + name" overstates what exists: nothing a caller can send reaches them. They are + declared-out-of-scope markers, and the code did not say so. + +Separately, and user-visible: `duration_head_path` is accepted by +`kKnownLoadExtras` (`src/vllm/multimodal/ltx2_video.cpp:257`) and **read by +nothing**. A caller who points at a duration head gets the recipe default and is +told nothing. + +## 1. What the references actually contain + +### 1.1 `multishot` — FABRICATED + +Searched as a **subject**, not as our own phrasing (the self-confirming-grep trap, +#604) — and **every absence below was re-derived on 2026-08-14 with a POSITIVE +CONTROL in the same command**, after two of the claims in this section shipped +false and a third was offered as exhaustive when it was not. A +grep that returns nothing proves nothing on its own — the same command has to +return the hits you know are there. The two commands, verbatim: + +```sh +# LTX-2 @ fd4ded7f — `multishot` ABSENCE, controls `scene cuts` / +# `single continuous take` / `DurationPredictor` in the SAME command. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|single continuous take|DurationPredictor' +# -> 0 lines matching multi[_ -]?shot, 22 control lines. +# LTX-2 @ fd4ded7f — every `shot` FILE, control `DurationPredictor`. +git grep -l -E '\bshots?\b' # -> 7 files (see below) +# LTX-2 @ fd4ded7f — `storyboard` ABSENCE, control `shot list` same command. +git grep -n -i -E 'storyboard|shot list' # -> 1 line, README.md:136, the control. +# LTX-2 @ fd4ded7f — what the `multi` prefix actually attaches to. +git grep -h -o -i -E 'multi[_ -]?(gpu|scale|stage|shot|clip|prompt|segment)' \ + | sort | uniq -c | sort -rn +# -> 94 multigpu, 44 multi-GPU, 25 Multi-GPU, 4 MULTI_GPU, 2 multi-prompt. +# diffusers @ 3a2f35d4 — same absence in the ltx2 pipelines package. +git grep -n -i -E 'multi[_ -]?shot|scene cuts|hard cuts|single continuous take' \ + -- src/diffusers/pipelines/ltx2/ +# -> 0 lines matching multi[_ -]?shot, 4 control lines. +``` + +| Query | LTX-2 `fd4ded7f` | diffusers `3a2f35d4` | +|---|---|---| +| `multi[_ -]?shot` (case-insensitive) | 0 hits, alongside 22 control hits | 0 hits, alongside 4 control hits | +| `\bshots?\b`, whole repository | **7 files**, none a generation mode | — | +| `multi[_ -]?(gpu\|scale\|stage\|shot\|clip\|prompt\|segment)` | `multi[-_ ]?gpu` **167**, `multi-prompt` **2** — and nothing else | — | +| `scene`, whole repository | **three senses, none a generation mode** — see §1.1a | only the prompt-guidance sense inside `pipelines/ltx2/` (`scene cuts` ×3, `scene opens` ×4) | +| `storyboard` | 0 hits, alongside its control | 0 hits | + +An earlier revision of this table said the `multi` prefix attaches to +`multi-GPU`/`multi_gpu` **only**. It does not: `multi-prompt` appears twice +(`ltx-pipelines/utils/args.py:618`, "Helps multi-prompt enhance"; and +`utils/denoisers.py:174`, "multi-prompt benchmark panels"). Both are *batching +several captions through one run* — the enhancer's KV cache and a benchmark +panel — not composing several takes into one output, so the conclusion is +unaffected and the "only" was still wrong. Recorded rather than quietly +corrected, for the same reason as everything else in this section. + +The **seven** `shot` files, with what the word means in each. An earlier revision +of this list said five and presented that as exhaustive (review finding M1). The +cause is recorded because it is reusable: that query was +`\bshots?\b` **restricted to `*.py *.md *.json *.yaml`**, and upstream's prompt +files are `.txt`. The filter, not the pattern, hid the answer — and it hid +exactly the two files that matter most here, because they are the ones that ship +at inference. Re-run without a path filter it is seven: + +- `ltx-core/duration_head/duration_head.py:1,5` — "predicts **shot** duration", + i.e. the natural length of ONE camera take. +- `ltx-core/duration_head/__init__.py:1` — same. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_i2v_system_prompt.txt:13,14,17` + — camera **shot type** in the prompt-enhancer's instructions. +- `ltx-core/text_encoders/gemma/encoders/prompts/gemma4_t2v_system_prompt.txt:11,12,15` + — same. +- `ltx-pipelines/utils/blocks.py:804` — `DurationPredictor`, "Predicts **shot** + duration (in frames)". +- `ltx-trainer/src/ltx_trainer/captioning.py:41,42,54,57` — captioning prose, for + describing an EXISTING video as training data. +- `LTX-2/README.md:59,136` — an example prompt ("a medium close-up **shot**") and + the prompting guide ("think like a cinematographer describing a **shot** list"). + +Upstream's "shot" is a single continuous take. There is no multi-shot generation +mode, and nothing that composes several takes into one output. + +The nearest thing in either tree runs the other direction: +`ltx-trainer/captioning.py:54` tells the CAPTIONER "the video contains multiple +shots, describe each one in turn" — an instruction for describing an input video +during dataset preparation, not for generating one. Named here so the next porter +who greps `shots` does not have to re-derive that it is the trainer. + +**The `shot` claim, corrected on the diffusers side too.** This section used to +call `utils.py:217` — "Shot type (exactly one: extreme wide shot / wide shot / +medium shot / medium close-up / close-up / extreme close-up)" — the "stronger +check" because it came from a second, independent reference. It is not +independent: that sentence is **verbatim** LTX-2's own +`gemma4_t2v_system_prompt.txt:12`, and `utils.py:251` is verbatim +`gemma4_i2v_system_prompt.txt:14`. diffusers vendored Lightricks' gemma4 system +prompts wholesale. It corroborates that upstream's `shot` is a camera shot type, +and it does so at a second pin, but it is one source quoted twice, not two +sources agreeing. diffusers' own contribution is the duration head's docstring, +`duration_head.py:83`, "Predicts the natural duration of the **shot** implied by +a caption", and its module list: `pipeline_ltx2`, `_condition`, +`_diffusion_decode`, `_hdr_lora`, `_ic_lora`, `_image2video`, `_latent_upsample`, +plus the components. None is multi-shot. + +`ltx-pipelines`' actual entry points, from `docs/pipelines.md` ("Full reference +for all 11 pipelines") and the module list: +`ti2vid_one_stage`, `ti2vid_two_stages`, `ti2vid_two_stages_hq`, +`ti2vid_two_stages_mgpu`, `ti2vid_two_stages_hq_mgpu`, `distilled`, +`distilled_mgpu`, `ic_lora`, `hdr_ic_lora`, `a2vid_two_stage`, `t2a_one_stage`, +`dubit`, `retake`, `dfr_pipeline`, `keyframe_interpolation`. None is multi-shot. + +### 1.1a `scene` — three senses, and the one that was missed twice + +**This is the third #604 instance this row shipped, and the one that took three +review rounds to find** (review finding B1). The header, this table and the PR +body all said "the only `scene` hit is PySceneDetect in the TRAINER". That is +false at `fd4ded7f`, and it was false in a SHIPPED header — the failure scenario +being a porter who greps `scene`, finds "scene cuts" prohibited in an inference +prompt, and concludes we missed a multi-shot path our own header denied. + +`scene` has three senses upstream, established by reading **every** hit of +`git grep -n -i scene` — no path filter, since a path filter is what hid the +`shot` answer above — rather than by grepping our own vocabulary. The anchors +below are representative of each sense, not a transcript of every line; the +command above is the transcript: + +| Sense | Where | Is it a generation mode? | +|---|---|---| +| `scene-linear` HDR colour (scene-referred light) | `ltx-core/color/hlg.py:1,5,44,74,93,256`, `ltx-core/hdr.py:29,132,146,162,171`, `ltx-pipelines/docs/hdr.md:15,16,34`, `utils/media_io/{color_config,encode,exr}.py`, `utils/args.py:813,861`, `ic_lora.py:422`, `retake.py:178`, `hdr_ic_lora.py:18`, `CHANGELOG.md:24` | no — colour science | +| PySceneDetect splitting | `ltx-trainer/scripts/split_scenes.py`, `ltx-trainer/docs/*`, `ltx-trainer/pyproject.toml:28` (`scenedetect>=0.6.5.2`), `ltx-trainer/AGENTS.md:85`, `.claude/skills/train-model/**` | no — a **training-data** preprocessor, and the only CODE sense | +| narrative prose, i.e. prompt-writing guidance | **`ltx-core`, which ships at INFERENCE**: `text_encoders/gemma/encoders/prompts/gemma3_i2v_system_prompt.txt:5,6,18,20`, `gemma3_t2v_system_prompt.txt:1,5,24,25`, `gemma4_i2v_system_prompt.txt:7,15,23,27`, `gemma4_t2v_system_prompt.txt:5,13,21,25`; plus `README.md:136` and `ltx-trainer/captioning.py:39` | no — instructions to the prompt enhancer | + +The third sense is the one the old sentence denied, and `README.md:136` is +**the same line** the header cites two clauses earlier for "a cinematographer +describing a shot list" — it reads "chronological descriptions of actions and +scenes". + +**The disposition does not move; it is STRENGTHENED.** Read as a subject rather +than as a keyword, the shipped enhancer prompts instruct the model *against* +multi-shot output: + +- `gemma3_i2v_system_prompt.txt:6` — "Inaccurate descriptions may cause scene cuts." +- `gemma3_i2v_system_prompt.txt:18`, `gemma3_t2v_system_prompt.txt:24` — "No + timestamps or cuts: DO NOT use timestamps or describe scene cuts unless + explicitly requested." +- `gemma4_i2v_system_prompt.txt:3` — "**Single continuous take — no hard cuts.**" + +diffusers carries the same four lines (`utils.py:121,161,182,240`), because it +vendored the same prompts. An upstream whose own prompt enhancer is told to +produce one continuous take is affirmative evidence that no multi-shot generation +mode exists — which is what `kMultishot` claimed to refuse. + +**Disposition: RETIRE.** This is a defect in our record, not a gap in our port. +There is nothing to owe, so recording it as owed is the error. The enumerator is +removed and the retirement recorded in the header, in this spec, and in the +commit message — which is where an exception's reason lives (AGENTS.md, "there is +no waiver registry"). + +### 1.2 `int8-convrot` — real absence, deliberately out of scope + +The inference quantization kinds LTX-2 defines, exhaustively +(`ltx-pipelines/utils/quantization_factory.py:22-26`, a `str`-valued enum with an +`assert_never` at `:50`, the file's last line — re-derived 2026-08-14, review +finding L1. The block quoted below starts at the `class` statement on `:22` and +its four members are `:23-26`, which is the range the header comment and the +shipped refusal message cite; `:23-27` was wrong at both ends and this spec was +the last place still carrying it): + +```python +class QuantizationKind(str, Enum): + FP8_CAST = "fp8-cast" + FP8_SCALED_MM = "fp8-scaled-mm" + NVFP4_CAST = "nvfp4-cast" + NVFP4_PREQUANT = "nvfp4-prequant" +``` + +No int8 arm. `convrot` / `conv_rot` / `quarot` / `spinquant` are 0 hits across the +repository, re-run at these pins. + +**Correction, 2026-08-13 (review finding F2).** An earlier revision of this +section put `hadamard` in that 0-hit list and concluded "no rotation of any kind". +That was itself the #604 pattern this row exists to close — an absence asserted +from a grep of our own vocabulary — and it is false. LTX-2 vendors Tri Dao's +fast-Hadamard-transform family in `ltx-kernels` +(`csrc/ops/include/fast_hadamard_transform.h`, `_common.h`, `_special.h`), and it +is explicitly **quantization-coupled**: `DequantHadamardParamsBase:54`, +`QuantHadamardParamsBase:71`, and `QuantMax = 127.0` +(`fast_hadamard_transform_common.h:14`). A fast Hadamard transform *is* the "rot" +of rotation-based quantization, so the word we should have searched for was there +all along. + +What the conclusion rests on instead, re-derived here: + +- **Nothing calls it.** `hadamard_mult_thread` / `hadamard_mult_warp` are defined + at `fast_hadamard_transform_common.h:102,123` and `_special.h:12,31,58,93` and + invoked at **no site in `csrc/`**. The `fast_hadamard_transform_cuda` + instantiations are commented out (`rms_norm_rope_cuda.cu:225-228`), and + `DequantHadamardParamsBase` / `QuantHadamardParamsBase` / `QuantMax` are + declared and referenced nowhere. +- **The live kernel that carries the Hadamard NAME does not rotate.** The only + params struct in use is `NormRopeHadamardParamsBase` + (`rms_norm_rope.cpp:12,19,95-98`, `rms_norm_rope_cuda.cu:188`), and + `norm_rope_cvt_kernel` (`rms_norm_rope_cuda.cu:51-170`) is RMS-norm → RoPE → + store, with no Hadamard step. The name is vestigial, inherited from the fork's + ancestry. +- **And where it is instantiated it is fp8 or bf16, never int8**: + `` and `` + (`rms_norm_rope_cuda.cu:219-223`), matching `rms_norm_split_rope_cuda.cu:196,200`. + +Corrected statement, same conclusion: **upstream vendors a +quantization-coupled Hadamard family, wires none of it, and never exposes a +rotation as a `QuantizationKind`.** So `int8-convrot` is still unreachable +upstream — but by dead code, not by absence. + +`int8` upstream is **unreachable, not absent**. Everything *wired* is trainer-only: + +- `ltx-trainer/src/ltx_trainer/gemma_8bit.py:33-36` — bitsandbytes `LLM.int8()` + for the Gemma backbone during LoRA training. +- `ltx-trainer/src/ltx_trainer/quantization.py:11-15` — optimum-quanto precisions + (`int8-quanto`, `int4-quanto`, …) for the trainer. + +**But one int8 kernel lives in an INFERENCE package** (review finding F1, and the +same false-absence pattern shipped inside the user-visible refusal message): +`ltx-kernels/src/ltx_kernels/blockwise/triton_ops.py` defines `_kernel` at +`:25-37` and `run_quantize_kernel` at `:40-50` — a per-row int8 quantize with fp32 +scales, output dtype `torch.int8` at `:43`, not a `uint8` buffer — aliased +`rowwise_int_quantize_triton` at `:436`. It is **dead**: that alias is its only +reference anywhere in the repository, and `blockwise/functional.py:12-18` +re-exports five other names and not this one. Context for why it is there at all: +the package is a fork of Lightricks' int8 kernel library retargeted to +fp8/fp6/nvfp4, and its custom-op namespace is still literally `q8_kernels_ops` +(`functional.py:25`). + +Every other `int8` match in the repository is `uint8`: pixel buffers, packed NVFP4 +nibbles, block-streaming staging. + +**Disposition: KEEP, re-anchored.** Unchanged by the two corrections above — +nothing wired upstream reaches int8, and no rotation is exposed as a quantization +kind. What changes is the sentence: the refusal now says int8 is UNREACHABLE +(trainer-only for anything wired, plus one dead kernel in `ltx-kernels`) instead +of claiming it appears "only in the trainer". Its *kind* also changes: it is a +declared-out-of-scope marker, not a reachable refusal. + +### 1.3 CFG parallelism — the name describes something upstream does not do + +`cfg` (case-insensitive, word) is **0 hits** in both multi-GPU trees: +`ltx-pipelines/src/ltx_pipelines/multigpu/` and +`ltx-core/src/ltx_core/multigpu/`. + +The three parallelisms upstream actually implements: + +| Form | Anchor | What it splits | +|---|---|---| +| Sequence parallel | `multigpu/sp_builder.py:25` (`SequenceParallelBuilder`), `ltx-core multigpu/transformer/sequence_parallel.py`, all-to-all attention | the token axis of one denoise step | +| Tiled data parallel | `multigpu/tdp_builder.py:25` (`TiledDataParallelBuilder`) | spatial tiles, **upscale stage only** | +| Distributed VAE decode | `ltx-core multigpu/vae/distributed_decoder.py:204-256` (`DistributedVideoDecoder.decode_video`) | latent tiles across ranks, driver blends | + +Upstream states the purpose in its own words +(`ltx-pipelines/docs/multigpu/README.md:7`, inside the ⚠️ block at `:5-16`): + +> **Multi-GPU (MGPU) is a latency tool, not a memory tool.** + +and adds that the mutable transformer is a **full replica on every GPU**, so it +cannot make a checkpoint fit. + +And CFG is not in the path this port runs at all. The distilled recipe denoises +with `SimpleDenoiser` at both stages (`ltx-pipelines/distilled.py:266`, `:295`), +documented as "**single transformer call, no guidance**" +(`utils/denoisers.py:3`); the guider it degenerates to is +`MultiModalGuiderParams(cfg_scale=1.0, stg_scale=0.0, modality_scale=1.0)` +(`utils/denoisers.py:25-26`), "only runs the conditioned pass and returns cond +unchanged". A `cfg_scale` of 1.0 is one pass, so there is no second pass to place +on a second GPU. + +Stronger still, and true even for the recipes that *do* guide: upstream's guided +path is not distributed either. `utils/denoisers.py:7-8` records that +`GuidedDenoiser` and `FactoryGuidedDenoiser` share `_guided_denoise`, "which +batches all guidance passes into a single transformer call". So upstream's CFG is +a batch dimension, never a second device — which is why no amount of `multigpu` +code was ever going to be CFG parallelism, and why the old enumerator name could +not have been right for any recipe. + +**Disposition: RENAME + re-anchor.** `kCfgParallelism` → `kMultiGpuParallelism`, +anchored to the three real forms, with the reason it is out of scope stated as +what it is: a single-node multi-GPU **latency** feature, on a port whose target is +one GB10. + +### 1.4 `kLoraFusion` — real upstream, correctly refused + +Verified present: `ltx-core/loader/primitives.py:160` +(`class LoraPathStrengthAndSDOps(NamedTuple)`), exported at +`loader/__init__.py:14,46`, consumed by `loader/single_gpu_model_builder.py:21` +and `block_streaming/builder.py:34,90`, fused by `loader/fuse_loras.py`. The +anchor stands; only its *kind* is corrected (marker, not reachable refusal). + +### 1.5 `kVideoEngineWiring` — LANDED + +`cefacd2d0` ("feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that +is no longer MiniMax-only (#435) (#641)", 2026-08-13) shipped exactly this: the +composition through `vllm::multimodal::VideoEngine`, reachable through the C ABI +as video family `ltx-2.5`. `include/vllm/multimodal/ltx2_video.h:12` already +speaks of the refusal in the past tense. + +**Disposition: RETIRE.** A refusal whose subject shipped is a false statement, not +a record of debt. + +### 1.6 The five with no product call site + +`grep` over `src/`, `include/`, `examples/`, `tests/` for every enumerator: + +| Enumerator | Product call site | +|---|---| +| `kTemporalUpsampler` | `src/vllm/model_executor/models/ltx2_upsampler.cpp:395` | +| `kBetaScheduler` | `src/vllm/model_executor/models/ltx2_pipeline.cpp:199` | +| `kLoraFusion` | **none** | +| `kMultishot` | **none** | +| `kInt8ConvRot` | **none** | +| `kCfgParallelism` | **none** | +| `kVideoEngineWiring` | **none** | + +The five are enumerated only by `tests/vllm/models/test_ltx2_pipeline.cpp:1237-1244`. +There is no request field, load extra, or CLI flag that asks for a LoRA, an +int8-convrot checkpoint, or a second GPU, so no caller can trip them. + +That is not a defect on its own — a declared boundary is worth having. The defect +is calling it a refusal. The header and the messages now distinguish: + +- **reachable refusal** — a product path constructs the condition and throws; +- **declared-out-of-scope marker** — a record of what upstream has and this port + does not, reached only by the ledger test. + +## 2. `duration_head_path` (#611) + +### 2.1 The full `kKnownLoadExtras` audit + +Every key the family accepts, and whether any code reads it. Reader anchors are in +`src/vllm/multimodal/ltx2_video.cpp` unless noted, **as of `3c6706cd0`** — this +table is a dated record, not a live one; see the note below it. + +| Key | Constant | Reader | Status | +|---|---|---|---| +| `audio_prompt_embeds_path` | `kLtx2AudioPromptEmbedsExtra` | `:969`, `:982` | READ | +| `pipeline_kind` | `kLtx2PipelineKindExtra` | `:805` | READ | +| `model_version` | `kLtx2ModelVersionExtra` | `:789` | READ | +| `allow_unported_modules` | `kLtx2AllowUnportedExtra` | `:638` | READ | +| `max_phase` | `kLtx2MaxPhaseExtra` | `:807` | READ | +| `dit_config_path` | `kLtx2DitConfigPathExtra` | `:693` | READ | +| `prompt_embeds_valid_rows` | `kLtx2PromptValidRowsExtra` | `:1010` | READ | +| `encoder_config_path` | `kLtx2EncoderConfigPathExtra` | `:864` | READ | +| `upsampler_path` | (literal) | `:839` | READ | +| `duration_head_path` | `kLtx2DurationHeadPathExtra` | **none** | **was ACCEPTED AND IGNORED; now refused by name** | + +Nine of ten are wired. `duration_head_path` is the only defect, so the sweep this +row owes is complete and closes the "the sweep that found this one did not cover +them all" clause of #611. + +**These numbers rotted twice inside this one PR (review finding F3).** The first +set shipped off by 37 lines and named nine lines that read nothing — cited, worst +of all, *in the file they were wrong about*. Correcting them, a merge of +`origin/main` moved the real ones by another 25. A hand-written `file:line` is +stale by the next commit, so the anchors are no longer maintained by hand: the +`READER ANCHORS` comment in `ltx2_video.cpp` carries the one live copy, and +`test_ltx2_video`'s "the recorded reader anchors are the ones in the source" case +derives them from the source on every run and fails with the replacement list +printed. The table above stays because a spec is a dated record and the SHA is +named; the code comment is the live one. + +One documentation gap found by the same sweep and fixed here: +`docs/USAGE.md:1650-1654` lists the LTX-2.5 extras and omits `encoder_config_path` +entirely, though it is defined and read. + +### 2.2 Why it is inert + +`ltx2_duration_head.h` / `ltx2_duration_head.cpp` port `DurationHead` and +`AttentionPooler` and gate them as bricks. Nothing in `ltx2_video.cpp` constructs +one. The AUTO-duration path (`resolve_num_frames`, `ltx-pipelines/utils/blocks.py`) +therefore cannot run, and `Generate` computes `frames` from +`duration_seconds * fps` directly (`ltx2_video.cpp:1152`). + +The stated reason for this moved twice already (#604's pattern, and the finding +that produced #611). The reason recorded here is the current one: **no head is +constructed**, so a supplied path is a file the engine never opens. + +### 2.3 Fix + +Refuse `duration_head_path` **by name** when it is supplied and non-empty, with a +message that says the head is unported, that the recipe default would otherwise be +substituted silently, and what to use instead (`num_frames`, or `duration`, which +is exact arithmetic against the recipe frame rate). + +Rejected alternative: dropping the key from `kKnownLoadExtras`. That produces +"unknown load extra", which is *wrong* — the key is defined by this family and its +meaning is understood; what is missing is the implementation. AGENTS.md requires +"a message naming the missing piece", and "unknown key" does not name it. + +Rejected alternative: reading it and constructing the head. That is the real fix +and it stays owed — it needs the connector-output plumbing the head consumes +(`duration_head.py:89-118` takes audio and/or video connector token states), which +is a different row. Refusing is the cheap correct answer until then. + +## 3. Scope + +**In.** + +1. Retire `kMultishot` and `kVideoEngineWiring`; record both retirements in the + header, this spec, and the commit message. +2. Rename `kCfgParallelism` → `kMultiGpuParallelism` and re-anchor its message to + the three real parallelisms plus the reason CFG is not in our path. +3. Re-anchor `kInt8ConvRot`'s comment and message to record the verified absence + at these pins. +4. Split the ledger into reachable refusals and declared-out-of-scope markers, in + the header comment, in the messages, and in the test. +5. Refuse `duration_head_path` by name (#611). +6. `docs/USAGE.md`: the extras paragraph, corrected on both counts. +7. Added by the review repair: correct the int8 sentence in the shipped refusal + (§1.2), correct this spec's own false `hadamard` absence (§1.2), derive the + reader anchors instead of writing them by hand (§2.1), and split the + `docs/FEATURES.md` row so the public surface stops calling the two REACHABLE + arms unrequestable (§1.6). No disposition changes. +8. Added by the THIRD review round: correct the false `scene` absence in the + shipped header, in §1.1's table and in the PR body (§1.1a); correct the `shot` + enumeration from five files to seven and record the path filter that caused it + (§1.1); correct `quantization_factory.py:23-27` to `:22-26` (§1.2); replace + §7's "every claim in §1 was re-derived" with what actually happened; and gate + the header's retirement note with a RED-first case so the sentence cannot come + back. No disposition changes — the `scene` evidence strengthens the + `kMultishot` retirement rather than weakening it. + +**Out.** + +- `.agents/specs/ltx-2-5.md` — operator-owned. This row does not edit it. Its §2 + "Out" list still names `multishot`; correcting that is the operator's edit, and + this spec is the record it would cite. Filed as + [#692](https://github.com/mudler/vllm.cpp/issues/692) by the review repair so + `main` stops carrying the claim unowned. +- Making the reachable/marker split gate REACHABILITY rather than message text. + The ledger test asserts what each message says; nothing asserts that a marker + still has no product call site, so giving `kLoraFusion` one would leave the test + green while the shipped refusal told a caller it is not requestable immediately + after they requested it. Filed as + [#691](https://github.com/mudler/vllm.cpp/issues/691), with the extras inventory + gate named as the anti-tautological shape to copy. +- Constructing a duration head, and the AUTO-duration path. Stays owed (#611 + remains open after this row, retitled by the fix rather than closed by it — + see §7). +- `kTemporalUpsampler` (row `LTX25-TEMPORAL-UPSAMPLER`), image conditioning (row + `LTX25-IMAGE-COND-FIX`), tiled decode (row `LTX25-TILED-DECODE`), AdaLN claims + (row `LTX25-ADALN-CLAIMS`). No file in this diff is theirs. + +## 4. Tests + +RED first for the behavioural change; the records changes are gated by the ledger +test's own assertions. + +1. **`ltx2 duration_head_path is REFUSED by name` (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`).** Builds a valid load, adds + `extras["duration_head_path"]`, and requires a throw whose message names the + key, names the duration head as the missing piece, and names the alternative. + **RED before the fix**: the load succeeds, because nothing reads the key. That + is the defect stated as a test. +2. **`ltx2 an accepted load extra is READ by something` (new).** Asserts the + inventory of §2.1 does not silently grow: every key in `kKnownLoadExtras` + either round-trips through a reader or is refused by name. Implemented as the + two known-inert keys being refused and the rest being accepted, so adding a + tenth decorative key fails. +3. **`ltx2 every out-of-scope feature is refused BY NAME`** (existing, updated). + The list drops from 7 to 5 entries — a CHANGED COUNT, reported as such — and + splits into `reachable` and `markers`, with the marker messages required to say + they are not requestable. Adds a guard that no refusal message mentions + `multishot` again. + +Three more added by the review repair, each RED before its fix: + +4. **The int8 marker's own evidence** (an addition to case 3). The message shipped + the sentence "int8 appears upstream only in the trainer", which §1.2 shows is + false. The case now requires the message NOT to contain it and to name + `ltx-kernels`. **RED before**: both assertions failed against the shipped text. +5. **`ltx2 video: the recorded reader anchors are the ones in the source`** (new, + `tests/vllm/multimodal/test_ltx2_video.cpp`). Derives each served key's reader + line from `ltx2_video.cpp` itself and compares it with the `READER ANCHORS` + comment, and separately requires every `kLtx2DurationHeadPathExtra` reference + after the array to sit inside `CheckUnservedExtras`. It hard-codes no line + number of its own, so it cannot go stale; the obligation it creates falls on + whoever moves a reader, in the file they are already editing. **RED before**: + `Recorded: []. Actual: [638 693 789 805 807 839 864 969 1010]`. +6. **`ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable`** (new, + `tests/vllm/models/test_ltx2_pipeline.cpp`). The public surface had re-merged + the reachable/marker split into one "Declared, not requestable" row naming the + temporal upsampler and `BetaScheduler`. Anti-vacuous: it also requires exactly + one such LTX-2.5 row to exist, so renaming the row away fails it. **RED + before**: both reachable arms found on `FEATURES.md:328`. + +One more added by the third review round: + +7. **`ltx2 the kMultishot retirement note states the scene evidence correctly`** + (new, `tests/vllm/models/test_ltx2_pipeline.cpp`). Reads + `include/vllm/model_executor/models/ltx2_pipeline.h` through a new + `LTX2_PIPELINE_HEADER_PATH` compile definition, flattens the comment block so a + reflow cannot make it vacuous, and requires the retired false sentence to be + absent and the replacement evidence — `scene cuts`, `ltx-core`, + `system_prompt` — to be present. Anti-vacuous: it first requires **exactly one** + ``kMultishot` — FABRICATED` note in the file, so deleting or renaming the + paragraph fails rather than passes. **RED before**: 4 of its 8 assertions + failed on `b718f580f` — `flat.find("the only \`scene\` hit is PySceneDetect")` + returned 29407 where npos was required, and `scene cuts` / `system_prompt` + both returned npos. **Anti-vacuity proven by mutation**: renaming the note's + marker in a staged copy gives `REQUIRE( 0 == 1 )`; the header was restored and + verified by sha256. + + What it can and cannot prove is stated in the case's own comment: it holds the + header's TEXT, so the false sentence cannot return, but it cannot verify the + upstream claim — no upstream checkout exists in this tree. That verification is + §1.1's commands with their positive controls, run against the two pins, and it + is not automatable here. Saying so is better than a test that appears to gate + it and does not. + +## 5. Risks + +| Risk | Mitigation | +|---|---| +| Removing an enumerator breaks an out-of-tree caller | `Ltx2UnportedPipelineFeature` is internal (`include/vllm/model_executor/`), not part of `include/vllm.h`. Grep shows 3 call sites, all in-tree. | +| Refusing `duration_head_path` breaks a working caller | It cannot: no code reads it, so no caller was getting anything from it. A caller who passes it today is being silently ignored, which is the bug. | +| The ledger test's case count changes and reads as a regression | Stated up front in the gate report: 7 → 5 entries in one case, plus 2 new cases. The exit code is the authority. | +| A concurrent LTX row edits the same header | Four live rows named; none owns `ltx2_pipeline.h:575-590`, `ltx2_video.cpp:250-300`, or the ledger test case. `LTX25-TEMPORAL-UPSAMPLER` owns `kTemporalUpsampler`, which this row does not touch. | + +## 6. Stop conditions + +- Return `NEEDS_DECISION` if `multishot` turns out to exist under vocabulary not + searched here — the retirement is then wrong and the arm must be re-anchored + instead. +- Return `NEEDS_DECISION` on a collision with a live LTX row. +- Do not close #611 if the refusal cannot be made to fail RED first. + +## 7. Outcome + +**Three of the claims in §1 were NOT re-derived, and three rounds of review found +them one at a time.** The sentence that stood here said "every claim in §1 was +re-derived"; it was written before the second and third rounds, and it was not +true when it was written either. What actually happened: + +- Round 1 found the `hadamard` absence in §1.2 false, and the "int8 appears only + in the trainer" sentence false **in the shipped refusal message**. +- Round 2 found nine reader anchors in §2.1 stale, cited in the file they were + wrong about. +- Round 3 found `scene` asserted trainer-only **in the shipped header**, the + `shot` file list presented as exhaustive at five when it is seven, and a + `quantization_factory.py` range wrong at both ends. Re-deriving the rest of + §1.1's table in the same pass turned up a fourth: the `multi` prefix was + recorded as `multi-GPU`/`multi_gpu` **only**, and `multi-prompt` is there twice. + Nobody had asked about that row; it was wrong because the row had never been + re-run either. + +Every one of the three is the same defect: an absence asserted from our own +vocabulary, with no positive control, and in two cases behind a path filter that +excluded the files carrying the answer. **This row's stated purpose is retiring +instances of [#604](https://github.com/mudler/vllm.cpp/issues/604), and it +shipped three of them** — two of those into files a user or a porter reads. That +self-record is worth more than the claim it replaces, because the claim was the +mechanism: writing "every claim was re-derived" is what made the next reader stop +checking. §1.1 now carries the commands with their positive controls inline, so +the claim is reproducible instead of asserted, and +`test_ltx2_pipeline`'s "the kMultishot retirement note states the scene evidence +correctly" holds the header's text so the false sentence cannot come back. + +Results, as they now stand: + +- `multishot`: **fabricated, confirmed** — and the confirmation is now stronger + than the original. 0 hits for the term in either reference against 22 and 4 + control hits in the same commands; the only upstream sense of "shot" is one + camera take across all seven files; and the shipped prompt-enhancer prompts + instruct the model to keep a "Single continuous take — no hard cuts" + (§1.1, §1.1a). Retired. +- `int8-convrot`: **unreachable upstream, and the first wording of that was wrong.** + Four inference quantization kinds exist and none is int8, and no rotation is + exposed as one — but int8 is not trainer-only (a dead per-row int8 kernel lives + in the `ltx-kernels` inference package) and `hadamard` is not 0 hits (a whole + quantization-coupled Hadamard family is vendored, and wired to nothing). Both + corrections are in §1.2; neither moves the disposition. Kept, re-anchored. +- CFG parallelism: **the name was wrong, the exclusion is right.** 0 `cfg` hits in + either multi-GPU tree; upstream's own README calls MGPU a latency tool; the + distilled recipe runs `SimpleDenoiser` with no guidance at all, so there is no + CFG pass to parallelize (§1.3). Renamed and re-anchored. +- `kVideoEngineWiring`: **stale, confirmed.** L7 landed in `cefacd2d0`. Retired. +- Five enumerators had no product call site (§1.6). The ledger now says which + kind each is, because "refused by name" overstated a marker. +- `duration_head_path` was the **only** unread key of ten (§2.1). Now refused by + name. The full inventory is the durable half of this row: it means the next + person asking "which extras are decorative?" reads a table instead of grepping. + +What this row deliberately did **not** do: construct a duration head. #611 stays +open for that, with its user-visible half — silent substitution of the recipe +default — closed. + +**The review's own findings, recorded because they are the point of the row.** +Across three rounds every finding was evidence accuracy, not logic: a false +absence inside the shipped refusal (§1.2), a false absence in this spec's §1.2 — +the one search term that mattered — nine stale reader anchors cited in the file +they were wrong about (§2.1), a public-doc row that re-merged the split the ledger +had just made, a false absence inside the shipped HEADER (§1.1a), a five-item list +offered as exhaustive when it is seven (§1.1), and a line range wrong at both ends +(§1.2). A row whose subject is #604 committed three instances of #604, and each +one survived at least one review that had already been told to look for exactly +this. That is not irony to note and move past; it is why the anchors are now +derived rather than written, why the int8 absence is stated as UNREACHABLE with +its dead kernel named, why the `hadamard` search this spec got wrong is spelled +out in full rather than quietly corrected, and why every absence in §1.1 now ships +with the command that proves it AND the control hits that prove the command ran. + +The transferable lesson, in one line: **a path filter is an absence claim too.** +The `shot` enumeration is the case where this is provable rather than inferred — +the pattern `\bshots?\b` was right and the filter `*.py *.md *.json *.yaml` was +wrong, over a tree whose prompt files are `.txt`, and it returns exactly the five +files that were recorded. (The `scene` claim's original command was never written +down, so why *it* missed `ltx-core` is not recoverable; that it was never +reproduced is the point.) A grep that returns nothing has to be re-run without its +narrowing, and with a control, before it is written down as an absence. + +## Now + +Row `LTX25-RETIRE-DEAD-ARMS` is `DONE`. The ledger carries five entries, split by +kind; `duration_head_path` is refused by name with a RED-first test; the +`kKnownLoadExtras` inventory is recorded in §2.1; the header's `kMultishot` +retirement note is gated by `test_ltx2_pipeline` against the false `scene` +absence it shipped (§1.1a). `.agents/specs/ltx-2-5.md` §2 "Out" still lists +`multishot` and is the operator's to correct, citing §1.1 here. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 1ecf0a10a..35c7abfe7 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -329,6 +329,9 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the |---|---|---| | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | +| LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | +| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The temporal x2 latent upsampler and `BetaScheduler`. A product path constructs each condition and throws, naming the missing piece | +| LTX-2.5 arms nothing can request | Declared, not requestable | LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU. No request field, load extra or CLI flag asks for these. `multishot` was RETIRED: no such entry point exists in either reference | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started | diff --git a/docs/USAGE.md b/docs/USAGE.md index 3e1cd9d28..d0695fb46 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1678,14 +1678,37 @@ knobs from `extras`. H3 takes `partition`. LTX-2.5 takes `audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind` (default `distilled_two_stage`), `model_version` (only for a checkpoint that -declares none), `dit_config_path`, `allow_unported_modules`, `max_phase`, -`prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a -family does not define is refused, never ignored. One caveat inside that set: -`duration_head_path` is accepted but INERT — the duration head is ported and gated -as a brick, nothing in the video engine constructs one, and no code reads that -key, so supplying it neither loads a head nor enables an AUTO duration. Give +declares none), `dit_config_path`, `encoder_config_path`, +`allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`, +`upsampler_path` and `duration_head_path`. An extra a family does not define is +refused, never ignored. One caveat inside that set: `duration_head_path` is +defined but UNSERVED — the duration head is ported and gated as a brick, and +nothing in the video engine constructs one — so supplying it is **refused by +name** at load rather than accepted. It used to be accepted and read by nothing, +which silently substituted the recipe default for the file you named. Give `num_frames` (or `duration`, which is exact arithmetic against the recipe's frame -rate) instead. +rate) instead. Every other key in that list reaches a reader. + +Two LTX-2.5 arms are refused where a render would otherwise silently downgrade: +the temporal x2 latent upsampler and `BetaScheduler`. Both are reachable — a +render asking for either gets a refusal naming the missing piece. Three more are +recorded as out of scope but are **not requestable**, so no flag or extra can +reach them: LoRA fusion, `int8-convrot`, and single-node multi-GPU. Their messages +say `DECLARED, NOT REQUESTABLE` so the two kinds are not confused. `int8-convrot` +in particular is a ComfyUI-ecosystem format: upstream LTX-2's own inference +quantization kinds are `fp8-cast`, `fp8-scaled-mm`, `nvfp4-cast` and +`nvfp4-prequant`, and nothing wired upstream reaches int8 at all. + +What is **not** on that list, and why: **multi-shot or multi-scene generation.** +A request that composes several camera takes into one output has no flag here +because upstream LTX-2 has no such mode to mirror — its `shot` is one continuous +take, and its own prompt-enhancement prompts instruct the model to keep a "single +continuous take" and not to describe scene cuts. `scene` does appear across the +upstream tree, in three unrelated senses (`scene-linear` HDR colour, PySceneDetect +in the trainer's dataset preprocessor, and that prompt-writing guidance); none of +them is a generation mode. This port carried a `multishot` refusal until +2026-08-13, which was a defect in our own record rather than a gap, and it was +retired. Generate one take per request. `prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real tokens; absent, every row is. It matters because the embeddings connector diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index e17bb6953..3ec787a3e 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -168,6 +168,9 @@ int main(int argc, char** argv) { else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + // Kept although the library REFUSES this extra by name (#611): the duration + // head is unported, and forwarding the flag gets the caller that named + // refusal instead of "unknown option", which says nothing about why. else if (f == "--duration-head") SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str())); else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str())); diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 264668416..d33ff281b 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -571,20 +571,72 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, const std::string& model_version); // --------------------------------------------------------------------------- -// Out of scope for L5, refused by name (spec section 2, "Out") +// Out of scope, refused by name (spec section 2 "Out"; the 2026-08-13 grounding +// pass is .agents/specs/ltx25-retire-dead-arms.md, row LTX25-RETIRE-DEAD-ARMS) // --------------------------------------------------------------------------- // Each of these renders something plausible if it is silently downgraded, which // is why none of them falls back. `Ltx2RefuseUnportedPipelineFeature` throws with -// a message naming the missing piece and the phase or row that owes it. +// a message naming the missing piece and the row that owes it. +// +// TWO KINDS live here, and conflating them overstated what this port refuses: +// +// REACHABLE REFUSAL — a product path constructs the condition and throws, so a +// caller CAN trip it. `kTemporalUpsampler` (ltx2_upsampler.cpp:395) and +// `kBetaScheduler` (ltx2_pipeline.cpp:199) are the two. +// +// DECLARED-OUT-OF-SCOPE MARKER — no request field, load extra or CLI flag asks +// for it, so nothing outside the ledger test reaches it. It is a record of what +// upstream HAS and this port does NOT, which is worth keeping; calling it a +// refusal is what was wrong. `kLoraFusion`, `kInt8ConvRot` and +// `kMultiGpuParallelism` are markers, and their messages say so. +// +// TWO ENUMERATORS WERE RETIRED on 2026-08-13, recorded here because the +// retirement IS the record — a reader who finds them in git history needs to know +// they did not simply move: +// +// `kMultishot` — FABRICATED. It refused "multishot generation" and cited +// "ltx-pipelines multishot entry points". No such entry point, symbol or string +// exists in Lightricks/LTX-2 @ fd4ded7f or huggingface/diffusers @ 3a2f35d4. +// Searched as a SUBJECT rather than by our own phrasing: upstream's only sense +// of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration"; +// README.md:136 "a cinematographer describing a shot list"). `scene` has THREE +// senses upstream and none is a generation mode: `scene-linear` HDR colour +// (ltx-core color/hlg.py, hdr.py), PySceneDetect in the TRAINER — the only CODE +// sense — and prompt-writing guidance, which ships at INFERENCE inside +// `ltx-core`, in text_encoders/gemma/encoders/prompts/ as +// gemma{3,4}_{i2v,t2v}_system_prompt.txt. That third sense is why the +// retirement HOLDS rather than being undermined: those prompts tell the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:6,18, gemma3_t2v:24, gemma4_i2v:3). A defect in our record is not +// a gap in our port, so there was nothing to owe. +// +// Recorded because it is the row's own subject: this paragraph used to claim +// that `scene` appeared upstream ONLY as PySceneDetect in the trainer. It was +// an absence asserted from our own vocabulary with no positive control — #604 — +// shipped in the header of the row that exists to retire #604 instances, and it +// took a third review round to find. The derivation, with its positive control +// in the same command, is .agents/specs/ltx25-retire-dead-arms.md §1.1. +// +// `kVideoEngineWiring` — LANDED. It said the end-to-end composition through +// `vllm::multimodal::VideoEngine` "is phase L7, not L5"; L7 shipped in +// `cefacd2d0`. A refusal whose subject shipped is a false statement. enum class Ltx2UnportedPipelineFeature { - kTemporalUpsampler, // model/upsampler with temporal_upsample=True - kLoraFusion, // loader/LoraPathStrengthAndSDOps - kMultishot, // ltx-pipelines multishot entry points - kInt8ConvRot, // ComfyUI-only quantization - kCfgParallelism, // ltx-pipelines/multigpu - kVideoEngineWiring, // end-to-end through vllm::multimodal::VideoEngine (L7) - kBetaScheduler, // components/schedulers.py:91-120 + // Reachable refusals. + kTemporalUpsampler, // model/upsampler with temporal_upsample=True + kBetaScheduler, // components/schedulers.py:91-120 + // Declared-out-of-scope markers. + kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps), + // fused by loader/fuse_loras.py + kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four + // inference kinds upstream defines are fp8-cast / fp8-scaled-mm / + // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-26). + // `convrot` is nowhere at all; int8 is UNREACHABLE rather than + // absent — trainer-only for anything wired, plus one DEAD kernel + // in ltx-kernels (triton_ops.py:35,43). §1.2 of the row spec + kMultiGpuParallelism, // ltx-pipelines/multigpu — sequence-parallel, tiled data + // parallel and distributed VAE decode. NOT CFG batching: + // zero `cfg` hits in either multigpu tree }; [[noreturn]] void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature); diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 0953aebff..11c4045b9 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -9,10 +9,13 @@ // Phases L2-L6 shipped a DiT forward, a text feature extractor, two VAEs, a // vocoder, an upsampler, a duration head, and a pipeline COMPONENT library — // schedules, noisers, steppers, guiders, patchifiers, recipes. Nothing drove -// them: `Ltx2RefuseUnportedPipelineFeature(kVideoEngineWiring)` refused the +// them: an `Ltx2UnportedPipelineFeature::kVideoEngineWiring` refusal named the // composition BY NAME and named this phase as its owner. This TU is that // composition and nothing else. It adds no numerics; every line either resolves -// a parameter, moves a buffer, or calls a brick that already has a golden. +// a parameter, moves a buffer, or calls a brick that already has a golden. (That +// enumerator was RETIRED in row LTX25-RETIRE-DEAD-ARMS once L7 landed in +// `cefacd2d0` — a refusal whose subject shipped is a false statement, not a +// record of debt. It is named here in the past tense on purpose.) // // ─── WHAT IT IS A PORT OF (file:line on BOTH sides) ────────────────────────── // Upstream: Lightricks/LTX-2 @ fd4ded7, packages/ltx-pipelines/src/ltx_pipelines/ diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index e8d61fa1d..bea3ea40c 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1138,30 +1138,63 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { const std::string owed = - " Not ported by phase L5; recorded as owed in .agents/specs/ltx-2-5.md."; + " Recorded as owed in .agents/specs/ltx-2-5.md; grounded against Lightricks/LTX-2 " + "fd4ded7f in .agents/specs/ltx25-retire-dead-arms.md."; + // A marker is not a refusal a caller can trip, and saying so is the point: this + // enum used to read as six live refusals when only two had a product call site. + const std::string marker = + " DECLARED, NOT REQUESTABLE: no request field or load extra asks for this, so nothing " + "but the out-of-scope ledger reaches this message."; switch (feature) { case Ltx2UnportedPipelineFeature::kTemporalUpsampler: Refuse("ltx2: the temporal x2 latent upsampler (model/upsampler/model.py:55-72, " "temporal_upsample=True) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kLoraFusion: - Refuse("ltx2: LoRA fusion (loader/LoraPathStrengthAndSDOps) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kMultishot: - Refuse("ltx2: multishot generation is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kInt8ConvRot: - Refuse("ltx2: the int8-convrot quantization (ComfyUI-only) is out of scope." + owed); - case Ltx2UnportedPipelineFeature::kCfgParallelism: - Refuse("ltx2: CFG / multi-GPU parallelism (ltx-pipelines/multigpu) is out of scope." + - owed); - case Ltx2UnportedPipelineFeature::kVideoEngineWiring: - Refuse("ltx2: end-to-end wiring through vllm::multimodal::VideoEngine is phase L7, not " - "L5." + - owed); case Ltx2UnportedPipelineFeature::kBetaScheduler: Refuse("ltx2: BetaScheduler (components/schedulers.py:91-120) is not ported. It inverts " "a Beta CDF through scipy.stats.beta.ppf, and no ltx-pipelines entry point " "constructs it." + owed); + case Ltx2UnportedPipelineFeature::kLoraFusion: + Refuse("ltx2: LoRA fusion (ltx-core loader/primitives.py:160 LoraPathStrengthAndSDOps, " + "fused by loader/fuse_loras.py) is out of scope." + + marker + owed); + case Ltx2UnportedPipelineFeature::kInt8ConvRot: + // VERIFIED UNREACHABLE so nobody re-audits it, and stated as UNREACHABLE rather + // than ABSENT because absent is what the first version of this message claimed + // and it was false. At LTX-2 @ fd4ded7f: `convrot` / `conv_rot` / `quarot` / + // `spinquant` really are 0 hits, and the four inference quantization kinds are + // exhaustive (quantization_factory.py:23-26, `assert_never` at :50). But int8 + // is NOT trainer-only. `ltx-kernels` — an inference package — carries a per-row + // int8 quantize kernel with fp32 scales (blockwise/triton_ops.py:25-50, out + // dtype `torch.int8` at :43), aliased `rowwise_int_quantize_triton` at :436. + // That alias is its ONLY reference: blockwise/functional.py:12-18 re-exports + // five names and not this one, so nothing constructs it. The package is a fork + // of Lightricks' int8 kernel library retargeted to fp8/fp6/nvfp4 — its custom-op + // namespace is still literally `q8_kernels_ops` (functional.py:25) — and the + // int8 half is what was left behind. Nothing wired reaches int8, which is why + // the disposition is unchanged; only the sentence was wrong. + Refuse("ltx2: the int8-convrot quantization is out of scope. It is a ComfyUI-ecosystem " + "format, not an LTX-2 arm: upstream's own inference kinds are fp8-cast, " + "fp8-scaled-mm, nvfp4-cast and nvfp4-prequant (ltx-pipelines/utils/" + "quantization_factory.py:23-26), and int8 is UNREACHABLE upstream — trainer-only " + "for anything wired (ltx-trainer gemma_8bit.py:33-36, quantization.py:11-15), plus " + "one DEAD per-row int8 quantize kernel in the ltx-kernels inference package " + "(blockwise/triton_ops.py:35,43, reached only by its own alias at :436)." + + marker + owed); + case Ltx2UnportedPipelineFeature::kMultiGpuParallelism: + // The old spelling was `kCfgParallelism`, which named something upstream does + // not do. There is no CFG pass to split here in the first place: the distilled + // recipe denoises with SimpleDenoiser at both stages (distilled.py:266,295), + // "single transformer call, no guidance" (utils/denoisers.py:3). + Refuse("ltx2: single-node multi-GPU parallelism (ltx-pipelines/multigpu) is out of " + "scope. Upstream has three forms and none of them is CFG batching: " + "sequence-parallel (multigpu/sp_builder.py:25), tiled data parallel " + "(multigpu/tdp_builder.py:25, upscale stage only) and distributed VAE decode " + "(ltx-core multigpu/vae/distributed_decoder.py:204-256). It is a LATENCY tool, " + "not a memory tool (docs/multigpu/README.md:5-16), and this port targets one " + "GB10." + + marker + owed); } Refuse("ltx2: unknown unported pipeline feature." + owed); } diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index c0eaf389f..d134855cc 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -247,15 +247,36 @@ int64_t ExtraInt(const std::map& extras, const std::st } } +// The one key this family DEFINES and does not SERVE. `Ltx2DurationPredict` is +// ported and gated as a brick (`ltx2_duration_head.h`), but nothing here +// constructs one, so a supplied path names a file the engine never opens. +constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; + // Every extra key this family DEFINES. An extra outside this set is refused // rather than ignored, for the same reason H3 refuses one // (minimax_h3_video.cpp): a mistyped knob that is silently dropped renders the // DEFAULT and looks like the feature not working. +// +// DEFINED IS NOT THE SAME AS SERVED, and conflating the two was #611: nine of +// these ten reach a reader, and `duration_head_path` reached none, so supplying a +// duration head substituted the recipe default in silence — the failure mode this +// very list exists to prevent, one level in. It stays in the list because the +// family DOES define the key and DOES know what it means; `CheckUnservedExtras` +// refuses it by name instead, which is a different and truer message than +// "unknown load extra". The full audit is in +// .agents/specs/ltx25-retire-dead-arms.md §2.1. +// +// The first hand-written set of these anchors named nine lines that were readers +// of NOTHING, in this very file, and a later merge moved the real ones again. So +// they are no longer trusted: the list below is derived from this file on every +// run and compared, and the failure prints the replacement to paste in. +// READER ANCHORS (derived and gated by test_ltx2_video): +// 638 693 789 805 807 839 864 969 1010 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, - "upsampler_path", "duration_head_path", + "upsampler_path", kLtx2DurationHeadPathExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -294,6 +315,27 @@ void CheckKnownExtras(const std::map& extras) { } } +// A key this family DEFINES but does not SERVE, refused BY NAME when supplied +// (#611). The alternative — accepting it — is the worst of the three options: +// worse than refusing, and worse than not defining the key, because the caller +// pointed at a specific file and got the recipe default with no diagnostic. +// +// Deliberately NOT the "unknown load extra" path above. That message says the +// family does not define the key, which is false here and would send the reader +// looking for a typo instead of for the unported head. +void CheckUnservedExtras(const std::map& extras) { + const std::string duration_head = VideoExtra(extras, kLtx2DurationHeadPathExtra); + if (!duration_head.empty()) { + Fail("the '" + std::string(kLtx2DurationHeadPathExtra) + "' extra names '" + duration_head + + "', but the duration head is NOT WIRED into this engine: `Ltx2DurationPredict` is ported " + "and gated as a brick (ltx2_duration_head.h, upstream duration_head.py:89-118) and " + "nothing here constructs one, so that file would never be opened and an AUTO duration " + "would fall back to the recipe default. Give 'num_frames', or 'duration' (exact " + "arithmetic against the recipe frame rate), instead. Refused rather than ignored; " + "recorded as owed in .agents/specs/ltx25-retire-dead-arms.md (#611)."); + } +} + // `detect_model_version` normalizes the separator before parsing // (utils/constants.py:161), and the recipe table is keyed on the two-component // spelling ("2.5"), not on the checkpoint's three-component "2.5.0". Reduce it @@ -511,6 +553,7 @@ Ltx2ConditioningTrace Ltx2VideoEngine::last_conditioning() const { std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& params) { if (params.dit_path.empty()) Fail("dit_path is required"); CheckKnownExtras(params.extras); + CheckUnservedExtras(params.extras); auto engine = std::unique_ptr(new Ltx2VideoEngine()); engine->impl_ = std::make_unique(); @@ -1171,12 +1214,12 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // needs an encoded prompt this engine cannot produce", and since `has_encoder` // above the engine produces exactly that. What is missing now is the head // itself — `ltx2_duration_head.h` is ported and gated as a brick, but nothing - // here constructs one, and `duration_head_path` is accepted in - // `kKnownLoadExtras` while NO code reads it (grep: it appears at that one - // site). So the extra is inert rather than wired, and that is recorded as owed - // rather than left to be discovered by someone who supplies it and gets the - // recipe default. An explicit duration is exact arithmetic, so it is served; - // the AUTO path is what is missing, and `num_frames` is how to avoid it. + // here constructs one. `duration_head_path` used to be ACCEPTED while no code + // read it, so a caller who supplied a head silently landed on this line + // instead; `CheckUnservedExtras` now refuses that key by name at load (#611, + // .agents/specs/ltx25-retire-dead-arms.md §2). What remains owed is the head + // itself. An explicit duration is exact arithmetic, so it is served; the AUTO + // path is what is missing, and `num_frames` is how to avoid it. frames = static_cast(std::llround(gen.duration_seconds * fps)); } if (frames < 1) Fail("num_frames resolved to " + std::to_string(frames)); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f67d71e78..ccd05a2e1 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -189,6 +189,13 @@ target_compile_definitions(test_video_engine PRIVATE # LTX-2.5 L7: the family behind the VideoEngine seam and the driving loop, run # over a reduced-dimension checkpoint set written in the SHIPPED file format. vllm_cpp_add_test(test_ltx2_video vllm/multimodal/test_ltx2_video.cpp) +# The reader-anchor claim this suite gates is a claim ABOUT a source file, so it +# reads that file. Recorded `file:line` anchors go stale within the same PR here +# (a merge of origin/main moved all nine of these by 25 lines mid-review), which +# is why the anchors are derived from the source and compared with the recorded +# ones rather than trusted. +target_compile_definitions(test_ltx2_video PRIVATE + LTX2_VIDEO_SOURCE_PATH="${CMAKE_SOURCE_DIR}/src/vllm/multimodal/ltx2_video.cpp") # LTX-2.5 DiT parity gate (row MODEL-DIFFUSION-LTX25, spec phase L2). The goldens # live next to the test source, like every other generated .inc. vllm_cpp_add_test(test_ltx2 vllm/models/test_ltx2.cpp) @@ -211,6 +218,16 @@ target_include_directories(test_ltx2_text_encoder PRIVATE ${CMAKE_CURRENT_SOURCE # next to the test source. vllm_cpp_add_test(test_ltx2_pipeline vllm/models/test_ltx2_pipeline.cpp) target_include_directories(test_ltx2_pipeline PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) +# The out-of-scope ledger separates a REACHABLE refusal from a declared marker, and +# docs/FEATURES.md is where that distinction reaches a user. It had re-merged the two +# into one "Declared, not requestable" row, so the public surface contradicted the +# ledger it projects; the case below reads the doc and holds it to the split. +# The ledger's RETIREMENT note carries an absence claim about upstream, and it +# shipped false twice in the same header. The case that guards it reads the header +# itself rather than a copy of the sentence kept in the test. +target_compile_definitions(test_ltx2_pipeline PRIVATE + VLLM_CPP_FEATURES_DOC_PATH="${CMAKE_SOURCE_DIR}/docs/FEATURES.md" + LTX2_PIPELINE_HEADER_PATH="${CMAKE_SOURCE_DIR}/include/vllm/model_executor/models/ltx2_pipeline.h") # LTX-2.5 L6: the quantized loaders, gated three ways — the SHIPPED checkpoints' # own manifests, a few hundred of their own bytes against torch, and synthetic # files for whole-model materialization. All three .inc files live beside the diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 11996d926..024917a36 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -27,11 +27,14 @@ #include #include +#include #include #include #include +#include #include #include +#include #include #include #include @@ -1231,26 +1234,214 @@ TEST_CASE("ltx2 the two references disagree on the default negative prompt, and // Out-of-scope refusals // =========================================================================== -TEST_CASE("ltx2 every L5 out-of-scope feature is refused BY NAME") { - // Spec section 2 "Out", plus the L7 boundary. A silent downgrade of any of - // these produces a video, which is exactly why none of them may fall back. - const std::vector> owed = { +// THE LIST SHRANK FROM SEVEN TO FIVE on 2026-08-13, and the shrink is the point. +// Two enumerators were retired by row LTX25-RETIRE-DEAD-ARMS (#644): +// `kMultishot`, which refused a feature that exists in NEITHER reference, and +// `kVideoEngineWiring`, whose subject shipped in `cefacd2d0`. See +// .agents/specs/ltx25-retire-dead-arms.md §1.1 and §1.5. A CHANGED CASE COUNT +// here is that retirement, not a lost assertion. +TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { + // Spec section 2 "Out". A silent downgrade of any of these produces a video, + // which is exactly why none of them may fall back. + // + // REACHABLE REFUSALS: a product path constructs the condition, so a caller can + // trip these. The call sites are `ltx2_upsampler.cpp:395` and + // `ltx2_pipeline.cpp:199`. + const std::vector> reachable = { {vllm::Ltx2UnportedPipelineFeature::kTemporalUpsampler, "temporal"}, + {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, + }; + // DECLARED-OUT-OF-SCOPE MARKERS: nothing a caller can send reaches these, so + // the message must not claim otherwise. Recording them as refusals overstated + // what this port has, which is the defect this row closes. + const std::vector> markers = { {vllm::Ltx2UnportedPipelineFeature::kLoraFusion, "LoRA"}, - {vllm::Ltx2UnportedPipelineFeature::kMultishot, "multishot"}, {vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot, "int8-convrot"}, - {vllm::Ltx2UnportedPipelineFeature::kCfgParallelism, "parallelism"}, - {vllm::Ltx2UnportedPipelineFeature::kVideoEngineWiring, "VideoEngine"}, - {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, + {vllm::Ltx2UnportedPipelineFeature::kMultiGpuParallelism, "multi-GPU"}, }; - for (const auto& item : owed) { + + std::vector> all = reachable; + all.insert(all.end(), markers.begin(), markers.end()); + for (const auto& item : all) { const std::string message = RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); INFO("feature = ", item.second, " refusal = ", message); CHECK(Mentions(message, item.second)); // Naming WHERE the work is owed is what keeps it from being rediscovered. CHECK(Mentions(message, "ltx-2-5.md")); + // The RETIRED arm must not come back: a refusal that cites a feature neither + // reference has sends the next reader looking upstream for it. + CHECK_FALSE(Mentions(message, "multishot")); + } + for (const auto& item : reachable) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("reachable = ", item.second, " refusal = ", message); + CHECK_FALSE(Mentions(message, "DECLARED, NOT REQUESTABLE")); + } + for (const auto& item : markers) { + const std::string message = + RefusalMessage([&] { vllm::Ltx2RefuseUnportedPipelineFeature(item.first); }); + INFO("marker = ", item.second, " refusal = ", message); + CHECK(Mentions(message, "DECLARED, NOT REQUESTABLE")); + } + + // THE ABSENCE THIS MESSAGE STATES IS ITSELF EVIDENCE, so it is gated. The first + // version of this marker ended "int8 appears upstream only in the trainer" — a + // false-absence claim of exactly the kind row LTX25-RETIRE-DEAD-ARMS exists to + // retire (#604), shipped inside a user-visible refusal. LTX-2 @ fd4ded7f carries + // a per-row int8 quantize kernel with fp32 scales in `ltx-kernels`, which is an + // INFERENCE package, not the trainer: `blockwise/triton_ops.py:35,43`, aliased + // `rowwise_int_quantize_triton` at `:436`. It is dead — that alias is its only + // reference and `blockwise/functional.py:12-18` does not re-export it — so the + // disposition is unchanged and only the sentence was wrong. See + // .agents/specs/ltx25-retire-dead-arms.md §1.2. + { + const std::string message = RefusalMessage([] { + vllm::Ltx2RefuseUnportedPipelineFeature(vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot); + }); + INFO("int8 marker = ", message); + CHECK_FALSE(Mentions(message, "only in the trainer")); + // The true statement names where the one inference-side int8 lives, so a + // reader who greps upstream and finds it is not left thinking we missed it. + CHECK(Mentions(message, "ltx-kernels")); + } +} + +// THE PUBLIC SURFACE MUST NOT RE-MERGE WHAT THE LEDGER SPLIT. `docs/FEATURES.md` +// is where the reachable/marker distinction reaches a user, and it carried ONE +// "Declared, not requestable" row listing all five arms — including the two a +// caller CAN trip. That is the same conflation the ledger above stopped making, +// reintroduced one surface out. Gated here rather than left to review because a +// doc row and an enum drift silently. +// +// Deliberately narrow: it asserts only that no row calling something +// unrequestable names a REACHABLE arm. It does not police the wording of the doc. +TEST_CASE("ltx2 docs/FEATURES.md never calls a REACHABLE refusal unrequestable") { + std::ifstream in(VLLM_CPP_FEATURES_DOC_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << VLLM_CPP_FEATURES_DOC_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string doc = buf.str(); + REQUIRE(doc.size() > 1000); + + // The two arms with a product call site: `ltx2_upsampler.cpp` constructs the + // temporal-upsampler condition and `ltx2_pipeline.cpp` the BetaScheduler one. + // Named by the words the doc uses for them, since that is what a reader sees. + const std::vector reachable_words = {"upsampler", "BetaScheduler"}; + + size_t rows_examined = 0; + size_t line_no = 0; + size_t at = 0; + while (at <= doc.size()) { + const size_t end = doc.find('\n', at); + const std::string line = doc.substr(at, end == std::string::npos ? std::string::npos : end - at); + ++line_no; + // A table row that makes the not-requestable claim, in either casing the doc + // uses for it. + const bool is_row = !line.empty() && line[0] == '|'; + const bool claims_unrequestable = + line.find("not requestable") != std::string::npos || + line.find("NOT REQUESTABLE") != std::string::npos; + if (is_row && claims_unrequestable && line.find("LTX-2.5") != std::string::npos) { + ++rows_examined; + for (const std::string& word : reachable_words) { + INFO("FEATURES.md:" << line_no << " = " << line); + CHECK_MESSAGE(line.find(word) == std::string::npos, + "a row claiming 'not requestable' names the REACHABLE arm '" + << word + << "'; split the reachable refusals out, per " + ".agents/specs/ltx25-retire-dead-arms.md §1.6"); + } + } + if (end == std::string::npos) break; + at = end + 1; + } + // ANTI-VACUOUS. Without this the case passes when the row is renamed away and + // proves nothing — the failure mode this whole row is about. + CHECK_MESSAGE(rows_examined == 1, + "expected exactly ONE LTX-2.5 'not requestable' row in docs/FEATURES.md, found " + << rows_examined); +} + +// THE RETIREMENT NOTE'S OWN EVIDENCE, held to what upstream actually says. +// +// Retiring `kMultishot` rests on an ABSENCE claim about upstream, and the header +// above the enum is where a porter reads it. That sentence shipped wrong: it said +// the only `scene` hit upstream was PySceneDetect in the TRAINER. At +// Lightricks/LTX-2 @ fd4ded7f `scene` has THREE senses, and the third — +// prompt-writing guidance — lives in `ltx-core`, which ships at INFERENCE. So a +// porter greps `scene`, finds "scene cuts" in a shipped prompt-enhancer prompt, +// and concludes we missed a multi-shot path that our own header told them did not +// exist. Third instance of #604 inside the row whose subject is retiring #604. +// +// The disposition did not move — it got STRONGER. Those prompts instruct the +// enhancer NOT to describe scene cuts and to keep a "Single continuous take" +// (gemma3_i2v:18, gemma3_t2v:24, gemma4_i2v:3), which is affirmative evidence +// that no multi-shot generation mode exists. +// +// WHAT THIS CASE CAN AND CANNOT PROVE. It reads the shipped header and holds its +// text, so the false sentence cannot come back and the true evidence cannot be +// dropped. It CANNOT verify the upstream claim — no upstream checkout exists in +// this tree — so the derivation, with the positive control in the same command, +// lives in .agents/specs/ltx25-retire-dead-arms.md §1.1. +TEST_CASE("ltx2 the kMultishot retirement note states the scene evidence correctly") { + std::ifstream in(LTX2_PIPELINE_HEADER_PATH); + REQUIRE_MESSAGE(in.good(), "cannot open " << LTX2_PIPELINE_HEADER_PATH); + std::stringstream buf; + buf << in.rdbuf(); + const std::string header = buf.str(); + REQUIRE(header.size() > 1000); + + // The note is prose wrapped across comment lines, so every claim below is + // matched against a flattened copy: comment markers dropped, runs of whitespace + // collapsed to one space. Without this a reflow of the paragraph would silently + // turn every assertion vacuous. + std::string flat; + flat.reserve(header.size()); + bool pending_space = false; + for (size_t i = 0; i < header.size(); ++i) { + const char c = header[i]; + if (c == '/' && i + 1 < header.size() && header[i + 1] == '/') { + i += 1; + pending_space = true; + continue; + } + if (std::isspace(static_cast(c)) != 0) { + pending_space = true; + continue; + } + if (pending_space && !flat.empty()) flat += ' '; + pending_space = false; + flat += c; + } + + // ANTI-VACUOUS. Every assertion below is about ONE paragraph; if that paragraph + // is renamed or removed they all pass while proving nothing. + size_t notes = 0; + for (size_t at = flat.find("`kMultishot` — FABRICATED"); at != std::string::npos; + at = flat.find("`kMultishot` — FABRICATED", at + 1)) { + ++notes; } + REQUIRE_MESSAGE(notes == 1, + "expected exactly ONE `kMultishot` — FABRICATED retirement note in " + << LTX2_PIPELINE_HEADER_PATH << ", found " << notes); + + // The sentence that was false. It asserted an upstream absence from our own + // vocabulary, with no positive control, in a SHIPPED header. + CHECK_MESSAGE(flat.find("the only `scene` hit is PySceneDetect") == std::string::npos, + "the retired false claim is back: `scene` is not trainer-only upstream — " + "ltx-core's shipped gemma prompt files carry it, see " + ".agents/specs/ltx25-retire-dead-arms.md §1.1"); + CHECK(flat.find("only `scene` hit") == std::string::npos); + + // And the evidence that replaced it, by the three things a porter needs: that + // the narrative sense is PROMPT GUIDANCE, that it ships in `ltx-core` rather + // than only the trainer, and that the guidance FORBIDS scene cuts. + CHECK_MESSAGE(flat.find("scene cuts") != std::string::npos, + "the retirement note must name the prompt guidance it now rests on"); + CHECK(flat.find("ltx-core") != std::string::npos); + CHECK(flat.find("system_prompt") != std::string::npos); } // =========================================================================== diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index b065fbe7b..1122469ab 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -25,8 +25,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -488,6 +490,266 @@ TEST_CASE("ltx2 video: an unknown extra is refused, not ignored") { } } +// A key this family DEFINES but does not serve is the worse half of the same +// defect, and the one an "unknown extra" check cannot see. `duration_head_path` +// was in `kKnownLoadExtras` and read by NOTHING (#611): supplying a duration head +// loaded no head, opened no file, and handed back the recipe default with no +// diagnostic. AGENTS.md requires an unimplemented arm to be refused with a +// message naming the missing piece, so it is refused rather than accepted. +// +// Dropping the key from `kKnownLoadExtras` instead would produce "unknown load +// extra", which is a DIFFERENT and wrong claim — the family defines the key and +// understands what it means; what is missing is the head. Hence the assertion on +// the missing piece and the alternative, not only on the key. +TEST_CASE("ltx2 video: duration_head_path is REFUSED by name, not silently ignored") { + Workspace ws; + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + // Any path at all: the point is that NOTHING opens it. Naming a file that does + // exist keeps a not-found error from standing in for the refusal. + mp.extras["duration_head_path"] = ws.paths.dit; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("duration_head_path is served by no code; accepting it substitutes the recipe default"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("duration_head_path") != std::string::npos); + // The MISSING PIECE, which is what separates this from "unknown key". + CHECK(msg.find("duration head") != std::string::npos); + // And what to use instead, so the refusal is actionable. + CHECK(msg.find("num_frames") != std::string::npos); + // Not the unknown-key message: that one would say the family does not define + // it, and this family does. + CHECK(msg.find("unknown load extra") == std::string::npos); + } +} + +// The INVENTORY, so the defect above cannot come back as a different key. Every +// extra this family accepts is either read by something or refused by name; a +// tenth decorative key fails this case rather than waiting to be discovered by +// the caller who supplies it. +// +// The audit behind it is in .agents/specs/ltx25-retire-dead-arms.md §2.1: nine of +// the ten keys have a reader and `duration_head_path` was the only one with none. +// The reader LINES are deliberately not repeated here — they moved twice while +// this row was in review. They live in one place, the READER ANCHORS comment in +// `ltx2_video.cpp`, and the case below derives them and holds that comment to it. +TEST_CASE("ltx2 video: every accepted load extra is READ by something") { + Workspace ws; + // The keys with a reader. + const std::vector served = { + vllm::multimodal::kLtx2AudioPromptEmbedsExtra, vllm::multimodal::kLtx2PipelineKindExtra, + vllm::multimodal::kLtx2ModelVersionExtra, vllm::multimodal::kLtx2AllowUnportedExtra, + vllm::multimodal::kLtx2MaxPhaseExtra, vllm::multimodal::kLtx2DitConfigPathExtra, + vllm::multimodal::kLtx2PromptValidRowsExtra, vllm::multimodal::kLtx2EncoderConfigPathExtra, + "upsampler_path", + }; + // The keys the family defines and does NOT serve. Growing this list is a + // deliberate act; growing it silently is the defect #611 records. + const std::vector refused = {"duration_head_path"}; + + // THE HANDLE ON THE REAL ARRAY. The unknown-extra refusal builds its listing + // from `kKnownLoadExtras` itself, so parsing that listing gates the ACTUAL + // accepted set rather than a copy of it maintained here. Without this the two + // vectors above would be true by construction and would gate nothing. + std::string listing; + { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras["definitely_not_a_key"] = "1"; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an unknown extra must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + const size_t at = msg.find("This family defines: "); + REQUIRE(at != std::string::npos); + listing = msg.substr(at + std::string("This family defines: ").size()); + } + } + INFO("listing = " << listing); + // Every name this row inventoried is still accepted... + for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); + for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); + // ...and there is no ELEVENTH name that this inventory has never seen. The + // separator is ", ", so the count is one more than the separators. + size_t names = 1; + for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { + ++names; + } + CHECK_MESSAGE(names == served.size() + refused.size(), + "kKnownLoadExtras grew; add the key to `served` (with its reader) or to " + "`refused` (with a by-name refusal), per .agents/specs/ltx25-retire-dead-arms.md"); + + // And the unserved half is refused rather than accepted. + for (const std::string& key : refused) { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[key] = ws.paths.dit; + INFO("key = " << key); + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an accepted-but-unread extra must be refused by name"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find(key) != std::string::npos); + CHECK(msg.find("unknown load extra") == std::string::npos); + } + } +} + +// THE ANCHORS FOR THAT INVENTORY, DERIVED RATHER THAN TRUSTED. +// +// The case above proves each key is accepted or refused; it cannot prove WHERE a +// key is read, and "nine of ten reach a reader at these lines" is the claim this +// row rests on. That claim shipped wrong: the recorded anchors were nine lines +// that named no reader at all, in the very file they were recorded in. Then, in +// review, a merge of `origin/main` moved all nine again. A `file:line` written by +// hand is stale by the next commit, so this derives them from the source and +// holds the recorded list to what it finds. When it fails it prints the answer. +// +// It does NOT assert absolute line numbers of its own — nothing here to rot. The +// only obligation it creates is on whoever moves a reader: update the one comment +// block in the same file they are already editing. +namespace { + +std::string ReadSourceFile(const char* path) { + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot open " << path); + std::stringstream buf; + buf << in.rdbuf(); + return buf.str(); +} + +std::vector SplitLines(const std::string& text) { + std::vector lines; + size_t at = 0; + while (at <= text.size()) { + const size_t end = text.find('\n', at); + lines.push_back(text.substr(at, end == std::string::npos ? std::string::npos : end - at)); + if (end == std::string::npos) break; + at = end + 1; + } + return lines; +} + +// 1-based index of the ONLY line containing `needle`, or 0. Uniqueness is the +// point: an anchor that matches twice anchors nothing, and existence alone is +// what let the stale numbers survive. +size_t UniqueLineWith(const std::vector& lines, const std::string& needle) { + size_t found = 0; + size_t count = 0; + for (size_t i = 0; i < lines.size(); ++i) { + if (lines[i].find(needle) != std::string::npos) { + ++count; + found = i + 1; + } + } + return count == 1 ? found : 0; +} + +std::string JoinNumbers(const std::vector& v) { + std::string s; + for (size_t n : v) s += (s.empty() ? "" : " ") + std::to_string(n); + return s; +} + +} // namespace + +TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") { + const std::string source = ReadSourceFile(LTX2_VIDEO_SOURCE_PATH); + const std::vector lines = SplitLines(source); + REQUIRE(lines.size() > 500); + + // Everything is measured relative to the accepted-keys array, so a stray + // mention in the file header cannot be mistaken for a reader. + const size_t array_line = UniqueLineWith(lines, "const char* const kKnownLoadExtras[] = {"); + REQUIRE_MESSAGE(array_line != 0, "kKnownLoadExtras[] declaration is not unique in the source"); + size_t array_end = 0; + for (size_t i = array_line; i < lines.size(); ++i) { + if (lines[i] == "};") { + array_end = i + 1; + break; + } + } + REQUIRE(array_end > array_line); + + // The nine SERVED keys, by the token each is spelled with in the source. Order + // is irrelevant — the comparison is on the sorted set — so this list is not a + // second place the anchors live. + const std::vector served_tokens = { + "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", + "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", + "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", + }; + std::vector derived; + for (const std::string& token : served_tokens) { + size_t first = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find(token) != std::string::npos) { + first = i + 1; + break; + } + } + INFO("token = " << token); + CHECK_MESSAGE(first != 0, "no reader found after kKnownLoadExtras for " << token); + if (first != 0) derived.push_back(first); + } + REQUIRE(derived.size() == served_tokens.size()); + std::sort(derived.begin(), derived.end()); + + // The RECORDED list, parsed out of the one comment line that carries it. + const size_t marker = UniqueLineWith(lines, "READER ANCHORS (derived and gated by"); + REQUIRE_MESSAGE(marker != 0, + "the READER ANCHORS comment marker is missing or not unique in the source"); + const std::string recorded_line = lines[marker]; // the line AFTER the marker (1-based) + std::vector recorded; + for (size_t i = 0; i < recorded_line.size();) { + if (std::isdigit(static_cast(recorded_line[i]))) { + size_t j = i; + while (j < recorded_line.size() && std::isdigit(static_cast(recorded_line[j]))) { + ++j; + } + recorded.push_back(static_cast(std::stoul(recorded_line.substr(i, j - i)))); + i = j; + } else { + ++i; + } + } + std::sort(recorded.begin(), recorded.end()); + CHECK_MESSAGE(recorded == derived, "the reader anchors recorded in ltx2_video.cpp are STALE. " + "Recorded: [" + << JoinNumbers(recorded) << "]. Actual: [" + << JoinNumbers(derived) + << "]. Paste the actual list into the READER ANCHORS " + "comment; the spec's §2.1 table is dated and stays."); + + // And the UNSERVED key is touched only by the refusal, never by a reader. This + // is the half a "has a reader" sweep cannot express. + const size_t refuse_line = UniqueLineWith(lines, "void CheckUnservedExtras("); + REQUIRE(refuse_line != 0); + size_t refuse_end = 0; + for (size_t i = refuse_line; i < lines.size(); ++i) { + if (lines[i] == "}") { + refuse_end = i + 1; + break; + } + } + REQUIRE(refuse_end > refuse_line); + size_t duration_hits = 0; + for (size_t i = array_end; i < lines.size(); ++i) { + if (lines[i].find("kLtx2DurationHeadPathExtra") == std::string::npos) continue; + ++duration_hits; + const size_t at = i + 1; + const bool inside_refusal = (at >= refuse_line) && (at <= refuse_end); + CHECK_MESSAGE(inside_refusal, + "ltx2_video.cpp:" << at + << " touches the duration-head extra OUTSIDE " + "CheckUnservedExtras; if it now has a real reader, move " + "it to the served list and drop the refusal (#611)"); + } + CHECK(duration_hits > 0); +} + // ─── the config the SHAPES cannot see ─────────────────────────────────────── // // WHY THESE ASSERT ON THE ENGINE AND NOT ON A LOCAL. The L7 repair is one line in