Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .agents/roadmap_v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,9 @@ issue is not yet placed. Keyed record: update in place, never append.
| [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug |
| [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug |
| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature |
| [#611](https://github.com/mudler/vllm.cpp/issues/611) | `ROAD-V1-LTX25` | `duration_head_path` is accepted by `kKnownLoadExtras` and read by NOTHING, so a caller who points at a duration head silently gets the recipe default. Row `LTX25-RETIRE-DEAD-ARMS` (spec [`ltx25-retire-dead-arms.md`](specs/ltx25-retire-dead-arms.md)) refuses it by name and records the full accepted/read inventory of all ten extras | bug |
| [#691](https://github.com/mudler/vllm.cpp/issues/691) | `ROAD-V1-LTX25` | The LTX-2.5 out-of-scope ledger's reachable/marker split gates the MESSAGE TEXT, not reachability: give `kLoraFusion` a product call site and the ledger test still passes while the shipped refusal tells the caller `DECLARED, NOT REQUESTABLE` right after they requested it. Copy the anti-tautological shape of the extras inventory gate | bug |
| [#692](https://github.com/mudler/vllm.cpp/issues/692) | `ROAD-V1-LTX25` | Campaign spec `ltx-2-5.md:291` still records the FABRICATED `multishot` arm as owed after row `LTX25-RETIRE-DEAD-ARMS` retired it; same line needs `CFG parallelism` renamed to multi-GPU. Operator-owned file, so filed rather than edited by that row | bug |
| [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug |
| [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug |
| [#264](https://github.com/mudler/vllm.cpp/issues/264) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprob_token_ids`: generative scoring over an explicit token set is unported | feature |
Expand Down
659 changes: 659 additions & 0 deletions .agents/specs/ltx25-retire-dead-arms.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,9 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
|---|---|---|
| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) |
| Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 |
| LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` |
| LTX-2.5 arms a request CAN reach | Refused by name at the call site | The temporal x2 latent upsampler and `BetaScheduler`. A product path constructs each condition and throws, naming the missing piece |
| LTX-2.5 arms nothing can request | Declared, not requestable | LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU. No request field, load extra or CLI flag asks for these. `multishot` was RETIRED: no such entry point exists in either reference |
| Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it |
| LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server |
| Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started |
Expand Down
37 changes: 30 additions & 7 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1678,14 +1678,37 @@ knobs from `extras`. H3 takes `partition`. LTX-2.5 takes
`audio_prompt_embeds_path` (the audio stream's conditioning, the twin of the
seam's `prompt_embeds_path`, which carries the video stream), `pipeline_kind`
(default `distilled_two_stage`), `model_version` (only for a checkpoint that
declares none), `dit_config_path`, `allow_unported_modules`, `max_phase`,
`prompt_embeds_valid_rows`, `upsampler_path` and `duration_head_path`. An extra a
family does not define is refused, never ignored. One caveat inside that set:
`duration_head_path` is accepted but INERT — the duration head is ported and gated
as a brick, nothing in the video engine constructs one, and no code reads that
key, so supplying it neither loads a head nor enables an AUTO duration. Give
declares none), `dit_config_path`, `encoder_config_path`,
`allow_unported_modules`, `max_phase`, `prompt_embeds_valid_rows`,
`upsampler_path` and `duration_head_path`. An extra a family does not define is
refused, never ignored. One caveat inside that set: `duration_head_path` is
defined but UNSERVED — the duration head is ported and gated as a brick, and
nothing in the video engine constructs one — so supplying it is **refused by
name** at load rather than accepted. It used to be accepted and read by nothing,
which silently substituted the recipe default for the file you named. Give
`num_frames` (or `duration`, which is exact arithmetic against the recipe's frame
rate) instead.
rate) instead. Every other key in that list reaches a reader.

Two LTX-2.5 arms are refused where a render would otherwise silently downgrade:
the temporal x2 latent upsampler and `BetaScheduler`. Both are reachable — a
render asking for either gets a refusal naming the missing piece. Three more are
recorded as out of scope but are **not requestable**, so no flag or extra can
reach them: LoRA fusion, `int8-convrot`, and single-node multi-GPU. Their messages
say `DECLARED, NOT REQUESTABLE` so the two kinds are not confused. `int8-convrot`
in particular is a ComfyUI-ecosystem format: upstream LTX-2's own inference
quantization kinds are `fp8-cast`, `fp8-scaled-mm`, `nvfp4-cast` and
`nvfp4-prequant`, and nothing wired upstream reaches int8 at all.

What is **not** on that list, and why: **multi-shot or multi-scene generation.**
A request that composes several camera takes into one output has no flag here
because upstream LTX-2 has no such mode to mirror — its `shot` is one continuous
take, and its own prompt-enhancement prompts instruct the model to keep a "single
continuous take" and not to describe scene cuts. `scene` does appear across the
upstream tree, in three unrelated senses (`scene-linear` HDR colour, PySceneDetect
in the trainer's dataset preprocessor, and that prompt-writing guidance); none of
them is a generation mode. This port carried a `multishot` refusal until
2026-08-13, which was a defect in our own record rather than a gap, and it was
retired. Generate one take per request.

`prompt_embeds_valid_rows` is how many of the supplied conditioning rows are real
tokens; absent, every row is. It matters because the embeddings connector
Expand Down
3 changes: 3 additions & 0 deletions examples/ltx2_gen/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,9 @@ int main(int argc, char** argv) {
else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str()));
else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str()));
else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str()));
// Kept although the library REFUSES this extra by name (#611): the duration
// head is unported, and forwarding the flag gets the caller that named
// refusal instead of "unknown option", which says nothing about why.
else if (f == "--duration-head")
SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str()));
else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str()));
Expand Down
70 changes: 61 additions & 9 deletions include/vllm/model_executor/models/ltx2_pipeline.h
Original file line number Diff line number Diff line change
Expand Up @@ -571,20 +571,72 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind,
const std::string& model_version);

// ---------------------------------------------------------------------------
// Out of scope for L5, refused by name (spec section 2, "Out")
// Out of scope, refused by name (spec section 2 "Out"; the 2026-08-13 grounding
// pass is .agents/specs/ltx25-retire-dead-arms.md, row LTX25-RETIRE-DEAD-ARMS)
// ---------------------------------------------------------------------------

// Each of these renders something plausible if it is silently downgraded, which
// is why none of them falls back. `Ltx2RefuseUnportedPipelineFeature` throws with
// a message naming the missing piece and the phase or row that owes it.
// a message naming the missing piece and the row that owes it.
//
// TWO KINDS live here, and conflating them overstated what this port refuses:
//
// REACHABLE REFUSAL — a product path constructs the condition and throws, so a
// caller CAN trip it. `kTemporalUpsampler` (ltx2_upsampler.cpp:395) and
// `kBetaScheduler` (ltx2_pipeline.cpp:199) are the two.
//
// DECLARED-OUT-OF-SCOPE MARKER — no request field, load extra or CLI flag asks
// for it, so nothing outside the ledger test reaches it. It is a record of what
// upstream HAS and this port does NOT, which is worth keeping; calling it a
// refusal is what was wrong. `kLoraFusion`, `kInt8ConvRot` and
// `kMultiGpuParallelism` are markers, and their messages say so.
//
// TWO ENUMERATORS WERE RETIRED on 2026-08-13, recorded here because the
// retirement IS the record — a reader who finds them in git history needs to know
// they did not simply move:
//
// `kMultishot` — FABRICATED. It refused "multishot generation" and cited
// "ltx-pipelines multishot entry points". No such entry point, symbol or string
// exists in Lightricks/LTX-2 @ fd4ded7f or huggingface/diffusers @ 3a2f35d4.
// Searched as a SUBJECT rather than by our own phrasing: upstream's only sense
// of "shot" is ONE camera take (duration_head.py:1,5 "predicts shot duration";
// README.md:136 "a cinematographer describing a shot list"). `scene` has THREE
// senses upstream and none is a generation mode: `scene-linear` HDR colour
// (ltx-core color/hlg.py, hdr.py), PySceneDetect in the TRAINER — the only CODE
// sense — and prompt-writing guidance, which ships at INFERENCE inside
// `ltx-core`, in text_encoders/gemma/encoders/prompts/ as
// gemma{3,4}_{i2v,t2v}_system_prompt.txt. That third sense is why the
// retirement HOLDS rather than being undermined: those prompts tell the
// enhancer NOT to describe scene cuts and to keep a "Single continuous take"
// (gemma3_i2v:6,18, gemma3_t2v:24, gemma4_i2v:3). A defect in our record is not
// a gap in our port, so there was nothing to owe.
//
// Recorded because it is the row's own subject: this paragraph used to claim
// that `scene` appeared upstream ONLY as PySceneDetect in the trainer. It was
// an absence asserted from our own vocabulary with no positive control — #604 —
// shipped in the header of the row that exists to retire #604 instances, and it
// took a third review round to find. The derivation, with its positive control
// in the same command, is .agents/specs/ltx25-retire-dead-arms.md §1.1.
//
// `kVideoEngineWiring` — LANDED. It said the end-to-end composition through
// `vllm::multimodal::VideoEngine` "is phase L7, not L5"; L7 shipped in
// `cefacd2d0`. A refusal whose subject shipped is a false statement.
enum class Ltx2UnportedPipelineFeature {
kTemporalUpsampler, // model/upsampler with temporal_upsample=True
kLoraFusion, // loader/LoraPathStrengthAndSDOps
kMultishot, // ltx-pipelines multishot entry points
kInt8ConvRot, // ComfyUI-only quantization
kCfgParallelism, // ltx-pipelines/multigpu
kVideoEngineWiring, // end-to-end through vllm::multimodal::VideoEngine (L7)
kBetaScheduler, // components/schedulers.py:91-120
// Reachable refusals.
kTemporalUpsampler, // model/upsampler with temporal_upsample=True
kBetaScheduler, // components/schedulers.py:91-120
// Declared-out-of-scope markers.
kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps),
// fused by loader/fuse_loras.py
kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four
// inference kinds upstream defines are fp8-cast / fp8-scaled-mm /
// nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-26).
// `convrot` is nowhere at all; int8 is UNREACHABLE rather than
// absent — trainer-only for anything wired, plus one DEAD kernel
// in ltx-kernels (triton_ops.py:35,43). §1.2 of the row spec
kMultiGpuParallelism, // ltx-pipelines/multigpu — sequence-parallel, tiled data
// parallel and distributed VAE decode. NOT CFG batching:
// zero `cfg` hits in either multigpu tree
};
[[noreturn]] void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature);

Expand Down
7 changes: 5 additions & 2 deletions include/vllm/multimodal/ltx2_video.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,13 @@
// Phases L2-L6 shipped a DiT forward, a text feature extractor, two VAEs, a
// vocoder, an upsampler, a duration head, and a pipeline COMPONENT library —
// schedules, noisers, steppers, guiders, patchifiers, recipes. Nothing drove
// them: `Ltx2RefuseUnportedPipelineFeature(kVideoEngineWiring)` refused the
// them: an `Ltx2UnportedPipelineFeature::kVideoEngineWiring` refusal named the
// composition BY NAME and named this phase as its owner. This TU is that
// composition and nothing else. It adds no numerics; every line either resolves
// a parameter, moves a buffer, or calls a brick that already has a golden.
// a parameter, moves a buffer, or calls a brick that already has a golden. (That
// enumerator was RETIRED in row LTX25-RETIRE-DEAD-ARMS once L7 landed in
// `cefacd2d0` — a refusal whose subject shipped is a false statement, not a
// record of debt. It is named here in the past tense on purpose.)
//
// ─── WHAT IT IS A PORT OF (file:line on BOTH sides) ──────────────────────────
// Upstream: Lightricks/LTX-2 @ fd4ded7, packages/ltx-pipelines/src/ltx_pipelines/
Expand Down
Loading
Loading