feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that is no longer MiniMax-only (#435) - #641
Merged
Merged
Conversation
…s no longer MiniMax-only (#435) Ports Lightricks LTX-2.5 -- a 21.00B joint video+audio flow-matching DiT, 48 blocks, two coupled streams (video 4096 / audio 2048) joined by explicit audio<->video cross-attention, per-head gated attention, gelu-approximate FFN -- and generalizes the diffusion lane so a second model was ADDITIVE rather than a fork. MiniMax-H3 moves behind the new `vllm::multimodal::VideoEngine` seam with its fold gate holding it to the same golden bytes. WHAT RUNS. The DiT and both VAEs including their ENCODERS and the mel front-end; the pipeline layer (sigma schedule, guidance, latent spatial x2 upsampler, duration head, embeddings connector); the quantized loaders, with FP8, torchao NVFP4 and the first-party NVFP4 file all loading; the Gemma-4 12B text tower with its embedded tokenizer, token-exact against HuggingFace on the shipped 262144-entry vocab; and a typed prompt reaching cross-attention through the connector. Reachable as video family `ltx-2.5` through the C ABI (v18, 36 exports), the `ltx2-gen` example and the server's /v1/videos, which registers family-agnostically. WHAT IS OWED, stated because a reader will otherwise over-read the above. The 320x192/25f render IS a coherent scene but was REGISTER-conditioned -- synthetic N(0, 0.2) input with 104 of 128 rows the connector's own trained learnable_registers -- so it is the checkpoint's learned default reached through the real connector, not a depiction of a prompt. The typed-prompt path is gated on a reduced-dimension synthetic fixture; a real-checkpoint prompted render has never been run, in either direction. The conditioning composition has NO value oracle: scaling it x1.5 or reversing its rows still passes every assertion, so `last_conditioning()` is a change detector and says so. The shipped NVFP4 tower's output is never compared to an independent oracle. Speed is structurally PENDING: vLLM-Omni carries no native 2.5 and its diffusers adapter is a black box, so no production-configuration denominator exists. ALSO FIXED, found while building the above. The shared scratch pool was DEVICE-BLIND: one process-wide free list held blocks from every backend, so CUDA->CPU was a SIGSEGV and CPU->CUDA was SILENT -- a uniform 0x7fff0000 all-NaN no token gate can see (#516, and #486 is the same bug). A DevicePool is now bound to one backend at construction and the no-argument accessors are REMOVED, which is what caught a live instance of the same defect landing in this very campaign. VERIFIED. Local: clean from-scratch Release build, zero warnings, 423/423 with the run's own denominator asserted against `ctest -N`. dgx.casa (GB10, sm_121a confirmed by cuobjdump on the binary, every fast-path arm enabled): 453 of 453 tests RAN under one `flock $HOME/gpu.lock` held 14:36:09Z-15:29:37Z; 9 failures, all nine drawn from the pre-existing set tracked as #233/#248/#305/#614, no new failure; test_minimax_h3 Passed 19.02s where it previously SEGFAULTed, and test_device_pool and all seven LTX suites pass. Issue: #435, #516, #486 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This was referenced Aug 13, 2026
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
…urce, and the USAGE anchor rotted in this branch's own rebase (#608, #649) Fresh-review findings on #643. Records only: no `src/`, `include/` or `tests/` file is touched and no row state moves. F1 - the USAGE.md anchor rotted during this branch's own rebase. The spec cited `docs/USAGE.md:902` for the parser-name count. That was right at the pre-rebase base `a20cdac30`; #641 (`cefacd2d0`) changed `docs/USAGE.md` by 572 lines and the count line moved to `:1126`. Re-derived and READ at this head: 1126: | `--tool-call-parser <name>` | `hermes` | Tool-call dialect (41 names over 37 families). `auto` detects from the chat template ... `:902` at this head is a container-tag table row (`| :main-cuda / -vulkan / -cpu | moves with main ...`). The spec now says `:1126` and carries the reason it moved, because nothing gates anchor line numbers (#632): a rebase moves them silently and a re-run preflight still passes, which is exactly how this survived. The PR body's "every anchor was re-derived and read at this head" is corrected there; it was not true of this one. F2 - the new characterisation was wrong where the old one was right. W0's COUNT correction stands: five upstream-only names, not the "three Rust/Harmony-backed ones" `docs/STATUS.md` claimed. Its restatement that `openai`/`cohere_command3`/`cohere_command4` are "plain Python" does not, and it is wrong in the more dangerous direction - plain Python is not the same as portable. Read in the pinned checkout 5559679229bc961848b121ccdeaa8fa5d79bec98: * vllm/tool_parsers/gptoss_tool_parser.py:17 - `GptOssToolParser` is a declared stub ("All output parsing is handled by HarmonyParser. This stub exists as a capability declaration via HarmonyParser.tool_parser_cls"), and BOTH methods, :31 and :45, raise NotImplementedError("GptOssToolParser is a stub. Use HarmonyParser for tool parsing."). Harmony-backed by explicit declaration, not a text parser. * vllm/tool_parsers/cohere_command_tool_parser.py:6-13 - both Cohere classes (base :34, :125, :138) are shims over the out-of-tree `cohere_melody` package (`PyFilter`/`PyFilterOptions`), with a hard ImportError if it is absent. All parsing is delegated; NO grammar exists in vLLM source to port. * minimax_m3_tool_parser.py:7,18 - `RustToolParser` + rust_parser_name = "MinimaxM3ToolParser". The grammar is in the Rust crate. * inkling_tool_parser.py:4,7 - a ParserEngine adapter over `InklingParserToolAdapter`. So of the five upstream-only names, only `inkling` is portable from vLLM source. That is load-bearing, so W1/W2 are RESCOPED, not reworded. W1 is three different jobs: `inkling` is the ordinary port; `minimax_m3` is written from the wire format its docstring describes, recorded from-scratch in porting-inventory.md section 9, with its upstream test as the fidelity gate; and `openai` has no upstream text parser at all, so it owes a decision recorded in the spec before code - mirror the Harmony seam, or register the name with an explicit refusal naming the missing piece, per AGENTS.md. W2's Cohere scope carries an undisclosed external dependency and likewise owes a decision rather than a port. `docs/STATUS.md` and the `roadmap_v1.md` #608 row move to the same reading, since AGENTS.md requires the three surfaces to agree. F3 - filed as #649 rather than repaired here. The `TOOLS-CALLING-CORE` row (.agents/engine-matrix.md:156) still records `tool_parser_names()` 40 / `reasoning_parser_names()` 7; re-derived at this head they are 41 (src/vllm/entrypoints/openai/tool_parsers/abstract.cpp:269) and 12 (src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp:72). It was flagged in the PR body with no issue behind it, which is the thing AGENTS.md forbids. #649 now tracks it and is linked from the roadmap_v1.md open-issues table. It is not fixed in this commit: it is a third row's prose and its reasoning half belongs to SAMPLE-REASONING (#605), so repairing it inside a records-only TOOLS-PARSER-BREADTH change is the "would surprise a reviewer" case the in-flow rule carves out. Code and tests are already correct - test_detect.cpp:221 pins 41 - so only the record drifted. F4 was informational. The PR body cited `TOOLS-STRUCTURAL-TAG` as the `PARTIAL` precedent, but that row carries `planned: specs/structural-tag.md` and no such file exists, so it is PARTIAL WITHOUT a committed spec - the opposite of the reason argued. The body is corrected; the `PARTIAL` state on this row is unchanged and remains correct on its own argument. scripts/agent-preflight.sh --staged: exit 0, "All gates green". scripts/check-windows-portability.py: exit 1, INHERITED - three errors in src/vllm/multimodal/video_engine.cpp introduced by #641, tracked by #648 (the fix) and #646 (the checker runs in no CI job, hook or preflight). `git diff origin/main -- src/ include/ tests/ scripts/` is empty and video_engine.cpp is byte-identical to origin/main. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
…L, and the issue table still held the retracted premise (#607) Fresh-review findings on #642. Records only: one spec section and two roadmap_v1.md rows. No src/, include/ or tests/ change, no row-state change. F1 - the seam map omitted the ENFORCEMENT consumer. Section 1.5 mapped the memory consequence (interfaces.py:293) and the kernel gate (qwen3_next.py:325) but not what makes a limit a limit. Added to the seam table, each line read in the pinned checkout 5559679229bc961848b121ccdeaa8fa5d79bec98: * vllm/multimodal/processing/context.py:409-428 - MultiModalProcessingInfo.validate_num_items raises VLLMValidationError("At most {limit} {modality}(s) may be provided in one prompt."), appending " Set `--limit-mm-per-prompt` to increase this limit." at :425-426 when the MODEL supports more than the user allowed. * its two call sites: context.py:461 inside parse_mm_data (:430), and vllm/entrypoints/chat_utils.py:662, whose only escape is enable_mm_embeds plus a *_embeds modality at limit 0 (chat_utils.py:653-660). * context.py:392-405 - allowed_mm_limits folds the USER limit with the model's supported_mm_limits by min(), so a user limit never raises a model's ceiling. * further consumers: vllm/multimodal/registry.py:126 and vllm/v1/worker/encoder_cudagraph.py:139. Following that chain changes what the flag DOES: get_limit_per_prompt returns 0 for every modality (multimodal.py:321-327), so validate_num_items computes limit = 0 and raises for any request carrying one or more items. Upstream --language-model-only REFUSES every multimodal request. The refusal is now placed explicitly in L1, with the reason it belongs there rather than in L3: it is the limits' own semantics, and a limit nothing enforces is not a limit. That retires my own overly generous claim. The section said "L2 without L3 is honest and shippable: the flag would be accepted and would correctly zero the limits". The honesty caveat I attached guarded only "frees VRAM", leaving the larger divergence unguarded - we would ACCEPT an image request upstream REFUSES, which is the flag's main observable behaviour. The claim is now conditional: with L1's refusal L2 ships honestly and only the memory win is owed; without it L2 is not shippable at all, because a flag that is accepted and inert is worse than the abort it replaces - an abort is visible and a silently served image request is not. F2 - one of the three mandated surfaces still stated the retracted premise. AGENTS.md requires the issue, the row's spec and the PR body to agree. The issue title was corrected; .agents/roadmap_v1.md's #607 row was not. It showed the Row column as "-" and still described the flag as "43 recipes skip the vision encoder to hand its VRAM to the KV cache". Now placed on ENG-MM-INPUT-PIPELINE and re-described from the corrected mechanism, naming BOTH consequences. Verified there is exactly ONE #607 row in that table (grep -c '^| \[#607\]' == 1), because two branches adding the same keyed row merge cleanly and define it twice. The recipe-surface sweep table's own --language-model-only line carried the same "free encoder VRAM" framing and is reconciled with it. F3 - wrong emphasis in a citation. The seam row labelled "--limit-mm-per-prompt / --language-model-only" cited arg_utils.py:555,1276,1691, all three of which are the language_model_only lines. Since section 1.5's thesis is that the limits come first, the two flags now have a row each and both triples are cited, all six read at the pin: 555 language_model_only: bool = MultiModalConfig.language_model_only 556 limit_mm_per_prompt: dict[str, int | dict[str, int]] = get_field( 1276 "--language-model-only", **multimodal_kwargs["language_model_only"] 1279 "--limit-mm-per-prompt", **multimodal_kwargs["limit_per_prompt"] 1691 language_model_only=self.language_model_only, 1692 limit_mm_per_prompt=self.limit_mm_per_prompt, scripts/agent-preflight.sh --staged: exit 0, "All gates green". scripts/check-windows-portability.py: exit 0, "Windows portability contract OK" - this branch is based on a20cdac, BEFORE #641 added src/vllm/multimodal/video_engine.cpp, so it does not carry that inherited red (#648 / #646). It will appear on a rebase onto current main, and it is not this branch's: no src/ file is touched here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This was referenced Aug 13, 2026
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
FOLLOWING_AGENTS_PROTOCOL main advanced to b81a5e4 (#641's LTX-2.5, then #650's oracle policy) and roadmap_v1.md's Open-issues table conflicted. That table is a KEYED RECORD, so AGENTS.md forbids resolving it by automatic three-way merge: took main's file WHOLESALE and reapplied this branch's two scoped edits. The two edits are NOT the same operation, and treating them alike is the trap I walked into first. #608 already exists on main -- it landed via #612 -- so it is an UPDATE IN PLACE. #649 is new, so it appends. My first attempt appended both and produced a duplicate #608, caught by counting rather than by any gate: the uniqueness check the checker does run would have caught it too, but only after I had already written the wrong thing. Verified rather than asserted: diff against main is +2/-1 -- one row replaced, one added -- and grep gives exactly one #608 row and one #649 row. One gate needed a second look. test_cpu_x86_llamacpp_floor's contended-leg case failed once here at load 26.97, then passed twice consecutively in this same worktree, and passes on clean main at the same load; the test file and tools/bench are byte-identical to main. That is #618's known flakiness, not this resolution. Re-ran the full staged gate afterwards for a clean result rather than reasoning past the red. scripts/agent-preflight.sh --staged: exit 0, all gates green. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 13, 2026
…, and rescope W1/W2 (#608, #649) (#643) FOLLOWING_AGENTS_PROTOCOL The row said "Qwen-Coder XML, Mistral, pythonic, and remaining parsers", "Our code: -", INVENTORIED, with a spec that was never written. All three parsers it named by title were implemented, alongside 38 others. This is a RECORD REPAIR, not an achievement: nothing new ships here. Three counts were wrong and two of them cancelled, which is why they survived a spec, a merge and a report. Ours is 41, not 40 -- the audit's regex used [a-z0-9_]+ and silently dropped the hyphenated granite-20b-fc, while test_detect.cpp:221 has asserted 41 since MODEL-MUSE-GLIMMER-W7. Upstream is 44, not 43. And two of OUR names exist in no upstream registry at the pin (qwen3, a local alias, and muse_glimmer, the off-pin unmerged-PR exception recorded in porting-inventory.md §16). 44 - 5 = 39 shared, + 2 = 41. The old "40 = 43 - 3" balanced only because two errors cancelled. The five upstream-only names are confirmed, but the characterisation of them was wrong in the dangerous direction -- "plain Python" reads as portable. Reading the method bodies rather than the imports: GptOssToolParser is a declared stub whose methods raise NotImplementedError and defer to Harmony; both Cohere classes hard-ImportError without the out-of-tree cohere_melody package and delegate all parsing to it. So only `inkling` is portable from vLLM source. W1/W2 are RESCOPED accordingly -- one ordinary port, one from-scratch write gated on the upstream test, and two names that each owe a recorded decision before any code. State INVENTORIED -> PARTIAL, not ANCHOR-BACKFILL: that state asserts a missing leaf spike and the spec is committed, while PARTIAL's "known to omit upstream behavior" is exactly true of the five. Summary counts re-derived from the rows. Reviewed twice. The first review FAILed it on four record findings -- a docs/USAGE.md anchor that rotted during this branch's own rebase (#641 moved that file by 572 lines), the portability characterisation, a defect flagged only in a PR body, and a backwards precedent citation. All four repaired by a fresh implementer; #649 filed for the stale TOOLS-CALLING-CORE counts. The scoped re-review returned PASS with no findings. Records only: 4 markdown files, no src/, no tests/, no checker weakened. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
FOLLOWING_AGENTS_PROTOCOL This branch was based on a20cdac and main has since reached 67e53e7 -- #641's LTX-2.5, #650's oracle policy, and #643's own tool-parser backfill. roadmap_v1.md conflicted. Resolved by the keyed-record rule rather than by three-way merge: took main's file WHOLESALE, then reapplied this branch's two edits IN PLACE -- the #607 issue row and the recipe-sweep table's --language-model-only line, both of which existed already and both of which this branch UPDATES rather than adds. Diff against main is +2/-2, exactly two replacements and no additions. The reason wholesale mattered here and not just as ceremony: this branch's tree still carried the PRE-CORRECTION #608 row, which said four of the five parsers were portable at the pin. #643 has since corrected that to "only inkling is portable from vLLM source". An automatic merge would have been free to keep either side; taking main's file wholesale means the corrected row is the one that survives, and I checked that it did rather than trusting the mechanism. Exactly one #607 row and one #608 row afterwards. scripts/agent-preflight.sh --staged: exit 0, all gates green. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This was referenced Aug 13, 2026
feat(TOOLS-PARSER-BREADTH): register
inkling — the ported engine had no registry face (#608 W1)
#683
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the LTX-2.5 campaign as a single squashed commit. One commit, authored
Ettore Di Giacinto <mudler@localai.io>— squash-merging a single-author PR preserves that author, which is why this is a fresh branch rather than a merge of the 100-commit campaign branch.What runs
The 21.00B joint video+audio DiT (48 blocks, video 4096 / audio 2048 with explicit audio↔video cross-attention, per-head gated attention); both VAEs including their encoders and the mel front-end; the pipeline layer (sigma schedule, guidance, latent spatial ×2 upsampler, duration head, embeddings connector); the quantized loaders with FP8, torchao NVFP4 and the first-party NVFP4 file all loading; the Gemma-4 12B text tower with its embedded tokenizer, token-exact against HuggingFace on the shipped 262144-entry vocab; and a typed prompt reaching cross-attention through the connector.
MiniMax-H3 moves behind the new
vllm::multimodal::VideoEngineseam, with its fold gate holding it to the same golden bytes — the second model was additive, not a fork.What is owed — stated because the above is easy to over-read
learnable_registers. It is the checkpoint's learned default reached through the real connector, not a depiction of a prompt.last_conditioning()is a change detector and says so in the public docs.PENDING— vLLM-Omni carries no native 2.5 and its diffusers adapter is a black box, so no production-configuration denominator exists.Also fixed
The shared scratch pool was device-blind (#516, and #486 is the same bug): one process-wide free list held blocks from every backend, so CUDA→CPU was a SIGSEGV and CPU→CUDA was silent — a uniform
0x7fff0000all-NaN no token gate can see. ADevicePoolis now bound to one backend at construction and the no-argument accessors are removed, which is what caught a live instance of the same defect landing inside this very campaign.Verification
Local — clean from-scratch Release build, zero warnings, 423/423, with the run's own denominator asserted against
ctest -Nrather than assumed.dgx.casa (GB10) —
sm_121aconfirmed bycuobjdumpon the built library, every fast-path arm enabled (cutlass-nvfp4,cutlass-fp8,marlin-nvfp4,fa2,fp4-mma, Triton AOT). 453 of 453 tests ran under oneflock $HOME/gpu.lock, held 14:36:09Z–15:29:37Z. Nine failures, all nine from the pre-existing set tracked as #233 / #248 / #305 / #614 — no new failure.test_minimax_h3Passed 19.02s where it previously SEGFAULTed;test_device_pooland all seven LTX suites pass.Issues filed for causes rather than instances during this work: #595, #598, #604, #611, #614.