port(ltx-2.5): L13 — a typed PROMPT conditions the render, and the third stale refusal goes away - #600
Merged
Conversation
…le L3 said did not exist (#435) LTX-2.5 advertises text-to-video and could not be typed at. `has_encoder()` was false and a prompt-carrying request was refused by name. This makes the prompt path RUN: a real string tokenizes with the tokenizer the checkpoint ships as a TENSOR, the torchao-NVFP4 Gemma-4 tower is materialized and forwarded, all 49 hidden states feed the existing aggregation, and both caption projections emit the two conditioning streams. One commit because the doc-checkpoint gate is per-commit and every part owes the same two documents; the four blocks below are how a reviewer should read it. ──────────────────────────────────────────────────────────────────────────────── 1. THE ORACLE L3 RECORDED AS IMPOSSIBLE WAS A `transformers` VERSION Phase L3 recorded that the tower could not be gated: no `gemma4_unified` in `CONFIG_MAPPING`, so it could not be built at reduced dims and there was nothing to compare against. That was never a property of the tower. MEASURED both ways on this box: /usr/bin/python3 transformers 5.3.0 -> KeyError 'gemma4_unified' /home/mudler/recon-cpu/venv transformers 5.12.1 -> builds and RUNS the tower So `Gemma4Model::ForwardHiddenStates` is now held to a RUNNING upstream instead of to invariants derived from its own output. gemma4.h has said "grounded + compiles" since it landed; compiling is not running. THE TOLERANCE IS A MEASUREMENT. Our stream carries bf16 and widens only on the way out; bf16 is also upstream's own resolved model dtype (base_encoder.py:41). So "is a difference a defect?" has a measurable answer: how far does UPSTREAM's answer move when the SAME upstream code runs at bf16 instead of f32? The generator measures exactly that, per state, and emits it as the bound. Nobody picked it, and it cannot be loosened to rescue a failure — loosening it means regenerating it, which means the oracle moved. state ours vs bf16 oracle the floor ratio 0 0 0.000923 0.00x 1 0.03125 0.072665 0.43x 3 0.191406 0.305009 0.63x 6 0.34375 0.559325 0.61x 9 0.46875 0.683670 0.69x 11 0.650391 1.098824 0.59x 12 0.149658 0.210947 0.71x Worst 0.71x over all 13 states. We are closer to upstream-in-bf16 than upstream-in-bf16 is to upstream-in-f32, at every layer; state 0 is bit-identical. The f32 arm is gated too, at the triangle bound, because a bf16 store is exactly what absorbs a reduction-order defect. THE FIXTURE DISCRIMINATES (§7.0(c)). The shipped 12B is not one repeated layer: (sliding x 5, full) eight times, and the two kinds differ — 16 q heads x 256 with 8 kv heads and a `v_proj`, against 16 x 512 with ONE kv head and no `v_proj` at all (`attention_k_eq_v` aliases V onto K), plus two rope types at two thetas and partial rotary on the full arm only. The reduced arch keeps every one and repeats the pattern twice, so a full layer is neither first nor last and a sliding layer FOLLOWS a full one. MUTATION-PROVEN: dropping `layer_scalar` -> 34 assertions RED; capturing the RAW last layer instead of the final-normed one -> 3 RED. Tree restored byte-for-byte (md5 040b97bf6b753f04be1855eaefc76ea1) and green re-confirmed after each. THE CONFIG IS NOT GUESSED. The shipped vonkaiser build has NO `__metadata__`, so its Gemma config cannot come from the file — which is also why upstream's own `GemmaAssets.from_single_file` raises on it. The committed config is the `__metadata__["gemma_config"]` of the OFFICIAL bf16 text encoder, read with an 84 KB HTTP RANGE request over the safetensors header: no large download, and the authoritative source rather than a reconstruction. It independently confirms §1.4's corrected width — `video_aggregate_embed.weight` is BF16 [4096, 188160] there, unpacked, where the NVFP4 build stores [4096, 94080]. `ParseHfConfig(json, source)` exists for that: a config does not always arrive as a sibling config.json, and writing one to a temp file to read it back would put the temp directory on a model path. ──────────────────────────────────────────────────────────────────────────────── 2. THE PROMPT PATH, AND A TOKENIZER THAT SHIPS AS A TENSOR LTX-2.5's text encoder embeds its tokenizer as `tokenizer_json` U8 [32169626] inside the one safetensors file (gemma_assets.py:34-36). Every loader in this tree assumed a sibling `tokenizer.json`, so none could read this checkpoint; hence `Tokenizer::FromHfJsonBytes`. GATE 1 — the WRAPPER, inline fixture, no asset. Strip; BOS prepended CONDITIONALLY (prepending unconditionally doubles it and drops the last token); LEFT padding so the valid run is the TAIL; truncate-then-prepend-then-re-truncate so a maximal prompt loses its LAST token rather than its BOS; an empty prompt still carrying exactly one BOS; and a refusal with no bos id, where upstream also raises (tokenizer.py:34-36). GATE 2 — REAL prompts, the REAL 262144-entry vocab, TOKEN-EXACT. Four prompts x 1024 positions, ids and mask, vs HuggingFace `tokenizers` 0.22.2: prompt 0 (10 valid tokens): first mismatching index = -1 (-1 = EXACT) prompt 1 (11 valid tokens): -1 prompt 2 ( 1 valid token ): -1 prompt 3 (43 valid tokens): -1 No tolerance appears in that gate. The prompts discriminate: one with leading and trailing whitespace, one EMPTY, one 43-token cinematic prompt with punctuation and digits. THE DISAGREEMENT, followed to the model author and recorded. `ltx_core` prepends BOS explicitly and says why — "Gemma 3 already emits it via post_processor; Gemma 4 does not, so we prepend" (tokenizer.py:12-15). diffusers instead passes `add_special_tokens=True` (pipeline_ltx2.py:339). MEASURED on the shipped tokenizer: its post_processor is a TemplateProcessing with ZERO special_tokens, so it adds nothing and diffusers would produce every prompt without a BOS. That measurement is emitted into the goldens header, so the reason is attached to the artifact. ──────────────────────────────────────────────────────────────────────────────── 3. THE TOWER ITSELF, CLOSING L6's RECORDED DEBT L6 recorded verbatim: "the torchao arm of the Gemma TOWER itself (`Gemma4Weights`) is not wired ... L6 loads the two caption projections, the asset pack and the geometry, and VALIDATES every tower module without materializing it." `Ltx2LoadGemmaTowerFromSafetensors` closes it, reusing L6's own `Ltx2DequantTorchaoNvfp4ToBf16` unchanged. WHY NOT `gemma4_weights.cpp`. The LTX file ships FLAT names (`model.layers.{i}.self_attn.q_proj.weight`); that loader reads the multimodal-wrapper form `model.language_model.layers.{i}....`. And every U8 width is HALF its logical one, so each layer's geometry is resolved from the CONFIG and the stored shapes are checked against it, never the reverse — a loader that trusted stored widths would build a tower of exactly half the right width whose matmuls all still conform among themselves. REFUSALS, each naming a specific wrong model: a PLE config against a checkpoint with no PLE tensors; `num_kv_shared_layers` != 0, where a shared layer reads a cache nobody wrote; a `v_proj` present or absent against what `attention_k_eq_v` declares, which is 16 kv heads of difference that every shape check passes; and a module in neither the BF16 nor the torchao-NVFP4 form. THE COST THE PORT DOES NOT PAY. Upstream pads every prompt to 1024 and runs all 1024 rows through a 12B tower; this runs the prompt's OWN length at the original absolute positions. Both halves of that equivalence are MEASURED, not argued: upstream's own padded-vs-short f32 spread is 3.2e-05 over a max|value| of 14.35, and our short run reproduces the padded run's valid rows at 0.5x the dtype bound. Writing that gate the obvious wrong way is kept in the test rather than quietly fixed: declaring seq_len = P + T while writing only T keys has attention read P slots of zeroes and call them keys — max|diff| 17.97 against max|value| 14.35. MEASURED END TO END on the real checkpoint (opt-in; ~33 GB host, ~26 min CPU): 329 torchao-NVFP4 modules dequantized; 48 layers, 8 of them full-attention at head_dim 512 / 1 kv head / no v_proj, 40 sliding at 256 / 8 kv heads prompt "a red fox running through deep snow at sunrise" -> 10 tokens -> video [1024, 4096] max|v| = 34.0715 mean|v| = 0.0401384 -> audio [1024, 2048] max|v| = 27.1555 a DIFFERENT prompt differs by max|diff| = 33.7318 6,292,805 assertions, 0 failed VALUES, not `isfinite`: a conditioning tensor of zeros would satisfy every shape and finiteness check and render an unconditioned clip, and a tower that ignored its input entirely would pass everything except the two-prompt arm. ──────────────────────────────────────────────────────────────────────────────── 4. WHY `has_encoder()` IS STILL FALSE Upstream does not hand the caption projections to cross-attention. Each stream goes through an `Embeddings1DConnector` first (embeddings_processor.py:70-117). Its MATH is ported — `Ltx2ConnectorForward` — but its WEIGHTS are not loaded: they ship inside the DiT file as `video_embeddings_connector.*` / `audio_embeddings_connector.*` and remain among the modules `Ltx2LoadDitFromSafetensors` REFUSES (ltx2_loader.h:96-99). MEASURED in the shipped FP8 DiT header: 372 such tensors, `learnable_registers [128, 2048]` on the audio arm, and `attn2.to_k.weight [4096, 4096]` confirming the video stream's 4096 is exactly the width the projections emit. So the conditioning this change computes has nowhere to go, and a `true` here would promise a render that cannot complete. The refusal therefore MOVES rather than lifts: `encoder_path` is still refused, but the message now names the connector weights instead of the tower, because the old message — "the Gemma-4 TOWER itself is owed" — is no longer true, and a stale refusal is worse than a blunt one. Flipping the flag is one change away, in files this row does not own. ──────────────────────────────────────────────────────────────────────────────── BASELINES, all matching, none moved test_ltx2 29/1615 test_ltx2_vae 16/1816 test_ltx2_pipeline 35/2358 test_ltx2_loader 20/2363 test_ltx2_video 17/170 test_ops_attention_cross 9/32 test_minimax_h3 79/57395 test_minimax_h3_video_fold 6/137 test_video_engine 11/254 test_capi 55/505 test_gemma4_honesty 2/6 test_gemma4_rocm_fp8_seams 2/10 test_gemma3_forward 3/503 test_tool_parser_gemma4 72/161 test_gemma4_vision_tower / registry_e2e / audio_tower / paged_engine 1/0 each test_ltx2_text_encoder 17/3350 -> 23/3583 (+6 cases; +16 assertions with CHECKPOINT_ROOT, +6,289,222 with the e2e opt-in) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…row/LTX25-L10-TEXT-TOWER
…e false Docs conflict was docs/FEATURES.md for the FOURTH time on the same keyed row (see #595). Took the campaign side wholesale: it is strictly newer and a superset, since L10's own text still said "STRUCTURAL e2e only, no render-quality claim", which stopped being true when L9C wired the connector. The substantive finding is in the merge, not the docs. ltx2_video.cpp:672 refuses a supplied encoder_path, and its reason is stated precisely: the Embeddings1DConnector's MATH is ported but its WEIGHTS are not loaded, they ship inside the DiT file, and Ltx2LoadDitFromSafetensors "still lists them among the modules it REFUSES". That was accurate when L10 was written against its base. It is NOT accurate on this merged tree. L9c landed exactly that loader: Ltx2LoadConnectorWeights is called at ltx2_video.cpp:792,794 and ltx2_loader.cpp:417 says in terms "NOT unported: phase L9c materializes them". So the two halves of prompt conditioning have met for the first time. L10 makes the tower RUN -- tokenizer token-exact on the shipped 262144-entry vocab, all 49 hidden states within the oracle's bf16 noise floor, both caption projections producing the 4096 and 2048 streams. L9c makes the connector those streams must pass through real, with the checkpoint's own weights. Neither branch alone could see that, which is exactly the class of thing a merge is supposed to surface and usually does not. Deliberately NOT lifting the refusal in this merge commit. Whether the last hop is complete is a question for a fresh implementer with a gate, not for the coordinating session to assert. The stale tensor-list comment at ltx2_loader.h:96-99 is owed the same correction. This is the third refusal in this campaign whose REASON went false while the refusal itself stayed correct. A refusal that names its cause is worth far more than one that does not -- but only if something re-reads the cause when the tree moves. Issue: #435 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ird stale refusal goes away
The last hop of the LTX-2.5 campaign. Two branches had each built half of it and
neither could see the other:
* L10 made the Gemma-4 tower RUN — the embedded tokenizer reaches a prompt
string, the torchao-NVFP4 tower materializes onto `Gemma4Weights`, all 49
hidden states come out within the oracle's own bf16 noise floor, and both
caption projections turn them into the 4096/2048 conditioning streams.
* L9c put the `Embeddings1DConnector` on the render path with the checkpoint's
OWN weights.
L10's `encoder_path` refusal said the connector weights were "still among the
modules `Ltx2LoadDitFromSafetensors` refuses (ltx2_loader.h:96-99)". On the
merged tree that was FALSE — `ltx2_loader.cpp:417` already recorded them as
loaded elsewhere by L9c. That is the THIRD refusal in this campaign whose stated
reason went stale, so the header comment it cited is corrected in the same
change rather than left to mislead a fourth reader.
WHAT NOW HAPPENS. `encoder_path` loads the tower, the tokenizer and both caption
projections; `Ltx2SelectTextFeatureVariant` resolves the V1/V2 shape from the
DiT's own transformer config; and a request's `prompt` is tokenized, encoded,
projected, run through the connector and handed to cross-attention, per request.
`has_encoder()` is true. `ltx2-gen` grows `--encoder`, `--encoder-config` and
`--prompt`, so the capability is reachable through the ABI rather than only from
a test.
ONE GENUINE GAP, refused rather than defaulted: the only shipped LTX-2.5 text
encoder carries NO `__metadata__` at all, so the Gemma config is an INPUT. It
comes from the checkpoint's `__metadata__["gemma_config"]` when present and from
the new `encoder_config_path` extra when not; neither source is a refusal, both
sources is a refusal. `layer_types`, `global_head_dim`,
`num_global_key_value_heads` and `attention_k_eq_v` each resolve a DIFFERENT
tower out of a byte-identical tensor set, which is the same polarity
`dit_config_path` already has.
TWO THINGS RECORDED AT THE CODE rather than left for a reader to rediscover.
`Ltx2TextEncoderConditioning` and `Ltx2ConnectorCreateEmbeddings` are two ports
of overlapping halves of `embeddings_processor.py:70-117` and BOTH carry the
right-pad sort, so composing them sorts an already-sorted stream; that is the
identity because a stable argsort of a 0/1 key is idempotent, and the engine now
ASSERTS the precondition instead of relying on the claim. And the tower runs on a
CPU queue even on the device arm, because everything in `ltx2_text_encoder.h` is
f32 by declaration — stated as owed, not hidden.
THE INSTRUMENT, and why it can see the difference. `Ltx2VideoEngine::last_conditioning()`
reports an FNV-1a digest and an absmax over the exact f32 buffers
`Ltx2ModalityInput::context` pointed at, after the connector and immediately
before the denoise loop. A frame statistic was deliberately NOT used: L9c's
reviewer found a scene and a colour field INDISTINGUISHABLE to the existing frame
analyzer (neighbour |dx|/sd 0.093 vs 0.033) and needed contact sheets. A digest
is a function of the bytes rather than a summary of them, so it has no such blind
spot; the absmax is what separates "the tower ran" from "the tower returned
zeros", which would otherwise satisfy a difference check for the wrong reason.
GATE. tests/vllm/multimodal/test_ltx2_video.cpp 24 -> 29 cases, 365 -> 485
assertions, on a reduced-dimension text encoder written in the SHIPPED format:
bf16 tower with the mixed sliding/full geometry and no `v_proj` on the full
layer, torchao-NVFP4 caption projections with a PADDED swizzled scale, the
tokenizer and HF sidecars stored as tensors, and no `__metadata__`. The fixture's
DiT config also gains the three V2 marker keys it was missing — it carried only
`caption_proj_before_connector`, which is the PARTIAL set upstream refuses, so it
gated no variant selection at all. The three added values are read from the
first-party NVFP4 DiT's own metadata.
RED before GREEN: against the pre-L13 engine the new cases do not compile —
`kLtx2EncoderConfigPathExtra`, `Ltx2ConditioningTrace` and `last_conditioning`
do not exist — which is the honest statement that the surface could not be
expressed. Mutation evidence, each applied alone and the tree restored
byte-identically (sha256 verified):
M1 tower encodes a CONSTANT instead of `gen.prompt`
-> 3 RED: both digests equal across two prompts, and the frames equal
M2 prompt path skips `RunConnector`
-> 3 RED in "goes through the CONNECTOR": same prompt, same tower, same
DiT weights, only the connector differs, and it stopped mattering
M3 drop the missing-gemma-config refusal
-> 2 RED. Worth recording: an EARLIER version of that subcase asserted
only that the message named `encoder_config_path`, and M3 left it GREEN
— the load still failed, but with "cannot open " from a fallthrough
reading the empty path. A refusal firing for the wrong reason is
exactly what this phase exists to stop shipping, so the assertion was
strengthened to the discriminating clauses before being called a gate.
NOT gated, and named rather than claimed unreachable: the additive-mask
right-pad guard. I did not construct a probe that reaches it, and per this
campaign's own rule a mutation that moves nothing is not evidence of
unreachability.
Full gate on this worktree, CPU Release build, x86_64: BUILD_EXIT=0, no
"No space left"/"BFD assertion" in the build log, 61G free on /. ctest -N 408,
`ctest -j8` 408/408 passed 0 failed (1 skipped: test_voxtral_e2e). Focused:
test_ltx2_video 29/485, test_ltx2_loader 20/2371, test_ltx2_text_encoder
23/3583, test_ltx2_pipeline 37/2382, test_ltx2 29/1615, test_video_engine
11/254, test_capi 55/505 — all SUCCESS.
`scripts/agent-preflight.sh --staged` passes `doc-checkpoint --staged` and
`now-current --staged`. Its `doc-checkpoint range` failure is INHERITED: all
three commits it names (b0aa475, d67f812, aa6aa0e) are ancestors of the
base SHA 67a7b1c and the same failure reproduces on the untouched base.
Issue #435.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
…that was only a witness Review findings S1, S2, S4-S7 on `43aa5837`. The reviewer's verdict on the code was that it is the strongest of this campaign and that the docs cell over-reads it; nothing here redesigns anything. S1 (BLOCKING) — A PUBLIC DOC ASSERTED A RENDER NO RUN SUPPORTS. `docs/FEATURES.md` read "e2e at 320x192/25f from a TYPED PROMPT via Gemma-4: coherent scene, valid MP4+WAV". Both halves were attached to a run that did not happen. The 320x192/25f arm is L9c's, and this record's own L9c section says it plainly: the conditioning was `--prompt-valid-rows 24` over SYNTHETIC N(0, 0.2), 104 of 128 rows were the connector's trained `learnable_registers`, and "It is not a depiction of a prompt." L10's real-checkpoint run produced conditioning only, no frames. L13's own gate is fixture-only, CPU Release. OPTION (b) TAKEN — state what happened, do not manufacture the run. The arithmetic is the reason, not squeamishness: L9c's 320x192/25f arm bottomed out at 68.2 GiB MemAvailable on a 119 GiB box that REBOOTS rather than OOM-kills, the tower is a further ~24 GB of host bf16 (ltx2_video.cpp:846), and at the time of writing dgx.casa's `$HOME/gpu.lock` was held by another session's `ctest` with root at 99% (62 GiB free). Risking the box for one number, with other agents live, is not a trade this finding asks for. The cells now say the render was register-conditioned and that a prompted one is OWED. S2 — `docs/STATUS.md` still said "the Gemma-4 tower is owed, so no prompt encodes yet", which this PR made false. Corrected without growing the page: the new cell is 216 chars, under the 220 that `oversized_cells` counts, so the ratchet is untouched. S4 — THE COMPOSITION'S VALUE ORACLE IS OWED, and is now recorded where a reader meets the instrument. `last_conditioning()` is a WITNESS, not a gate: the reviewer's R6 (video conditioning x1.5 after the connector) and R7 (conditioning rows REVERSED) each passed all 485 assertions with exit 0. A digest detects CHANGE; nothing at this level pins VALUES. The per-brick oracles are real and untouched; the two JOINS - `Ltx2ConnectorCreateEmbeddings` and the `Generate` composition - have none, and both mutations live in exactly that gap. Recorded rather than closed, with the closure specified and its one prerequisite VERIFIED so the next implementer does not have to: upstream's counterpart is `EmbeddingsProcessor.process_hidden_states` (embeddings_processor.py:97-117), in the same package `gen-ltx2-pipeline-goldens.py` already executes at section 10, and that generator reproduces its committed output BYTE-FOR-BYTE on the pinned upstream (md5 53e2a6aba8885d7d58302ad0b7b09eb4 both sides, LTX-2 clean at fd4ded7f, vllm-omni at a4ea67a2), so a section can be added without disturbing anything already gated. S5 — A DATA RACE ON THE API THE HEADER ADVERTISES, fixed rather than documented. `last_conditioning()` read `impl_->trace` without the mutex `Generate` holds. It now returns BY VALUE under that mutex; a reference could not be made safe, since the lock is released before the caller reads. The second half is real too: the trace is filled before the denoise loop, so a `Generate` that throws later left a healthy-looking trace for a render that never completed. New `completed` flag, set at the single successful return. Gated in BOTH directions rather than asserted, on a REAL refusal - keyframe / reference conditioning is refused after the trace is written, so a prompted request carrying a reference image fills the trace and then fails: * drop `completed = true` -> 3 cases RED, exit 1 * set it at trace-fill time (pre-fix) -> the new case RED, exit 1 Tree restored byte-identically after each (sha256 22342f84...c546 before and after). S6 — TRUE VALUES, NOW RECORDED. The four V2 markers were correct but nothing in the repo carried the header they came from, so the next reader needed the 18.72 GB checkpoint mounted. The observed `__metadata__["config"]["transformer"]` is now recorded with its path, byte size and revision, at the fixture and in the record. All four present, none drifted; `text_encoder_norm_type: PER_TOKEN_RMS` corroborates independently and the selector deliberately does not read it. And a CAUSE CORRECTED. The fixture said an earlier partial marker set "gated no variant selection at all". `Ltx2SelectTextFeatureVariant` DOES refuse a partial set (ltx2_text_encoder.cpp:184-192). It never fired because no production path CALLED the selector before L13 - the keys and the first caller landed in the same commit. S7 — ONE GUARD PRESENTED AS A POSITIVE IS UNGATED. The right-pad precondition guard gets the same sentence the mask-value guard already had. No reachability is claimed: no probe was built that fails to reach it, and a mutation that moves nothing is not evidence of unreachability. ONE SELF-CORRECTION worth naming, since it is this campaign's own failure mode. I first replaced "+~24 GB text tower" with an on-disk figure, believing the number ungrounded. It is grounded - 12B params at bf16, stated at ltx2_video.cpp:846 and in docs/USAGE.md - and the resident figure is the useful one for a memory cell. Reverted. GATE. CPU Release, clean configure, BUILD_EXIT=0, build logs clean of `No space left` / `BFD assertion`, df 53-67G free throughout. `ctest -N` 414 registered; full `ctest -j4` 414/414 passed, 0 failed, exit 0 (test_voxtral_e2e skipped). test_ltx2_video 29/485 -> 30/499 (the one new case), test_ltx2_loader 20/2371, test_ltx2_text_encoder 23/3583, test_ltx2_pipeline 37/2382, test_capi 55/505 - all unchanged and all SUCCESS with exit 0 read separately from the summary. `scripts/agent-preflight.sh --staged` passes `doc-checkpoint --staged`, `now-current --staged` and `check-public-doc-tables`. Its `doc-checkpoint range` failure is INHERITED and re-verified here: b0aa475, d67f812 and aa6aa0e are each `--is-ancestor` of the base SHA 67a7b1c. Issue #435. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
S8, the one blocking finding of the re-review. `docs/USAGE.md` still described the pre-L13 world in two places, and this PR is what falsified them. `:487` claimed "the Gemma-4 text tower is still not ported"; `:499-510` restated that a prompt is refused because the connector weights sit among the modules the DiT loader refuses, so "`encoder_path` is still refused" and "`has_encoder()` is still false". At this head `im.has_encoder = true` on the `encoder_path` branch (`ltx2_video.cpp:893`), and the connector families load under their own contract (`ltx2_loader.cpp:416-427`). The file also contradicted itself: `:372`, added by this PR, opens "A typed prompt works." Provenance, since a later reader should be able to re-check the cause rather than trust the repair: `:487` arrived at `e48c86253` (L9c) and `:499` at `ab6671394` (L10), BOTH TRUE when written; `43aa58377` in this PR made them false and left them standing. The "renders a scene, not YOUR scene" material is RE-ATTRIBUTED, not deleted. It is still true of the L9c render and of the embeds path, so it now says which run it describes: `--prompt-embeds` with `--prompt-valid-rows 24` over synthetic N(0, 0.2) rows, no tower on the path, 104 of 128 connector rows the connector's own trained `learnable_registers`. What it must stop being is a property of the `--encoder` + `--prompt` command L13 rewrote the example into, because that is the S1 defect in the other direction: NO prompted render has been run in either direction, so the page now claims nothing about what that command renders -- not that it puts a fox on the screen, and not that it fails to. Two accuracy repairs in the same sweep: - The `Ltx2ConditioningTrace` header said both composition mutations "passed all 485 assertions ... on this exact head". 485 was `43aa58377`'s count; this head is 499, so a comment whose whole purpose is precision named a number no run of it could produce. RE-MEASURED here rather than transcribed, because a reviewer report is an input and not a gate result: video conditioning scaled x1.5 after the connector, and the conditioning rows reversed, each applied alone, each recompiled and relinked (verified in the build log), the tree restored byte-for-byte and re-verified green between legs. BOTH still pass, at 30 cases / 499 assertions, exit 0. The verdict did not move; the COUNT had drifted. - `last_conditioning()` was documented as safe to call from a server thread while another renders. True as to safety, but `Generate` holds that same mutex for the whole render, so the caller blocks for minutes rather than getting a stale-but-immediate answer. Stated. One sibling found by sweeping the whole file rather than fixing the instance in front of me, which is what #604 says has failed four rounds running. The AUTO duration comment said the head "needs the encoded prompt this engine cannot produce" -- a fifth refusal-reason that went stale while its refusal stayed correct, since L13 produces exactly that. The real reason is now recorded: no duration head is ever constructed here, and `duration_head_path` is accepted in `kKnownLoadExtras` while no code reads it, so the extra is inert. Named in `docs/USAGE.md` too, where a reader would otherwise supply it and silently get the recipe default. NOT PAID, and named rather than left to be found: `.agents/benchmark-record.md`'s L13 section carries the same superseded 485/485 pair. Correcting it is not free -- `check-doc-checkpoint` classes any benchmark-record edit as a measurement and demands `docs/STATUS.md` + `docs/BENCHMARKS.md`, and those are the surfaces the re-review passed and told this repair not to disturb, with STATUS at its `oversized_cells` ratchet. Manufacturing an edit to a blessed surface to turn a gate green is the thing this protocol forbids, so it is reported instead. Gate (CPU Release, no CUDA): BUILD_EXIT=0, 0 warnings, no `No space left` / `BFD assertion`, disk 87%. `ctest -N` 414 registered; full `ctest -j8` 413 pass / 1 skip, the single failure `test_engine_core_proc` a known `-j` flake that passes serially. `test_ltx2_video` 30 cases / 499 assertions, exit 0. `agent-preflight.sh --staged` doc-checkpoint and now-current both ok; the `doc-checkpoint range` failure is pre-existing on this branch (`b0aa475a3`, `d67f8125e`, `aa6aa0ecd`) and untouched here. Refs #435, #604. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The previous repair (56a917c) fixed the two blocks it was pointed at and then proved completeness by grepping for `still not ported`, `` `encoder_path` is still refused `` and `` `has_encoder()` is still false ``. All three returned nothing -- but those were the phrasings it had just written, so the search could only confirm itself. Searching the vocabulary of the CLAIM instead of the sentence found six more instances of the same false statement, in three docs/USAGE.md sections that were never swept. The file asserted both "A typed prompt works." (:372, added by this PR) and "there is no text encoder" (:1817). Both cannot be true. CLOSED, each against the code site that makes it true: 1. :1816-1821, a bolded present-tense block whose six sub-claims are all false at this head. `ltx-2.5` is one of exactly two registered video families (REGISTER_VLLM_VIDEO_FAMILY, ltx2_video.cpp:1529); --encoder loads the tower and sets has_encoder (ltx2_video.cpp:893); both VAEs and the pipeline layer ship (ltx2_video_vae.cpp, ltx2_audio_vae.cpp, ltx2_pipeline.cpp); /v1/videos registers family-agnostically through LoadVideoEngine; and ltx2-gen renders, documented at :436-450 of this same file. It arrived at 3d89f6f -- the first LTX commit, where it was true -- and was never revisited as L3 through L13 built each of the six pieces it denied. 2-3. :1920 heading "(no render path yet)" and :1924-1925 "There is still no render path, so these are library entry points and not a command". Those loaders are exactly what the render path drives: --dit / --video-dit reaches Ltx2StreamDitToDevice / Ltx2LoadDitFromSafetensors at ltx2_video.cpp:576-577, and --encoder / --video-encoder reaches Ltx2LoadTextEncoderFromSafetensors at ltx2_video.cpp:851. 4-5. :1963 heading "(no render path yet)" and :1965 "There is no LTX-2.5 render path. This section documents one brick." It is one brick OF the shipped path. 6. :1927-1932 said the DiT loader refuses FIVE families, "and the two *_embeddings_connector towers", "which still reports every one of them in Ltx2DitCheckpoint::unported". There are THREE. UnportedFamilies filters the connectors at ltx2_loader.cpp:439 via LoadedElsewhere, and RefuseUnported's own message says so in capitals at ltx2_loader.cpp:461-464 -- the checker's message is the authority on what it enforces. This is the identical claim the previous repair retired 400 lines earlier, left standing precisely because the sweep was built from the text that had just been fixed. Completeness was proved this time from the claim's vocabulary and its subject rather than from the repaired sentences: no render path, no text encoder, no VAE, no pipeline, will not work, one brick, not a command, unported, refuses, embeddings_connector, not ported, is refused, still no, yet, library entry point, layout and forward, asking the video engine -- over docs/, README.md, examples/ and both --help surfaces a user sees. 189 hits read. REPORTED, not fixed, because fixing half of it is worse than reporting it: this branch bumped VLLM_ABI_VERSION 17 -> 18 and added vllm_video_engine_family (36 VLLM_API declarations, was 35) at 3db9233, and left docs/USAGE.md:1464-1465 and README.md:189 and :388 all saying "17" and "35". origin/main is self- consistent at 17/35, so this PR is what falsified them, and docs/STATUS.md:119 already says v18 -- the tree disagrees with itself today. The USAGE half alone would pass check-doc-checkpoint, but check-readme-structure refuses the README half without a landing-source edit, and no landing-source edit is legitimately owed here: examples/server/main.cpp:15's "published as vllm_server_main at ABI v17" is a statement about WHEN that symbol shipped, verified correct at c1716fd. Manufacturing an edit to a blessed surface to turn a gate green is what this protocol forbids, and correcting one of two surfaces that carry the same number would leave a reader unable to tell which is right. The owed change is three lines: 17 -> 18 and 35 -> 36 at those three sites, together. Gate, CPU Release, x86_64, this worktree, no CUDA: - CONFIGURE_EXIT=0, BUILD_EXIT=0, 0 compiler warnings; build log clean of "No space left" and "BFD assertion"; disk 79G -> 64G free, 71G after cleanup. - ctest -N: 414 registered. - full ctest -j8: 413 passed, 1 skipped (test_voxtral_e2e), 1 failed (test_engine_core_proc) -- the known -j flake, which PASSES serially: "1/1 Test #195: test_engine_core_proc ... Passed". - test_ltx2_video: 30 cases / 499 assertions, exit 0 -- unchanged from this head's recorded counts, as a documentation-only change must be. - check-doc-checkpoint --staged: OK. check-public-doc-tables: OK. The preflight "doc-checkpoint range" failure is INHERITED and re-verified: b0aa475, d67f812 and aa6aa0e are each an ancestor of the base 67a7b1c. This is instance sixteen in the campaign #604 tracks, and the second in this file and this PR. Refs #435, #604. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
…#435) The last round corrected the post_processor rationale in the HEADER and left the SOURCE saying the opposite thing, twelve lines above the `Encode` call it annotates. The two made contradictory predictions from the same premise: the header said a future post_processor makes us WRONG, the source said it keeps us RIGHT. F7's whole content was "the rationale was inverted", so half of it was still open. VERIFIED AGAINST UPSTREAM, not taken on trust. The reviewer could not reach the anchor; this was read at /home/mudler/_git/LTX-2 pin fd4ded7f: packages/ltx-core/src/ltx_core/text_encoders/gemma/tokenizer.py:37-43 encoded = self.tokenizer( text, padding=False, truncation=True, max_length=self.max_length, return_tensors="pt", ) That is `__call__` with its DEFAULT `add_special_tokens=True`, so upstream DOES run the post_processor. The HEADER had the direction right; the SOURCE was the wrong one, and it is the source that is corrected here. A THIRD claim in the same comment was also false. "Doing both would double the BOS" is not true: upstream's own guard at :45 (`not input_ids or input_ids[0] != bos_id`), mirrored at ltx2_text_encoder.cpp:712, absorbs a leading BOS from either call. The real exposure is a post_processor that adds anything ELSE, and the comment now says that instead. SIBLING SWEEP, built from the SUBJECT (post_processor, BOS, add_special_tokens, Encode) rather than from the sentences being fixed, since a grep built from the new text only confirms itself. Two more hits, both corrected: * gen-ltx2-prompt-tokens-goldens.py:86 annotated its `add_special_tokens=False` as transcribing ":38-43" -- the same conflation, in the generator that produces the committed goldens. Anchor corrected to :37-43 and the deliberate flag flip is now stated. * ltx2_text_encoder.h:421 called the BOS prepend "unconditional", which contradicts test_ltx2_text_encoder.cpp:2398 ("i.e. conditional") and :45 itself. It is conditional, and the guard is why. ──────────────────────────────────────────────────────────────────────────────── TWO COMMENT OVERCLAIMS, PROVED BY THE REVIEWER'S MUTATION Replacing `Ltx2ComputeRightPadOrder`'s stable partition with the identity permutation reds two OTHER cases (22 assertions) and leaves the new conditioning case entirely green. `out.conditioning` and `want_f32` both reach `sort_index` and `additive_mask` through the SAME function over the SAME `out.mask`, so a defect there cancels and those ~40 assertions cannot fire. "Held EXACTLY" was an assertion true by construction wearing a gate's label. This is NOT a coverage hole and no test changes. Both are genuinely gated by "ltx2 text: additive mask, right-pad ordering and the binary mask" and "ltx2 text: the encoder -> conditioning hand-off", which compare against committed upstream goldens rather than against a second call of the code under test. The comment now says what its assertions do establish (the mask is threaded through unaltered, and the mask itself is held to the golden) and names the two cases that actually gate the ordering. ──────────────────────────────────────────────────────────────────────────────── TWO COVERAGE LIMITS RECORDED, NOT FIXED * The loader case's "the q|k|v concat ORDER" claim holds for the LTX path ONLY. `Ltx2LoadGemmaTowerFromSafetensors` uses its own `TowerConcat` (ltx2_text_encoder.cpp:943-945); `gemma4_weights.cpp:281-295` is a SEPARATE implementation this suite never loads, and mutating it leaves every case here green. That path owes its own gate. * The rope-table instrument covers the FULL-ATTENTION arm only. The sliding layers' `default` rope at theta 1e4 has no equivalent f32 instrument and is reached only through the hidden states, where the same bf16 noise-floor argument works against resolving a config-carried angle defect. ──────────────────────────────────────────────────────────────────────────────── DOCS Comment-only in src/include/tests, so AGENTS.md owes nothing here -- but check-doc-checkpoint is path-derived and demands FEATURES.md (from src/vllm/model_executor/models/) and USAGE.md (from include/vllm/). The prior repair at 9fbc682 paid it the same way. Both edits are GENUINE rather than gate-appeasement: they record the tokenization divergence this commit just verified, which is a real user-facing caveat -- our prompt tokenization mirrors upstream only while the checkpoint's post_processor stays empty. USAGE.md:454 and :473 are left alone; they are inherited from the base branch and belong to PR #600's lane. The FEATURES.md note is folded into an existing paragraph rather than added as a new one, because check-public-doc-tables budgets prose paragraphs (21) as well as table cells (220 chars) and rows (600), and the LTX row's cell is already at 217. CODE IS BYTE-IDENTICAL. The only non-comment line in the diff is a trailing comment on an unchanged statement. GATE, CPU, Release, VLLM_CPP_CUDA=OFF: BUILD_EXIT=0, no "No space left" / "BFD assertion", df 86% used test_ltx2_text_encoder: 26 cases / 4115 assertions, Status SUCCESS, exit 0 (matches the declared baseline exactly) ctest -N: 414 (the brief said 409; this configure enumerates 414 at the unmodified head, and a comment-only diff cannot register a test) ctest: 414/414 passed, 0 failed, exit 0, first pass under -j 8, no flaky re-runs needed binary md5 unchanged across the post-docs rebuild (8b48f39b...) check-doc-checkpoint "range" still reports b0aa475, d67f812 and aa6aa0e. All three are ancestors of 9fbc682, inherited, and not repairable from here without editing #600's lane. Issue: #435, #604 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
, #604) The re-review found `test_ltx2_text_encoder.cpp:635` citing `ltx2_text_encoder.cpp:943-945` for "the LTX tower loader's own `TowerConcat`". Lines 943-945 are the `q_proj` `TowerModule` load. The concat calls are :951 and :953; the helper is defined at :835. The substantive claim was measured and is true, and only the number was wrong -- which is the worst shape for this defect, because a reader who follows the anchor concludes a TRUE claim is unsupported. Fixing one number and stopping is what let the class survive three review rounds, so every `file:line` this branch ADDS over `row/MODEL-DIFFUSION-LTX25` was resolved against the tree it points into. Extracted mechanically by SHAPE off the added lines of `git diff origin/row/MODEL-DIFFUSION-LTX25...HEAD` (`name.ext:NN`, `name.ext:NN-MM`, bare `:NN` / `:NN-MM`, "line(s) NN", "SS N"), not by grepping the sentences already known to be wrong -- a grep built from text you just fixed only confirms itself. 19 citations added, in 6 files. 19 resolved. 8 wrong, at 6 comment sites. site cited is ------------------------------------------------------------------------ test_ltx2_text_encoder.cpp:635 ltx2_te.cpp:943-945 q_proj load test_ltx2_text_encoder.cpp:1959 rope_utils.py:187-245 ends at 254 test_ltx2_text_encoder.cpp:1992 rope_utils.py:233 blank line gemma4.h:260 gemma4_unified.py:257-274 prev fn tail gemma4.h:260 rope_utils.py:187-245 ends at 254 gen-ltx2-gemma-tower-goldens.py:378 gemma4_unified.py:214-218 call, no lookup gen-ltx2-gemma-tower-goldens.py:378 rope_utils.py:187-245 ends at 254 ltx2_text_encoder.cpp:1055 gemma4.cpp:300-312 cuts the cast The three `modeling_rope_utils.py:187-245` sites are one anchor repeated. The function is `_compute_proportional_rope_parameters`, :187-254, and all three sentences name its ZERO PADDING -- which is `torch.zeros(nope_angles, ...)` at :246, ONE LINE past the cited end. Following the anchor is following it to the `torch.cat(` that opens two lines earlier and stopping before the argument that matters. Corrected to :187-254, with :246 named where the padding is the claim. `modeling_rope_utils.py:233` is a blank line. `rope_angles = int(rope_proportion * head_dim // 2)`, which the comment quotes, is :234. `modeling_gemma4_unified.py:257-274` opens on the PREVIOUS function's `return` (:257) and stops before `forward`'s own (:275). The `emb = torch.cat((freqs, freqs), dim=-1)` the sentence mirrors is :271, so the anchor does contain it, but both ends land in the wrong construct. Corrected to the whole method, :259-275, with :271 named. `modeling_gemma4_unified.py:214-218` is the CALL, `rope_init_fn(self.config, **rope_init_fn_kwargs)`. The sentence claims the class "routes `rope_type: proportional` to `_compute_proportional_rope_parameters`", and the routing is `ROPE_INIT_FUNCTIONS[rope_type]` at :207, outside the range -- a reader at :214-218 cannot see what `rope_init_fn` resolves to, so the claim reads unsupported. Widened to :206-218. `gemma4.cpp:300-312` opens on the weight-less V `RmsNorm` at :300 and closes on the `} else {` at :312, which EXCLUDES the `vt::CastF32` pair at :313-314 that the sentence is about ("run a CastF32 on every K and V it wrote"). The `kv.dtype != adt` arm is :306-315. Corrected, with the two `DBuf`s and the casts named separately, because those are the two costs the sentence charges. The eleven that resolve as written, checked one by one, not assumed: LTX-2 `tokenizer.py:37-43` (three sites, the `self.tokenizer(...)` call), `:44-46`, `:45`, `:12-15`, `gemma_assets.py:335-386` (`build_text_encoder_tensors_from_gemma_root`, :335 to its `return` at :386), `base_encoder.py:41`; diffusers `pipeline_ltx2.py:339` (literally `add_special_tokens=True`); ours `gemma4_weights.cpp:281-295`, which does span all three qkv arms as the previous round reported. Two non-line anchors added by the F7 commit -- the two test-case NAMES it cites as the ones that gate the ordering -- exist verbatim at :1115 and :1166. PINS. LTX-2 at `fd4ded7f`, tree clean. diffusers at `3a2f35d4`. transformers 5.12.1 at `/home/mudler/recon-cpu/venv`, the interpreter the tower generator actually ran under; both files are md5-IDENTICAL to `/home/mudler/_git/ transformers` @ `7d06b1a5`, so the numbering above is not an artifact of which install was read. F10's anchor. `:951-953` and not `:835`, because the sentence is the LTX half of a contrast whose other half is `gemma4_weights.cpp:281-295` -- assembly SITES, not a helper definition -- and because the qkv concat ORDER is what the case gates and what the previous round's mutation (`q,k,v` -> `q,v,k`, 14 assertions RED) perturbed at :951. `:835` is named alongside it so the helper is still one hop away. DOCS. The code is byte-identical; this is comments and two documents. `check-doc-checkpoint` is path-derived and demands `FEATURES.md` (from `src/vllm/model_executor/models/`) and `USAGE.md` (from `include/vllm/`), which is the same bill `9fbc68256` and `ddfcb516a` paid. Both edits are genuine. `USAGE.md` still opened its LTX-2.5 gate section with "**There is no LTX-2.5 render path yet** ... no text encoder, no VAE, no pipeline and no `/v1/videos` route for it -- asking the video engine for LTX-2.5 will not work." Every clause is false, and the SAME DOCUMENT contradicts all of them: `ltx2-gen` renders end to end at :412, `--video-family ltx-2.5` pins the family on the server at :435, and the text tower is this row. Rewritten to say what the section is for. The refusal list that follows is left byte-for-byte alone: it is inherited text, it is not part of this finding, and `ltx2.h:38` and `:47-49` do support the two clauses of it I checked. `FEATURES.md` pointed at "`ltx2_text_encoder.cpp` names the line that would have to change" -- an unresolvable pointer, in a commit about unresolvable pointers. It now names `Ltx2TokenizeGemmaPrompt`. A symbol, deliberately, because a line number in a public document has nothing that can keep it true. Folded into the existing paragraph: the 700-char prose cap and the 220-char cell cap make an addition an EVICTION of someone else's content, and the LTX row's two cells are already at 217 and 215. `docs/USAGE.md:454` and `:473` untouched -- #600's lane. GATE -- CPU, Release, `VLLM_CPP_CUDA=OFF`, `build-f10`, deleted after. cmake --build build-f10 -j 16 BUILD_EXIT=0; 0 hits for 'No space left|BFD assertion'; df 85% ctest -N 414 ctest -j 8 --output-on-failure 414/414 passed, 0 failed, exit 0 ./build-f10/tests/test_ltx2_text_encoder 26 cases | 4115 assertions | 0 failed Status: SUCCESS, exit 0 (baseline 26 / 4115 matched exactly) First pass under `-j 8`; no suite needed a serial re-run. `doc-checkpoint --staged` and `now-current --staged` both ok. `doc-checkpoint range` still reports `b0aa475a3`, `d67f8125e` and `aa6aa0ecd` -- the identical three failures this worktree recorded BEFORE any edit, all ancestors of `9fbc68256`, inherited and not repairable from here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last hop of the LTX-2.5 campaign. Issue #435; spec
.agents/specs/ltx-2-5.md(operator-owned — see "what the spec needs" below).This PR carries L10 as well. It branches from
integrate/l10@67a7b1cd9(the campaign merged intorow/LTX25-L10-TEXT-TOWER), which is not an ancestor ofrow/MODEL-DIFFUSION-LTX25, so the diff includesab6671394and5b5bd69e7. That is deliberate: L13 cannot exist without L10, and the whole point of this change is what happens when the two halves finally meet.The finding
Two branches had each built half of the hop and neither could see the other.
Gemma4Weights, all 49 hidden states come out within the oracle's own bf16 noise floor, and both caption projections turn them into the 4096/2048 conditioning streams.Embeddings1DConnectoron the render path with the checkpoint's OWN weights.L10's
encoder_pathrefusal said the connector weights were "still among the modulesLtx2LoadDitFromSafetensorsrefuses (ltx2_loader.h:96-99)". On the merged tree that is false —ltx2_loader.cpp:417already records them as loaded elsewhere by L9c. That is the third refusal in this campaign whose stated reason went stale, so the header comment it cited is corrected here rather than left to mislead a fourth reader, along with the two other places that repeated it (ltx2_video.h'sallow_unported_modulesnote andltx2_text_encoder.h's "the connector is NOT ported" owed-note, which stopped being true at L5).Does a typed prompt now condition the render?
Yes.
encoder_pathloads the tower, the tokenizer and both caption projections;Ltx2SelectTextFeatureVariantresolves the V1/V2 shape from the DiT's own transformer config; and a request'spromptis tokenized, encoded, projected, run through the connector and handed to cross-attention, per request.has_encoder()is true.ltx2-gengrows--encoder,--encoder-configand--prompt, so the capability is reachable through the ABI rather than only from a test.One genuine gap, refused rather than defaulted
The only shipped LTX-2.5 text encoder (
vonkaiser'sgemma4-12b-with-proj-nvfp4-torchao.safetensors) carries no__metadata__at all, so the Gemma config is an INPUT. It comes from__metadata__["gemma_config"]when the checkpoint has one and from the newencoder_config_pathextra when it does not; neither source is a refusal, both sources is a refusal.layer_types,global_head_dim,num_global_key_value_headsandattention_k_eq_veach resolve a different tower out of a byte-identical tensor set — the same polaritydit_config_pathalready has.Two things recorded at the code
Ltx2TextEncoderConditioningandLtx2ConnectorCreateEmbeddingsare two ports of overlapping halves ofembeddings_processor.py:70-117and both carry the right-pad sort, so composing them sorts an already-sorted stream. That is the identity because a stable argsort of a 0/1 key is idempotent — and "correct because idempotent" is a claim, so the engine now asserts the precondition rather than relying on it.ltx2_text_encoder.his f32 by declaration and its device arm is owed. Stated, not hidden.The instrument, and why it can see the difference
Ltx2VideoEngine::last_conditioning()reports an FNV-1a digest and an absmax over the exact f32 buffersLtx2ModalityInput::contextpointed at, taken after the connector and immediately before the denoise loop.A frame statistic was deliberately not used. L9c's reviewer found that a scene and a colour field were indistinguishable to the existing frame analyzer (neighbour |dx|/sd 0.093 vs 0.033, block-mean ratios nearly identical) and needed contact sheets to tell them apart. A digest is a function of the bytes rather than a summary of them, so it has no such blind spot: any change to any element changes it, and no plausible-looking wrong tensor satisfies it by accident. The absmax is the companion it needs — two prompts whose conditioning both collapsed to zeros would have equal digests and RED the dependence check for the wrong reason.
The frame-byte comparison is the second half: the digest says the conditioning depends on the prompt, the frames say the conditioning reaches the render.
RED before GREEN
Against the pre-L13 engine the new cases do not compile —
kLtx2EncoderConfigPathExtra,Ltx2ConditioningTraceandlast_conditioningdo not exist. That is the honest statement that the surface could not be expressed, not a substitute for behavioural evidence, which the mutations below supply.Each mutation applied alone, tree restored byte-identically afterwards (sha256 verified):
gen.promptRunConnectorM3 is worth reading. An earlier version of that subcase asserted only that the message named
encoder_config_path, and M3 left it GREEN — the load still failed, but with"encoder_config_path: cannot open "from a fallthrough reading the empty path. A refusal firing for the wrong reason is precisely what this phase exists to stop shipping, so the assertion was strengthened to the discriminating clauses before being called a gate.Not gated, and named rather than claimed unreachable: the additive-mask right-pad guard. I did not construct a probe that reaches it, and per this campaign's own rule a mutation that moves nothing is not evidence of unreachability.
The fixture
A reduced-dimension text encoder written in the shipped format: bf16 tower with the mixed sliding/full geometry and no
v_projon the full layer, torchao-NVFP4 caption projections with a padded swizzled scale (the loader-test's copyREQUIREsrows % 128 == 0, which a 16-wide projection never satisfies), tokenizer + HF sidecars stored as tensors, and no__metadata__.The fixture's DiT config also gains the three V2 marker keys it was missing. It carried only
caption_proj_before_connector— which is the PARTIAL set upstream refuses as config drift — so it gated no variant selection at all and would have refused the moment the engine asked for one. The three added values are read from the first-partyLightricks/LTX-2.5NVFP4 DiT's own__metadata__["config"]["transformer"].Gate
CPU Release build, x86_64, this worktree.
BUILD_EXIT=0; noNo space left/BFD assertionin the build log; 61G free on/.ctest -N→ 408;ctest -j8→ 408/408 passed, 0 failed (1 skipped:test_voxtral_e2e).test_ltx2_videotest_ltx2_loadertest_ltx2_text_encodertest_ltx2_pipelinetest_ltx2test_video_enginetest_capiAll
Status: SUCCESS!.scripts/agent-preflight.sh --staged:doc-checkpoint --stagedok,now-current --stagedok. Itsdoc-checkpoint rangefailure is inherited — all three commits it names (b0aa475a3,d67f8125e,aa6aa0ecd) are ancestors of the base SHA67a7b1cd9, and the same failure reproduces on the untouched base.What the spec needs (operator-owned, not edited here)
has_encoder()is still false, and this is the honest part" is now superseded and should say so rather than be deleted.Review round 2 — findings S1, S2, S4-S7 on
43aa5837(head nowd2cbe605)The reviewer's verdict on the code was that it is the strongest of this campaign and that the docs cell over-reads it. Nothing below redesigns anything.
S1 (BLOCKING) — CLOSED, option (b)
docs/FEATURES.mdread "e2e at 320x192/25f from a TYPED PROMPT via Gemma-4: coherent scene, valid MP4+WAV". No recorded run supports that sentence, and it is withdrawn rather than softened.--prompt-valid-rows 24over synthetic N(0, 0.2)learnable_registersOption (b) taken: state what actually happened. The arithmetic is the reason, not squeamishness. L9c's arm bottomed out at 68.2 GiB MemAvailable on a 119 GiB box that reboots rather than OOM-kills; the tower is a further ~24 GB of host bf16 (
ltx2_video.cpp:846); and at the time of writing dgx.casa's$HOME/gpu.lockwas held by another session'sctestwith root at 99% (62 GiB free) against a ~21 GiB build tree. Risking the box for one number, with other agents live, is not a trade this finding asks for.The precise sentences now shipped:
docs/FEATURES.md(correctness cell, 220 chars, at the limit):`SPIKE`. DiT, VAEs, conditioning, pipeline, quant loaders gated vs upstream at reduced dims. Typed prompt -> Gemma-4 -> cross-attn, FIXTURE-gated. The 320x192/25f scene was register-conditioned; a prompted render is OWEDdocs/STATUS.md:A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING`S2 — CLOSED
docs/STATUS.mdno longer says "the Gemma-4 tower is owed, so no prompt encodes yet". The new cell is 216 chars, under the 220 thatoversized_cellscounts, so the shrink-only ratchet (44) is untouched.docs/BENCHMARKS.md's LTX-2.5 axes row carries the same correction.S4 — RECORDED (not closed), with the closure specified and its prerequisite verified
The finding is accepted exactly as written:
last_conditioning()is a witness, not a gate. R6 (video x1.5) and R7 (rows reversed) each passed all 485 assertions, exit 0. A digest detects CHANGE; nothing at this level pins VALUES. The per-brick oracles are real and untouched; the two joins —Ltx2ConnectorCreateEmbeddingsand theGeneratecomposition — have none, and both mutations live in that gap.Recorded at
Ltx2ConditioningTrace, atLtx2ConnectorCreateEmbeddings, and in the append-only record. The reviewer's path is named as the closure: upstream's counterpart isEmbeddingsProcessor.process_hidden_states(embeddings_processor.py:97-117), in the same packagegen-ltx2-pipeline-goldens.pyalready executes at section 10.Prerequisite verified so the next implementer does not have to: that generator reproduces its committed output byte-for-byte on the pinned upstream —
md5 53e2a6aba8885d7d58302ad0b7b09eb4for both the regenerated file andtests/vllm/models/ltx2_pipeline_goldens.inc, withLTX-2clean atfd4ded7f...andvllm-omniata4ea67a2..., the SHA the C++ suite pins. A section can be added without disturbing anything already gated.S5 — CLOSED, both halves fixed and gated in both directions
last_conditioning()now returns by value under the mutexGenerateholds. A reference could not be made safe: the lock is released before the caller reads. And the trace is filled before the denoise loop, so aGeneratethat throws later left a healthy-looking trace for a render that never completed — newcompletedflag, set at the single successful return.Gated on a real refusal, not an injected one: keyframe/reference conditioning is refused after the trace is written, so a prompted request carrying a reference image fills the trace and then fails.
completed = trueTree restored byte-identically after each (
sha256 22342f84...c546before and after).S6 — CLOSED, both halves
The four V2 markers are correct; nothing in the repo recorded the header. Now recorded at the fixture and in the record, with path, byte size and revision:
/mnt/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors, 18,721,432,024 bytes,Lightricks/LTX-2.5rev8a4ff96f...(HF download record; the LFS oid is the upstream sha256 and was not re-verified locally). All four present, none drifted.text_encoder_norm_type: PER_TOKEN_RMScorroborates independently, and the selector deliberately does not read it.Cause corrected. "It gated no variant selection at all" is wrong —
Ltx2SelectTextFeatureVariantdoes refuse a partial set (ltx2_text_encoder.cpp:184-192). It never fired because no production path called the selector before L13; the keys and the first caller landed in the same commit.S7 — CLOSED
The right-pad precondition guard now carries the same sentence the mask-value guard had. No reachability is claimed — no probe was built that fails to reach it, and a mutation that moves nothing is not evidence of unreachability. "Ungated, no reachability claim" is the statement.
One self-correction, named because it is this campaign's own failure mode
I first replaced
+~24 GB text towerwith an on-disk figure, believing it ungrounded. It is grounded — 12B params at bf16, stated atltx2_video.cpp:846and indocs/USAGE.md— and the resident figure is the useful one for a memory cell. Reverted before commit.Gate at
d2cbe605CPU Release, clean configure,
BUILD_EXIT=0, build logs clean ofNo space left/BFD assertion,df53-67G free throughout.ctest -N: 414 registeredctest -j4: 414/414 passed, 0 failed, exit 0 (test_voxtral_e2eskipped)test_ltx2_videotest_ltx2_loadertest_ltx2_text_encodertest_ltx2_pipelinetest_capiExit codes read separately from the doctest summary. Every unchanged suite is byte-for-byte the same count; the only movement is the one new case.
scripts/agent-preflight.sh --stagedpassesdoc-checkpoint --staged,now-current --stagedandcheck-public-doc-tables. Itsdoc-checkpoint rangefailure is inherited and re-verified:b0aa475a3,d67f8125eandaa6aa0ecdare eachgit merge-base --is-ancestorof the base SHA67a7b1cd9.Re-review round: S8 — CLOSED at
56a917c69The re-review passed everything else and returned FAIL on one blocking finding.
docs/USAGE.mdstill described the pre-L13 world in two blocks, and this PR is what made them false.What was false, and why it blocked
:487-489im.has_encoder = trueon theencoder_pathbranch,ltx2_video.cpp:893:499-510encoder_pathis still refused ...has_encoder()is still false. Conditioning meanwhile comes fromprompt_embeds_path"ltx2_loader.cpp:416-427The file also contradicted itself:
:372, added by this PR, opens "A typed prompt works." and 115 lines later the same document said the tower was unported.Provenance, so a later reader can re-check the cause rather than trust the repair:
:487arrived ate48c86253(L9c) and:499atab6671394(L10), both true when written;43aa58377in this PR falsified them and left them standing.The repair, and what it deliberately did NOT delete
"It renders a scene, and it does not render YOUR scene" is re-attributed, not dropped. It is still true of the L9c render and of the embeds path, so it now names the run it describes —
--prompt-embedswith--prompt-valid-rows 24over synthetic N(0, 0.2) rows, no tower on the path, 104 of 128 connector rows the connector's own trainedlearnable_registers.What it had to stop being is a property of the
--encoder+--promptcommand L13 rewrote the example into. That framing is the S1 defect in the other direction: no prompted render has been run in either direction, so the page now claims nothing about what that command renders — not that it puts a fox on the screen, and not that it fails to. A real-checkpoint prompted render stays OWED.A third item was added to the memory paragraph, because it was measured without the tower:
--encoderadds ~24 GB of resident host bf16 on top of the ~44 GB DiT staging, and the figures quoted there came from an embeds-path run.Two accuracy nits — both FIXED
1. The assertion count. The
Ltx2ConditioningTraceheader said both composition mutations "passed all 485 assertions ... on this exact head". 485 was43aa58377's count; this head is 499 — this PR's own gate table above records the 485 -> 499 move. A comment whose whole purpose is precision named a number no run of it could produce.Re-measured here rather than transcribed, because a reviewer report is an input and never a gate result:
56a917c69Each applied alone from a pristine copy with an asserted-unique anchor; each leg's build log verified to contain
Building CXX object ... ltx2_video.cpp.o+Linking, so no leg ran a stale binary; the tree restored byte-for-byte (diffclean) and re-verified green between legs. The verdict did not move — the point is that the count had drifted, not the conclusion.2.
last_conditioning()blocking. It was documented as safe to call from a server thread while another renders. True as to safety, butGenerateholds that same mutex for the whole render (ltx2_video.cpp:980, function-scopelock_guard), so the caller blocks for minutes rather than getting a stale-but-immediate answer. Stated.One sibling, found by sweeping instead of fixing the instance in front of me
This is what #604 says has failed four rounds running. The AUTO-duration comment claimed the duration head "needs the encoded prompt this engine cannot produce" — a fifth refusal-reason that went stale while its refusal stayed correct, since L13 produces exactly that. The true reason is now recorded: no duration head is ever constructed here, and
duration_head_pathis accepted inkKnownLoadExtraswhile no code reads it (it appears at that one site;upstream_path, by contrast, is read at:771). So the extra is inert rather than wired. Also named indocs/USAGE.md, where a reader would otherwise supply it and silently get the recipe default.NOT paid, and named rather than left to be found
.agents/benchmark-record.md's L13 section carries the same superseded 485/485 pair. Correcting it is not free:check-doc-checkpointclasses any benchmark-record edit as a measurement (MEASUREMENT_RECORDS-> thelifecycleclass) and demandsdocs/STATUS.md+docs/BENCHMARKS.md. Those are the surfaces the re-review passed and told this repair not to disturb, with STATUS already at itsoversized_cellsratchet of 44. Manufacturing an edit to a blessed surface to turn a gate green is the thing this protocol forbids, so it is reported instead. One-line follow-up: swap both485/485 assertionscells for30 cases / 499 assertionsalongside the next legitimate STATUS/BENCHMARKS write.Proof the false claims are gone
The only surviving
485is the sentence that records the drift and names the head it belonged to.Gate at
56a917c69CPU Release, no CUDA.
BUILD_EXIT=0, 0 warnings, build logs clean ofNo space left/BFD assertion, disk 87% used / 59G free.ctest -N: 414 registeredctest -j8: 413 passed / 1 skipped (test_voxtral_e2e); the single failuretest_engine_core_procis a known-jflake and passes serially (1/1 Test #195 ... Passed)test_ltx2_video: 30 cases / 499 assertions, exit 0scripts/agent-preflight.sh --staged:doc-checkpoint --stagedok,now-current --stagedokThe
doc-checkpoint rangefailure remains inherited and untouched — it names onlyb0aa475a3,d67f8125e,aa6aa0ecd, all pre-existing branch commits.Refs #435, #604.
🤖 Generated with Claude Code
Re-review round: S9 — CLOSED at
4e5d4a2eaThe re-review confirmed the code is sound and that the previous repair's own two blocks are correct. It returned one blocking finding, and it is about the proof, not the patch:
56a917c69proved its completeness with a grep built from the text it had just written.It searched for
still not ported,`encoder_path` is still refusedand`has_encoder()` is still false. All three returned nothing — because all three were its own phrasings. Searching the vocabulary of the claim rather than the sentence found six more instances of the same false statement, in threedocs/USAGE.mdsections that were never swept.At
56a917c69the file asserted both "A typed prompt works." (:372, added by this PR) and "there is no text encoder" (:1817). Both cannot be true.The six, each against the code that disproves it
:1816-1821/v1/videosroute for it — asking the video engine for LTX-2.5 will not work"ltx2_video.cpp:1529registers the family,:893setshas_encoder = true,ltx2_video_vae.cpp+ltx2_audio_vae.cppship,ltx2_pipeline.cppships,/v1/videosregisters through the family-agnosticLoadVideoEngine, andltx2-genrenders — documented at:436-450of this same file:1920:1924-1925ltx2_video.cpp:576-577(--dit/--video-dit) and:851(--encoder/--video-encoder):1963:1965:1927-1932*_embeddings_connectortowers", "which still reports every one of them inLtx2DitCheckpoint::unported"UnportedFamiliesfilters the connectors atltx2_loader.cpp:439(LoadedElsewhere), andRefuseUnported's own message says so in capitals at:461-464— the checker's message is the authority#6 is the decisive one. It is the identical claim this PR's previous repair retired at
:515-530, left standing 400 lines further down — which is precisely what a self-confirming grep cannot find.Provenance is written into the page itself, not only into this body, so a later reader can re-check the cause instead of trusting the repair: block 1 arrived at
3d89f6fc4(the first LTX commit, true when written) and was never revisited as L3→L13 built each of the six pieces it denied; the "five" arrived at5966ffef3and stopped being true ate48c86253.The completeness proof, built from the claim and not from the fix
Patterns searched over
docs/,README.md,examples/and both--helpsurfaces a user sees:no render path,no text encoder,no VAE,no pipeline,will not work,one brick,not a command,unported,refuses,embeddings_connector,not ported,is refused,still no,yet,library entry point,layout and forward,asking the video engine189 hits read. Everything else was checked and left alone because it is accurate:
STATUS.md:119,FEATURES.md:156,BENCHMARKS.md:426,ENVIRONMENT.md:164,USAGE.md:349-540and:1432-1458, andltx2-gen --help, which already describes both conditioning paths correctly.USAGE.md:551("Still no reference decode") is Muse Glimmer, not LTX. The server's--helpnames no video flags at all..agents/carries none of these phrasings.docs/USAGE.mdis now internally consistent with:372end to end.One finding beyond the six — REPORTED, not fixed
This branch bumped the ABI and left three public claims at the old numbers.
3db9233ecmovedVLLM_ABI_VERSION17 → 18 and addedvllm_video_engine_family(36VLLM_APIdeclarations, was 35).docs/USAGE.md:1464-1465,README.md:189andREADME.md:388all still say 17 and 35.origin/mainis self-consistent at 17/35, so this PR is what falsified them — anddocs/STATUS.md:119already says v18, so the tree disagrees with itself today.It is reported rather than half-fixed. The USAGE half alone passes
check-doc-checkpoint, butcheck-readme-structurerefuses the README half:and no landing-source edit is legitimately owed here —
examples/server/main.cpp:15's "published asvllm_server_mainat ABI v17" is a statement about when that symbol shipped, verified correct atc1716fd0c. Manufacturing an edit to a blessed surface to turn a gate green is what this protocol forbids, and correcting one of two surfaces that carry the same number leaves a reader unable to tell which is right. The owed change is three lines — 17 → 18 and 35 → 36 at those three sites, landed together.Gate at
4e5d4a2eaCPU Release, x86_64, no CUDA, clean configure into a fresh build dir (deleted afterwards).
CONFIGURE_EXIT=0,BUILD_EXIT=0, 0 compiler warnings; build log clean ofNo space left/BFD assertion; disk 79G → 64G free during, 71G after cleanup.ctest -N: 414 registeredctest -j8: 413 passed / 1 skipped (test_voxtral_e2e). The single failuretest_engine_core_procis the known-jflake and passes serially:1/1 Test #195: test_engine_core_proc … Passed.test_ltx2_video: 30 cases / 499 assertions, exit 0 — identical to this head's recorded counts, as a documentation-only change must be. Exit code read separately from the doctest summary.scripts/agent-preflight.sh --staged:doc-checkpoint --stagedok,now-current --stagedok;check-public-doc-tablesok.The
doc-checkpoint rangefailure remains inherited and was re-verified withgit merge-base --is-ancestor:b0aa475a3,d67f8125eandaa6aa0ecdare each an ancestor of the base SHA67a7b1cd9.This is instance sixteen in the campaign #604 tracks, and the second in this same file and PR.
Refs #435, #604.