spec(MODEL-MM-indextts2): scope IndexTTS-2.5, our first audio-generating lane (#634) - #637
Merged
Conversation
…entory that says otherwise (#633) FOLLOWING_AGENTS_PROTOCOL The pin protocol covers one repository. Every architecture living only in `vllm-project/vllm-omni` therefore has no oracle this protocol can NAME — not a hard comparison, an unavailable one. Two rows already carry it as their blocker: `minimax-h3.md:205-208` ("a vllm-omni pin is a prerequisite for W3+") and the H3 row in `model-matrix.md`, which closes with "OPEN: there is no vllm-omni parity PIN". #435 (LTX-2.5) is the third. The surface is ~40 modules wide, not three rows: the whole TTS family (IndexTTS2, Fish Speech, GLM-TTS, Ming-TTS, MOSS-TTS, Qwen3-TTS, VoxCPM2, Voxtral-TTS, Higgs Audio v2/v3, Step Audio2, CosyVoice3, OmniVoice) plus Qwen2.5/3-Omni, Aura-Omni, Ming-Flash-Omni and Hunyuan Image3/Video. So this is also a RECORDS defect: `model-matrix.md` claims an EXHAUSTIVE architecture inventory of 329 rows, and that claim is scoped to one repository without saying so. Design, in the spec: a SECOND pin block rather than a second value of the first. vllm-omni requires vLLM 0.27.0+ against our 0.26.0.dev0 core pin, so the two are allowed to disagree, and `vllm_core_commit_used` records which core commit the omni oracle actually ran against. Two rules follow — an omni-gated number is labeled with BOTH commits, and is never evidence about the core pin's surface. Gateability stays per ARCHITECTURE: the oracle must build, run and EMIT OUTPUT for that model, because `assert_oracle_commit` exists precisely because an oracle once resolved and was a rollback. The spec answers the question it would otherwise leave to be discovered: an omni pin advance does NOT re-open the vLLM-side binding grids, conditional on isolation (own virtualenv, no core-pinned row citing an omni number, no mutation of ${VLLM_SOURCE} or the environment the core pin measures itself from). Fail any condition and the denominator's own dependency tree moved, so it is re-validated as a core sync cycle. W1-W2 (the `upstream-sync.md` block schema, concept and rules) land here. W3-W5 are owed and unclaimed: the checker with its five RED-first assertions, the measured pin values, and a per-architecture gateability demonstration. The pin values are deliberately NOT invented here — they need an eligible host, and an unavailable value stays PENDING rather than becoming an assumption. `ENGINE_ROWS` 152 -> 153 for a genuinely new protocol row, never to make a failing transition pass. Gates: `agent-preflight.sh` and `--staged` both green. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Two keyed-record conflicts, both resolved by taking the target branch version WHOLESALE and reapplying the scoped edit, per AGENTS.md §Records. `main` had independently bumped `ENGINE_ROWS` 152 -> 153 for `SERVE-RECIPE-ARGS` while this branch bumped the same constant 152 -> 153 for `ENG-UPSTREAM-OMNI-PIN`. Both rows are real and neither replaces the other, so the merged value is 154 — taking either side's 153 would have silently dropped one row while leaving the matrix looking consistent. The `Serving, API, CLI, library` rollup is merged the same way: 32 -> 34 rows, main's +1 ACTIVE and this branch's +1 READY both kept. Unrelated keys verified byte-for-byte: the whole engine-matrix diff against origin/main is this row plus those two rollup cells, and the roadmap issue table carries no duplicate issue key after its automerge. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…top overclaiming the inventory gap (#633) FOLLOWING_AGENTS_PROTOCOL Two corrections found while writing the IndexTTS-2.5 row against this spec. The spec cited `a4ea67a2` as the one source audit of vllm-omni. `main` has since landed a second, newer one: #609/#610 anchor their omni rows to `bbe6ccc512a404a2df8c977ea29003002f2683e8`. Both are now cited, with the point that neither is a pin — a source read establishes what EXISTS, never what RUNS, which is the whole distinction this row turns on. The spec also said the omni surface is "uninventoried and uncounted". That was true when written and is no longer: those same PRs rowed several omni architectures explicitly as out-of-repo. The residual defect is narrower and the text now says the narrower thing — the document's scope sentence still does not state the boundary those rows work around, and each of them repeats the same missing-pin blocker in its evidence cell. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ing lane (#634) FOLLOWING_AGENTS_PROTOCOL vLLM-Omni supports IndexTTS-2.5 (`vllm-omni serve IndexTeam/IndexTTS-2.5 --omni`) and we have no record of it at all. Every audio path we ship CONSUMES audio (Parakeet, Voxtral, `audio_processor.cpp`); nothing synthesizes it, and `/v1/audio/speech` does not exist. This scopes the port; it implements nothing. Upstream registers TWO architectures, so this rows both rather than collapsing them: `IndexTTS2TalkerForConditionalGeneration` (`indextts2/indextts2_talker.py`) and `IndexTTS2S2MelDecoder` (`indextts2/indextts2_s2mel_decoder.py`), at `registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8` — the same anchor the Moss-TTS rows use. Stage 0 is a ~0.8B GPT-2 AR talker; stage 1 is EnhancedCodec (2.5 replaces RepCodec, `use_gpt_latent=false`) -> S2Mel CFM/DiT -> BigVGAN at 22.05 kHz, ~6 GB VRAM. The reuse is real and is why this is worth scoping now: BigVGAN is ALREADY ported (`minimax_h3_audio_vae.cpp`, gated 4.2e-9 vs the checkpoint's own remote code), as are WAV serialization, the flow-matching denoise loop, a Conformer encoder (w2v-bert-2.0 is a Conformer) and the `/v1/videos` runner seam. `opt.cpp` is the GPT-2 family, so the talker is an additive delta rather than a fresh transformer. Three things the spec pins down rather than leaving to be discovered: - REFERENCE AUDIO IS MANDATORY. Upstream states IndexTTS-2 has no text-only synthesis, so w2v-bert-2.0 + MaskGCT + CAMPPlus are required port surface, not an optional voice-cloning extra. - THE GATE CANNOT BE TOKEN-EXACT END TO END. A seed controls both AR sampling and per-request CFM noise, and upstream does not guarantee a bit-identical waveform across differing concurrent batch composition. So: binding per-stage numerics vs the checkpoint's own remote code plus token-exact mel codes at fixed seed / c1 / fixed batch composition, with an ASR-round-trip + speaker-similarity ratchet that bounds max relative error from BOTH sides. Explicitly not a correlation gate (scale-invariant) and not a count-based tolerance. - WE MIRROR vLLM-OMNI, WHICH ITSELF DEVIATES from IndexTeam: upstream's stage 0 uses plain vLLM sampling, not the official `num_beams=3`. Recorded, not silently inherited. Also recorded: the two upstream doc surfaces disagree on language support, so nothing reaches `docs/FEATURES.md` until `indextts2_5.yaml` settles it, and the checkpoint is under a custom bilibili-model-license. Both rows land `INVENTORIED`, not `SPIKE`: they are unclaimed and hard-blocked on #633 (no vllm-omni pin, and vllm-omni wants vLLM 0.27.0+ against our 0.26.0.dev0). `SPIKE` would owe a `CLAIM-*` owner these rows do not have, and the checklist legend reserves a per-row mark for engaged rows — so they sit in the rollup tail with the spec linked from their Spike cell. Rollup reconciled in the same commit: INVENTORIED 321 -> 323, total 369 -> 371, checklist prose 336 -> 338 with engaged unchanged at 45 and the tail 291 -> 293. Quantization is bf16-only for v1 with every other arm refused at load naming the missing piece and recorded as owed — vLLM-Omni ships no quantized IndexTTS arm, and the model fits GB10 unquantized. Gates: `agent-preflight.sh` and `--staged` green. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL This row is STACKED on the pin row, not independent of it: the IndexTTS-2.5 spec cross-references `specs/upstream-omni-pin.md`, and #634 is hard-blocked on #633. Merging the base branch in keeps those links resolvable in this tree rather than landing a spec whose central dependency is a dangling reference. No conflict; the only incoming change is the pin spec's own correction commit. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…d-bearing (#633) FOLLOWING_AGENTS_PROTOCOL `check-pr-size.py` correctly refused the branch: a change to `scripts/check-agent-record.py` owes semantic mutation evidence in `tests/scripts/test_agent_record.py`, and the count bump had none. The new assertion names BOTH rows the constant now covers, which matters more here than in the #117 and #606 precedents it copies, because this bump COLLIDED. `main` took the constant 152 -> 153 for `SERVE-RECIPE-ARGS` while this branch took the same 152 -> 153 for `ENG-UPSTREAM-OMNI-PIN`, so both sides read 153 and the merge looked clean. Keeping either 153 would have dropped a real row while leaving the matrix internally consistent. RED-first, against exactly that bad merge: drop the omni-pin row, rewind `ENGINE_ROWS` to 153 and rewind the serving rollup, so every count agrees with every other count. The full suite then reports 53 tests with ONE failure, and it is this test — nothing else in the record can see a row that vanished into a matching number. Scratch copy restored byte-for-byte, verified by comparison. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…het evidence) FOLLOWING_AGENTS_PROTOCOL Picks up the engine-ratchet mutation evidence check-pr-size required on the base branch, so this stacked branch carries it too. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…load-bearing (#634) FOLLOWING_AGENTS_PROTOCOL The mutation evidence `check-pr-size.py` requires for the `check-agent-record.py` MODEL count change, mirroring the #609/#610 assertion it copies. IndexTTS-2.5 is registered by vLLM-Omni as TWO architectures (talker + S2Mel decoder), so it moves the pin by two rather than one. That is the specific hazard: a port described in prose as "a model" is exactly the shape that lands ONE row with a bump of TWO, and no count can tell that apart from two rows landing. Both rows are named, and both are asserted `INVENTORIED` rather than `SPIKE` — unclaimed and blocked on #633, where `SPIKE` would owe a `CLAIM-*` owner they do not have. RED-first against that exact half-landed shape: drop the S2Mel row, rewind the MODEL pin to 370, and rewind the rollup and the checklist prose to match, so every number in the record agrees with every other number. The suite reports 54 tests with ONE failure and it is this test. Scratch copy restored byte-for-byte, verified by comparison. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…t superseded FOLLOWING_AGENTS_PROTOCOL Three keyed-record conflicts, all resolved by taking main WHOLESALE and reapplying the scoped edit, per AGENTS.md §Records. The automatic merge of `model-matrix.md` was the instructive one: it produced a file with TWO `Total` rows and prose reading "plus 12 rows ... = 338 total. Those 10 are", because main had restructured the same paragraph (326 at the pin -> 355, and a separate rollup total). Discarded and rebuilt: 366 -> 368 architecture rows, rollup 370 -> 372, INVENTORIED 321 -> 323, both IndexTTS rows re-added. `docs/FEATURES.md` and the MODEL count in `check-agent-record.py` were rebuilt the same way, so main's LTX-2.5 line and its 363/370 comment history survive intact and this branch's edits sit on top. The ratchet-test docstring now reads 370 -> 372 rather than 369 -> 371, so the assertion's stated reason matches the number it guards. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ed (#634) FOLLOWING_AGENTS_PROTOCOL Resolving the `check-agent-record.py` conflict by taking main WHOLESALE also took main's `ENGINE_ROWS = 153`, discarding the 154 this branch inherits from its base `row/ENG-UPSTREAM-OMNI-PIN`. The matrix still carried 154 rows, so the gate read "154 engine rows; expected 153" and was right to. Restored verbatim from the base branch, justification comment included, so the two branches state the same number for the same reason. Exactly the failure mode "take the target version wholesale" is supposed to be paired with re-applying EVERY scoped edit, not just the ones the conflict marker pointed at. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… neither exists (#634) FOLLOWING_AGENTS_PROTOCOL The spec carried the serving surface as a one-line W6, which understated it. A capability that is not reachable through the shared surface is not done, so the seam and the ABI are scope, and the spec now says what shape they take. VERIFIED state, not assumed: `include/vllm.h` is at `VLLM_ABI_VERSION 18` and contains NO speech or TTS entry point; `ApiServer` registers chat, completions, embeddings, models, `/v1/audio/transcriptions`, `/v1/videos` and `/v1/videos/sync`; `audio/speech` and `audio/voices` have ZERO hits across `src/`, `include/` and `examples/`. Every audio route we serve consumes audio. None produces it. The precedent is the video lane, which solved this for a generative modality: an abstract `vllm::multimodal::VideoEngine` (`video_engine.h:135`), the assembly library-owned behind `vllm_video_*` ABI entries, `/v1/videos` routed through that same seam, and the example reduced to a thin `vllm.h` client. H3 and LTX-2.5 both sit behind it, which is the evidence the shape survives a second lane. So this lane mirrors it rather than inventing one: a `SpeechEngine` seam whose first implementation is IndexTTS-2.5 but which must not be its ONLY possible one (the omni TTS family is ~10 further architectures), ABI v19 with `vllm_synthesize` named symmetrically to the existing `vllm_transcribe`, the two routes through the same seam, and refusal-by-name as part of the surface. W6 splits into W6a (seam + ABI + `test_capi`) and W6b (routes + example). Neither needs the vllm-omni pin: a seam and an ABI are gateable against their own contract, so they are NOT deferred behind #633 the way the e2e and speed claims are. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL #635 landed as squash `782264c0c`, so the pin content this branch carried as its own commits arrives on main as a DIFFERENT commit. That produced an add/add conflict on `specs/upstream-omni-pin.md` plus content conflicts on `upstream-sync.md` and the roadmap issue table. Resolved by taking MAIN's version for everything the pin PR owned — the squash is the merged truth for those files, and keeping this branch's pre-squash copies would resurrect a superseded draft — then reapplying only this row's own scoped edit, the #634 issue-table entry. Verified on the merged tree rather than inferred: check-agent-record, check-model-checklist, check-oracle-pins, check-doc-checkpoint, check-public-doc-tables, check-now-current and the 54-case record suite all green, with `ENGINE_ROWS = 154` (from the merged pin row) and MODEL = 372 (this row's two architectures) both intact. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scopes #634. Stacked on
#635; retargets to
mainwhenthat merges.
Why
vLLM-Omni supports IndexTTS-2.5 and we had no record of it. Every audio path we
ship CONSUMES audio (Parakeet, Voxtral,
audio_processor.cpp); nothingsynthesizes it, and
/v1/audio/speechdoes not exist. Scoping only, no productcode.
Two architectures, so two rows
IndexTTS2TalkerForConditionalGeneration(indextts2/indextts2_talker.py) andIndexTTS2S2MelDecoder(indextts2/indextts2_s2mel_decoder.py), atregistry.py@bbe6ccc512a404a2df8c977ea29003002f2683e8. Stage 0 is a ~0.8BGPT-2 AR talker; stage 1 is EnhancedCodec (2.5 replaces RepCodec,
use_gpt_latent=false) then S2Mel CFM/DiT then BigVGAN at 22.05 kHz. ~6 GB VRAM.Two reuse claims CORRECTED after LTX-2.5 (#641) merged
The first version of this spec said BigVGAN "needs generalizing out of the
minimax_h3_*namespace, which converges with #435". Both halves were wrong:vllm::multimodal::VideoEngine), notthe vocoder.
Activation1d) is already shared between H3 and LTX-2.5, published fromminimax_h3.hand gated by both suites.ltx2_audio_vae.cpp:223-230recordswhy it was not copied: a second copy of the alias-free trim geometry "goes
wrong quietly, because each copy keeps its own green gate while the two audio
VAEs drift apart."
IndexTTS-2.5 would be the third consumer, so W1 shrank from "generalize" to
"relocate that shared core to a neutral home", with a live precedent to follow.
Two mel front-ends now exist as well, and which one w2v-bert-2.0 needs is flagged
as a W3 question rather than assumed.
Three things the spec pins down
w2v-bert-2.0 + MaskGCT + CAMPPlus are required port surface.
per-request CFM noise, and upstream does not guarantee a bit-identical waveform
across differing batch composition. Binding gate: per-stage numerics vs the
checkpoint's own remote code plus token-exact mel codes at fixed seed / c1 /
fixed batch composition, with an ASR-round-trip and speaker-similarity ratchet
bounding max relative error from BOTH sides. Not a correlation gate
(scale-invariant), not a count-based tolerance (bounds nothing).
sampling, not
num_beams=3). Recorded, not silently inherited. Also: the twoupstream doc surfaces disagree on language support, so nothing reaches
docs/FEATURES.mduntilindextts2_5.yamlsettles it, and the checkpoint isunder a custom bilibili-model-license.
INVENTORIED, not SPIKE
Both rows are unclaimed and blocked on the UNPINNED vllm-omni oracle
(
.agents/oracles/vllm-omni.md:gateable = no,evidence = #633).SPIKEwould owe a
CLAIM-*owner they do not have.Merge note for the reviewer
model-matrix.mdauto-merged into a broken file — twoTotalrows and prosereading "plus 12 rows ... = 338 total. Those 10 are", because main had
restructured the same paragraph. Discarded and rebuilt on main: 366 -> 368
architecture rows, rollup 370 -> 372, INVENTORIED 321 -> 323. A follow-up commit
also restores the
ENGINE_ROWS = 154this branch inherits from its base, whichthe wholesale resolution had wiped while the matrix still carried 154 rows — the
gate caught it and was right to.
Gates
agent-preflight.shgreen excepttest_cpu_x86_llamacpp_floor, exit 4(
NO_QUIET_WINDOW) at load 74-127 while another session builds LTX-2.5 on thesame box; it never measured anything. Record suite 54/54.