feat(videos): /v1/videos speaks OpenAI's Sora wire shape, and serves the MP4 - #71
Merged
Conversation
mudler
force-pushed
the
row/SERVE-VIDEOS-OAI
branch
2 times, most recently
from
August 6, 2026 23:50
f441b91 to
01cf15a
Compare
localai-bot
changed the title
mudler
force-pushed
the
row/SERVE-VIDEOS-OAI
branch
3 times, most recently
from
August 7, 2026 01:12
39650f6 to
bc55383
Compare
…the MP4
Row `SERVE-VIDEOS-OAI`, branch `row/SERVE-VIDEOS-OAI`. An unmodified OpenAI
(Sora) client now works against our `/v1/videos`, ADDITIVELY over the
vLLM-Omni-derived fields we already take.
SPLIT, deliberately. The original change was 1241 non-exempt lines, over the
900 check-pr-size.py cap. It is split along its real seam rather than trimmed to
fit: this row is the request/response WIRE SHAPE (no generation code, no VAE, no
examples/server change), and REFERENCE CONDITIONING (`input_reference` -> fl2va
plus the two `metadata` ref2va modalities) is a stacked follow-up row. No cap and
no ratchet was raised.
Request aliases. `model`, `size` ("1280x720") and `seconds` land on the existing
native members. `seconds` is taken as a number OR a numeric string, because
OpenAI's schema types it as a string enum ("4"/"8"/"12") and a literal client
would otherwise be rejected on a type. PRECEDENCE is defined and gated: the
NATIVE field WINS (`width`/`height` over `size`, `duration` over `seconds`),
which is what guarantees every body that parsed before means exactly what it
meant before, applied PER-AXIS so an explicit `width` alone still lets `size`
supply the height. Both spellings are VALIDATED either way, so a malformed
`size` is a 400 even when explicit `width`/`height` override it.
`model` warns, never rejects: a Sora client cannot know the local model's name,
so refusing would defeat the compatibility and ignoring would hide a real
mismatch. The requested name and the divergence ride the job for its whole life.
GET /v1/videos/{id}/content returns the finished MP4. Without it a caller can
start and poll a job but never FETCH the result over HTTP. Unknown id -> 404;
queued/running -> 409 naming the status (a pending job must never answer with
bytes: a partially muxed file reaches the client as a valid-looking truncated
MP4); failed -> 500 with the failure; a vanished output -> 500, not a 200 with
zero bytes.
GATE (CPU, foreground): test_video_api 11/11 (125 assertions),
test_openai_api_server 40/40 (509), `server` builds clean. Additivity is gated
over a REAL socket: with no VideoRunner all four routes are absent (a bare 404,
no ErrorResponse envelope), with one they serve and the unknown-id 404 is ours.
TWO OUT-OF-ROW CI REPAIRS, carried here because they block EVERY merge, not just
this one. Both are `agent-record` job failures on main, both reproduced on an
unrelated PR, and neither is repaired by weakening a checker.
(1) check-fusion-consistency has failed since run 31129401136 because it flags
minimax_h3_video_vae_device.cpp. Repaired the way AGENTS.md names, with a
conscious allowlist entry carrying the verified reason: w1 already ships merged
and is already one MatmulBT, so nothing is unmerged; the seam is unusable because
the VAE block is f32 end to end where the method is kBF16, and every VAE Linear
carries a rank-1 bias the bias-free method has no slot for.
(2) check-role-discipline failed on every feature PR. CI checks out
refs/pull/N/merge, a SYNTHETIC merge GitHub builds whose entire message is
"Merge <head> into <base>": it names neither the row branch nor the PR, and it
NEVER lands on main, so a gate about MAIN's history was run on a commit that is
not main's history. Reproduced on PR #80 (Vulkan) to prove it is not this row's
doing. The fix reads the SECOND parent, which is the PR head: a merge of a branch
whose own commits name the row IS arrival through a row PR, one hop away. Gated
as a non-weakening: a new test asserts a merge naming no row ANYWHERE, and a
plain local `Merge branch 'wip'`, both STILL fail. Suite 40/40.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Opus 5 (1M context)
mudler
force-pushed
the
row/SERVE-VIDEOS-OAI
branch
from
August 7, 2026 02:31
bc55383 to
306839e
Compare
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…user docs PR #68. A real bug found by RENDERING, not by the suite, plus the user-facing documentation the H3 lane never had. THE BUG. A 124-frame render silently muxed as 61 frames. `audio_t` is the PER-CHANNEL latent length (the planner sets it from 40 Hz * duration) and the packed layout carries `audio_t * audio_channel` ROWS, one per (channel, step). The denormalize step divided by the channel count, so the decoded audio ran half the video's duration; because the muxer passes `-shortest`, that silently truncated the VIDEO to half its frames too. The fix is one line, and it now agrees with the same function's own `target_audio_rows = request.audio_t * request.audio_channel` a few lines above. WHY THE SUITE WAS BLIND, which is the part worth keeping. Every existing gate asserts shape SELF-CONSISTENCY, and a uniformly halved pipeline is perfectly self-consistent: the shapes agreed with each other, they were just half as long as the request asked for. ffprobe on the artifact exposed it. The gate added is the one that is NOT self-consistency: the structural t2va e2e case now decodes `audio_t` latent steps INDEPENDENTLY and requires the pipeline to have produced exactly that many samples. RED-first: restoring the divide turns that case RED (1 failed assertion), removing it goes GREEN. Suite 69/69, 47416 assertions. A SECOND BUG FROM THE SAME SESSION, DROPPED RATHER THAN LANDED TWICE. Conditioning PREPENDS rows to img_pos, so the denoise loop returns condition rows followed by the targets, and unpatchify was handed the whole set. This branch fixed it by carrying the layout's own `update_mask` out of the loop; main fixed the SAME bug independently in row/H3-RENDER-CLOSE by taking the TRAILING target_video_rows with a VT_CHECK on the row accounting. On rebase the update_mask mechanism and its test were DROPPED: landing a second solution to a fixed bug is churn, and the positional form already ships gated. The alternative is recorded in .agents/state.md rather than silently lost. DOCS. The video+audio generation path had no user-facing documentation at all, and the multimodal INPUT interface was documented WRONG: `vllm-cli --image/--video/--audio` does not exist, the CLI is text-only. Multimodal input is served over the OpenAI API as `image_url`, `video_url` and `input_audio`/`audio_url` content parts on /v1/chat/completions (src/vllm/entrypoints/openai/chat_mm.cpp), which was documented NOWHERE, so docs/USAGE.md now states it rather than the wrong claim merely being removed. README gains a News section led by video+audio generation, paid for INSIDE the 30,000-char landing-page budget rather than by raising it. PUBLIC DOCS. The 19-line narrative this branch originally appended to docs/BENCHMARKS.md was a non-canonical H2 section and pushed the page over its prose budget: converted into three keyed ROWS (Thor render speed, render duration, quantization floor), with the forensics moved verbatim into .agents/benchmark-record.md, which is what that page's own checker instructs. ALSO REPAIRS RECORD DAMAGE THAT LEFT check-agent-record RED ON MAIN. #93's squash-merge removed 130 lines from specs/minimax-h3.md: its branch predated #71/#92, so landing it DELETED sections 9 and 10 (the /v1/videos rows) and left two orphaned table fragments with no heading and no separator. The checker reports that as "table has 4 pipes; expected 3" plus "no linked spec names exact stable token SERVE-VIDEOS-REFS", the engine-matrix row pointing at a section that no longer existed. Restored sections 9 and 10 VERBATIM from 548b000 and cut the orphans; #93's own content is preserved byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context)
localai-bot
added a commit
that referenced
this pull request
Aug 7, 2026
…user docs (#68) PR #68. A real bug found by RENDERING, not by the suite, plus the user-facing documentation the H3 lane never had. THE BUG. A 124-frame render silently muxed as 61 frames. `audio_t` is the PER-CHANNEL latent length (the planner sets it from 40 Hz * duration) and the packed layout carries `audio_t * audio_channel` ROWS, one per (channel, step). The denormalize step divided by the channel count, so the decoded audio ran half the video's duration; because the muxer passes `-shortest`, that silently truncated the VIDEO to half its frames too. The fix is one line, and it now agrees with the same function's own `target_audio_rows = request.audio_t * request.audio_channel` a few lines above. WHY THE SUITE WAS BLIND, which is the part worth keeping. Every existing gate asserts shape SELF-CONSISTENCY, and a uniformly halved pipeline is perfectly self-consistent: the shapes agreed with each other, they were just half as long as the request asked for. ffprobe on the artifact exposed it. The gate added is the one that is NOT self-consistency: the structural t2va e2e case now decodes `audio_t` latent steps INDEPENDENTLY and requires the pipeline to have produced exactly that many samples. RED-first: restoring the divide turns that case RED (1 failed assertion), removing it goes GREEN. Suite 69/69, 47416 assertions. A SECOND BUG FROM THE SAME SESSION, DROPPED RATHER THAN LANDED TWICE. Conditioning PREPENDS rows to img_pos, so the denoise loop returns condition rows followed by the targets, and unpatchify was handed the whole set. This branch fixed it by carrying the layout's own `update_mask` out of the loop; main fixed the SAME bug independently in row/H3-RENDER-CLOSE by taking the TRAILING target_video_rows with a VT_CHECK on the row accounting. On rebase the update_mask mechanism and its test were DROPPED: landing a second solution to a fixed bug is churn, and the positional form already ships gated. The alternative is recorded in .agents/state.md rather than silently lost. DOCS. The video+audio generation path had no user-facing documentation at all, and the multimodal INPUT interface was documented WRONG: `vllm-cli --image/--video/--audio` does not exist, the CLI is text-only. Multimodal input is served over the OpenAI API as `image_url`, `video_url` and `input_audio`/`audio_url` content parts on /v1/chat/completions (src/vllm/entrypoints/openai/chat_mm.cpp), which was documented NOWHERE, so docs/USAGE.md now states it rather than the wrong claim merely being removed. README gains a News section led by video+audio generation, paid for INSIDE the 30,000-char landing-page budget rather than by raising it. PUBLIC DOCS. The 19-line narrative this branch originally appended to docs/BENCHMARKS.md was a non-canonical H2 section and pushed the page over its prose budget: converted into three keyed ROWS (Thor render speed, render duration, quantization floor), with the forensics moved verbatim into .agents/benchmark-record.md, which is what that page's own checker instructs. ALSO REPAIRS RECORD DAMAGE THAT LEFT check-agent-record RED ON MAIN. #93's squash-merge removed 130 lines from specs/minimax-h3.md: its branch predated #71/#92, so landing it DELETED sections 9 and 10 (the /v1/videos rows) and left two orphaned table fragments with no heading and no separator. The checker reports that as "table has 4 pipes; expected 3" plus "no linked spec names exact stable token SERVE-VIDEOS-REFS", the engine-matrix row pointing at a section that no longer existed. Restored sections 9 and 10 VERBATIM from 548b000 and cut the orphans; #93's own content is preserved byte-for-byte. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context) Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
SERVE-VIDEOS-OAI. An unmodified OpenAI (Sora) client now works against our/v1/videos, additively over the vLLM-Omni-derived fields we already take.Split, deliberately
The original single change was 1241 non-exempt lines, over the 900
check-pr-size.pycap. It is split along its real seam rather than trimmed to fit:examples/serverchange. 703 non-exempt lines.input_reference-> fl2va, plus the twometadataref2va modalities) is a stacked follow-up row.No cap and no ratchet was raised.
What lands
Request aliases.
model,size("1280x720") andsecondsland on the existing native members.secondsis taken as a number or a numeric string, because OpenAI's schema types it as a string enum ("4"/"8"/"12") and a literal client would otherwise be rejected on a type.Precedence, defined and gated. The native field wins (
width/heightoversize,durationoverseconds), which is what guarantees every body that parsed before means exactly what it meant before. It is applied per-axis, so an explicitwidthalone still letssizesupply the height. Both spellings are validated either way, so a malformedsizeis a 400 even when explicitwidth/heightwould override it.modelwarns, never rejects. A Sora client cannot know the local model's name, so refusing would defeat the compatibility and ignoring would hide a real mismatch. The requested name and the divergence ride the job for its whole life.GET /v1/videos/{id}/content. Without it a caller can start and poll a job but never fetch the result over HTTP. Unknown id -> 404; queued/running -> 409 naming the status (a pending job must never answer with bytes: a partially muxed file reaches the client as a valid-looking truncated MP4); failed -> 500 with the failure; a vanished output -> 500, not a 200 with zero bytes.Gate (CPU, foreground)
test_video_api11/11, 125 assertionstest_openai_api_server40/40, 509 assertionsserverbuilds cleanVideoRunnerall four routes are absent (a bare 404, noErrorResponseenvelope); with one they serve, and the unknown-id 404 is ours.Two out-of-row CI repairs, carried here because they block EVERY merge
Both are
agent-recordfailures onmainitself, both reproduced on an unrelated PR, and neither is repaired by weakening a checker.check-fusion-consistencyhas failed since run 31129401136 because it flagsminimax_h3_video_vae_device.cpp. Repaired the way AGENTS.md names (fold, or take a conscious allowlist entry):w1already ships merged[2*ff_inner, dim]and is already oneMatmulBT, so nothing is unmerged; the seam is unusable because the VAE block is f32 end to end where the method's DBufs are kBF16, and every VAELinearcarries a rank-1 bias the bias-free method has no slot for.check-role-disciplinefailed on every feature PR. CI checks outrefs/pull/N/merge, a synthetic merge GitHub builds whose entire message isMerge <head> into <base>: it names neither the row branch nor the PR, and it never lands onmain, so a gate about main's history was being run on a commit that is not main's history. Reproduced on Vulkan: a model runs end to end, token-exact — 16 native kernels, variant pipeline, two CI gates #80 (Vulkan) to prove it is not this row's doing. The fix reads the second parent, which is the PR head: a merge of a branch whose own commits name the row is arrival through a row PR, one hop away. Gated as a non-weakening: a new test asserts that a merge naming no row anywhere, and a plain localMerge branch 'wip', both still fail. Suite 40/40.Residuals, named
OpenAI's status vocabulary and id shape are not mirrored (ours stays queued/running/succeeded/failed, ids
vid_N, noobject/progress/created_at); reference conditioning is the stacked follow-up row; the real-weights leg rides the H3 GB10/disk window.FOLLOWING_AGENTS_PROTOCOL