From b2bdae6115b598fb4de59c64e579b14899e0e67a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 23:58:34 +0000 Subject: [PATCH] feat(videos): reference conditioning over /v1/videos, image and clip and voice Row `SERVE-VIDEOS-REFS`, branch `row/SERVE-VIDEOS-REFS`, stacked on `row/SERVE-VIDEOS-OAI`. That row made an OpenAI client's body PARSE; this one makes its REFERENCES do something. Before it, no reference modality was reachable over HTTP at all: an image-to-video request generated silently from the prompt alone. `input_reference` -> fl2va, NOT ref2va. OpenAI documents it as the image the generated video STARTS FROM, which is exactly what fl2va expresses: MiniMaxH3EncodeKeyframeCondRows pins frame 0 OF THE OUTPUT to the supplied image. MiniMaxH3EncodeReferenceImages means something else entirely, whole reference images PREPENDED as their own blocks (subject or style guidance that never becomes a frame), so mapping input_reference there would have silently changed what the API promises. The source is a filesystem path or an RFC 2397 data: URL, decoded by the SAME DecodeDataUri the chat multimodal parts use rather than a second, subtly different decoder; an http(s) URL is refused BY NAME rather than stat-ed as a path and failing later with a confusing message. The two modalities OpenAI has no slot for ride `metadata`. H3 supports three (image, silent video, audio) and the Sora schema carries one, so the other two enter through the standard free-form string map every strict client already tolerates, rather than invented top-level fields that would fail a client's schema validation. metadata.input_reference_video is a DIRECTORY of frame_%06d.ppm (the exact layout minimax-h3-gen and this server write, so clips chain) because no demuxer is vendored; metadata.input_reference_audio is a 16-bit PCM WAV path or data: URL. A video reference is SILENT, by construction and stated: MiniMaxH3EncodeReferenceVideo emits ref_audio_t == 0. An audio reference supplied alongside ATTACHES to that same kVideoAudio block, the layout packed_sequence.py builds; alone it is its own block. Legality is enforced in the PARSER, not left to the pipeline. The rule is the pipeline's own (minimax_h3_pipeline.cpp:251: fl2va keyframes and ref2va blocks are EXCLUSIVE), so an illegal pair is a 400 naming it rather than a failed job, and never a silently dropped reference, which is the failure that looks like it worked. Two refusals up front rather than deep in the denoise: a reference image must be a binary PPM (P6), since no PNG/JPEG codec is vendored (the same NAMED residual the chat multimodal path carries), and it must already be at the resolved output geometry, since no resampler is vendored. Both VAE encoder halves load LAZILY and ONCE each under one mutex. GATE (CPU, foreground): test_video_api 14/14 (167 assertions), test_openai_api_server 41/41 (525), `server` builds clean. Each modality is gated as ARRIVING at the runner, and an illegal pair is a 400 that generates nothing. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Opus 5 (1M context) --- .agents/NOW.md | 2 +- .agents/coordination.md | 3 +- .agents/engine-matrix.md | 5 +- .agents/specs/minimax-h3.md | 83 +++++- .agents/state.md | 76 ++++++ docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- docs/USAGE.md | 61 ++++- examples/server/main.cpp | 248 +++++++++++++++++- include/vllm/entrypoints/openai/video_api.h | 47 +++- scripts/check-agent-record.py | 10 +- src/vllm/entrypoints/openai/video_api.cpp | 82 ++++++ .../entrypoints/openai/test_api_server.cpp | 59 ++++- .../entrypoints/openai/test_video_api.cpp | 116 +++++++- 15 files changed, 776 insertions(+), 22 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 576dbf46f..5dc3d19e8 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -25,7 +25,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | | Supported-models list (`row/DOCS-SUPPORTED-MODELS-MATRIX`) | **DRAFT PR**: FEATURES per-arch table CI-bound to registry (30 archs) | Reviewer merge | -| `/v1/videos` OpenAI shape (`row/SERVE-VIDEOS-OAI`) | **PR open**: Sora `model`/`size`/`seconds` + `GET /{id}/content`, CPU-gated | Follow-up row: reference conditioning | +| `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna diff --git a/.agents/coordination.md b/.agents/coordination.md index 4a2ef9b0a..5630a8b59 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1629,7 +1629,8 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-ROCM-W0-1` | `BACKEND-ROCM` (INVENTORIED→ACTIVE) | Claude Code (opus-5) | worktree `/home/mudler/_git/vllm.cpp-rocm-w0`, base `main` `4cfeee13`; CPU-only Release build, NO GPU of any kind | branches `row/backend-rocm-seam` (PR #87) + `row/backend-rocm-w0` (PR #88, stacked) | Owns ONLY the ROCm skeleton and the platform-SELECTION gate it uncovered: NEW `include/vt/rocm/{rocm_arch,rocm_runtime}.h`, NEW `src/vt/rocm/{rocm_backend,rocm_rmsnorm,rocm_ops}.hip`, NEW `src/vllm/platforms/rocm.cpp`, NEW `tests/vt/{test_rocm_arch,test_rocm_backend}.cpp`, NEW `.agents/specs/rocm-backend-w0.md`; ADDITIVE edits: `include/vt/device.h` (`kROCM`), `include/vllm/platforms/interface.h` + `src/vllm/platforms/platform.cpp` (`CurrentPlatformPriority()` + kROCM in the walk), `tests/vllm/platforms/test_platform.cpp`, `tests/vt/test_backend_cross_device.cpp` (kROCM in the discovery list), `CMakeLists.txt`/`tests/CMakeLists.txt`, this row + the `BACKEND-ROCM` matrix row + README/BUILD/ROCM/STATUS/FEATURES/BENCHMARKS + NOW/state. **NON-COLLISION:** every new file is ROCm-only; the only shared-code edits are one enum entry and one selection-array entry, neither of which changes behaviour for any registered backend (nothing registers kROCM in a non-HIP build). | `ACTIVE` | 2026-08-06 — **W0 LANDED, CPU-GATED, HIP UNBUILT.** The three `.hip` TUs have never been compiled by anyone (no AMD hardware here) — weaker than `build-supported`, and labelled UNBUILT everywhere. Verified: clean `-Werror` build, `ctest` full suite, `test_rocm_arch` 40/40 (upstream's own worked examples), `test_platform` incl. the new walk gate, device-leakage unchanged at 32; the two non-HIP legs are object-compiled in every build as a bit-rot guard. PENDING (hardware): every HIP compile, `test_rocm_backend`, the cross-device RmsNorm-vs-CPU-oracle comparison, and both `BACKEND-GATE-ROCM-*` rows. Next event is a contributor's first `-DVLLM_CPP_HIP=ON` on #41. | | `CLAIM-KERNEL-CPU-ELEM-WIDE-1` | `KERNEL-GEMM-CPU-ELEM-X86WIDE` + `KERNEL-GEMM-CPU-TILED` | Claude Code (opus-5) | worktree `/home/mudler/_git/vllm.cpp-x86wide`, base `cbc56f12` | branch `row/KERNEL-GEMM-CPU-ELEM-X86WIDE` | see ownership above | `SPIKE` | 2026-08-06 spike committed; W4 (widen the test tier sweep) is the first implementation step, deliberately before any kernel | -| `CLAIM-SERVE-VIDEOS-OAI` | `SERVE-VIDEOS-OAI` (NEW engine-matrix row; the MiniMax-H3 model rows and the `row/H3-FP4-SPEED` fp4 lane are UNTOUCHED) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-OAI`, base `main` `add5e9f9` | `/v1/videos` in OpenAI's Sora WIRE SHAPE (`model`, `size`, `seconds`) + `GET /v1/videos/{id}/content`. Owns `{include,src}/vllm/entrypoints/openai/video_api.*`, the video handlers/routes in `api_server.{h,cpp}`, both video test files, `specs/minimax-h3.md` §9 and the record surfaces. ADDITIVE: no generation kernel, no model forward, no VAE and no non-video route touched; the routes still register only with a `VideoRunner`. SPLIT: reference conditioning is a stacked follow-up row (spec §10), which is why this one carries no `examples/server/main.cpp` change. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 11/11-125, `test_openai_api_server` 40/40-509, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | +| `CLAIM-SERVE-VIDEOS-OAI` | `SERVE-VIDEOS-OAI` (NEW engine-matrix row; the MiniMax-H3 model rows and the `row/H3-FP4-SPEED` fp4 lane are UNTOUCHED) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-OAI`, base `main` `add5e9f9` | `/v1/videos` in OpenAI's Sora WIRE SHAPE (`model`, `size`, `seconds`) + `GET /v1/videos/{id}/content`. Owns `{include,src}/vllm/entrypoints/openai/video_api.*`, the video handlers/routes in `api_server.{h,cpp}`, both video test files, `specs/minimax-h3.md` §9 and the record surfaces. ADDITIVE: no generation kernel, no model forward, no VAE and no non-video route touched; the routes still register only with a `VideoRunner`. SPLIT: reference conditioning is the stacked row `SERVE-VIDEOS-REFS` (spec §10), which is why this one carries no `examples/server/main.cpp` change. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 11/11-125, `test_openai_api_server` 40/40-509, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | +| `CLAIM-SERVE-VIDEOS-REFS` | `SERVE-VIDEOS-REFS` (NEW engine-matrix row, stacked on `SERVE-VIDEOS-OAI`) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-REFS`, base `row/SERVE-VIDEOS-OAI` | Reference conditioning over `/v1/videos`: `input_reference` -> fl2va first-frame conditioning, the `metadata` video/audio references -> ref2va blocks, the exclusivity rule at the request boundary, and the `examples/server` runner wiring with lazily-loaded VAE encoder halves. Owns the reference members of `{include,src}/vllm/entrypoints/openai/video_api.*`, the video runner in `examples/server/main.cpp`, both video test files, `specs/minimax-h3.md` §10 and the record surfaces. ADDITIVE: no generation kernel and no model forward touched. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 14/14-167, `test_openai_api_server` 41/41-525, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | ## Handoff queue diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 3c217a49a..36d0b8d46 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 11 | -| Serving, API, CLI, library | 23 | 10 | 2 | 1 | 0 | 3 | 2 | 1 | 4 | +| Serving, API, CLI, library | 24 | 10 | 2 | 1 | 0 | 4 | 2 | 1 | 4 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 3 | 3 | 0 | 0 | 0 | 1 | 1 | 1 | -| **Total** | **141** | **36** | **16** | **4** | **7** | **19** | **8** | **9** | **41** | +| **Total** | **142** | **36** | **16** | **4** | **7** | **20** | **8** | **9** | **41** | ## Engine core and scheduling @@ -214,6 +214,7 @@ claims it. | `SERVE-RESPONSES-MESSAGES` | Responses, Anthropic messages, audio | T2 | `vllm/entrypoints/openai/responses/api_router.py:48`; `vllm/entrypoints/anthropic/api_router.py:49`; `vllm/entrypoints/speech_to_text/transcription/api_router.py:1` | - | - | `planned: specs/responses-messages-endpoints.md` | `INVENTORIED` | - | | `SERVE-ADMIN` | Abort-requests, sleep, pause/resume, profiling, RL weight updates. **`/abort_requests` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `POST /abort_requests` (from the dev/rlhf admin router) parses `{request_ids:[...]}` and aborts exactly those (external) ids via an injected abort callback wired to the engine abort path (`AsyncLLM::abort`); an empty/missing list means "abort all in-flight" (the callback decides). Response `{"status":"aborted","aborted":}`; malformed JSON → 400 `{"detail":"Invalid JSON format"}`; abort failure → 500 `{"error":...}` — all three shapes mirror the upstream router verbatim. ADDITIVE + opt-in (route registered only when the abort callback is attached → 404 otherwise). **PRODUCTION `main.cpp` WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary now wires `/abort_requests` to the LIVE `AsyncLLM::abort` through the shared `ConfigureUtilityEndpoints` seam, DEV-mode gated behind the new `--enable-server-dev-mode` flag — mirroring vLLM registering the dev/rlhf router only under `if envs.VLLM_SERVER_DEV_MODE` (api_server.py:238; envs.py:157 default 0). Explicit-id abort tears the request down and reports the exact drop in unfinished requests (before−after); empty `request_ids` (abort-ALL) reports 0 — NAMED RESIDUAL (AsyncLLM exposes no active-request-id accessor). RESIDUAL: the abort-ALL enumeration (missing `AsyncLLM::active_request_ids()`); `/sleep`/`/wake_up`/`/is_sleeping`, `/pause`/`/resume`, `/start_profile`/`/stop_profile`, weight-update/EP endpoints still INVENTORIED | T2/T3 | `vllm/entrypoints/serve/dev/rlhf/api_router.py:94-138` (abort_requests); dev-mode gate `vllm/entrypoints/openai/api_server.py:238-240`, `vllm/entrypoints/serve/__init__.py:35`, `vllm/envs.py:157`; `vllm/entrypoints/serve/dev/sleep/api_router.py:21`; `vllm/entrypoints/serve/dev/rlhf/api_router.py:29,74,136`; `vllm/entrypoints/serve/profile/api_router.py:21` | handler `src/vllm/entrypoints/openai/api_server.cpp:488` (`handle_abort_requests`); opt-in setter `include/vllm/entrypoints/openai/api_server.h:156` (`set_abort_requests`); production seam `src/vllm/entrypoints/openai/api_server.cpp` (`ConfigureUtilityEndpoints`, before/after delta-count) + `examples/server/main.cpp` (`--enable-server-dev-mode`); engine abort path `include/vllm/v1/engine/async_llm.h:115` (`abort`) | `tests/vllm/entrypoints/openai/test_api_server.cpp:1104` (shape + callback wiring: explicit ids passthrough, empty→abort-all branch, malformed→400),`:1143` (aborts an in-flight AsyncLLM request → `has_unfinished_requests()` false),`:1250` (opt-in route gate: 404 no-callback → 200 attached, RED-first),`:1319` (**production seam: dev-mode gate 404→200, live abort exact delta-count==1, empty→0**) — in the 32/32 / 420-assertion suite | [admin-endpoints.md](specs/admin-endpoints.md) | `ANCHOR-BACKFILL` | `CLAIM-C8-SERVE-PROD-WIRING` | | `SERVE-VIDEOS-OAI` | `/v1/videos` in OpenAI's Sora WIRE SHAPE, over the vLLM-Omni-derived job endpoints. **CPU-LANDED + GATED 2026-08-06 (`CLAIM-SERVE-VIDEOS-OAI`):** the OpenAI request spellings (`model`, `size` "WxH", `seconds` as a number OR the string enum OpenAI actually types) parse as ALIASES onto the existing native members, NATIVE-wins precedence applied PER-AXIS, both spellings validated either way so a malformed alias is a 400 even when overridden; an unserved `model` is a job `warning` echoed for the job's whole life, never a rejection (a Sora client cannot know the local model's name); and `GET /v1/videos/{id}/content` serves the finished MP4 (404 unknown / 409 unfinished / 500 failed / 500 vanished), without which a caller could start and poll a job but never fetch the result over HTTP. All four routes still register ONLY with a `VideoRunner` attached, now gated over a REAL socket. RESIDUALS (named): OpenAI's status vocabulary/id shape is not mirrored; reference conditioning (`input_reference`, the `metadata` video/audio references) is a stacked follow-up row; the real-weights leg rides the H3 GB10/disk window. | T2 | OpenAI Sora video API (`POST /v1/videos`, `GET /v1/videos/{video_id}/content`); vLLM-Omni `vllm/entrypoints/openai/video/api_router.py` (the async/sync job pair we already mirror) | `include/vllm/entrypoints/openai/video_api.h:31`; `src/vllm/entrypoints/openai/video_api.cpp:98`; `src/vllm/entrypoints/openai/api_server.cpp:279` | `tests/vllm/entrypoints/openai/test_video_api.cpp:64`; `tests/vllm/entrypoints/openai/test_api_server.cpp:1751` | [minimax-h3.md §9](specs/minimax-h3.md) | `ACTIVE` | `CLAIM-SERVE-VIDEOS-OAI` | +| `SERVE-VIDEOS-REFS` | REFERENCE CONDITIONING over `/v1/videos`: the image an OpenAI request starts from, plus the two modalities OpenAI's schema has no slot for. **CPU-LANDED + GATED 2026-08-06 (`CLAIM-SERVE-VIDEOS-REFS`, stacked on `SERVE-VIDEOS-OAI`):** OpenAI's `input_reference` (a filesystem path or an RFC 2397 `data:` URL, decoded by the SAME `DecodeDataUri` the chat multimodal parts use) maps to fl2va FIRST-FRAME conditioning via `MiniMaxH3EncodeKeyframeCondRows`, because OpenAI documents it as the frame the video starts from and ref2va would silently change what the API promises; the silent-video and audio references ride the standard free-form `metadata` map (`input_reference_video`, a DIRECTORY of `frame_%06d.ppm` since no demuxer is vendored; `input_reference_audio`, a 16-bit PCM WAV) and become ref2va blocks, an audio reference ATTACHING to the video block when both are given. fl2va-vs-ref2va exclusivity is enforced in the PARSER, mirroring `minimax_h3_pipeline.cpp:251`, so an illegal pair is a 400 naming it rather than a dropped reference. Both VAE ENCODER halves load lazily and once. RESIDUALS (named): reference images are binary PPM at the output resolution (no PNG/JPEG codec, no resampler vendored); a video reference is a frame directory; OpenAI's real upload is multipart, ours is the JSON spelling. | T2 | OpenAI Sora video API (`input_reference`, `metadata`); exclusivity rule `src/vllm/model_executor/models/minimax_h3_pipeline.cpp:251` | `include/vllm/entrypoints/openai/video_api.h:51`; `src/vllm/entrypoints/openai/video_api.cpp:66`; `examples/server/main.cpp:96` | `tests/vllm/entrypoints/openai/test_video_api.cpp:172`; `tests/vllm/entrypoints/openai/test_api_server.cpp:1827` | [minimax-h3.md §10](specs/minimax-h3.md) | `ACTIVE` | `CLAIM-SERVE-VIDEOS-REFS` | | `SERVE-OTLP` | OpenTelemetry traces | T2 | `vllm/config/observability.py:18,36,128` | - | - | `planned: specs/otlp-tracing.md` | `INVENTORIED` | - | | `SERVE-BATCH-API` | Offline OpenAI Batch API runner — read a JSONL of `BatchRequestInput` (`custom_id`/`method`/`url`/`body`), dispatch each line to the matching serving handler, collect `BatchRequestOutput` rows (custom_id echoed, per-line error isolation), write the response JSONL. **W0 SPIKE + W1 CPU BRICK LANDED + CPU-GATED 2026-07-29 (`CLAIM-BATCH-API`, NOT pushed):** `RunBatch` (RunLine/RunLines/Run) + `RunBatchFile` (local paths) as a pure ORCHESTRATOR over the existing `OpenAIServingChat::create_chat_completion` (the SAME handler `handle_chat_completions` drives — NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map. `/v1/chat/completions` wired; `BatchResponseData`/`BatchRequestOutput` schema + `vllm-`/`vllm-batch-` ids; the `run_request` AllResponse/ErrorResponse/stream branches; the unsupported-endpoint (handler None) + unsupported-url error rows. RECORDED DEVIATION: a malformed input line is ISOLATED into an error row (batch continues) where upstream aborts the job (deviation lives in the library; the abort-on-bad-line CLI exit code is a W2 residual). RESIDUALS (named, spec §Work breakdown): the `vllm run-batch` CLI + `BatchFrontendArgs` (W2); embeddings/score/rerank dispatch (W3, rides `SERVE-POOLING-ENDPOINTS`); audio transcription/translation + media fetch (W4); http(s)/data-URL file I/O, metrics server, overlapped `AsyncLLM` submission (W5) | T2 | `vllm/entrypoints/openai/run_batch.py:148-228` (schema),`:508-570` (run_request/make_error),`:722-777,815-847` (dispatch/run loop); `tests/entrypoints/openai/test_run_batch.py:375,402,432` | `include/vllm/entrypoints/openai/run_batch.h`; `src/vllm/entrypoints/openai/run_batch.cpp:70,84,87,142` | `tests/vllm/entrypoints/openai/test_run_batch.cpp:434` (7 cases / 80 assertions, CPU, RED-first: dropping the custom_id echo fails 9 assertions) | [batch-api.md](specs/batch-api.md) | `ANCHOR-BACKFILL` | `CLAIM-BATCH-API` | diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md index 985beef25..e801e504e 100644 --- a/.agents/specs/minimax-h3.md +++ b/.agents/specs/minimax-h3.md @@ -787,8 +787,8 @@ before means exactly what it meant before. SPLIT, deliberately: this row is the REQUEST/RESPONSE SHAPE only (`model`, `size`, `seconds`, and the MP4 download route). It touches no generation code and loads no VAE. The REFERENCE CONDITIONING half (`input_reference` -> fl2va, plus -the two `metadata` reference modalities -> ref2va) is a stacked follow-up row -(spec §10), because it is a separate capability that pulls in the VAE +the two `metadata` reference modalities -> ref2va) is §10, row +`SERVE-VIDEOS-REFS`, because it is a separate capability that pulls in the VAE encoder halves and the runner. Each half is independently reviewable and gated. ### 9.0 Spike contract (`SERVE-VIDEOS-OAI`) @@ -801,7 +801,7 @@ encoder halves and the runner. Each half is independently reviewable and gated. | Port map | Request contract -> `include/vllm/entrypoints/openai/video_api.h` (`VideoRequest::model` + `ParseVideoSize`) and `src/vllm/entrypoints/openai/video_api.cpp` (`ParseVideoRequest`, `ReadDuration`, `ParseWholeNumber`). Job record -> `VideoJobStore::Create(model, warning)` + `VideoJobStatusJson`. Download route -> `ApiServer::handle_video_content` + `video_model_warning` + their registration in `src/vllm/entrypoints/openai/api_server.cpp`. | | Tests to port | No upstream test module exists for this surface (OpenAI publishes an API, not tests; vLLM-Omni's video endpoint has no ported test). The contract is gated in-tree instead, extending the existing files: `tests/vllm/entrypoints/openai/test_video_api.cpp` (parsing, precedence, the job record) and `tests/vllm/entrypoints/openai/test_api_server.cpp` (routes, content behaviour, additivity over a real socket). Every assertion uses values that DIFFER from the field default. | | Gates | CPU, foreground: `test_video_api` 11/11 (125 assertions), `test_openai_api_server` 40/40 (509), `server` builds clean. Content route: 404 unknown / 409 unfinished (no bytes leaked) / 500 failed / 500 vanished / 200 byte-exact `video/mp4`. Additivity: with no `VideoRunner`, `POST /v1/videos` is 404 over a real socket with no `ErrorResponse` envelope; with one, it is 200 and the unknown-id 404 IS ours. Commands: `cmake --build build --target test_video_api test_openai_api_server server -j12`. Real-weights e2e rides §8's GB10/disk window. | -| Dependencies | Row IDs: the MiniMax-H3 model rows and `row/H3-FP4-SPEED` (UNTOUCHED - no generation code changed); the stacked reference-conditioning row (§10) builds on this one. No new download, no GPU, no toolchain change for the CPU gate. | +| Dependencies | Row IDs: the MiniMax-H3 model rows and `row/H3-FP4-SPEED` (UNTOUCHED - no generation code changed); `SERVE-VIDEOS-REFS` (§10) stacks on this row. No new download, no GPU, no toolchain change for the CPU gate. | | Work breakdown | (1) alias parsing + precedence + `ParseVideoSize`; (2) `model` recording + the job `warning`; (3) `handle_video_content` + its route; (4) both test files; (5) docs + record. | | Risks/decisions | NATIVE-wins precedence: the only direction that leaves every previously-parsing body meaning what it meant. `model` mismatch WARNS rather than 404s: a Sora client cannot know the local model name, so a rejection would defeat the compatibility; silence would hide it. A 409 (never bytes) on an unfinished job: a partially muxed file would reach the client as a valid-looking, truncated MP4. No vLLM-defined behaviour is reopened. | @@ -833,5 +833,80 @@ failure; a vanished output -> 500, not a 200 with zero bytes. (a 404 with no `ErrorResponse` envelope), with one they serve. - **Residuals, named.** OpenAI's status vocabulary is not mirrored (ours stays queued/running/succeeded/failed, ids `vid_N`, no `object`/`progress`/ - `created_at`); reference conditioning is §10, not this row. + `created_at`); reference conditioning is §10 (`SERVE-VIDEOS-REFS`), not this row. +- **Real-weights leg** rides the same GB10/disk window as §8. + +## 10. W-REFS — reference conditioning over `/v1/videos`, 2026-08-06 + +Row `SERVE-VIDEOS-REFS` (engine matrix, Serving surface), claim +`CLAIM-SERVE-VIDEOS-REFS`, branch `row/SERVE-VIDEOS-REFS`, stacked on §9. + +§9 made an OpenAI client's request PARSE. This row makes its REFERENCES do +something: an image the video starts from, a clip it continues, a voice it +carries. Before it, no reference modality was reachable over HTTP at all. + +### 10.0 Spike contract (`SERVE-VIDEOS-REFS`) + +| Section | Content | +|---|---| +| Scope | IN: OpenAI's `input_reference` mapped to fl2va first-frame conditioning; the two reference modalities OpenAI has no slot for carried in `metadata` (`input_reference_video`, `input_reference_audio`) mapped to ref2va blocks; the fl2va/ref2va combination rule enforced at the request boundary; the `examples/server` runner wiring (PPM decode, frame-directory clip, WAV, lazily-loaded VAE encoder halves). OUT: the ref2va IMAGE modality (reachable via the native `task` + the CLI, deliberately not bound to `input_reference`); any change to generation, the DiT, the VAEs or the muxer; OpenAI's multipart upload. | +| Upstream chain | OpenAI documents `input_reference` as the image the generated video STARTS FROM. The conditioning entry points are ours and already gated: `MiniMaxH3EncodeKeyframeCondRows` (fl2va), `MiniMaxH3EncodeReferenceVideo` / `MiniMaxH3EncodeReferenceAudio` (ref2va), `MiniMaxH3ReadWav`. The exclusivity rule is `src/vllm/model_executor/models/minimax_h3_pipeline.cpp:251`. | +| Our baseline | After §9 the OpenAI wire shape parses, but every reference field is absent: an image-to-video request silently generated from the prompt alone. | +| Port map | Request contract -> `include/vllm/entrypoints/openai/video_api.h` (`input_reference*`, `metadata`, the `has_*` predicates) and `src/vllm/entrypoints/openai/video_api.cpp` (`ReadReferenceSource`, `ReadMetadata`, the combination `VT_CHECK`); the `data:` decode REUSES `entrypoints::openai::DecodeDataUri` (chat_mm) rather than a second decoder. Reference wiring (the process boundary keeps it out of the library) -> `examples/server/main.cpp`: `DecodePpmChw`, `ReadReferenceClipChw` (the CLI's `DIR/frame_%06d.ppm` convention), `ReadReferenceBytes`, and the fl2va / ref2va branches with lazily-loaded VAE encoder halves. | +| Tests to port | No upstream test module exists for this surface. The contract is gated in-tree, extending the same two files: `test_video_api.cpp` (reference parsing, `metadata` passthrough, combination legality) and `test_api_server.cpp` (each modality ARRIVES at the runner, and an illegal pair is a 400 that generates nothing). | +| Gates | CPU, foreground: `test_video_api` 14/14 (167 assertions), `test_openai_api_server` 41/41 (525), `server` builds clean. Commands: `cmake --build build --target test_video_api test_openai_api_server server -j12`. Real-weights e2e rides §8's GB10/disk window. | +| Dependencies | Row `SERVE-VIDEOS-OAI` (§9), stacked. Code: `MiniMaxH3Encode{KeyframeCondRows,ReferenceVideo,ReferenceAudio}`, `MiniMaxH3ReadWav`, `DecodeDataUri`. Runtime: `--video-vae` for an image or video reference, `--audio-vae` for an audio reference (both encoder halves, loaded lazily and once). No new download, no GPU. | +| Work breakdown | (1) `input_reference` parsing (path or `data:` URL) -> fl2va, with the geometry refusal; (2) the `metadata` map + the video/audio reference keys; (3) the combination rule in the parser; (4) the `examples/server` runner branches; (5) both test files; (6) docs + record. | +| Risks/decisions | `input_reference` -> fl2va, NOT ref2va: OpenAI documents it as the frame the video starts from; ref2va would silently change what the API promises. The two extra modalities go in `metadata` rather than new top-level fields, so a strict client's schema validation still passes. Combination legality is enforced in the PARSER, not left to the pipeline, so a supplied reference is never silently dropped. | + +### 10.1 `input_reference` is fl2va, not ref2va + +OpenAI documents it as the image the video STARTS FROM, which is what +`MiniMaxH3EncodeKeyframeCondRows` expresses (frame 0 of the output pinned to the +image, `imgvid_noise_aug = 1.0`). `MiniMaxH3EncodeReferenceImages` prepends whole +reference images as their own blocks — guidance that never becomes a frame — so +mapping it there would have changed what the API promises. With a reference image +and no explicit `task`, the task IS `fl2va`, and the image's aspect drives the +default resolution through `MiniMaxH3ResolveShape`. + +Two limits, both refused up front rather than deep in the denoise: the image must +be a binary PPM (P6), because no PNG/JPEG codec is vendored (the same NAMED +residual the chat multimodal path carries), and it must already be at the +resolved output geometry, because no image resampler is vendored. The refusal +names both geometries. + +### 10.2 The two reference modalities OpenAI has no slot for + +H3 has three (image, silent video, audio); the Sora schema carries one. The other +two enter through `metadata`, the standard OpenAI free-form string map that +strict clients tolerate, rather than invented top-level fields that would fail a +client's schema validation. The whole map is kept verbatim. + +- `metadata.input_reference_video` — a DIRECTORY of `frame_%06d.ppm`, the exact + layout `minimax-h3-gen` and the server WRITE, so clips chain. No demuxer is + vendored, hence a frame directory rather than a container; a `data:` URL cannot + name a directory and is refused by name. + `MiniMaxH3EncodeReferenceVideo` emits `ref_audio_t == 0`: the clip is SILENT. +- `metadata.input_reference_audio` — a 16-bit PCM WAV path or `data:` URL, read + by the existing `MiniMaxH3ReadWav` and encoded by + `MiniMaxH3EncodeReferenceAudio`. Supplied with a video reference it ATTACHES to + that block (one `kVideoAudio` block carrying both, the layout + `packed_sequence.py` builds); alone it is its own block. + +LEGALITY is the pipeline's own rule (`minimax_h3_pipeline.cpp:251`: fl2va +keyframes and ref2va blocks are exclusive), enforced in the PARSER so it is a 400 +naming the pair rather than a failed job — and never a silently dropped +reference, which is the failure that looks like it worked. Legal: none / image / +video / audio / video+audio. Illegal: `input_reference` with either metadata +reference. + +### 10.3 Status + +- **CPU-LANDED + gated.** `test_video_api` 14/14 (167 assertions), + `test_openai_api_server` 41/41 (525), `server` builds clean. Each modality is + gated as ARRIVING at the runner, and an illegal pair is a 400 that generates + nothing (`calls == 0`). +- **Residuals, named.** Reference images are binary PPM at the output resolution; + a video reference is a frame DIRECTORY; OpenAI's real `input_reference` upload + is multipart, ours is the JSON spelling. - **Real-weights leg** rides the same GB10/disk window as §8. diff --git a/.agents/state.md b/.agents/state.md index 5766f6b6d..009c8ec16 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -40501,3 +40501,79 @@ The fix reads the SECOND parent, which is the PR head: a merge of a branch whose own commits name the row IS arrival through a row PR, one hop away. NOT a weakening, and gated as such: a new test asserts that a merge naming no row anywhere (and a plain local `Merge branch 'wip'`) STILL fails. Suite 40/40. + +## 2026-08-07T04:40 - reference CONDITIONING over `/v1/videos`: `input_reference` -> fl2va, `metadata` -> ref2va + + + +Row `SERVE-VIDEOS-REFS`, claim `CLAIM-SERVE-VIDEOS-REFS`, branch +`row/SERVE-VIDEOS-REFS`, STACKED on `SERVE-VIDEOS-OAI`. That row made an OpenAI +client's body PARSE; this one makes its REFERENCES do something. Before it, no +reference modality was reachable over HTTP at all: an image-to-video request +generated silently from the prompt alone. + +**`input_reference` -> fl2va, NOT ref2va.** OpenAI documents it as the image the +generated video STARTS FROM, which is exactly what fl2va expresses: +`MiniMaxH3EncodeKeyframeCondRows` pins frame 0 OF THE OUTPUT to the supplied +image (`imgvid_noise_aug = 1.0`). `MiniMaxH3EncodeReferenceImages` means +something else entirely: whole reference images PREPENDED as their own blocks, +i.e. subject or style guidance that never becomes a frame of the result. Mapping +`input_reference` there would have silently changed what the API promises. With a +reference and no explicit `task`, the task IS fl2va, and the image's aspect +drives the default geometry through `MiniMaxH3ResolveShape`. The source is a +filesystem path OR an RFC 2397 `data:` URL, decoded by the SAME +`entrypoints::openai::DecodeDataUri` the chat multimodal parts use rather than a +second, subtly different decoder; an http(s) URL is REFUSED BY NAME rather than +stat-ed as a path and failing later with a confusing message. + +**The two modalities OpenAI has no slot for ride `metadata`.** H3 supports three +(image, silent video, audio) and the Sora schema carries one, so the other two +enter through the standard free-form string map every strict client already +tolerates, rather than invented top-level fields that would fail a client's +schema validation. The whole map is kept verbatim (unknown keys pass through +untouched); two keys are lifted into typed fields. +`metadata.input_reference_video` is a DIRECTORY of `frame_%06d.ppm` -- the exact +layout `minimax-h3-gen` and this server WRITE, so one run's frames chain straight +into the next request -- because no container demuxer is vendored; a `data:` URL +cannot name a directory and is refused saying so. +`metadata.input_reference_audio` is a 16-bit PCM WAV path or `data:` URL. + +**A video reference is SILENT, by construction and stated.** +`MiniMaxH3EncodeReferenceVideo` emits `ref_audio_t == 0`, because the clip's own +soundtrack would need the audio VAE encoder run over it. An audio reference +supplied alongside ATTACHES to that same `kVideoAudio` block (its `ref_audio_t` +claims exactly the rows just encoded), which is the layout `packed_sequence.py` +builds; alone it is its own block. + +**Legality is enforced in the PARSER, not left to the pipeline.** The rule is the +pipeline's own (`minimax_h3_pipeline.cpp:251`: fl2va keyframes and ref2va blocks +are EXCLUSIVE), so an illegal pair is a 400 naming it rather than a failed job -- +and never a silently dropped reference, which is the failure that looks like it +worked. Legal: none / image / video / audio / video+audio. Illegal: +`input_reference` with either metadata reference. + +**Two refusals up front rather than deep in the denoise.** A reference image must +be a binary PPM (P6): no PNG/JPEG codec is vendored, the same NAMED residual the +chat multimodal path carries. And it must already be at the RESOLVED output +geometry, because no image resampler is vendored -- a mis-sized keyframe would +either abort inside the denoise or pin the wrong latent rows, so the refusal +names both geometries. Both VAE ENCODER halves (video and audio) load LAZILY and +ONCE each under one mutex: a text-to-video server must not pay for weights it +never uses, and a server that does use them must not reload per request. + +**GATE (CPU, foreground).** `test_video_api` 14/14 (167 assertions), +`test_openai_api_server` 41/41 (525), `server` builds clean. Each modality is +gated as ARRIVING at the runner (a reference that parsed and then never reached +the pipeline is the failure that looks like it worked), and an illegal pair is a +400 that generates nothing (`calls == 0`). + +**Records.** spec §10, engine-matrix `SERVE-VIDEOS-REFS` (+1 row, +`check-agent-record.py` ENGINE_ROWS 141 -> 142), coordination claim, USAGE +(reference fields, the `metadata` block, the legality table), FEATURES/BENCHMARKS +/STATUS rows updated IN PLACE (keyed tables), STATUS ratchet lowered +284043 -> 284037 in the same change, NOW. + +**Residuals, named.** Reference images are binary PPM at the output resolution; a +video reference is a frame DIRECTORY; OpenAI's real `input_reference` upload is +multipart, ours is the JSON spelling; the real-weights leg rides the H3 GB10/disk +window. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 96e029024..3c214c75c 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | DeepSeek-V4-Flash vs vLLM | Infeasible on one Spark | 2x GB10 with TP2 over the NCCL seam | | DFlash speculative decode | **CLOSED 2026-07-27 (D14)**: warp-scoped draft attention (242.9 → 77.9 ms), c1 our-on 29.32 vs vLLM-on 29.24 tok/s, non-overlapping 3-rep bands, 1.003x | none, closed | | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | -| `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (the `model`/`size`/`seconds` request aliases + the MP4 content route), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | +| `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d993ea7fa..c23c1c103 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -227,7 +227,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Plugin / out-of-tree model registration | ✅ in-tree factory `DONE` + plugin seam | ✅ | ◐ | ☐ | | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ | | Embedding / pooling endpoints | ◐ engine only | ✅ | ✅ | ✅ | -| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases, native fields win, + `GET /{id}/content` | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | +| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ | ## Parallelism and scale-out diff --git a/docs/STATUS.md b/docs/STATUS.md index 0add0aeef..5aa4d18b0 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -111,7 +111,7 @@ token-for-token correctness against the pinned oracle. | Tool-call parsing | 36 parser families / 40 accepted names, streaming | Every vLLM tool parser at the pin except the three Rust/Harmony-backed ones: pure-text parsers ported 1:1, the six engine-backed families reimplemented from their wire formats, all held to the upstream test suites. Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md | | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | | Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md | -| OpenAI server | Supported (subset) | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | +| OpenAI server | Supported (subset) | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`, conditioned on an `input_reference` image or the `metadata` video/audio references. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | | Pooling task class (embeddings / classify / score / rerank) | Spiked; pooler op + heads composite + pooling runner path landed (CPU), not yet servable end-to-end (no concrete model / endpoints) | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** — `EmbeddingPoolerHead` (projector→matryoshka→normalize), `ClassifierPoolerHead` (classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` task routing, and the `PoolerConfig`/`PoolingParams` structs (`test_pooler_heads` 27/27, 240 asserts, RED-first). **W3 landed the pooling RUNNER path** — `PoolingRunner` applies the model's `Pooler` to the last hidden state and returns pooled embeddings instead of sampled tokens, gated by a STRUCTURAL cosine-parity check vs a double-precision LAST+normalize reference (`test_pooling_runner` 5/5, 14 asserts, RED-first). NOT yet servable / honest residuals: a concrete pooling MODEL forward + the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) — no cosine-vs-oracle number is fabricated (W3-model); the `/v1/embeddings` + score/rerank/classify endpoints (W4); tokwise AllPool/StepPool (W5). See docs/BENCHMARKS.md | | Plugin system (out-of-core registration) | Spiked; first CPU brick landed, not yet wired into any production path | The extensibility-first discovery layer. W0 spike over vLLM's plugin surface (general / platform / io_processor / endpoint groups, the `register_model` an out-of-tree plugin calls, the invocation seams) is committed (`.agents/specs/plugin-system.md`, `ENG-PLUGIN-SYSTEM` ACTIVE, `CLAIM-PLUGIN-SYSTEM`). W1 landed `vllm::plugins::LoadGeneralPlugins()` + the out-of-core general-plugin registration seam (`RegisterGeneralPlugin` / `REGISTER_VLLM_GENERAL_PLUGIN`) over the existing `REGISTER_VLLM_MODEL`-style registries (the in-tree factory `MODEL-FACTORY-registry` is record-repaired `DONE` 2026-08-05: 28 self-registering TUs, dgx debt paid by the 2026-07-23 seven-gate run): a 1:1 mirror of `load_general_plugins` (load-once idempotence, the `VLLM_PLUGINS` allowlist, per-plugin failure isolation). Proven by an out-of-core toy-model plugin that registers a toy architecture through the public `RegisterModel` seam — unit-gated RED-first (`test_plugin_system` 1 case / 29 assertions: the toy arch resolves ONLY after LoadGeneralPlugins runs it, and not under `VLLM_PLUGINS=""`). Python entry points have no C++20 analogue, so discovery is the project's static-init/`dlopen` registration idiom (recorded porting-inventory §9). NOT yet wired: real shared-object `dlopen` + the C-ABI `vllm_plugin_register` entry (W2), the engine/CLI `--load-plugins` wiring that calls LoadGeneralPlugins from the construction paths (W3), the platform/quant plugin kinds (W4), and the io_processor/stat_logger/endpoint groups (W5) are named residuals. See docs/BENCHMARKS.md | | Offline Batch API (JSONL file runner) | Spiked; first CPU brick landed, not yet exposed as a CLI | The offline OpenAI Batch API: read a JSONL of OpenAI-format requests, run each through the engine, write a JSONL of responses. W0 spike over vLLM's `run_batch.py` (schema, endpoint dispatch, run loop, file I/O) is committed (`.agents/specs/batch-api.md`, `SERVE-BATCH-API` ACTIVE, `CLAIM-BATCH-API`). W1 landed `RunBatch` (`RunLine`/`RunLines`/`Run`) + `RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map: `/v1/chat/completions` dispatch, the `BatchResponseData`/`BatchRequestOutput` schema (`vllm-` ids, custom_id echoed), the `run_request` AllResponse/ErrorResponse/stream branches, and the unsupported-endpoint/url error rows. Unit-gated RED-first (`test_openai_run_batch` 7 cases / 80 assertions over the synthetic serving engine: ordered rows + custom_id echo + per-line BatchRequestOutput schema round-trip, a malformed line isolated into an error row so the batch continues, dispatch + 404 error rows; dropping the custom_id echo fails 9 assertions). Recorded deviation: a malformed line is isolated (batch continues) where upstream aborts the job. NOT yet exposed: the `vllm run-batch` CLI + `BatchFrontendArgs` (W2), embeddings/score/rerank dispatch (W3, rides pooling endpoints), audio transcription/translation + media fetch (W4), and http(s)/data-URL file I/O + metrics + overlapped `AsyncLLM` submission (W5) are named residuals. See docs/BENCHMARKS.md | diff --git a/docs/USAGE.md b/docs/USAGE.md index 100a076af..0647e4370 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -148,9 +148,11 @@ open("out.mp4", "wb").write(client.videos.download_content(video.id).read()) | `model` | OpenAI | Recorded and echoed back. A name this server does not serve is a `warning` on the job, never a rejection: the video model is chosen at startup | | `size` | OpenAI | `"x"`, e.g. `"1280x720"`. Whole pixels, both positive | | `seconds` | OpenAI | Duration, as a number or a numeric string (`8` and `"8"` both work) | +| `input_reference` | OpenAI | The image the video starts from. A filesystem path or a `data:` URL | +| `metadata` | OpenAI | Free-form string map, passed through untouched. Two keys are acted on: `input_reference_video` and `input_reference_audio` (see below) | | `width`, `height` | native | Output geometry in pixels | | `duration` | native | Duration in seconds | -| `task` | native | `t2va`, `fl2va`, `ref2va`; defaults to `t2va` | +| `task` | native | `t2va`, `fl2va`, `ref2va`; resolved from the inputs when omitted | | `num_frames`, `num_inference_steps`, `flow_shift`, `audio_flow_shift`, `seed` | native | The H3 generation knobs. Accepted at the top level or nested under `extra_params` | **Precedence.** When a body carries both spellings of one value, the **native @@ -159,6 +161,63 @@ direction keeps every request that parses today meaning exactly what it meant before. Both spellings are validated either way, so a malformed `size` is a 400 even when explicit `width`/`height` would have overridden it. +**`input_reference` maps to fl2va first-frame conditioning.** OpenAI documents +it as the image the generated video starts from, which is what fl2va expresses: +the supplied image is pinned as frame 0 of the output. H3's other image mode, +ref2va, prepends whole reference images as their own blocks (subject or style +guidance that never becomes a frame), so it stays reachable only through the +native `task` field and the `minimax-h3-gen` CLI. Two limits: the image must be +a **binary PPM (P6)** (no PNG or JPEG codec is vendored, the same residual the +chat multimodal path carries), and it must already be at the output resolution +(no image resampler is vendored). A mismatch is refused with the resolved +geometry in the message. + +### Video and audio references (`metadata`) + +H3 supports three reference modalities and OpenAI's schema has a slot for one, +so the other two enter through `metadata`, the standard OpenAI free-form string +map. Strict clients tolerate it, and no invented top-level field breaks their +schema validation. Unknown metadata keys are passed through untouched. + +```jsonc +{ + "prompt": "the same scene, at dusk", + "metadata": { + "input_reference_video": "/tmp/vllm_h3_videos/job0", // DIR of frame_%06d.ppm + "input_reference_audio": "/tmp/voice.wav" // 16-bit PCM WAV, or a data: URL + } +} +``` + +`input_reference_video` is a **directory of `frame_%06d.ppm`**, which is exactly +what this server and `minimax-h3-gen` write, so one run's frames chain straight +into the next request. It is not a container file: no demuxer is vendored. + +**A video reference is SILENT.** `MiniMaxH3EncodeReferenceVideo` emits a +`kVideoAudio` block with `ref_audio_t == 0`, so the clip contributes no sound of +its own. Supplying `input_reference_audio` alongside it attaches the audio to +that same block (one block carrying both, the layout upstream builds); without +it the reference is picture only. That is a real limitation, not an omission. + +**Legal combinations.** fl2va keyframes and ref2va reference blocks are +exclusive in the pipeline itself +([`minimax_h3_pipeline.cpp`](../src/vllm/model_executor/models/minimax_h3_pipeline.cpp)), +so the request parser enforces the same rule and returns a 400 naming the +offending pair rather than dropping a reference you supplied. + +| `input_reference` | `metadata.input_reference_video` | `metadata.input_reference_audio` | | +|---|---|---|---| +| (none) | (none) | (none) | t2va, prompt only | +| image | (none) | (none) | fl2va, the image is frame 0 | +| (none) | clip | (none) | ref2va, silent video reference | +| (none) | (none) | WAV | ref2va, audio reference | +| (none) | clip | WAV | ref2va, one block carrying both | +| image | clip and/or WAV | | **400**: keyframe and reference conditioning are exclusive | + +The video reference needs `--video-vae` (the encoder half of the same file) and +the audio reference needs `--audio-vae`; both load lazily, once, on the first +request that asks for them. + ### The job lifecycle `POST /v1/videos` returns immediately with `{"id": "vid_1", "status": "queued"}`; diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 3284b22cb..9544e21df 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -32,8 +32,11 @@ #include #include #include +#include #include +#include #include +#include #include #include #include @@ -90,6 +93,105 @@ namespace fs = std::filesystem; using vllm::HfConfig; using vllm::Qwen3_5MoeWeights; +// Decode a binary PPM (P6) into [3, H, W] floats in [-1, 1] — the layout the H3 +// video-VAE encoder takes. It reads from BYTES rather than a path so one decoder +// serves both spellings of `input_reference` (a filesystem path and an inline +// data: URL). PPM is the only still-image container this tree can read: no PNG / +// JPEG codec is vendored, which is the same NAMED residual the chat multimodal +// path carries (see chat_mm.h), not a limitation of this endpoint. +std::vector DecodePpmChw(const std::string& bytes, int64_t* out_h, + int64_t* out_w) { + std::istringstream in(bytes, std::ios::binary); + std::string magic; + in >> magic; + if (magic != "P6") { + throw std::runtime_error( + "input_reference: not a binary PPM (P6); no PNG/JPEG codec is vendored, " + "so a reference image must be supplied as binary PPM"); + } + auto next_int = [&]() { + int v = 0; + while (in >> std::ws, in.peek() == '#') { std::string skip; std::getline(in, skip); } + in >> v; + return v; + }; + const int w = next_int(), h = next_int(), maxv = next_int(); + if (w <= 0 || h <= 0 || maxv <= 0) { + throw std::runtime_error("input_reference: bad PPM header"); + } + in.get(); // the single whitespace byte before the payload + std::vector rgb(static_cast(w) * h * 3); + in.read(reinterpret_cast(rgb.data()), static_cast(rgb.size())); + if (!in) throw std::runtime_error("input_reference: truncated PPM payload"); + std::vector chw(rgb.size()); + const int64_t plane = static_cast(w) * h; + for (int64_t i = 0; i < plane; ++i) { + for (int64_t c = 0; c < 3; ++c) { + chw[static_cast(c * plane + i)] = + static_cast(rgb[static_cast(i * 3 + c)]) / (maxv * 0.5f) - 1.0f; + } + } + if (out_h != nullptr) *out_h = h; + if (out_w != nullptr) *out_w = w; + return chw; +} + +// The bytes behind one reference: the parser hands us either a path or the +// already-decoded payload of a data: URL. +std::string ReadReferenceBytes(const std::string& field, const std::string& path, + const std::vector& inline_bytes) { + if (!inline_bytes.empty()) return std::string(inline_bytes.begin(), inline_bytes.end()); + std::ifstream in(path, std::ios::binary); + if (!in) throw std::runtime_error(field + ": cannot open " + path); + return std::string((std::istreambuf_iterator(in)), std::istreambuf_iterator()); +} + +// A ref2va VIDEO reference: DIR/frame_%06d.ppm, the exact layout minimax_h3_gen +// and this server's own muxer WRITE, so one run's frames chain straight into the +// next request. Returns [C, T, H, W] in [-1, 1] (the encoder's clip layout); +// no container demuxer is vendored, which is why this is a frame directory. +std::vector ReadReferenceClipChw(const std::string& dir, int64_t* out_t, + int64_t* out_h, int64_t* out_w) { + std::vector per_frame; // frame-major [T][C,H,W] + int64_t frames = 0, fh = 0, fw = 0; + for (int64_t k = 0;; ++k) { + char name[512]; + std::snprintf(name, sizeof(name), "%s/frame_%06lld.ppm", dir.c_str(), + static_cast(k)); + std::ifstream probe(name, std::ios::binary); + if (!probe) break; + const std::string bytes((std::istreambuf_iterator(probe)), + std::istreambuf_iterator()); + int64_t h = 0, w = 0; + const std::vector frame = DecodePpmChw(bytes, &h, &w); + if (frames == 0) { fh = h; fw = w; } + if (h != fh || w != fw) { + throw std::runtime_error( + "metadata.input_reference_video: every frame_%06d.ppm must have the same size"); + } + per_frame.insert(per_frame.end(), frame.begin(), frame.end()); + ++frames; + } + if (frames == 0) { + throw std::runtime_error("metadata.input_reference_video: no frame_%06d.ppm files in " + dir); + } + // [T][C,H,W] -> [C,T,H,W], the causal 3-D encoder's layout. + std::vector chw(per_frame.size()); + const int64_t plane = fh * fw; + for (int64_t c = 0; c < 3; ++c) { + for (int64_t k = 0; k < frames; ++k) { + for (int64_t e = 0; e < plane; ++e) { + chw[static_cast((c * frames + k) * plane + e)] = + per_frame[static_cast(k * 3 * plane + c * plane + e)]; + } + } + } + *out_t = frames; + *out_h = fh; + *out_w = fw; + return chw; +} + struct Args { std::string model_dir; std::string host = "0.0.0.0"; @@ -613,6 +715,18 @@ int main(int argc, char** argv) { vllm::MiniMaxH3PartitionInfo partition_info; vt::Device device; std::atomic counter{0}; + // The two VAEs' ENCODER halves, for the reference modalities: the video VAE + // encodes an `input_reference` image and a `metadata.input_reference_video` + // clip, the audio VAE encodes a `metadata.input_reference_audio` waveform. + // Loaded LAZILY and ONCE each: a text-to-video server must not pay for + // weights it never uses, and a server that does use them must not reload + // per request. + std::string video_vae_path, audio_vae_path; + std::mutex encoder_mutex; + bool video_encoder_loaded = false, audio_encoder_loaded = false; + vllm::MiniMaxH3AudioVaeWeights video_encoder_weights, audio_encoder_weights; + vllm::MiniMaxH3EncoderFcn3dConfig video_encoder_cfg; + vllm::MiniMaxH3AudioVaeEncoderConfig audio_encoder_cfg; }; std::shared_ptr video; if (!args.video_dit.empty()) { @@ -677,6 +791,8 @@ int main(int argc, char** argv) { video->workdir = args.video_workdir; video->ffmpeg = args.video_ffmpeg; video->partition_info = vllm::MiniMaxH3PartitionFromFlag(args.video_partition); + video->video_vae_path = args.video_vae; + video->audio_vae_path = args.audio_vae; if (args.video_device == "cuda") { video->device = vt::GetBackend(vt::DeviceType::kCUDA).CreateQueue().device; } @@ -722,13 +838,50 @@ int main(int argc, char** argv) { "video generation needs conditioning: start the server with " "--video-encoder (to condition on the prompt) or --video-prompt-embeds"); } + // OpenAI `input_reference` -> fl2va FIRST-FRAME conditioning. + // + // WHY fl2va and not ref2va: OpenAI documents input_reference as the image + // the generated video STARTS FROM (image-to-video), which is exactly what + // fl2va expresses — MiniMaxH3EncodeKeyframeCondRows pins frame 0 OF THE + // OUTPUT to the supplied image. ref2va + // (MiniMaxH3EncodeReferenceImages) means something else: whole reference + // images PREPENDED to the sequence as their own blocks, i.e. subject or + // style guidance that never becomes a frame of the result. Mapping + // input_reference there would silently change what the API promises. The + // ref2va modalities OpenAI has no slot for enter through `metadata` + // (input_reference_video / input_reference_audio) instead; the parser has + // already refused the combinations the pipeline forbids. + std::vector reference_chw; + int64_t reference_h = 0, reference_w = 0; + if (req.has_input_reference()) { + reference_chw = DecodePpmChw( + ReadReferenceBytes("input_reference", req.input_reference_path, + req.input_reference_bytes), + &reference_h, &reference_w); + } + std::vector reference_clip; + int64_t clip_t = 0, clip_h = 0, clip_w = 0; + if (req.has_input_reference_video()) { + reference_clip = ReadReferenceClipChw(req.input_reference_video_dir, &clip_t, + &clip_h, &clip_w); + } + const vllm::MiniMaxH3DitParams& p = video->dit.params; vllm::MiniMaxH3T2vaRequest r; r.partition = video->partition_info; // #77 guard: MiniMaxH3GenerateT2va // refuses a task this partition can't serve. + // With a reference image and no explicit task, the task IS fl2va; a + // metadata reference means ref2va. The image aspect also drives the + // default resolution (_resolve_shape). + const bool has_ref2va = + req.has_input_reference_video() || req.has_input_reference_audio(); + const std::string task = + !req.task.empty() + ? req.task + : (req.has_input_reference() ? "fl2va" : (has_ref2va ? "ref2va" : "t2va")); const vllm::MiniMaxH3ShapePlan plan = vllm::MiniMaxH3ResolveShape( - req.task.empty() ? "t2va" : req.task, req.duration_seconds, req.num_frames, - req.height, req.width, 0, 0); + task, req.duration_seconds, req.num_frames, req.height, req.width, + reference_w, reference_h); r.latent_t = plan.latent_t; r.num_frames = plan.num_frames; r.latent_h = plan.height / vllm::kMiniMaxH3VaeRatio; @@ -744,6 +897,97 @@ int main(int argc, char** argv) { r.audio_latents_std = video->audio_stats.std_dev; r.text_len = static_cast(conditioning.size()) / p.text_dim; + // Both VAE encoder halves load at most once, under one lock, whichever + // reference modality asks for them first. + auto ensure_video_encoder = [&]() { + if (video->video_vae_path.empty()) { + throw std::runtime_error( + "a video/image reference needs the video VAE ENCODER half: start " + "the server with --video-vae"); + } + if (video->video_encoder_loaded) return; + const vllm::SafetensorsFile vf = vllm::SafetensorsFile::Open(video->video_vae_path); + video->video_encoder_weights = vllm::LoadMiniMaxH3VideoVaeEncoderWeights(vf); + video->video_encoder_cfg = vllm::MiniMaxH3EncoderFcn3dConfig{}; + video->video_encoder_cfg.z_channels = 2 * p.latents_dim; // mean | logvar + video->video_encoder_loaded = true; + }; + auto ensure_audio_encoder = [&]() { + if (video->audio_vae_path.empty()) { + throw std::runtime_error( + "metadata.input_reference_audio needs the audio VAE ENCODER half: " + "start the server with --audio-vae"); + } + if (video->audio_encoder_loaded) return; + const vllm::SafetensorsFile af = vllm::SafetensorsFile::Open(video->audio_vae_path); + video->audio_encoder_weights = vllm::LoadMiniMaxH3AudioVaeEncoderWeights(af); + video->audio_encoder_cfg = vllm::MiniMaxH3AudioVaeEncoderConfig{}; + video->audio_encoder_cfg.vae_latent_channels = p.audio_latents_dim; + video->audio_encoder_loaded = true; + }; + + if (req.has_input_reference()) { + if (reference_h != plan.height || reference_w != plan.width) { + // No image resampler is vendored, and a mis-sized keyframe would + // either abort deep in the denoise or pin the wrong latent rows. Say + // so up front, with the geometry we resolved. + throw std::runtime_error( + "input_reference is " + std::to_string(reference_w) + "x" + + std::to_string(reference_h) + " but this request resolved to " + + std::to_string(plan.width) + "x" + std::to_string(plan.height) + + "; supply the reference at the output size (or pass a matching " + "`size`): no image resampler is vendored"); + } + std::lock_guard guard(video->encoder_mutex); + ensure_video_encoder(); + r.keyframe_frame_indices = {0}; // FIRST frame; upstream also allows {-1}/{0,-1} + r.imgvid_noise_aug = 1.0; // pin the frame exactly + r.keyframe_cond_rows = vllm::MiniMaxH3EncodeKeyframeCondRows( + video->video_encoder_cfg, video->video_encoder_weights, p, {reference_chw}, + reference_h, reference_w, r.latent_t, r.imgvid_noise_aug, {}); + } else if (has_ref2va) { + // ── ref2va REFERENCE BLOCKS. Exclusive with the fl2va branch above + // (minimax_h3_pipeline.cpp:251), which the parser already enforced. ── + std::lock_guard guard(video->encoder_mutex); + std::vector blocks; + if (req.has_input_reference_video()) { + ensure_video_encoder(); + vllm::MiniMaxH3RefBlock block{}; + r.keyframe_cond_rows = vllm::MiniMaxH3EncodeReferenceVideo( + video->video_encoder_cfg, video->video_encoder_weights, p, reference_clip, + clip_t, clip_h, clip_w, &block); + // SILENT by construction: MiniMaxH3EncodeReferenceVideo emits + // ref_audio_t == 0 because the clip's own soundtrack would need the + // audio VAE encoder run over it. An audio reference below ATTACHES to + // this block, which is the layout packed_sequence.py builds. + blocks.push_back(block); + } + if (req.has_input_reference_audio()) { + ensure_audio_encoder(); + const std::string wav_bytes = ReadReferenceBytes( + "metadata.input_reference_audio", req.input_reference_audio_path, + req.input_reference_audio_bytes); + int64_t samples_per_channel = 0; + const std::vector waveform = vllm::MiniMaxH3ReadWav( + wav_bytes, vllm::kMiniMaxH3AudioChannels, vllm::kMiniMaxH3AudioSampleRate, + &samples_per_channel); + vllm::MiniMaxH3RefBlock audio_block{}; + r.audio_ref_rows = vllm::MiniMaxH3EncodeReferenceAudio( + video->audio_encoder_cfg, video->audio_encoder_weights, waveform, + vllm::kMiniMaxH3AudioChannels, samples_per_channel, video->audio_stats.mean, + video->audio_stats.std_dev, /*noise_aug=*/1.0, {}, &audio_block); + if (!blocks.empty() && + blocks[0].kind == vllm::MiniMaxH3RefBlock::Kind::kVideoAudio) { + // The reference video now HAS sound: one kVideoAudio block carries + // both, so its ref_audio_t must claim exactly the rows just encoded. + blocks[0].ref_audio_t = audio_block.ref_audio_t; + } else { + blocks.push_back(audio_block); + } + } + r.ref_blocks = blocks; + } + const int64_t frame_rows = (r.latent_h / p.patch_size_h) * (r.latent_w / p.patch_size_w); std::vector nv(static_cast(r.latent_t * frame_rows * diff --git a/include/vllm/entrypoints/openai/video_api.h b/include/vllm/entrypoints/openai/video_api.h index 89a16aeb1..655d2c779 100644 --- a/include/vllm/entrypoints/openai/video_api.h +++ b/include/vllm/entrypoints/openai/video_api.h @@ -5,7 +5,7 @@ // POST /v1/videos/sync -> run to completion, return the MP4 in the body // // It ALSO speaks OpenAI's Sora video shape, so an OpenAI client works unmodified: -// POST /v1/videos {model, prompt, size:"WxH", seconds} +// POST /v1/videos {model, prompt, size:"WxH", seconds, input_reference} // GET /v1/videos/{id} job status // GET /v1/videos/{id}/content the finished MP4 bytes (video/mp4) // The OpenAI spellings are ALIASES onto the native fields, never replacements — @@ -29,8 +29,8 @@ namespace vllm::openai { // One parsed /v1/videos request. Mirrors the fields vLLM-Omni accepts; anything // absent falls back to H3's documented defaults via the shape planner. The -// OpenAI-spelled fields (`model`, `size`, `seconds`) land on the SAME members, -// so nothing downstream learns a second vocabulary. +// OpenAI-spelled fields (`model`, `size`, `seconds`, `input_reference`) land on +// the SAME members, so nothing downstream learns a second vocabulary. struct VideoRequest { std::string prompt; // OpenAI `model` ("sora-2-pro", ...). RECORDED, never a hard failure: the video @@ -47,6 +47,38 @@ struct VideoRequest { double audio_flow_shift = 3.0; // audio int64_t seed = 0; bool has_seed = false; + + // OpenAI `input_reference` — the image an image-to-video request starts from. + // Exactly one of these is populated (see ParseVideoRequest): a filesystem path, + // or the bytes of an inline RFC 2397 `data:` URL plus its declared media type. + std::string input_reference_path; + std::vector input_reference_bytes; + std::string input_reference_media_type; + + bool has_input_reference() const { + return !input_reference_path.empty() || !input_reference_bytes.empty(); + } + + // OpenAI `metadata`: a free-form string->string map every strict client already + // tolerates. H3 supports THREE reference modalities and OpenAI's schema has a + // slot for exactly one (the image), so the other two enter here rather than as + // invented top-level fields that would fail a client's schema validation. The + // whole map is kept verbatim (unknown keys are passed through untouched); the + // two keys we act on are lifted out below. + // metadata.input_reference_video — a DIRECTORY of frame_%06d.ppm, the exact + // layout `minimax-h3-gen` writes, so one run's output chains into the next. + // SILENT: the clip carries no audio unless input_reference_audio is also + // given (the audio VAE's encoder is a separate call). + // metadata.input_reference_audio — a 16-bit PCM WAV path or `data:` URL. + std::map metadata; + std::string input_reference_video_dir; + std::string input_reference_audio_path; + std::vector input_reference_audio_bytes; + + bool has_input_reference_video() const { return !input_reference_video_dir.empty(); } + bool has_input_reference_audio() const { + return !input_reference_audio_path.empty() || !input_reference_audio_bytes.empty(); + } }; // Parse OpenAI's `size` — "x", e.g. "1280x720" — into its two @@ -60,6 +92,15 @@ void ParseVideoSize(const std::string& size, int64_t* width, int64_t* height); // malformed input rather than silently defaulting, so a bad request is a 400 with // a reason instead of a surprising generation. // +// REFERENCE COMBINATIONS are checked here, not left to the pipeline, because a +// reference the caller supplied and we quietly dropped is the failure that looks +// like it worked. The rule is the pipeline's own +// (minimax_h3_pipeline.cpp:251): fl2va keyframes and ref2va reference blocks are +// EXCLUSIVE. `input_reference` is fl2va; the metadata video/audio references are +// ref2va. So legal: nothing; input_reference alone; video alone; audio alone; +// video+audio (one kVideoAudio block carrying both). Illegal, and a 400 naming +// the pair: input_reference together with either metadata reference. +// // PRECEDENCE, when a body carries both spellings of one value: the NATIVE field // WINS (`width`/`height` over `size`, `duration` over `seconds`). The OpenAI // spelling is a compatibility shim, so this guarantees that every body which diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index ceb58b86d..ece193332 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -264,7 +264,15 @@ # fl2va/ref2va exclusivity is enforced at the request boundary. CPU-landed + # gated, `PARTIAL`, `CLAIM-SERVE-VIDEOS-OAI`, spec `specs/minimax-h3.md` §9. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 141 +# 142 since 2026-08-06: +`SERVE-VIDEOS-REFS` (reference CONDITIONING over +# `/v1/videos`: `input_reference` -> fl2va, plus the two `metadata` ref2va +# modalities) — a real new capability stacked on `SERVE-VIDEOS-OAI`, not a +# restatement of it: that row made an OpenAI body PARSE, this one makes its +# references reach the pipeline. Before it no reference modality was reachable +# over HTTP at all. CPU-landed + gated, `CLAIM-SERVE-VIDEOS-REFS`, spec +# `specs/minimax-h3.md` §10. +# Bumped for a real new row, never to make a failing state transition pass. +ENGINE_ROWS = 142 MATRIX_PATHS = [ENGINE_MATRIX, *(path for path, _ in MATRICES.values())] REQUIRED = [ diff --git a/src/vllm/entrypoints/openai/video_api.cpp b/src/vllm/entrypoints/openai/video_api.cpp index 6315b0c1f..a1d741184 100644 --- a/src/vllm/entrypoints/openai/video_api.cpp +++ b/src/vllm/entrypoints/openai/video_api.cpp @@ -8,6 +8,11 @@ #include #include #include + +// DecodeDataUri: the RFC 2397 `data:` decode the chat multimodal parts already +// use. `input_reference` accepting a data: URL means the SAME decoder serves both +// surfaces rather than a second, subtly different one. +#include "vllm/entrypoints/openai/chat_mm.h" #include "vt/dtype.h" namespace vllm::openai { @@ -65,6 +70,65 @@ double ReadDuration(const nlohmann::json& body, const char* key) { return 0.0; // unreachable; VT_CHECK(false) throws } +// A reference SOURCE: a filesystem path, or an inline `data:` URL. An http(s) +// URL is REFUSED by name rather than treated as a path, because silently +// stat-ing "https://..." would fail much later with a confusing message. +// `field` names the offending field so the 400 says which one. +void ReadReferenceSource(const std::string& field, const std::string& value, + std::string* out_path, std::vector* out_bytes, + std::string* out_media_type) { + VT_CHECK(!value.empty(), "video request: `" + field + "` must not be empty"); + if (value.compare(0, 5, "data:") == 0) { + try { + const entrypoints::openai::DecodedMedia media = + entrypoints::openai::DecodeDataUri(value); + VT_CHECK(!media.bytes.empty(), + "video request: `" + field + "` data: URL decoded to no bytes"); + *out_bytes = media.bytes; + if (out_media_type != nullptr) *out_media_type = media.media_type; + } catch (const std::exception& e) { + VT_CHECK(false, "video request: `" + field + "` is not a valid data: URL: " + + e.what()); + } + return; + } + VT_CHECK(value.compare(0, 7, "http://") != 0 && value.compare(0, 8, "https://") != 0, + "video request: `" + field + + "` must be a filesystem path or a data: URL; fetching an http(s) " + "URL is not supported"); + *out_path = value; +} + +// OpenAI `metadata`: a free-form string->string map. Everything is kept, and the +// two H3 reference keys OpenAI has no slot for are lifted into typed fields. +void ReadMetadata(const nlohmann::json& value, VideoRequest* out) { + VT_CHECK(value.is_object(), "video request: `metadata` must be an object"); + for (const auto& [key, item] : value.items()) { + VT_CHECK(item.is_string(), + "video request: `metadata." + key + + "` must be a string (metadata is a string map)"); + out->metadata[key] = item.get(); + } + const auto video = out->metadata.find("input_reference_video"); + if (video != out->metadata.end()) { + VT_CHECK(!video->second.empty(), + "video request: `metadata.input_reference_video` must not be empty"); + // A DIRECTORY of frame_%06d.ppm, not a container file: no demuxer is + // vendored, and this is exactly the layout `minimax-h3-gen` writes, so one + // run's frames chain into the next request. `data:` cannot name a directory. + VT_CHECK(video->second.compare(0, 5, "data:") != 0, + "video request: `metadata.input_reference_video` must be a directory " + "of frame_%06d.ppm files, so a data: URL cannot express it"); + out->input_reference_video_dir = video->second; + } + const auto audio = out->metadata.find("input_reference_audio"); + if (audio != out->metadata.end()) { + ReadReferenceSource("metadata.input_reference_audio", audio->second, + &out->input_reference_audio_path, + &out->input_reference_audio_bytes, nullptr); + } +} + } // namespace void ParseVideoSize(const std::string& size, int64_t* width, int64_t* height) { @@ -109,6 +173,24 @@ VideoRequest ParseVideoRequest(const std::string& body) { out.model = json.at("model").get(); VT_CHECK(!out.model.empty(), "video request: `model` must not be empty"); } + if (Has(json, "input_reference")) { + VT_CHECK(json.at("input_reference").is_string(), + "video request: `input_reference` must be a string (a filesystem path " + "or a data: URL)"); + ReadReferenceSource("input_reference", json.at("input_reference").get(), + &out.input_reference_path, &out.input_reference_bytes, + &out.input_reference_media_type); + } + if (Has(json, "metadata")) ReadMetadata(json.at("metadata"), &out); + // The pipeline's own exclusion (minimax_h3_pipeline.cpp:251), enforced HERE so + // it is a 400 with the offending pair named rather than a failed job — and so a + // supplied reference is never silently dropped. + VT_CHECK(!(out.has_input_reference() && + (out.has_input_reference_video() || out.has_input_reference_audio())), + "video request: `input_reference` (fl2va first-frame conditioning) cannot " + "be combined with `metadata.input_reference_video`/" + "`metadata.input_reference_audio` (ref2va reference blocks); keyframe and " + "reference conditioning are exclusive"); // vLLM-Omni carries the generation knobs under `extra_params`; accept them at // the top level too so a plain client does not have to nest. const nlohmann::json& extra = diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 023e40be0..065bcb354 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -1808,13 +1808,15 @@ TEST_CASE("api_server: the OpenAI request aliases reach the runner unchanged") { // the field default, so this proves the parser ran end to end. ApiServer::DispatchResult r = h.server.handle_videos_sync(R"({ "model": "test-model", "prompt": "a cat on a skateboard", - "size": "1280x720", "seconds": "8" + "size": "1280x720", "seconds": "8", "input_reference": "/tmp/frame0.ppm" })"); REQUIRE(r.status == 200); CHECK(seen.prompt == "a cat on a skateboard"); CHECK(seen.width == 1280); CHECK(seen.height == 720); CHECK(seen.duration_seconds == doctest::Approx(8.0)); + CHECK(seen.input_reference_path == "/tmp/frame0.ppm"); + CHECK(seen.has_input_reference()); CHECK(seen.model == "test-model"); // The served model was named, so nothing is warned about. nlohmann::json body = nlohmann::json::parse(r.body); @@ -1822,6 +1824,61 @@ TEST_CASE("api_server: the OpenAI request aliases reach the runner unchanged") { CHECK_FALSE(body.contains("warning")); } +TEST_CASE("api_server: every reference modality reaches the runner, or is refused") { + // H3 has three reference modalities and OpenAI's schema carries one, so the + // other two ride in `metadata`. What matters is that each one ARRIVES at the + // runner (the library gates already prove a reference changes the output); + // a reference that parsed and then never reached the pipeline is the failure + // that looks like it worked. + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + vllm::openai::VideoRequest seen; + int calls = 0; + h.server.set_video_runner( + [&](const vllm::openai::VideoRequest& req) -> std::string { + seen = req; + ++calls; + return "/tmp/out.mp4"; + }); + + SUBCASE("the metadata video + audio references arrive together") { + REQUIRE(h.server + .handle_videos_sync(R"({"prompt":"x","metadata":{ + "input_reference_video":"/tmp/prev_job", + "input_reference_audio":"/tmp/voice.wav", + "trace_id":"abc-123"}})") + .status == 200); + CHECK(calls == 1); + CHECK(seen.input_reference_video_dir == "/tmp/prev_job"); + CHECK(seen.input_reference_audio_path == "/tmp/voice.wav"); + CHECK(seen.metadata.at("trace_id") == "abc-123"); // free-form keys pass through + CHECK_FALSE(seen.has_input_reference()); + } + + SUBCASE("the audio reference arrives alone, as inline bytes") { + REQUIRE(h.server + .handle_videos_sync( + R"({"prompt":"x","metadata":{"input_reference_audio":"data:audio/wav;base64,aGk="}})") + .status == 200); + CHECK(calls == 1); + REQUIRE(seen.input_reference_audio_bytes.size() == 2); + CHECK(seen.input_reference_audio_bytes[0] == 'h'); + CHECK_FALSE(seen.has_input_reference_video()); + } + + SUBCASE("fl2va + ref2va is a 400 and the runner is never reached") { + ApiServer::DispatchResult r = h.server.handle_videos_sync( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_audio":"/tmp/a.wav"}})"); + CHECK(r.status == 400); + CHECK(calls == 0); // nothing generated from a half-honoured request + const std::string message = + nlohmann::json::parse(r.body).at("error").at("message").get(); + CHECK(message.find("exclusive") != std::string::npos); + } +} + TEST_CASE("api_server: an unserved `model` warns on the job but still generates") { const HfConfig c = MakeConfig(); const Qwen3_5MoeWeights w = MakeWeights(c); diff --git a/tests/vllm/entrypoints/openai/test_video_api.cpp b/tests/vllm/entrypoints/openai/test_video_api.cpp index cc9f076de..a232b959a 100644 --- a/tests/vllm/entrypoints/openai/test_video_api.cpp +++ b/tests/vllm/entrypoints/openai/test_video_api.cpp @@ -63,7 +63,7 @@ TEST_CASE("video api: request parsing applies H3 defaults and rejects bad input" // --------------------------------------------------------------------------- // OpenAI's Sora video shape (developers.openai.com/api/docs/guides/video-generation): -// {model, prompt, size:"x", seconds}. These are ALIASES +// {model, prompt, size:"x", seconds, input_reference}. These are ALIASES // onto the native fields, so an OpenAI client works unmodified while every // existing body keeps its exact meaning. // @@ -168,6 +168,114 @@ TEST_CASE("video api: `model` is recorded, never a parse failure") { CHECK(ParseVideoRequest(R"({"prompt":"x"})").model.empty()); } +TEST_CASE("video api: `input_reference` accepts a path or a data: URL") { + const VideoRequest path = + ParseVideoRequest(R"({"prompt":"x","input_reference":"/tmp/first.ppm"})"); + CHECK(path.input_reference_path == "/tmp/first.ppm"); + CHECK(path.input_reference_bytes.empty()); + CHECK(path.has_input_reference()); + + // "hi" base64-encoded, with the media type preserved for the runner. + const VideoRequest inline_data = ParseVideoRequest( + R"({"prompt":"x","input_reference":"data:image/x-portable-pixmap;base64,aGk="})"); + CHECK(inline_data.input_reference_path.empty()); + CHECK(inline_data.input_reference_media_type == "image/x-portable-pixmap"); + REQUIRE(inline_data.input_reference_bytes.size() == 2); + CHECK(inline_data.input_reference_bytes[0] == 'h'); + CHECK(inline_data.input_reference_bytes[1] == 'i'); + + CHECK_FALSE(ParseVideoRequest(R"({"prompt":"x"})").has_input_reference()); + + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":""})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":5})")); + // A broken data: URL must be a specific 400, not a path named "data:...". + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"data:image/png;base64,@@@"})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"data:image/png,raw"})")); + // An http(s) URL is refused BY NAME rather than treated as a filesystem path. + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"https://x/i.png"})")); +} + +// --------------------------------------------------------------------------- +// The two reference modalities OpenAI has no slot for. H3 supports THREE (image, +// silent video, audio) and the Sora schema carries one, so the other two ride in +// `metadata` — a standard OpenAI free-form string map, which strict clients +// tolerate, rather than invented top-level fields that would fail validation. +// --------------------------------------------------------------------------- + +TEST_CASE("video api: `metadata` carries the video and audio references") { + const VideoRequest r = ParseVideoRequest(R"({ + "prompt":"x", + "metadata":{"input_reference_video":"/tmp/prev_job", + "input_reference_audio":"/tmp/voice.wav", + "trace_id":"abc-123"} + })"); + // A DIRECTORY of frame_%06d.ppm — the layout minimax-h3-gen writes, so clips chain. + CHECK(r.input_reference_video_dir == "/tmp/prev_job"); + CHECK(r.has_input_reference_video()); + CHECK(r.input_reference_audio_path == "/tmp/voice.wav"); + CHECK(r.has_input_reference_audio()); + CHECK_FALSE(r.has_input_reference()); // the image slot stayed empty + // Free-form means free-form: keys we do not act on survive untouched. + CHECK(r.metadata.at("trace_id") == "abc-123"); + CHECK(r.metadata.size() == 3); + + // The audio reference takes an inline data: URL too ("hi" base64). + const VideoRequest inline_audio = ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_audio":"data:audio/wav;base64,aGk="}})"); + CHECK(inline_audio.input_reference_audio_path.empty()); + REQUIRE(inline_audio.input_reference_audio_bytes.size() == 2); + CHECK(inline_audio.has_input_reference_audio()); + + // Absent metadata leaves every reference empty. + const VideoRequest none = ParseVideoRequest(R"({"prompt":"x"})"); + CHECK(none.metadata.empty()); + CHECK_FALSE(none.has_input_reference_video()); + CHECK_FALSE(none.has_input_reference_audio()); + + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":"nope"})")); // not an object + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"k":7}})")); // not a string map + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"input_reference_video":""}})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"input_reference_audio":""}})")); + // A directory cannot be a data: URL, and saying so beats a confusing open() later. + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_video":"data:video/mp4;base64,aGk="}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_audio":"https://x/a.wav"}})")); +} + +TEST_CASE("video api: only the pipeline's legal reference combinations are accepted") { + // The rule is minimax_h3_pipeline.cpp:251 — fl2va keyframes and ref2va + // reference blocks are EXCLUSIVE. `input_reference` is fl2va; the metadata + // references are ref2va blocks. + auto parses = [](const char* body) { + ParseVideoRequest(body); + return true; + }; + + // LEGAL: each alone, and video+audio together (one kVideoAudio block). + CHECK(parses(R"({"prompt":"x","input_reference":"/tmp/f0.ppm"})")); + CHECK(parses(R"({"prompt":"x","metadata":{"input_reference_video":"/tmp/clip"}})")); + CHECK(parses(R"({"prompt":"x","metadata":{"input_reference_audio":"/tmp/a.wav"}})")); + const VideoRequest both = ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_video":"/tmp/clip", + "input_reference_audio":"/tmp/a.wav"}})"); + CHECK(both.has_input_reference_video()); + CHECK(both.has_input_reference_audio()); + + // ILLEGAL: fl2va + ref2va. REJECTED, never silently dropped — a dropped + // reference is the failure that looks like it worked. + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_video":"/tmp/clip"}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_audio":"/tmp/a.wav"}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_video":"/tmp/clip", + "input_reference_audio":"/tmp/a.wav"}})")); +} + TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields") { // The exact shape an unmodified OpenAI client sends, plus our native knobs // alongside: this is ADDITIVE compatibility, not a replacement. @@ -176,7 +284,8 @@ TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields "prompt": "a cat on a skateboard", "size": "1280x720", "seconds": "8", - "task": "t2va", + "input_reference": "/tmp/frame0.ppm", + "task": "fl2va", "extra_params": {"num_frames": 97, "num_inference_steps": 30, "flow_shift": 9.5, "audio_flow_shift": 2.5, "seed": 4242} })"); @@ -185,7 +294,8 @@ TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields CHECK(r.width == 1280); CHECK(r.height == 720); CHECK(r.duration_seconds == doctest::Approx(8.0)); - CHECK(r.task == "t2va"); + CHECK(r.input_reference_path == "/tmp/frame0.ppm"); + CHECK(r.task == "fl2va"); CHECK(r.num_frames == 97); CHECK(r.num_inference_steps == 30); // default 50 CHECK(r.flow_shift == doctest::Approx(9.5)); // default 12.0