diff --git a/.agents/NOW.md b/.agents/NOW.md index 576dbf46..5dc3d19e 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -25,7 +25,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): async mirror on classic dense Qwen3; SACRED 184/184 | Residual: sibling scope one-liner | | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | | Supported-models list (`row/DOCS-SUPPORTED-MODELS-MATRIX`) | **DRAFT PR**: FEATURES per-arch table CI-bound to registry (30 archs) | Reviewer merge | -| `/v1/videos` OpenAI shape (`row/SERVE-VIDEOS-OAI`) | **PR open**: Sora `model`/`size`/`seconds` + `GET /{id}/content`, CPU-gated | Follow-up row: reference conditioning | +| `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna diff --git a/.agents/coordination.md b/.agents/coordination.md index 4a2ef9b0..5630a8b5 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1629,7 +1629,8 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-ROCM-W0-1` | `BACKEND-ROCM` (INVENTORIED→ACTIVE) | Claude Code (opus-5) | worktree `/home/mudler/_git/vllm.cpp-rocm-w0`, base `main` `4cfeee13`; CPU-only Release build, NO GPU of any kind | branches `row/backend-rocm-seam` (PR #87) + `row/backend-rocm-w0` (PR #88, stacked) | Owns ONLY the ROCm skeleton and the platform-SELECTION gate it uncovered: NEW `include/vt/rocm/{rocm_arch,rocm_runtime}.h`, NEW `src/vt/rocm/{rocm_backend,rocm_rmsnorm,rocm_ops}.hip`, NEW `src/vllm/platforms/rocm.cpp`, NEW `tests/vt/{test_rocm_arch,test_rocm_backend}.cpp`, NEW `.agents/specs/rocm-backend-w0.md`; ADDITIVE edits: `include/vt/device.h` (`kROCM`), `include/vllm/platforms/interface.h` + `src/vllm/platforms/platform.cpp` (`CurrentPlatformPriority()` + kROCM in the walk), `tests/vllm/platforms/test_platform.cpp`, `tests/vt/test_backend_cross_device.cpp` (kROCM in the discovery list), `CMakeLists.txt`/`tests/CMakeLists.txt`, this row + the `BACKEND-ROCM` matrix row + README/BUILD/ROCM/STATUS/FEATURES/BENCHMARKS + NOW/state. **NON-COLLISION:** every new file is ROCm-only; the only shared-code edits are one enum entry and one selection-array entry, neither of which changes behaviour for any registered backend (nothing registers kROCM in a non-HIP build). | `ACTIVE` | 2026-08-06 — **W0 LANDED, CPU-GATED, HIP UNBUILT.** The three `.hip` TUs have never been compiled by anyone (no AMD hardware here) — weaker than `build-supported`, and labelled UNBUILT everywhere. Verified: clean `-Werror` build, `ctest` full suite, `test_rocm_arch` 40/40 (upstream's own worked examples), `test_platform` incl. the new walk gate, device-leakage unchanged at 32; the two non-HIP legs are object-compiled in every build as a bit-rot guard. PENDING (hardware): every HIP compile, `test_rocm_backend`, the cross-device RmsNorm-vs-CPU-oracle comparison, and both `BACKEND-GATE-ROCM-*` rows. Next event is a contributor's first `-DVLLM_CPP_HIP=ON` on #41. | | `CLAIM-KERNEL-CPU-ELEM-WIDE-1` | `KERNEL-GEMM-CPU-ELEM-X86WIDE` + `KERNEL-GEMM-CPU-TILED` | Claude Code (opus-5) | worktree `/home/mudler/_git/vllm.cpp-x86wide`, base `cbc56f12` | branch `row/KERNEL-GEMM-CPU-ELEM-X86WIDE` | see ownership above | `SPIKE` | 2026-08-06 spike committed; W4 (widen the test tier sweep) is the first implementation step, deliberately before any kernel | -| `CLAIM-SERVE-VIDEOS-OAI` | `SERVE-VIDEOS-OAI` (NEW engine-matrix row; the MiniMax-H3 model rows and the `row/H3-FP4-SPEED` fp4 lane are UNTOUCHED) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-OAI`, base `main` `add5e9f9` | `/v1/videos` in OpenAI's Sora WIRE SHAPE (`model`, `size`, `seconds`) + `GET /v1/videos/{id}/content`. Owns `{include,src}/vllm/entrypoints/openai/video_api.*`, the video handlers/routes in `api_server.{h,cpp}`, both video test files, `specs/minimax-h3.md` §9 and the record surfaces. ADDITIVE: no generation kernel, no model forward, no VAE and no non-video route touched; the routes still register only with a `VideoRunner`. SPLIT: reference conditioning is a stacked follow-up row (spec §10), which is why this one carries no `examples/server/main.cpp` change. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 11/11-125, `test_openai_api_server` 40/40-509, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | +| `CLAIM-SERVE-VIDEOS-OAI` | `SERVE-VIDEOS-OAI` (NEW engine-matrix row; the MiniMax-H3 model rows and the `row/H3-FP4-SPEED` fp4 lane are UNTOUCHED) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-OAI`, base `main` `add5e9f9` | `/v1/videos` in OpenAI's Sora WIRE SHAPE (`model`, `size`, `seconds`) + `GET /v1/videos/{id}/content`. Owns `{include,src}/vllm/entrypoints/openai/video_api.*`, the video handlers/routes in `api_server.{h,cpp}`, both video test files, `specs/minimax-h3.md` §9 and the record surfaces. ADDITIVE: no generation kernel, no model forward, no VAE and no non-video route touched; the routes still register only with a `VideoRunner`. SPLIT: reference conditioning is the stacked row `SERVE-VIDEOS-REFS` (spec §10), which is why this one carries no `examples/server/main.cpp` change. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 11/11-125, `test_openai_api_server` 40/40-509, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | +| `CLAIM-SERVE-VIDEOS-REFS` | `SERVE-VIDEOS-REFS` (NEW engine-matrix row, stacked on `SERVE-VIDEOS-OAI`) | Claude Opus 5 (1M context) sub-agent | isolated worktree `/home/mudler/_git/vllm.cpp/.claude/worktrees/agent-a152dd219c723cb12` (CPU-only; no GPU, no download; foreground) | branch `row/SERVE-VIDEOS-REFS`, base `row/SERVE-VIDEOS-OAI` | Reference conditioning over `/v1/videos`: `input_reference` -> fl2va first-frame conditioning, the `metadata` video/audio references -> ref2va blocks, the exclusivity rule at the request boundary, and the `examples/server` runner wiring with lazily-loaded VAE encoder halves. Owns the reference members of `{include,src}/vllm/entrypoints/openai/video_api.*`, the video runner in `examples/server/main.cpp`, both video test files, `specs/minimax-h3.md` §10 and the record surfaces. ADDITIVE: no generation kernel and no model forward touched. | `ACTIVE` | 2026-08-06 CPU-LANDED + gated (`test_video_api` 14/14-167, `test_openai_api_server` 41/41-525, `server` -Werror clean). Real-weights leg rides the H3 GB10/disk window. | ## Handoff queue diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 3c217a49..36d0b8d4 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -42,11 +42,11 @@ forensics: roadmap_v1.md and the parity ledger. | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 3 | 0 | 1 | 5 | | Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 | | Speculative decoding | 21 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 11 | -| Serving, API, CLI, library | 23 | 10 | 2 | 1 | 0 | 3 | 2 | 1 | 4 | +| Serving, API, CLI, library | 24 | 10 | 2 | 1 | 0 | 4 | 2 | 1 | 4 | | LoRA and adapters | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 9 | 3 | 3 | 0 | 0 | 0 | 1 | 1 | 1 | -| **Total** | **141** | **36** | **16** | **4** | **7** | **19** | **8** | **9** | **41** | +| **Total** | **142** | **36** | **16** | **4** | **7** | **20** | **8** | **9** | **41** | ## Engine core and scheduling @@ -214,6 +214,7 @@ claims it. | `SERVE-RESPONSES-MESSAGES` | Responses, Anthropic messages, audio | T2 | `vllm/entrypoints/openai/responses/api_router.py:48`; `vllm/entrypoints/anthropic/api_router.py:49`; `vllm/entrypoints/speech_to_text/transcription/api_router.py:1` | - | - | `planned: specs/responses-messages-endpoints.md` | `INVENTORIED` | - | | `SERVE-ADMIN` | Abort-requests, sleep, pause/resume, profiling, RL weight updates. **`/abort_requests` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `POST /abort_requests` (from the dev/rlhf admin router) parses `{request_ids:[...]}` and aborts exactly those (external) ids via an injected abort callback wired to the engine abort path (`AsyncLLM::abort`); an empty/missing list means "abort all in-flight" (the callback decides). Response `{"status":"aborted","aborted":}`; malformed JSON → 400 `{"detail":"Invalid JSON format"}`; abort failure → 500 `{"error":...}` — all three shapes mirror the upstream router verbatim. ADDITIVE + opt-in (route registered only when the abort callback is attached → 404 otherwise). **PRODUCTION `main.cpp` WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary now wires `/abort_requests` to the LIVE `AsyncLLM::abort` through the shared `ConfigureUtilityEndpoints` seam, DEV-mode gated behind the new `--enable-server-dev-mode` flag — mirroring vLLM registering the dev/rlhf router only under `if envs.VLLM_SERVER_DEV_MODE` (api_server.py:238; envs.py:157 default 0). Explicit-id abort tears the request down and reports the exact drop in unfinished requests (before−after); empty `request_ids` (abort-ALL) reports 0 — NAMED RESIDUAL (AsyncLLM exposes no active-request-id accessor). RESIDUAL: the abort-ALL enumeration (missing `AsyncLLM::active_request_ids()`); `/sleep`/`/wake_up`/`/is_sleeping`, `/pause`/`/resume`, `/start_profile`/`/stop_profile`, weight-update/EP endpoints still INVENTORIED | T2/T3 | `vllm/entrypoints/serve/dev/rlhf/api_router.py:94-138` (abort_requests); dev-mode gate `vllm/entrypoints/openai/api_server.py:238-240`, `vllm/entrypoints/serve/__init__.py:35`, `vllm/envs.py:157`; `vllm/entrypoints/serve/dev/sleep/api_router.py:21`; `vllm/entrypoints/serve/dev/rlhf/api_router.py:29,74,136`; `vllm/entrypoints/serve/profile/api_router.py:21` | handler `src/vllm/entrypoints/openai/api_server.cpp:488` (`handle_abort_requests`); opt-in setter `include/vllm/entrypoints/openai/api_server.h:156` (`set_abort_requests`); production seam `src/vllm/entrypoints/openai/api_server.cpp` (`ConfigureUtilityEndpoints`, before/after delta-count) + `examples/server/main.cpp` (`--enable-server-dev-mode`); engine abort path `include/vllm/v1/engine/async_llm.h:115` (`abort`) | `tests/vllm/entrypoints/openai/test_api_server.cpp:1104` (shape + callback wiring: explicit ids passthrough, empty→abort-all branch, malformed→400),`:1143` (aborts an in-flight AsyncLLM request → `has_unfinished_requests()` false),`:1250` (opt-in route gate: 404 no-callback → 200 attached, RED-first),`:1319` (**production seam: dev-mode gate 404→200, live abort exact delta-count==1, empty→0**) — in the 32/32 / 420-assertion suite | [admin-endpoints.md](specs/admin-endpoints.md) | `ANCHOR-BACKFILL` | `CLAIM-C8-SERVE-PROD-WIRING` | | `SERVE-VIDEOS-OAI` | `/v1/videos` in OpenAI's Sora WIRE SHAPE, over the vLLM-Omni-derived job endpoints. **CPU-LANDED + GATED 2026-08-06 (`CLAIM-SERVE-VIDEOS-OAI`):** the OpenAI request spellings (`model`, `size` "WxH", `seconds` as a number OR the string enum OpenAI actually types) parse as ALIASES onto the existing native members, NATIVE-wins precedence applied PER-AXIS, both spellings validated either way so a malformed alias is a 400 even when overridden; an unserved `model` is a job `warning` echoed for the job's whole life, never a rejection (a Sora client cannot know the local model's name); and `GET /v1/videos/{id}/content` serves the finished MP4 (404 unknown / 409 unfinished / 500 failed / 500 vanished), without which a caller could start and poll a job but never fetch the result over HTTP. All four routes still register ONLY with a `VideoRunner` attached, now gated over a REAL socket. RESIDUALS (named): OpenAI's status vocabulary/id shape is not mirrored; reference conditioning (`input_reference`, the `metadata` video/audio references) is a stacked follow-up row; the real-weights leg rides the H3 GB10/disk window. | T2 | OpenAI Sora video API (`POST /v1/videos`, `GET /v1/videos/{video_id}/content`); vLLM-Omni `vllm/entrypoints/openai/video/api_router.py` (the async/sync job pair we already mirror) | `include/vllm/entrypoints/openai/video_api.h:31`; `src/vllm/entrypoints/openai/video_api.cpp:98`; `src/vllm/entrypoints/openai/api_server.cpp:279` | `tests/vllm/entrypoints/openai/test_video_api.cpp:64`; `tests/vllm/entrypoints/openai/test_api_server.cpp:1751` | [minimax-h3.md §9](specs/minimax-h3.md) | `ACTIVE` | `CLAIM-SERVE-VIDEOS-OAI` | +| `SERVE-VIDEOS-REFS` | REFERENCE CONDITIONING over `/v1/videos`: the image an OpenAI request starts from, plus the two modalities OpenAI's schema has no slot for. **CPU-LANDED + GATED 2026-08-06 (`CLAIM-SERVE-VIDEOS-REFS`, stacked on `SERVE-VIDEOS-OAI`):** OpenAI's `input_reference` (a filesystem path or an RFC 2397 `data:` URL, decoded by the SAME `DecodeDataUri` the chat multimodal parts use) maps to fl2va FIRST-FRAME conditioning via `MiniMaxH3EncodeKeyframeCondRows`, because OpenAI documents it as the frame the video starts from and ref2va would silently change what the API promises; the silent-video and audio references ride the standard free-form `metadata` map (`input_reference_video`, a DIRECTORY of `frame_%06d.ppm` since no demuxer is vendored; `input_reference_audio`, a 16-bit PCM WAV) and become ref2va blocks, an audio reference ATTACHING to the video block when both are given. fl2va-vs-ref2va exclusivity is enforced in the PARSER, mirroring `minimax_h3_pipeline.cpp:251`, so an illegal pair is a 400 naming it rather than a dropped reference. Both VAE ENCODER halves load lazily and once. RESIDUALS (named): reference images are binary PPM at the output resolution (no PNG/JPEG codec, no resampler vendored); a video reference is a frame directory; OpenAI's real upload is multipart, ours is the JSON spelling. | T2 | OpenAI Sora video API (`input_reference`, `metadata`); exclusivity rule `src/vllm/model_executor/models/minimax_h3_pipeline.cpp:251` | `include/vllm/entrypoints/openai/video_api.h:51`; `src/vllm/entrypoints/openai/video_api.cpp:66`; `examples/server/main.cpp:96` | `tests/vllm/entrypoints/openai/test_video_api.cpp:172`; `tests/vllm/entrypoints/openai/test_api_server.cpp:1827` | [minimax-h3.md §10](specs/minimax-h3.md) | `ACTIVE` | `CLAIM-SERVE-VIDEOS-REFS` | | `SERVE-OTLP` | OpenTelemetry traces | T2 | `vllm/config/observability.py:18,36,128` | - | - | `planned: specs/otlp-tracing.md` | `INVENTORIED` | - | | `SERVE-BATCH-API` | Offline OpenAI Batch API runner — read a JSONL of `BatchRequestInput` (`custom_id`/`method`/`url`/`body`), dispatch each line to the matching serving handler, collect `BatchRequestOutput` rows (custom_id echoed, per-line error isolation), write the response JSONL. **W0 SPIKE + W1 CPU BRICK LANDED + CPU-GATED 2026-07-29 (`CLAIM-BATCH-API`, NOT pushed):** `RunBatch` (RunLine/RunLines/Run) + `RunBatchFile` (local paths) as a pure ORCHESTRATOR over the existing `OpenAIServingChat::create_chat_completion` (the SAME handler `handle_chat_completions` drives — NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map. `/v1/chat/completions` wired; `BatchResponseData`/`BatchRequestOutput` schema + `vllm-`/`vllm-batch-` ids; the `run_request` AllResponse/ErrorResponse/stream branches; the unsupported-endpoint (handler None) + unsupported-url error rows. RECORDED DEVIATION: a malformed input line is ISOLATED into an error row (batch continues) where upstream aborts the job (deviation lives in the library; the abort-on-bad-line CLI exit code is a W2 residual). RESIDUALS (named, spec §Work breakdown): the `vllm run-batch` CLI + `BatchFrontendArgs` (W2); embeddings/score/rerank dispatch (W3, rides `SERVE-POOLING-ENDPOINTS`); audio transcription/translation + media fetch (W4); http(s)/data-URL file I/O, metrics server, overlapped `AsyncLLM` submission (W5) | T2 | `vllm/entrypoints/openai/run_batch.py:148-228` (schema),`:508-570` (run_request/make_error),`:722-777,815-847` (dispatch/run loop); `tests/entrypoints/openai/test_run_batch.py:375,402,432` | `include/vllm/entrypoints/openai/run_batch.h`; `src/vllm/entrypoints/openai/run_batch.cpp:70,84,87,142` | `tests/vllm/entrypoints/openai/test_run_batch.cpp:434` (7 cases / 80 assertions, CPU, RED-first: dropping the custom_id echo fails 9 assertions) | [batch-api.md](specs/batch-api.md) | `ANCHOR-BACKFILL` | `CLAIM-BATCH-API` | diff --git a/.agents/specs/minimax-h3.md b/.agents/specs/minimax-h3.md index 985beef2..e801e504 100644 --- a/.agents/specs/minimax-h3.md +++ b/.agents/specs/minimax-h3.md @@ -787,8 +787,8 @@ before means exactly what it meant before. SPLIT, deliberately: this row is the REQUEST/RESPONSE SHAPE only (`model`, `size`, `seconds`, and the MP4 download route). It touches no generation code and loads no VAE. The REFERENCE CONDITIONING half (`input_reference` -> fl2va, plus -the two `metadata` reference modalities -> ref2va) is a stacked follow-up row -(spec §10), because it is a separate capability that pulls in the VAE +the two `metadata` reference modalities -> ref2va) is §10, row +`SERVE-VIDEOS-REFS`, because it is a separate capability that pulls in the VAE encoder halves and the runner. Each half is independently reviewable and gated. ### 9.0 Spike contract (`SERVE-VIDEOS-OAI`) @@ -801,7 +801,7 @@ encoder halves and the runner. Each half is independently reviewable and gated. | Port map | Request contract -> `include/vllm/entrypoints/openai/video_api.h` (`VideoRequest::model` + `ParseVideoSize`) and `src/vllm/entrypoints/openai/video_api.cpp` (`ParseVideoRequest`, `ReadDuration`, `ParseWholeNumber`). Job record -> `VideoJobStore::Create(model, warning)` + `VideoJobStatusJson`. Download route -> `ApiServer::handle_video_content` + `video_model_warning` + their registration in `src/vllm/entrypoints/openai/api_server.cpp`. | | Tests to port | No upstream test module exists for this surface (OpenAI publishes an API, not tests; vLLM-Omni's video endpoint has no ported test). The contract is gated in-tree instead, extending the existing files: `tests/vllm/entrypoints/openai/test_video_api.cpp` (parsing, precedence, the job record) and `tests/vllm/entrypoints/openai/test_api_server.cpp` (routes, content behaviour, additivity over a real socket). Every assertion uses values that DIFFER from the field default. | | Gates | CPU, foreground: `test_video_api` 11/11 (125 assertions), `test_openai_api_server` 40/40 (509), `server` builds clean. Content route: 404 unknown / 409 unfinished (no bytes leaked) / 500 failed / 500 vanished / 200 byte-exact `video/mp4`. Additivity: with no `VideoRunner`, `POST /v1/videos` is 404 over a real socket with no `ErrorResponse` envelope; with one, it is 200 and the unknown-id 404 IS ours. Commands: `cmake --build build --target test_video_api test_openai_api_server server -j12`. Real-weights e2e rides §8's GB10/disk window. | -| Dependencies | Row IDs: the MiniMax-H3 model rows and `row/H3-FP4-SPEED` (UNTOUCHED - no generation code changed); the stacked reference-conditioning row (§10) builds on this one. No new download, no GPU, no toolchain change for the CPU gate. | +| Dependencies | Row IDs: the MiniMax-H3 model rows and `row/H3-FP4-SPEED` (UNTOUCHED - no generation code changed); `SERVE-VIDEOS-REFS` (§10) stacks on this row. No new download, no GPU, no toolchain change for the CPU gate. | | Work breakdown | (1) alias parsing + precedence + `ParseVideoSize`; (2) `model` recording + the job `warning`; (3) `handle_video_content` + its route; (4) both test files; (5) docs + record. | | Risks/decisions | NATIVE-wins precedence: the only direction that leaves every previously-parsing body meaning what it meant. `model` mismatch WARNS rather than 404s: a Sora client cannot know the local model name, so a rejection would defeat the compatibility; silence would hide it. A 409 (never bytes) on an unfinished job: a partially muxed file would reach the client as a valid-looking, truncated MP4. No vLLM-defined behaviour is reopened. | @@ -833,5 +833,80 @@ failure; a vanished output -> 500, not a 200 with zero bytes. (a 404 with no `ErrorResponse` envelope), with one they serve. - **Residuals, named.** OpenAI's status vocabulary is not mirrored (ours stays queued/running/succeeded/failed, ids `vid_N`, no `object`/`progress`/ - `created_at`); reference conditioning is §10, not this row. + `created_at`); reference conditioning is §10 (`SERVE-VIDEOS-REFS`), not this row. +- **Real-weights leg** rides the same GB10/disk window as §8. + +## 10. W-REFS — reference conditioning over `/v1/videos`, 2026-08-06 + +Row `SERVE-VIDEOS-REFS` (engine matrix, Serving surface), claim +`CLAIM-SERVE-VIDEOS-REFS`, branch `row/SERVE-VIDEOS-REFS`, stacked on §9. + +§9 made an OpenAI client's request PARSE. This row makes its REFERENCES do +something: an image the video starts from, a clip it continues, a voice it +carries. Before it, no reference modality was reachable over HTTP at all. + +### 10.0 Spike contract (`SERVE-VIDEOS-REFS`) + +| Section | Content | +|---|---| +| Scope | IN: OpenAI's `input_reference` mapped to fl2va first-frame conditioning; the two reference modalities OpenAI has no slot for carried in `metadata` (`input_reference_video`, `input_reference_audio`) mapped to ref2va blocks; the fl2va/ref2va combination rule enforced at the request boundary; the `examples/server` runner wiring (PPM decode, frame-directory clip, WAV, lazily-loaded VAE encoder halves). OUT: the ref2va IMAGE modality (reachable via the native `task` + the CLI, deliberately not bound to `input_reference`); any change to generation, the DiT, the VAEs or the muxer; OpenAI's multipart upload. | +| Upstream chain | OpenAI documents `input_reference` as the image the generated video STARTS FROM. The conditioning entry points are ours and already gated: `MiniMaxH3EncodeKeyframeCondRows` (fl2va), `MiniMaxH3EncodeReferenceVideo` / `MiniMaxH3EncodeReferenceAudio` (ref2va), `MiniMaxH3ReadWav`. The exclusivity rule is `src/vllm/model_executor/models/minimax_h3_pipeline.cpp:251`. | +| Our baseline | After §9 the OpenAI wire shape parses, but every reference field is absent: an image-to-video request silently generated from the prompt alone. | +| Port map | Request contract -> `include/vllm/entrypoints/openai/video_api.h` (`input_reference*`, `metadata`, the `has_*` predicates) and `src/vllm/entrypoints/openai/video_api.cpp` (`ReadReferenceSource`, `ReadMetadata`, the combination `VT_CHECK`); the `data:` decode REUSES `entrypoints::openai::DecodeDataUri` (chat_mm) rather than a second decoder. Reference wiring (the process boundary keeps it out of the library) -> `examples/server/main.cpp`: `DecodePpmChw`, `ReadReferenceClipChw` (the CLI's `DIR/frame_%06d.ppm` convention), `ReadReferenceBytes`, and the fl2va / ref2va branches with lazily-loaded VAE encoder halves. | +| Tests to port | No upstream test module exists for this surface. The contract is gated in-tree, extending the same two files: `test_video_api.cpp` (reference parsing, `metadata` passthrough, combination legality) and `test_api_server.cpp` (each modality ARRIVES at the runner, and an illegal pair is a 400 that generates nothing). | +| Gates | CPU, foreground: `test_video_api` 14/14 (167 assertions), `test_openai_api_server` 41/41 (525), `server` builds clean. Commands: `cmake --build build --target test_video_api test_openai_api_server server -j12`. Real-weights e2e rides §8's GB10/disk window. | +| Dependencies | Row `SERVE-VIDEOS-OAI` (§9), stacked. Code: `MiniMaxH3Encode{KeyframeCondRows,ReferenceVideo,ReferenceAudio}`, `MiniMaxH3ReadWav`, `DecodeDataUri`. Runtime: `--video-vae` for an image or video reference, `--audio-vae` for an audio reference (both encoder halves, loaded lazily and once). No new download, no GPU. | +| Work breakdown | (1) `input_reference` parsing (path or `data:` URL) -> fl2va, with the geometry refusal; (2) the `metadata` map + the video/audio reference keys; (3) the combination rule in the parser; (4) the `examples/server` runner branches; (5) both test files; (6) docs + record. | +| Risks/decisions | `input_reference` -> fl2va, NOT ref2va: OpenAI documents it as the frame the video starts from; ref2va would silently change what the API promises. The two extra modalities go in `metadata` rather than new top-level fields, so a strict client's schema validation still passes. Combination legality is enforced in the PARSER, not left to the pipeline, so a supplied reference is never silently dropped. | + +### 10.1 `input_reference` is fl2va, not ref2va + +OpenAI documents it as the image the video STARTS FROM, which is what +`MiniMaxH3EncodeKeyframeCondRows` expresses (frame 0 of the output pinned to the +image, `imgvid_noise_aug = 1.0`). `MiniMaxH3EncodeReferenceImages` prepends whole +reference images as their own blocks — guidance that never becomes a frame — so +mapping it there would have changed what the API promises. With a reference image +and no explicit `task`, the task IS `fl2va`, and the image's aspect drives the +default resolution through `MiniMaxH3ResolveShape`. + +Two limits, both refused up front rather than deep in the denoise: the image must +be a binary PPM (P6), because no PNG/JPEG codec is vendored (the same NAMED +residual the chat multimodal path carries), and it must already be at the +resolved output geometry, because no image resampler is vendored. The refusal +names both geometries. + +### 10.2 The two reference modalities OpenAI has no slot for + +H3 has three (image, silent video, audio); the Sora schema carries one. The other +two enter through `metadata`, the standard OpenAI free-form string map that +strict clients tolerate, rather than invented top-level fields that would fail a +client's schema validation. The whole map is kept verbatim. + +- `metadata.input_reference_video` — a DIRECTORY of `frame_%06d.ppm`, the exact + layout `minimax-h3-gen` and the server WRITE, so clips chain. No demuxer is + vendored, hence a frame directory rather than a container; a `data:` URL cannot + name a directory and is refused by name. + `MiniMaxH3EncodeReferenceVideo` emits `ref_audio_t == 0`: the clip is SILENT. +- `metadata.input_reference_audio` — a 16-bit PCM WAV path or `data:` URL, read + by the existing `MiniMaxH3ReadWav` and encoded by + `MiniMaxH3EncodeReferenceAudio`. Supplied with a video reference it ATTACHES to + that block (one `kVideoAudio` block carrying both, the layout + `packed_sequence.py` builds); alone it is its own block. + +LEGALITY is the pipeline's own rule (`minimax_h3_pipeline.cpp:251`: fl2va +keyframes and ref2va blocks are exclusive), enforced in the PARSER so it is a 400 +naming the pair rather than a failed job — and never a silently dropped +reference, which is the failure that looks like it worked. Legal: none / image / +video / audio / video+audio. Illegal: `input_reference` with either metadata +reference. + +### 10.3 Status + +- **CPU-LANDED + gated.** `test_video_api` 14/14 (167 assertions), + `test_openai_api_server` 41/41 (525), `server` builds clean. Each modality is + gated as ARRIVING at the runner, and an illegal pair is a 400 that generates + nothing (`calls == 0`). +- **Residuals, named.** Reference images are binary PPM at the output resolution; + a video reference is a frame DIRECTORY; OpenAI's real `input_reference` upload + is multipart, ours is the JSON spelling. - **Real-weights leg** rides the same GB10/disk window as §8. diff --git a/.agents/state.md b/.agents/state.md index 5766f6b6..009c8ec1 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -40501,3 +40501,79 @@ The fix reads the SECOND parent, which is the PR head: a merge of a branch whose own commits name the row IS arrival through a row PR, one hop away. NOT a weakening, and gated as such: a new test asserts that a merge naming no row anywhere (and a plain local `Merge branch 'wip'`) STILL fails. Suite 40/40. + +## 2026-08-07T04:40 - reference CONDITIONING over `/v1/videos`: `input_reference` -> fl2va, `metadata` -> ref2va + + + +Row `SERVE-VIDEOS-REFS`, claim `CLAIM-SERVE-VIDEOS-REFS`, branch +`row/SERVE-VIDEOS-REFS`, STACKED on `SERVE-VIDEOS-OAI`. That row made an OpenAI +client's body PARSE; this one makes its REFERENCES do something. Before it, no +reference modality was reachable over HTTP at all: an image-to-video request +generated silently from the prompt alone. + +**`input_reference` -> fl2va, NOT ref2va.** OpenAI documents it as the image the +generated video STARTS FROM, which is exactly what fl2va expresses: +`MiniMaxH3EncodeKeyframeCondRows` pins frame 0 OF THE OUTPUT to the supplied +image (`imgvid_noise_aug = 1.0`). `MiniMaxH3EncodeReferenceImages` means +something else entirely: whole reference images PREPENDED as their own blocks, +i.e. subject or style guidance that never becomes a frame of the result. Mapping +`input_reference` there would have silently changed what the API promises. With a +reference and no explicit `task`, the task IS fl2va, and the image's aspect +drives the default geometry through `MiniMaxH3ResolveShape`. The source is a +filesystem path OR an RFC 2397 `data:` URL, decoded by the SAME +`entrypoints::openai::DecodeDataUri` the chat multimodal parts use rather than a +second, subtly different decoder; an http(s) URL is REFUSED BY NAME rather than +stat-ed as a path and failing later with a confusing message. + +**The two modalities OpenAI has no slot for ride `metadata`.** H3 supports three +(image, silent video, audio) and the Sora schema carries one, so the other two +enter through the standard free-form string map every strict client already +tolerates, rather than invented top-level fields that would fail a client's +schema validation. The whole map is kept verbatim (unknown keys pass through +untouched); two keys are lifted into typed fields. +`metadata.input_reference_video` is a DIRECTORY of `frame_%06d.ppm` -- the exact +layout `minimax-h3-gen` and this server WRITE, so one run's frames chain straight +into the next request -- because no container demuxer is vendored; a `data:` URL +cannot name a directory and is refused saying so. +`metadata.input_reference_audio` is a 16-bit PCM WAV path or `data:` URL. + +**A video reference is SILENT, by construction and stated.** +`MiniMaxH3EncodeReferenceVideo` emits `ref_audio_t == 0`, because the clip's own +soundtrack would need the audio VAE encoder run over it. An audio reference +supplied alongside ATTACHES to that same `kVideoAudio` block (its `ref_audio_t` +claims exactly the rows just encoded), which is the layout `packed_sequence.py` +builds; alone it is its own block. + +**Legality is enforced in the PARSER, not left to the pipeline.** The rule is the +pipeline's own (`minimax_h3_pipeline.cpp:251`: fl2va keyframes and ref2va blocks +are EXCLUSIVE), so an illegal pair is a 400 naming it rather than a failed job -- +and never a silently dropped reference, which is the failure that looks like it +worked. Legal: none / image / video / audio / video+audio. Illegal: +`input_reference` with either metadata reference. + +**Two refusals up front rather than deep in the denoise.** A reference image must +be a binary PPM (P6): no PNG/JPEG codec is vendored, the same NAMED residual the +chat multimodal path carries. And it must already be at the RESOLVED output +geometry, because no image resampler is vendored -- a mis-sized keyframe would +either abort inside the denoise or pin the wrong latent rows, so the refusal +names both geometries. Both VAE ENCODER halves (video and audio) load LAZILY and +ONCE each under one mutex: a text-to-video server must not pay for weights it +never uses, and a server that does use them must not reload per request. + +**GATE (CPU, foreground).** `test_video_api` 14/14 (167 assertions), +`test_openai_api_server` 41/41 (525), `server` builds clean. Each modality is +gated as ARRIVING at the runner (a reference that parsed and then never reached +the pipeline is the failure that looks like it worked), and an illegal pair is a +400 that generates nothing (`calls == 0`). + +**Records.** spec §10, engine-matrix `SERVE-VIDEOS-REFS` (+1 row, +`check-agent-record.py` ENGINE_ROWS 141 -> 142), coordination claim, USAGE +(reference fields, the `metadata` block, the legality table), FEATURES/BENCHMARKS +/STATUS rows updated IN PLACE (keyed tables), STATUS ratchet lowered +284043 -> 284037 in the same change, NOW. + +**Residuals, named.** Reference images are binary PPM at the output resolution; a +video reference is a frame DIRECTORY; OpenAI's real `input_reference` upload is +multipart, ours is the JSON spelling; the real-weights leg rides the H3 GB10/disk +window. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 96e02902..3c214c75 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | DeepSeek-V4-Flash vs vLLM | Infeasible on one Spark | 2x GB10 with TP2 over the NCCL seam | | DFlash speculative decode | **CLOSED 2026-07-27 (D14)**: warp-scoped draft attention (242.9 → 77.9 ms), c1 our-on 29.32 vs vLLM-on 29.24 tok/s, non-overlapping 3-rep bands, 1.003x | none, closed | | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | -| `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (the `model`/`size`/`seconds` request aliases + the MP4 content route), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | +| `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d993ea7f..c23c1c10 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -227,7 +227,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Plugin / out-of-tree model registration | ✅ in-tree factory `DONE` + plugin seam | ✅ | ◐ | ☐ | | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ | | Embedding / pooling endpoints | ◐ engine only | ✅ | ✅ | ✅ | -| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases, native fields win, + `GET /{id}/content` | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | +| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ | ## Parallelism and scale-out diff --git a/docs/STATUS.md b/docs/STATUS.md index 0add0aee..5aa4d18b 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -111,7 +111,7 @@ token-for-token correctness against the pinned oracle. | Tool-call parsing | 36 parser families / 40 accepted names, streaming | Every vLLM tool parser at the pin except the three Rust/Harmony-backed ones: pure-text parsers ported 1:1, the six engine-backed families reimplemented from their wire formats, all held to the upstream test suites. Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md | | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | | Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md | -| OpenAI server | Supported (subset) | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | +| OpenAI server | Supported (subset) | `/v1/completions`, `/v1/chat/completions`, streaming SSE, `/v1/models`, `/health`, `/version`, `/ping`, `/metrics`, `/tokenize` (raw-`prompt` and chat-`messages`), `/detokenize`, `/tokenizer_info`, `/server_info`, `/reset_prefix_cache`, `/abort_requests`; `/v1/videos` in OpenAI's Sora shape + `GET /v1/videos/{id}/content`, conditioned on an `input_reference` image or the `metadata` video/audio references. `/tokenizer_info` and `/abort_requests` are flag-gated; `/metrics` and `/reset_prefix_cache` have handlers but no live backing on the async path. Endpoint list and flags: docs/USAGE.md. Depth-2 async serving no longer corrupts the host heap under `ignore_eos` | | Pooling task class (embeddings / classify / score / rerank) | Spiked; pooler op + heads composite + pooling runner path landed (CPU), not yet servable end-to-end (no concrete model / endpoints) | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** — `EmbeddingPoolerHead` (projector→matryoshka→normalize), `ClassifierPoolerHead` (classifier→`(logit-mean)/sigma`→activation), the `SequencePooler` + `PoolerForEmbed`/`PoolerForClassify` factories, the `DispatchPooler` task routing, and the `PoolerConfig`/`PoolingParams` structs (`test_pooler_heads` 27/27, 240 asserts, RED-first). **W3 landed the pooling RUNNER path** — `PoolingRunner` applies the model's `Pooler` to the last hidden state and returns pooled embeddings instead of sampled tokens, gated by a STRUCTURAL cosine-parity check vs a double-precision LAST+normalize reference (`test_pooling_runner` 5/5, 14 asserts, RED-first). NOT yet servable / honest residuals: a concrete pooling MODEL forward + the REAL-model oracle cosine gate (`vllm.LLM(task="embed").encode`) — no cosine-vs-oracle number is fabricated (W3-model); the `/v1/embeddings` + score/rerank/classify endpoints (W4); tokwise AllPool/StepPool (W5). See docs/BENCHMARKS.md | | Plugin system (out-of-core registration) | Spiked; first CPU brick landed, not yet wired into any production path | The extensibility-first discovery layer. W0 spike over vLLM's plugin surface (general / platform / io_processor / endpoint groups, the `register_model` an out-of-tree plugin calls, the invocation seams) is committed (`.agents/specs/plugin-system.md`, `ENG-PLUGIN-SYSTEM` ACTIVE, `CLAIM-PLUGIN-SYSTEM`). W1 landed `vllm::plugins::LoadGeneralPlugins()` + the out-of-core general-plugin registration seam (`RegisterGeneralPlugin` / `REGISTER_VLLM_GENERAL_PLUGIN`) over the existing `REGISTER_VLLM_MODEL`-style registries (the in-tree factory `MODEL-FACTORY-registry` is record-repaired `DONE` 2026-08-05: 28 self-registering TUs, dgx debt paid by the 2026-07-23 seven-gate run): a 1:1 mirror of `load_general_plugins` (load-once idempotence, the `VLLM_PLUGINS` allowlist, per-plugin failure isolation). Proven by an out-of-core toy-model plugin that registers a toy architecture through the public `RegisterModel` seam — unit-gated RED-first (`test_plugin_system` 1 case / 29 assertions: the toy arch resolves ONLY after LoadGeneralPlugins runs it, and not under `VLLM_PLUGINS=""`). Python entry points have no C++20 analogue, so discovery is the project's static-init/`dlopen` registration idiom (recorded porting-inventory §9). NOT yet wired: real shared-object `dlopen` + the C-ABI `vllm_plugin_register` entry (W2), the engine/CLI `--load-plugins` wiring that calls LoadGeneralPlugins from the construction paths (W3), the platform/quant plugin kinds (W4), and the io_processor/stat_logger/endpoint groups (W5) are named residuals. See docs/BENCHMARKS.md | | Offline Batch API (JSONL file runner) | Spiked; first CPU brick landed, not yet exposed as a CLI | The offline OpenAI Batch API: read a JSONL of OpenAI-format requests, run each through the engine, write a JSONL of responses. W0 spike over vLLM's `run_batch.py` (schema, endpoint dispatch, run loop, file I/O) is committed (`.agents/specs/batch-api.md`, `SERVE-BATCH-API` ACTIVE, `CLAIM-BATCH-API`). W1 landed `RunBatch` (`RunLine`/`RunLines`/`Run`) + `RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map: `/v1/chat/completions` dispatch, the `BatchResponseData`/`BatchRequestOutput` schema (`vllm-` ids, custom_id echoed), the `run_request` AllResponse/ErrorResponse/stream branches, and the unsupported-endpoint/url error rows. Unit-gated RED-first (`test_openai_run_batch` 7 cases / 80 assertions over the synthetic serving engine: ordered rows + custom_id echo + per-line BatchRequestOutput schema round-trip, a malformed line isolated into an error row so the batch continues, dispatch + 404 error rows; dropping the custom_id echo fails 9 assertions). Recorded deviation: a malformed line is isolated (batch continues) where upstream aborts the job. NOT yet exposed: the `vllm run-batch` CLI + `BatchFrontendArgs` (W2), embeddings/score/rerank dispatch (W3, rides pooling endpoints), audio transcription/translation + media fetch (W4), and http(s)/data-URL file I/O + metrics + overlapped `AsyncLLM` submission (W5) are named residuals. See docs/BENCHMARKS.md | diff --git a/docs/USAGE.md b/docs/USAGE.md index 100a076a..0647e437 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -148,9 +148,11 @@ open("out.mp4", "wb").write(client.videos.download_content(video.id).read()) | `model` | OpenAI | Recorded and echoed back. A name this server does not serve is a `warning` on the job, never a rejection: the video model is chosen at startup | | `size` | OpenAI | `"x"`, e.g. `"1280x720"`. Whole pixels, both positive | | `seconds` | OpenAI | Duration, as a number or a numeric string (`8` and `"8"` both work) | +| `input_reference` | OpenAI | The image the video starts from. A filesystem path or a `data:` URL | +| `metadata` | OpenAI | Free-form string map, passed through untouched. Two keys are acted on: `input_reference_video` and `input_reference_audio` (see below) | | `width`, `height` | native | Output geometry in pixels | | `duration` | native | Duration in seconds | -| `task` | native | `t2va`, `fl2va`, `ref2va`; defaults to `t2va` | +| `task` | native | `t2va`, `fl2va`, `ref2va`; resolved from the inputs when omitted | | `num_frames`, `num_inference_steps`, `flow_shift`, `audio_flow_shift`, `seed` | native | The H3 generation knobs. Accepted at the top level or nested under `extra_params` | **Precedence.** When a body carries both spellings of one value, the **native @@ -159,6 +161,63 @@ direction keeps every request that parses today meaning exactly what it meant before. Both spellings are validated either way, so a malformed `size` is a 400 even when explicit `width`/`height` would have overridden it. +**`input_reference` maps to fl2va first-frame conditioning.** OpenAI documents +it as the image the generated video starts from, which is what fl2va expresses: +the supplied image is pinned as frame 0 of the output. H3's other image mode, +ref2va, prepends whole reference images as their own blocks (subject or style +guidance that never becomes a frame), so it stays reachable only through the +native `task` field and the `minimax-h3-gen` CLI. Two limits: the image must be +a **binary PPM (P6)** (no PNG or JPEG codec is vendored, the same residual the +chat multimodal path carries), and it must already be at the output resolution +(no image resampler is vendored). A mismatch is refused with the resolved +geometry in the message. + +### Video and audio references (`metadata`) + +H3 supports three reference modalities and OpenAI's schema has a slot for one, +so the other two enter through `metadata`, the standard OpenAI free-form string +map. Strict clients tolerate it, and no invented top-level field breaks their +schema validation. Unknown metadata keys are passed through untouched. + +```jsonc +{ + "prompt": "the same scene, at dusk", + "metadata": { + "input_reference_video": "/tmp/vllm_h3_videos/job0", // DIR of frame_%06d.ppm + "input_reference_audio": "/tmp/voice.wav" // 16-bit PCM WAV, or a data: URL + } +} +``` + +`input_reference_video` is a **directory of `frame_%06d.ppm`**, which is exactly +what this server and `minimax-h3-gen` write, so one run's frames chain straight +into the next request. It is not a container file: no demuxer is vendored. + +**A video reference is SILENT.** `MiniMaxH3EncodeReferenceVideo` emits a +`kVideoAudio` block with `ref_audio_t == 0`, so the clip contributes no sound of +its own. Supplying `input_reference_audio` alongside it attaches the audio to +that same block (one block carrying both, the layout upstream builds); without +it the reference is picture only. That is a real limitation, not an omission. + +**Legal combinations.** fl2va keyframes and ref2va reference blocks are +exclusive in the pipeline itself +([`minimax_h3_pipeline.cpp`](../src/vllm/model_executor/models/minimax_h3_pipeline.cpp)), +so the request parser enforces the same rule and returns a 400 naming the +offending pair rather than dropping a reference you supplied. + +| `input_reference` | `metadata.input_reference_video` | `metadata.input_reference_audio` | | +|---|---|---|---| +| (none) | (none) | (none) | t2va, prompt only | +| image | (none) | (none) | fl2va, the image is frame 0 | +| (none) | clip | (none) | ref2va, silent video reference | +| (none) | (none) | WAV | ref2va, audio reference | +| (none) | clip | WAV | ref2va, one block carrying both | +| image | clip and/or WAV | | **400**: keyframe and reference conditioning are exclusive | + +The video reference needs `--video-vae` (the encoder half of the same file) and +the audio reference needs `--audio-vae`; both load lazily, once, on the first +request that asks for them. + ### The job lifecycle `POST /v1/videos` returns immediately with `{"id": "vid_1", "status": "queued"}`; diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 3284b22c..9544e21d 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -32,8 +32,11 @@ #include #include #include +#include #include +#include #include +#include #include #include #include @@ -90,6 +93,105 @@ namespace fs = std::filesystem; using vllm::HfConfig; using vllm::Qwen3_5MoeWeights; +// Decode a binary PPM (P6) into [3, H, W] floats in [-1, 1] — the layout the H3 +// video-VAE encoder takes. It reads from BYTES rather than a path so one decoder +// serves both spellings of `input_reference` (a filesystem path and an inline +// data: URL). PPM is the only still-image container this tree can read: no PNG / +// JPEG codec is vendored, which is the same NAMED residual the chat multimodal +// path carries (see chat_mm.h), not a limitation of this endpoint. +std::vector DecodePpmChw(const std::string& bytes, int64_t* out_h, + int64_t* out_w) { + std::istringstream in(bytes, std::ios::binary); + std::string magic; + in >> magic; + if (magic != "P6") { + throw std::runtime_error( + "input_reference: not a binary PPM (P6); no PNG/JPEG codec is vendored, " + "so a reference image must be supplied as binary PPM"); + } + auto next_int = [&]() { + int v = 0; + while (in >> std::ws, in.peek() == '#') { std::string skip; std::getline(in, skip); } + in >> v; + return v; + }; + const int w = next_int(), h = next_int(), maxv = next_int(); + if (w <= 0 || h <= 0 || maxv <= 0) { + throw std::runtime_error("input_reference: bad PPM header"); + } + in.get(); // the single whitespace byte before the payload + std::vector rgb(static_cast(w) * h * 3); + in.read(reinterpret_cast(rgb.data()), static_cast(rgb.size())); + if (!in) throw std::runtime_error("input_reference: truncated PPM payload"); + std::vector chw(rgb.size()); + const int64_t plane = static_cast(w) * h; + for (int64_t i = 0; i < plane; ++i) { + for (int64_t c = 0; c < 3; ++c) { + chw[static_cast(c * plane + i)] = + static_cast(rgb[static_cast(i * 3 + c)]) / (maxv * 0.5f) - 1.0f; + } + } + if (out_h != nullptr) *out_h = h; + if (out_w != nullptr) *out_w = w; + return chw; +} + +// The bytes behind one reference: the parser hands us either a path or the +// already-decoded payload of a data: URL. +std::string ReadReferenceBytes(const std::string& field, const std::string& path, + const std::vector& inline_bytes) { + if (!inline_bytes.empty()) return std::string(inline_bytes.begin(), inline_bytes.end()); + std::ifstream in(path, std::ios::binary); + if (!in) throw std::runtime_error(field + ": cannot open " + path); + return std::string((std::istreambuf_iterator(in)), std::istreambuf_iterator()); +} + +// A ref2va VIDEO reference: DIR/frame_%06d.ppm, the exact layout minimax_h3_gen +// and this server's own muxer WRITE, so one run's frames chain straight into the +// next request. Returns [C, T, H, W] in [-1, 1] (the encoder's clip layout); +// no container demuxer is vendored, which is why this is a frame directory. +std::vector ReadReferenceClipChw(const std::string& dir, int64_t* out_t, + int64_t* out_h, int64_t* out_w) { + std::vector per_frame; // frame-major [T][C,H,W] + int64_t frames = 0, fh = 0, fw = 0; + for (int64_t k = 0;; ++k) { + char name[512]; + std::snprintf(name, sizeof(name), "%s/frame_%06lld.ppm", dir.c_str(), + static_cast(k)); + std::ifstream probe(name, std::ios::binary); + if (!probe) break; + const std::string bytes((std::istreambuf_iterator(probe)), + std::istreambuf_iterator()); + int64_t h = 0, w = 0; + const std::vector frame = DecodePpmChw(bytes, &h, &w); + if (frames == 0) { fh = h; fw = w; } + if (h != fh || w != fw) { + throw std::runtime_error( + "metadata.input_reference_video: every frame_%06d.ppm must have the same size"); + } + per_frame.insert(per_frame.end(), frame.begin(), frame.end()); + ++frames; + } + if (frames == 0) { + throw std::runtime_error("metadata.input_reference_video: no frame_%06d.ppm files in " + dir); + } + // [T][C,H,W] -> [C,T,H,W], the causal 3-D encoder's layout. + std::vector chw(per_frame.size()); + const int64_t plane = fh * fw; + for (int64_t c = 0; c < 3; ++c) { + for (int64_t k = 0; k < frames; ++k) { + for (int64_t e = 0; e < plane; ++e) { + chw[static_cast((c * frames + k) * plane + e)] = + per_frame[static_cast(k * 3 * plane + c * plane + e)]; + } + } + } + *out_t = frames; + *out_h = fh; + *out_w = fw; + return chw; +} + struct Args { std::string model_dir; std::string host = "0.0.0.0"; @@ -613,6 +715,18 @@ int main(int argc, char** argv) { vllm::MiniMaxH3PartitionInfo partition_info; vt::Device device; std::atomic counter{0}; + // The two VAEs' ENCODER halves, for the reference modalities: the video VAE + // encodes an `input_reference` image and a `metadata.input_reference_video` + // clip, the audio VAE encodes a `metadata.input_reference_audio` waveform. + // Loaded LAZILY and ONCE each: a text-to-video server must not pay for + // weights it never uses, and a server that does use them must not reload + // per request. + std::string video_vae_path, audio_vae_path; + std::mutex encoder_mutex; + bool video_encoder_loaded = false, audio_encoder_loaded = false; + vllm::MiniMaxH3AudioVaeWeights video_encoder_weights, audio_encoder_weights; + vllm::MiniMaxH3EncoderFcn3dConfig video_encoder_cfg; + vllm::MiniMaxH3AudioVaeEncoderConfig audio_encoder_cfg; }; std::shared_ptr video; if (!args.video_dit.empty()) { @@ -677,6 +791,8 @@ int main(int argc, char** argv) { video->workdir = args.video_workdir; video->ffmpeg = args.video_ffmpeg; video->partition_info = vllm::MiniMaxH3PartitionFromFlag(args.video_partition); + video->video_vae_path = args.video_vae; + video->audio_vae_path = args.audio_vae; if (args.video_device == "cuda") { video->device = vt::GetBackend(vt::DeviceType::kCUDA).CreateQueue().device; } @@ -722,13 +838,50 @@ int main(int argc, char** argv) { "video generation needs conditioning: start the server with " "--video-encoder (to condition on the prompt) or --video-prompt-embeds"); } + // OpenAI `input_reference` -> fl2va FIRST-FRAME conditioning. + // + // WHY fl2va and not ref2va: OpenAI documents input_reference as the image + // the generated video STARTS FROM (image-to-video), which is exactly what + // fl2va expresses — MiniMaxH3EncodeKeyframeCondRows pins frame 0 OF THE + // OUTPUT to the supplied image. ref2va + // (MiniMaxH3EncodeReferenceImages) means something else: whole reference + // images PREPENDED to the sequence as their own blocks, i.e. subject or + // style guidance that never becomes a frame of the result. Mapping + // input_reference there would silently change what the API promises. The + // ref2va modalities OpenAI has no slot for enter through `metadata` + // (input_reference_video / input_reference_audio) instead; the parser has + // already refused the combinations the pipeline forbids. + std::vector reference_chw; + int64_t reference_h = 0, reference_w = 0; + if (req.has_input_reference()) { + reference_chw = DecodePpmChw( + ReadReferenceBytes("input_reference", req.input_reference_path, + req.input_reference_bytes), + &reference_h, &reference_w); + } + std::vector reference_clip; + int64_t clip_t = 0, clip_h = 0, clip_w = 0; + if (req.has_input_reference_video()) { + reference_clip = ReadReferenceClipChw(req.input_reference_video_dir, &clip_t, + &clip_h, &clip_w); + } + const vllm::MiniMaxH3DitParams& p = video->dit.params; vllm::MiniMaxH3T2vaRequest r; r.partition = video->partition_info; // #77 guard: MiniMaxH3GenerateT2va // refuses a task this partition can't serve. + // With a reference image and no explicit task, the task IS fl2va; a + // metadata reference means ref2va. The image aspect also drives the + // default resolution (_resolve_shape). + const bool has_ref2va = + req.has_input_reference_video() || req.has_input_reference_audio(); + const std::string task = + !req.task.empty() + ? req.task + : (req.has_input_reference() ? "fl2va" : (has_ref2va ? "ref2va" : "t2va")); const vllm::MiniMaxH3ShapePlan plan = vllm::MiniMaxH3ResolveShape( - req.task.empty() ? "t2va" : req.task, req.duration_seconds, req.num_frames, - req.height, req.width, 0, 0); + task, req.duration_seconds, req.num_frames, req.height, req.width, + reference_w, reference_h); r.latent_t = plan.latent_t; r.num_frames = plan.num_frames; r.latent_h = plan.height / vllm::kMiniMaxH3VaeRatio; @@ -744,6 +897,97 @@ int main(int argc, char** argv) { r.audio_latents_std = video->audio_stats.std_dev; r.text_len = static_cast(conditioning.size()) / p.text_dim; + // Both VAE encoder halves load at most once, under one lock, whichever + // reference modality asks for them first. + auto ensure_video_encoder = [&]() { + if (video->video_vae_path.empty()) { + throw std::runtime_error( + "a video/image reference needs the video VAE ENCODER half: start " + "the server with --video-vae"); + } + if (video->video_encoder_loaded) return; + const vllm::SafetensorsFile vf = vllm::SafetensorsFile::Open(video->video_vae_path); + video->video_encoder_weights = vllm::LoadMiniMaxH3VideoVaeEncoderWeights(vf); + video->video_encoder_cfg = vllm::MiniMaxH3EncoderFcn3dConfig{}; + video->video_encoder_cfg.z_channels = 2 * p.latents_dim; // mean | logvar + video->video_encoder_loaded = true; + }; + auto ensure_audio_encoder = [&]() { + if (video->audio_vae_path.empty()) { + throw std::runtime_error( + "metadata.input_reference_audio needs the audio VAE ENCODER half: " + "start the server with --audio-vae"); + } + if (video->audio_encoder_loaded) return; + const vllm::SafetensorsFile af = vllm::SafetensorsFile::Open(video->audio_vae_path); + video->audio_encoder_weights = vllm::LoadMiniMaxH3AudioVaeEncoderWeights(af); + video->audio_encoder_cfg = vllm::MiniMaxH3AudioVaeEncoderConfig{}; + video->audio_encoder_cfg.vae_latent_channels = p.audio_latents_dim; + video->audio_encoder_loaded = true; + }; + + if (req.has_input_reference()) { + if (reference_h != plan.height || reference_w != plan.width) { + // No image resampler is vendored, and a mis-sized keyframe would + // either abort deep in the denoise or pin the wrong latent rows. Say + // so up front, with the geometry we resolved. + throw std::runtime_error( + "input_reference is " + std::to_string(reference_w) + "x" + + std::to_string(reference_h) + " but this request resolved to " + + std::to_string(plan.width) + "x" + std::to_string(plan.height) + + "; supply the reference at the output size (or pass a matching " + "`size`): no image resampler is vendored"); + } + std::lock_guard guard(video->encoder_mutex); + ensure_video_encoder(); + r.keyframe_frame_indices = {0}; // FIRST frame; upstream also allows {-1}/{0,-1} + r.imgvid_noise_aug = 1.0; // pin the frame exactly + r.keyframe_cond_rows = vllm::MiniMaxH3EncodeKeyframeCondRows( + video->video_encoder_cfg, video->video_encoder_weights, p, {reference_chw}, + reference_h, reference_w, r.latent_t, r.imgvid_noise_aug, {}); + } else if (has_ref2va) { + // ── ref2va REFERENCE BLOCKS. Exclusive with the fl2va branch above + // (minimax_h3_pipeline.cpp:251), which the parser already enforced. ── + std::lock_guard guard(video->encoder_mutex); + std::vector blocks; + if (req.has_input_reference_video()) { + ensure_video_encoder(); + vllm::MiniMaxH3RefBlock block{}; + r.keyframe_cond_rows = vllm::MiniMaxH3EncodeReferenceVideo( + video->video_encoder_cfg, video->video_encoder_weights, p, reference_clip, + clip_t, clip_h, clip_w, &block); + // SILENT by construction: MiniMaxH3EncodeReferenceVideo emits + // ref_audio_t == 0 because the clip's own soundtrack would need the + // audio VAE encoder run over it. An audio reference below ATTACHES to + // this block, which is the layout packed_sequence.py builds. + blocks.push_back(block); + } + if (req.has_input_reference_audio()) { + ensure_audio_encoder(); + const std::string wav_bytes = ReadReferenceBytes( + "metadata.input_reference_audio", req.input_reference_audio_path, + req.input_reference_audio_bytes); + int64_t samples_per_channel = 0; + const std::vector waveform = vllm::MiniMaxH3ReadWav( + wav_bytes, vllm::kMiniMaxH3AudioChannels, vllm::kMiniMaxH3AudioSampleRate, + &samples_per_channel); + vllm::MiniMaxH3RefBlock audio_block{}; + r.audio_ref_rows = vllm::MiniMaxH3EncodeReferenceAudio( + video->audio_encoder_cfg, video->audio_encoder_weights, waveform, + vllm::kMiniMaxH3AudioChannels, samples_per_channel, video->audio_stats.mean, + video->audio_stats.std_dev, /*noise_aug=*/1.0, {}, &audio_block); + if (!blocks.empty() && + blocks[0].kind == vllm::MiniMaxH3RefBlock::Kind::kVideoAudio) { + // The reference video now HAS sound: one kVideoAudio block carries + // both, so its ref_audio_t must claim exactly the rows just encoded. + blocks[0].ref_audio_t = audio_block.ref_audio_t; + } else { + blocks.push_back(audio_block); + } + } + r.ref_blocks = blocks; + } + const int64_t frame_rows = (r.latent_h / p.patch_size_h) * (r.latent_w / p.patch_size_w); std::vector nv(static_cast(r.latent_t * frame_rows * diff --git a/include/vllm/entrypoints/openai/video_api.h b/include/vllm/entrypoints/openai/video_api.h index 89a16aeb..655d2c77 100644 --- a/include/vllm/entrypoints/openai/video_api.h +++ b/include/vllm/entrypoints/openai/video_api.h @@ -5,7 +5,7 @@ // POST /v1/videos/sync -> run to completion, return the MP4 in the body // // It ALSO speaks OpenAI's Sora video shape, so an OpenAI client works unmodified: -// POST /v1/videos {model, prompt, size:"WxH", seconds} +// POST /v1/videos {model, prompt, size:"WxH", seconds, input_reference} // GET /v1/videos/{id} job status // GET /v1/videos/{id}/content the finished MP4 bytes (video/mp4) // The OpenAI spellings are ALIASES onto the native fields, never replacements — @@ -29,8 +29,8 @@ namespace vllm::openai { // One parsed /v1/videos request. Mirrors the fields vLLM-Omni accepts; anything // absent falls back to H3's documented defaults via the shape planner. The -// OpenAI-spelled fields (`model`, `size`, `seconds`) land on the SAME members, -// so nothing downstream learns a second vocabulary. +// OpenAI-spelled fields (`model`, `size`, `seconds`, `input_reference`) land on +// the SAME members, so nothing downstream learns a second vocabulary. struct VideoRequest { std::string prompt; // OpenAI `model` ("sora-2-pro", ...). RECORDED, never a hard failure: the video @@ -47,6 +47,38 @@ struct VideoRequest { double audio_flow_shift = 3.0; // audio int64_t seed = 0; bool has_seed = false; + + // OpenAI `input_reference` — the image an image-to-video request starts from. + // Exactly one of these is populated (see ParseVideoRequest): a filesystem path, + // or the bytes of an inline RFC 2397 `data:` URL plus its declared media type. + std::string input_reference_path; + std::vector input_reference_bytes; + std::string input_reference_media_type; + + bool has_input_reference() const { + return !input_reference_path.empty() || !input_reference_bytes.empty(); + } + + // OpenAI `metadata`: a free-form string->string map every strict client already + // tolerates. H3 supports THREE reference modalities and OpenAI's schema has a + // slot for exactly one (the image), so the other two enter here rather than as + // invented top-level fields that would fail a client's schema validation. The + // whole map is kept verbatim (unknown keys are passed through untouched); the + // two keys we act on are lifted out below. + // metadata.input_reference_video — a DIRECTORY of frame_%06d.ppm, the exact + // layout `minimax-h3-gen` writes, so one run's output chains into the next. + // SILENT: the clip carries no audio unless input_reference_audio is also + // given (the audio VAE's encoder is a separate call). + // metadata.input_reference_audio — a 16-bit PCM WAV path or `data:` URL. + std::map metadata; + std::string input_reference_video_dir; + std::string input_reference_audio_path; + std::vector input_reference_audio_bytes; + + bool has_input_reference_video() const { return !input_reference_video_dir.empty(); } + bool has_input_reference_audio() const { + return !input_reference_audio_path.empty() || !input_reference_audio_bytes.empty(); + } }; // Parse OpenAI's `size` — "x", e.g. "1280x720" — into its two @@ -60,6 +92,15 @@ void ParseVideoSize(const std::string& size, int64_t* width, int64_t* height); // malformed input rather than silently defaulting, so a bad request is a 400 with // a reason instead of a surprising generation. // +// REFERENCE COMBINATIONS are checked here, not left to the pipeline, because a +// reference the caller supplied and we quietly dropped is the failure that looks +// like it worked. The rule is the pipeline's own +// (minimax_h3_pipeline.cpp:251): fl2va keyframes and ref2va reference blocks are +// EXCLUSIVE. `input_reference` is fl2va; the metadata video/audio references are +// ref2va. So legal: nothing; input_reference alone; video alone; audio alone; +// video+audio (one kVideoAudio block carrying both). Illegal, and a 400 naming +// the pair: input_reference together with either metadata reference. +// // PRECEDENCE, when a body carries both spellings of one value: the NATIVE field // WINS (`width`/`height` over `size`, `duration` over `seconds`). The OpenAI // spelling is a compatibility shim, so this guarantees that every body which diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index ceb58b86..ece19333 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -264,7 +264,15 @@ # fl2va/ref2va exclusivity is enforced at the request boundary. CPU-landed + # gated, `PARTIAL`, `CLAIM-SERVE-VIDEOS-OAI`, spec `specs/minimax-h3.md` §9. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 141 +# 142 since 2026-08-06: +`SERVE-VIDEOS-REFS` (reference CONDITIONING over +# `/v1/videos`: `input_reference` -> fl2va, plus the two `metadata` ref2va +# modalities) — a real new capability stacked on `SERVE-VIDEOS-OAI`, not a +# restatement of it: that row made an OpenAI body PARSE, this one makes its +# references reach the pipeline. Before it no reference modality was reachable +# over HTTP at all. CPU-landed + gated, `CLAIM-SERVE-VIDEOS-REFS`, spec +# `specs/minimax-h3.md` §10. +# Bumped for a real new row, never to make a failing state transition pass. +ENGINE_ROWS = 142 MATRIX_PATHS = [ENGINE_MATRIX, *(path for path, _ in MATRICES.values())] REQUIRED = [ diff --git a/src/vllm/entrypoints/openai/video_api.cpp b/src/vllm/entrypoints/openai/video_api.cpp index 6315b0c1..a1d74118 100644 --- a/src/vllm/entrypoints/openai/video_api.cpp +++ b/src/vllm/entrypoints/openai/video_api.cpp @@ -8,6 +8,11 @@ #include #include #include + +// DecodeDataUri: the RFC 2397 `data:` decode the chat multimodal parts already +// use. `input_reference` accepting a data: URL means the SAME decoder serves both +// surfaces rather than a second, subtly different one. +#include "vllm/entrypoints/openai/chat_mm.h" #include "vt/dtype.h" namespace vllm::openai { @@ -65,6 +70,65 @@ double ReadDuration(const nlohmann::json& body, const char* key) { return 0.0; // unreachable; VT_CHECK(false) throws } +// A reference SOURCE: a filesystem path, or an inline `data:` URL. An http(s) +// URL is REFUSED by name rather than treated as a path, because silently +// stat-ing "https://..." would fail much later with a confusing message. +// `field` names the offending field so the 400 says which one. +void ReadReferenceSource(const std::string& field, const std::string& value, + std::string* out_path, std::vector* out_bytes, + std::string* out_media_type) { + VT_CHECK(!value.empty(), "video request: `" + field + "` must not be empty"); + if (value.compare(0, 5, "data:") == 0) { + try { + const entrypoints::openai::DecodedMedia media = + entrypoints::openai::DecodeDataUri(value); + VT_CHECK(!media.bytes.empty(), + "video request: `" + field + "` data: URL decoded to no bytes"); + *out_bytes = media.bytes; + if (out_media_type != nullptr) *out_media_type = media.media_type; + } catch (const std::exception& e) { + VT_CHECK(false, "video request: `" + field + "` is not a valid data: URL: " + + e.what()); + } + return; + } + VT_CHECK(value.compare(0, 7, "http://") != 0 && value.compare(0, 8, "https://") != 0, + "video request: `" + field + + "` must be a filesystem path or a data: URL; fetching an http(s) " + "URL is not supported"); + *out_path = value; +} + +// OpenAI `metadata`: a free-form string->string map. Everything is kept, and the +// two H3 reference keys OpenAI has no slot for are lifted into typed fields. +void ReadMetadata(const nlohmann::json& value, VideoRequest* out) { + VT_CHECK(value.is_object(), "video request: `metadata` must be an object"); + for (const auto& [key, item] : value.items()) { + VT_CHECK(item.is_string(), + "video request: `metadata." + key + + "` must be a string (metadata is a string map)"); + out->metadata[key] = item.get(); + } + const auto video = out->metadata.find("input_reference_video"); + if (video != out->metadata.end()) { + VT_CHECK(!video->second.empty(), + "video request: `metadata.input_reference_video` must not be empty"); + // A DIRECTORY of frame_%06d.ppm, not a container file: no demuxer is + // vendored, and this is exactly the layout `minimax-h3-gen` writes, so one + // run's frames chain into the next request. `data:` cannot name a directory. + VT_CHECK(video->second.compare(0, 5, "data:") != 0, + "video request: `metadata.input_reference_video` must be a directory " + "of frame_%06d.ppm files, so a data: URL cannot express it"); + out->input_reference_video_dir = video->second; + } + const auto audio = out->metadata.find("input_reference_audio"); + if (audio != out->metadata.end()) { + ReadReferenceSource("metadata.input_reference_audio", audio->second, + &out->input_reference_audio_path, + &out->input_reference_audio_bytes, nullptr); + } +} + } // namespace void ParseVideoSize(const std::string& size, int64_t* width, int64_t* height) { @@ -109,6 +173,24 @@ VideoRequest ParseVideoRequest(const std::string& body) { out.model = json.at("model").get(); VT_CHECK(!out.model.empty(), "video request: `model` must not be empty"); } + if (Has(json, "input_reference")) { + VT_CHECK(json.at("input_reference").is_string(), + "video request: `input_reference` must be a string (a filesystem path " + "or a data: URL)"); + ReadReferenceSource("input_reference", json.at("input_reference").get(), + &out.input_reference_path, &out.input_reference_bytes, + &out.input_reference_media_type); + } + if (Has(json, "metadata")) ReadMetadata(json.at("metadata"), &out); + // The pipeline's own exclusion (minimax_h3_pipeline.cpp:251), enforced HERE so + // it is a 400 with the offending pair named rather than a failed job — and so a + // supplied reference is never silently dropped. + VT_CHECK(!(out.has_input_reference() && + (out.has_input_reference_video() || out.has_input_reference_audio())), + "video request: `input_reference` (fl2va first-frame conditioning) cannot " + "be combined with `metadata.input_reference_video`/" + "`metadata.input_reference_audio` (ref2va reference blocks); keyframe and " + "reference conditioning are exclusive"); // vLLM-Omni carries the generation knobs under `extra_params`; accept them at // the top level too so a plain client does not have to nest. const nlohmann::json& extra = diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index 023e40be..065bcb35 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -1808,13 +1808,15 @@ TEST_CASE("api_server: the OpenAI request aliases reach the runner unchanged") { // the field default, so this proves the parser ran end to end. ApiServer::DispatchResult r = h.server.handle_videos_sync(R"({ "model": "test-model", "prompt": "a cat on a skateboard", - "size": "1280x720", "seconds": "8" + "size": "1280x720", "seconds": "8", "input_reference": "/tmp/frame0.ppm" })"); REQUIRE(r.status == 200); CHECK(seen.prompt == "a cat on a skateboard"); CHECK(seen.width == 1280); CHECK(seen.height == 720); CHECK(seen.duration_seconds == doctest::Approx(8.0)); + CHECK(seen.input_reference_path == "/tmp/frame0.ppm"); + CHECK(seen.has_input_reference()); CHECK(seen.model == "test-model"); // The served model was named, so nothing is warned about. nlohmann::json body = nlohmann::json::parse(r.body); @@ -1822,6 +1824,61 @@ TEST_CASE("api_server: the OpenAI request aliases reach the runner unchanged") { CHECK_FALSE(body.contains("warning")); } +TEST_CASE("api_server: every reference modality reaches the runner, or is refused") { + // H3 has three reference modalities and OpenAI's schema carries one, so the + // other two ride in `metadata`. What matters is that each one ARRIVES at the + // runner (the library gates already prove a reference changes the output); + // a reference that parsed and then never reached the pipeline is the failure + // that looks like it worked. + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + ServerHarness h(c, w, Fixture()); + vllm::openai::VideoRequest seen; + int calls = 0; + h.server.set_video_runner( + [&](const vllm::openai::VideoRequest& req) -> std::string { + seen = req; + ++calls; + return "/tmp/out.mp4"; + }); + + SUBCASE("the metadata video + audio references arrive together") { + REQUIRE(h.server + .handle_videos_sync(R"({"prompt":"x","metadata":{ + "input_reference_video":"/tmp/prev_job", + "input_reference_audio":"/tmp/voice.wav", + "trace_id":"abc-123"}})") + .status == 200); + CHECK(calls == 1); + CHECK(seen.input_reference_video_dir == "/tmp/prev_job"); + CHECK(seen.input_reference_audio_path == "/tmp/voice.wav"); + CHECK(seen.metadata.at("trace_id") == "abc-123"); // free-form keys pass through + CHECK_FALSE(seen.has_input_reference()); + } + + SUBCASE("the audio reference arrives alone, as inline bytes") { + REQUIRE(h.server + .handle_videos_sync( + R"({"prompt":"x","metadata":{"input_reference_audio":"data:audio/wav;base64,aGk="}})") + .status == 200); + CHECK(calls == 1); + REQUIRE(seen.input_reference_audio_bytes.size() == 2); + CHECK(seen.input_reference_audio_bytes[0] == 'h'); + CHECK_FALSE(seen.has_input_reference_video()); + } + + SUBCASE("fl2va + ref2va is a 400 and the runner is never reached") { + ApiServer::DispatchResult r = h.server.handle_videos_sync( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_audio":"/tmp/a.wav"}})"); + CHECK(r.status == 400); + CHECK(calls == 0); // nothing generated from a half-honoured request + const std::string message = + nlohmann::json::parse(r.body).at("error").at("message").get(); + CHECK(message.find("exclusive") != std::string::npos); + } +} + TEST_CASE("api_server: an unserved `model` warns on the job but still generates") { const HfConfig c = MakeConfig(); const Qwen3_5MoeWeights w = MakeWeights(c); diff --git a/tests/vllm/entrypoints/openai/test_video_api.cpp b/tests/vllm/entrypoints/openai/test_video_api.cpp index cc9f076d..a232b959 100644 --- a/tests/vllm/entrypoints/openai/test_video_api.cpp +++ b/tests/vllm/entrypoints/openai/test_video_api.cpp @@ -63,7 +63,7 @@ TEST_CASE("video api: request parsing applies H3 defaults and rejects bad input" // --------------------------------------------------------------------------- // OpenAI's Sora video shape (developers.openai.com/api/docs/guides/video-generation): -// {model, prompt, size:"x", seconds}. These are ALIASES +// {model, prompt, size:"x", seconds, input_reference}. These are ALIASES // onto the native fields, so an OpenAI client works unmodified while every // existing body keeps its exact meaning. // @@ -168,6 +168,114 @@ TEST_CASE("video api: `model` is recorded, never a parse failure") { CHECK(ParseVideoRequest(R"({"prompt":"x"})").model.empty()); } +TEST_CASE("video api: `input_reference` accepts a path or a data: URL") { + const VideoRequest path = + ParseVideoRequest(R"({"prompt":"x","input_reference":"/tmp/first.ppm"})"); + CHECK(path.input_reference_path == "/tmp/first.ppm"); + CHECK(path.input_reference_bytes.empty()); + CHECK(path.has_input_reference()); + + // "hi" base64-encoded, with the media type preserved for the runner. + const VideoRequest inline_data = ParseVideoRequest( + R"({"prompt":"x","input_reference":"data:image/x-portable-pixmap;base64,aGk="})"); + CHECK(inline_data.input_reference_path.empty()); + CHECK(inline_data.input_reference_media_type == "image/x-portable-pixmap"); + REQUIRE(inline_data.input_reference_bytes.size() == 2); + CHECK(inline_data.input_reference_bytes[0] == 'h'); + CHECK(inline_data.input_reference_bytes[1] == 'i'); + + CHECK_FALSE(ParseVideoRequest(R"({"prompt":"x"})").has_input_reference()); + + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":""})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":5})")); + // A broken data: URL must be a specific 400, not a path named "data:...". + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"data:image/png;base64,@@@"})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"data:image/png,raw"})")); + // An http(s) URL is refused BY NAME rather than treated as a filesystem path. + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","input_reference":"https://x/i.png"})")); +} + +// --------------------------------------------------------------------------- +// The two reference modalities OpenAI has no slot for. H3 supports THREE (image, +// silent video, audio) and the Sora schema carries one, so the other two ride in +// `metadata` — a standard OpenAI free-form string map, which strict clients +// tolerate, rather than invented top-level fields that would fail validation. +// --------------------------------------------------------------------------- + +TEST_CASE("video api: `metadata` carries the video and audio references") { + const VideoRequest r = ParseVideoRequest(R"({ + "prompt":"x", + "metadata":{"input_reference_video":"/tmp/prev_job", + "input_reference_audio":"/tmp/voice.wav", + "trace_id":"abc-123"} + })"); + // A DIRECTORY of frame_%06d.ppm — the layout minimax-h3-gen writes, so clips chain. + CHECK(r.input_reference_video_dir == "/tmp/prev_job"); + CHECK(r.has_input_reference_video()); + CHECK(r.input_reference_audio_path == "/tmp/voice.wav"); + CHECK(r.has_input_reference_audio()); + CHECK_FALSE(r.has_input_reference()); // the image slot stayed empty + // Free-form means free-form: keys we do not act on survive untouched. + CHECK(r.metadata.at("trace_id") == "abc-123"); + CHECK(r.metadata.size() == 3); + + // The audio reference takes an inline data: URL too ("hi" base64). + const VideoRequest inline_audio = ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_audio":"data:audio/wav;base64,aGk="}})"); + CHECK(inline_audio.input_reference_audio_path.empty()); + REQUIRE(inline_audio.input_reference_audio_bytes.size() == 2); + CHECK(inline_audio.has_input_reference_audio()); + + // Absent metadata leaves every reference empty. + const VideoRequest none = ParseVideoRequest(R"({"prompt":"x"})"); + CHECK(none.metadata.empty()); + CHECK_FALSE(none.has_input_reference_video()); + CHECK_FALSE(none.has_input_reference_audio()); + + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":"nope"})")); // not an object + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"k":7}})")); // not a string map + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"input_reference_video":""}})")); + CHECK_THROWS(ParseVideoRequest(R"({"prompt":"x","metadata":{"input_reference_audio":""}})")); + // A directory cannot be a data: URL, and saying so beats a confusing open() later. + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_video":"data:video/mp4;base64,aGk="}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_audio":"https://x/a.wav"}})")); +} + +TEST_CASE("video api: only the pipeline's legal reference combinations are accepted") { + // The rule is minimax_h3_pipeline.cpp:251 — fl2va keyframes and ref2va + // reference blocks are EXCLUSIVE. `input_reference` is fl2va; the metadata + // references are ref2va blocks. + auto parses = [](const char* body) { + ParseVideoRequest(body); + return true; + }; + + // LEGAL: each alone, and video+audio together (one kVideoAudio block). + CHECK(parses(R"({"prompt":"x","input_reference":"/tmp/f0.ppm"})")); + CHECK(parses(R"({"prompt":"x","metadata":{"input_reference_video":"/tmp/clip"}})")); + CHECK(parses(R"({"prompt":"x","metadata":{"input_reference_audio":"/tmp/a.wav"}})")); + const VideoRequest both = ParseVideoRequest( + R"({"prompt":"x","metadata":{"input_reference_video":"/tmp/clip", + "input_reference_audio":"/tmp/a.wav"}})"); + CHECK(both.has_input_reference_video()); + CHECK(both.has_input_reference_audio()); + + // ILLEGAL: fl2va + ref2va. REJECTED, never silently dropped — a dropped + // reference is the failure that looks like it worked. + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_video":"/tmp/clip"}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_audio":"/tmp/a.wav"}})")); + CHECK_THROWS(ParseVideoRequest( + R"({"prompt":"x","input_reference":"/tmp/f0.ppm", + "metadata":{"input_reference_video":"/tmp/clip", + "input_reference_audio":"/tmp/a.wav"}})")); +} + TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields") { // The exact shape an unmodified OpenAI client sends, plus our native knobs // alongside: this is ADDITIVE compatibility, not a replacement. @@ -176,7 +284,8 @@ TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields "prompt": "a cat on a skateboard", "size": "1280x720", "seconds": "8", - "task": "t2va", + "input_reference": "/tmp/frame0.ppm", + "task": "fl2va", "extra_params": {"num_frames": 97, "num_inference_steps": 30, "flow_shift": 9.5, "audio_flow_shift": 2.5, "seed": 4242} })"); @@ -185,7 +294,8 @@ TEST_CASE("video api: a whole OpenAI-shaped body parses, keeping our rich fields CHECK(r.width == 1280); CHECK(r.height == 720); CHECK(r.duration_seconds == doctest::Approx(8.0)); - CHECK(r.task == "t2va"); + CHECK(r.input_reference_path == "/tmp/frame0.ppm"); + CHECK(r.task == "fl2va"); CHECK(r.num_frames == 97); CHECK(r.num_inference_steps == 30); // default 50 CHECK(r.flow_shift == doctest::Approx(9.5)); // default 12.0