diff --git a/.agents/NOW.md b/.agents/NOW.md index 2bce1e1e8..1e1d4a525 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | -| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 2 LANDED (#123)**: H3 video on the surface (`vllm_video_*` v12, `/v1/videos` via the seam, both examples thin clients, ratchet 11→9); ROW 1 (#121) before it | GB10 re-verify residual; next fold row | +| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 IN REVIEW (`row/DEVICE-KNOB`)**: ABI v14 `device` knob (0=auto/1=cpu/2=cuda), absent-cuda fails loud, `--device` on server+cli, #123's 3 minors folded | Merge after review; rebased (#122); CUDA A/B residual | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/specs/one-surface-abi.md b/.agents/specs/one-surface-abi.md index ad4a273b4..7eaa31d0c 100644 --- a/.agents/specs/one-surface-abi.md +++ b/.agents/specs/one-surface-abi.md @@ -1,6 +1,6 @@ # ONE SURFACE — every capability ships through the C ABI -Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign).** +Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; the #123 review's three MINOR findings closed in the same change (c_header_compile.c now references the v11+v12 surface + the v14 field; the v12 changelog block moved to chronological position; the fold fixture's flag list gained `--keep-quant`). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** ## The defect diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index eccf47aee..5b0a37dbd 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -141,7 +141,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows). | 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner | | 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | — | M | pooler live-wiring | | 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`→`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual | -| 8 | Device-selection knob | `device` field on `vllm_model_params` (absent today) → `EngineParams` → `SelectQueue`; `--device` on server + cli | additive | S | mirror vLLM `--device`/`DeviceConfig` | +| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue` explicit arms via `LoadedEngine::ResolveExplicitDeviceType`; explicit cpu never probes, explicit ABSENT cuda fails LOUD before any model I/O (device.py:61-66 never-substitute mirror); `--device` on server + cli | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | | 9 | Voxtral + audio chat seam | register `VoxtralForConditionalGeneration` + fold `VoxtralGenerateGreedy` into the registry forward; audio-capable chat fn + an engine consumer for `AudioKwargs` mm_features | rewrite tests→clients | M | mirror upstream `voxtral.py:309`, `SupportsTranscription` | | 10 | Gemma-4 audio e2e | bf16 device audio forward + audio→text merge (residual `gemma4_audio.h:41`); USM log-mel front end | fold into the registered mm forward | M | `MM-SERVE-E2E` | | 11 | Tokenizer/bench ABI + real-load gate | `vllm_tokenize`/`vllm_detokenize`; token-id/count fields on the stream callback for bench; **gate `vllm_engine_load` on a REAL tiny checkpoint at least once** (today bad-path only, `test_capi.cpp:474`) | rewrite `tokenize`/`bench` as clients | S-M | `/tokenize` route exists (`api_server.cpp:432`) | diff --git a/.agents/state.md b/.agents/state.md index 200b027af..02468dc19 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -42368,3 +42368,84 @@ The full digits and file:line map live in spec §21; the binding facts: capture pushed the unified pool below the floor at 00:25 — reproduces the §19 measured box-safety violation; NOT retried per the safety mandate). Denominator stays the #111 recorded ~21 floor. Box recovered clean; worker auto-restored (--restart=always). + +## 2026-08-08 — ARCH-ONE-SURFACE ROW 8: explicit device selection through the ONE surface (`row/DEVICE-KNOB`, task #284) + + +**What.** The smallest fold row: an embedder can now ASK for a device instead +of inheriting the accelerator-first probe. Mirrors vLLM's `DeviceConfig.device` +names (`${VLLM_SOURCE}/vllm/config/device.py:13` `Device = Literal["auto", +"cuda", "cpu", ...]`; explicit assignment is VERBATIM, never substituted, +device.py:61-66; upstream's `--device` CLI flag itself was removed in v0.10, +`bc8a8ce5e`, so the ABI field mirrors the surviving config surface). + +- W1 ABI: `vllm_model_params.device` appended (int32, 0=auto/1=cpu/2=cuda — + 0 MUST be auto for the zero-value contract; cpu-before-cuda follows the v12 + `vllm_video_model_params.device` precedent shifted by the auto slot). + `VLLM_ABI_VERSION` -> 14 per the operator's version-collision correction + (#122 Kimi `vllm_complete_tokens` reconciles to v13 at its landing; this + branch pre-rebase shows v12 -> v14 with the gap named in the changelog). + Floor pin advanced to `>= 14` (the #121 == lesson). New + `include/vllm/config/device.h` + `src/vllm/config/device.cpp` + (`vllm::Device`, `DeviceFromString`, `DeviceName`), ported from + vllm/config/device.py @ 555967922. +- W2 plumb: `EngineParams::device` -> `SelectQueue(architecture, device)`. + Explicit arms route through `LoadedEngine::ResolveExplicitDeviceType(device, + cuda_platform_registered)` — pure, so the CPU tier gates the full matrix + including "explicit cpu beats a REGISTERED accelerator". Explicit cpu never + consults the probe; explicit cuda requires the registered kCUDA platform + (cuda.cpp Registrar == usable GPU) and THROWS the pinned message otherwise; + queue-creation failure on the explicit arm PROPAGATES (no try/catch), unlike + the auto arm's byte-identical catch-and-fall-back. `FromModelDir` resolves + an explicit device BEFORE any path/config I/O (mirror of DeviceConfig + resolving at config time, arg_utils.py:1878), so the device error is never + masked by a path error — which is also what makes the capi plumb pinnable + with no loadable checkpoint. capi validates 0/1/2 (else + VLLM_ERR_INVALID_ARGUMENT) and refuses explicit cuda on the CPU-hosted + transcription stack instead of silently downgrading. +- W3 thin clients: `--device auto|cpu|cuda` on `vllm-cli` (ABI field only, + local name->int map) and `examples/server` (`vllm::DeviceFromString` -> + `EngineParams.device`; `--video-device` untouched — separate engine, + separate checkpoint, may legitimately differ). +- W5 (the #123 review's 3 MINORS, closed): c_header_compile.c now actually + references the v11 transcription + v12 video entry points + the v14 field + (its "references every ABI entry point" claim went stale at v10); the v12 + changelog block moved to chronological position (after v11); the H3 fold + fixture's flag list gained `--keep-quant` (matches test:85 + the capture). + +**Gates.** Full CPU build -Werror clean (fresh tree). test_capi 44 cases +(4 new v14 + floor >= 14) green; test_loaded_engine_dense +3 device cases +green; test_dlopen green; test_openai_api_server + the new explicit-cpu +serving smoke green; test_minimax_h3_video_fold green (vllm.h touched); +test_model_loader_gguf + test_platform green. check-surface-coverage green +(capability row added, ratchet UNTOUCHED — 8 after the #122 rebase below; +9 at first write). STATUS ratchet: paid for the +new line by removing a stale back-to-back merge DUPLICATE of the metrics +narrative (nothing lost), ratchet 279200 -> 279150 (measured 279108). + +**Mutation kills (each RED then reverted).** M1 default device=2 -> +zero-contract 3 fails; M2 capi 0->kCUDA -> zero-contract bogus-path arm RED +(the auto arm must report the PATH, not a device); M3 range validation +dropped -> range test 6 fails; M4 capi drops the device=2 mapping -> plumb +pin RED; M5 FromModelDir early resolve removed -> capi plumb pin + the +device-before-path test both RED; M6 ctor passes kAuto -> EngineParams-seam +ctor-throw arm RED; M7 SelectQueue explicit arm catch-and-fallback -> same +test RED; M8 policy hands explicit-cpu to a registered accelerator -> matrix +RED; M9 DeviceFromString "cuda"->kCPU -> parse test RED; M10 ABI macro left +at 12 -> floor pin RED. Not uniquely killable (disclosed): the api_server +smoke (same seams as M6-M8), c_header_compile (a compile-time reference), +the two doc-only minors. + +**Residuals (honest).** (1) CUDA-build A/B — auto->CUDA vs explicit-cpu->CPU +vs explicit-cuda->CUDA on a GPU box — not run (no dgx in scope); the CPU tier +pins that half via the pure matrix. (2) The 0-as-explicit-cpu vs 0-as-auto +distinction is behaviorally invisible on a CPU-only tier at the queue level; +pinned structurally (matrix + the bogus-path capi arm) and named for the +CUDA-build pass. (3) EXECUTED post-#122 (9f772cad): rebased onto main — v14 sits +on v13 (changelog now v10..v14 chronological, the minor-#2 v12 move kept), +c_header_compile composes main's `vllm_complete_tokens` reference with the +v14 field, floor pin stays >= 14, allowlist ratchet inherited at 8, keyed +records main-wholesale + these rows re-applied. (4) Explicit +names for xpu/vulkan/metal/rocm are additive later; auto reaches them today. +(5) The transcription stack refuses explicit cuda (CPU pipeline) — revisit +when an accelerated transcription path exists. diff --git a/CMakeLists.txt b/CMakeLists.txt index a7906a079..7d20ebea7 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -513,6 +513,7 @@ add_library(vllm STATIC src/vllm/sampling_params.cpp src/vllm/lora/punica_cpu.cpp src/vllm/config/scheduler.cpp + src/vllm/config/device.cpp src/vllm/config/kv_transfer.cpp src/vllm/config/speculative.cpp src/vllm/outputs.cpp diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index acd083f44..a0178ffe5 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Track | Status | Next gate | |---|---|---| -| Surface coverage (`ARCH-ONE-SURFACE`) | **No number owed**: a CPU records/tooling change (the ONE SURFACE audit + `scripts/check-surface-coverage.py` guard, two axes, preflight + CI); no kernel or generation path touched | Fold-plan lanes carry their own gates; guard keeps CLI-only capabilities from landing untracked | +| Surface coverage (`ARCH-ONE-SURFACE`) | **No number owed**: CPU tooling/plumbing (guard; ROW 8 device knob, ABI v14 `device`, zero value = the probe, byte-identical); no kernel path touched | Fold lanes carry their own gates; ROW 8 CUDA-build A/B = named residual | | 35B prefill TTFT | 0.93x to 0.98x at every concurrency (2026-08-05) | Attribute the residual, then close | | 35B low-batch MoE decode | CLOSED at low batch (c1 0.975x, c4 wins); c16 0.93x. `VT_ASYNC_DEVICE_MIRROR` **default ON for correctness**. `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) A/B'd speed-NEUTRAL | c16 lever is prefill glue (task #61), not the decode drain. `test_qwen36_async_serving` GREEN | | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 21694513b..5b22a6ed1 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -247,6 +247,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Embeddings / pooling | none | embedder-unreachable | | Audio transcription (Parakeet ASR) | `vllm_transcribe`, `vllm_transcription_params_default`, `vllm_transcription_free` | reachable | | Video+audio generation (MiniMax-H3) | `vllm_video_engine_load`, `vllm_video_generate`, `vllm_video_result_free`, `vllm_video_mux_argv` | reachable | +| Explicit device selection (auto/cpu/cuda) | `device` field on `vllm_model_params` (ABI v14; 0=auto keeps the probe, explicit absent device fails loud) | reachable | | Multimodal input (image/audio/video) | none | embedder-unreachable | ## Parallelism and scale-out diff --git a/docs/STATUS.md b/docs/STATUS.md index d0b72bde2..1db2bd261 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1563,6 +1563,9 @@ runtime-verified yet. lanes. Guard `scripts/check-surface-coverage.py` (two axes, preflight + CI): every `examples/*` unit is a client of `include/vllm.h` or tracked to a fold row; every `FEATURES.md` C-ABI capability names an entry point or is tracked. + Fold ROW 8: `vllm_model_params.device` (ABI v14, 0=auto/1=cpu/2=cuda); + explicit cpu forces CPU, an absent named device fails loud (never + substituted); `--device` on server+cli. CUDA-build A/B = named residual. - **Automatic prefix caching (APC)** is on by default for dense models (hybrid / GDN and attention-free default off, mirroring vLLM), and it now has an end-to-end cache-ON gate on `Qwen/Qwen3-4B` (a shared common prefix reused @@ -1594,10 +1597,6 @@ runtime-verified yet. remaining work is the async production-serving path wiring, the chat/completion response-body timing surface, and the config-gated metric families (speculative decoding, KV connector, multimodal cache, LoRA). - matching vLLM's own mapping. A behavioural CPU gate drives the reference engine - for several steps and checks the values track the run. The remaining work is - the async production-serving path wiring and the config-gated metric families - (speculative decoding, KV connector, multimodal cache, LoRA). - **SGLang RadixAttention behavior parity** — scoped 2026-07-27, W1+W2 flags now IMPLEMENTED (CPU-gated). SGLang's radix-tree prefix cache is functionally equivalent to our block-hash APC (both do automatic longest-prefix KV sharing diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index 337bbc9ac..3e1969e3d 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -6,7 +6,7 @@ // vllm_last_error(). // // vllm-cli --model --prompt "" -// [--tokenizer-config ] +// [--tokenizer-config ] [--device auto|cpu|cuda] // [--max-tokens N] [--temperature T] [--top-p P] [--top-k K] // [--seed S] [--stream] // @@ -36,13 +36,17 @@ struct Args { bool have_seed = false; bool stream = false; std::string speculative_config; // vLLM --speculative-config JSON; "" => off. + // --device (ABI v14): "auto" (default probe), "cpu", or "cuda" — the names + // of vLLM's DeviceConfig.device this build serves. Mapped to the int the ABI + // takes (0/1/2) in ParseArgs; an unknown name is rejected there. + int32_t device = 0; }; void Usage(const char* argv0, std::FILE* out) { std::fprintf( out, "usage: %s --model --prompt \"\"\n" - " [--tokenizer-config ]\n" + " [--tokenizer-config ] [--device auto|cpu|cuda]\n" " [--max-tokens N] [--temperature T] [--top-p P] [--top-k K]\n" " [--seed S] [--stream]\n" " [--speculative-config '']\n" @@ -90,6 +94,26 @@ bool ParseArgs(int argc, char** argv, Args& a, int& exit_code) { a.stream = true; } else if (flag == "--speculative-config") { a.speculative_config = NextArg(argc, argv, i); + } else if (flag == "--device") { + // The vLLM DeviceConfig.device names (auto/cpu/cuda) -> the ABI int + // (vllm_model_params.device: 0=auto, 1=cpu, 2=cuda). An unknown name is + // a usage error, mirroring vLLM rejecting a non-Literal device value. + const std::string device = NextArg(argc, argv, i); + if (device == "auto") { + a.device = 0; + } else if (device == "cpu") { + a.device = 1; + } else if (device == "cuda") { + a.device = 2; + } else { + std::fprintf(stderr, + "vllm-cli: unknown --device '%s' (expected auto, cpu, or " + "cuda)\n", + device.c_str()); + Usage(argv[0], stderr); + exit_code = 2; + return false; + } } else if (flag == "-h" || flag == "--help") { Usage(argv[0], stdout); exit_code = 0; @@ -150,6 +174,10 @@ int main(int argc, char** argv) { if (!args.speculative_config.empty()) { mp.speculative_config = args.speculative_config.c_str(); } + // --device: explicit device selection (ABI v14). 0 (the default) keeps the + // accelerator-first probe; an explicitly named absent device fails the load + // below with the library's message (never a silent fallback). + mp.device = args.device; vllm_engine* engine = nullptr; std::fprintf(stderr, "vllm-cli: loading model from %s\n", diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 66777c48c..a06d944d4 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -53,6 +53,7 @@ #include #endif +#include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" #include "vllm/config/scheduler.h" #include "vllm/entrypoints/chat_template.h" @@ -133,6 +134,14 @@ struct Args { int max_model_len = 0; // 0 => config.max_position_embeddings int max_num_seqs = 8; int max_num_batched_tokens = 0; // 0 => per-architecture default. + // --device: explicit device selection for the TEXT engine (ARCH-ONE-SURFACE + // ROW 8), the vLLM DeviceConfig.device names this build serves: "auto" + // (default — the accelerator-first probe, byte-identical to before the flag), + // "cpu" (force the CPU queue), "cuda" (require CUDA; an absent device fails + // startup LOUDLY, never a silent fallback). The video engine keeps its own + // --video-device below: the two engines are loaded from different + // checkpoints and may legitimately serve on different devices. + std::string device = "auto"; // --- MiniMax-H3 video generation (opt-in; absent => /v1/videos is unregistered // and the server behaves exactly as before). --- std::string video_dit, video_vae, video_vae_config, audio_vae, audio_vae_config; @@ -198,6 +207,7 @@ struct Args { "[--num-blocks N] [--max-model-len N]\n" " [--max-num-seqs N] " "[--max-num-batched-tokens N]\n" + " [--device auto|cpu|cuda]\n" " [--cuda-profile-graph-replays N]\n" " [--cuda-profile-graph-batch N]\n" " [--benchmark-shutdown-fifo F]\n" @@ -244,6 +254,11 @@ Args ParseArgs(int argc, char** argv) { a.max_num_seqs = std::stoi(NextArg(argc, argv, i, argv[0])); } else if (flag == "--max-num-batched-tokens") { a.max_num_batched_tokens = std::stoi(NextArg(argc, argv, i, argv[0])); + } else if (flag == "--device") { + // Text-engine device selection (mirrors vLLM's DeviceConfig.device + // names). Validated by vllm::DeviceFromString at engine construction; + // --video-device (below) stays the video engine's separate knob. + a.device = NextArg(argc, argv, i, argv[0]); } else if (flag == "--cuda-profile-graph-replays") { a.cuda_profile_graph_replays = std::stoi(NextArg(argc, argv, i, argv[0])); @@ -457,6 +472,12 @@ int main(int argc, char** argv) { engine_params.max_num_seqs = args.max_num_seqs; engine_params.max_num_batched_tokens = args.max_num_batched_tokens; engine_params.enable_prefix_caching = args.enable_prefix_caching; + // --device: explicit device selection (ARCH-ONE-SURFACE ROW 8). "auto" + // (default) keeps the accelerator-first probe byte-identical; an unknown + // name throws HERE (a startup error), and an explicitly named ABSENT + // device fails FromModelDir loudly — never a silent fallback + // (vllm/config/device.py:61-66). + engine_params.device = vllm::DeviceFromString(args.device); // Reject an unknown policy string (mirrors upstream SchedulingPolicy(value)). engine_params.policy = vllm::SchedulerPolicyFromString(args.scheduling_policy); // ENG-SGLANG-BEHAVIOR-FLAG (SW1): `lpm` needs prefix caching to have any diff --git a/include/vllm.h b/include/vllm.h index 1880348e3..a276381cb 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -82,6 +82,15 @@ extern "C" { * default, so zero-filling the struct growth keeps a v9 engine byte-identical. * Scheduler policy (incl. SGLang's cache-aware LPM) is selected through the v9 * string field .scheduling_policy = "lpm" — there is no separate int knob. + * v11: AUDIO TRANSCRIPTION (ARCH-ONE-SURFACE fold #4) — vllm_transcribe / + * vllm_transcription_params(_default) / vllm_transcription(_free), appended so + * zero values preserve behaviour. vllm_engine_load now RESOLVES a + * transcription-only checkpoint (Parakeet CTC/RNNT/TDT, the vLLM + * SupportsTranscription mirror) into a transcription engine: the text entry + * points on such a handle report VLLM_ERR_INVALID_ARGUMENT with an actionable + * message instead of serving, and vllm_transcribe on a TEXT handle does the + * same. A pre-v11 caller that never loads a Parakeet directory is + * byte-identical. * v12: VIDEO+AUDIO GENERATION (ARCH-ONE-SURFACE ROW 2, MiniMax-H3) — the * ratified video slice: an opaque vllm_video_engine loaded from the H3 * checkpoint set (vllm_video_engine_load/free, vllm_video_model_params + @@ -94,23 +103,23 @@ extern "C" { * handles refuse each other's checkpoints LOUDLY: vllm_video_engine_load on a * text-model directory names vllm_engine_load, and vllm_engine_load on an H3 * checkpoint directory keeps failing exactly as at v11 (no config.json). - * v11: AUDIO TRANSCRIPTION (ARCH-ONE-SURFACE fold #4) — vllm_transcribe / - * vllm_transcription_params(_default) / vllm_transcription(_free), appended so - * zero values preserve behaviour. vllm_engine_load now RESOLVES a - * transcription-only checkpoint (Parakeet CTC/RNNT/TDT, the vLLM - * SupportsTranscription mirror) into a transcription engine: the text entry - * points on such a handle report VLLM_ERR_INVALID_ARGUMENT with an actionable - * message instead of serving, and vllm_transcribe on a TEXT handle does the - * same. A pre-v11 caller that never loads a Parakeet directory is - * byte-identical. * v13: vllm_complete_tokens — blocking completion from a PRE-TOKENIZED prompt * (vLLM's TokensPrompt), returning the generated token ids (and optionally the * detokenized vllm_completion). The entry point for embedders that manage * their own tokenization and for token-exact gates/benchmarks that compare * whole token streams against a reference (the Kimi-Linear paged-runner fold * battery, ARCH-ONE-SURFACE ROW 7, is the first consumer). Purely additive — - * no struct changed. */ -#define VLLM_ABI_VERSION 13 + * no struct changed. + * v14: vllm_model_params.device — EXPLICIT DEVICE SELECTION (ARCH-ONE-SURFACE + * fold ROW 8), the mirror of vLLM's DeviceConfig.device names + * (vllm/config/device.py:13): 0=auto (the byte-identical default — the + * accelerator-first platform probe that has always selected the queue), 1=cpu + * (force the CPU queue even on an accelerator build), 2=cuda (require the + * CUDA platform; when it is absent the load FAILS with VLLM_ERR_MODEL_LOAD — + * an explicitly named device is never silently substituted, device.py:61-66). + * Appended at the END of vllm_model_params so a zero-initialized struct keeps + * the pre-v14 engine byte-identical. */ +#define VLLM_ABI_VERSION 14 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -257,6 +266,24 @@ typedef struct vllm_model_params { * SGLang's cache-aware LPM) is a SEPARATE knob — the v9 string field * .scheduling_policy = "lpm", not an int here. */ int32_t enable_jump_forward; + /* ── Device selection (ABI v14) ──────────────────────────────────────────── + * Which device the text-generation engine serves on, mirroring vLLM's + * DeviceConfig.device names (vllm/config/device.py:13 — Device = + * Literal["auto", "cuda", "cpu", ...]): + * 0 => AUTO (the byte-identical default): the accelerator-first platform + * probe that has always selected the queue (CUDA first, CPU + * fallback; on other builds ROCm/XPU/Vulkan/Metal probe in between); + * 1 => CPU: force the CPU queue even when an accelerator is available; + * 2 => CUDA: require the CUDA platform. When it is absent in this + * build/process the load FAILS with VLLM_ERR_MODEL_LOAD and a + * message naming the device — an explicitly named device is NEVER + * silently replaced by another (mirror of vLLM assigning an explicit + * device verbatim, device.py:61-66). + * 0 must stay auto so a zero-initialized struct preserves pre-v14 behaviour; + * the cpu-before-cuda value order follows the v12 precedent + * (vllm_video_model_params.device: 0 cpu, 1 cuda) shifted by the auto slot. + * Any other value fails vllm_engine_load with VLLM_ERR_INVALID_ARGUMENT. */ + int32_t device; } vllm_model_params; /* ── Custom logits processor (ABI v8) ───────────────────────────────────────── diff --git a/include/vllm/config/device.h b/include/vllm/config/device.h new file mode 100644 index 000000000..db90a7732 --- /dev/null +++ b/include/vllm/config/device.h @@ -0,0 +1,47 @@ +// Ported from: vllm/config/device.py @ 555967922 (Device Literal:13, +// DeviceConfig:16-78). +// +// Scope (ARCH-ONE-SURFACE fold ROW 8): the explicit device-selection surface — +// the mirror of vLLM's `DeviceConfig.device` names. Upstream the value set is +// `Device = Literal["auto", "cuda", "cpu", "tpu", "xpu"]` (device.py:13); +// "auto" resolves through the platform probe (device.py:49-60) and an explicit +// name is assigned VERBATIM, never silently substituted (device.py:61-66). +// +// PORT NOTES (recorded deviations): +// - Only the members this build can serve are selectable: auto/cpu/cuda. +// "tpu" has no backend here; xpu/vulkan/metal/rocm are reachable through +// the AUTO accelerator-first probe (platform.cpp kCurrentPriority) and +// gain explicit names ADDITIVELY when a lane needs them. +// - The integer values are the C-ABI wire contract +// (vllm_model_params.device, ABI v14): 0 MUST be auto so a zero-initialized +// struct preserves the pre-v14 accelerator-first behaviour byte for byte; +// cpu-before-cuda then follows the shipped v12 precedent +// (vllm_video_model_params.device: 0 cpu, 1 cuda) shifted by the auto slot. +#ifndef VLLM_CONFIG_DEVICE_H_ +#define VLLM_CONFIG_DEVICE_H_ + +#include +#include + +namespace vllm { + +// The device selection an engine is asked to serve on. kAuto is the default +// and the zero value: the accelerator-first platform probe that has always +// selected the queue (CurrentPlatform(), src/vllm/platforms/platform.cpp). +enum class Device : int32_t { + kAuto = 0, // platform-probed, upstream's "auto" default (device.py:20). + kCPU = 1, // force the CPU queue even when an accelerator is present. + kCUDA = 2, // require the CUDA platform; absent => the load fails LOUD. +}; + +// Parse the wire/CLI name ("auto" | "cpu" | "cuda" — the supported subset of +// upstream's Device Literal, device.py:13). Throws std::invalid_argument on +// any other name, mirroring pydantic rejecting a non-Literal value. +Device DeviceFromString(const std::string& value); + +// The canonical name for a selection (static storage; never nullptr). +const char* DeviceName(Device device); + +} // namespace vllm + +#endif // VLLM_CONFIG_DEVICE_H_ diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index 4cf230df1..bdb97e01c 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -13,6 +13,7 @@ #include #include +#include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" #include "vllm/config/scheduler.h" #include "vllm/config/speculative.h" @@ -114,6 +115,17 @@ struct EngineParams { // only enables the seam. Non-safetensors (GGUF) checkpoints lack `mtp.*`, so an // MTP config over a GGUF source is rejected. std::optional speculative_config = std::nullopt; + + // ARCH-ONE-SURFACE fold ROW 8: explicit device selection, the mirror of + // vLLM's DeviceConfig.device (vllm/config/device.py). kAuto (default) keeps + // the accelerator-first probe that has always selected the queue — the + // byte-identical default. kCPU forces the CPU queue without consulting the + // probe; kCUDA requires the CUDA platform and the load fails LOUD when it is + // absent (never a silent fallback — an explicit device is assigned verbatim + // upstream, device.py:61-66). Exposed on the C ABI as + // vllm_model_params.device (ABI v14: 0=auto, 1=cpu, 2=cuda) and on the + // server as --device. + vllm::Device device = vllm::Device::kAuto; }; // Owns the full V1 engine stack (config + weights + tokenizer + Scheduler + @@ -170,6 +182,23 @@ class LoadedEngine { int max_model_len, bool is_dense_arch); static bool ResolveEnablePrefixCaching(const EngineParams& params, const ModelInfo& model_info); + // ARCH-ONE-SURFACE ROW 8: the EXPLICIT arms of the device-selection policy + // behind SelectQueue, factored pure over the "is the CUDA platform + // registered" probe answer so the CPU tier can gate the whole matrix without + // registering fake global platforms: + // * kCPU -> vt::DeviceType::kCPU unconditionally — an explicit CPU ask + // never consults the accelerator probe, even when CUDA is registered; + // * kCUDA -> vt::DeviceType::kCUDA when cuda_platform_registered, else + // THROWS std::runtime_error naming the device (fail LOUD; the mirror of + // vLLM assigning an explicit device verbatim and never substituting + // another — vllm/config/device.py:61-66); + // * kAuto is NOT resolved here (it resolves through the accelerator-first + // probe inside SelectQueue, byte-identical to pre-ROW-8) and throws + // std::invalid_argument if passed. + // SelectQueue routes its explicit arms through THIS function, so the gate on + // it pins the production policy, not a parallel copy. + static vt::DeviceType ResolveExplicitDeviceType(vllm::Device requested, + bool cuda_platform_registered); vllm::v1::LLMEngine& engine() { return engine_; } // Lazily start W2's EngineCoreProc + output-handler threads. Once created, diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index 721bdee4f..cf23d414b 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -375,7 +375,14 @@ def features_errors(text: str) -> list[str]: # subgroup-reduction outcome, paid for by dropping the native-kernel COUNT, # which is derivable from the linked campaign doc while a void result is not # derivable from anywhere. Net -0. - "chars": 279200, + # + # 279150 since 2026-08-08 (measured 279111 after the #122 rebase): the ONE-SURFACE ROW 8 device + # knob owes this page a status line, paid for by removing a STALE MERGE + # DUPLICATE in the metrics paragraph (the same "matching vLLM's own + # mapping ... behavioural CPU gate ... remaining work" narrative appeared + # twice back to back; the longer, newer version stays). Nothing was lost, + # only de-duplicated. Net -92. + "chars": 279150, "h2_sections": 11, "long_paragraphs": 89, "oversized_cells": 47, diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index ae5e90613..372e52e54 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -464,6 +464,7 @@ VLLM_API vllm_model_params vllm_model_params_default(void) { p.scheduling_policy = nullptr; // NULL => "fcfs" (ABI v9). p.kv_transfer_config = nullptr; // NULL => no connector (ABI v9). p.enable_jump_forward = 0; // 0 => env-resolved, default OFF (ABI v10). + p.device = 0; // 0 => auto: the accelerator-first probe (ABI v14). return p; } @@ -601,6 +602,26 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, "or 2 (off)"); return VLLM_ERR_INVALID_ARGUMENT; } + // ABI v14: explicit device selection (0=auto, 1=cpu, 2=cuda), mirroring + // vLLM's DeviceConfig.device names (vllm/config/device.py:13). 0 leaves + // ep.device at kAuto — the byte-identical accelerator-first probe. An + // explicitly named device that is ABSENT fails inside FromModelDir, before + // any model I/O, and reports VLLM_ERR_MODEL_LOAD with a message naming the + // device (never a silent fallback — device.py:61-66). + switch (params->device) { + case 0: + break; // auto (the default probe) — ep.device stays kAuto. + case 1: + ep.device = vllm::Device::kCPU; + break; + case 2: + ep.device = vllm::Device::kCUDA; + break; + default: + SetError( + "vllm_engine_load: device must be 0 (auto), 1 (cpu), or 2 (cuda)"); + return VLLM_ERR_INVALID_ARGUMENT; + } // ABI v11 task dispatch: a directory whose config.json architectures // resolve to a SupportsTranscription-ONLY registration (Parakeet @@ -621,6 +642,17 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, peek = nullptr; } if (peek != nullptr && peek->info.supports_transcription_only) { + // ABI v14: the transcription stack is a CPU-hosted pipeline; an + // explicit CUDA ask cannot be served and is REFUSED rather than + // silently downgraded (the same never-substitute rule as the text + // engine, vllm/config/device.py:61-66). + if (ep.device == vllm::Device::kCUDA) { + SetError( + "vllm_engine_load: device 'cuda' was requested but this " + "transcription-only checkpoint serves on the CPU pipeline; use " + "device=auto or device=cpu"); + return VLLM_ERR_INVALID_ARGUMENT; + } auto* handle = new vllm_engine; handle->transcriber = std::make_unique( diff --git a/src/vllm/config/device.cpp b/src/vllm/config/device.cpp new file mode 100644 index 000000000..a73969553 --- /dev/null +++ b/src/vllm/config/device.cpp @@ -0,0 +1,40 @@ +// See include/vllm/config/device.h. Ported from: vllm/config/device.py @ +// 555967922 (Device Literal:13 — the name set; DeviceConfig:61-66 — an explicit +// device is assigned verbatim, never substituted). +#include "vllm/config/device.h" + +#include + +namespace vllm { + +Device DeviceFromString(const std::string& value) { + if (value == "auto") { + return Device::kAuto; + } + if (value == "cpu") { + return Device::kCPU; + } + if (value == "cuda") { + return Device::kCUDA; + } + // Mirrors pydantic rejecting a value outside the Device Literal + // (vllm/config/device.py:13). "tpu"/"xpu" are upstream names this build + // cannot serve explicitly yet; they are rejected with the same message shape + // rather than mapped to something else. + throw std::invalid_argument("Unknown device: " + value + + " (expected one of: auto, cpu, cuda)"); +} + +const char* DeviceName(Device device) { + switch (device) { + case Device::kAuto: + return "auto"; + case Device::kCPU: + return "cpu"; + case Device::kCUDA: + return "cuda"; + } + return "invalid"; +} + +} // namespace vllm diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 00c778eee..b9685d8a7 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -46,7 +46,25 @@ namespace { // a PARTIAL backend decline a model whose kernels it has not registered, instead // of being selected and then failing deep inside a kernel bind. Empty means "no // model resolved yet", which is treated as no constraint. -vt::Queue SelectQueue(std::string_view architecture) { +// +// ARCH-ONE-SURFACE ROW 8: `device` is the caller's explicit selection +// (EngineParams::device / vllm_model_params.device). kAuto keeps the +// accelerator-first probe below byte-identical; an EXPLICIT selection routes +// through LoadedEngine::ResolveExplicitDeviceType and — unlike the auto arm — +// a failure to serve the named device PROPAGATES instead of falling back to +// CPU (mirror of vLLM never substituting an explicitly named device, +// vllm/config/device.py:61-66). +vt::Queue SelectQueue(std::string_view architecture, vllm::Device device) { + if (device != vllm::Device::kAuto) { + const vt::DeviceType resolved = LoadedEngine::ResolveExplicitDeviceType( + device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + if (resolved == vt::DeviceType::kCPU) { + return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + } + // No try/catch here on purpose: an explicit accelerator whose queue cannot + // be created must FAIL the load loudly, never silently serve on CPU. + return vt::GetBackend(resolved).CreateQueue(); + } // M2.2b: run the engine forward on the ACCELERATOR when one is available, so // (on CUDA/GB10) the fp4-resident MoE/lm_head weights hit vt::MatmulNvfp4 // on-device instead of the CPU dequant reference. @@ -458,6 +476,39 @@ bool LoadedEngine::ResolveEnablePrefixCaching(const EngineParams& params, return !model_info.is_hybrid && !model_info.has_inner_state; } +// ARCH-ONE-SURFACE ROW 8: the explicit arms of the device-selection policy — +// see the contract in model_loader.h. Ported semantics: +// vllm/config/device.py:61-66 @ 555967922 (an explicit device string is +// assigned VERBATIM — never substituted), with the loud failure upstream +// raises when the named device cannot serve (torch/worker init on an absent +// CUDA device; our analogue is the unregistered kCUDA platform, +// src/vllm/platforms/cuda.cpp Registrar — kCUDA registers only when a usable +// GPU probed). +vt::DeviceType LoadedEngine::ResolveExplicitDeviceType( + vllm::Device requested, bool cuda_platform_registered) { + switch (requested) { + case vllm::Device::kCPU: + // Explicit CPU never consults the accelerator probe: even on a + // CUDA-capable build/process this selects the CPU queue. + return vt::DeviceType::kCPU; + case vllm::Device::kCUDA: + if (!cuda_platform_registered) { + throw std::runtime_error( + "device 'cuda' was requested but no CUDA platform is available in " + "this build/process (an explicitly named device is never silently " + "replaced — mirror of vllm/config/device.py:61-66; use device=auto " + "or device=cpu, or run a CUDA build on a machine with a usable " + "GPU)"); + } + return vt::DeviceType::kCUDA; + case vllm::Device::kAuto: + break; // auto resolves through the probe in SelectQueue, not here. + } + throw std::invalid_argument( + "ResolveExplicitDeviceType resolves only explicit device selections " + "(cpu/cuda); auto resolves through the accelerator-first probe"); +} + bool LoadedEngine::EnsureNoneHash() { // Idempotent: init_none_hash just (re)assigns the NONE_HASH global. // @@ -648,7 +699,8 @@ LoadedEngine::LoadedEngine(HfConfig config, runner_(config_, *model_, kv_cfg_, preselected_queue != nullptr ? *preselected_queue - : SelectQueue(model_->registration().architecture), + : SelectQueue(model_->registration().architecture, + params.device), /*max_num_reqs=*/params.max_num_seqs > 0 ? params.max_num_seqs : 8, max_model_len_, /*max_num_batched_tokens=*/max_num_batched_tokens_, @@ -813,6 +865,19 @@ vllm::v1::AsyncLLM& LoadedEngine::async_engine() { std::unique_ptr LoadedEngine::FromModelDir( const std::string& model_dir, const EngineParams& params) { + // ARCH-ONE-SURFACE ROW 8: resolve an EXPLICIT device selection up front, + // BEFORE any path/config/weight I/O — the mirror of vLLM resolving + // DeviceConfig at config-creation time, ahead of the model load + // (vllm/engine/arg_utils.py:1878 builds DeviceConfig first; + // device.py __post_init__ resolves immediately). An explicitly named absent + // device therefore fails HERE, loudly, and is never masked by a later + // path/tokenizer error. The result is discarded: SelectQueue re-runs the + // SAME ResolveExplicitDeviceType when it actually creates the queue, so the + // policy has exactly one owner. + if (params.device != vllm::Device::kAuto) { + (void)ResolveExplicitDeviceType( + params.device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + } const fs::path dir(model_dir); // A single `.gguf` file: config + weights + tokenizer all come from the @@ -997,7 +1062,7 @@ std::unique_ptr LoadedEngine::FromModelDir( // Select before loading so an eligible discrete-CUDA dense loader stages each // completed layer to the exact queue the runner will use. If construction // fails before the runner takes over, destroy the selected native stream. - vt::Queue load_queue = SelectQueue(registration.architecture); + vt::Queue load_queue = SelectQueue(registration.architecture, params.device); try { std::unique_ptr model = ModelRegistry::Load( config, ModelSource::FromSafetensorsOwned(shards, &load_queue)); diff --git a/tests/capi/c_header_compile.c b/tests/capi/c_header_compile.c index e5a332941..669e4f013 100644 --- a/tests/capi/c_header_compile.c +++ b/tests/capi/c_header_compile.c @@ -17,7 +17,10 @@ static bool c_header_token_cb(const char* delta_text, bool finished, } /* Instantiate the POD structs + a status value so the C compiler actually lays - * them out, and reference every ABI entry point so the declarations are used. */ + * them out, and reference every ABI entry point so the declarations are used — + * including the v11 transcription slice, the v12 video slice, and the v14 + * device field (this file went stale between v10 and v12; the claim above is + * only honest if new surface lands HERE in the same change). */ int vllm_capi_c_header_check(vllm_engine* eng, const char* prompt) { vllm_model_params mp = vllm_model_params_default(); vllm_sampling_params sp = vllm_sampling_params_default(); @@ -26,6 +29,8 @@ int vllm_capi_c_header_check(vllm_engine* eng, const char* prompt) { vllm_request* request = NULL; vllm_status st = VLLM_OK; + mp.device = 0; /* ABI v14: 0=auto (the accelerator-first probe default). */ + st = vllm_engine_load(&mp, &eng); if (st == VLLM_OK) { st = vllm_complete(eng, prompt, &sp, &out); @@ -48,9 +53,48 @@ int vllm_capi_c_header_check(vllm_engine* eng, const char* prompt) { st = vllm_request_wait(request); vllm_request_free(request); } + + /* Chat entry points (ABI v3). */ + { + char* response_json = NULL; + st = vllm_chat(eng, "{}", &response_json); + vllm_string_free(response_json); + st = vllm_chat_stream(eng, "{}", cb, /*user_data=*/NULL); + } + + /* Audio transcription (ABI v11). */ + { + vllm_transcription_params tp = vllm_transcription_params_default(); + vllm_transcription transcript; + st = vllm_transcribe(eng, &tp, &transcript); + vllm_transcription_free(&transcript); + } + vllm_engine_free(eng); } + /* Video+audio generation (ABI v12): the separate vllm_video_engine handle + * plus the engine-free mux-argv composer. */ + { + vllm_video_model_params vmp = vllm_video_model_params_default(); + vllm_video_params vp = vllm_video_params_default(); + vllm_video_engine* veng = NULL; + st = vllm_video_engine_load(&vmp, &veng); + if (st == VLLM_OK) { + vllm_video_result vres; + st = vllm_video_generate(veng, &vp, &vres); + vllm_video_result_free(&vres); + vllm_video_engine_free(veng); + } + { + vllm_video_mux_params mux = vllm_video_mux_params_default(); + char** mux_argv = NULL; + int32_t mux_argc = 0; + st = vllm_video_mux_argv(&mux, &mux_argv, &mux_argc); + vllm_video_mux_argv_free(mux_argv, mux_argc); + } + } + (void)vllm_last_error(); (void)vllm_version(); (void)vllm_abi_version(); diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index 96df56eca..c25d842b7 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -25,7 +26,9 @@ #include #include "capi/engine_handle.h" +#include "vllm/config/device.h" #include "vllm/entrypoints/model_loader.h" +#include "vllm/platforms/interface.h" #include "vllm/entrypoints/openai/serving_utils.h" #include "vllm/model_executor/models/qwen3_5_weights.h" #include "vllm/tokenizer/bpe.h" @@ -1291,10 +1294,13 @@ TEST_CASE("capi: version and abi-version are exposed") { // The engine-config growth (max_num_batched_tokens / scheduling_policy / // kv_transfer_config) is ABI v9; the jump-forward toggle is ABI v10; the // transcription slice (vllm_transcribe) is ABI v11; the video-generation - // slice (vllm_video_*) is ABI v12. The >= pin is the one check that can - // catch a WRONG bump: the == VLLM_ABI_VERSION assertions here and in - // test_dlopen compare against the same macro and move with it. - CHECK(vllm_abi_version() >= 13); + // slice (vllm_video_*) is ABI v12; the pre-tokenized completion entry + // point (vllm_complete_tokens) is ABI v13; the device-selection field + // (vllm_model_params.device) is ABI v14. The >= pin is the one check that + // can catch a WRONG bump: the == VLLM_ABI_VERSION assertions here and in + // test_dlopen compare against the same macro and move with it (the #121 + // lesson: an == floor moves with the macro and proves nothing). + CHECK(vllm_abi_version() >= 14); } // ─── ABI v11: audio transcription (ARCH-ONE-SURFACE ROW 1) ─────────────────── @@ -1694,3 +1700,142 @@ TEST_CASE("capi v12: video argument contract") { vllm_video_mux_argv_free(nullptr, 3); // no-op } + +// ─── ABI v14: explicit device selection (ARCH-ONE-SURFACE ROW 8) ───────────── +// The device knob: 0=auto (the byte-identical accelerator-first probe), 1=cpu, +// 2=cuda — the vLLM DeviceConfig.device names (vllm/config/device.py:13). The +// pure policy matrix (explicit cpu beats a registered accelerator; explicit +// cuda never falls back) is gated in test_loaded_engine_dense.cpp; here the +// C-ABI wire contract and the params->EngineParams->SelectQueue plumb. + +namespace { +// A synthetic engine over an explicit EngineParams device selection, sharing +// MakeSyntheticEngine's stack. +vllm_engine* MakeSyntheticEngineWithDevice(vllm::Device device) { + const HfConfig c = MakeConfig(); + EngineParams params = SyntheticParams(); + params.device = device; + auto loaded = std::make_unique(c, MakeWeights(c), BuildFixture(), + params); + return vllm::capi::MakeEngineHandle(std::move(loaded)); +} +} // namespace + +TEST_CASE("capi v14: the device zero-value/default contract") { + // The default is 0 == auto — a zero-initialized struct and + // vllm_model_params_default() agree, so a pre-v14 caller's zero-filled + // growth keeps the accelerator-first probe engine byte-identical. + const vllm_model_params def = vllm_model_params_default(); + CHECK(def.device == 0); + vllm_model_params zeroed; + std::memset(&zeroed, 0, sizeof(zeroed)); + CHECK(zeroed.device == def.device); + + // The zero value maps to AUTO, not to an explicit device: with device left + // at the default and a bogus path, the load must report the PATH (the auto + // arm defers to the probe and never resolves an explicit device up front). + // A mutation that maps 0 to an explicit cuda would surface the device error + // here instead — on the CPU tier that is the distinguishable half; 0-as- + // explicit-cpu is behaviorally identical on this tier and is pinned by the + // pure policy matrix in test_loaded_engine_dense.cpp plus the CUDA-build + // residual named in the spec. + vllm_model_params bogus = vllm_model_params_default(); + bogus.model_path = "/nonexistent/vllm-cpp/model/dir"; + vllm_engine* probe_eng = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&bogus, &probe_eng) == VLLM_ERR_MODEL_LOAD); + CHECK(probe_eng == nullptr); + CHECK(std::string(vllm_last_error()).find("not a directory") != + std::string::npos); + + // Behavior: on a CPU-only process the auto probe resolves CPU, so an + // explicit-cpu engine must generate EXACTLY what the default (auto) engine + // generates. (Guarded: on an accelerator build auto legitimately selects the + // accelerator, and byte-equality with a CPU run is not the contract.) + if (!vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + vllm_engine* auto_eng = MakeSyntheticEngine(); // device unset == kAuto + vllm_engine* cpu_eng = MakeSyntheticEngineWithDevice(vllm::Device::kCPU); + REQUIRE(auto_eng != nullptr); + REQUIRE(cpu_eng != nullptr); + vllm_sampling_params sp = GreedyParams(6); + vllm_completion a{}; + vllm_completion b{}; + REQUIRE(vllm_complete(auto_eng, "hello world", &sp, &a) == VLLM_OK); + REQUIRE(vllm_complete(cpu_eng, "hello world", &sp, &b) == VLLM_OK); + CHECK(std::string(a.text) == std::string(b.text)); + CHECK(a.completion_tokens == b.completion_tokens); + vllm_completion_free(&a); + vllm_completion_free(&b); + vllm_engine_free(auto_eng); + vllm_engine_free(cpu_eng); + } +} + +TEST_CASE("capi v14: device range validates before any load work") { + // An out-of-range device is a CALLER error caught before FromModelDir — the + // bogus path must NOT be what fails here. + for (const int32_t bad : {static_cast(3), static_cast(-1), + static_cast(7)}) { + vllm_model_params p = vllm_model_params_default(); + p.model_path = "/nonexistent/vllm-cpp/model/dir"; + p.device = bad; + vllm_engine* eng = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&p, &eng) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(eng == nullptr); + CHECK(std::string(vllm_last_error()) + .find("device must be 0 (auto), 1 (cpu), or 2 (cuda)") != + std::string::npos); + } +} + +TEST_CASE("capi v14: explicit cuda on a CUDA-less process fails LOUD (plumb pin)") { + if (vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + return; // CUDA build/box: the explicit-cuda arm resolves; nothing to pin. + } + // device=2 with a bogus path: the DEVICE error must surface, not the path + // error — FromModelDir resolves an explicit device BEFORE any I/O, so this + // pins the whole capi -> EngineParams -> FromModelDir plumb (an implementation + // that drops params->device would report the path instead and go RED here). + vllm_model_params p = vllm_model_params_default(); + p.model_path = "/nonexistent/vllm-cpp/model/dir"; + p.device = 2; // cuda + vllm_engine* eng = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&p, &eng) == VLLM_ERR_MODEL_LOAD); + CHECK(eng == nullptr); + CHECK(std::string(vllm_last_error()) + .find("device 'cuda' was requested but no CUDA platform") != + std::string::npos); + + // device=1 (cpu) on the same bogus path proceeds to the path and reports IT — + // the plumb forwards the field's VALUE, not a constant. + p.device = 1; // cpu + eng = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&p, &eng) == VLLM_ERR_MODEL_LOAD); + CHECK(eng == nullptr); + CHECK(std::string(vllm_last_error()).find("not a directory") != + std::string::npos); +} + +TEST_CASE("capi v14: explicit cpu forces the CPU queue at the EngineParams seam") { + // The observable queue seam: an engine constructed with device=kCPU runs its + // runner on the CPU device. On the CPU tier this is trivially true of auto as + // well; on a CUDA build it is the force-CPU pin (the runner would otherwise + // sit on the CUDA queue). The CPU-tier statement of "cpu beats a registered + // accelerator" is the pure matrix in test_loaded_engine_dense.cpp. + const HfConfig c = MakeConfig(); + EngineParams params = SyntheticParams(); + params.device = vllm::Device::kCPU; + LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), params); + CHECK(loaded.runner().device().type == vt::DeviceType::kCPU); + + // And the ctor-path plumb (LoadedEngine's own SelectQueue call, the arm the + // GGUF/MoE branches take): explicit cuda on a CUDA-less process must throw + // the pinned message out of construction — never silently build on CPU. + if (!vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + EngineParams cuda_params = SyntheticParams(); + cuda_params.device = vllm::Device::kCUDA; + CHECK_THROWS_WITH_AS( + LoadedEngine(MakeConfig(), MakeWeights(c), BuildFixture(), cuda_params), + doctest::Contains("device 'cuda' was requested but no CUDA platform"), + std::runtime_error); + } +} diff --git a/tests/vllm/entrypoints/openai/test_api_server.cpp b/tests/vllm/entrypoints/openai/test_api_server.cpp index f0123466d..7a02c86ad 100644 --- a/tests/vllm/entrypoints/openai/test_api_server.cpp +++ b/tests/vllm/entrypoints/openai/test_api_server.cpp @@ -46,7 +46,9 @@ #include #endif +#include "vllm/config/device.h" #include "vllm/config/scheduler.h" +#include "vllm/entrypoints/model_loader.h" #include "vllm/entrypoints/openai/serving_chat.h" #include "vllm/entrypoints/openai/serving_completion.h" #include "vllm/entrypoints/openai/serving_models.h" @@ -2309,3 +2311,45 @@ TEST_CASE("api_server: the audio routes do not exist on a TEXT server") { h.server.stop(); server_thread.join(); } + +// ─── ARCH-ONE-SURFACE ROW 8: the server's --device seam ────────────────────── +// The exact chain examples/server/main.cpp drives for `--device cpu`: +// vllm::DeviceFromString -> EngineParams.device -> LoadedEngine (SelectQueue's +// explicit arm) -> async_engine() -> the OpenAI serving stack — here over the +// synthetic in-memory model (no disk), asserting the device-selected engine +// SERVES and sits on the CPU queue. The policy matrix (cpu beats a registered +// accelerator; explicit cuda never falls back) is test_loaded_engine_dense.cpp; +// the C-ABI plumb is test_capi.cpp. +TEST_CASE("api_server: an explicit-cpu device-selected engine serves /v1/completions") { + const HfConfig c = MakeConfig(); + vllm::entrypoints::EngineParams params; + params.block_size = kBlockSize; + params.num_blocks = 32; + params.max_model_len = kMaxModelLen; + params.max_num_seqs = 8; + // The server's own parse of `--device cpu` (an unknown name throws there at + // startup; pinned in test_loaded_engine_dense.cpp). + params.device = vllm::DeviceFromString("cpu"); + vllm::entrypoints::LoadedEngine loaded(c, MakeWeights(c), BuildFixture(), + params); + // The observable seam: the runner of the explicitly-cpu engine is on the CPU + // device (on a CUDA build this is the force-CPU pin; auto would select CUDA). + CHECK(loaded.runner().device().type == vt::DeviceType::kCPU); + + OpenAIServingModels models("test-model"); + OpenAIServingCompletion completion(loaded.async_engine(), "test-model", + /*enable_force_include_usage=*/false); + OpenAIServingChat chat(loaded.async_engine(), "test-model", InVocabChatPrompt, + "hermes", /*reasoning_parser_name=*/std::string(), + /*enable_force_include_usage=*/false); + ApiServer server(completion, chat, models, "9.9.9"); + + const std::string body = + R"({"model":"test-model","prompt":"hello","max_tokens":5,"temperature":0.0})"; + ApiServer::DispatchResult r = server.handle_completions(body); + CHECK(r.status == 200); + json j = json::parse(r.body); + CHECK(j.at("object") == "text_completion"); + CHECK(j.at("choices").at(0).at("finish_reason") == "length"); + CHECK(j.at("usage").at("completion_tokens") == 5); +} diff --git a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp index c8f1d18ee..5d4974b5b 100644 --- a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp +++ b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp @@ -27,7 +27,9 @@ #include #include +#include "vllm/config/device.h" #include "vllm/config/scheduler.h" +#include "vllm/platforms/interface.h" #include "vllm/v1/core/sched/async_scheduler.h" #include @@ -447,3 +449,88 @@ TEST_CASE("loaded_engine: prefix caching mirrors model-capability defaults") { params.enable_prefix_caching = false; CHECK_FALSE(LoadedEngine::ResolveEnablePrefixCaching(params, decoder)); } + +// ─── ARCH-ONE-SURFACE ROW 8: explicit device selection ─────────────────────── +// The policy matrix behind SelectQueue's explicit arms, gated PURE over the +// "is the CUDA platform registered" probe answer so the CPU tier pins the +// whole contract — including the CUDA-build half ("explicit cpu beats a +// registered accelerator") that a CPU-only process could otherwise never +// exercise. SelectQueue routes its explicit arms through THIS function +// (model_loader.cpp), so these pins bind the production policy, not a copy. +TEST_CASE("loaded_engine: ResolveExplicitDeviceType — cpu never probes, cuda never falls back") { + using vllm::Device; + + // Explicit CPU resolves CPU regardless of what is registered. The `true` arm + // is the CUDA-build pin: a registered accelerator must NOT win over an + // explicit cpu ask (the fold-ROW-8 defect was that an embedder could not ASK + // for CPU at all). + CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, true) == + vt::DeviceType::kCPU); + CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, false) == + vt::DeviceType::kCPU); + + // Explicit CUDA with the platform registered resolves CUDA. + CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, true) == + vt::DeviceType::kCUDA); + + // Explicit CUDA WITHOUT the platform throws the pinned message — never a + // silent CPU fallback (mirror of vLLM assigning an explicit device verbatim, + // vllm/config/device.py:61-66). + CHECK_THROWS_WITH_AS( + LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, false), + doctest::Contains("device 'cuda' was requested but no CUDA platform"), + std::runtime_error); + + // kAuto is not an explicit selection: it resolves through the probe inside + // SelectQueue, and this seam refuses it rather than guessing. + CHECK_THROWS_AS(LoadedEngine::ResolveExplicitDeviceType(Device::kAuto, true), + std::invalid_argument); +} + +TEST_CASE("loaded_engine: DeviceFromString mirrors the vLLM Device names") { + using vllm::Device; + // The supported subset of upstream's Device Literal (vllm/config/device.py:13) + // — the strings the server's --device flag consumes. + CHECK(vllm::DeviceFromString("auto") == Device::kAuto); + CHECK(vllm::DeviceFromString("cpu") == Device::kCPU); + CHECK(vllm::DeviceFromString("cuda") == Device::kCUDA); + CHECK_THROWS_WITH_AS(vllm::DeviceFromString("tpu"), + doctest::Contains("Unknown device: tpu"), + std::invalid_argument); + CHECK_THROWS_AS(vllm::DeviceFromString(""), std::invalid_argument); + + // The wire contract (vllm_model_params.device, ABI v14): 0 MUST stay auto so + // a zero-initialized struct preserves pre-v14 behaviour; cpu/cuda follow the + // v12 vllm_video_model_params.device precedent (0 cpu, 1 cuda) shifted by + // the auto slot. + CHECK(static_cast(Device::kAuto) == 0); + CHECK(static_cast(Device::kCPU) == 1); + CHECK(static_cast(Device::kCUDA) == 2); + CHECK(std::string(vllm::DeviceName(Device::kAuto)) == "auto"); + CHECK(std::string(vllm::DeviceName(Device::kCPU)) == "cpu"); + CHECK(std::string(vllm::DeviceName(Device::kCUDA)) == "cuda"); +} + +TEST_CASE("loaded_engine: FromModelDir resolves an explicit absent device BEFORE any path I/O") { + // The device error must win over the path error (the mirror of vLLM building + // DeviceConfig at config-creation time, before the model load — + // arg_utils.py:1878, device.py __post_init__). This is also what makes the + // EngineParams->FromModelDir plumb pinnable on the CPU tier with no loadable + // checkpoint: a bogus path + device=cuda must report the DEVICE, not the path. + if (vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + return; // CUDA build/box: the explicit-cuda arm resolves; nothing to pin. + } + EngineParams params; + params.device = vllm::Device::kCUDA; + CHECK_THROWS_WITH_AS( + LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/model/dir", params), + doctest::Contains("device 'cuda' was requested but no CUDA platform"), + std::runtime_error); + + // An explicit CPU ask is legal and proceeds to the path (the path error, not + // a device error, surfaces) — the plumb forwards the field, not a constant. + params.device = vllm::Device::kCPU; + CHECK_THROWS_WITH_AS( + LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/model/dir", params), + doctest::Contains("not a directory"), std::runtime_error); +} diff --git a/tests/vllm/models/minimax_h3_video_fold_fixture.h b/tests/vllm/models/minimax_h3_video_fold_fixture.h index 2a850eb93..4562f8955 100644 --- a/tests/vllm/models/minimax_h3_video_fold_fixture.h +++ b/tests/vllm/models/minimax_h3_video_fold_fixture.h @@ -19,7 +19,9 @@ // hidden 64, latents 8/6) + a video-VAE whose PARSED config reproduces the // reduced ViT3D decoder of the "WHOLE t2va path composes end to end" case, and // the same reduced BigVGAN audio VAE. Render request: --partition fl2va -// --frames 5 --height 32 --width 32 --steps 3 (latent 2x2x2, audio_t 8). +// --keep-quant --frames 5 --height 32 --width 32 --steps 3 (latent 2x2x2, +// audio_t 8) — keep-quant is the arm the goldens were captured on +// (FixtureModelParams sets dequant_bf16 = 0, test_minimax_h3_video_fold.cpp). #pragma once #include