From e0e2d6451377e547b561867134d14af45fe28083 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 01:37:34 +0000 Subject: [PATCH 1/2] chore(surface): claim device-leakage repair Materialize the isolated ARCH-ONE-SURFACE helper claim and record the inherited PR #136 DSR regression as a pending correctness-only checkpoint. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/coordination.md | 14 ++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 6 +++--- 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/.agents/coordination.md b/.agents/coordination.md index c256ee1ea..f14a2faae 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,6 +118,20 @@ without the selected contention proof for their entire run are discarded. ## Active claims +**ONE-SURFACE device-selection leakage repair (`ARCH-ONE-SURFACE`, +2026-08-08, `CLAIM-ARCH-ONE-SURFACE-DEVICE-LEAKAGE`).** Codex (GPT-5), +isolated worktree +`/home/mudler/_git/vllm.cpp-arch-one-surface-device-leakage`, branch +`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`, base `origin/main` `b44ad337`. +Repairs the inherited PR #136 DSR regression without weakening the baseline or +allowlist: generic device configuration and model loading must route through +the platform/backend abstraction while preserving ABI-v14 auto/CPU/CUDA +selection, loud absent-CUDA failure, and the restored H3 seam. Owns +`include/vllm/config/device.h`, `src/vllm/config/device.cpp`, +`src/vllm/entrypoints/model_loader.cpp`, focused CPU tests, the existing +ONE-SURFACE specs/records and required public checkpoint surfaces. CPU-only; +no CUDA execution, model download, benchmark, or service change. + **CPU grouped keep-quant GEMM activation-dtype P0 (`QUANT-GGUF-CIQ-GEMM`, 2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`).** Claude Code (claude-opus-5), isolated worktree `/home/mudler/_git/vllmcpp-ciq-grouped`, diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index e39a2dc17..eb2a83513 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Track | Status | Next gate | |---|---|---| -| Surface coverage (`ARCH-ONE-SURFACE`) | **No number owed**: CPU tooling/plumbing (guard; ROW 8 device knob, ABI v14 `device`, zero value = the probe, byte-identical); no kernel path touched | Fold lanes carry their own gates; ROW 8 CUDA-build A/B = named residual | +| Surface coverage (`ARCH-ONE-SURFACE`) | **PENDING correctness-only:** #136 regressed device-seam DSR 32->39; repair removes seven shared CUDA literals through generic seams. ABI-v14 selection is unchanged; no speed number | DSR 32 with unchanged baseline/allowlist; CPU selector/loader tests + mutation review; CUDA A/B remains residual | | 35B prefill TTFT | 0.93x to 0.98x at every concurrency (2026-08-05) | Attribute the residual, then close | | 35B low-batch MoE decode | CLOSED at low batch (c1 0.975x, c4 wins); c16 0.93x. `VT_ASYNC_DEVICE_MIRROR` **default ON for correctness**. `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) A/B'd speed-NEUTRAL | c16 lever is prefill glue (task #61), not the decode drain. `test_qwen36_async_serving` GREEN | | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | diff --git a/docs/STATUS.md b/docs/STATUS.md index 640deaa94..e3c0d15c4 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1563,9 +1563,9 @@ runtime-verified yet. lanes. Guard `scripts/check-surface-coverage.py` (two axes, preflight + CI): every `examples/*` unit is a client of `include/vllm.h` or tracked to a fold row; every `FEATURES.md` C-ABI capability names an entry point or is tracked. - Fold ROW 8: `vllm_model_params.device` (ABI v14, 0=auto/1=cpu/2=cuda); - explicit cpu forces CPU, an absent named device fails loud (never - substituted); `--device` on server+cli. CUDA-build A/B = named residual. + ROW 8 ABI-v14 device selection is behavior-complete, but #136 regressed its + shared-layer DSR 32->39; generic-seam repair is pending with the baseline + unchanged. CUDA-build A/B remains the named residual. - **Automatic prefix caching (APC)** is on by default for dense models (hybrid / GDN and attention-free default off, mirroring vLLM), and it now has an end-to-end cache-ON gate on `Qwen/Qwen3-4B` (a shared common prefix reused From 285987a6c83d4f93b0698d2041851588f12b3bd4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 01:52:34 +0000 Subject: [PATCH 2/2] fix(surface): resolve explicit device through platform registry Preserve ABI-v14 device values and public names while removing shared CUDA literals from device selection. Resolve the canonical device name through the platform registry and propagate its DeviceType, with mutation-sensitive CPU gates. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:gpt-5 [Codex] --- .agents/NOW.md | 4 +-- .agents/coordination.md | 27 +++++++------- .agents/parity-ledger.md | 1 + .agents/specs/one-surface-abi.md | 23 +++++++++++- .agents/specs/surface-coverage-2026-08-07.md | 2 +- .agents/state.md | 27 ++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 7 ++-- include/vllm/config/device.h | 4 ++- include/vllm/entrypoints/model_loader.h | 8 +++-- include/vllm/platforms/interface.h | 6 ++++ src/capi/vllm_c.cpp | 4 +-- src/vllm/config/device.cpp | 4 +-- src/vllm/entrypoints/model_loader.cpp | 21 +++++++---- src/vllm/platforms/platform.cpp | 10 ++++++ tests/capi/test_capi.cpp | 2 +- .../entrypoints/test_loaded_engine_dense.cpp | 36 +++++++++++-------- tests/vllm/platforms/test_platform.cpp | 16 +++++++++ 18 files changed, 152 insertions(+), 52 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 6d2a7ee60..3a6c150e6 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -1,6 +1,6 @@ # NOW — the one-Read resume surface - + Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live, the gate being chased, what to do next. Never a log — evidence lives in the @@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | -| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 IN REVIEW (`row/DEVICE-KNOB`)**: ABI v14 `device` knob, absent-cuda fails loud, `--device` on server+cli; **#135 H3 device-seam repair IN** (ABI 0/1 via `DeviceType`, DSR 32) | Merge after review; CUDA A/B | +| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 LANDED; #138 repair CPU-GREEN**: ABI v14 stable; named platform resolves via registry; DSR 39→32 (`kcuda=0`), H3 untouched | Review #138; CUDA A/B residual | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/coordination.md b/.agents/coordination.md index f14a2faae..840e0c4d5 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -118,20 +118,6 @@ without the selected contention proof for their entire run are discarded. ## Active claims -**ONE-SURFACE device-selection leakage repair (`ARCH-ONE-SURFACE`, -2026-08-08, `CLAIM-ARCH-ONE-SURFACE-DEVICE-LEAKAGE`).** Codex (GPT-5), -isolated worktree -`/home/mudler/_git/vllm.cpp-arch-one-surface-device-leakage`, branch -`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`, base `origin/main` `b44ad337`. -Repairs the inherited PR #136 DSR regression without weakening the baseline or -allowlist: generic device configuration and model loading must route through -the platform/backend abstraction while preserving ABI-v14 auto/CPU/CUDA -selection, loud absent-CUDA failure, and the restored H3 seam. Owns -`include/vllm/config/device.h`, `src/vllm/config/device.cpp`, -`src/vllm/entrypoints/model_loader.cpp`, focused CPU tests, the existing -ONE-SURFACE specs/records and required public checkpoint surfaces. CPU-only; -no CUDA execution, model download, benchmark, or service change. - **CPU grouped keep-quant GEMM activation-dtype P0 (`QUANT-GGUF-CIQ-GEMM`, 2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`).** Claude Code (claude-opus-5), isolated worktree `/home/mudler/_git/vllmcpp-ciq-grouped`, @@ -1684,6 +1670,19 @@ this claim will meet. The tiled row is speed-gatable on dgx. ## Closing and archival +- 2026-08-08: `CLAIM-ARCH-ONE-SURFACE-DEVICE-LEAKAGE` CLOSED on draft PR + #138. Base `b44ad337` inherited PR #136's seven shared-layer CUDA literals + (DSR 39 vs baseline 32). The repair keeps ABI-v14 0/1/2 and the public + `auto`/`cpu`/`cuda` names, represents slot 2 internally as a named-platform + selection, resolves its canonical name through `FindPlatformByName`, and + propagates the registry's `DeviceType` without a backend-specific shared + branch. RED-first compiler failure pinned the new seam; the non-CUDA `kXPU` + resolver assertion kills a hidden constant return. GREEN: DSR 32 + (`kcuda=0`) with baseline/allowlist unchanged; checker mutations 25/25; + CPU Release `-Werror` `test_platform` 11/11·85, + `test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. H3 source was + untouched; no CUDA execution, download, benchmark or service change. + A block closes only when every row in its declared scope is `DONE`. In the same change: diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 9b72bdaec..bd2740ebc 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -926,3 +926,4 @@ Columns: | 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; MEASUREMENT, lifecycle unchanged) | **MiniMax-H3 - THE NUMBER: what quantizing the TEXT ENCODER to Q4_K_M does to the conditioning.** Same prompt (wuxia, 233 tokens), same tokenizer, same 50-layer truncation, same `MiniMaxH3EncoderTextForwardDevice`, same f32 activations - only the weight bytes differ (Q4_K_M ggml blocks vs the original bf16 14-shard release); both arms self-report IDENTICAL geometry (50/5120/64/8/128/25600), which is what establishes they are the same model. Conditioning `[233, 5120]` f32 via `--encoder-only --save-embeds`. | Not a vLLM-parity change: H3 is BEYOND-PIN (vllm-omni), and vLLM-Omni serves NO quantized H3 at all (BF16-only), so there is no upstream arm to compare against - the bf16 release IS the reference here, and it is the one the loader added in `6d454b00` makes runnable. The quantization-sensitivity premise is ComfyUI PR 15298 (H3's partial split-half RoPE produces channel-wise magnitude outliers that corrupt even INT8), and the measurement CONFIRMS its mechanism concretely: token 0 is an attention sink at norm 15,522 vs a 366 mean (42x), carrying 68% of the total squared error with its DIRECTION intact (cos 0.99962). | **MEASURED on Thor sm_110, build `d1085374` (built and measured as `d1085374`, amended for the row-branch trailer; IDENTICAL tree `dd9283cf`, so the measurement binary IS this commit), GPU idle.** Q4_K_M vs bf16: max|diff| 154.0, RMS 0.5045, rel RMS **0.03403** (0.06849 excluding the sink token), per-token cosine min 0.90916 / mean **0.99745** / median 0.99810, rotation median 3.535 deg / max 24.61 deg, 232 of 233 tokens below cosine 0.999. NOT a scale change: norm ratio mean 0.99010 but the best global rescale only moves 0.03403 -> 0.03280, so it is DIRECTIONAL. CALIBRATION arm (bf16 encoder, ONE-WORD prompt edit `at night`->`at dawn`, also 233 tokens): rel RMS 0.01897 / 0.06666 excl. sink, cosine mean 0.99769 median 0.99963, 172 of 233 tokens above 0.999. So quantization moves the conditioning by the SAME total energy as rewriting a word of the prompt (6.85% vs 6.67%) but with the opposite SHAPE - diffuse over every token instead of concentrated on the words that changed. Cost: Q4 arm 40 s / 18.0 GiB peak; bf16 arm 40 s / 45.41 GiB uploaded / host conversion peak 0.0195 MiB / 51.95 GiB total peak, streamer counters `layers=50 tensors=400 direct=350 converted=200 fused=100` proving the shard path ran. `benchmark_binding=false` (no throughput claim). EXPLICITLY NOT ESTABLISHED: that the RENDER changes - nothing here measures the DiT's sensitivity to a 3.5-degree median rotation; the owed follow-up is a same-DiT/same-seed render A/B over the two saved embeds. | | 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 ABI-v12 device dispatch follow-up; lifecycle unchanged) | **MiniMax-H3 video engine device selection is backend-parameterized.** The stable public selector remains 0=CPU / 1=CUDA, but shared code maps it once to `vt::DeviceType`, creates one queue through `GetBackend(device_type)`, and uses that queue's device instead of naming CUDA twice. | vLLM-Omni pipeline ownership remains unchanged; this is a vllm.cpp C-ABI/backend-seam correction with no upstream behavioral delta. | RED: DSR 34 (`kcuda=2`) vs baseline 32. GREEN: DSR 32 with baseline/allowlist unchanged; checker mutations 25/25. New fold unit pins 0/1 and invalid selectors; CPU compile/test NOT RUN locally because the shared filesystem reached 100%, pending GitHub CI. No GPU, download, or performance claim. | | 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 device-dispatch mutation-gate follow-up; lifecycle unchanged) | **Pins the single-queue and device-provenance contract through the real `MiniMaxH3VideoEngine::Load` path.** Adds a read-only engine `device()` query and a counting CUDA-backend fixture whose queue reports a distinctive CPU:7 device. | No vLLM-Omni behavior changes; this is test observability for the vllm.cpp backend seam above. | Reviewer mutation reproduced first: replacing `im.device = stream_queue.device` with a second `CreateQueue().device` remained GREEN at 5 cases / 135 assertions. RED-first test then reported `2 == 1`; restoring reuse is GREEN at 6/137. Independent mutation deleting queue-device provenance fails the CPU:7 assertion. DSR remains 32 and all 25 leakage-checker mutations pass. No GPU, download, generation, or performance claim. | +| 2026-08-08 (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`; `ARCH-ONE-SURFACE` ROW 8 follow-up; PR #138; CPU-only; lifecycle unchanged) | **Removes PR #136's seven shared CUDA literals without changing ABI-v14 device selection.** Wire 0/1/2 and public `auto`/`cpu`/`cuda` stay exact; internal slot 2 becomes a named-platform tag, `FindPlatformByName` resolves the canonical name, and shared loading propagates the registered `DeviceType`. Explicit CPU, absent-CUDA-before-I/O and CAPI slot 2 are preserved; H3 is untouched. | vLLM `config/device.py:13,61-66` @ `555967922`: supported device name assigned verbatim and never silently substituted. The registry lookup is the in-tree Platform/Backend portability seam; no upstream behavior delta. | RED: inherited DSR 39 (`kcuda=7`) and focused compile failures for the new enum/API/signature. Mutation pin: resolver input `kXPU` must return `kXPU`. GREEN: DSR 32 (`kcuda=0`) with baseline/allowlist unchanged; checker suite 25/25; CPU Release `-Werror` `test_platform` 11/11·85, `test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. No CUDA runtime or performance claim; GPU A/B remains residual. | diff --git a/.agents/specs/one-surface-abi.md b/.agents/specs/one-surface-abi.md index 3e99de527..426061684 100644 --- a/.agents/specs/one-surface-abi.md +++ b/.agents/specs/one-surface-abi.md @@ -1,6 +1,6 @@ # ONE SURFACE — every capability ships through the C ABI -Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; the #123 review's three MINOR findings closed in the same change (c_header_compile.c now references the v11+v12 surface + the v14 field; the v12 changelog block moved to chronological position; the fold fixture's flag list gained `--keep-quant`). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** +Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; PR #138 follow-up removes PR #136's seven shared CUDA literals by resolving the stable public name through the platform registry and propagating its `DeviceType` (DSR 39 -> 32, `kcuda=0`, baseline/allowlist unchanged). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** ## The defect @@ -202,3 +202,24 @@ must report that exact device after exactly one `CreateQueue` call. The original second-queue mutation was false-GREEN at 5/135, then RED at `2 == 1`; deleting the queue-device reuse is independently RED. The final CPU fold target is GREEN at 6 cases / 137 assertions in the isolated `/dev/shm` build. + +### ROW 8 shared-device follow-up (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE`, PR #138) + +PR #136 implemented ABI-v14 explicit selection correctly but encoded its CUDA +identity seven times in shared configuration/loading code, regressing DSR from +32 to 39. The repair preserves the wire values and public names exactly: +`Device::kNamedPlatform` remains integer 2 and `DeviceName()` remains `"cuda"`. +Shared loading now passes that canonical name to `FindPlatformByName`, then +propagates the registered platform's `DeviceType`; no CUDA `DeviceType` literal +remains in the shared selector. This is a real abstraction rather than textual +evasion: the pure resolver test supplies `kXPU` and requires `kXPU` back, while +the registry test walks every registered canonical device name. Explicit CPU +still ignores the accelerator lookup, absent CUDA still throws before model +path I/O, the C ABI still maps slot 2, and H3 dispatch is untouched. + +RED was the focused compiler failure for the missing enum/API/signature plus +the inherited checker result (`kcuda=7`, DSR 39). GREEN is DSR 32 with +`kcuda=0`, unchanged baseline/allowlist, all 25 checker mutations, and a CPU +Release `-Werror` build: `test_platform` 11/11·85, +`test_loaded_engine_dense` 9/9·65 and `test_capi` 45/45·428. No CUDA runtime was +used; the existing GPU A/B residual remains. diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index 5b0a37dbd..8b7a3b3ca 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -141,7 +141,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows). | 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner | | 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | — | M | pooler live-wiring | | 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`→`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual | -| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue` explicit arms via `LoadedEngine::ResolveExplicitDeviceType`; explicit cpu never probes, explicit ABSENT cuda fails LOUD before any model I/O (device.py:61-66 never-substitute mirror); `--device` on server + cli | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | +| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`; leakage follow-up PR #138)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue`; the stable public name resolves through `FindPlatformByName` and its registered `DeviceType` is propagated without a shared CUDA literal; explicit cpu never probes, explicit ABSENT cuda fails LOUD before model I/O; DSR 32 / `kcuda=0` | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | | 9 | Voxtral + audio chat seam | register `VoxtralForConditionalGeneration` + fold `VoxtralGenerateGreedy` into the registry forward; audio-capable chat fn + an engine consumer for `AudioKwargs` mm_features | rewrite tests→clients | M | mirror upstream `voxtral.py:309`, `SupportsTranscription` | | 10 | Gemma-4 audio e2e | bf16 device audio forward + audio→text merge (residual `gemma4_audio.h:41`); USM log-mel front end | fold into the registered mm forward | M | `MM-SERVE-E2E` | | 11 | Tokenizer/bench ABI + real-load gate | `vllm_tokenize`/`vllm_detokenize`; token-id/count fields on the stream callback for bench; **gate `vllm_engine_load` on a REAL tiny checkpoint at least once** (today bad-path only, `test_capi.cpp:474`) | rewrite `tokenize`/`bench` as clients | S-M | `/tokenize` route exists (`api_server.cpp:432`) | diff --git a/.agents/state.md b/.agents/state.md index 7ad7f10fd..a0815a4f5 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -42468,3 +42468,30 @@ records main-wholesale + these rows re-applied. (4) Explicit names for xpu/vulkan/metal/rocm are additive later; auto reaches them today. (5) The transcription stack refuses explicit cuda (CPU pipeline) — revisit when an accelerated transcription path exists. + +## 2026-08-08 — ARCH-ONE-SURFACE ROW 8 device leakage repaired through the platform registry (PR #138) + + +Current main `b44ad337` inherited PR #136 with seven CUDA enum literals in the +shared device config/loader: DSR was 39 vs the immutable baseline 32 +(`kcuda=7`). RED-first focused tests required a platform-neutral ABI enum tag, +a canonical-name registry lookup and an optional resolved `DeviceType`; the +old API failed to compile on each missing seam. + +The ABI and public behavior are unchanged: values remain 0=auto, 1=cpu, +2=cuda; parsing and `DeviceName()` still expose `"cuda"`; absent CUDA still +fails loudly before model-path I/O; explicit CPU still bypasses accelerator +selection. Internally slot 2 is `Device::kNamedPlatform`; shared loading calls +`FindPlatformByName(DeviceName(device))` and propagates the registered +platform's type. A mutation-sensitive test supplies `kXPU` and requires +`kXPU` back, so replacing the propagation with a hidden CUDA constant is RED. +The platform test walks every registered canonical name and pins missing-name +failure. + +GREEN in `/dev/shm/vllm-device-leakage-build`, CPU Release, CUDA OFF, +`-Werror`: `test_platform` 11/11 cases · 85 assertions; +`test_loaded_engine_dense` 9/9 · 65; `test_capi` 45/45 · 428. The leakage +checker reports DSR 32 (`kcuda=0`, `is_cuda=0`, `cuda_inc=0`, `vt_ifdef=32`) +with baseline/allowlist unchanged, and its mutation suite is 25/25. H3 source +is untouched. No CUDA runtime, model download, service action, benchmark or +release artifact occurred; the existing CUDA-build A/B remains pending. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index eb2a83513..b0caf7e4d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Track | Status | Next gate | |---|---|---| -| Surface coverage (`ARCH-ONE-SURFACE`) | **PENDING correctness-only:** #136 regressed device-seam DSR 32->39; repair removes seven shared CUDA literals through generic seams. ABI-v14 selection is unchanged; no speed number | DSR 32 with unchanged baseline/allowlist; CPU selector/loader tests + mutation review; CUDA A/B remains residual | +| Surface coverage (`ARCH-ONE-SURFACE`) | **CORRECTNESS COMPLETE:** #138 restores DSR 32 (`kcuda=0`) via registry/name resolution; ABI-v14 selection unchanged; no speed claim | CPU selector/platform/C-ABI tests green; CUDA A/B remains residual | | 35B prefill TTFT | 0.93x to 0.98x at every concurrency (2026-08-05) | Attribute the residual, then close | | 35B low-batch MoE decode | CLOSED at low batch (c1 0.975x, c4 wins); c16 0.93x. `VT_ASYNC_DEVICE_MIRROR` **default ON for correctness**. `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) A/B'd speed-NEUTRAL | c16 lever is prefill glue (task #61), not the decode drain. `test_qwen36_async_serving` GREEN | | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | diff --git a/docs/STATUS.md b/docs/STATUS.md index e3c0d15c4..fc4b36603 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1563,9 +1563,10 @@ runtime-verified yet. lanes. Guard `scripts/check-surface-coverage.py` (two axes, preflight + CI): every `examples/*` unit is a client of `include/vllm.h` or tracked to a fold row; every `FEATURES.md` C-ABI capability names an entry point or is tracked. - ROW 8 ABI-v14 device selection is behavior-complete, but #136 regressed its - shared-layer DSR 32->39; generic-seam repair is pending with the baseline - unchanged. CUDA-build A/B remains the named residual. + ROW 8 ABI-v14 device selection is behavior-complete; #138 repairs #136's + shared-layer DSR 39->32 through registry/name resolution (`kcuda=0`) with the + baseline and allowlist unchanged. CPU selector/platform/C-ABI gates are + green; CUDA-build A/B remains the named residual. - **Automatic prefix caching (APC)** is on by default for dense models (hybrid / GDN and attention-free default off, mirroring vLLM), and it now has an end-to-end cache-ON gate on `Qwen/Qwen3-4B` (a shared common prefix reused diff --git a/include/vllm/config/device.h b/include/vllm/config/device.h index db90a7732..e0f9495b0 100644 --- a/include/vllm/config/device.h +++ b/include/vllm/config/device.h @@ -31,7 +31,9 @@ namespace vllm { enum class Device : int32_t { kAuto = 0, // platform-probed, upstream's "auto" default (device.py:20). kCPU = 1, // force the CPU queue even when an accelerator is present. - kCUDA = 2, // require the CUDA platform; absent => the load fails LOUD. + // ABI v14 slot 2 currently names "cuda". Keep the enum platform-neutral: + // shared selection resolves DeviceName() through the platform registry. + kNamedPlatform = 2, }; // Parse the wire/CLI name ("auto" | "cpu" | "cuda" — the supported subset of diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index bdb97e01c..e47f54df7 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -188,7 +188,8 @@ class LoadedEngine { // registering fake global platforms: // * kCPU -> vt::DeviceType::kCPU unconditionally — an explicit CPU ask // never consults the accelerator probe, even when CUDA is registered; - // * kCUDA -> vt::DeviceType::kCUDA when cuda_platform_registered, else + // * kNamedPlatform -> the DeviceType returned by the canonical-name + // platform lookup, else // THROWS std::runtime_error naming the device (fail LOUD; the mirror of // vLLM assigning an explicit device verbatim and never substituting // another — vllm/config/device.py:61-66); @@ -197,8 +198,9 @@ class LoadedEngine { // std::invalid_argument if passed. // SelectQueue routes its explicit arms through THIS function, so the gate on // it pins the production policy, not a parallel copy. - static vt::DeviceType ResolveExplicitDeviceType(vllm::Device requested, - bool cuda_platform_registered); + static vt::DeviceType ResolveExplicitDeviceType( + vllm::Device requested, + std::optional named_platform_type); vllm::v1::LLMEngine& engine() { return engine_; } // Lazily start W2's EngineCoreProc + output-handler threads. Once created, diff --git a/include/vllm/platforms/interface.h b/include/vllm/platforms/interface.h index 68b77a5a6..ba21a058a 100644 --- a/include/vllm/platforms/interface.h +++ b/include/vllm/platforms/interface.h @@ -303,6 +303,12 @@ void RegisterPlatform(DeviceType type, Platform* platform); Platform& GetPlatform(DeviceType type); bool HasPlatform(DeviceType type); +// Find a registered platform by vt::DeviceTypeName(), or nullptr when that +// canonical user-facing name has no usable platform in this process. This is +// the shared explicit-device seam: callers name a platform without branching +// on a backend-specific DeviceType literal. +Platform* FindPlatformByName(std::string_view name); + // The process's active compute platform: the highest-priority registered // accelerator, else CPU — mirrors vLLM resolving `current_platform` by probing // for an accelerator and falling back to CPU. This answers ONLY the process-level diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 372e52e54..00071ea42 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -615,7 +615,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, ep.device = vllm::Device::kCPU; break; case 2: - ep.device = vllm::Device::kCUDA; + ep.device = vllm::Device::kNamedPlatform; break; default: SetError( @@ -646,7 +646,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, // explicit CUDA ask cannot be served and is REFUSED rather than // silently downgraded (the same never-substitute rule as the text // engine, vllm/config/device.py:61-66). - if (ep.device == vllm::Device::kCUDA) { + if (ep.device == vllm::Device::kNamedPlatform) { SetError( "vllm_engine_load: device 'cuda' was requested but this " "transcription-only checkpoint serves on the CPU pipeline; use " diff --git a/src/vllm/config/device.cpp b/src/vllm/config/device.cpp index a73969553..f312f081d 100644 --- a/src/vllm/config/device.cpp +++ b/src/vllm/config/device.cpp @@ -15,7 +15,7 @@ Device DeviceFromString(const std::string& value) { return Device::kCPU; } if (value == "cuda") { - return Device::kCUDA; + return Device::kNamedPlatform; } // Mirrors pydantic rejecting a value outside the Device Literal // (vllm/config/device.py:13). "tpu"/"xpu" are upstream names this build @@ -31,7 +31,7 @@ const char* DeviceName(Device device) { return "auto"; case Device::kCPU: return "cpu"; - case Device::kCUDA: + case Device::kNamedPlatform: return "cuda"; } return "invalid"; diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index b9685d8a7..2549fff5a 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -56,8 +56,12 @@ namespace { // vllm/config/device.py:61-66). vt::Queue SelectQueue(std::string_view architecture, vllm::Device device) { if (device != vllm::Device::kAuto) { + const vllm::platforms::Platform* named_platform = + vllm::platforms::FindPlatformByName(vllm::DeviceName(device)); const vt::DeviceType resolved = LoadedEngine::ResolveExplicitDeviceType( - device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + device, named_platform == nullptr + ? std::nullopt + : std::optional{named_platform->device_type()}); if (resolved == vt::DeviceType::kCPU) { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } @@ -485,14 +489,15 @@ bool LoadedEngine::ResolveEnablePrefixCaching(const EngineParams& params, // src/vllm/platforms/cuda.cpp Registrar — kCUDA registers only when a usable // GPU probed). vt::DeviceType LoadedEngine::ResolveExplicitDeviceType( - vllm::Device requested, bool cuda_platform_registered) { + vllm::Device requested, + std::optional named_platform_type) { switch (requested) { case vllm::Device::kCPU: // Explicit CPU never consults the accelerator probe: even on a // CUDA-capable build/process this selects the CPU queue. return vt::DeviceType::kCPU; - case vllm::Device::kCUDA: - if (!cuda_platform_registered) { + case vllm::Device::kNamedPlatform: + if (!named_platform_type.has_value()) { throw std::runtime_error( "device 'cuda' was requested but no CUDA platform is available in " "this build/process (an explicitly named device is never silently " @@ -500,7 +505,7 @@ vt::DeviceType LoadedEngine::ResolveExplicitDeviceType( "or device=cpu, or run a CUDA build on a machine with a usable " "GPU)"); } - return vt::DeviceType::kCUDA; + return *named_platform_type; case vllm::Device::kAuto: break; // auto resolves through the probe in SelectQueue, not here. } @@ -875,8 +880,12 @@ std::unique_ptr LoadedEngine::FromModelDir( // SAME ResolveExplicitDeviceType when it actually creates the queue, so the // policy has exactly one owner. if (params.device != vllm::Device::kAuto) { + const vllm::platforms::Platform* named_platform = + vllm::platforms::FindPlatformByName(vllm::DeviceName(params.device)); (void)ResolveExplicitDeviceType( - params.device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + params.device, named_platform == nullptr + ? std::nullopt + : std::optional{named_platform->device_type()}); } const fs::path dir(model_dir); diff --git a/src/vllm/platforms/platform.cpp b/src/vllm/platforms/platform.cpp index 32e4a9697..20d245717 100644 --- a/src/vllm/platforms/platform.cpp +++ b/src/vllm/platforms/platform.cpp @@ -73,6 +73,16 @@ Platform& GetPlatform(DeviceType type) { bool HasPlatform(DeviceType type) { return Registry()[Index(type)] != nullptr; } +Platform* FindPlatformByName(std::string_view name) { + for (size_t i = 0; i < vt::kNumDeviceTypes; ++i) { + Platform* platform = Registry()[i]; + if (platform == nullptr) continue; + const DeviceType type = static_cast(i); + if (name == vt::DeviceTypeName(type)) return platform; + } + return nullptr; +} + Platform& CurrentPlatform() { for (DeviceType type : kCurrentPriority) { Platform* p = Registry()[static_cast(type)]; diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index c25d842b7..a241c933b 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1832,7 +1832,7 @@ TEST_CASE("capi v14: explicit cpu forces the CPU queue at the EngineParams seam" // the pinned message out of construction — never silently build on CPU. if (!vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { EngineParams cuda_params = SyntheticParams(); - cuda_params.device = vllm::Device::kCUDA; + cuda_params.device = vllm::Device::kNamedPlatform; CHECK_THROWS_WITH_AS( LoadedEngine(MakeConfig(), MakeWeights(c), BuildFixture(), cuda_params), doctest::Contains("device 'cuda' was requested but no CUDA platform"), diff --git a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp index 5d4974b5b..72912e33c 100644 --- a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp +++ b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp @@ -457,33 +457,39 @@ TEST_CASE("loaded_engine: prefix caching mirrors model-capability defaults") { // registered accelerator") that a CPU-only process could otherwise never // exercise. SelectQueue routes its explicit arms through THIS function // (model_loader.cpp), so these pins bind the production policy, not a copy. -TEST_CASE("loaded_engine: ResolveExplicitDeviceType — cpu never probes, cuda never falls back") { +TEST_CASE("loaded_engine: ResolveExplicitDeviceType uses the named platform without fallback") { using vllm::Device; - // Explicit CPU resolves CPU regardless of what is registered. The `true` arm - // is the CUDA-build pin: a registered accelerator must NOT win over an + // Explicit CPU resolves CPU regardless of what the name lookup found. The + // non-CPU value is the accelerator-build pin: a registered accelerator must + // NOT win over an // explicit cpu ask (the fold-ROW-8 defect was that an embedder could not ASK // for CPU at all). - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, true) == + CHECK(LoadedEngine::ResolveExplicitDeviceType( + Device::kCPU, std::optional{vt::DeviceType::kXPU}) == vt::DeviceType::kCPU); - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, false) == + CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, std::nullopt) == vt::DeviceType::kCPU); - // Explicit CUDA with the platform registered resolves CUDA. - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, true) == - vt::DeviceType::kCUDA); + // The explicit named-platform arm returns what the registry found. kXPU is + // deliberate mutation sensitivity: a hidden CUDA constant cannot satisfy it. + CHECK(LoadedEngine::ResolveExplicitDeviceType( + Device::kNamedPlatform, std::optional{vt::DeviceType::kXPU}) == + vt::DeviceType::kXPU); // Explicit CUDA WITHOUT the platform throws the pinned message — never a // silent CPU fallback (mirror of vLLM assigning an explicit device verbatim, // vllm/config/device.py:61-66). CHECK_THROWS_WITH_AS( - LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, false), + LoadedEngine::ResolveExplicitDeviceType(Device::kNamedPlatform, + std::nullopt), doctest::Contains("device 'cuda' was requested but no CUDA platform"), std::runtime_error); // kAuto is not an explicit selection: it resolves through the probe inside // SelectQueue, and this seam refuses it rather than guessing. - CHECK_THROWS_AS(LoadedEngine::ResolveExplicitDeviceType(Device::kAuto, true), + CHECK_THROWS_AS( + LoadedEngine::ResolveExplicitDeviceType(Device::kAuto, std::nullopt), std::invalid_argument); } @@ -493,7 +499,7 @@ TEST_CASE("loaded_engine: DeviceFromString mirrors the vLLM Device names") { // — the strings the server's --device flag consumes. CHECK(vllm::DeviceFromString("auto") == Device::kAuto); CHECK(vllm::DeviceFromString("cpu") == Device::kCPU); - CHECK(vllm::DeviceFromString("cuda") == Device::kCUDA); + CHECK(vllm::DeviceFromString("cuda") == Device::kNamedPlatform); CHECK_THROWS_WITH_AS(vllm::DeviceFromString("tpu"), doctest::Contains("Unknown device: tpu"), std::invalid_argument); @@ -505,10 +511,10 @@ TEST_CASE("loaded_engine: DeviceFromString mirrors the vLLM Device names") { // the auto slot. CHECK(static_cast(Device::kAuto) == 0); CHECK(static_cast(Device::kCPU) == 1); - CHECK(static_cast(Device::kCUDA) == 2); + CHECK(static_cast(Device::kNamedPlatform) == 2); CHECK(std::string(vllm::DeviceName(Device::kAuto)) == "auto"); CHECK(std::string(vllm::DeviceName(Device::kCPU)) == "cpu"); - CHECK(std::string(vllm::DeviceName(Device::kCUDA)) == "cuda"); + CHECK(std::string(vllm::DeviceName(Device::kNamedPlatform)) == "cuda"); } TEST_CASE("loaded_engine: FromModelDir resolves an explicit absent device BEFORE any path I/O") { @@ -517,11 +523,11 @@ TEST_CASE("loaded_engine: FromModelDir resolves an explicit absent device BEFORE // arg_utils.py:1878, device.py __post_init__). This is also what makes the // EngineParams->FromModelDir plumb pinnable on the CPU tier with no loadable // checkpoint: a bogus path + device=cuda must report the DEVICE, not the path. - if (vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + if (vllm::platforms::FindPlatformByName("cuda") != nullptr) { return; // CUDA build/box: the explicit-cuda arm resolves; nothing to pin. } EngineParams params; - params.device = vllm::Device::kCUDA; + params.device = vllm::Device::kNamedPlatform; CHECK_THROWS_WITH_AS( LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/model/dir", params), doctest::Contains("device 'cuda' was requested but no CUDA platform"), diff --git a/tests/vllm/platforms/test_platform.cpp b/tests/vllm/platforms/test_platform.cpp index 198017fdc..d3f7c3540 100644 --- a/tests/vllm/platforms/test_platform.cpp +++ b/tests/vllm/platforms/test_platform.cpp @@ -15,6 +15,7 @@ using vllm::platforms::CurrentPlatform; using vllm::platforms::DeviceCapability; +using vllm::platforms::FindPlatformByName; using vllm::platforms::GetPlatform; using vllm::platforms::HasPlatform; using vllm::platforms::Platform; @@ -102,6 +103,21 @@ TEST_CASE("CPU platform is self-registered and advertises CPU capabilities") { CHECK(cpu.get_attn_backend_priority() == cpu_priority); } +TEST_CASE("platform registry resolves canonical names without device-specific callers") { + REQUIRE(FindPlatformByName("cpu") != nullptr); + CHECK(FindPlatformByName("cpu")->device_type() == DeviceType::kCPU); + CHECK(FindPlatformByName("not-a-platform") == nullptr); + + size_t count = 0; + const DeviceType* priority = vllm::platforms::CurrentPlatformPriority(count); + for (size_t i = 0; i < count; ++i) { + if (!HasPlatform(priority[i])) continue; + CAPTURE(vt::DeviceTypeName(priority[i])); + CHECK(FindPlatformByName(vt::DeviceTypeName(priority[i])) == + &GetPlatform(priority[i])); + } +} + TEST_CASE("CurrentPlatform resolves accelerator-first, else falls back to CPU") { // CurrentPlatform() answers the PROCESS-level "what accelerator is this // process on" question (interface.h:104): accelerator-first, CPU fallback. It