diff --git a/.agents/NOW.md b/.agents/NOW.md index 6d2a7ee6..cdea5550 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -1,6 +1,6 @@ # NOW — the one-Read resume surface - + Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live, the gate being chased, what to do next. Never a log — evidence lives in the @@ -28,7 +28,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | | `BACKEND-ROCM` W0 | Skeleton in; **HIP never compiled** (no AMD HW) | #41 contributors build it; a compile error IS the deliverable | -| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 IN REVIEW (`row/DEVICE-KNOB`)**: ABI v14 `device` knob, absent-cuda fails loud, `--device` on server+cli; **#135 H3 device-seam repair IN** (ABI 0/1 via `DeviceType`, DSR 32) | Merge after review; CUDA A/B | +| Surface coverage (`ARCH-ONE-SURFACE`) | **ROW 8 LANDED; #139 repair CPU-GREEN**: ABI v14 stable; registry-resolved named platform; DSR 39→32; execution guard 52/52 | Fresh re-review #139; CUDA A/B residual | In-flight (default-OFF, not pushed): `laguna-fp4proj-prod`, laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/coordination.md b/.agents/coordination.md index c256ee1e..fb94f1c7 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1670,6 +1670,19 @@ this claim will meet. The tiled row is speed-gatable on dgx. ## Closing and archival +- 2026-08-08: `CLAIM-ARCH-ONE-SURFACE-DEVICE-LEAKAGE-V2` CLOSED on draft PR + #139. Base `b44ad337` inherited PR #136's seven shared-layer CUDA literals + (DSR 39 vs baseline 32). The repair keeps ABI-v14 0/1/2 and the public + `auto`/`cpu`/`cuda` names, represents slot 2 internally as a named-platform + selection, resolves its canonical name through `FindPlatformByName`, and + propagates the registry's `DeviceType` without a backend-specific shared + branch. RED-first compiler failure pinned the new seam; the non-CUDA `kXPU` + resolver assertion kills a hidden constant return. GREEN: DSR 32 + (`kcuda=0`) with baseline/allowlist unchanged; checker mutations 25/25; + CPU Release `-Werror` `test_platform` 11/11·85, + `test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. H3 source was + untouched; no CUDA execution, download, benchmark or service change. + A block closes only when every row in its declared scope is `DONE`. In the same change: diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 9b72bdae..f99fe847 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -926,3 +926,4 @@ Columns: | 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`; `ROAD-V1-H3`; model `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit`; MEASUREMENT, lifecycle unchanged) | **MiniMax-H3 - THE NUMBER: what quantizing the TEXT ENCODER to Q4_K_M does to the conditioning.** Same prompt (wuxia, 233 tokens), same tokenizer, same 50-layer truncation, same `MiniMaxH3EncoderTextForwardDevice`, same f32 activations - only the weight bytes differ (Q4_K_M ggml blocks vs the original bf16 14-shard release); both arms self-report IDENTICAL geometry (50/5120/64/8/128/25600), which is what establishes they are the same model. Conditioning `[233, 5120]` f32 via `--encoder-only --save-embeds`. | Not a vLLM-parity change: H3 is BEYOND-PIN (vllm-omni), and vLLM-Omni serves NO quantized H3 at all (BF16-only), so there is no upstream arm to compare against - the bf16 release IS the reference here, and it is the one the loader added in `6d454b00` makes runnable. The quantization-sensitivity premise is ComfyUI PR 15298 (H3's partial split-half RoPE produces channel-wise magnitude outliers that corrupt even INT8), and the measurement CONFIRMS its mechanism concretely: token 0 is an attention sink at norm 15,522 vs a 366 mean (42x), carrying 68% of the total squared error with its DIRECTION intact (cos 0.99962). | **MEASURED on Thor sm_110, build `d1085374` (built and measured as `d1085374`, amended for the row-branch trailer; IDENTICAL tree `dd9283cf`, so the measurement binary IS this commit), GPU idle.** Q4_K_M vs bf16: max|diff| 154.0, RMS 0.5045, rel RMS **0.03403** (0.06849 excluding the sink token), per-token cosine min 0.90916 / mean **0.99745** / median 0.99810, rotation median 3.535 deg / max 24.61 deg, 232 of 233 tokens below cosine 0.999. NOT a scale change: norm ratio mean 0.99010 but the best global rescale only moves 0.03403 -> 0.03280, so it is DIRECTIONAL. CALIBRATION arm (bf16 encoder, ONE-WORD prompt edit `at night`->`at dawn`, also 233 tokens): rel RMS 0.01897 / 0.06666 excl. sink, cosine mean 0.99769 median 0.99963, 172 of 233 tokens above 0.999. So quantization moves the conditioning by the SAME total energy as rewriting a word of the prompt (6.85% vs 6.67%) but with the opposite SHAPE - diffuse over every token instead of concentrated on the words that changed. Cost: Q4 arm 40 s / 18.0 GiB peak; bf16 arm 40 s / 45.41 GiB uploaded / host conversion peak 0.0195 MiB / 51.95 GiB total peak, streamer counters `layers=50 tensors=400 direct=350 converted=200 fused=100` proving the shard path ran. `benchmark_binding=false` (no throughput claim). EXPLICITLY NOT ESTABLISHED: that the RENDER changes - nothing here measures the DiT's sensitivity to a 3.5-degree median rotation; the owed follow-up is a same-DiT/same-seed render A/B over the two saved embeds. | | 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 ABI-v12 device dispatch follow-up; lifecycle unchanged) | **MiniMax-H3 video engine device selection is backend-parameterized.** The stable public selector remains 0=CPU / 1=CUDA, but shared code maps it once to `vt::DeviceType`, creates one queue through `GetBackend(device_type)`, and uses that queue's device instead of naming CUDA twice. | vLLM-Omni pipeline ownership remains unchanged; this is a vllm.cpp C-ABI/backend-seam correction with no upstream behavioral delta. | RED: DSR 34 (`kcuda=2`) vs baseline 32. GREEN: DSR 32 with baseline/allowlist unchanged; checker mutations 25/25. New fold unit pins 0/1 and invalid selectors; CPU compile/test NOT RUN locally because the shared filesystem reached 100%, pending GitHub CI. No GPU, download, or performance claim. | | 2026-08-08 (`row/ARCH-ONE-SURFACE`; H3 device-dispatch mutation-gate follow-up; lifecycle unchanged) | **Pins the single-queue and device-provenance contract through the real `MiniMaxH3VideoEngine::Load` path.** Adds a read-only engine `device()` query and a counting CUDA-backend fixture whose queue reports a distinctive CPU:7 device. | No vLLM-Omni behavior changes; this is test observability for the vllm.cpp backend seam above. | Reviewer mutation reproduced first: replacing `im.device = stream_queue.device` with a second `CreateQueue().device` remained GREEN at 5 cases / 135 assertions. RED-first test then reported `2 == 1`; restoring reuse is GREEN at 6/137. Independent mutation deleting queue-device provenance fails the CPU:7 assertion. DSR remains 32 and all 25 leakage-checker mutations pass. No GPU, download, generation, or performance claim. | +| 2026-08-08 (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE-V2`; `ARCH-ONE-SURFACE` ROW 8 follow-up; PR #139; CPU-only; lifecycle unchanged) | **Removes PR #136's seven shared CUDA literals without changing ABI-v14 device selection.** Wire 0/1/2 and public `auto`/`cpu`/`cuda` stay exact; internal slot 2 becomes a named-platform tag, `FindPlatformByName` resolves the canonical name, and shared loading propagates the registered `DeviceType`. Explicit CPU, absent-CUDA-before-I/O and CAPI slot 2 are preserved; H3 is untouched. | vLLM `config/device.py:13,61-66` @ `555967922`: supported device name assigned verbatim and never silently substituted. The registry lookup is the in-tree Platform/Backend portability seam; no upstream behavior delta. | RED: inherited DSR 39 (`kcuda=7`) and focused compile failures for the new enum/API/signature. Mutation pin: resolver input `kXPU` must return `kXPU`. GREEN: DSR 32 (`kcuda=0`) with baseline/allowlist unchanged; checker suite 25/25; CPU Release `-Werror` `test_platform` 11/11·85, `test_loaded_engine_dense` 9/9·65, `test_capi` 45/45·428. No CUDA runtime or performance claim; GPU A/B remains residual. | diff --git a/.agents/specs/one-surface-abi.md b/.agents/specs/one-surface-abi.md index 3e99de52..5997426b 100644 --- a/.agents/specs/one-surface-abi.md +++ b/.agents/specs/one-surface-abi.md @@ -1,6 +1,6 @@ # ONE SURFACE — every capability ships through the C ABI -Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; the #123 review's three MINOR findings closed in the same change (c_header_compile.c now references the v11+v12 surface + the v14 field; the v12 changelog block moved to chronological position; the fold fixture's flag list gained `--keep-quant`). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** +Row: `ARCH-ONE-SURFACE`. Status: **AUDIT DONE; remediation IN PROGRESS — ROW 1 (Parakeet ASR / audio transcription) LANDED 2026-08-07: ABI v11 `vllm_transcribe`, live `/v1/audio/transcriptions`, registry refuse-by-task, example folded, ratchet 12 -> 11. ROW 2 (MiniMax-H3 video+audio generation) LANDED 2026-08-08 (`row/H3-VIDEO-ABI`, task #283): ABI v12 `vllm_video_engine_load`/`vllm_video_generate`/`vllm_video_result_free` + `vllm_video_mux_argv` over the `MiniMaxH3VideoEngine` library seam, `/v1/videos` routed through the SAME seam, both H3 examples rewritten as `vllm.h` clients byte-identical to the pre-fold binary, ratchet 11 -> 9. GB10 real-video re-verification is a NAMED RESIDUAL (box on the Kimi campaign). ROW 8 (explicit device selection) LANDED 2026-08-08 (`row/DEVICE-KNOB`, task #284): ABI v14 `vllm_model_params.device` (0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names) -> `EngineParams::device` -> `SelectQueue`; explicit cpu forces the CPU queue without probing, an explicitly named ABSENT device fails LOUD before any model I/O (the vllm/config/device.py:61-66 never-substitute mirror), `--device` on server + cli as pure field consumers; PR #139 follow-up removes PR #136's seven shared CUDA literals by resolving the stable public name through the platform registry and propagating its `DeviceType` (DSR 39 -> 32, `kcuda=0`, baseline/allowlist unchanged). CUDA-build A/B (auto->CUDA vs explicit-cpu->CPU on a GPU box) is a NAMED RESIDUAL — the CPU tier pins that half through the pure `ResolveExplicitDeviceType` matrix instead.** ## The defect @@ -202,3 +202,59 @@ must report that exact device after exactly one `CreateQueue` call. The original second-queue mutation was false-GREEN at 5/135, then RED at `2 == 1`; deleting the queue-device reuse is independently RED. The final CPU fold target is GREEN at 6 cases / 137 assertions in the isolated `/dev/shm` build. + +### ROW 8 shared-device follow-up (`row/ARCH-ONE-SURFACE-DEVICE-LEAKAGE-V2`, PR #139) + +PR #136 implemented ABI-v14 explicit selection correctly but encoded its CUDA +identity seven times in shared configuration/loading code, regressing DSR from +32 to 39. The repair preserves the wire values and public names exactly: +`Device::kNamedPlatform` remains integer 2 and `DeviceName()` remains `"cuda"`. +Shared loading now passes that canonical name to `FindPlatformByName`, then +propagates the registered platform's `DeviceType`; no CUDA `DeviceType` literal +remains in the shared selector. This is a real abstraction rather than textual +evasion: the pure resolver test supplies `kXPU` and requires `kXPU` back, while +an isolated integration target registers a distinctive XPU-shaped platform and +backend, requires exact canonical-name lookup (rejecting prefix and malformed +near-matches), and proves the production selector propagates that platform type +into the created queue. Explicit CPU still ignores the accelerator lookup, +absent CUDA still throws before model path I/O, the C ABI still maps slot 2, +and H3 dispatch is untouched. + +RED was the focused compiler failure for the missing enum/API/signature plus +the inherited checker result (`kcuda=7`, DSR 39). GREEN is DSR 32 with +`kcuda=0`, unchanged baseline/allowlist, all 25 checker mutations, and a CPU +Release `-Werror` build: `test_platform` 11/11·85, +`test_device_selection` 2/2·11, `test_loaded_engine_dense` 9/9·65 and +`test_capi` 45/45·428. Each of the three review mutations (wrong returned +platform identity, forced-CPU selector input and prefix name matching) is RED. +The standalone selector target is non-vacuous under +`scripts/check-test-registration.py`: a disposable CPU-only CMake configure and +File-API codemodel query prove that the target exists with its exact source, +then the checker selects the Release configuration, materializes that +configuration's disposable artifact, and requires `ctest -C Release` JSON to +resolve the enabled test to the exact one-argument executable command. This +works for single- and multi-config generators. The workflow guard structurally +normalizes quoted/spaced YAML mapping keys and accepts only the dedicated direct +argv block owned by a job/step with no `if`, `continue-on-error`, or custom +`shell`. Preflight is not inferred from loop text: the checker executes a +disposable copy with instrumented `python3`/`git` shims and requires the checker +and suite argv to occur exactly once, so empty bodies, `continue`, an outer false +branch, and an unset array are all red. + +The 52/52 suite covers the earlier registration deaths plus disabled CTest, +non-gating Actions variants, and the preflight execution variants. Its mutation +inventory is protected by a third, independent layer in the production checker: +the canonical path and content SHA-256 are pinned there, as are the named +integrity method and wrapper-body contracts. Deleting or renaming M18 in both the +suite and manifest, redirecting the suite to a byte-identical manifest, or +deleting the integrity test is therefore red. A distinct direct test passes the +unchanged canonical suite source with a byte-identical alternate manifest path +and requires only the exact path-specific error; deleting the production path +guard now makes that test red. The production-owned AST contract separately +requires M42 to retain `assertEqual(errors, [the exact path diagnostic])`, so +deleting or replacing that outcome assertion cannot be masked by M42's +independent byte-identity assertion. This is precisely a claim of +resistance to tandem two-layer suite/manifest shrinkage; it does not claim to +resist an arbitrary simultaneous rewrite of the suite, manifest, and production +checker. +No CUDA runtime was used; the existing GPU A/B residual remains. diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index 5b0a37db..94cd98da 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -141,7 +141,7 @@ lanes are leaves of `ARCH-ONE-SURFACE` (do not open parallel rows). | 5 | Kimi-Linear incremental | expose the incremental decode path through the runner/engine (the recompute forward already routes) | rewrite `kimi_linear_gen` | S–M | `KimiDecodeCache` on the runner | | 6 | Embeddings/pooling | `vllm_embed`/pooling entry point + live `/v1/embeddings`; register a pooling arch (`is_pooling_model=true`); invoke `PoolingRunner` in the step | — | M | pooler live-wiring | | 7 | Multimodal input | multimodal-content entry point on `vllm_chat`; run the vision/audio tower in the engine step (`mm_features`→`ModelForwardInput.mm`) | wire `chat_mm` seam into the ABI | L | `MM-SERVE-E2E` engine mm-forward residual | -| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue` explicit arms via `LoadedEngine::ResolveExplicitDeviceType`; explicit cpu never probes, explicit ABSENT cuda fails LOUD before any model I/O (device.py:61-66 never-substitute mirror); `--device` on server + cli | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | +| 8 | Device-selection knob | **DONE (ROW 8, 2026-08-08, `row/DEVICE-KNOB`; leakage follow-up PR #139)**: `vllm_model_params.device` (ABI v14: 0=auto/1=cpu/2=cuda, the vLLM `DeviceConfig.device` names, device.py:13) → `EngineParams::device` → `SelectQueue`; the stable public name resolves through `FindPlatformByName` and its registered `DeviceType` is propagated without a shared CUDA literal; explicit cpu never probes, explicit ABSENT cuda fails LOUD before model I/O; DSR 32 / `kcuda=0` | **DONE**: both thin clients consume the field; zero value byte-identical (auto probe) | S | mirror vLLM `--device`/`DeviceConfig` | | 9 | Voxtral + audio chat seam | register `VoxtralForConditionalGeneration` + fold `VoxtralGenerateGreedy` into the registry forward; audio-capable chat fn + an engine consumer for `AudioKwargs` mm_features | rewrite tests→clients | M | mirror upstream `voxtral.py:309`, `SupportsTranscription` | | 10 | Gemma-4 audio e2e | bf16 device audio forward + audio→text merge (residual `gemma4_audio.h:41`); USM log-mel front end | fold into the registered mm forward | M | `MM-SERVE-E2E` | | 11 | Tokenizer/bench ABI + real-load gate | `vllm_tokenize`/`vllm_detokenize`; token-id/count fields on the stream callback for bench; **gate `vllm_engine_load` on a REAL tiny checkpoint at least once** (today bad-path only, `test_capi.cpp:474`) | rewrite `tokenize`/`bench` as clients | S-M | `/tokenize` route exists (`api_server.cpp:432`) | diff --git a/.agents/state.md b/.agents/state.md index 7ad7f10f..c62c6a42 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -42468,3 +42468,152 @@ records main-wholesale + these rows re-applied. (4) Explicit names for xpu/vulkan/metal/rocm are additive later; auto reaches them today. (5) The transcription stack refuses explicit cuda (CPU pipeline) — revisit when an accelerated transcription path exists. + +## 2026-08-08 — ARCH-ONE-SURFACE ROW 8 device leakage repaired through the platform registry (PR #139) + + +Current main `b44ad337` inherited PR #136 with seven CUDA enum literals in the +shared device config/loader: DSR was 39 vs the immutable baseline 32 +(`kcuda=7`). RED-first focused tests required a platform-neutral ABI enum tag, +a canonical-name registry lookup and an optional resolved `DeviceType`; the +old API failed to compile on each missing seam. + +The ABI and public behavior are unchanged: values remain 0=auto, 1=cpu, +2=cuda; parsing and `DeviceName()` still expose `"cuda"`; absent CUDA still +fails loudly before model-path I/O; explicit CPU still bypasses accelerator +selection. Internally slot 2 is `Device::kNamedPlatform`; shared loading calls +`FindPlatformByName(DeviceName(device))` and propagates the registered +platform's type. A mutation-sensitive test supplies `kXPU` and requires +`kXPU` back, so replacing the propagation with a hidden CUDA constant is RED. +The platform test walks every registered canonical name and pins missing-name +failure. + +GREEN in `/dev/shm/vllm-device-leakage-build`, CPU Release, CUDA OFF, +`-Werror`: `test_platform` 11/11 cases · 85 assertions; +`test_loaded_engine_dense` 9/9 · 65; `test_capi` 45/45 · 428. The leakage +checker reports DSR 32 (`kcuda=0`, `is_cuda=0`, `cuda_inc=0`, `vt_ifdef=32`) +with baseline/allowlist unchanged, and its mutation suite is 25/25. H3 source +is untouched. No CUDA runtime, model download, service action, benchmark or +release artifact occurred; the existing CUDA-build A/B remains pending. + +## 2026-08-08 — PR #139 device-selector test registration made non-vacuous + + +Review found that deleting `test_device_selection` from `tests/CMakeLists.txt` +left preflight green: the standalone integration test existed but no tree gate +required CI to build or execute it. `scripts/check-test-registration.py` now +requires the exact target/source invocation and verifies that the shared +`vllm_cpp_add_test` helper both creates the executable and registers it with +CTest. The checker and its 15-test mutation suite run in preflight and the +explicit agent-record CI lane. Deleting the target invocation, substituting its +source, deleting either helper registration, duplicating the target, or dropping +the guard's preflight/CI wiring is RED. The loader header now states explicitly +that internal `Device::kNamedPlatform` is the stable public/wire value and name +`2="cuda"`; behavior and ABI are unchanged. No GPU, model, benchmark or speed +claim is involved; CUDA A/B remains pending. + +## 2026-08-08 — PR #139 registration guard follows configured semantics + + +Mutation re-review proved the text scanner vacuous under five semantic deaths: +the target in a CMake bracket comment, false conditional, or quoted string; the +helper registration in a false conditional; and both CI commands behind `if +false`. The checker now configures a disposable CPU-only project, reads CMake's +File-API codemodel for the exact executable/source pair, and reads CTest JSON for +the registered test. The CI half parses the dedicated YAML literal block into +exact direct argv, rejecting control flow and inert quoted text. The expanded +23-test suite makes all five survivors RED, pins every mutation case's outcome +assertion, and preserves the original 15 cases. A fresh CPU Release `-Werror` +configure exposes `test_device_selection` through both build-target help and +CTest JSON, builds it, and runs the focused selector/platform/loader/C-ABI set +4/4 green. No behavior, ABI, performance, model, GPU, or CUDA A/B claim changed. + +## 2026-08-08 — PR #139 registration guard proves execution, not presence + + +Final mutation review found five more critical survivors in the guard itself. +CTest accepted a required test by name while its helper executed `cmake -E +true`; GitHub Actions commands counted inside a step/job disabled by `if`; +preflight names counted in an inert array instead of the executed +`CHECKERS`/`SUITES`; deleting both mutation assertion-helper assertions left the +suite green; and deleting M19 plus shrinking the in-file numeric range made a +smaller suite self-approve. + +The repair binds every claim to execution. After CMake File-API identifies the +single executable artifact, the disposable configure materializes that path and +CTest JSON must resolve to it as the exact sole argv. The Actions parser accepts +only a canonical literal run block owned by an unconditional job and step. +Preflight parses the named arrays and requires the actual checker/suite loops to +consume them. Wrapper bodies are inspected for their direct production call and +`assertTrue`, and the 27 numbered mutations are pinned in an external manifest; +the 34-test suite also mutates both integrity mechanisms directly. + +All five real review mutations are RED in a scratch exact-head tree (each rc=1), +as are wrong-binary, extra-argv, job-level condition and loop-rebinding nearby +weakenings. Baseline checker + suite are GREEN. This is CPU-only test/governance +hardening: selector behavior, ABI, DSR 32, performance, model state, CUDA A/B and +release state are unchanged. + +## 2026-08-08 — PR #139 registration guard executes every claimed layer + + +The preceding checkpoint was later mutation-reviewed as insufficient: it still +accepted a disabled CTest, non-gating quoted/spaced Actions fields, preflight +loop headers whose bodies never executed, and tandem suite/manifest shrinkage. +Its 34-test conclusion is superseded by this checkpoint, without rewriting that +historical entry. The comprehensive repair replaces source-shape inference at +each boundary. +CMake File-API now selects Release from a multi-config codemodel and the CTest +query uses the same explicit configuration; the exact executable command must +be present and the test must not be `DISABLED`. The Actions subset parser +normalizes quoted/spaced keys and accepts the dedicated exact-argv block only +when its job/step has no `if`, `continue-on-error`, or custom `shell`. Preflight +is executed in a disposable tree with instrumented `python3`/`git` shims; the +checker and suite argv must each be observed exactly once, making a `true` body, +immediate `continue`, false outer branch, or unset array red. + +Mutation-suite integrity is now owned by the production checker, a third layer: +it pins the canonical manifest path plus SHA-256 and the suite's named integrity +method/wrapper contracts. Tandem suite+manifest deletion/rename of M18, redirect +to a byte-identical alternate manifest, and integrity-test deletion all fail. +This proves resistance to those two-layer tandem shrinks, not to an arbitrary +simultaneous rewrite of all three checker layers. RED-first was 50 discovered +tests with 11 failures + 9 missing-contract errors; GREEN is 50/50 plus the +shipped-tree checker. CPU-only governance hardening; selector behavior, ABI, DSR +32, performance, model state, CUDA A/B and release state remain unchanged. + +## 2026-08-08 — PR #139 canonical manifest-path guard made independently observable + + +Final mutation review deleted only the production check that rejects a +non-canonical mutation-manifest path and the 50-test suite remained green. M40 +was not discriminating: it rewrote the suite wrapper source, while its broad +`canonical manifest` assertion could be satisfied by the independent wrapper +shape error. + +M42 now calls `mutation_suite_integrity_errors` with the unchanged canonical +suite source and a byte-for-byte copy of the manifest at an alternate path, and +requires the sole exact error `mutation suite must use the canonical manifest +path`. The external inventory includes M42 and its production-owned SHA-256 was +updated, so dropping the test or its manifest entry is independently red. The +exact delete-only production mutation now fails only M42 (1/51), while the +baseline checker and suite are green at 51/51. CPU-only governance hardening; +selector behavior, ABI, DSR 32, performance, model state, CUDA A/B and release +state remain unchanged. + +## 2026-08-08 — PR #139 M42 outcome assertion is production-pinned + + +Scoped re-review deleted M42's sole path-outcome `assertEqual`, but the suite +stayed green because its unrelated byte-identity `assertTrue` satisfied the +generic mutation-case shape check. The production checker now parses M42 and +requires `assertEqual(errors, ["mutation suite must use the canonical manifest +path"])` exactly. M43 deletes that assertion and replaces its diagnostic in +turn; both mutations produce the dedicated M42 contract error. The external +inventory and production-owned digest now pin M43 as well. + +The direct destructive proofs are red: deleting M42's outcome assertion fails +the shipped-tree check and M43, while deleting the production path guard still +fails M42 itself. Baseline checker plus suite are green at 52/52. This remains +CPU-only governance hardening; selector behavior, ABI, DSR 32, performance, +model state, CUDA A/B and release state are unchanged. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 995864f5..cae4467f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,10 @@ jobs: run: | python3 scripts/check-surface-coverage.py python3 tests/scripts/test_check_surface_coverage.py + - name: Critical regression tests remain executable and CTest-registered + run: | + python3 scripts/check-test-registration.py + python3 tests/scripts/test_check_test_registration.py - name: cuBLASLt GEMM invocation stays dtype-faithful (template parity) run: | python3 scripts/check-gemv-invocation-consistency.py diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index e39a2dc1..0d009eb8 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Track | Status | Next gate | |---|---|---| -| Surface coverage (`ARCH-ONE-SURFACE`) | **No number owed**: CPU tooling/plumbing (guard; ROW 8 device knob, ABI v14 `device`, zero value = the probe, byte-identical); no kernel path touched | Fold lanes carry their own gates; ROW 8 CUDA-build A/B = named residual | +| Surface coverage (`ARCH-ONE-SURFACE`) | **CORRECTNESS COMPLETE:** #139 restores DSR 32 (`kcuda=0`) via registry/name resolution; ABI-v14 selection unchanged; no speed claim | Selector 2/2·11 plus execution-bound CMake/File-API/CTest + CI/preflight + manifest-integrity guard 52/52; CPU platform/loader/C-ABI tests green; CUDA A/B remains residual | | 35B prefill TTFT | 0.93x to 0.98x at every concurrency (2026-08-05) | Attribute the residual, then close | | 35B low-batch MoE decode | CLOSED at low batch (c1 0.975x, c4 wins); c16 0.93x. `VT_ASYNC_DEVICE_MIRROR` **default ON for correctness**. `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) A/B'd speed-NEUTRAL | c16 lever is prefill glue (task #61), not the decode drain. `test_qwen36_async_serving` GREEN | | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | diff --git a/docs/STATUS.md b/docs/STATUS.md index 640deaa9..2e9f88d7 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1563,9 +1563,10 @@ runtime-verified yet. lanes. Guard `scripts/check-surface-coverage.py` (two axes, preflight + CI): every `examples/*` unit is a client of `include/vllm.h` or tracked to a fold row; every `FEATURES.md` C-ABI capability names an entry point or is tracked. - Fold ROW 8: `vllm_model_params.device` (ABI v14, 0=auto/1=cpu/2=cuda); - explicit cpu forces CPU, an absent named device fails loud (never - substituted); `--device` on server+cli. CUDA-build A/B = named residual. + ROW 8 ABI-v14 device selection is behavior-complete; #139 repairs #136's + shared-layer DSR 39->32 through registry/name resolution (`kcuda=0`) with the + baseline unchanged. CPU: selector 2/2·11; semantic execution guard 52/52 + (CTest config/enabled, CI/preflight, manifest integrity). CUDA A/B remains. - **Automatic prefix caching (APC)** is on by default for dense models (hybrid / GDN and attention-free default off, mirroring vLLM), and it now has an end-to-end cache-ON gate on `Qwen/Qwen3-4B` (a shared common prefix reused diff --git a/include/vllm/config/device.h b/include/vllm/config/device.h index db90a773..e0f9495b 100644 --- a/include/vllm/config/device.h +++ b/include/vllm/config/device.h @@ -31,7 +31,9 @@ namespace vllm { enum class Device : int32_t { kAuto = 0, // platform-probed, upstream's "auto" default (device.py:20). kCPU = 1, // force the CPU queue even when an accelerator is present. - kCUDA = 2, // require the CUDA platform; absent => the load fails LOUD. + // ABI v14 slot 2 currently names "cuda". Keep the enum platform-neutral: + // shared selection resolves DeviceName() through the platform registry. + kNamedPlatform = 2, }; // Parse the wire/CLI name ("auto" | "cpu" | "cuda" — the supported subset of diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index bdb97e01..a911bf09 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -12,6 +12,7 @@ #include #include #include +#include #include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" @@ -120,7 +121,9 @@ struct EngineParams { // vLLM's DeviceConfig.device (vllm/config/device.py). kAuto (default) keeps // the accelerator-first probe that has always selected the queue — the // byte-identical default. kCPU forces the CPU queue without consulting the - // probe; kCUDA requires the CUDA platform and the load fails LOUD when it is + // probe. The INTERNAL value Device::kNamedPlatform is the tag for the stable + // PUBLIC/WIRE request whose value and name remain 2="cuda"; it resolves that + // canonical name through the platform registry and fails LOUD when CUDA is // absent (never a silent fallback — an explicit device is assigned verbatim // upstream, device.py:61-66). Exposed on the C ABI as // vllm_model_params.device (ABI v14: 0=auto, 1=cpu, 2=cuda) and on the @@ -128,6 +131,13 @@ struct EngineParams { vllm::Device device = vllm::Device::kAuto; }; +// The shared queue-selection seam used by every LoadedEngine construction +// path. Exposed from this internal header so the explicit named-platform path +// can be gated with a distinctive registered platform/backend rather than a +// parallel pure-policy copy. +vt::Queue SelectQueueForModel(std::string_view architecture, + vllm::Device device); + // Owns the full V1 engine stack (config + weights + tokenizer + Scheduler + // runner -> Executor -> EngineCore; Input/OutputProcessor -> LLMEngine) for a // registered model. The concrete weights/forward are held behind LoadedModel; @@ -188,7 +198,8 @@ class LoadedEngine { // registering fake global platforms: // * kCPU -> vt::DeviceType::kCPU unconditionally — an explicit CPU ask // never consults the accelerator probe, even when CUDA is registered; - // * kCUDA -> vt::DeviceType::kCUDA when cuda_platform_registered, else + // * kNamedPlatform -> the DeviceType returned by the canonical-name + // platform lookup, else // THROWS std::runtime_error naming the device (fail LOUD; the mirror of // vLLM assigning an explicit device verbatim and never substituting // another — vllm/config/device.py:61-66); @@ -197,8 +208,9 @@ class LoadedEngine { // std::invalid_argument if passed. // SelectQueue routes its explicit arms through THIS function, so the gate on // it pins the production policy, not a parallel copy. - static vt::DeviceType ResolveExplicitDeviceType(vllm::Device requested, - bool cuda_platform_registered); + static vt::DeviceType ResolveExplicitDeviceType( + vllm::Device requested, + std::optional named_platform_type); vllm::v1::LLMEngine& engine() { return engine_; } // Lazily start W2's EngineCoreProc + output-handler threads. Once created, diff --git a/include/vllm/platforms/interface.h b/include/vllm/platforms/interface.h index 68b77a5a..ba21a058 100644 --- a/include/vllm/platforms/interface.h +++ b/include/vllm/platforms/interface.h @@ -303,6 +303,12 @@ void RegisterPlatform(DeviceType type, Platform* platform); Platform& GetPlatform(DeviceType type); bool HasPlatform(DeviceType type); +// Find a registered platform by vt::DeviceTypeName(), or nullptr when that +// canonical user-facing name has no usable platform in this process. This is +// the shared explicit-device seam: callers name a platform without branching +// on a backend-specific DeviceType literal. +Platform* FindPlatformByName(std::string_view name); + // The process's active compute platform: the highest-priority registered // accelerator, else CPU — mirrors vLLM resolving `current_platform` by probing // for an accelerator and falling back to CPU. This answers ONLY the process-level diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 561dc99c..f85f4fb0 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -66,6 +66,7 @@ CHECKERS=( check-fusion-consistency check-runner-routing-consistency check-surface-coverage + check-test-registration check-protocol-consistency check-state-order check-now-current @@ -87,6 +88,7 @@ SUITES=( test_check_fusion_consistency test_check_runner_routing_consistency test_check_surface_coverage + test_check_test_registration test_check_protocol_consistency test_check_state_order test_check_now_current diff --git a/scripts/check-test-registration.py b/scripts/check-test-registration.py new file mode 100644 index 00000000..5613279e --- /dev/null +++ b/scripts/check-test-registration.py @@ -0,0 +1,783 @@ +#!/usr/bin/env python3 +"""Require critical regression tests to remain buildable and CTest-registered. + +The CPU CI lane builds every target and runs ``ctest``, but both promises become +vacuous when a regression test is accidentally removed from ``tests/CMakeLists.txt``. +This tree gate pins the small set of tests whose review explicitly requires a +non-vacuous registration guard. It also verifies that the shared helper still +creates an executable *and* registers that executable with CTest. +""" + +from __future__ import annotations + +import ast +import hashlib +import json +import os +import re +import shlex +import subprocess +import sys +import tempfile +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +TESTS_CMAKE = ROOT / "tests/CMakeLists.txt" +PREFLIGHT = ROOT / "scripts/agent-preflight.sh" +CI = ROOT / ".github/workflows/ci.yml" +MUTATION_SUITE = ROOT / "tests/scripts/test_check_test_registration.py" +MUTATION_MANIFEST = ROOT / "tests/scripts/check_test_registration_mutations.txt" +MUTATION_MANIFEST_SHA256 = ( + "46ac35fc533e345aa7735aeffde9d49524598998bdca1b784bd872d21a012803" +) + +REQUIRED_TESTS = { + "test_device_selection": "vllm/entrypoints/test_device_selection.cpp", +} + +def _without_line_comments(text: str) -> str: + """Remove ``#`` comments while preserving quoted ``#`` characters.""" + + cleaned: list[str] = [] + for line in text.splitlines(): + quoted = False + escaped = False + kept: list[str] = [] + for char in line: + if escaped: + kept.append(char) + escaped = False + continue + if char == "\\" and quoted: + kept.append(char) + escaped = True + continue + if char == '"': + quoted = not quoted + kept.append(char) + continue + if char == "#" and not quoted: + break + kept.append(char) + cleaned.append("".join(kept)) + return "\n".join(cleaned) + + +def _configure( + source_dir: Path, build_dir: Path, extra_args: list[str] | None = None +) -> subprocess.CompletedProcess[str]: + """Configure CMake after requesting its semantic target codemodel.""" + + query = build_dir / ".cmake/api/v1/query/codemodel-v2" + query.parent.mkdir(parents=True, exist_ok=True) + query.touch() + command = ["cmake", "-S", str(source_dir), "-B", str(build_dir)] + if extra_args: + command.extend(extra_args) + return subprocess.run(command, text=True, capture_output=True, check=False) + + +def _codemodel_targets( + build_dir: Path, +) -> tuple[str | None, dict[str, dict[str, object]]]: + """Return one configuration and its targets from the CMake File API. + + Multi-config generators describe several artifact paths. Release is the + CI contract and must be selected consistently for both the codemodel and + the subsequent CTest query. + """ + + reply = build_dir / ".cmake/api/v1/reply" + indexes = sorted(reply.glob("index-*.json")) + if not indexes: + return None, {} + index = json.loads(indexes[-1].read_text(encoding="utf-8")) + codemodel_file = index["reply"]["codemodel-v2"]["jsonFile"] + codemodel = json.loads((reply / codemodel_file).read_text(encoding="utf-8")) + configurations = codemodel.get("configurations", []) + if not configurations: + return None, {} + configuration = next( + (entry for entry in configurations if entry.get("name") == "Release"), + configurations[0], + ) + targets: dict[str, dict[str, object]] = {} + for summary in configuration.get("targets", []): + detail = json.loads((reply / summary["jsonFile"]).read_text(encoding="utf-8")) + targets[summary["name"]] = detail + name = configuration.get("name") + return (name if isinstance(name, str) and name else None), targets + + +def _ctest_tests( + build_dir: Path, configuration: str | None +) -> dict[str, dict[str, object]]: + """Return configured CTest command/properties keyed by test name. + + CTest omits ``command`` from JSON when a target executable does not exist + yet. The caller materializes disposable placeholders at the File-API + artifact paths before asking for this document, so the command is the + resolved executable path rather than an uninterpreted CMake token. + """ + + command = ["ctest", "--test-dir", str(build_dir)] + if configuration is not None: + command.extend(["-C", configuration]) + command.append("--show-only=json-v1") + result = subprocess.run( + command, + text=True, + capture_output=True, + check=False, + ) + if result.returncode != 0: + return {} + try: + document = json.loads(result.stdout) + except json.JSONDecodeError: + return {} + return { + test["name"]: { + "command": test.get("command", []), + "properties": test.get("properties", []), + } + for test in document.get("tests", []) + if isinstance(test.get("name"), str) + } + + +def _cmake_truthy(value: object) -> bool: + """Interpret CMake's documented false constants; unknown values fail closed.""" + + if isinstance(value, bool): + return value + if value is None: + return False + normalized = str(value).strip().upper() + return normalized not in { + "", + "0", + "FALSE", + "OFF", + "NO", + "N", + "IGNORE", + "NOTFOUND", + } and not normalized.endswith("-NOTFOUND") + + +def _target_artifact(build_dir: Path, detail: dict[str, object]) -> Path | None: + """Resolve the single configured executable artifact for a File-API target.""" + + artifacts = detail.get("artifacts", []) + if detail.get("type") != "EXECUTABLE" or not isinstance(artifacts, list): + return None + paths = [entry.get("path") for entry in artifacts if isinstance(entry, dict)] + if len(paths) != 1 or not isinstance(paths[0], str): + return None + return (build_dir / paths[0]).resolve() + + +def _materialize_ctest_targets( + build_dir: Path, targets: dict[str, dict[str, object]], required: dict[str, str] +) -> dict[str, Path]: + """Make unbuilt configured target paths resolvable to CTest JSON. + + Only the disposable configure directory is touched. Existing artifacts are + never replaced; absent artifacts become empty executable placeholders long + enough for ``ctest --show-only=json-v1`` to resolve target-name commands. + """ + + artifacts: dict[str, Path] = {} + for target in required: + detail = targets.get(target) + if detail is None: + continue + artifact = _target_artifact(build_dir, detail) + if artifact is None: + continue + artifacts[target] = artifact + if artifact.exists(): + continue + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.touch() + artifact.chmod(0o700) + return artifacts + + +def _configured_contract_errors( + source_dir: Path, + build_dir: Path, + required: dict[str, str], + extra_args: list[str] | None = None, +) -> list[str]: + """Ask CMake/CTest what exists instead of interpreting CMake source text.""" + + configured = _configure(source_dir, build_dir, extra_args) + if configured.returncode != 0: + transcript = configured.stdout + configured.stderr + errors = ["CMake configure failed while proving required test registration"] + for target in sorted(required): + errors.append(f"missing required test target {target} in configured codemodel") + if "already exists" in transcript: + for target in sorted(required): + errors.append(f"required test target {target} is registered 2 times") + errors.append( + "vllm_cpp_add_test does not create an executable with its configured sources" + ) + return errors + + configuration, targets = _codemodel_targets(build_dir) + artifacts = _materialize_ctest_targets(build_dir, targets, required) + tests = _ctest_tests(build_dir, configuration) + errors: list[str] = [] + for target, expected_source in sorted(required.items()): + detail = targets.get(target) + if detail is None: + errors.append(f"missing required test target {target} in configured codemodel") + errors.append( + "vllm_cpp_add_test does not create an executable with its configured sources" + ) + continue + actual_sources = { + Path(source["path"]).as_posix() for source in detail.get("sources", []) + } + if expected_source not in actual_sources: + actual = ", ".join(sorted(actual_sources)) or "" + errors.append( + f"required test target {target} must compile {expected_source}; got {actual}" + ) + artifact = artifacts.get(target) + if artifact is None: + errors.append( + f"required test target {target} has no single configured executable artifact" + ) + if target not in tests: + errors.append( + f"required test target {target} is not registered with CTest; " + "vllm_cpp_add_test does not register that executable with CTest" + ) + elif artifact is not None: + test = tests[target] + command = test.get("command", []) + if not isinstance(command, list): + command = [] + actual_command: Path | None = None + if len(command) == 1: + candidate = Path(command[0]) + actual_command = ( + candidate.resolve() + if candidate.is_absolute() + else (build_dir / candidate).resolve() + ) + if actual_command != artifact: + rendered = shlex.join(command) if command else "" + errors.append( + f"CTest test {target} must execute configured target {target} exactly; " + f"got {rendered}" + ) + properties = test.get("properties", []) + if not isinstance(properties, list): + properties = [] + disabled = [ + prop.get("value") + for prop in properties + if isinstance(prop, dict) and prop.get("name") == "DISABLED" + ] + if any(_cmake_truthy(value) for value in disabled): + errors.append(f"CTest test {target} must not be DISABLED") + return errors + + +def registration_errors( + cmake_text: str, required: dict[str, str] | None = None +) -> list[str]: + """Return violations of the executable + CTest registration contract.""" + + if required is None: + required = REQUIRED_TESTS + with tempfile.TemporaryDirectory(prefix="vllm-registration-unit-") as temporary: + root = Path(temporary) + (root / "CMakeLists.txt").write_text(cmake_text, encoding="utf-8") + for source in { + *required.values(), + "vllm/entrypoints/other.cpp", + }: + path = root / source + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("int registration_guard_dummy;\n", encoding="utf-8") + return _configured_contract_errors(root, root / "build", required) + + +def _indent(line: str) -> int: + return len(line) - len(line.lstrip()) + + +def _literal_block(lines: list[str], header_index: int) -> list[str]: + parent_indent = _indent(lines[header_index]) + raw: list[str] = [] + for candidate in lines[header_index + 1 :]: + if not candidate.strip(): + raw.append("") + continue + if _indent(candidate) <= parent_indent: + break + raw.append(candidate) + nonblank = [line for line in raw if line.strip()] + if not nonblank: + return [] + content_indent = min(_indent(line) for line in nonblank) + return [line[content_indent:] if line.strip() else "" for line in raw] + + +def _yaml_mapping(line: str, indent: int, sequence: bool = False) -> tuple[str, str] | None: + """Parse one direct YAML mapping key in the workflow's structural subset.""" + + if _indent(line) != indent: + return None + content = line[indent:] + if sequence: + if not content.startswith("-"): + return None + content = content[1:].lstrip() + if not content: + return None + match = re.match(r"^(?P'[^']*'|\"[^\"]*\"|[^:#]+?)\s*:\s*(?P.*)$", content) + if match is None: + return None + key = match.group("key").strip() + if len(key) >= 2 and key[0] == key[-1] and key[0] in {"'", '"'}: + key = key[1:-1] + return key.strip(), match.group("value").strip() + + +def _unconditional_ci_run_blocks(text: str) -> list[list[str]]: + """Return literal run blocks owned by unconditional Actions jobs/steps. + + This is deliberately a narrow GitHub-Actions structural parser, not a YAML + implementation: it recognizes the canonical ``jobs -> job -> steps -> -`` + hierarchy and direct ``if``/``run`` fields. A run block in prose, a sibling + mapping, or a conditional job/step never enters the result. + """ + + lines = text.splitlines() + blocks: list[list[str]] = [] + jobs_index = next( + (i for i, line in enumerate(lines) if line == "jobs:"), None + ) + if jobs_index is None: + return blocks + + job_starts = [ + i + for i in range(jobs_index + 1, len(lines)) + if (mapping := _yaml_mapping(lines[i], 2)) is not None + and mapping[1] == "" + ] + for job_pos, job_start in enumerate(job_starts): + job_end = job_starts[job_pos + 1] if job_pos + 1 < len(job_starts) else len(lines) + job_lines = lines[job_start + 1 : job_end] + job_fields = { + mapping[0] + for line in job_lines + if (mapping := _yaml_mapping(line, 4)) is not None + } + if "if" in job_fields: + continue + steps_offset = next( + ( + i + for i, line in enumerate(job_lines) + if _yaml_mapping(line, 4) == ("steps", "") + ), + None, + ) + if steps_offset is None: + continue + steps_start = job_start + 1 + steps_offset + 1 + step_starts = [ + i + for i in range(steps_start, job_end) + if _indent(lines[i]) == 6 and lines[i][6:].startswith("-") + ] + for step_pos, step_start in enumerate(step_starts): + step_end = ( + step_starts[step_pos + 1] + if step_pos + 1 < len(step_starts) + else job_end + ) + step_fields: dict[str, tuple[str, int]] = {} + first = _yaml_mapping(lines[step_start], 6, sequence=True) + if first is not None: + step_fields[first[0]] = (first[1], step_start) + for index in range(step_start + 1, step_end): + mapping = _yaml_mapping(lines[index], 8) + if mapping is not None: + step_fields[mapping[0]] = (mapping[1], index) + if {"if", "continue-on-error", "shell"} & step_fields.keys(): + continue + run = step_fields.get("run") + if run is not None and re.fullmatch(r"\|[-+]?", run[0]): + run_index = run[1] + blocks.append(_literal_block(lines, run_index)) + return blocks + + +def _direct_commands(block: list[str]) -> list[list[str]] | None: + """Parse a literal block that contains only direct, unconditional commands.""" + + commands: list[list[str]] = [] + for line in block: + stripped = line.strip() + if not stripped or stripped.startswith("#"): + continue + if line != line.lstrip(): + return None + try: + argv = shlex.split(stripped, comments=True, posix=True) + except ValueError: + return None + if not argv or any(token in {";", "&&", "||", "|", "&"} for token in argv): + return None + commands.append(argv) + return commands + + +def _ci_has_active_guard_step(ci_text: str) -> bool: + expected = [ + ["python3", "scripts/check-test-registration.py"], + ["python3", "tests/scripts/test_check_test_registration.py"], + ] + return any( + _direct_commands(block) == expected + for block in _unconditional_ci_run_blocks(ci_text) + ) + + +def _active_ci_commands(ci_text: str) -> set[tuple[str, ...]]: + commands: set[tuple[str, ...]] = set() + for block in _unconditional_ci_run_blocks(ci_text): + parsed = _direct_commands(block) + if parsed is not None: + commands.update(tuple(command) for command in parsed) + return commands + + +def _bash_array_values(text: str, name: str) -> list[str] | None: + """Read one top-level Bash array assignment by its actual variable name.""" + + lines = text.splitlines() + starts = [i for i, line in enumerate(lines) if line.strip() == f"{name}=("] + if len(starts) != 1: + return None + values: list[str] = [] + for line in lines[starts[0] + 1 :]: + if line.strip() == ")": + return values + try: + tokens = shlex.split(line, comments=True, posix=True) + except ValueError: + return None + values.extend(tokens) + return None + + +def _trace_preflight_commands(text: str) -> tuple[int, list[tuple[str, ...]]]: + """Execute preflight with Python/Git shims and return Python argv traces.""" + + with tempfile.TemporaryDirectory(prefix="vllm-preflight-trace-") as temporary: + root = Path(temporary) + script = root / "scripts/agent-preflight.sh" + script.parent.mkdir(parents=True) + script.write_text(text, encoding="utf-8") + script.chmod(0o700) + (root / ".agents").mkdir() + (root / ".agents/NOW.md").write_text("trace-only\n", encoding="utf-8") + shim_dir = root / "shim" + shim_dir.mkdir() + trace = root / "python.trace" + python = shim_dir / "python3" + python.write_text( + "#!/bin/sh\n" + "printf '%s\\0' \"$@\" >> \"$VLLM_REGISTRATION_TRACE\"\n" + "printf '\\0' >> \"$VLLM_REGISTRATION_TRACE\"\n", + encoding="utf-8", + ) + python.chmod(0o700) + git = shim_dir / "git" + git.write_text("#!/bin/sh\nexit 1\n", encoding="utf-8") + git.chmod(0o700) + environment = os.environ.copy() + environment["PATH"] = f"{shim_dir}{os.pathsep}{environment.get('PATH', '')}" + environment["VLLM_REGISTRATION_TRACE"] = str(trace) + result = subprocess.run( + ["bash", str(script), "--quiet", "--no-require-role"], + cwd=root, + env=environment, + text=True, + capture_output=True, + check=False, + ) + raw = trace.read_bytes() if trace.exists() else b"" + invocations = [] + for record in raw.split(b"\0\0"): + if not record: + continue + invocations.append( + tuple(token.decode("utf-8") for token in record.split(b"\0") if token) + ) + return result.returncode, invocations + + +def _preflight_execution_errors(text: str) -> list[str]: + returncode, invocations = _trace_preflight_commands(text) + errors: list[str] = [] + checker = ("scripts/check-test-registration.py",) + suite = ("tests/scripts/test_check_test_registration.py",) + if invocations.count(checker) != 1: + errors.append("preflight does not execute CHECKERS through its checker loop") + if invocations.count(suite) != 1: + errors.append("preflight does not execute SUITES through its suite loop") + if returncode != 0: + errors.append(f"instrumented preflight execution failed with rc={returncode}") + return errors + + +def wiring_errors(preflight_text: str, ci_text: str) -> list[str]: + """Return missing preflight/CI wiring for this checker and its mutations.""" + + preflight_source = preflight_text + preflight_text = _without_line_comments(preflight_source) + errors: list[str] = [] + checkers = _bash_array_values(preflight_text, "CHECKERS") + suites = _bash_array_values(preflight_text, "SUITES") + if checkers is None or "check-test-registration" not in checkers: + errors.append("check-test-registration is missing from preflight CHECKERS") + if suites is None or "test_check_test_registration" not in suites: + errors.append("test_check_test_registration is missing from preflight SUITES") + errors.extend(_preflight_execution_errors(preflight_source)) + active_ci_commands = _active_ci_commands(ci_text) + if ("python3", "scripts/check-test-registration.py") not in active_ci_commands: + errors.append("check-test-registration is missing from the explicit CI checker step") + if ( + "python3", + "tests/scripts/test_check_test_registration.py", + ) not in active_ci_commands: + errors.append("test_check_test_registration is missing from the CI mutation suite") + if not _ci_has_active_guard_step(ci_text): + errors.append( + "CI guard step must contain the checker and mutation suite as direct active commands" + ) + return errors + + +def _method_calls(method: ast.AST) -> list[ast.Call]: + return [call for call in ast.walk(method) if isinstance(call, ast.Call)] + + +def mutation_suite_integrity_errors( + source: str, + manifest_text: str | None = None, + *, + manifest_path: Path = MUTATION_MANIFEST, +) -> list[str]: + """Pin the suite/manifest pair from this independent production layer.""" + + errors: list[str] = [] + if manifest_path.resolve() != MUTATION_MANIFEST.resolve(): + errors.append("mutation suite must use the canonical manifest path") + if manifest_text is None: + try: + manifest_text = manifest_path.read_text(encoding="utf-8") + except OSError as exc: + return errors + [f"mutation manifest is unreadable: {exc}"] + digest = hashlib.sha256(manifest_text.encode("utf-8")).hexdigest() + if digest != MUTATION_MANIFEST_SHA256: + errors.append("mutation manifest differs from the production-pinned digest") + manifest = { + line + for line in manifest_text.splitlines() + if line and not line.startswith("#") + } + try: + tree = ast.parse(source) + except SyntaxError as exc: + return errors + [f"mutation suite is not valid Python: {exc}"] + methods = { + node.name: node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + actual = {name for name in methods if name.startswith("test_M")} + if actual != manifest: + errors.append("mutation inventory differs from fixed manifest") + + for name in sorted(actual & manifest): + calls = { + call.func.attr + for call in _method_calls(methods[name]) + if isinstance(call.func, ast.Attribute) + } + if not {"assert_error", "assert_wiring_error", "assertTrue"} & calls: + errors.append(f"{name} has no semantic outcome assertion") + + m42 = methods.get("test_M42_byte_identical_alternate_manifest_path_fails") + m42_path_assertion = False + if m42 is not None: + for call in _method_calls(m42): + if not ( + isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "self" + and call.func.attr == "assertEqual" + and len(call.args) == 2 + and isinstance(call.args[0], ast.Name) + and call.args[0].id == "errors" + and isinstance(call.args[1], ast.List) + and len(call.args[1].elts) == 1 + and isinstance(call.args[1].elts[0], ast.Constant) + and call.args[1].elts[0].value + == "mutation suite must use the canonical manifest path" + ): + continue + m42_path_assertion = True + break + if not m42_path_assertion: + errors.append( + "test_M42 must assert the exact canonical-manifest path diagnostic" + ) + + for name, production_call in { + "assert_error": "registration_errors", + "assert_wiring_error": "wiring_errors", + }.items(): + method = methods.get(name) + if method is None: + errors.append(f"{name} helper is missing") + continue + calls = _method_calls(method) + if not any( + isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "self" + and call.func.attr == "assertTrue" + for call in calls + ): + errors.append(f"{name} has no direct semantic assertion") + if not any( + isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "mod" + and call.func.attr == production_call + for call in calls + ): + errors.append(f"{name} does not call {production_call}") + + wrapper = methods.get("_suite_integrity_errors") + wrapper_ok = False + if wrapper is not None: + for call in _method_calls(wrapper): + if not ( + isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "mod" + and call.func.attr == "mutation_suite_integrity_errors" + ): + continue + path_keywords = [kw for kw in call.keywords if kw.arg == "manifest_path"] + wrapper_ok = ( + len(call.args) == 1 + and isinstance(call.args[0], ast.Name) + and call.args[0].id == "source" + and len(path_keywords) == 1 + and isinstance(path_keywords[0].value, ast.Attribute) + and isinstance(path_keywords[0].value.value, ast.Name) + and path_keywords[0].value.value.id == "mod" + and path_keywords[0].value.attr == "MUTATION_MANIFEST" + ) + if not wrapper_ok: + errors.append("suite integrity wrapper does not use the canonical manifest") + + integrity = methods.get("test_suite_integrity_contract_is_pinned") + if integrity is None: + errors.append("required mutation-suite integrity method is missing") + else: + calls = _method_calls(integrity) + called_wrapper = any( + isinstance(call.func, ast.Name) and call.func.id == "_suite_integrity_errors" + for call in calls + ) + asserted = any( + isinstance(call.func, ast.Attribute) + and isinstance(call.func.value, ast.Name) + and call.func.value.id == "self" + and call.func.attr == "assertEqual" + for call in calls + ) + if not (called_wrapper and asserted): + errors.append("mutation-suite integrity method has no pinned assertion") + return errors + + +def check_tree(root: Path = ROOT) -> list[str]: + paths = { + "tests/CMakeLists.txt": root / "tests/CMakeLists.txt", + "scripts/agent-preflight.sh": root / "scripts/agent-preflight.sh", + ".github/workflows/ci.yml": root / ".github/workflows/ci.yml", + "tests/scripts/test_check_test_registration.py": root + / "tests/scripts/test_check_test_registration.py", + "tests/scripts/check_test_registration_mutations.txt": root + / "tests/scripts/check_test_registration_mutations.txt", + } + missing = [relative for relative, path in paths.items() if not path.is_file()] + if missing: + return [f"required registration-guard input is missing: {path}" for path in missing] + + with tempfile.TemporaryDirectory(prefix="vllm-registration-tree-") as temporary: + registration = _configured_contract_errors( + root, + Path(temporary) / "build", + { + target: f"tests/{source}" + for target, source in REQUIRED_TESTS.items() + }, + [ + "-DVLLM_CPP_CUDA=OFF", + "-DVLLM_CPP_HIP=OFF", + "-DVLLM_CPP_VULKAN=OFF", + "-DVLLM_CPP_METAL=OFF", + "-DVLLM_CPP_MLX=OFF", + "-DVLLM_CPP_TRITON=OFF", + "-DVLLM_CPP_BUILD_TESTS=ON", + "-DVLLM_CPP_BUILD_EXAMPLES=OFF", + "-DVLLM_CPP_SERVER=OFF", + "-DCMAKE_BUILD_TYPE=Release", + ], + ) + integrity = mutation_suite_integrity_errors( + paths["tests/scripts/test_check_test_registration.py"].read_text(encoding="utf-8"), + paths["tests/scripts/check_test_registration_mutations.txt"].read_text(encoding="utf-8"), + manifest_path=paths["tests/scripts/check_test_registration_mutations.txt"], + ) + return registration + wiring_errors( + paths["scripts/agent-preflight.sh"].read_text(encoding="utf-8"), + paths[".github/workflows/ci.yml"].read_text(encoding="utf-8"), + ) + integrity + + +def main() -> int: + errors = check_tree() + if errors: + for error in errors: + print(f"ERROR: {error}", file=sys.stderr) + return 1 + print( + "OK: required regression tests have executable + CTest registration " + "and the guard is wired into preflight/CI." + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 372e52e5..00071ea4 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -615,7 +615,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, ep.device = vllm::Device::kCPU; break; case 2: - ep.device = vllm::Device::kCUDA; + ep.device = vllm::Device::kNamedPlatform; break; default: SetError( @@ -646,7 +646,7 @@ VLLM_API vllm_status vllm_engine_load(const vllm_model_params* params, // explicit CUDA ask cannot be served and is REFUSED rather than // silently downgraded (the same never-substitute rule as the text // engine, vllm/config/device.py:61-66). - if (ep.device == vllm::Device::kCUDA) { + if (ep.device == vllm::Device::kNamedPlatform) { SetError( "vllm_engine_load: device 'cuda' was requested but this " "transcription-only checkpoint serves on the CPU pipeline; use " diff --git a/src/vllm/config/device.cpp b/src/vllm/config/device.cpp index a7396955..f312f081 100644 --- a/src/vllm/config/device.cpp +++ b/src/vllm/config/device.cpp @@ -15,7 +15,7 @@ Device DeviceFromString(const std::string& value) { return Device::kCPU; } if (value == "cuda") { - return Device::kCUDA; + return Device::kNamedPlatform; } // Mirrors pydantic rejecting a value outside the Device Literal // (vllm/config/device.py:13). "tpu"/"xpu" are upstream names this build @@ -31,7 +31,7 @@ const char* DeviceName(Device device) { return "auto"; case Device::kCPU: return "cpu"; - case Device::kCUDA: + case Device::kNamedPlatform: return "cuda"; } return "invalid"; diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index b9685d8a..846d77eb 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -40,8 +40,6 @@ namespace vllm::entrypoints { namespace fs = std::filesystem; -namespace { - // `architecture` is the model's registered architecture string. It is what lets // a PARTIAL backend decline a model whose kernels it has not registered, instead // of being selected and then failing deep inside a kernel bind. Empty means "no @@ -54,10 +52,15 @@ namespace { // a failure to serve the named device PROPAGATES instead of falling back to // CPU (mirror of vLLM never substituting an explicitly named device, // vllm/config/device.py:61-66). -vt::Queue SelectQueue(std::string_view architecture, vllm::Device device) { +vt::Queue SelectQueueForModel(std::string_view architecture, + vllm::Device device) { if (device != vllm::Device::kAuto) { + const vllm::platforms::Platform* named_platform = + vllm::platforms::FindPlatformByName(vllm::DeviceName(device)); const vt::DeviceType resolved = LoadedEngine::ResolveExplicitDeviceType( - device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + device, named_platform == nullptr + ? std::nullopt + : std::optional{named_platform->device_type()}); if (resolved == vt::DeviceType::kCPU) { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } @@ -96,6 +99,8 @@ vt::Queue SelectQueue(std::string_view architecture, vllm::Device device) { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } +namespace { + bool DirectDeviceLoadRequested() { const char* release = std::getenv("VT_RELEASE_HOST_WEIGHTS"); if (release != nullptr && release[0] == '0') return false; @@ -485,14 +490,15 @@ bool LoadedEngine::ResolveEnablePrefixCaching(const EngineParams& params, // src/vllm/platforms/cuda.cpp Registrar — kCUDA registers only when a usable // GPU probed). vt::DeviceType LoadedEngine::ResolveExplicitDeviceType( - vllm::Device requested, bool cuda_platform_registered) { + vllm::Device requested, + std::optional named_platform_type) { switch (requested) { case vllm::Device::kCPU: // Explicit CPU never consults the accelerator probe: even on a // CUDA-capable build/process this selects the CPU queue. return vt::DeviceType::kCPU; - case vllm::Device::kCUDA: - if (!cuda_platform_registered) { + case vllm::Device::kNamedPlatform: + if (!named_platform_type.has_value()) { throw std::runtime_error( "device 'cuda' was requested but no CUDA platform is available in " "this build/process (an explicitly named device is never silently " @@ -500,7 +506,7 @@ vt::DeviceType LoadedEngine::ResolveExplicitDeviceType( "or device=cpu, or run a CUDA build on a machine with a usable " "GPU)"); } - return vt::DeviceType::kCUDA; + return *named_platform_type; case vllm::Device::kAuto: break; // auto resolves through the probe in SelectQueue, not here. } @@ -699,8 +705,8 @@ LoadedEngine::LoadedEngine(HfConfig config, runner_(config_, *model_, kv_cfg_, preselected_queue != nullptr ? *preselected_queue - : SelectQueue(model_->registration().architecture, - params.device), + : SelectQueueForModel(model_->registration().architecture, + params.device), /*max_num_reqs=*/params.max_num_seqs > 0 ? params.max_num_seqs : 8, max_model_len_, /*max_num_batched_tokens=*/max_num_batched_tokens_, @@ -871,12 +877,16 @@ std::unique_ptr LoadedEngine::FromModelDir( // (vllm/engine/arg_utils.py:1878 builds DeviceConfig first; // device.py __post_init__ resolves immediately). An explicitly named absent // device therefore fails HERE, loudly, and is never masked by a later - // path/tokenizer error. The result is discarded: SelectQueue re-runs the - // SAME ResolveExplicitDeviceType when it actually creates the queue, so the - // policy has exactly one owner. + // path/tokenizer error. The result is discarded: SelectQueueForModel re-runs + // the SAME ResolveExplicitDeviceType when it actually creates the queue, so + // the policy has exactly one owner. if (params.device != vllm::Device::kAuto) { + const vllm::platforms::Platform* named_platform = + vllm::platforms::FindPlatformByName(vllm::DeviceName(params.device)); (void)ResolveExplicitDeviceType( - params.device, vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)); + params.device, named_platform == nullptr + ? std::nullopt + : std::optional{named_platform->device_type()}); } const fs::path dir(model_dir); @@ -1062,7 +1072,8 @@ std::unique_ptr LoadedEngine::FromModelDir( // Select before loading so an eligible discrete-CUDA dense loader stages each // completed layer to the exact queue the runner will use. If construction // fails before the runner takes over, destroy the selected native stream. - vt::Queue load_queue = SelectQueue(registration.architecture, params.device); + vt::Queue load_queue = + SelectQueueForModel(registration.architecture, params.device); try { std::unique_ptr model = ModelRegistry::Load( config, ModelSource::FromSafetensorsOwned(shards, &load_queue)); diff --git a/src/vllm/platforms/platform.cpp b/src/vllm/platforms/platform.cpp index 32e4a969..20d24571 100644 --- a/src/vllm/platforms/platform.cpp +++ b/src/vllm/platforms/platform.cpp @@ -73,6 +73,16 @@ Platform& GetPlatform(DeviceType type) { bool HasPlatform(DeviceType type) { return Registry()[Index(type)] != nullptr; } +Platform* FindPlatformByName(std::string_view name) { + for (size_t i = 0; i < vt::kNumDeviceTypes; ++i) { + Platform* platform = Registry()[i]; + if (platform == nullptr) continue; + const DeviceType type = static_cast(i); + if (name == vt::DeviceTypeName(type)) return platform; + } + return nullptr; +} + Platform& CurrentPlatform() { for (DeviceType type : kCurrentPriority) { Platform* p = Registry()[static_cast(type)]; diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index fbfe8319..7b075be4 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -59,6 +59,8 @@ target_compile_definitions(test_minimax_h3_video_fold PRIVATE vllm_cpp_add_test(test_video_api vllm/entrypoints/openai/test_video_api.cpp) target_include_directories(test_model_registry PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_platform vllm/platforms/test_platform.cpp) +vllm_cpp_add_test(test_device_selection + vllm/entrypoints/test_device_selection.cpp) vllm_cpp_add_test(test_linear_method vllm/model_executor/layers/test_linear_method.cpp) target_include_directories(test_linear_method PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_decode_graph_sizes vllm/models/test_decode_graph_sizes.cpp) diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index c25d842b..a241c933 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1832,7 +1832,7 @@ TEST_CASE("capi v14: explicit cpu forces the CPU queue at the EngineParams seam" // the pinned message out of construction — never silently build on CPU. if (!vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { EngineParams cuda_params = SyntheticParams(); - cuda_params.device = vllm::Device::kCUDA; + cuda_params.device = vllm::Device::kNamedPlatform; CHECK_THROWS_WITH_AS( LoadedEngine(MakeConfig(), MakeWeights(c), BuildFixture(), cuda_params), doctest::Contains("device 'cuda' was requested but no CUDA platform"), diff --git a/tests/scripts/check_test_registration_mutations.txt b/tests/scripts/check_test_registration_mutations.txt new file mode 100644 index 00000000..4f301f50 --- /dev/null +++ b/tests/scripts/check_test_registration_mutations.txt @@ -0,0 +1,47 @@ +# Fixed mutation inventory for test_check_test_registration.py. +# +# Keep this separate from the suite's discovery logic: deleting a mutation case +# and weakening an in-file range must not make the smaller suite self-approve. +test_M1_deleting_test_invocation_fails +test_M2_changing_test_source_fails +test_M3_deleting_add_executable_from_helper_fails +test_M4_deleting_add_test_from_helper_fails +test_M5_duplicate_target_fails +test_M6_commented_out_invocation_fails +test_M7_commented_out_add_executable_fails +test_M8_commented_out_add_test_fails +test_M9_deleting_preflight_checker_fails +test_M10_deleting_preflight_suite_fails +test_M11_deleting_ci_checker_fails +test_M12_deleting_ci_suite_fails +test_M13_bracket_commented_target_fails +test_M14_target_in_false_conditional_fails +test_M15_quoted_target_text_fails +test_M16_helper_registration_in_false_conditional_fails +test_M17_helper_executable_in_false_conditional_fails +test_M18_ci_commands_behind_false_shell_branch_fail +test_M19_ci_commands_as_quoted_text_fail +test_M20_ctest_noop_command_fails +test_M21_ctest_wrong_binary_fails +test_M22_ctest_extra_argument_fails +test_M23_ci_step_with_false_condition_fails +test_M24_ci_job_with_false_condition_fails +test_M25_preflight_names_in_inert_arrays_fail +test_M26_preflight_checker_loop_rebound_fails +test_M27_preflight_suite_loop_rebound_fails +test_M28_disabled_ctest_fails +test_M29_ci_continue_on_error_fails +test_M30_ci_quoted_false_condition_fails +test_M31_ci_spaced_false_condition_fails +test_M32_ci_inert_shell_fails +test_M33_preflight_checker_loop_noop_body_fails +test_M34_preflight_suite_loop_noop_body_fails +test_M35_preflight_checker_loop_immediate_continue_fails +test_M36_preflight_loops_behind_false_outer_branch_fail +test_M37_preflight_unsets_checkers_before_loop_fails +test_M38_deleting_M18_and_manifest_entry_fails +test_M39_renaming_M18_and_manifest_entry_fails +test_M40_redirecting_suite_to_alternate_manifest_fails +test_M41_deleting_manifest_integrity_test_fails +test_M42_byte_identical_alternate_manifest_path_fails +test_M43_M42_keeps_exact_path_outcome_assertion diff --git a/tests/scripts/test_check_test_registration.py b/tests/scripts/test_check_test_registration.py new file mode 100644 index 00000000..b1799d04 --- /dev/null +++ b/tests/scripts/test_check_test_registration.py @@ -0,0 +1,590 @@ +#!/usr/bin/env python3 +"""Mutation checks for the required CMake/CTest registration guard.""" + +from __future__ import annotations + +import ast +import importlib.util +import os +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-test-registration.py" +SPEC = importlib.util.spec_from_file_location("check_test_registration", CHECKER) +assert SPEC is not None and SPEC.loader is not None +mod = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = mod +SPEC.loader.exec_module(mod) + + +PASSING_CMAKE = """ +cmake_minimum_required(VERSION 3.20) +project(registration_guard LANGUAGES CXX) +enable_testing() +add_library(vllm_core INTERFACE) +add_library(vllm::vllm ALIAS vllm_core) +add_library(vllm_test_main INTERFACE) + +function(vllm_cpp_add_test name) + add_executable(${name} ${ARGN}) + target_link_libraries(${name} PRIVATE vllm::vllm vllm_test_main) + add_test(NAME ${name} COMMAND ${name}) +endfunction() + +vllm_cpp_add_test(test_device_selection + vllm/entrypoints/test_device_selection.cpp) +""" + +PASSING_CI = """jobs: + checks: + steps: + - name: Critical regression tests remain executable and CTest-registered + run: | + python3 scripts/check-test-registration.py + python3 tests/scripts/test_check_test_registration.py +""" + +PASSING_PREFLIGHT = """CHECKERS=( + check-test-registration +) +SUITES=( + test_check_test_registration +) +for checker in "${CHECKERS[@]}"; do + python3 "scripts/$checker.py" +done +for suite in "${SUITES[@]}"; do + python3 "tests/scripts/$suite.py" +done +""" + + +def _suite_integrity_errors(source: str) -> list[str]: + """Exercise the production-owned, canonical mutation-suite contract.""" + + return mod.mutation_suite_integrity_errors( + source, manifest_path=mod.MUTATION_MANIFEST + ) + + +class RegistrationMutationTests(unittest.TestCase): + def assert_error(self, text: str, needle: str) -> None: + errors = mod.registration_errors(text) + self.assertTrue(any(needle in error for error in errors), errors) + + def test_minimal_complete_registration_passes(self) -> None: + self.assertEqual(mod.registration_errors(PASSING_CMAKE), []) + + def test_M1_deleting_test_invocation_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)\n", + "", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M2_changing_test_source_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm/entrypoints/test_device_selection.cpp", "vllm/entrypoints/other.cpp" + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "must compile vllm/entrypoints/test_device_selection.cpp") + + def test_M3_deleting_add_executable_from_helper_fails(self) -> None: + mutated = PASSING_CMAKE.replace(" add_executable(${name} ${ARGN})\n", "") + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "does not create an executable") + + def test_M4_deleting_add_test_from_helper_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})\n", "" + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "does not register that executable with CTest") + + def test_M5_duplicate_target_fails(self) -> None: + mutated = PASSING_CMAKE + ( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)\n" + ) + self.assert_error(mutated, "registered 2 times") + + def test_M6_commented_out_invocation_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)", + "# vllm_cpp_add_test(test_device_selection " + "vllm/entrypoints/test_device_selection.cpp)", + ) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M7_commented_out_add_executable_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_executable(${name} ${ARGN})", " # add_executable(${name} ${ARGN})" + ) + self.assert_error(mutated, "does not create an executable") + + def test_M8_commented_out_add_test_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " # add_test(NAME ${name} COMMAND ${name})", + ) + self.assert_error(mutated, "does not register that executable with CTest") + + def test_M13_bracket_commented_target_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)", + "#[[\nvllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)\n]]", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M14_target_in_false_conditional_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)", + "if(FALSE)\n" + " vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)\n" + "endif()", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M15_quoted_target_text_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + "vllm_cpp_add_test(test_device_selection\n" + " vllm/entrypoints/test_device_selection.cpp)", + 'set(dead "vllm_cpp_add_test(test_device_selection ' + 'vllm/entrypoints/test_device_selection.cpp)")', + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M16_helper_registration_in_false_conditional_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " if(FALSE)\n" + " add_test(NAME ${name} COMMAND ${name})\n" + " endif()", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "is not registered with CTest") + + def test_M17_helper_executable_in_false_conditional_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_executable(${name} ${ARGN})", + " if(FALSE)\n" + " add_executable(${name} ${ARGN})\n" + " endif()", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "missing required test target test_device_selection") + + def test_M20_ctest_noop_command_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " add_test(NAME ${name} COMMAND ${CMAKE_COMMAND} -E true)", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error( + mutated, "must execute configured target test_device_selection exactly" + ) + + def test_M21_ctest_wrong_binary_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " add_test(NAME ${name} COMMAND other)", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error( + mutated, "must execute configured target test_device_selection exactly" + ) + + def test_M22_ctest_extra_argument_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " add_test(NAME ${name} COMMAND ${name} --list-test-cases)", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error( + mutated, "must execute configured target test_device_selection exactly" + ) + + def test_M28_disabled_ctest_fails(self) -> None: + mutated = PASSING_CMAKE.replace( + " add_test(NAME ${name} COMMAND ${name})", + " add_test(NAME ${name} COMMAND ${name})\n" + " set_tests_properties(${name} PROPERTIES DISABLED TRUE)", + ) + self.assertNotEqual(mutated, PASSING_CMAKE) + self.assert_error(mutated, "must not be DISABLED") + + def test_ninja_multi_config_resolves_release_ctest_command(self) -> None: + old_generator = os.environ.get("CMAKE_GENERATOR") + os.environ["CMAKE_GENERATOR"] = "Ninja Multi-Config" + try: + self.assertEqual(mod.registration_errors(PASSING_CMAKE), []) + finally: + if old_generator is None: + os.environ.pop("CMAKE_GENERATOR", None) + else: + os.environ["CMAKE_GENERATOR"] = old_generator + + +class WiringMutationTests(unittest.TestCase): + def setUp(self) -> None: + self.preflight = PASSING_PREFLIGHT + self.ci = PASSING_CI + + def assert_wiring_error(self, preflight: str, ci: str, needle: str) -> None: + errors = mod.wiring_errors(preflight, ci) + self.assertTrue(any(needle in error for error in errors), errors) + + def test_complete_preflight_and_ci_wiring_passes(self) -> None: + self.assertEqual(mod.wiring_errors(self.preflight, self.ci), []) + + def test_M9_deleting_preflight_checker_fails(self) -> None: + mutated = self.preflight.replace(" check-test-registration\n", "") + self.assert_wiring_error(mutated, self.ci, "preflight CHECKERS") + + def test_M10_deleting_preflight_suite_fails(self) -> None: + mutated = self.preflight.replace(" test_check_test_registration\n", "") + self.assert_wiring_error(mutated, self.ci, "preflight SUITES") + + def test_M11_deleting_ci_checker_fails(self) -> None: + mutated = self.ci.replace( + " python3 scripts/check-test-registration.py\n", "" + ) + self.assert_wiring_error(self.preflight, mutated, "CI checker") + + def test_M12_deleting_ci_suite_fails(self) -> None: + mutated = self.ci.replace( + " python3 tests/scripts/test_check_test_registration.py\n", "" + ) + self.assert_wiring_error(self.preflight, mutated, "CI mutation suite") + + def test_M18_ci_commands_behind_false_shell_branch_fail(self) -> None: + mutated = self.ci.replace( + " python3 scripts/check-test-registration.py\n" + " python3 tests/scripts/test_check_test_registration.py\n", + " if false; then\n" + " python3 scripts/check-test-registration.py\n" + " python3 tests/scripts/test_check_test_registration.py\n" + " fi\n", + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M19_ci_commands_as_quoted_text_fail(self) -> None: + mutated = self.ci.replace( + " python3 scripts/check-test-registration.py\n", + ' echo "python3 scripts/check-test-registration.py"\n', + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M23_ci_step_with_false_condition_fails(self) -> None: + mutated = self.ci.replace( + " - name: Critical regression tests remain executable and CTest-registered\n" + " run: |\n", + " - name: Critical regression tests remain executable and CTest-registered\n" + " if: ${{ false }}\n" + " run: |\n", + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M24_ci_job_with_false_condition_fails(self) -> None: + mutated = self.ci.replace( + " checks:\n", + " checks:\n" + " if: ${{ false }}\n", + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M25_preflight_names_in_inert_arrays_fail(self) -> None: + mutated = self.preflight.replace( + " check-test-registration\n", "" + ).replace( + " test_check_test_registration\n", "" + ) + ( + "INERT=(\n" + " check-test-registration\n" + " test_check_test_registration\n" + ")\n" + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "preflight CHECKERS") + self.assert_wiring_error(mutated, self.ci, "preflight SUITES") + + def test_M26_preflight_checker_loop_rebound_fails(self) -> None: + mutated = self.preflight.replace( + 'for checker in "${CHECKERS[@]}"; do', + 'for checker in "${INERT[@]}"; do', + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute CHECKERS") + + def test_M27_preflight_suite_loop_rebound_fails(self) -> None: + mutated = self.preflight.replace( + 'for suite in "${SUITES[@]}"; do', + 'for suite in "${INERT[@]}"; do', + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute SUITES") + + def test_M29_ci_continue_on_error_fails(self) -> None: + mutated = self.ci.replace( + " run: |\n", " continue-on-error: true\n run: |\n" + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M30_ci_quoted_false_condition_fails(self) -> None: + mutated = self.ci.replace( + " run: |\n", " 'if': ${{ false }}\n run: |\n" + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M31_ci_spaced_false_condition_fails(self) -> None: + mutated = self.ci.replace( + " run: |\n", " if : ${{ false }}\n run: |\n" + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M32_ci_inert_shell_fails(self) -> None: + mutated = self.ci.replace( + " run: |\n", " shell: /bin/true {0}\n run: |\n" + ) + self.assertNotEqual(mutated, self.ci) + self.assert_wiring_error(self.preflight, mutated, "direct active commands") + + def test_M33_preflight_checker_loop_noop_body_fails(self) -> None: + mutated = self.preflight.replace( + ' python3 "scripts/$checker.py"', " true" + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute CHECKERS") + + def test_M34_preflight_suite_loop_noop_body_fails(self) -> None: + mutated = self.preflight.replace( + ' python3 "tests/scripts/$suite.py"', " true" + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute SUITES") + + def test_M35_preflight_checker_loop_immediate_continue_fails(self) -> None: + mutated = self.preflight.replace( + 'for checker in "${CHECKERS[@]}"; do\n', + 'for checker in "${CHECKERS[@]}"; do\n continue\n', + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute CHECKERS") + + def test_M36_preflight_loops_behind_false_outer_branch_fail(self) -> None: + loop_block = ( + 'for checker in "${CHECKERS[@]}"; do\n' + ' python3 "scripts/$checker.py"\n' + "done\n" + 'for suite in "${SUITES[@]}"; do\n' + ' python3 "tests/scripts/$suite.py"\n' + "done\n" + ) + mutated = self.preflight.replace(loop_block, "if false; then\n" + loop_block + "fi\n") + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute CHECKERS") + self.assert_wiring_error(mutated, self.ci, "execute SUITES") + + def test_M37_preflight_unsets_checkers_before_loop_fails(self) -> None: + mutated = self.preflight.replace( + 'for checker in "${CHECKERS[@]}"; do', + 'unset CHECKERS\nfor checker in "${CHECKERS[@]}"; do', + ) + self.assertNotEqual(mutated, self.preflight) + self.assert_wiring_error(mutated, self.ci, "execute CHECKERS") + + +class ShippedTreeTests(unittest.TestCase): + def test_shipped_tree_is_registered_and_wired(self) -> None: + self.assertEqual(mod.check_tree(ROOT), []) + + +class SuiteIntegrityTests(unittest.TestCase): + def test_deleting_wrapper_assertions_is_caught(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + mutated = source.replace( + " self.assertTrue(any(needle in error for error in errors), errors)\n", + "", + ) + self.assertNotEqual(mutated, source) + errors = _suite_integrity_errors(mutated) + self.assertTrue(any("assert_error has no direct semantic assertion" in e for e in errors)) + self.assertTrue( + any("assert_wiring_error has no direct semantic assertion" in e for e in errors) + ) + + def test_deleting_a_manifested_mutation_is_caught(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + tree = ast.parse(source) + method = next( + node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name == "test_M19_ci_commands_as_quoted_text_fail" + ) + lines = source.splitlines(keepends=True) + mutated = "".join(lines[: method.lineno - 1] + lines[method.end_lineno :]) + errors = _suite_integrity_errors(mutated) + self.assertTrue(any("fixed manifest" in error for error in errors), errors) + + def test_mutation_cases_keep_their_outcome_assertions(self) -> None: + """Make deleting a mutation's only meaningful assertion turn RED.""" + + source = Path(__file__).read_text(encoding="utf-8") + self.assertEqual(_suite_integrity_errors(source), []) + + def test_assertion_helpers_keep_semantic_outcome_assertions(self) -> None: + """The mutation wrappers must themselves retain a real assertion.""" + + source = Path(__file__).read_text(encoding="utf-8") + errors = _suite_integrity_errors(source) + self.assertFalse(any("assert_error" in error for error in errors), errors) + self.assertFalse(any("assert_wiring_error" in error for error in errors), errors) + + tree = ast.parse(source) + methods = { + node.name: node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + for name in { + "test_minimal_complete_registration_passes", + "test_complete_preflight_and_ci_wiring_passes", + "test_shipped_tree_is_registered_and_wired", + }: + method = methods[name] + calls = { + call.func.attr + for call in ast.walk(method) + if isinstance(call, ast.Call) and isinstance(call.func, ast.Attribute) + } + self.assertIn("assertEqual", calls, name) + + def test_suite_integrity_contract_is_pinned(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + self.assertEqual(_suite_integrity_errors(source), []) + + def test_M38_deleting_M18_and_manifest_entry_fails(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + tree = ast.parse(source) + method = next( + node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name == "test_M18_ci_commands_behind_false_shell_branch_fail" + ) + lines = source.splitlines(keepends=True) + mutated_source = "".join(lines[: method.lineno - 1] + lines[method.end_lineno :]) + manifest = mod.MUTATION_MANIFEST.read_text(encoding="utf-8") + mutated_manifest = manifest.replace( + "test_M18_ci_commands_behind_false_shell_branch_fail\n", "" + ) + errors = mod.mutation_suite_integrity_errors( + mutated_source, manifest_text=mutated_manifest + ) + self.assertTrue(any("pinned digest" in error for error in errors), errors) + + def test_M39_renaming_M18_and_manifest_entry_fails(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + old = "test_M18_ci_commands_behind_false_shell_branch_fail" + new = "test_M18_ci_commands_hidden_by_false_shell_branch_fail" + mutated_source = source.replace(old, new) + manifest = mod.MUTATION_MANIFEST.read_text(encoding="utf-8") + mutated_manifest = manifest.replace(old, new) + errors = mod.mutation_suite_integrity_errors( + mutated_source, manifest_text=mutated_manifest + ) + self.assertTrue(any("pinned digest" in error for error in errors), errors) + + def test_M40_redirecting_suite_to_alternate_manifest_fails(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + mutated = source.replace( + "source, manifest_path=mod.MUTATION_MANIFEST", + 'source, manifest_path=ROOT / "tests/scripts/alternate_mutations.txt"', + 1, + ) + self.assertNotEqual(mutated, source) + errors = mod.mutation_suite_integrity_errors( + mutated, manifest_text=mod.MUTATION_MANIFEST.read_text(encoding="utf-8") + ) + self.assertTrue(any("canonical manifest" in error for error in errors), errors) + + def test_M41_deleting_manifest_integrity_test_fails(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + tree = ast.parse(source) + method = next( + node + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + and node.name == "test_suite_integrity_contract_is_pinned" + ) + lines = source.splitlines(keepends=True) + mutated = "".join(lines[: method.lineno - 1] + lines[method.end_lineno :]) + errors = mod.mutation_suite_integrity_errors( + mutated, manifest_text=mod.MUTATION_MANIFEST.read_text(encoding="utf-8") + ) + self.assertTrue(any("integrity method is missing" in error for error in errors), errors) + + def test_M42_byte_identical_alternate_manifest_path_fails(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + manifest = mod.MUTATION_MANIFEST.read_text(encoding="utf-8") + with tempfile.TemporaryDirectory( + prefix="vllm-registration-manifest-" + ) as temporary: + alternate = Path(temporary) / mod.MUTATION_MANIFEST.name + alternate.write_text(manifest, encoding="utf-8") + self.assertTrue( + alternate.read_bytes() == mod.MUTATION_MANIFEST.read_bytes() + ) + errors = mod.mutation_suite_integrity_errors( + source, manifest_path=alternate + ) + self.assertEqual( + errors, ["mutation suite must use the canonical manifest path"] + ) + + def test_M43_M42_keeps_exact_path_outcome_assertion(self) -> None: + source = Path(__file__).read_text(encoding="utf-8") + assertion = ( + " self.assertEqual(\n" + ' errors, ["mutation suite must use the canonical manifest path"]\n' + " )\n" + ) + for label, replacement in { + "deleted": "", + "wrong diagnostic": " self.assertEqual(errors, [])\n", + }.items(): + with self.subTest(mutation=label): + mutated = source.replace(assertion, replacement, 1) + self.assertNotEqual(mutated, source) + errors = _suite_integrity_errors(mutated) + self.assertTrue( + "test_M42 must assert the exact canonical-manifest path diagnostic" + in errors, + errors, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/vllm/entrypoints/test_device_selection.cpp b/tests/vllm/entrypoints/test_device_selection.cpp new file mode 100644 index 00000000..4d0b27b9 --- /dev/null +++ b/tests/vllm/entrypoints/test_device_selection.cpp @@ -0,0 +1,106 @@ +// ARCH-ONE-SURFACE ROW 8: mutation-sensitive integration gate for canonical +// platform lookup and explicit queue selection. This is a separate executable +// because it deliberately registers a distinctive fake platform/backend in the +// otherwise-unused XPU slot; process isolation keeps the global registries from +// affecting the general platform and loader suites. +#include + +#include +#include +#include +#include + +#include "vllm/config/device.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/platforms/interface.h" +#include "vt/backend.h" + +namespace { + +class FakeXpuBackend final : public vt::Backend { + public: + void* Alloc(size_t bytes) override { + return std::malloc(bytes == 0 ? 1 : bytes); + } + void Free(void* p) override { std::free(p); } + void Memset(vt::Queue&, void* p, int value, size_t bytes) override { + std::memset(p, value, bytes); + } + void Copy(vt::Queue&, void* dst, const void* src, size_t bytes) override { + std::memcpy(dst, src, bytes); + } + vt::Queue CreateQueue() override { + ++create_queue_calls; + return vt::Queue{vt::Device{vt::DeviceType::kXPU, 17}, nullptr}; + } + bool UnifiedMemory() const override { return true; } + + int create_queue_calls = 0; +}; + +class FakeXpuPlatform final : public vllm::platforms::Platform { + public: + explicit FakeXpuPlatform(FakeXpuBackend& backend) : backend_(backend) {} + + vt::DeviceType device_type() const override { return vt::DeviceType::kXPU; } + vt::Backend& backend() const override { return backend_; } + vllm::platforms::DeviceCapability get_device_capability() const override { + return {}; + } + std::vector supported_dtypes() const override { + return {vt::DType::kBF16}; + } + vllm::platforms::ResidencyPolicy residency_policy() const override { + return {}; + } + + private: + FakeXpuBackend& backend_; +}; + +FakeXpuBackend& Backend() { + static FakeXpuBackend backend; + return backend; +} + +FakeXpuPlatform& Platform() { + static FakeXpuPlatform platform(Backend()); + return platform; +} + +void RegisterDistinctivePlatform() { + vt::RegisterBackend(vt::DeviceType::kXPU, &Backend()); + vllm::platforms::RegisterPlatform(vt::DeviceType::kXPU, &Platform()); +} + +} // namespace + +TEST_CASE("platform lookup preserves a distinctive non-CPU platform identity") { + RegisterDistinctivePlatform(); + + vllm::platforms::Platform* found = + vllm::platforms::FindPlatformByName("xpu"); + REQUIRE(found == &Platform()); + CHECK(found->device_type() == vt::DeviceType::kXPU); + + for (std::string_view invalid : {"xpu-junk", "xpu ", " xpu", "XPU", + "xp", "xpu-extra"}) { + CAPTURE(invalid); + CHECK(vllm::platforms::FindPlatformByName(invalid) == nullptr); + } +} + +TEST_CASE("explicit named-device queue propagates the found platform type") { + RegisterDistinctivePlatform(); + // Slot 2 remains publicly named "cuda". Registering the XPU-shaped fake in + // that lookup slot separates the canonical public name from the platform's + // returned type: a hidden CPU/CUDA constant cannot satisfy this assertion. + vllm::platforms::RegisterPlatform(vt::DeviceType::kCUDA, &Platform()); + Backend().create_queue_calls = 0; + + vt::Queue queue = vllm::entrypoints::SelectQueueForModel( + "DistinctiveArchitecture", vllm::Device::kNamedPlatform); + CHECK(queue.device.type == vt::DeviceType::kXPU); + CHECK(queue.device.index == 17); + CHECK(Backend().create_queue_calls == 1); +} diff --git a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp index 5d4974b5..72912e33 100644 --- a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp +++ b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp @@ -457,33 +457,39 @@ TEST_CASE("loaded_engine: prefix caching mirrors model-capability defaults") { // registered accelerator") that a CPU-only process could otherwise never // exercise. SelectQueue routes its explicit arms through THIS function // (model_loader.cpp), so these pins bind the production policy, not a copy. -TEST_CASE("loaded_engine: ResolveExplicitDeviceType — cpu never probes, cuda never falls back") { +TEST_CASE("loaded_engine: ResolveExplicitDeviceType uses the named platform without fallback") { using vllm::Device; - // Explicit CPU resolves CPU regardless of what is registered. The `true` arm - // is the CUDA-build pin: a registered accelerator must NOT win over an + // Explicit CPU resolves CPU regardless of what the name lookup found. The + // non-CPU value is the accelerator-build pin: a registered accelerator must + // NOT win over an // explicit cpu ask (the fold-ROW-8 defect was that an embedder could not ASK // for CPU at all). - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, true) == + CHECK(LoadedEngine::ResolveExplicitDeviceType( + Device::kCPU, std::optional{vt::DeviceType::kXPU}) == vt::DeviceType::kCPU); - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, false) == + CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCPU, std::nullopt) == vt::DeviceType::kCPU); - // Explicit CUDA with the platform registered resolves CUDA. - CHECK(LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, true) == - vt::DeviceType::kCUDA); + // The explicit named-platform arm returns what the registry found. kXPU is + // deliberate mutation sensitivity: a hidden CUDA constant cannot satisfy it. + CHECK(LoadedEngine::ResolveExplicitDeviceType( + Device::kNamedPlatform, std::optional{vt::DeviceType::kXPU}) == + vt::DeviceType::kXPU); // Explicit CUDA WITHOUT the platform throws the pinned message — never a // silent CPU fallback (mirror of vLLM assigning an explicit device verbatim, // vllm/config/device.py:61-66). CHECK_THROWS_WITH_AS( - LoadedEngine::ResolveExplicitDeviceType(Device::kCUDA, false), + LoadedEngine::ResolveExplicitDeviceType(Device::kNamedPlatform, + std::nullopt), doctest::Contains("device 'cuda' was requested but no CUDA platform"), std::runtime_error); // kAuto is not an explicit selection: it resolves through the probe inside // SelectQueue, and this seam refuses it rather than guessing. - CHECK_THROWS_AS(LoadedEngine::ResolveExplicitDeviceType(Device::kAuto, true), + CHECK_THROWS_AS( + LoadedEngine::ResolveExplicitDeviceType(Device::kAuto, std::nullopt), std::invalid_argument); } @@ -493,7 +499,7 @@ TEST_CASE("loaded_engine: DeviceFromString mirrors the vLLM Device names") { // — the strings the server's --device flag consumes. CHECK(vllm::DeviceFromString("auto") == Device::kAuto); CHECK(vllm::DeviceFromString("cpu") == Device::kCPU); - CHECK(vllm::DeviceFromString("cuda") == Device::kCUDA); + CHECK(vllm::DeviceFromString("cuda") == Device::kNamedPlatform); CHECK_THROWS_WITH_AS(vllm::DeviceFromString("tpu"), doctest::Contains("Unknown device: tpu"), std::invalid_argument); @@ -505,10 +511,10 @@ TEST_CASE("loaded_engine: DeviceFromString mirrors the vLLM Device names") { // the auto slot. CHECK(static_cast(Device::kAuto) == 0); CHECK(static_cast(Device::kCPU) == 1); - CHECK(static_cast(Device::kCUDA) == 2); + CHECK(static_cast(Device::kNamedPlatform) == 2); CHECK(std::string(vllm::DeviceName(Device::kAuto)) == "auto"); CHECK(std::string(vllm::DeviceName(Device::kCPU)) == "cpu"); - CHECK(std::string(vllm::DeviceName(Device::kCUDA)) == "cuda"); + CHECK(std::string(vllm::DeviceName(Device::kNamedPlatform)) == "cuda"); } TEST_CASE("loaded_engine: FromModelDir resolves an explicit absent device BEFORE any path I/O") { @@ -517,11 +523,11 @@ TEST_CASE("loaded_engine: FromModelDir resolves an explicit absent device BEFORE // arg_utils.py:1878, device.py __post_init__). This is also what makes the // EngineParams->FromModelDir plumb pinnable on the CPU tier with no loadable // checkpoint: a bogus path + device=cuda must report the DEVICE, not the path. - if (vllm::platforms::HasPlatform(vt::DeviceType::kCUDA)) { + if (vllm::platforms::FindPlatformByName("cuda") != nullptr) { return; // CUDA build/box: the explicit-cuda arm resolves; nothing to pin. } EngineParams params; - params.device = vllm::Device::kCUDA; + params.device = vllm::Device::kNamedPlatform; CHECK_THROWS_WITH_AS( LoadedEngine::FromModelDir("/nonexistent/vllm-cpp/model/dir", params), doctest::Contains("device 'cuda' was requested but no CUDA platform"), diff --git a/tests/vllm/platforms/test_platform.cpp b/tests/vllm/platforms/test_platform.cpp index 198017fd..d3f7c354 100644 --- a/tests/vllm/platforms/test_platform.cpp +++ b/tests/vllm/platforms/test_platform.cpp @@ -15,6 +15,7 @@ using vllm::platforms::CurrentPlatform; using vllm::platforms::DeviceCapability; +using vllm::platforms::FindPlatformByName; using vllm::platforms::GetPlatform; using vllm::platforms::HasPlatform; using vllm::platforms::Platform; @@ -102,6 +103,21 @@ TEST_CASE("CPU platform is self-registered and advertises CPU capabilities") { CHECK(cpu.get_attn_backend_priority() == cpu_priority); } +TEST_CASE("platform registry resolves canonical names without device-specific callers") { + REQUIRE(FindPlatformByName("cpu") != nullptr); + CHECK(FindPlatformByName("cpu")->device_type() == DeviceType::kCPU); + CHECK(FindPlatformByName("not-a-platform") == nullptr); + + size_t count = 0; + const DeviceType* priority = vllm::platforms::CurrentPlatformPriority(count); + for (size_t i = 0; i < count; ++i) { + if (!HasPlatform(priority[i])) continue; + CAPTURE(vt::DeviceTypeName(priority[i])); + CHECK(FindPlatformByName(vt::DeviceTypeName(priority[i])) == + &GetPlatform(priority[i])); + } +} + TEST_CASE("CurrentPlatform resolves accelerator-first, else falls back to CPU") { // CurrentPlatform() answers the PROCESS-level "what accelerator is this // process on" question (interface.h:104): accelerator-first, CPU fallback. It