diff --git a/.agents/NOW.md b/.agents/NOW.md index db3c38fdc..b88299745 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -29,7 +29,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`. | Vulkan 27B | decode **MET 4.36 vs 4.35** (barriers OFF). **LOADMEM: load held the model TWICE, VmRSS 100.759 -> 53.413 GiB** | Load-phase host build is the new peak | | `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed (contributor, authorship-preserved); W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) | | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) | -| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** | #141; artifacts pending | +| Release | **ACTIVE; required W1-W11/W13 implemented in #196** | Finish hosted ten-SM proof; rebase/push; run full eight-tuple dry run | | Surface coverage (`ARCH-ONE-SURFACE`) | ROW 8 + #139; **embeddings live (#137): model, runner, ABI v15, endpoint, fold 4/4-231** | Real-checkpoint oracle cosine | In-flight, default-OFF, not pushed: see the row's spec. diff --git a/.agents/completed/state-events/2026-08/STATE-20260809T150001-001.md b/.agents/completed/state-events/2026-08/STATE-20260809T150001-001.md new file mode 100644 index 000000000..dc7810a3f --- /dev/null +++ b/.agents/completed/state-events/2026-08/STATE-20260809T150001-001.md @@ -0,0 +1,30 @@ +# W6 server package claimed after empty release diagnosis + + +## Context + +The `v0.0.2-alpha1-ci-test` release proved the expected W5 boundary: GitHub +created the release with zero assets and launched zero release-event workflows. +PR #141 explicitly excluded W1-W4 and W6-W13; it merged only the manifest +schema and tooling at `bd20da30`. + +## Outcome + +`CLAIM-ENG-RELEASE-BINARIES-W6` owns the first missing executable boundary: +canonical `vllm-server`, a server-only install component, deterministic +staging/archive targets, and an empty-prefix to archive to empty-extraction to +`bin/vllm-server --help` gate. The existing library/header install surface must +remain unchanged. W1-W4, W7-W13, publication, GPU execution, runtime-model +correctness/performance, downloads, services, and channel advancement remain +pending at this checkpoint. + +## Evidence + +The claim is PR #196 on branch `row/ENG-RELEASE-BINARIES`, rebased onto +`origin/main` `26a128c2`. The accepted dependency order and release contract are +in `.agents/specs/release-binary-matrix.md`. + +## Next action + +Implement the focused W6 test and prove it fails because the current target is +named `server` and no server install/archive rule exists. diff --git a/.agents/completed/state-events/2026-08/STATE-20260809T160000-001.md b/.agents/completed/state-events/2026-08/STATE-20260809T160000-001.md new file mode 100644 index 000000000..a04cf4f17 --- /dev/null +++ b/.agents/completed/state-events/2026-08/STATE-20260809T160000-001.md @@ -0,0 +1,33 @@ +# W6 installed server package green + + +## Context + +PR #196 implements W6 on rebased claim base `26a128c2`. The server remains a +thin public C-ABI client while linking the static core, installs as +`bin/vllm-server` through component `vllm-server`, and exposes deterministic +`vllm-server-stage` and `vllm-server-archive` targets. Packaging installs into +a new empty prefix and normalizes archive ordering, owner, modes, and timestamps. + +## Outcome + +The row remains `ACTIVE`. W1-W4 and W7-W13 remain pending. This development +archive has no W7 allowlist/dependency/RPATH audit, VERSION, manifest, SBOM, +licenses, provenance, or final checksum and has not been published. No model +runtime, correctness, performance, or channel claim advances. + +## Evidence + +RED: `python3 tests/scripts/test_server_package.py` failed because target +`vllm-server-archive` did not exist. GREEN: the clean CPU test completed 1/1 in +105.000 seconds; a second run against the prebuilt tree completed in 3.285 +seconds. It reproduced the archive SHA256, extracted into a second empty tree, +ran canonical `bin/vllm-server --help` with loader-path overrides removed, +proved Linux `ldd` has no `libvllm`, and confirmed the existing `vllm.h`, static, +and shared library installs remain present. GitHub Actions repeated the package +gate successfully at the pre-rebase immutable head. + +## Next action + +Continue PR #196 with W1-W4 and W7-W13, beginning with the cross-family CUDA +fat-build red gate from the accepted dependency graph. diff --git a/.agents/coordination.md b/.agents/coordination.md index 8e80d874d..d7181a5be 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -134,19 +134,22 @@ ships DEFAULT-OFF pending an operator re-measurement. dgx build/gate roots `dgx.casa:~/vkbase` (main control) and `~/vkbar` (this branch), both `-DVLLM_CPP_VULKAN=ON`, every GPU stage under `flock $HOME/gpu.lock`. -**Server binary release manifest W5 (`ENG-RELEASE-BINARIES`, 2026-08-08, -`CLAIM-ENG-RELEASE-BINARIES-W5`).** Codex (GPT-5), isolated worktree +**Server binary release W1-W13 (`ENG-RELEASE-BINARIES`, 2026-08-09, +`CLAIM-ENG-RELEASE-BINARIES-W1-W13`).** Codex (GPT-5), isolated worktree `/home/mudler/_git/vllm.cpp-release-binaries`, branch -`row/ENG-RELEASE-BINARIES`, base `origin/main` `b38f78a7`. PR #129 merged the -accepted release matrix; the current claim is W5 only: a versioned manifest -schema, deterministic generator/validator, canonical fixtures, mutation tests, -and the narrow release-contract/record surfaces those tests require. No W1-W4 -CUDA/CPU implementation, W6 archive/install target, release workflow, -published artifact, download, GPU, model, kernel, or service change is in -scope. Draft PR #141 is the claim. W5 is implemented and green at 19/19 after -the fresh review's ten production-removal mutations were killed; the row is -`ACTIVE`, while every archive, staged smoke, runtime, correctness, performance, -install/package and publication claim remains pending. +`row/ENG-RELEASE-BINARIES`, base `origin/main` `81291a89`. W5 merged in PR #141 +at `bd20da30`; developer direction binds every remaining W1-W13 checkpoint to +the single draft PR #196. W6's clean CPU gate is GREEN: the canonical +static-core binary installs through component `vllm-server`; two package runs +produce identical archive bytes; an empty extraction runs `--help` without a +dynamic `libvllm`; and the existing library/header install remains present. +Required W1-W11/W13 implementation is now complete in the same draft PR: +ten-SM gencode/AOT, adaptive CPU tiers, extracted-archive and supply-chain gates, +least-privilege immutable handoff, eight primary bundles, generated indexes, +attestation, and exact-file publication. Local CPU/Vulkan/mutation gates are +green. The row remains `ACTIVE`; hosted ten-SM completion, the full eight-tuple +dry run, matching-hardware evidence, merge, and tagged publication remain +pending. W12 stays optional/non-primary. **CPU grouped keep-quant GEMM activation-dtype P0 (`QUANT-GGUF-CIQ-GEMM`, 2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`).** Claude Code @@ -1425,6 +1428,7 @@ table, tests, CMake. Details in the state-log entry of the same date. | Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | |---|---|---|---|---|---|---|---| +| `CLAIM-ENG-RELEASE-BINARIES-W1-W13` | `ENG-RELEASE-BINARIES` (`ACTIVE`; complete release matrix) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; inline execution in the existing isolated worktree | `row/ENG-RELEASE-BINARIES`, base `origin/main` `81291a89`, draft PR #196 | Complete W1-W13 contract in one PR: fat-CUDA gencode and AOT, adaptive CPU tiers, staged validation and supply chain, least-privilege release automation, all declared bundles, generated index, and tagged-release audit. W5 is inherited from main; ROCm remains blocked by contract | `ACTIVE` | 2026-08-09 — required W1-W11/W13 implementation complete; local CPU/Vulkan/mutation gates green; hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, rebase/merge, and tagged publication pending; W12 optional/non-primary | | `CLAIM-ENG-RELEASE-BINARIES-W5` | `ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; CPU/build-time metadata tooling only, no GPU/download/service work | `row/ENG-RELEASE-BINARIES`, base `b38f78a7`, claim commit `29107d0b`, draft PR #141 | W5 implemented: versioned manifest schema + deterministic generator/validator, canonical synthetic CPU/CUDA fixtures, fail-closed mutation tests, release checker/registration and required record/doc checkpoints. Excludes W1-W4, W6-W13, archives, install/package/publish workflows and runtime artifacts | `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; no archive or real runtime/correctness/performance evidence | | `CLAIM-KIMI-LINEAR-W0` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — dedicated W0 spike) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a771cc029e6843be0`; CPU-only, records-only — NO build, NO GPU, NO download (two GPU jobs queued ahead; the W0 GPU golden capture is a SEPARATE later step) | branch `worktree-agent-a771cc029e6843be0`, base `origin/main` HEAD `10dd23ee` | The FULL dedicated W0 spike for `KimiLinearForCausalLM` (Kimi-Linear-48B-A3B) per the spike-first protocol, so W1 implementation can start immediately. Owns ONLY: NEW `.agents/specs/kimi-linear.md`, the `MODEL-TEXT-kimi-linear-*` matrix row (Spike link → the dedicated spec + note; STAYS `SPIKE` because it is actively claimed — protocol forbids a claimed row from being `READY`; checklist mark stays `📋`), this claim row + the narrative block below, the roadmap breadth note, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, the `.agents/NOW.md` live-claim row + stamp, and one `.agents/state.md` entry. **NON-COLLISION:** records-only — touches NO model/kernel/registry/loader source, NO CMake, NO test. Co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` (MLA half) + `CLAIM-KDA-KERNEL` (KDA host refs), exactly as the K3 row is co-owned by `CLAIM-KIMI-K3-SCOPE`/`CLAIM-KIMI-K3-W2-W5`. | `DONE` | 2026-08-05 — **W0 DEDICATED SPIKE LANDED (records-only, NOT pushed).** Authoritative `config.json` fetched (27 layers = 20 KDA + 7 NoPE-MLA; 256e/top-8/1-shared sigmoid `noaux_tc` `routed_scaling=2.446` `first_k_dense_replace=1`; `num_nextn_predict_layers=0` ⇒ no MTP); reuse-vs-new map with our `file:line`; HW-fit (FITS one GB10, 91.5 GiB / 0.77× pool — real e2e SACRED gate reachable); W0-W7 breakdown + GPU golden-capture recipe. NET-NEW = KDA device kernel (host refs landed, `CLAIM-KDA-KERNEL`), NoPE-MLA branch, hybrid schedule/het-KV, loader name-map. Record checkers green (`check-model-checklist`, `check-agent-record`, `check-doc-checkpoint`, `check-now-current`). | | `CLAIM-KIMI-LINEAR-W1` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W1 registry/config/loader scaffolding; forward REFUSES-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a11ae231a71e8694b`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release); NO GPU, NO download — additive TU + one REGISTER line, forward is a `VT_CHECK(false)` stub | branch `worktree-agent-a11ae231a71e8694b`, base `origin/main` HEAD `053116df` | Kimi-Linear-48B-A3B W1 — registry + config + loader + KV-spec scaffolding so the W3-W6 forward can start. Owns ONLY: NEW `include/vllm/model_executor/models/kimi_linear.h`, NEW `src/vllm/model_executor/models/{kimi_linear_registry,kimi_linear_weights,kimi_linear}.cpp`, NEW `tests/vllm/models/test_kimi_linear_scaffold.cpp`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `KimiLinearForCausalLM` sorted-set + error-message + model-property inserts in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the forward refuses, rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` refuse stub so NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W1 CPU SCAFFOLDING LANDED (foreground, NOT pushed).** Additive registry (`kimi_linear`/`KimiLinearForCausalLM`, `is_hybrid`, `supports_multimodal=false`) + `ParseKimiLinearParams` (20 KDA + 7 NoPE-MLA schedule, MLA 512/128/64/128 q_lora-null, 256e/top-8/1-shared sigmoid `noaux_tc` 2.446, `first_k_dense_replace=1`; asserts `mla_use_nope`/`q_lora==null`) + `EnumerateKimiLinearTensors` name-map VERIFIED vs the real HF safetensors index (MoE = `block_sparse_moe.*`, the correction of the DERIVED K3 `mlp.*`) + loader THROWS BY NAME on missing/mis-shaped tensors + het KV spec (MLA latent-576 + KDA mamba `12288×3`/`32×128×128`) + REFUSE-by-name forward. `test_kimi_linear_scaffold` **9/9·83** + `test_model_registry` **24/24** + `test_kimi_k3_scaffold` 6/6 unchanged; clean CPU Release build; `check-fusion-consistency`/`check-runner-routing-consistency` green (refuse stub skipped); record checkers rc=0. RESIDUAL (W3-W6): KDA device kernel + NoPE-MLA route + sigmoid-noaux MoE + het-KV born-on-runner forward + the e2e SACRED gate (spec §8 recipe). | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 032e54388..b7f27b099 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -207,7 +207,7 @@ claims it. | `SERVE-CLI-BENCH` | Serve and latency/throughput/serve benchmark modes | T0 | `vllm/entrypoints/cli/serve.py:44`; `vllm/entrypoints/cli/benchmark/main.py:29`; production queue `vllm/v1/engine/core.py:200-231,622-669` | separate binaries + explicit scheduler-capacity flags `examples/server/main.cpp:63,96,116,170`; production `AsyncLLM` benchmark frontend + auditable scheduler depth `examples/bench/bench_core.h:426,495,595`; `examples/bench/main.cpp:51` | server help contract `examples/CMakeLists.txt:34`; production-frontend and metric assertions `tests/examples/test_bench.cpp:18,29-32,61,81,97` | [CLI/serve/benchmark spike](specs/cli-serve-bench.md) | `PARTIAL` | - | | `SERVE-GATE-ONLINE` | Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | `vllm/benchmarks/serve.py:1,581-615`; [v0.25 audit](sync/2026-07-12-702f481.md); `tests/benchmarks/test_serve_cli.py:1` | Schema-v5 harness plus [trace controller](../include/vt/cuda/cuda_profiler_control.h#L13), [production component driver](../scripts/dgx-gdn-packed-component.sh), and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py) | **BINDING `9ecd9d0`: 114/124** (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; `benchmark_binding` refers here, superseding `3f256ab` 55/124 and `246a23c` 49/124, both retained immutable). Two-grid totality with `f0fb727` (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 `p99_itl` ~0.86 residual, ROOT-CAUSED (2026-07-18, `CLAIM-C8-P99-TAIL-1`, [spec](specs/c8-p99-itl-tail-2026-07-18.md)) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, `tests/vllm/v1/test_scheduler_wave.cpp:265`, [tail spec](specs/tail-stall-analysis-2026-07-16.md)). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit | [online serving gate](specs/cuda-online-serving-gate.md); [merged GDN projections](specs/gdn-merged-input-projections.md); [packed decode](specs/gdn-packed-decode.md) | `ANCHOR-BACKFILL` | CLAIM-SERVE-GATE-1 | | `SERVE-E2E-NIGHTLY` | Server conformance and real-model nightly suites for all release gates | T0 | `tests/entrypoints/openai/`; `tests/v1/e2e/`; `.buildkite/test-pipeline.yaml` | current unit/conformance tests only; no scheduled DGX suite | `tests/vllm/entrypoints/openai/test_conformance.cpp:1`; `tests/parity/test_qwen36_paged_engine.cpp:78`; `tests/parity/test_qwen27_paged_engine.cpp:110` | `planned: specs/server-e2e-nightly.md` | `INVENTORIED` | - | -| `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | W5 schema `release/manifest-v1.schema.json:1`, deterministic build-time generator/validator `scripts/release_manifest.py:21`, and CPU/CUDA contract fixtures implemented; CPU tier kernel-family/bit/OS-probe inventories are exact and schema const/enum comparisons are JSON-type-strict; server target remains `examples/CMakeLists.txt:54-64`; cross-family CUDA fat/per-source-gencode, multi-SM AOT, CPU forced-tier, install/archive/publish implementation and all real release evidence remain pending | W5 contract/mutation suite `tests/scripts/test_release_manifest.py:260` 19/19 plus ten fresh-review production removals killed independently; existing accepted release-contract suite 30/30; fixtures are synthetic and no archive, staged smoke, runtime, correctness or performance evidence exists | [release-binary-matrix.md](specs/release-binary-matrix.md) | `ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` | +| `ENG-RELEASE-BINARIES` | Downloadable host-ABI-specific `vllm-server` bundles: adaptive CPU and fat CUDA primary artifacts, optional per-SM diagnostics, and literal-static feasibility boundary | T0 | vLLM release lanes `.buildkite/release-pipeline.yaml:1-18,34-170` @ `555967922`; release-image dependency boundary `docker/Dockerfile.cpu:262-290` | Required W1-W11/W13 implementation is complete in draft PR #196: ten-SM gencode/AOT, adaptive CPU tiers, extracted-archive validation and supply chain, least-privilege immutable handoff, eight primary bundles, byte-derived indexes, attestation, and exact-file publication; W12 remains optional/non-primary | Local manifest/archive/workflow/mutation gates, full adaptive x86 tier execution, clean CPU archive, and Vulkan 35/35 backend + 11/11 cross-device archive run are green; hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, and tagged publication remain pending, so no published-binary claim exists | [release-binary-matrix.md](specs/release-binary-matrix.md) | `ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | no Dockerfile for the server exists; the only container asset is the arm64 CPU bench cross-check `docker/Dockerfile.arm64`, which builds `vllm-bench` and is not a release artifact; staging tree, install rule and publish workflow are owed by [`ENG-RELEASE-BINARIES`](engine-matrix.md) and unimplemented | issue `#170`; no image or registry evidence | `planned: specs/container-images.md` | `INVENTORIED` | - | | `SERVE-CLI-CHAT` | Interactive `chat` and `complete` commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation | T1 | registration `vllm/entrypoints/cli/main.py:17-37,73-98`; client/model resolution + stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312` at `5559679229` | current in-process completion only `examples/cli/main.cpp:1-207`; remote command implementation absent | C-ABI stream baseline `tests/capi/test_capi.cpp:567-711`; chat-template baseline `tests/capi/test_chat_prompt.cpp:37-89`; command/fake-server tests absent | [cli-chat-complete.md](specs/cli-chat-complete.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-CLI-CHAT-SPIKE` | | `SERVE-POOLING-ENDPOINTS` | Embeddings, pooling, score, rerank, classify HTTP surface (`/v1/embeddings`, `/pooling`, `/score`, `/rerank`, `/classify`). **SPIKED 2026-07-28 (`CLAIM-POOLING`):** the whole pooling task class is scoped in [pooling-task-class.md](specs/pooling-task-class.md). **`/v1/embeddings` LIVE 2026-08-08 (ARCH-ONE-SURFACE ROW 6, `CLAIM-EMBEDDINGS-ONE-SURFACE`):** task-conditional registration (embed/api_router.py:22-28 mirror; the route exists ONLY on a pooling-model server, and the generate routes do not — both directions socket-404-pinned), OpenAI request/response shape (string-or-array input; `dimensions`/base64/token-arrays are named-residual 400s), handler drives the ONE engine path (`LoadedEngine -> LLMEngine::embed -> registry forward -> PoolingRunner`) — the same path `vllm_embed` (ABI v15) drives. RESIDUALS: `/pooling`, `/score`, `/rerank`, `/classify` (need a classify arch) | T2 | `vllm/entrypoints/pooling/embed/api_router.py:28`; `vllm/entrypoints/pooling/embed/protocol.py:34,173-185`; `vllm/entrypoints/pooling/scoring/api_router.py:37,71`; `vllm/entrypoints/pooling/classify/api_router.py:26` | `src/vllm/entrypoints/openai/api_server.cpp` `handle_embeddings` + the `if (embedder_)` route gate; `examples/server/main.cpp` pooling task dispatch | `tests/vllm/entrypoints/openai/test_api_server.cpp` embeddings section (dispatch shape + socket smoke + BOTH-direction 404 pins) | [embeddings-one-surface.md](specs/embeddings-one-surface.md) | `ACTIVE` | `CLAIM-EMBEDDINGS-ONE-SURFACE` | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 5a9e19a3f..d3c20c919 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -112,7 +112,7 @@ models we already ship + benchmark. Full seam map + M0–M5 W-plan: | 14 | `ROAD-V1-D5` | LoRA, local KV/weight offload, expert streaming, wider model zoo | [engine matrix](engine-matrix.md), [model matrix](model-matrix.md) | corrected expert-streaming spike accepted (`ENG-EXPERT-STREAM` READY): bank-only safetensors→Marlin bank, fixed contiguous cache slots matching Marlin dense strides, logical→slot remap after explicit router D2H, chunked C` working everywhere `--dit ` did; gated CPU-only (72/72, 54497 post-rebase) on index/name mapping and on the REAL 535-tensor geometry read from a sparse 13-shard release. The DEVICE streamer landed 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`, spec §8.14): one tensor at a time, zero host buffer for the bulk, bit-exact vs the non-streamed reference (73/73, 55203). **ENCODER + THE NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`, spec §8.15)**: the 14-shard bf16 text encoder streams too and `--encoder-only` runs it alone; measured over 233 tokens, Q4_K_M-vs-bf16 conditioning is cos 0.99745 mean / 6.85% rel RMS excl. sink / 3.5 deg median rotation — as much as a one-word prompt edit, but DIFFUSE. Whether the RENDER changes is NOT established (75/75, 55609). This unblocks the bf16-vs-quantized quality A/B; no render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the NVFP4 DiT projections now keep FP4 PACKED and route through the shared `dense_nvfp4::MatmulNvfp4W4A16D` (Marlin W4A16 — vLLM's own forced-a16 selection; SAME kernel as Laguna/dense-Qwen3 NVFP4; no new quant code); fp4-vs-bf16 WIRING gate GREEN (62/62·30039, W4A16 dispatcher runs all 11 quantized GEMMs). **GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** Marlin W4A16 RAN on sm_121a (`dense_gemms==11` default / `marlin_gemms==11` VT_MARLIN_DENSE=0, `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT; fp4 is a MEMORY win (~16 vs ~66 GB), ~0.79–0.83× the bf16 arm per diffusion forward (compute-bound large M; 3.47× faster at small decode-like M). Real-checkpoint fp4-resident t2va e2e RUNS end-to-end (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-768×1344-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render-coherence bug (device VAE decode / denoise), separate from the fp4 speed work. vLLM-Omni serves NO quantized H3 (BF16-only) -> HW/loader-forced-indirect (4×B300 209f render 86.964 s vs 1×GB10 209 s/forward). **2026-08-08 ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** the public 0/1 selector is mapped once to generic `DeviceType`; DSR returns 34→32 with the baseline/allowlist unchanged; CPU compile/fold test pending in CI due shared-disk pressure. | | 15 | `ROAD-V1-D6` | **llama.cpp device breadth folded into scope (user-directed 2026-08-05):** the 11 ggml backends vLLM has no platform for — cann, musa, opencl, openvino, rpc, webgpu, zdnn, zendnn, hexagon, blas, virtgpu — inventoried as `BACKEND-GGML-*`. **SPIKES FIRST:** no implementation before each row's `.agents/specs/.md` clears the spike contract, per the standing directive. vLLM stays the mirror source; llama.cpp is the breadth reference. | [backend matrix](backend-matrix.md) | ☐ per-row spike required | `INVENTORIED` | first spike accepted | -| REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md) | `ACTIVE` | W5 versioned manifest schema/generator and fail-closed CPU/CUDA/backend/channel fixtures implemented in draft PR #141 (19/19; ten fresh-review production removals killed); W1-W4 and W6-W13 remain pending, including unavailable x86 CUDA-13 ten-SM fat-build hardware/toolchain, install/archive/publish work and every real archive/runtime/correctness/performance claim; no archive exists | +| REL | `ROAD-V1-RELEASE` | KISS downloads per OS+host ABI: one adaptive CPU binary and one fat CUDA binary covering every supported SM; per-SM CUDA artifacts are optional diagnostics; stable channels require matching runtime evidence and build-only paths stay preview | [`ENG-RELEASE-BINARIES`](engine-matrix.md) | [release binary matrix](specs/release-binary-matrix.md) | `ACTIVE` | Required W1-W11/W13 implementation is complete in draft PR #196: eight primary bundles, extracted-archive gates, manifests/SBOM/provenance, immutable verification and attestation, generated indexes, and exact-file publication. Local CPU/Vulkan and mutation gates are green. Hosted ten-SM completion, the full eight-tuple dry run, matching-hardware evidence, merge, and tagged publication remain pending; no published binary exists. W12 remains optional/non-primary | | IMG | `ROAD-V1-CONTAINERS` | **Published container images on GHCR, built by GitHub Actions (user-directed 2026-08-08).** The same staged bundle `ROAD-V1-RELEASE` defines, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane — and `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image, every supported SM), `vulkan`, `cpu`, plus `rocm` blocked-preview; version tags immutable, every `latest-` moves. Every lane is a `linux/amd64`+`linux/arm64` multi-arch manifest on native runners, because the project's own gate hardware (GB10, Thor, Orin) is arm64. Metal and MLX are NOT-CONTAINERIZABLE and stay static-binary-only — a recorded boundary, not pending work. Depends on the `ROAD-V1-RELEASE` install/stage tree: the image IS the bundle, so the two lanes must not grow separate layouts. No image, workflow or registry package exists. | [`ENG-RELEASE-CONTAINERS`](engine-matrix.md) | none yet: `planned: specs/container-images.md`; the row cannot leave `INVENTORIED` without it | `INVENTORIED` | Write the container-image spike spec (lane matrix, base images, manifest/provenance, retention, least-privilege publish flow) against the accepted release-binary contract; no image work starts before it | An area row cannot enter `READY` without a real spike under `specs/`, and cannot diff --git a/.agents/specs/release-binary-matrix.md b/.agents/specs/release-binary-matrix.md index 088d0836d..db6614a0e 100644 --- a/.agents/specs/release-binary-matrix.md +++ b/.agents/specs/release-binary-matrix.md @@ -1,20 +1,35 @@ # Downloadable server binary release matrix -Status: accepted contract with W5 manifest tooling implemented for -`ENG-RELEASE-BINARIES`. The release row is `ACTIVE`; no archive, release -workflow, or runtime result is claimed. +Status: accepted contract with required W1-W11/W13 implementation complete for +`ENG-RELEASE-BINARIES` in the single delivery PR #196. The release row remains +`ACTIVE`: local gates are green, while the hosted eight-tuple dry run, +matching-hardware evidence, and tagged publication are still pending. W12 is +optional/non-primary. Pins: vLLM parity source `555967922`; vllm.cpp spike baseline `f13c49ee`; request [#117](https://github.com/mudler/vllm.cpp/issues/117); claim `CLAIM-ENG-RELEASE-BINARIES-SPIKE` in draft PR [#129](https://github.com/mudler/vllm.cpp/pull/129); W5 implementation claim `CLAIM-ENG-RELEASE-BINARIES-W5` in draft PR -[#141](https://github.com/mudler/vllm.cpp/pull/141). +[#141](https://github.com/mudler/vllm.cpp/pull/141); W6 implementation claim +`CLAIM-ENG-RELEASE-BINARIES-W6` in draft PR +[#196](https://github.com/mudler/vllm.cpp/pull/196). + +## Delivery topology + +Developer direction on 2026-08-09 binds all remaining W1-W13 work to PR #196. +Each work unit remains an independently committed, red-first, executable +checkpoint, but no work unit is split into another implementation PR. PR #196 +stays draft until every required work unit, the full release dry run, fresh +review, and operator verification are green. W12 remains optional and cannot +block W13; its optional status does not permit moving it to another PR. identity=ENG-RELEASE-BINARIES lifecycle=ACTIVE manifest_schema=vllm.cpp.release-manifest.v1 +delivery_pull_request=196 +delivery_mode=single-pr-W1-W13 primary_cuda_artifact=one-fat-binary-per-os-host-abi primary_cuda_sms=80,86,87,89,90a,100a,103a,110,120a,121a per_sm_cuda=optional-non-primary @@ -30,7 +45,7 @@ musl_channel=experimental-preview musl_scope=cpu-only-no-gpu rocm_channel=blocked gpu_driver_boundary=external-host-never-bundled -required_anchor_paths=.agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md,.agents/coordination.md,.agents/completed/state-events/2026-08/STATE-20260808T220000-002.md,docs/STATUS.md,docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json,scripts/release_manifest.py,tests/scripts/test_release_manifest.py +required_anchor_paths=.agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md,.agents/coordination.md,.agents/completed/state-events/2026-08/STATE-20260809T160000-001.md,docs/STATUS.md,docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json,scripts/release_manifest.py,tests/scripts/test_release_manifest.py,examples/CMakeLists.txt,scripts/package-server.py,tests/scripts/test_server_package.py work_W1= work_W2=W1 work_W3= @@ -38,6 +53,7 @@ work_W4= work_W5= work_W5_status=implemented work_W6= +work_W6_status=implemented work_W7=W1,W2,W3,W4,W5,W6 work_W8=W5,W7 work_W9=W3,W4,W5,W6,W7 @@ -430,9 +446,10 @@ wildcards, and a failed lane cannot be replaced by an older workflow artifact. ## Work breakdown: helper-sized implementation plan -Each work unit is a separate claim with its own red-first checker change and -fresh review. W5 is implemented in draft PR #141 and advances the row to -`ACTIVE`; no later work unit or release evidence is implied. +Each work unit is a separate verified checkpoint inside the single active +claim and PR #196, with its own red-first checker change and review. W5 was +implemented in PR #141; required W1-W11/W13 are implemented in #196. Hosted +completion remains necessary before any publication or channel claim. | Work | Deps | Deliverable | Exit gate | |---|---|---|---| @@ -441,7 +458,7 @@ fresh review. W5 is implemented in draft PR #141 and advances the row to | W3 | — | x86_64 CPU ISA-dispatch inventory and completion | SSE2/portable baseline runs without AVX2; current F16C/AVX2/AVX-512 tiers forced and executed; exact OS-state probes; VNNI/AMX listed only for real gated kernels; no `-march=native` | | W4 | — | aarch64 CPU ISA-dispatch inventory and completion | NEON/portable baseline plus independently forced DotProd/i8mm where kernels exist; exact Linux HWCAP/Darwin sysctl gates; poor/rich host or emulation execution | | W5 | — | **IMPLEMENTED (#141):** versioned release-manifest generator and schema with independent per-SM and per-CPU-tier evidence | 19/19 tests: fixtures distinguish absent, false, failed and true; schema type/enum/const are independently live; booleans cannot satisfy integer types/constants; CPU tier kernel-family/bit/OS-probe inventories are exact; compiled tiers/SMs, fallback/AOT state, the named NVIDIA driver dependency, publication channels, static boundaries and host ABI are mandatory | -| W6 | — | canonical `vllm-server` output name, install component, and staging/package target for the existing static-core server | install into empty prefix; extracted help smoke; existing library install unchanged | +| W6 | — | **IMPLEMENTED (#196):** canonical `vllm-server` output name, install component, and deterministic staging/package target for the existing static-core server | clean CPU build installs into an empty prefix; archive bytes reproduce; extracted `--help` runs without dynamic `libvllm`; existing library/header install remains present | | W7 | W1, W2, W3, W4, W5, W6 | staged archive validator: allowlist, dependency/RPATH, fat-SM/AOT and adaptive-CPU audits, SHA256, VERSION, licenses and SPDX SBOM | Linux fixture/archive tests red-first; no build paths, missing SM, unsafe ISA tier or undeclared dependency accepted | | W8 | W5, W7 | least-privilege dry-run/tag workflow, immutable artifact handoff, provenance and protected publish stages | permissions checker plus dry run proves no release is created and publish cannot consume unverified bytes | | W9 | W3, W4, W5, W6, W7 | primary adaptive CPU bundles: Linux glibc x86_64+aarch64 and experimental x86_64 musl literal-static | glibc binaries execute baseline and supported rich tiers on matching hosts/emulation before stable; musl remains preview | @@ -482,7 +499,8 @@ The release program is feasible as backend-specific static-core bundles with a hybrid stable/preview channel. Literal-static scope is limited to the experimental musl CPU lane. ROCm is blocked. Primary downloads are adaptive CPU or fat CUDA per OS+host ABI; per-SM CUDA archives are optional diagnostics. The -W5 manifest schema/tooling is implemented and the row is `ACTIVE`, not `DONE`. -W1-W4 and W6-W13 remain pending; in particular no archive, install/package or -publish workflow, staged-archive evidence, runtime/correctness proof, or -performance result exists. +Required W1-W11/W13 are implemented and the row is `ACTIVE`, not `DONE`. Local +archive, CPU, Vulkan, workflow, and mutation gates are green. The hosted +eight-tuple dry run, matching-hardware evidence, and tagged publication remain +pending, so no published archive or release-channel advancement is claimed. +W12 remains the optional non-primary diagnostic lane. diff --git a/.agents/waivers.csv b/.agents/waivers.csv index 7ede002de..23531089e 100644 --- a/.agents/waivers.csv +++ b/.agents/waivers.csv @@ -1,3 +1,4 @@ waiver_id,checker,scope,owner,reason,evidence,expires WAIVER-PR-SIZE-001,scripts/check-pr-size.py,pr:128,maintainer,One-time policy consolidation and evidence-preserving archive migration exceeds the procedure and governance-test class budgets,PR-128 policy migration,2026-08-15 WAIVER-PR-SIZE-002,scripts/check-pr-size.py,pr:166,maintainer,One-time lossless structured-state migration exceeds review budgets through mechanical evidence fan-out,PR-166 structured state design,2026-08-15 +WAIVER-PR-SIZE-003,scripts/check-pr-size.py,pr:196,maintainer,One-time atomic W1-W13 release matrix intentionally ships through one delivery PR,release-binary-matrix.md delivery topology and developer direction 2026-08-09,2026-09-30 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9d50a4a99..3bc5da4bd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -51,6 +51,20 @@ jobs: - name: Binary release manifests are deterministic and fail closed run: | python3 tests/scripts/test_release_manifest.py + - name: Extracted release archives fail closed on content and metadata drift + run: | + python3 tests/scripts/test_release_archive.py + - name: Release workflow is dry-run safe and least privilege + run: | + python3 scripts/check-release-workflow.py + python3 tests/scripts/test_release_pipeline.py + python3 tests/scripts/test_release_index.py + - name: CPU release metadata and tier execution gates fail closed + run: | + python3 tests/scripts/test_release_metadata.py + python3 tests/scripts/test_release_accelerator_metadata.py + python3 tests/scripts/test_release_macos_metadata.py + python3 tests/scripts/test_cpu_release_gates.py - name: README stays a human-readable user-facing document run: | python3 scripts/check-readme-structure.py @@ -294,8 +308,54 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - name: CUDA feature table resolves per architecture - run: cmake -P cmake/CudaArchFeaturesTest.cmake + - name: CUDA feature table and per-source gencode resolve exactly + run: | + cmake -P cmake/CudaArchFeaturesTest.cmake + cmake -P cmake/CudaSourceGencodeTest.cmake + cmake -P cmake/TritonAOTMultiArchTest.cmake + python3 -m unittest tests.scripts.test_check_cuda_fat_gencode + python3 -m unittest tests.scripts.test_check_triton_aot_multiarch + cuda-fat-build: + # W1 release prerequisite: compile the primary heterogeneous CUDA archive, + # then inspect both compile_commands.json and the linked archive. No GPU is + # used; runtime/correctness evidence remains independent. + permissions: + contents: read + concurrency: + group: ci-cuda-fat-build-${{ github.ref }}-${{ github.repository }} + cancel-in-progress: true + runs-on: ubuntu-latest + timeout-minutes: 180 + container: nvidia/cuda:13.3.0-devel-ubuntu24.04 + steps: + - name: Install build tools + run: | + apt-get update + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + binutils ca-certificates cmake g++ git ninja-build python3 + rm -rf /var/lib/apt/lists/* + - uses: actions/checkout@v4 + - name: Configure ten-SM static library + run: | + cmake -S . -B build-cuda-fat -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + -DVLLM_CPP_CUDA=ON \ + -DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \ + -DVLLM_CPP_CUTLASS_FETCH=ON \ + -DVLLM_CPP_TRITON=ON \ + -DVLLM_CPP_BUILD_TESTS=OFF \ + -DVLLM_CPP_BUILD_EXAMPLES=OFF \ + -DVLLM_CPP_SERVER=OFF + - name: Build and audit exact per-source gencode + run: | + cmake --build build-cuda-fat --target vllm --parallel 2 + python3 scripts/check-cuda-fat-gencode.py \ + --compile-commands build-cuda-fat/compile_commands.json \ + --library build-cuda-fat/libvllm.a + python3 scripts/check-triton-aot-multiarch.py \ + --vendored-root src/vt/cuda/triton_aot_vendored \ + --library build-cuda-fat/libvllm.a vulkan-spirv-freshness: # The Vulkan backend commits its SPIR-V (src/vt/vulkan/vulkan_spirv.h) instead # of compiling shaders at build time, so the build needs no shader toolchain on @@ -407,13 +467,111 @@ jobs: steps: - uses: actions/checkout@v4 - name: Configure - run: cmake -S . -B build -DVLLM_CPP_BUILD_TESTS=ON + run: | + cmake -S . -B build \ + -DVLLM_CPP_BUILD_TESTS=ON \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON - name: Build # Bounded parallelism: a bare `-j` lets Make link ALL test executables at # once, which OOM-kills the runner (ld signal 9) during the parallel link. run: cmake --build build -j 2 + - name: Installed server archive is reproducible and runnable + env: + VLLM_CPP_PACKAGE_TEST_BUILD_DIR: ${{ github.workspace }}/build + run: python3 tests/scripts/test_server_package.py + - name: CPU ISA objects use only their declared feature flags + run: | + python3 scripts/check-cpu-isa-build.py \ + --compile-commands build/compile_commands.json - name: Test run: ctest --test-dir build --output-on-failure + - name: Force every x86 ISA tier available on this runner + run: | + tiers=(portable) + if [[ "$(uname -m)" == "x86_64" ]]; then + tiers+=(sse2) + if grep -qiw f16c /proc/cpuinfo && grep -qiw avx /proc/cpuinfo; then + tiers+=(sse2+f16c) + fi + if grep -qiw avx2 /proc/cpuinfo; then + tiers+=(avx2) + fi + if grep -qiw avx512f /proc/cpuinfo && \ + grep -qiw avx512bw /proc/cpuinfo && \ + grep -qiw avx512vl /proc/cpuinfo; then + tiers+=(avx512) + fi + fi + for tier in "${tiers[@]}"; do + echo "Running test_ops_matmul_elem with VT_CPU_MATMUL_TIER=${tier}" + VT_CPU_MATMUL_TIER="${tier}" build/tests/test_ops_matmul_elem + done + build-test-cpu-arm64: + # Independent Arm execution evidence: the x86 lane cannot prove HWCAP + # dispatch, Arm instructions, or the host ABI. The native runner exercises + # every available rich tier; qemu's Cortex-A53 model supplies the required + # feature-poor execution and illegal-tier refusal gate. + concurrency: + group: ci-build-test-cpu-arm64-${{ github.ref }}-${{ github.repository }} + cancel-in-progress: true + runs-on: ubuntu-24.04-arm + steps: + - uses: actions/checkout@v4 + - name: Configure focused Arm CPU gate + run: | + cmake -S . -B build-arm \ + -DVLLM_CPP_BUILD_TESTS=ON \ + -DVLLM_CPP_BUILD_EXAMPLES=OFF \ + -DVLLM_CPP_SERVER=OFF \ + -DVLLM_CPP_CUDA=OFF \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON + - name: Build Arm ISA and kernel gates + run: | + cmake --build build-arm \ + --target test_cpu_isa_arm test_ops_matmul_elem \ + test_ops_quant_dot test_ops_quant_repack \ + -j 2 + - name: Arm ISA objects use only their declared feature flags + run: | + python3 scripts/check-arm-isa-build.py \ + --compile-commands build-arm/compile_commands.json + - name: Force baseline and every rich tier available on the native runner + run: | + build-arm/tests/test_cpu_isa_arm + VT_CPU_MATMUL_TIER=portable build-arm/tests/test_ops_matmul_elem + VT_CPU_MATMUL_TIER=neon build-arm/tests/test_ops_matmul_elem + VT_CPU_Q8_DOT=portable \ + VT_CPU_QUANT_MMLA=portable \ + build-arm/tests/test_ops_quant_dot + VT_CPU_QUANT_REPACK=portable build-arm/tests/test_ops_quant_repack + if grep -qiw asimddp /proc/cpuinfo; then + VT_CPU_Q8_DOT=sdot build-arm/tests/test_ops_quant_dot + fi + if grep -qiw asimddp /proc/cpuinfo && grep -qiw i8mm /proc/cpuinfo; then + VT_CPU_QUANT_MMLA=i8mm build-arm/tests/test_ops_quant_dot + VT_CPU_QUANT_REPACK=i8mm build-arm/tests/test_ops_quant_repack + fi + - name: Feature-poor Cortex-A53 executes baseline and refuses rich tiers + run: | + sudo apt-get update + sudo apt-get install --yes qemu-user + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_cpu_isa_arm + VT_CPU_MATMUL_TIER=portable \ + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem + VT_CPU_MATMUL_TIER=neon \ + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_matmul_elem + VT_CPU_Q8_DOT=portable VT_CPU_QUANT_MMLA=portable \ + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot + if VT_CPU_Q8_DOT=sdot \ + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then + echo "Cortex-A53 unexpectedly accepted the DotProd tier" >&2 + exit 1 + fi + if VT_CPU_QUANT_MMLA=i8mm \ + qemu-aarch64 -cpu cortex-a53 build-arm/tests/test_ops_quant_dot; then + echo "Cortex-A53 unexpectedly accepted the i8mm tier" >&2 + exit 1 + fi sanitize-cpu: # The DYNAMIC detector lanes (VLLM_CPP_SANITIZE, see CMakeLists.txt). The # plain build-test-cpu job above proves the suite PASSES; it cannot see a diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 000000000..bae2f1440 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,481 @@ +name: release + +on: + workflow_dispatch: {} + push: + tags: ['v*'] + +permissions: + contents: read + +concurrency: + group: release-${{ github.ref }}-${{ github.repository }} + cancel-in-progress: false + +jobs: + plan: + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + publish: ${{ steps.plan.outputs.publish }} + release_tag: ${{ steps.plan.outputs.release_tag }} + version: ${{ steps.plan.outputs.version }} + steps: + - uses: actions/checkout@v4 + - name: Compute an explicit dry-run or exact-tag plan + id: plan + run: | + version=$(sed -n 's/^project(vllm_cpp VERSION \([^ ]*\).*/\1/p' CMakeLists.txt) + python3 scripts/release_pipeline.py plan \ + --event '${{ github.event_name }}' \ + --ref '${{ github.ref }}' \ + --sha '${{ github.sha }}' \ + --version "$version" \ + --matrix release/release-matrix.json \ + --output release-plan.json \ + --github-output "$GITHUB_OUTPUT" + - name: Upload immutable plan + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-plan-${{ github.sha }} + path: release-plan.json + if-no-files-found: error + overwrite: false + retention-days: 7 + cpu_x86: + needs: plan + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - name: Install feature-poor userspace emulator + run: | + sudo apt-get update + sudo apt-get install --yes qemu-user + - name: Build, execute, package, and validate adaptive x86 CPU bundle + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_DATE_EPOCH: ${{ github.event.repository.pushed_at }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + version=$(sed -n 's/^project(vllm_cpp VERSION \([^ ]*\).*/\1/p' CMakeLists.txt) + test "$version" = "$VERSION" + scripts/build-cpu-release.sh \ + linux-x86_64-glibc-cpu x86_64 stable build-release-cpu-x86 \ + "$(getconf GNU_LIBC_VERSION | awk '{print $2}')" \ + "$(command -v qemu-x86_64)" + - name: Upload exact validated x86 CPU asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-x86_64-glibc-cpu-${{ github.sha }} + path: | + build-release-cpu-x86/release/linux-x86_64-glibc-cpu.tar.gz + build-release-cpu-x86/release/linux-x86_64-glibc-cpu.tar.gz.sha256 + build-release-cpu-x86/release/linux-x86_64-glibc-cpu.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + cpu_arm64: + needs: plan + permissions: + contents: read + runs-on: ubuntu-24.04-arm + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - name: Install feature-poor userspace emulator + run: | + sudo apt-get update + sudo apt-get install --yes qemu-user + - name: Build, execute, package, and validate adaptive Arm CPU bundle + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + version=$(sed -n 's/^project(vllm_cpp VERSION \([^ ]*\).*/\1/p' CMakeLists.txt) + test "$version" = "$VERSION" + scripts/build-cpu-release.sh \ + linux-aarch64-glibc-cpu aarch64 stable build-release-cpu-arm64 \ + "$(getconf GNU_LIBC_VERSION | awk '{print $2}')" \ + "$(command -v qemu-aarch64)" + - name: Upload exact validated Arm CPU asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-aarch64-glibc-cpu-${{ github.sha }} + path: | + build-release-cpu-arm64/release/linux-aarch64-glibc-cpu.tar.gz + build-release-cpu-arm64/release/linux-aarch64-glibc-cpu.tar.gz.sha256 + build-release-cpu-arm64/release/linux-aarch64-glibc-cpu.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + cpu_musl: + needs: plan + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - name: Build and validate the literal-static musl experiment + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + version=$(sed -n 's/^project(vllm_cpp VERSION \([^ ]*\).*/\1/p' CMakeLists.txt) + test "$version" = "$VERSION" + docker run --rm \ + -e EVIDENCE_URL -e SOURCE_DATE_EPOCH -e SOURCE_SHA -e VERSION \ + -v "$GITHUB_WORKSPACE:/src" -w /src alpine:3.22 sh -euxc ' + apk add --no-cache binutils build-base cmake file ninja python3 qemu-x86_64 + scripts/build-cpu-release.sh \ + linux-x86_64-musl-cpu-static x86_64 experimental-preview \ + build-release-cpu-musl 1.2.5 /usr/bin/qemu-x86_64 + ' + sudo chown -R "$(id -u):$(id -g)" build-release-cpu-musl + - name: Upload exact validated musl CPU asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-x86_64-musl-cpu-static-${{ github.sha }} + path: | + build-release-cpu-musl/release/linux-x86_64-musl-cpu-static.tar.gz + build-release-cpu-musl/release/linux-x86_64-musl-cpu-static.tar.gz.sha256 + build-release-cpu-musl/release/linux-x86_64-musl-cpu-static.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + cuda_x86: + needs: plan + permissions: + contents: read + runs-on: ubuntu-latest + timeout-minutes: 360 + container: nvidia/cuda:13.3.0-devel-ubuntu24.04 + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - name: Install build and archive inspection tools + run: | + apt-get update + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + binutils ca-certificates cmake file g++ git ninja-build python3 + rm -rf /var/lib/apt/lists/* + - uses: actions/checkout@v4 + - name: Build and validate the x86 ten-SM CUDA bundle + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + scripts/build-linux-accelerator-release.sh \ + linux-x86_64-glibc-cuda-fat cuda build-release-cuda-x86 + - name: Upload exact validated x86 CUDA asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-x86_64-glibc-cuda-fat-${{ github.sha }} + path: | + build-release-cuda-x86/release/linux-x86_64-glibc-cuda-fat.tar.gz + build-release-cuda-x86/release/linux-x86_64-glibc-cuda-fat.tar.gz.sha256 + build-release-cuda-x86/release/linux-x86_64-glibc-cuda-fat.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + cuda_arm64: + needs: plan + permissions: + contents: read + runs-on: ubuntu-24.04-arm + timeout-minutes: 360 + container: nvidia/cuda:13.3.0-devel-ubuntu24.04 + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - name: Install build and archive inspection tools + run: | + apt-get update + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + binutils ca-certificates cmake file g++ git ninja-build python3 + rm -rf /var/lib/apt/lists/* + - uses: actions/checkout@v4 + - name: Build and validate the Arm ten-SM CUDA bundle + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + scripts/build-linux-accelerator-release.sh \ + linux-aarch64-glibc-cuda-fat cuda build-release-cuda-arm64 + - name: Upload exact validated Arm CUDA asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-aarch64-glibc-cuda-fat-${{ github.sha }} + path: | + build-release-cuda-arm64/release/linux-aarch64-glibc-cuda-fat.tar.gz + build-release-cuda-arm64/release/linux-aarch64-glibc-cuda-fat.tar.gz.sha256 + build-release-cuda-arm64/release/linux-aarch64-glibc-cuda-fat.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + vulkan_x86: + needs: plan + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - name: Install Vulkan software runtime and archive inspectors + run: | + sudo apt-get update + sudo apt-get install --yes mesa-vulkan-drivers + - name: Build, execute, package, and validate Vulkan preview + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + scripts/build-linux-accelerator-release.sh \ + linux-x86_64-glibc-vulkan vulkan build-release-vulkan-x86 + - name: Upload exact validated Vulkan asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-linux-x86_64-glibc-vulkan-${{ github.sha }} + path: | + build-release-vulkan-x86/release/linux-x86_64-glibc-vulkan.tar.gz + build-release-vulkan-x86/release/linux-x86_64-glibc-vulkan.tar.gz.sha256 + build-release-vulkan-x86/release/linux-x86_64-glibc-vulkan.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + metal_arm64: + needs: plan + permissions: + contents: read + runs-on: macos-15 + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - name: Build, execute, package, and validate native Metal bundle + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + scripts/build-macos-release.sh \ + macos-arm64-metal stable build-release-metal-arm64 + - name: Upload exact validated Metal asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-macos-arm64-metal-${{ github.sha }} + path: | + build-release-metal-arm64/release/macos-arm64-metal.tar.gz + build-release-metal-arm64/release/macos-arm64-metal.tar.gz.sha256 + build-release-metal-arm64/release/macos-arm64-metal.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + mlx_arm64: + needs: plan + permissions: + contents: read + runs-on: macos-15 + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.13' + - name: Install the exact redistributable MLX runtime + run: python3 -m pip install --disable-pip-version-check 'mlx==0.32.0' + - name: Build, execute, package, and validate MLX preview + env: + EVIDENCE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SOURCE_SHA: ${{ github.sha }} + VERSION: ${{ needs.plan.outputs.version }} + run: | + SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) + export SOURCE_DATE_EPOCH + mlx_root=$(python3 -c 'import mlx, pathlib; print(pathlib.Path(mlx.__file__).parent)') + mlx_license=$(python3 -c 'import importlib.metadata as m; d=m.distribution("mlx"); print(next(str(d.locate_file(f)) for f in d.files if f.name.lower() in {"license", "license.txt"}))') + test -f "$mlx_root/lib/libmlx.dylib" + test -f "$mlx_root/lib/mlx.metallib" -o -f "$mlx_root/mlx.metallib" + test -f "$mlx_license" + scripts/build-macos-release.sh \ + macos-arm64-metal-mlx preview build-release-mlx-arm64 \ + "$mlx_root" 0.32.0 "$mlx_license" + - name: Upload exact validated MLX asset triplet + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-macos-arm64-metal-mlx-${{ github.sha }} + path: | + build-release-mlx-arm64/release/macos-arm64-metal-mlx.tar.gz + build-release-mlx-arm64/release/macos-arm64-metal-mlx.tar.gz.sha256 + build-release-mlx-arm64/release/macos-arm64-metal-mlx.tar.gz.provenance.json + if-no-files-found: error + overwrite: false + retention-days: 7 + build: + needs: [plan, cpu_x86, cpu_arm64, cpu_musl, cuda_x86, cuda_arm64, vulkan_x86, metal_arm64, mlx_arm64] + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + artifact_digest: ${{ steps.upload.outputs.artifact-digest }} + steps: + - uses: actions/checkout@v4 + - name: Download the exact immutable plan by ID + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.plan.outputs.artifact_id }} + path: plan + - name: Download exact validated CPU artifacts by immutable IDs + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.cpu_x86.outputs.artifact_id }},${{ needs.cpu_arm64.outputs.artifact_id }},${{ needs.cpu_musl.outputs.artifact_id }},${{ needs.cuda_x86.outputs.artifact_id }},${{ needs.cuda_arm64.outputs.artifact_id }},${{ needs.vulkan_x86.outputs.artifact_id }},${{ needs.metal_arm64.outputs.artifact_id }},${{ needs.mlx_arm64.outputs.artifact_id }} + path: assets + merge-multiple: true + - name: Produce the byte-bound release handoff + run: | + python3 scripts/release_pipeline.py handoff \ + --plan plan/release-plan.json \ + --assets-dir assets \ + --output release-handoff.json + - name: Upload immutable unverified handoff + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-unverified-${{ github.sha }} + path: | + release-handoff.json + assets + if-no-files-found: error + overwrite: false + retention-days: 7 + verify: + needs: [plan, build] + permissions: + contents: read + runs-on: ubuntu-latest + outputs: + artifact_id: ${{ steps.upload.outputs.artifact-id }} + artifact_digest: ${{ steps.upload.outputs.artifact-digest }} + steps: + - uses: actions/checkout@v4 + - name: Download exact plan by ID + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.plan.outputs.artifact_id }} + path: plan + - name: Download exact unverified handoff by ID + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.build.outputs.artifact_id }} + path: unverified + - name: Verify handoff against plan and workflow SHA + run: | + mkdir -p verified + cp -a unverified/assets verified/assets + python3 scripts/release_pipeline.py verify \ + --plan plan/release-plan.json \ + --handoff unverified/release-handoff.json \ + --assets-dir verified/assets \ + --output verified/verified-handoff.json \ + --sha '${{ github.sha }}' + - name: Generate byte-derived release indexes + run: | + python3 scripts/release_index.py \ + --assets-dir verified/assets \ + --handoff verified/verified-handoff.json \ + --json-output verified/release-index.json \ + --markdown-output verified/RELEASE_INDEX.md \ + --retention-days 7 + - name: Upload immutable verified handoff + id: upload + uses: actions/upload-artifact@v4 + with: + name: release-verified-${{ github.sha }} + path: verified + if-no-files-found: error + overwrite: false + retention-days: 7 + attest: + needs: [plan, verify] + if: needs.plan.outputs.publish == 'true' && startsWith(github.ref, 'refs/tags/v') + permissions: + contents: read + id-token: write + attestations: write + artifact-metadata: write + runs-on: ubuntu-latest + steps: + - name: Download only the verified handoff by ID + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.verify.outputs.artifact_id }} + path: verified + - name: Attest verified bytes + uses: actions/attest@v4 + with: + subject-path: verified/assets/** + publish: + needs: [plan, verify, attest] + if: needs.plan.outputs.publish == 'true' && startsWith(github.ref, 'refs/tags/v') + environment: release + permissions: + contents: write + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Download only the verified handoff by ID + uses: actions/download-artifact@v4 + with: + artifact-ids: ${{ needs.verify.outputs.artifact_id }} + path: verified + - name: Publish the exact verified release assets + env: + GH_TOKEN: ${{ github.token }} + run: | + tag='${{ needs.plan.outputs.release_tag }}' + test "$tag" = "${GITHUB_REF#refs/tags/}" + python3 scripts/release_pipeline.py publish \ + --handoff verified/verified-handoff.json \ + --assets-dir verified/assets \ + --index-json verified/release-index.json \ + --index-markdown verified/RELEASE_INDEX.md \ + --tag "$tag" diff --git a/CMakeLists.txt b/CMakeLists.txt index 5fbbc096f..5d5b6dc47 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -28,6 +28,9 @@ option(VLLM_CPP_BUILD_EXAMPLES "Build examples" ON) # deviation, NOT a compute/ML dependency (consistent with the no-pytorch/no-ggml # rule). option(VLLM_CPP_SERVER "Build the OpenAI HTTP server (needs third_party/httplib/httplib.h)" ON) +option(VLLM_CPP_LITERAL_STATIC + "Link the CPU-only server as a literal static executable (experimental musl release lane only)" + OFF) if(VLLM_CPP_SERVER AND NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/third_party/httplib/httplib.h") message(WARNING "VLLM_CPP_SERVER=ON but third_party/httplib/httplib.h is missing; " "disabling the server target. Vendor cpp-httplib to enable it.") @@ -113,10 +116,10 @@ set(MLX_ROOT "" CACHE PATH "Root of an MLX install (include/ + lib/) for VLLM_CP # build-supported — they need a real fp16/non-tensor-core kernel # body, a scoped follow-up. See backend-matrix.md # BACKEND-CUDA-SM0{80,86,87,89}/SM10{0,3}/SM110 and §W10. -# A cross-FAMILY FAT list (e.g. "90a;121a") resolves its features honestly but -# does NOT compile — the sources are gencode'd for the whole list, so the sm12x -# fp4 PTX is rejected on compute_90a; per-source gencode narrowing (spec W7) is -# what a fat cross-family binary needs. See .agents/specs/cuda-arch-additivity.md. +# Cross-family fat lists use release-W1 per-source gencode: portable TUs receive +# every requested target while feature TUs receive their exact feature-table +# intersection. The ten-SM archive gate below proves both the commands and the +# linked cubins. See .agents/specs/release-binary-matrix.md W1. set(VLLM_CPP_CUDA_ARCHITECTURES "121a" CACHE STRING "Target CUDA architectures (GB10=121a, consumer Blackwell=120a, same-family fat binary=120a;121a; the 'a' arch-specific target is REQUIRED for the native block-scaled fp4xfp4 MMA — mma.sync kind::mxf4nvf4 is rejected on base sm_120/sm_121)") option(VLLM_CPP_BENCH_PROFILE_CONTROL "Compile trace-only SIGUSR2/cudaProfiler replay control plus FIFO graceful shutdown (never for production timing builds)" @@ -920,6 +923,8 @@ add_library(vllm STATIC src/vt/cpu/cpu_threadpool.cpp src/vt/cpu/cpu_ops.cpp src/vt/cpu/cpu_minimax_h3.cpp + src/vt/cpu/cpu_isa_arm.cpp + src/vt/cpu/cpu_isa_x86.cpp src/vt/cpu/cpu_matmul_elem.cpp src/vt/cpu/cpu_quant_dequant.cpp src/vt/cpu/cpu_quant_act.cpp @@ -974,16 +979,21 @@ if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|AMD64") COMPILE_OPTIONS "$<$:-mavx512f;-mavx512bw;-mavx512vl;-mf16c>") endif() if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|arm64") + # The portable Q8 DotProd C++ tier is valid on Linux and Darwin. Compile only + # its TU with DotProd enabled; runtime entry is gated by Linux HWCAP or Darwin + # sysctl. The separately scheduled A76 assembly remains Linux-only below. + set_source_files_properties( + src/vt/cpu/cpu_quant_dot_sdot.cpp PROPERTIES + COMPILE_DEFINITIONS VT_CPU_ARM_Q8_DOT=1 + COMPILE_OPTIONS "$<$:-march=armv8.2-a+dotprod+fp16>") # KERNEL-CPU-A76-Q8-DOT is Linux-only because its dispatch uses HWCAP/MIDR # and the hand-written file carries ELF assembler directives. Other arm64 # platforms compile the generic stub and retain their existing portable tier. if(CMAKE_SYSTEM_NAME STREQUAL "Linux") enable_language(ASM) target_sources(vllm PRIVATE src/vt/cpu/cpu_quant_dot_a76.S) - set_source_files_properties( - src/vt/cpu/cpu_quant_dot_sdot.cpp PROPERTIES - COMPILE_DEFINITIONS VT_CPU_A76_Q8_DOT=1 - COMPILE_OPTIONS "$<$:-march=armv8.2-a+dotprod+fp16>") + set_property(SOURCE src/vt/cpu/cpu_quant_dot_sdot.cpp APPEND PROPERTY + COMPILE_DEFINITIONS VT_CPU_A76_Q8_DOT=1) endif() # G6 mmla tier AND G7 repack tier (cpu_quant_repack_arm.cpp) both need i8mm. # The global -ffp-contract=off (top of this file) still applies on top of this @@ -1803,6 +1813,74 @@ if(VLLM_CPP_CUDA) "$<$:--expt-relaxed-constexpr;--expt-extended-lambda;-DFLASH_NAMESPACE=flash;-include;${_FA2_ROOT}/fa2_compat_prelude.h;-isystem;${_FA2_ROOT}/src;-isystem;${_FA2_ROOT}/stubs;-isystem;${VLLM_CPP_CUTLASS_DIR}/include>") message(STATUS "FlashAttention-2 prefill/decode: ENABLED for arch(es) [${VT_FA2_ARCHS}] (runtime toggles VT_FA2_PREFILL, VT_FA2_DECODE)") endif() + + # RELEASE W1: CMake's target-wide CUDA_ARCHITECTURES property would compile + # every CUDA TU for every requested SM, including architecture-specific + # CUTLASS/Marlin/FA2 bodies that ptxas rejects outside their feature-table + # cells. Disable target-wide emission for libvllm and give every CUDA source + # exactly one explicit gencode set: portable sources receive the complete + # requested list, feature sources receive only their resolved intersection. + # This mirrors vLLM's set_gencode_flags_for_srcs rather than inventing a + # release-only compilation path. + set_property(TARGET vllm PROPERTY CUDA_ARCHITECTURES OFF) + set(_VT_CUDA_FEATURE_SOURCES) + + if(VT_FP4_MMA_ARCHS) + target_sources(vllm PRIVATE src/vt/cuda/cuda_nvfp4_sm12x.cu) + vt_cuda_set_source_gencode("${VT_FP4_MMA_ARCHS}" + src/vt/cuda/cuda_nvfp4_sm12x.cu) + list(APPEND _VT_CUDA_FEATURE_SOURCES src/vt/cuda/cuda_nvfp4_sm12x.cu) + endif() + + if(_NVFP4_CUTLASS_SOURCES) + vt_cuda_set_source_gencode("${VT_CUTLASS_NVFP4_ARCHS}" + ${_NVFP4_CUTLASS_SOURCES}) + list(APPEND _VT_CUDA_FEATURE_SOURCES ${_NVFP4_CUTLASS_SOURCES}) + endif() + if(_FP8_CUTLASS_SOURCES) + vt_cuda_set_source_gencode("${VT_CUTLASS_FP8_ARCHS}" + ${_FP8_CUTLASS_SOURCES}) + list(APPEND _VT_CUDA_FEATURE_SOURCES ${_FP8_CUTLASS_SOURCES}) + endif() + if(VLLM_CPP_CUTLASS_NVFP4_SM100) + vt_cuda_set_source_gencode("${VT_CUTLASS_NVFP4_SM100_ARCHS}" + src/vt/cuda/cuda_matmul_nvfp4_sm100.cu) + list(APPEND _VT_CUDA_FEATURE_SOURCES + src/vt/cuda/cuda_matmul_nvfp4_sm100.cu) + endif() + if(VLLM_CPP_SCALEDMM_C3X_SM90) + vt_cuda_set_source_gencode("${VT_SCALEDMM_C3X_SM90_ARCHS}" + src/vt/cuda/cuda_scaled_mm_c3x_sm90.cu) + list(APPEND _VT_CUDA_FEATURE_SOURCES + src/vt/cuda/cuda_scaled_mm_c3x_sm90.cu) + endif() + if(VLLM_CPP_SCALEDMM_C3X_SM100) + vt_cuda_set_source_gencode("${VT_SCALEDMM_C3X_SM100_ARCHS}" + src/vt/cuda/cuda_scaled_mm_c3x_sm100.cu) + list(APPEND _VT_CUDA_FEATURE_SOURCES + src/vt/cuda/cuda_scaled_mm_c3x_sm100.cu) + endif() + if(_MARLIN_SRCS) + vt_cuda_set_source_gencode("${VT_MARLIN_NVFP4_ARCHS}" ${_MARLIN_SRCS}) + list(APPEND _VT_CUDA_FEATURE_SOURCES ${_MARLIN_SRCS}) + endif() + if(_FA2_KERNEL_SRCS) + set(_VT_FA2_SOURCES + src/vt/cuda/cuda_flash_attn_fa2.cu + ${_FA2_KERNEL_SRCS}) + vt_cuda_set_source_gencode("${VT_FA2_ARCHS}" ${_VT_FA2_SOURCES}) + list(APPEND _VT_CUDA_FEATURE_SOURCES ${_VT_FA2_SOURCES}) + endif() + + get_target_property(_VT_VLLM_TARGET_SOURCES vllm SOURCES) + set(_VT_CUDA_PORTABLE_SOURCES ${_VT_VLLM_TARGET_SOURCES}) + list(FILTER _VT_CUDA_PORTABLE_SOURCES INCLUDE REGEX "\\.cu$") + list(REMOVE_ITEM _VT_CUDA_PORTABLE_SOURCES ${_VT_CUDA_FEATURE_SOURCES}) + vt_cuda_set_source_gencode("${VLLM_CPP_CUDA_ARCHITECTURES}" + ${_VT_CUDA_PORTABLE_SOURCES}) + message(STATUS + "CUDA per-source gencode: ${VLLM_CPP_CUDA_ARCHITECTURES} portable; " + "feature TUs narrowed by VT_CUDA_FEATURE_TABLE") endif() # OpenAI HTTP server (M3.1 Task 4): the api_server.cpp translation unit pulls in diff --git a/README.md b/README.md index 650bc6e72..769818d98 100644 --- a/README.md +++ b/README.md @@ -305,7 +305,7 @@ hardware-blocked and why, is in [docs/STATUS.md](docs/STATUS.md). | Backend | Hardware | State | |---|---|---| | **CUDA** | GB10 / DGX Spark (sm_121a) | Runtime-gated. 27B at/above vLLM throughput, 35B prefill-pending | -| **CUDA** | Blackwell, Hopper, Ampere, Ada (sm_80 through sm_121a) | Build-supported, fast GDN path build-verified per-arch. Not runtime-proven here (no such boards) | +| **CUDA** | Blackwell, Hopper, Ampere, Ada (sm_80–sm_121a) | Per-arch builds pass; ten-SM archive candidate awaits hosted cubin audit; no runtime proof here | | **CPU** | x86-64, arm64 | Correctness / CI reference. At or ahead of llama.cpp on every GGUF axis, Arm i8mm quant-GEMM tier | | **Metal** | Apple Silicon | Two models end to end, 18 of 75 ops native. Prefill ahead of MLX-LM, warm total 97.6% with the MLX provider | | **Vulkan** | Portable GPU | `opt-125m` STRICT token-exact; Qwen3.6-27B decode **matches llama.cpp Vulkan** (4.36 vs 4.35). Op coverage: [docs/STATUS.md](docs/STATUS.md) | diff --git a/cmake/CudaArchFeatures.cmake b/cmake/CudaArchFeatures.cmake index b4b064a4a..1f470ce43 100644 --- a/cmake/CudaArchFeatures.cmake +++ b/cmake/CudaArchFeatures.cmake @@ -197,6 +197,54 @@ function(vt_cuda_archs_denormalize OUT_VAR IN_ARCHS) set(${OUT_VAR} "${_out}" PARENT_SCOPE) endfunction() +# vt_cuda_gencode_options( ) +# Convert an explicit CMake-form architecture list into source-scoped nvcc +# gencode options. W1 disables CMake's target-wide CUDA_ARCHITECTURES emission +# and applies these options exactly once to each CUDA source, using either the +# complete release list (portable TUs) or the feature-table intersection +# (architecture-specific TUs). This is the local equivalent of vLLM's +# set_gencode_flags_for_srcs helper in cmake/utils.cmake. +function(vt_cuda_gencode_options OUT_VAR ARCHS) + set(_options) + set(_archs ${ARCHS}) + list(REMOVE_DUPLICATES _archs) + list(SORT _archs COMPARE NATURAL ORDER ASCENDING) + foreach(_arch IN LISTS _archs) + set(_emit_ptx OFF) + if(_arch MATCHES "\\+PTX$") + set(_emit_ptx ON) + string(REGEX REPLACE "\\+PTX$" "" _arch "${_arch}") + endif() + if(NOT _arch MATCHES "^[0-9]+[af]?$") + message(FATAL_ERROR + "vt_cuda_gencode_options: invalid CUDA architecture '${_arch}'") + endif() + list(APPEND _options + "-gencode=arch=compute_${_arch},code=sm_${_arch}") + if(_emit_ptx) + list(APPEND _options + "-gencode=arch=compute_${_arch},code=compute_${_arch}") + endif() + endforeach() + set(${OUT_VAR} "${_options}" PARENT_SCOPE) +endfunction() + +# vt_cuda_set_source_gencode( ...) +# Apply one explicit architecture set to CUDA translation units without +# disturbing their existing source-scoped warnings, includes, definitions, +# or CUTLASS/FlashInfer options. +function(vt_cuda_set_source_gencode ARCHS) + if(NOT ARGN) + return() + endif() + vt_cuda_gencode_options(_gencode "${ARCHS}") + if(NOT _gencode) + message(FATAL_ERROR + "vt_cuda_set_source_gencode: no architectures for sources [${ARGN}]") + endif() + set_property(SOURCE ${ARGN} APPEND PROPERTY COMPILE_OPTIONS ${_gencode}) +endfunction() + # =========================================================================== # THE FEATURE TABLE — arch -> capability set. # @@ -354,19 +402,15 @@ function(vt_cuda_report_feature FEATURE RESOLVED_ARCHS) if(RESOLVED_ARCHS) message(STATUS " CUDA feature ${FEATURE}: ENABLED for [${RESOLVED_ARCHS}]") # A strict subset means the fat build contains archs with no tactic for this - # feature. Today that is a LOUD build-time compile failure on the untargeted - # arch (there are no cross-family tactic bodies yet), never a silent capability - # drop. Narrowing per-source gencode (vLLM's set_gencode_flags_for_srcs, - # cmake/utils.cmake:265-345) lands with the first cross-family tactic. + # feature. W1 source-scoped gencode compiles this feature only for the named + # intersection while the other archs retain their portable sources. set(_missing "${_tgt_cmake}") list(REMOVE_ITEM _missing ${RESOLVED_ARCHS}) if(_missing) message(WARNING "CUDA feature '${FEATURE}' has no tactic for requested arch(es) [${_missing}]. " - "It stays ENABLED for [${RESOLVED_ARCHS}]; the sources are still compiled for the " - "whole target list, so building this heterogeneous fat binary requires the per-arch " - "tactic bodies (and per-source gencode narrowing) first. See " - ".agents/specs/cuda-arch-additivity.md.") + "It stays ENABLED for [${RESOLVED_ARCHS}]; W1 compiles its source TUs only " + "for that intersection and preserves portable fallbacks for the other targets.") endif() else() message(STATUS " CUDA feature ${FEATURE}: DISABLED (no requested arch in [${_tgt_cmake}] provides it)") diff --git a/cmake/CudaArchFeaturesTest.cmake b/cmake/CudaArchFeaturesTest.cmake index 668057d35..bdf31cd81 100644 --- a/cmake/CudaArchFeaturesTest.cmake +++ b/cmake/CudaArchFeaturesTest.cmake @@ -75,10 +75,9 @@ foreach(_f IN LISTS _ALL_FEATURES) endforeach() # --- A cross-family target has NO tactic body here and must resolve EMPTY for -# itself while leaving the sm_12x arch fully enabled. (Such a heterogeneous list -# still cannot COMPILE — the sources are gencode'd for the whole list — but the -# resolution must be honest about which arch provides what; see -# .agents/specs/cuda-arch-additivity.md Risks #1.) +# itself while leaving the sm_12x arch fully enabled. W1's source-scoped +# gencode now makes this heterogeneous list compilable without widening the +# feature body; the resolution remains the authority for which arch provides it. foreach(_f IN LISTS _ALL_FEATURES) expect_feature("90a;121a" "${_f}" "121a") endforeach() diff --git a/cmake/CudaSourceGencodeTest.cmake b/cmake/CudaSourceGencodeTest.cmake new file mode 100644 index 000000000..05c39770e --- /dev/null +++ b/cmake/CudaSourceGencodeTest.cmake @@ -0,0 +1,62 @@ +# Toolkit-free executable contract for W1 per-source CUDA gencode. +cmake_minimum_required(VERSION 3.24) + +get_filename_component(_here "${CMAKE_CURRENT_LIST_DIR}" ABSOLUTE) +include("${_here}/CudaArchFeatures.cmake") + +set(_ten_sms "80;86;87;89;90a;100a;103a;110;120a;121a") + +function(expect_gencode ARCHS EXPECTED) + vt_cuda_gencode_options(_got "${ARCHS}") + if(NOT "${_got}" STREQUAL "${EXPECTED}") + message(FATAL_ERROR + "GENCODE MISMATCH for [${ARCHS}]: expected [${EXPECTED}], got [${_got}]") + endif() +endfunction() + +expect_gencode("90a;120a" + "-gencode=arch=compute_90a,code=sm_90a;-gencode=arch=compute_120a,code=sm_120a") +expect_gencode("${_ten_sms}" + "-gencode=arch=compute_80,code=sm_80;-gencode=arch=compute_86,code=sm_86;-gencode=arch=compute_87,code=sm_87;-gencode=arch=compute_89,code=sm_89;-gencode=arch=compute_90a,code=sm_90a;-gencode=arch=compute_100a,code=sm_100a;-gencode=arch=compute_103a,code=sm_103a;-gencode=arch=compute_110,code=sm_110;-gencode=arch=compute_120a,code=sm_120a;-gencode=arch=compute_121a,code=sm_121a") + +set(VLLM_CPP_CUDA_ARCHITECTURES "${_ten_sms}") +vt_cuda_feature_archs(_fp4 "fp4-mma") +vt_cuda_feature_archs(_sm90 "scaledmm-c3x-sm90") +vt_cuda_feature_archs(_sm100 "scaledmm-c3x-sm100") +vt_cuda_feature_archs(_fa2 "fa2") +expect_gencode("${_fp4}" + "-gencode=arch=compute_120a,code=sm_120a;-gencode=arch=compute_121a,code=sm_121a") +expect_gencode("${_sm90}" "-gencode=arch=compute_90a,code=sm_90a") +expect_gencode("${_sm100}" "-gencode=arch=compute_100a,code=sm_100a") +expect_gencode("${_fa2}" + "-gencode=arch=compute_80,code=sm_80;-gencode=arch=compute_86,code=sm_86;-gencode=arch=compute_87,code=sm_87;-gencode=arch=compute_89,code=sm_89;-gencode=arch=compute_120a,code=sm_120a;-gencode=arch=compute_121a,code=sm_121a") + +get_filename_component(_root "${_here}/.." ABSOLUTE) +file(READ "${_root}/CMakeLists.txt" _root_cmake) +foreach(_required IN ITEMS + "set_property(TARGET vllm PROPERTY CUDA_ARCHITECTURES OFF)" + "vt_cuda_set_source_gencode(\"\${VT_FP4_MMA_ARCHS}\"" + "src/vt/cuda/cuda_nvfp4_sm12x.cu" + "vt_cuda_set_source_gencode(\"\${VT_CUTLASS_NVFP4_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VT_CUTLASS_FP8_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VT_SCALEDMM_C3X_SM90_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VT_SCALEDMM_C3X_SM100_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VT_MARLIN_NVFP4_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VT_FA2_ARCHS}\"" + "vt_cuda_set_source_gencode(\"\${VLLM_CPP_CUDA_ARCHITECTURES}\"") + string(FIND "${_root_cmake}" "${_required}" _position) + if(_position EQUAL -1) + message(FATAL_ERROR "missing W1 source-gencode wiring: ${_required}") + endif() +endforeach() + +file(READ "${_root}/src/vt/cuda/cuda_matmul_nvfp4.cu" _portable_nvfp4) +file(READ "${_root}/src/vt/cuda/cuda_nvfp4_sm12x.cu" _native_nvfp4) +string(FIND "${_portable_nvfp4}" "kind::mxf4nvf4" _portable_native_opcode) +string(FIND "${_native_nvfp4}" "kind::mxf4nvf4" _native_opcode) +if(NOT _portable_native_opcode EQUAL -1 OR _native_opcode EQUAL -1) + message(FATAL_ERROR + "sm12x native MMA must exist only in cuda_nvfp4_sm12x.cu") +endif() + +message(STATUS "CUDA per-source gencode expectations: ALL PASS") diff --git a/cmake/TritonAOT.cmake b/cmake/TritonAOT.cmake index a5c7efcc2..796e00974 100644 --- a/cmake/TritonAOT.cmake +++ b/cmake/TritonAOT.cmake @@ -60,6 +60,8 @@ option(VLLM_CPP_TRITON option(VLLM_CPP_TRITON_REGEN "MAINTAINER: regenerate target-pinned vendored Triton AOT artifacts with Python+Triton+ptxas; see scripts/regen-triton-aot.sh" OFF) +include("${CMAKE_CURRENT_LIST_DIR}/TritonAOTMultiArch.cmake") + # Where the vendored artifacts live, one subdir per arch (sm_121a today; # sm_90/sm_80/gfx* slots in later). Each arch dir holds the generated .c/.h for # every kernel base plus a MANIFEST (generator versions + source hashes + @@ -90,6 +92,7 @@ set(VLLM_CPP_TRITON_TARGET "" CACHE STRING # add_triton_kernel call appends its exact manifest line in both builder and # regen modes; finalize compares it with the canonical contract and MANIFEST. set_property(GLOBAL PROPERTY VLLM_TRITON_AOT_EXPECTED_BASE_LINES "") +set_property(GLOBAL PROPERTY VLLM_TRITON_AOT_DISPATCH_DECLS "") # _triton_aot_arch_name(OUTVAR) -> active vendored arch directory name. # @@ -134,10 +137,31 @@ function(_triton_aot_arch_dir OUTVAR) set(${OUTVAR} "${VLLM_CPP_TRITON_VENDORED_DIR}/${_a}" PARENT_SCOPE) endfunction() +# _triton_aot_arch_names(OUTVAR) -> artifact trees embedded by a normal build. +# W2 deliberately embeds every complete vendored tree. Runtime selection is +# exact-SM, so sm_87/sm_103/sm_110/sm_120 use the portable CUDA fallback and +# can never attempt a neighboring cubin. Maintainer regeneration remains a +# one-tree operation selected by VLLM_CPP_TRITON_VENDORED_ARCH. +function(_triton_aot_arch_names OUTVAR) + if(VLLM_CPP_TRITON_REGEN) + _triton_aot_arch_name(_single) + set(_arches "${_single}") + else() + vt_triton_aot_available_arches(_arches) + endif() + foreach(_arch IN LISTS _arches) + if(NOT EXISTS "${VLLM_CPP_TRITON_VENDORED_DIR}/${_arch}/MANIFEST") + message(FATAL_ERROR + "Triton AOT W2 matrix names missing tree ${_arch}; regenerate or repair " + "${VLLM_CPP_TRITON_VENDORED_DIR}") + endif() + endforeach() + set(${OUTVAR} "${_arches}" PARENT_SCOPE) +endfunction() + # Resolve and validate the code-generation target from the vendored destination. # The current AOT toolchain is CUDA-only and accepts one target per artifact tree. -function(_triton_aot_resolved_target OUTVAR) - _triton_aot_arch_name(_arch) +function(_triton_aot_resolved_target_for_arch OUTVAR _arch) if(NOT _arch MATCHES "^sm_([0-9]+)a?$") message(FATAL_ERROR "Cannot derive a single CUDA Triton target from vendored arch '${_arch}'. " @@ -153,6 +177,11 @@ function(_triton_aot_resolved_target OUTVAR) set(${OUTVAR} "${_derived}" PARENT_SCOPE) endfunction() +function(_triton_aot_resolved_target OUTVAR) + _triton_aot_arch_name(_arch) + _triton_aot_resolved_target_for_arch(${OUTVAR} "${_arch}") +endfunction() + # add_triton_kernel(RESULT_VAR KERNEL_PY KERNEL_NAME OUT_BASE SIGNATURE GRID # [NUM_WARPS] [NUM_STAGES]) # @@ -185,7 +214,6 @@ function(add_triton_kernel RESULT_VAR KERNEL_PY KERNEL_NAME OUT_BASE SIGNATURE G message(FATAL_ERROR "add_triton_kernel: kernel file not found: ${KERNEL_PY}") endif() get_filename_component(_py_name "${KERNEL_PY}" NAME) - _triton_aot_arch_dir(_adir) # The MANIFEST line that describes THIS generation request. Written on regen; # compared against the vendored MANIFEST when consuming (a mismatch means the @@ -196,54 +224,73 @@ function(add_triton_kernel RESULT_VAR KERNEL_PY KERNEL_NAME OUT_BASE SIGNATURE G "${_manifest_line}") if(NOT VLLM_CPP_TRITON_REGEN) - # ── BUILDER path: consume the vendored artifacts. No Python. ───────────── - if(NOT EXISTS "${_adir}/MANIFEST") - message(FATAL_ERROR - "VLLM_CPP_TRITON=ON but there are no vendored Triton AOT artifacts for " - "this arch:\n ${_adir}\n" - "Options:\n" - " * build without -DVLLM_CPP_TRITON=ON (the portable C++ CUDA kernels are\n" - " the always-available fallback), or\n" - " * set -DVLLM_CPP_TRITON_VENDORED_ARCH= if the artifacts exist under\n" - " another name, or\n" - " * regenerate for this arch (MAINTAINER task; needs Python+Triton+ptxas):\n" - " scripts/regen-triton-aot.sh (configure with -DVLLM_CPP_TRITON_REGEN=ON)") - endif() - if(NOT EXISTS "${_adir}/${OUT_BASE}.c" OR NOT EXISTS "${_adir}/${OUT_BASE}.h") - message(FATAL_ERROR - "Vendored Triton AOT tree ${_adir} is missing the '${OUT_BASE}' kernel " - "(expected ${OUT_BASE}.c/.h). The vendored tree predates this kernel — " - "regenerate it: scripts/regen-triton-aot.sh") - endif() - file(GLOB _spec_sources "${_adir}/${OUT_BASE}.*.c") - if(NOT _spec_sources) - message(FATAL_ERROR - "Vendored Triton AOT tree ${_adir} has ${OUT_BASE}.c but no per-spec " - "${OUT_BASE}..c launchers — the tree is corrupt; regenerate it: " - "scripts/regen-triton-aot.sh") - endif() - set(_all_sources ${_spec_sources} "${_adir}/${OUT_BASE}.c") + # ── BUILDER path: consume and namespace all six vendored trees. ────────── + _triton_aot_arch_names(_arch_names) + set(_all_sources) + set(_dispatch_dir "${CMAKE_BINARY_DIR}/triton_aot_multiarch") + foreach(_arch IN LISTS _arch_names) + set(_adir "${VLLM_CPP_TRITON_VENDORED_DIR}/${_arch}") + if(NOT EXISTS "${_adir}/${OUT_BASE}.c" OR + NOT EXISTS "${_adir}/${OUT_BASE}.h") + message(FATAL_ERROR + "Vendored Triton AOT tree ${_adir} is missing '${OUT_BASE}'. " + "Regenerate: scripts/regen-triton-aot.sh") + endif() + file(GLOB _spec_sources "${_adir}/${OUT_BASE}.*.c") + if(NOT _spec_sources) + message(FATAL_ERROR + "Vendored Triton AOT tree ${_adir} has no ${OUT_BASE}..c") + endif() + set(_tree_sources ${_spec_sources} "${_adir}/${OUT_BASE}.c") - # Parameter staleness: find this base's line in the MANIFEST and compare. - set(_found_line "") - file(STRINGS "${_adir}/MANIFEST" _mlines) - foreach(_line IN LISTS _mlines) - string(FIND "${_line}" "base ${OUT_BASE} " _pos) - if(_pos EQUAL 0) - set(_found_line "${_line}") + set(_found_line "") + file(STRINGS "${_adir}/MANIFEST" _mlines) + foreach(_line IN LISTS _mlines) + string(FIND "${_line}" "base ${OUT_BASE} " _pos) + if(_pos EQUAL 0) + set(_found_line "${_line}") + endif() + endforeach() + if(NOT _found_line STREQUAL _manifest_line) + message(FATAL_ERROR + "STALE VENDORED TRITON AOT ARTIFACTS: '${OUT_BASE}' differs in " + "${_adir}/MANIFEST.\n expected: ${_manifest_line}\n" + " vendored: ${_found_line}") + endif() + + vt_triton_aot_namespace_sources( + "${_arch}" "${OUT_BASE}" "${_dispatch_dir}/namespaces" + _tree_sources_namespaced ${_tree_sources}) + list(APPEND _all_sources ${_tree_sources_namespaced}) + + file(STRINGS "${_adir}/${OUT_BASE}.h" _default_decl + REGEX "^CUresult ${OUT_BASE}_default\\(") + list(LENGTH _default_decl _decl_count) + if(NOT _decl_count EQUAL 1) + message(FATAL_ERROR + "${_adir}/${OUT_BASE}.h must declare one ${OUT_BASE}_default") endif() + vt_triton_aot_namespace_declaration( + _default_decl "${_arch}" "${OUT_BASE}_default" "${_default_decl}") + set_property(GLOBAL APPEND PROPERTY VLLM_TRITON_AOT_DISPATCH_DECLS + "${_default_decl}") + file(STRINGS "${_adir}/${OUT_BASE}.h" _load_decl + REGEX "^void load_${OUT_BASE}\\(") + list(LENGTH _load_decl _load_decl_count) + if(NOT _load_decl_count EQUAL 1) + message(FATAL_ERROR + "${_adir}/${OUT_BASE}.h must declare one load_${OUT_BASE}") + endif() + vt_triton_aot_namespace_declaration( + _load_decl "${_arch}" "load_${OUT_BASE}" "${_load_decl}") + set_property(GLOBAL APPEND PROPERTY VLLM_TRITON_AOT_DISPATCH_DECLS + "${_load_decl}") + vt_triton_aot_namespace_token( + _namespaced_default "${_arch}" "${OUT_BASE}_default") + message(STATUS + "Triton AOT: ${OUT_BASE} <- ${_arch} as ${_namespaced_default}") endforeach() - if(NOT _found_line STREQUAL _manifest_line) - message(FATAL_ERROR - "STALE VENDORED TRITON AOT ARTIFACTS: the generation parameters for " - "'${OUT_BASE}' differ from ${_adir}/MANIFEST.\n" - " expected: ${_manifest_line}\n" - " vendored: ${_found_line}\n" - "Refusing to build cubins that do not reflect the declared launch ABI. " - "Regenerate: " - "scripts/regen-triton-aot.sh") - endif() - message(STATUS "Triton AOT: ${OUT_BASE} <- vendored ${_adir} (no Python)") + set(_result_include_dir "${_dispatch_dir}") else() # ── MAINTAINER path: regenerate with the Python toolchain, then refresh ── # the vendored tree and compile THOSE files (what a builder will compile). @@ -254,6 +301,7 @@ function(add_triton_kernel RESULT_VAR KERNEL_PY KERNEL_NAME OUT_BASE SIGNATURE G "Point -DVLLM_CPP_TRITON_PYTHON= at a Python that has Triton.") endif() + _triton_aot_arch_dir(_adir) set(_outdir "${CMAKE_BINARY_DIR}/triton_aot") file(MAKE_DIRECTORY "${_outdir}") @@ -361,16 +409,17 @@ function(add_triton_kernel RESULT_VAR KERNEL_PY KERNEL_NAME OUT_BASE SIGNATURE G endif() endforeach() message(STATUS "Triton AOT: ${OUT_BASE} regenerated -> ${_adir}") + set(_result_include_dir "${_adir}") endif() # This is GENERATED codegen, not our code: compile with warnings suppressed # (-w). The launch stub deliberately has a missing-return on its empty-grid # branch; without -w that would trip a warning. The files end in .c, so CMake # compiles them as C automatically (enable_language(C) at the call site). - set_source_files_properties(${_all_sources} PROPERTIES COMPILE_OPTIONS "-w") + set_property(SOURCE ${_all_sources} APPEND PROPERTY COMPILE_OPTIONS "-w") set(${RESULT_VAR} "${_all_sources}" PARENT_SCOPE) - set(${RESULT_VAR}_INCLUDE_DIR "${_adir}" PARENT_SCOPE) + set(${RESULT_VAR}_INCLUDE_DIR "${_result_include_dir}" PARENT_SCOPE) set(${RESULT_VAR}_HEADER "${OUT_BASE}.h" PARENT_SCOPE) message(STATUS "Triton AOT: ${OUT_BASE} -> stable symbol ${OUT_BASE}_default()") @@ -386,9 +435,13 @@ endfunction() # * builder: verifies target, line-info policy, source hashes, generation # declarations, artifact inventory, and artifact hashes; any drift is fatal. function(triton_aot_finalize) - _triton_aot_arch_dir(_adir) - _triton_aot_arch_name(_arch_name) - _triton_aot_resolved_target(_resolved_target) + if(VLLM_CPP_TRITON_REGEN) + _triton_aot_arch_dir(_adir) + _triton_aot_arch_name(_arch_name) + _triton_aot_resolved_target(_resolved_target) + else() + _triton_aot_arch_names(_arch_names) + endif() get_property(_expected_bases GLOBAL PROPERTY VLLM_TRITON_AOT_EXPECTED_BASE_LINES) list(SORT _expected_bases) @@ -469,6 +522,25 @@ else: "(triton ${_triton_ver}, ptxas '${_ptxas_ver}'). " "Review + commit the vendored tree: git diff ${_adir}") else() + get_property(_dispatch_decls GLOBAL PROPERTY + VLLM_TRITON_AOT_DISPATCH_DECLS) + list(REMOVE_DUPLICATES _dispatch_decls) + list(SORT _dispatch_decls) + set(_dispatch_dir "${CMAKE_BINARY_DIR}/triton_aot_multiarch") + file(MAKE_DIRECTORY "${_dispatch_dir}") + set(_dispatch_header "${_dispatch_dir}/triton_aot_multiarch.h") + file(WRITE "${_dispatch_header}" + "/* Generated W2 declarations for six collision-free AOT trees. */\n" + "#pragma once\n#include \n#ifdef __cplusplus\nextern \"C\" {\n#endif\n") + foreach(_decl IN LISTS _dispatch_decls) + file(APPEND "${_dispatch_header}" "${_decl};\n") + endforeach() + file(APPEND "${_dispatch_header}" + "#ifdef __cplusplus\n}\n#endif\n") + + foreach(_arch_name IN LISTS _arch_names) + set(_adir "${VLLM_CPP_TRITON_VENDORED_DIR}/${_arch_name}") + _triton_aot_resolved_target_for_arch(_resolved_target "${_arch_name}") # Exact dispatcher/signature set for the vendored tree we just consumed. file(STRINGS "${_adir}/MANIFEST" _mlines) if(NOT "arch ${_arch_name}" IN_LIST _mlines) @@ -570,6 +642,10 @@ else: message(STATUS "Triton AOT: vendored tree ${_adir} matches triton_kernels/ " "(MANIFEST hashes OK)") endif() + endforeach() + message(STATUS + "Triton AOT W2: embedded trees [${_arch_names}]; exact runtime dispatch " + "header ${_dispatch_header}") endif() endfunction() diff --git a/cmake/TritonAOTMultiArch.cmake b/cmake/TritonAOTMultiArch.cmake new file mode 100644 index 000000000..d0a6ccc42 --- /dev/null +++ b/cmake/TritonAOTMultiArch.cmake @@ -0,0 +1,81 @@ +# W2 helpers for embedding all available Triton AOT cubin trees in one binary. +# Kept separate so the exact matrix and symbol namespace can be tested with +# `cmake -P` without enabling a compiler or CUDA. + +set(VT_TRITON_AOT_AVAILABLE_ARCHES + sm_80 sm_86 sm_89 sm_90a sm_100a sm_121a) + +function(vt_triton_aot_available_arches OUT_VAR) + set(${OUT_VAR} "${VT_TRITON_AOT_AVAILABLE_ARCHES}" PARENT_SCOPE) +endfunction() + +function(vt_triton_aot_arch_tree OUT_VAR ARCH) + set(_tree "sm_${ARCH}") + if(_tree IN_LIST VT_TRITON_AOT_AVAILABLE_ARCHES) + set(${OUT_VAR} "${_tree}" PARENT_SCOPE) + else() + set(${OUT_VAR} "" PARENT_SCOPE) + endif() +endfunction() + +function(vt_triton_aot_namespace_token OUT_VAR ARCH TOKEN) + if(NOT ARCH MATCHES "^sm_[0-9]+a?$") + message(FATAL_ERROR "invalid Triton AOT namespace architecture '${ARCH}'") + endif() + if(NOT TOKEN MATCHES "^[A-Za-z_][A-Za-z0-9_]*$") + message(FATAL_ERROR "invalid Triton AOT namespace token '${TOKEN}'") + endif() + set(${OUT_VAR} "vt_aot_${ARCH}_${TOKEN}" PARENT_SCOPE) +endfunction() + +# Return one namespaced declaration without its trailing semicolon. CMake uses +# semicolons as list separators, so storing raw C declarations in a GLOBAL +# property can drop the terminator and concatenate two declarations into a +# function definition. The final header writer adds exactly one semicolon. +function(vt_triton_aot_namespace_declaration OUT_VAR ARCH TOKEN DECLARATION) + vt_triton_aot_namespace_token(_namespaced "${ARCH}" "${TOKEN}") + string(REPLACE "${TOKEN}" "${_namespaced}" _declaration "${DECLARATION}") + # file(STRINGS) escapes a source semicolon as `\;` in a CMake list. Remove + # both characters before this value enters another list/property. + string(REGEX REPLACE "[\\\\;]+$" "" _declaration "${_declaration}") + set(${OUT_VAR} "${_declaration}" PARENT_SCOPE) +endfunction() + +# Prefix every generated external identifier containing BASE before compiling a +# vendored C launcher. This includes stable dispatchers, hash dispatchers, +# module/function globals, cubin arrays, and load/unload functions. The embedded +# kernel name string remains untouched, as required by cuModuleGetFunction. +function(vt_triton_aot_namespace_sources ARCH BASE OUTPUT_DIR OUT_VAR) + set(_sources ${ARGN}) + if(NOT _sources) + message(FATAL_ERROR "no Triton AOT sources to namespace for ${ARCH}/${BASE}") + endif() + file(MAKE_DIRECTORY "${OUTPUT_DIR}") + set(_tokens) + foreach(_source IN LISTS _sources) + file(STRINGS "${_source}" _symbol_lines REGEX "${BASE}") + string(JOIN "\n" _symbol_text ${_symbol_lines}) + string(REGEX MATCHALL + "[A-Za-z_][A-Za-z0-9_]*${BASE}[A-Za-z0-9_]*" + _prefixed_tokens "${_symbol_text}") + string(REGEX MATCHALL "${BASE}[A-Za-z0-9_]*" + _base_tokens "${_symbol_text}") + list(APPEND _tokens ${_prefixed_tokens} ${_base_tokens}) + endforeach() + list(REMOVE_DUPLICATES _tokens) + list(SORT _tokens) + if(NOT _tokens) + message(FATAL_ERROR "no generated symbols found for ${ARCH}/${BASE}") + endif() + + set(_header "${OUTPUT_DIR}/${ARCH}_${BASE}_namespace.h") + file(WRITE "${_header}" + "/* Generated by TritonAOTMultiArch.cmake: collision-free W2 symbols. */\n") + foreach(_token IN LISTS _tokens) + vt_triton_aot_namespace_token(_namespaced "${ARCH}" "${_token}") + file(APPEND "${_header}" "#define ${_token} ${_namespaced}\n") + endforeach() + set_property(SOURCE ${_sources} APPEND PROPERTY + COMPILE_OPTIONS "-include${_header}") + set(${OUT_VAR} "${_sources}" PARENT_SCOPE) +endfunction() diff --git a/cmake/TritonAOTMultiArchTest.cmake b/cmake/TritonAOTMultiArchTest.cmake new file mode 100644 index 000000000..257e44c80 --- /dev/null +++ b/cmake/TritonAOTMultiArchTest.cmake @@ -0,0 +1,75 @@ +cmake_minimum_required(VERSION 3.24) + +get_filename_component(_here "${CMAKE_CURRENT_LIST_DIR}" ABSOLUTE) +include("${_here}/TritonAOTMultiArch.cmake") + +vt_triton_aot_available_arches(_arches) +if(NOT _arches STREQUAL "sm_80;sm_86;sm_89;sm_90a;sm_100a;sm_121a") + message(FATAL_ERROR "unexpected W2 AOT tree order: [${_arches}]") +endif() + +foreach(_case IN ITEMS + "80=sm_80" "86=sm_86" "89=sm_89" "90a=sm_90a" + "100a=sm_100a" "121a=sm_121a") + string(REPLACE "=" ";" _parts "${_case}") + list(GET _parts 0 _arch) + list(GET _parts 1 _expected) + vt_triton_aot_arch_tree(_actual "${_arch}") + if(NOT _actual STREQUAL _expected) + message(FATAL_ERROR "${_arch}: expected ${_expected}, got ${_actual}") + endif() +endforeach() + +foreach(_arch IN ITEMS 87 103a 110 120a) + vt_triton_aot_arch_tree(_actual "${_arch}") + if(_actual) + message(FATAL_ERROR "${_arch}: unavailable tree must select fallback") + endif() +endforeach() + +vt_triton_aot_namespace_token(_token "sm_90a" "gdn_deltah_h48_default") +if(NOT _token STREQUAL "vt_aot_sm_90a_gdn_deltah_h48_default") + message(FATAL_ERROR "unexpected namespaced token: ${_token}") +endif() + +vt_triton_aot_namespace_declaration( + _declaration "sm_100a" "gdn_chunko_bf16_h48_default" + "CUresult gdn_chunko_bf16_h48_default(CUstream stream);") +if(NOT _declaration STREQUAL + "CUresult vt_aot_sm_100a_gdn_chunko_bf16_h48_default(CUstream stream)") + message(FATAL_ERROR "unexpected normalized declaration: ${_declaration}") +endif() + +# Exercise the same GLOBAL/list/write shape that lost a semicolon in the first +# hosted fat build. Every generated prototype must remain a declaration instead +# of running into the next line as a malformed function definition. +get_filename_component(_root "${_here}/.." ABSOLUTE) +set(_declarations) +foreach(_arch IN LISTS _arches) + file(STRINGS + "${_root}/src/vt/cuda/triton_aot_vendored/${_arch}/gdn_chunko_bf16_h48.h" + _raw REGEX "^CUresult gdn_chunko_bf16_h48_default\\(") + vt_triton_aot_namespace_declaration( + _normalized "${_arch}" "gdn_chunko_bf16_h48_default" "${_raw}") + list(APPEND _declarations "${_normalized}") +endforeach() +list(REMOVE_DUPLICATES _declarations) +list(SORT _declarations) +set(_generated "${CMAKE_CURRENT_BINARY_DIR}/triton-aot-declaration-test.h") +file(WRITE "${_generated}" "") +foreach(_decl IN LISTS _declarations) + file(APPEND "${_generated}" "${_decl};\n") +endforeach() +file(STRINGS "${_generated}" _generated_lines) +list(LENGTH _generated_lines _generated_count) +if(NOT _generated_count EQUAL 6) + message(FATAL_ERROR "expected six declarations, got ${_generated_count}") +endif() +foreach(_line IN LISTS _generated_lines) + if(NOT _line MATCHES ";$") + message(FATAL_ERROR "generated declaration lost semicolon: ${_line}") + endif() +endforeach() +file(REMOVE "${_generated}") + +message(STATUS "Triton AOT multi-arch matrix: ALL PASS") diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 44fb97f78..c48084c8d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,11 +1,12 @@ # Benchmarks -## At a glance: W5 manifest guarded; release artifacts pending +## At a glance: W5/W6 green; validated release artifacts pending | Reference | Workload | Headline | Tokens | |---|---|---|---| | **Structured state record (active)** | v1 scalar + relational + Git-history contracts | No benchmark. At `776c56f1`: 157 imports = 3,231,342 exact bytes; append preserved all 156 wrappers/rows. 95 tests: validator/core 44 (checker 20 + core 24), NOW 18, migration 22, cutover 11. New raw-row mutation guard. | n/a | -| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`: versioned deterministic manifest for primary host-ABI fat-CUDA + adaptive-CPU static-core bundles, optional per-SM diagnostics and experimental literal-static musl CPU | **W5 GREEN:** 19/19 + ten review mutations killed; contract 30/30; PR-size classes/budget GREEN. Synthetic fixtures only. **PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates | n/a | +| **Binary release matrix (ACTIVE; required W1-W11/W13 implemented in #196)** | Eight primary CPU/CUDA/Vulkan/Metal/MLX host tuples | Adaptive x86 tiers, Vulkan 35/35 + cross-device 11/11, and metadata/mutation gates green. **PENDING:** hosted full matrix, matching hardware, tagged publish | n/a | +| **Binary release delivery topology** | #196: read-only build/verify, OIDC attest, protected publish; generated indexes and explicit handoff-authenticated assets | Fixes the zero-binary release path by attaching all eight archive/checksum/provenance triplets plus indexes. Hosted proof pending; W12 diagnostics optional | n/a | | **Container images (inventoried)** | `ENG-RELEASE-CONTAINERS`: GHCR images from GitHub Actions; lanes `-cuda`/`-vulkan`/`-cpu`, amd64+arm64 manifests | **No number owed:** record-only. No Dockerfile, workflow, registry package or image exists; the image is the unimplemented `ENG-RELEASE-BINARIES` bundle | n/a | | **Developer agent entry point (implemented)** | `DOCS-AGENT-PROTOCOL-ENTRYPOINT`: public contribution guide + synchronized, mutation-gated pre-claim intake rule | Rebased documentation/protocol only; benchmark void | n/a | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 8b804a56d..1b7ff2610 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -97,7 +97,10 @@ portable/reference path. In normal operation leave them unset. | `VT_CPU_MATMUL_STEAL` | off | Skip the decode-shape chunk-grid collapse in the CPU matmul so work stealing stays live (the collapse is a faithful ggml port that turns a decode GEMV into `nth` equal static chunks gated by the slowest core). Same-binary A/B; byte-identical either way, since which thread computes which output changes no reduction order. Not yet measured, so the default stays off | | `VT_CPU_ELEM_KN_REPACK` | off | Transpose elementwise (f32/f16/bf16) `[N,K]` matmul weights to `[K,N]` at load, so `MatmulBT` takes the transpose-free CPU micro-kernels (1.16x to 1.30x, byte-identical). Off by default because the repacked bytes are transposed and only the CPU `MatmulBTKernel` honours the marker; enabling it for a non-CPU consumer would read garbage | | `VT_CPU_REF` | off | Set on to force the portable reference path (dequantize-everything oracle), the standard "is this a kernel bug?" bisect switch | -| `VT_CPU_Q8_DOT` | `auto` | Selects the CPU Q8_0 x Q8_0 inner dot: `portable`, compiler-generated `sdot`, or `a76-asm`. `auto` selects the gated assembly schedule only on Cortex-A76 with DotProd and otherwise stays portable. Unsupported ISA selections safely fall back; `vllm-cpu-kernel-bench --variant` fails loudly instead | +| `VT_CPU_MATMUL_TIER` | `auto` | Force the elementwise CPU GEMM tier in the same adaptive binary: `portable`/`neon` on arm64, or `portable`/`sse2`/`sse2+f16c`/`avx2`/`avx512` on x86_64. An unsupported or cross-architecture tier fails closed after the exact OS capability probe | +| `VT_CPU_Q8_DOT` | `auto` | Select the Arm Q8_0 x Q8_0 inner dot: `portable`, compiler-generated `sdot`, or the Linux-only `a76-asm`. `auto` selects the A76 schedule on Cortex-A76, otherwise SDOT on any Linux/Darwin DotProd-capable host, otherwise portable. A forced unsupported tier fails closed | +| `VT_CPU_QUANT_MMLA` | `auto` | Select the Arm i8mm quant-dot tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless Linux `HWCAP2_I8MM` or Darwin `hw.optional.arm.FEAT_I8MM` is present | +| `VT_CPU_QUANT_REPACK` | `auto` | Select the Arm i8mm+DotProd Q8_0 repack tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless the exact Linux HWCAP/HWCAP2 or Darwin sysctl capabilities are present | | `VT_DFLASH_PAGED` | on (CUDA, DFlash spec-decode) | The materialized `[context;block]` draft forward instead of the fixed-capacity paged draft-KV store read through `vt::DFlashPagedBlockAttention` (bit-identical; only the DFlash single-request propose path) | | `VT_DFLASH_GRAPH` | on (CUDA, DFlash spec-decode) | The eager paged draft step instead of the captured/replayed draft-step CUDA graph (replayed==eager bit-identical; only the DFlash single-request propose path) | | `VT_DFLASH_ATTN_BLOCK` | off (CUDA, DFlash spec-decode) | `=1` selects the D12/D13 block-per-(query,head) draft paged-attention kernel instead of the D14 default warp-scoped online-softmax kernel (same f32-softmax math within the bf16 envelope; the D14 warp kernel is ~3x faster and closed the ~2% speed residual; only the DFlash single-request propose path) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 201b368f8..8dc9453b7 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -26,7 +26,7 @@ are our reading of their documented behavior, not measurements. | Weight formats | Safetensors + GGUF | Safetensors | Safetensors | GGUF | | Correctness gate | token-exact vs vLLM | reference | own | own | | Architectures | 30 registered, 25+ gated | 130+ | 100+ | 100+ | -| Downloadable server binaries | ◐ W5 mutation-gated manifest contract implemented; no archives published | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries | +| Downloadable server binaries | ◐ eight-tuple CPU/CUDA/Vulkan/Metal/MLX release pipeline implemented in #196; full hosted dry run and first binary-bearing tag pending | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries | ## Serving and scheduling diff --git a/docs/RELEASES.md b/docs/RELEASES.md new file mode 100644 index 000000000..71571b5d6 --- /dev/null +++ b/docs/RELEASES.md @@ -0,0 +1,68 @@ +# Binary releases + +Tagged releases are built from one immutable source commit. The release workflow +builds each host/backend tuple independently, validates the freshly extracted +archive, assembles a byte-bound handoff, generates the indexes from the embedded +manifests, attests the verified archives, and only then publishes them. + +No release is created by pull requests or manual workflow runs. A tag must equal +the project version as `v`, the release matrix must be marked ready, and +the protected `release` environment must approve publication. + +## Primary downloads + +| Artifact | Channel | Contents and runtime boundary | +|---|---|---| +| `linux-x86_64-glibc-cpu` | stable | One conservative-baseline adaptive x86-64 binary with portable/SSE2, F16C, AVX2, and AVX-512 tiers | +| `linux-aarch64-glibc-cpu` | stable | One adaptive arm64 binary with portable/NEON and independently gated DotProd/i8mm kernels | +| `linux-x86_64-musl-cpu-static` | experimental-preview | Literal-static, CPU-only feasibility bundle; not a glibc replacement | +| `linux-x86_64-glibc-cuda-fat` | preview | One x86-64 binary containing all ten supported SMs and six exact-SM Triton AOT trees | +| `linux-aarch64-glibc-cuda-fat` | preview | The same complete CUDA architecture inventory for the arm64 host ABI | +| `linux-x86_64-glibc-vulkan` | preview | Vulkan bundle; the loader, ICD, and device driver remain host dependencies | +| `macos-arm64-metal` | stable | Native Apple Silicon Metal bundle; system frameworks remain host dependencies | +| `macos-arm64-metal-mlx` | preview | Metal plus the exact redistributable MLX dylib/metallib and license | + +The CUDA fat archives contain `sm_80`, `sm_86`, `sm_87`, `sm_89`, `sm_90a`, +`sm_100a`, `sm_103a`, `sm_110`, `sm_120a`, and `sm_121a`. NVIDIA's kernel and +driver ABI are never bundled. Optional per-SM diagnostic builds are not primary +downloads and cannot substitute for either fat-archive gate. + +Models, tokenizer data, certificates, GPU drivers, Python, PyTorch, compilers, +and source trees are not included. Text serving does not require `ffmpeg`; video +features require a compatible `ffmpeg` executable on `PATH` or an explicit +`--video-ffmpeg` path. + +## Verify a download + +Each archive is published with a `.sha256` checksum and a +`.provenance.json` sidecar. The generated `release-index.json` and +`RELEASE_INDEX.md` enumerate the exact published bytes, channel, host ABI, +compiled CPU tiers or CUDA SMs, runtime boundary, and known limitations. + +```sh +sha256sum --check linux-x86_64-glibc-cpu.tar.gz.sha256 +tar -xzf linux-x86_64-glibc-cpu.tar.gz +./bin/vllm-server --version +./bin/vllm-server --help +``` + +The archive also contains `release-manifest.json`, `VERSION`, an SPDX JSON SBOM, +third-party notices, and required redistributed licenses. + +## Retention + +Intermediate GitHub Actions artifacts are retained for seven days. Published +GitHub release assets remain available until a maintainer explicitly deletes +the release or its assets. The same policy is machine-readable in +`release/release-matrix.json` and copied into every generated release index. + +## Maintainer flow + +1. Set the project version and ensure `release/release-matrix.json` remains + publish-ready. +2. Run a manual `release` workflow on the intended commit. It is a non-publishing + full-matrix dry run. +3. Create and push the exact `v` tag only after the dry run is green. +4. Approve the protected `release` environment after build, verification, and + attestation succeed. Publication enumerates only files authenticated by the + verified handoff; shell globs and older workflow artifacts are rejected. diff --git a/docs/STATUS.md b/docs/STATUS.md index 5be0009f1..8324b55da 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -36,10 +36,10 @@ Startup-latency axis (2026-08-07): `MEASURED / provisional`. Cold launch to firs 6 legs contended, uncontended repeat died with a host reboot. [Detail](../.agents/specs/startup-latency-axis.md). -Container images (2026-08-08): `INVENTORIED`, nothing built. `ENG-RELEASE-CONTAINERS` records GHCR images -published by GitHub Actions - lanes `-cuda`/`-vulkan`/`-cpu` plus moving `latest-cuda`/`latest-vulkan`/`latest-cpu`, -each an amd64+arm64 manifest. The image is the `ENG-RELEASE-BINARIES` staged bundle, which does not exist yet, so -no Dockerfile, workflow or registry package is claimed. Metal/MLX are NOT-CONTAINERIZABLE. Issue #170. +Binary releases: required W1-W11/W13 are implemented in draft #196; local CPU, +Vulkan, archive, metadata and mutation gates are green. The hosted eight-tuple +dry run and tagged publication remain pending, so no binary is published. W12 +per-SM diagnostics remain optional. See [RELEASES](RELEASES.md). Protocol (2026-08-09): `776c56f1` has 157 imports = 3,231,342 exact bytes; append preserved prior 156 wrappers/rows. Archive/new raw-row mutation guards bind. @@ -123,7 +123,7 @@ token-for-token correctness against the pinned oracle. | Tool-call parsing | 36 parser families / 40 accepted names, streaming | Every vLLM tool parser at the pin except the three Rust/Harmony-backed ones: pure-text parsers ported 1:1, the six engine-backed families reimplemented from their wire formats, all held to the upstream test suites. Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md | | Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 9 parsers, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3 - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. Coverage: 9 of upstream's ~28 registered names (remaining text families + engine-backed adapters tracked as W2/W3 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case | | Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md | -| OpenAI server | Subset; #141 W5 19/19 ACTIVE ART∅ SIZE✓ | Completion/chat (streaming SSE), models, health/version/ping, metrics, tokenize/detokenize, tokenizer/server info, prefix-cache reset, abort, and Sora-shaped video creation/content. Tokenizer info and abort are flag-gated; metrics and cache reset lack live async backing. Details: docs/USAGE.md | +| OpenAI server | Subset; #196 binary pipeline implemented; no published binaries | Completion/chat (SSE), models, health/version/ping, metrics, tokenize/detokenize, tokenizer/server info, prefix-cache reset, abort, and Sora-shaped video creation/content. Tokenizer info and abort are flag-gated; metrics and cache reset lack live async backing. Details: docs/USAGE.md | | Pooling task class (embeddings / classify / score / rerank) | **EMBEDDINGS LIVE ON THE ONE SURFACE (ROW 6)**: `LlamaModel` registered, `PoolingRunner` in the engine step, `vllm_embed` (ABI v15) + live `/v1/embeddings`; classify/score/rerank engine-side only | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** (`EmbeddingPoolerHead`, `ClassifierPoolerHead`, `SequencePooler` + factories, `DispatchPooler` routing, `PoolerConfig`/`PoolingParams`; `test_pooler_heads` 27/27-240, RED-first). **W3 landed the pooling RUNNER path** (`PoolingRunner`: pooled embeddings instead of sampled tokens, structural cosine gate vs an f64 LAST+normalize reference, `test_pooling_runner` 5/5-14, RED-first). **ROW 6 (2026-08-08): embeddings LIVE** — fold gate `test_llama_embedding_fold` 4/4-231 (engine path == direct registry path, f64 LAST+normalize ref, chunked is_valid arm); residuals: REAL checkpoint + `LLM(task="embed")` oracle cosine (no number fabricated), score/rerank/classify endpoints, matryoshka/base64/token-array inputs, tokwise (W5). Detail: `.agents/specs/embeddings-one-surface.md` | | Plugin system (out-of-core registration) | Spiked; first CPU brick landed, not yet wired into any production path | The extensibility-first discovery layer. W0 spike over vLLM's plugin surface (general / platform / io_processor / endpoint groups, the `register_model` an out-of-tree plugin calls, the invocation seams) is committed (`.agents/specs/plugin-system.md`, `ENG-PLUGIN-SYSTEM` ACTIVE, `CLAIM-PLUGIN-SYSTEM`). W1 landed `vllm::plugins::LoadGeneralPlugins()` + the out-of-core general-plugin registration seam (`RegisterGeneralPlugin` / `REGISTER_VLLM_GENERAL_PLUGIN`) over the existing `REGISTER_VLLM_MODEL`-style registries (the in-tree factory `MODEL-FACTORY-registry` is record-repaired `DONE` 2026-08-05: 28 self-registering TUs, dgx debt paid by the 2026-07-23 seven-gate run): a 1:1 mirror of `load_general_plugins` (load-once idempotence, the `VLLM_PLUGINS` allowlist, per-plugin failure isolation). Proven by an out-of-core toy-model plugin that registers a toy architecture through the public `RegisterModel` seam — unit-gated RED-first (`test_plugin_system` 1 case / 29 assertions: the toy arch resolves ONLY after LoadGeneralPlugins runs it, and not under `VLLM_PLUGINS=""`). Python entry points have no C++20 analogue, so discovery is the project's static-init/`dlopen` registration idiom (recorded porting-inventory §9). NOT yet wired: real shared-object `dlopen` + the C-ABI `vllm_plugin_register` entry (W2), the engine/CLI `--load-plugins` wiring that calls LoadGeneralPlugins from the construction paths (W3), the platform/quant plugin kinds (W4), and the io_processor/stat_logger/endpoint groups (W5) are named residuals. See docs/BENCHMARKS.md | | Offline Batch API (JSONL file runner) | Spiked; first CPU brick landed, not yet exposed as a CLI | The offline OpenAI Batch API: read a JSONL of OpenAI-format requests, run each through the engine, write a JSONL of responses. W0 spike over vLLM's `run_batch.py` (schema, endpoint dispatch, run loop, file I/O) is committed (`.agents/specs/batch-api.md`, `SERVE-BATCH-API` ACTIVE, `CLAIM-BATCH-API`). W1 landed `RunBatch` (`RunLine`/`RunLines`/`Run`) + `RunBatchFile` — a pure orchestrator over the existing `OpenAIServingChat::create_chat_completion` (NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map: `/v1/chat/completions` dispatch, the `BatchResponseData`/`BatchRequestOutput` schema (`vllm-` ids, custom_id echoed), the `run_request` AllResponse/ErrorResponse/stream branches, and the unsupported-endpoint/url error rows. Unit-gated RED-first (`test_openai_run_batch` 7 cases / 80 assertions over the synthetic serving engine: ordered rows + custom_id echo + per-line BatchRequestOutput schema round-trip, a malformed line isolated into an error row so the batch continues, dispatch + 404 error rows; dropping the custom_id echo fails 9 assertions). Recorded deviation: a malformed line is isolated (batch continues) where upstream aborts the job. NOT yet exposed: the `vllm run-batch` CLI + `BatchFrontendArgs` (W2), embeddings/score/rerank dispatch (W3, rides pooling endpoints), audio transcription/translation + media fetch (W4), and http(s)/data-URL file I/O + metrics + overlapped `AsyncLLM` submission (W5) are named residuals. See docs/BENCHMARKS.md | diff --git a/docs/USAGE.md b/docs/USAGE.md index fbeb3e3b0..06a2a0e00 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -95,14 +95,132 @@ running. Any other dtype fails at load with a message naming what it saw. ## OpenAI-compatible server -`server` is a small HTTP server speaking the OpenAI API. Source: +`vllm-server` is a small HTTP server speaking the OpenAI API. Source: [`examples/server/main.cpp`](../examples/server/main.cpp) and [`src/vllm/entrypoints/openai/`](../src/vllm/entrypoints/openai/). ```sh -build/examples/server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 +build/examples/vllm-server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 ``` +The install component and deterministic archive target both stage from install +rules rather than copying the build tree: + +```sh +cmake --build build --target vllm-server-stage +cmake --build build --target vllm-server-archive +build/release/stage/bin/vllm-server --help +``` + +The basic CMake archive under `build/release/` includes the version, configured +backend, OS, and host architecture in its name. It is a developer package. The +release workflow separately produces host-ABI-specific archives with a +manifest, `VERSION`, SPDX SBOM, notices, licenses, and detached checksum and +provenance sidecars; no release download is claimed until that workflow has +completed on a release tag. + +To reproduce the W1 heterogeneous CUDA archive candidate, configure the exact +release architecture set. Portable translation units compile for all ten SMs; +architecture-specific kernels compile only for their supported intersection: + +```sh +cmake -S . -B build-cuda-fat -G Ninja \ + -DVLLM_CPP_CUDA=ON \ + -DVLLM_CPP_CUDA_ARCHITECTURES='80;86;87;89;90a;100a;103a;110;120a;121a' \ + -DVLLM_CPP_CUTLASS_FETCH=ON -DVLLM_CPP_TRITON=OFF +cmake --build build-cuda-fat --target vllm +python3 scripts/check-cuda-fat-gencode.py \ + --compile-commands build-cuda-fat/compile_commands.json \ + --library build-cuda-fat/libvllm.a +``` + +The release workflow applies this audit to independently linked x86_64 and +arm64 host executables, packages each as a preview `cuda-fat` archive, and then +runs the extracted-archive validator. Each archive must contain all ten SM +images and the six available exact-SM Triton AOT namespaces; the manifest keeps +runtime evidence separate per SM. These build-only preview candidates are not +a downloadable release claim until the tagged workflow publishes them. + +The complete primary download matrix and its runtime boundaries are documented +in [RELEASES.md](RELEASES.md). A manual workflow dispatch runs all eight tuples +without publication. An exact version tag runs the same build, produces +`release-index.json` and `RELEASE_INDEX.md` from the verified archive manifests, +attests the archive bytes, and publishes every archive/checksum/provenance +triplet through the protected release environment. + +### Selecting an x86 CPU ISA tier + +The x86_64 CPU library is one adaptive binary: portable, SSE2, +SSE2+F16C, AVX2, and AVX-512 elementwise matmul kernels are isolated in their +own translation units and selected only after CPUID plus the required XCR0 OS +state are checked. Leave `VT_CPU_MATMUL_TIER` unset for automatic selection, or +set it to `portable`, `sse2`, `sse2+f16c`, `avx2`, or `avx512` for a same-binary +correctness/performance check. A forced tier that the current CPU or OS cannot +execute fails closed instead of silently narrowing or risking an illegal +instruction. Release builds never use `-march=native`. + +On arm64, leave the same variable unset to select between portable and NEON +elementwise matmul, or force `portable`/`neon`. DotProd and i8mm kernels are +independently selectable with `VT_CPU_Q8_DOT`, `VT_CPU_QUANT_MMLA`, and +`VT_CPU_QUANT_REPACK`; `auto` uses Linux HWCAP/HWCAP2 or Darwin feature sysctls, +while an unavailable forced tier fails closed. The exact accepted values are +listed in [ENVIRONMENT.md](ENVIRONMENT.md). + +### Validating a staged release archive + +Release verification reads only a freshly extracted archive, never files from +the build tree. Pass the archive together with its final-byte SHA256 and SLSA +provenance sidecars: + +```sh +python3 scripts/validate-release-archive.py \ + --archive vllm.cpp-0.0.1-cpu-linux-x86_64.tar.gz \ + --checksum vllm.cpp-0.0.1-cpu-linux-x86_64.tar.gz.sha256 \ + --provenance vllm.cpp-0.0.1-cpu-linux-x86_64.tar.gz.provenance.json \ + --forbid-path "$PWD/build" +``` + +The validator checks the content allowlist, executable and host ABI, manifest, +`VERSION`, SPDX SBOM, licenses, ELF dependencies and RPATH/RUNPATH, extracted +`--help`/`--version` smokes, and backend-specific CUDA or adaptive-CPU claims. +The digest and provenance are sidecars because both describe the final archive +bytes; placing either inside those bytes would create a self-reference. + +The CPU release helper is the reproducible entry point used by CI. It requires +an explicit artifact tuple, architecture, channel, build directory, libc ABI, +and a QEMU userspace emulator. The gate executes every compiled tier under a +feature-rich CPU model, then executes the baseline and proves rich-tier refusal +under a feature-poor model before metadata can be generated: + +```sh +SOURCE_SHA=$(git rev-parse HEAD) \ +VERSION=0.0.1 \ +SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) \ +EVIDENCE_URL=https://github.com/mudler/vllm.cpp/actions/runs/EXAMPLE \ +scripts/build-cpu-release.sh \ + linux-x86_64-glibc-cpu x86_64 stable build-release-cpu-x86 \ + 2.39 /usr/bin/qemu-x86_64 +``` + +The corresponding arm64 tuple is `linux-aarch64-glibc-cpu`. The only literal +static tuple is the CPU-only `linux-x86_64-musl-cpu-static` experiment; normal +CPU and accelerator archives are static-core bundles with audited host runtime +dependencies. + +To exercise the release pipeline without publishing anything, trigger its +manual entry point: + +```sh +gh workflow run release.yml --ref main +``` + +Manual runs are always dry runs. Publication additionally requires an exact +`v` tag, a release matrix whose required lanes are all marked +ready, successful verification and attestation jobs, and approval of the +protected `release` environment. Build and verification jobs have read-only +repository permissions; only attestation receives OIDC authority, and only the +final protected job receives `contents: write`. + Any OpenAI client works by pointing its `base_url` at it: ```python diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 92275e965..075636842 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -68,11 +68,90 @@ endif() if(VLLM_CPP_SERVER) add_executable(server server/main.cpp) # ARCH-ONE-SURFACE: server/main.cpp is a THIN CLIENT of the public C ABI - # (vllm_server_main, v17) -- it includes vllm.h and nothing else, so it links the - # PACKAGED shared library exactly as an out-of-tree consumer would, not the - # internal C++ target. - target_link_libraries(server PRIVATE vllm::shared) + # (vllm_server_main, v17) -- it includes vllm.h and nothing else. The release + # executable links the static core below, but reaches it only through that same + # public surface. + # Release bundles carry one static-core server. The executable remains a + # thin C-ABI client; only the link form changes so an extracted archive does + # not depend on a separately installed libvllm. + target_link_libraries(server PRIVATE vllm::vllm) + if(VLLM_CPP_LITERAL_STATIC) + if(NOT CMAKE_SYSTEM_NAME STREQUAL "Linux" OR VLLM_CPP_CUDA OR + VLLM_CPP_METAL OR VLLM_CPP_MLX OR VLLM_CPP_VULKAN OR VLLM_CPP_HIP) + message(FATAL_ERROR + "VLLM_CPP_LITERAL_STATIC is restricted to the experimental Linux CPU-only lane") + endif() + target_link_options(server PRIVATE -static -static-libgcc -static-libstdc++) + endif() + set_target_properties(server PROPERTIES OUTPUT_NAME vllm-server) + if(VLLM_CPP_MLX) + set_target_properties(server PROPERTIES + BUILD_RPATH "${MLX_ROOT}/lib" + INSTALL_RPATH "@loader_path/../lib") + find_file(_vllm_mlx_metallib mlx.metallib + PATHS "${MLX_ROOT}/lib" "${MLX_ROOT}" NO_DEFAULT_PATH) + if(NOT _vllm_mlx_metallib) + message(FATAL_ERROR + "MLX release packaging requires mlx.metallib under MLX_ROOT") + endif() + install(FILES "${MLX_LIBRARY}" "${_vllm_mlx_metallib}" + DESTINATION ${CMAKE_INSTALL_LIBDIR} + COMPONENT vllm-server) + endif() vllm_cpp_set_warnings(server) + install(TARGETS server + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} + COMPONENT vllm-server) + + find_package(Python3 COMPONENTS Interpreter REQUIRED) + set(_vllm_server_backends) + if(VLLM_CPP_CUDA) + list(APPEND _vllm_server_backends cuda) + endif() + if(VLLM_CPP_METAL) + if(VLLM_CPP_MLX) + list(APPEND _vllm_server_backends metal-mlx) + else() + list(APPEND _vllm_server_backends metal) + endif() + endif() + if(VLLM_CPP_VULKAN) + list(APPEND _vllm_server_backends vulkan) + endif() + if(VLLM_CPP_HIP) + list(APPEND _vllm_server_backends rocm) + endif() + if(NOT _vllm_server_backends) + list(APPEND _vllm_server_backends cpu) + endif() + string(JOIN "-" _vllm_server_backend ${_vllm_server_backends}) + string(TOLOWER "${CMAKE_SYSTEM_NAME}" _vllm_server_os) + string(TOLOWER "${CMAKE_SYSTEM_PROCESSOR}" _vllm_server_arch) + set(_vllm_server_release_dir "${CMAKE_BINARY_DIR}/release") + set(_vllm_server_stage_dir "${_vllm_server_release_dir}/stage") + set(_vllm_server_archive + "${_vllm_server_release_dir}/vllm.cpp-${PROJECT_VERSION}-${_vllm_server_backend}-${_vllm_server_os}-${_vllm_server_arch}.tar.gz") + + add_custom_target(vllm-server-stage + COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/scripts/package-server.py + --build-dir ${CMAKE_BINARY_DIR} + --stage-dir ${_vllm_server_stage_dir} + --config $ + DEPENDS server ${CMAKE_SOURCE_DIR}/scripts/package-server.py + COMMENT "Installing the vllm-server component into an empty staging prefix" + VERBATIM) + + add_custom_target(vllm-server-archive + COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/scripts/package-server.py + --build-dir ${CMAKE_BINARY_DIR} + --stage-dir ${_vllm_server_stage_dir} + --archive ${_vllm_server_archive} + --config $ + DEPENDS server ${CMAKE_SOURCE_DIR}/scripts/package-server.py + BYPRODUCTS ${_vllm_server_archive} + COMMENT "Creating deterministic installed vllm-server archive" + VERBATIM) + if(VLLM_CPP_BUILD_TESTS) add_test(NAME test_server_help COMMAND $ --help) set_tests_properties(test_server_help PROPERTIES diff --git a/include/vt/quant.h b/include/vt/quant.h index 15906f05d..198e035b9 100644 --- a/include/vt/quant.h +++ b/include/vt/quant.h @@ -94,15 +94,15 @@ bool QuantQ8A76AsmActive(); // process runs on i8mm-capable aarch64 (compile-time `__ARM_FEATURE_MATMUL_INT8` // AND runtime `HWCAP2_I8MM`) AND the dtype is one of the four encodings upstream // gives an mmla path (Q8_0, Q4_0, Q4_K, Q6_K). Returns nullptr everywhere else — -// on any other CPU, when `VT_CPU_QUANT_MMLA=0`, and for q3_K/q5_K (no upstream +// on any other CPU, when `VT_CPU_QUANT_MMLA=portable`, and for q3_K/q5_K (no upstream // mmla) — so the caller falls back to the portable nrc==1 tier. A returned // kernel produces a 2x2 output tile: it MUST be called with nrc==2, two // consecutive weight rows (stride bx) and two consecutive activation rows // (stride by), writing s[0]=(w0,a0), s[1]=(w1,a0), s[bs]=(w0,a1), s[bs+1]=(w1,a1). VecDotFn QuantMmlaVecDot(DType dtype); -// True when the Arm i8mm mmla tier is live in this process (i8mm probed present -// and not defeated by VT_CPU_QUANT_MMLA). Always false off i8mm-capable aarch64. +// True when the Arm i8mm mmla tier is live in this process. A forced unsupported +// tier fails closed; auto uses the shared Linux HWCAP/Darwin sysctl probe. bool QuantMmlaActive(); // Bytes one quantized ACTIVATION row occupies for a given weight dtype, i.e. @@ -144,8 +144,8 @@ bool HasQuantDotKernel(DType dtype); // kMatmulBTQuant is q8_0); the k-quants keep the mmla tier. // True when the i8mm repack tier is LIVE in this process: compiled for aarch64 -// with i8mm, `HWCAP2_I8MM` probed present, and not disabled by -// `VT_CPU_QUANT_REPACK=0|off|false`. Always false off i8mm-capable aarch64, so +// with i8mm, the exact HWCAP/sysctl bits probed present, and not forced to +// `VT_CPU_QUANT_REPACK=portable`. Always false off i8mm-capable aarch64, so // the loader never repacks and every consumer keeps the portable/mmla path. bool QuantRepackActive(); diff --git a/release/manifest-v1.schema.json b/release/manifest-v1.schema.json index 8450bfa3c..6d32b6cfd 100644 --- a/release/manifest-v1.schema.json +++ b/release/manifest-v1.schema.json @@ -59,10 +59,11 @@ "artifact": { "type": "object", "additionalProperties": false, - "required": ["id", "version", "channel", "kind", "static_boundary"], + "required": ["id", "version", "c_abi_version", "channel", "kind", "static_boundary"], "properties": { "id": {"type": "string", "minLength": 1}, "version": {"type": "string", "minLength": 1}, + "c_abi_version": {"type": "integer"}, "channel": { "enum": ["stable", "preview", "experimental-preview"] }, @@ -92,6 +93,7 @@ "VLLM_CPP_CUDA_ARCHITECTURES", "VLLM_CPP_HIP", "VLLM_CPP_HIP_ARCHITECTURES", + "VLLM_CPP_LITERAL_STATIC", "VLLM_CPP_METAL", "VLLM_CPP_MLX", "VLLM_CPP_SERVER", @@ -114,6 +116,7 @@ "items": {"type": "string", "minLength": 1}, "uniqueItems": true }, + "VLLM_CPP_LITERAL_STATIC": {"type": "boolean"}, "VLLM_CPP_METAL": {"type": "boolean"}, "VLLM_CPP_MLX": {"type": "boolean"}, "VLLM_CPP_SERVER": {"type": "boolean"}, diff --git a/release/release-matrix.json b/release/release-matrix.json new file mode 100644 index 000000000..a30d3660b --- /dev/null +++ b/release/release-matrix.json @@ -0,0 +1,18 @@ +{ + "artifacts": [ + {"channel": "stable", "id": "linux-x86_64-glibc-cpu", "required": true}, + {"channel": "stable", "id": "linux-aarch64-glibc-cpu", "required": true}, + {"channel": "experimental-preview", "id": "linux-x86_64-musl-cpu-static", "required": true}, + {"channel": "preview", "id": "linux-x86_64-glibc-cuda-fat", "required": true}, + {"channel": "preview", "id": "linux-aarch64-glibc-cuda-fat", "required": true}, + {"channel": "stable", "id": "macos-arm64-metal", "required": true}, + {"channel": "preview", "id": "macos-arm64-metal-mlx", "required": true}, + {"channel": "preview", "id": "linux-x86_64-glibc-vulkan", "required": true} + ], + "release_ready": true, + "retention": { + "ci_artifacts_days": 7, + "github_release": "maintainer-deletion-only" + }, + "schema": "vllm.cpp.release-matrix.v1" +} diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 438926ab5..4074e0dc1 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -58,6 +58,7 @@ CHECKERS=( check-prompt-contract check-agent-record check-release-binary-contract + check-release-workflow check-role-discipline claim-view check-readme-structure @@ -79,6 +80,13 @@ SUITES=( test_agent_record test_check_release_binary_contract test_release_manifest + test_release_archive + test_release_pipeline + test_release_index + test_release_metadata + test_release_accelerator_metadata + test_release_macos_metadata + test_cpu_release_gates test_agent_role test_agent_onboard test_agent_start diff --git a/scripts/build-cpu-release.sh b/scripts/build-cpu-release.sh new file mode 100755 index 000000000..c87584727 --- /dev/null +++ b/scripts/build-cpu-release.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ $# -ne 6 ]]; then + echo "usage: $0 ARTIFACT_ID ARCH CHANNEL BUILD_DIR ABI_VERSION POOR_EMULATOR" >&2 + exit 2 +fi + +artifact_id=$1 +arch=$2 +channel=$3 +build_dir=$4 +abi_version=$5 +poor_emulator=$6 +: "${SOURCE_SHA:?SOURCE_SHA is required}" +: "${VERSION:?VERSION is required}" +: "${EVIDENCE_URL:?EVIDENCE_URL is required}" +: "${SOURCE_DATE_EPOCH:?SOURCE_DATE_EPOCH is required}" + +literal_static=OFF +if [[ "$artifact_id" == linux-x86_64-musl-cpu-static ]]; then + literal_static=ON +fi + +cmake -S . -B "$build_dir" -G Ninja \ + -DVLLM_CPP_BUILD_TESTS=ON \ + -DVLLM_CPP_BUILD_EXAMPLES=ON \ + -DVLLM_CPP_SERVER=ON \ + -DVLLM_CPP_CUDA=OFF \ + -DVLLM_CPP_CUDA_ARCHITECTURES= \ + -DVLLM_CPP_HIP=OFF \ + -DVLLM_CPP_HIP_ARCHITECTURES= \ + -DVLLM_CPP_LITERAL_STATIC="$literal_static" \ + -DVLLM_CPP_METAL=OFF \ + -DVLLM_CPP_MLX=OFF \ + -DMLX_ROOT= \ + -DVLLM_CPP_TRITON=OFF \ + -DVLLM_CPP_VULKAN=OFF \ + -DCMAKE_BUILD_TYPE=Release + +targets=(server test_ops_matmul_elem) +if [[ "$arch" == aarch64 ]]; then + targets+=(test_cpu_isa_arm test_ops_quant_dot test_ops_quant_repack) +fi +cmake --build "$build_dir" --target "${targets[@]}" -j 2 + +release_dir="$build_dir/release" +tier_report="$release_dir/cpu-tier-report.json" +stage_dir="$release_dir/stage" +metadata_dir="$release_dir/metadata" +archive="$release_dir/$artifact_id.tar.gz" +mkdir -p "$release_dir" + +python3 scripts/run-cpu-release-gates.py \ + --arch "$arch" \ + --tests-dir "$build_dir/tests" \ + --poor-emulator "$poor_emulator" \ + --rich-emulator "$poor_emulator" \ + --rich-cpu max \ + --output "$tier_report" \ + --evidence-url "$EVIDENCE_URL" + +python3 scripts/package-server.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" + +compiler=$(c++ --version | head -n 1) +toolchain="$(cmake --version | head -n 1); $(ninja --version)" +c_abi_version=$(sed -n 's/^#define VLLM_ABI_VERSION \([0-9][0-9]*\)$/\1/p' include/vllm.h) +if [[ -z "$c_abi_version" ]]; then + echo "could not resolve VLLM_ABI_VERSION" >&2 + exit 1 +fi + +python3 scripts/release_metadata.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --output-dir "$metadata_dir" \ + --tier-report "$tier_report" \ + --artifact-id "$artifact_id" \ + --channel "$channel" \ + --version "$VERSION" \ + --c-abi-version "$c_abi_version" \ + --source-commit "$SOURCE_SHA" \ + --source-clean \ + --abi-version "$abi_version" \ + --compiler "$compiler" \ + --toolchain "$toolchain" \ + --evidence-url "$EVIDENCE_URL" + +python3 scripts/package-server.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --metadata-dir "$metadata_dir" \ + --archive "$archive" + +python3 scripts/validate-release-archive.py \ + --archive "$archive" \ + --checksum "$archive.sha256" \ + --provenance "$archive.provenance.json" \ + --repo-root . diff --git a/scripts/build-linux-accelerator-release.sh b/scripts/build-linux-accelerator-release.sh new file mode 100755 index 000000000..5052e1aec --- /dev/null +++ b/scripts/build-linux-accelerator-release.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ $# -ne 3 ]]; then + echo "usage: $0 ARTIFACT_ID BACKEND BUILD_DIR" >&2 + exit 2 +fi + +artifact_id=$1 +backend=$2 +build_dir=$3 +: "${SOURCE_SHA:?SOURCE_SHA is required}" +: "${VERSION:?VERSION is required}" +: "${EVIDENCE_URL:?EVIDENCE_URL is required}" +: "${SOURCE_DATE_EPOCH:?SOURCE_DATE_EPOCH is required}" + +cuda=OFF +triton=OFF +vulkan=OFF +cuda_architectures= +if [[ "$backend" == cuda ]]; then + cuda=ON + triton=ON + cuda_architectures='80;86;87;89;90a;100a;103a;110;120a;121a' +elif [[ "$backend" == vulkan ]]; then + vulkan=ON +else + echo "unsupported Linux accelerator backend: $backend" >&2 + exit 2 +fi + +cmake -S . -B "$build_dir" -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + -DVLLM_CPP_BUILD_TESTS=ON \ + -DVLLM_CPP_BUILD_EXAMPLES=ON \ + -DVLLM_CPP_SERVER=ON \ + -DVLLM_CPP_CUDA="$cuda" \ + -DVLLM_CPP_CUDA_ARCHITECTURES="$cuda_architectures" \ + -DVLLM_CPP_CUTLASS_FETCH=ON \ + -DVLLM_CPP_HIP=OFF \ + -DVLLM_CPP_HIP_ARCHITECTURES= \ + -DVLLM_CPP_LITERAL_STATIC=OFF \ + -DVLLM_CPP_METAL=OFF \ + -DVLLM_CPP_MLX=OFF \ + -DMLX_ROOT= \ + -DVLLM_CPP_TRITON="$triton" \ + -DVLLM_CPP_VULKAN="$vulkan" + +targets=(server) +if [[ "$backend" == vulkan ]]; then + targets+=(test_vulkan_backend test_backend_cross_device) +fi +cmake --build "$build_dir" --target "${targets[@]}" -j 2 + +if [[ "$backend" == cuda ]]; then + python3 scripts/check-cuda-fat-gencode.py \ + --compile-commands "$build_dir/compile_commands.json" \ + --library "$build_dir/libvllm.a" + python3 scripts/check-triton-aot-multiarch.py \ + --vendored-root src/vt/cuda/triton_aot_vendored \ + --library "$build_dir/libvllm.a" +else + "$build_dir/tests/test_vulkan_backend" + "$build_dir/tests/test_backend_cross_device" +fi + +release_dir="$build_dir/release" +stage_dir="$release_dir/stage" +metadata_dir="$release_dir/metadata" +archive="$release_dir/$artifact_id.tar.gz" +mkdir -p "$release_dir" +python3 scripts/package-server.py --build-dir "$build_dir" --stage-dir "$stage_dir" + +compiler=$(c++ --version | head -n 1) +toolchain="$(cmake --version | head -n 1); $(ninja --version)" +c_abi_version=$(sed -n 's/^#define VLLM_ABI_VERSION \([0-9][0-9]*\)$/\1/p' include/vllm.h) +abi_version=$(getconf GNU_LIBC_VERSION | awk '{print $2}') +python3 scripts/release_accelerator_metadata.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --output-dir "$metadata_dir" \ + --artifact-id "$artifact_id" \ + --channel preview \ + --version "$VERSION" \ + --c-abi-version "$c_abi_version" \ + --source-commit "$SOURCE_SHA" \ + --source-clean \ + --abi-version "$abi_version" \ + --compiler "$compiler" \ + --toolchain "$toolchain" \ + --evidence-url "$EVIDENCE_URL" +python3 scripts/package-server.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --metadata-dir "$metadata_dir" \ + --archive "$archive" +python3 scripts/validate-release-archive.py \ + --archive "$archive" \ + --checksum "$archive.sha256" \ + --provenance "$archive.provenance.json" \ + --repo-root . \ + --forbid-path "$PWD/$build_dir" diff --git a/scripts/build-macos-release.sh b/scripts/build-macos-release.sh new file mode 100755 index 000000000..34759f6e6 --- /dev/null +++ b/scripts/build-macos-release.sh @@ -0,0 +1,85 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ $# -lt 3 || $# -gt 6 ]]; then + echo "usage: $0 ARTIFACT_ID CHANNEL BUILD_DIR [MLX_ROOT MLX_VERSION MLX_LICENSE]" >&2 + exit 2 +fi + +artifact_id=$1 +channel=$2 +build_dir=$3 +mlx_root=${4:-} +mlx_version=${5:-} +mlx_license=${6:-} +: "${SOURCE_SHA:?SOURCE_SHA is required}" +: "${VERSION:?VERSION is required}" +: "${EVIDENCE_URL:?EVIDENCE_URL is required}" +: "${SOURCE_DATE_EPOCH:?SOURCE_DATE_EPOCH is required}" + +mlx=OFF +mlx_metadata_args=() +if [[ "$artifact_id" == macos-arm64-metal-mlx ]]; then + mlx=ON + if [[ -z "$mlx_root" || -z "$mlx_version" || -z "$mlx_license" ]]; then + echo "MLX artifact requires an exact root, version, and license" >&2 + exit 2 + fi + mlx_metadata_args=(--mlx-license "$mlx_license") +fi + +cmake -S . -B "$build_dir" -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_BUILD_TESTS=ON \ + -DVLLM_CPP_BUILD_EXAMPLES=ON \ + -DVLLM_CPP_SERVER=ON \ + -DVLLM_CPP_CUDA=OFF \ + -DVLLM_CPP_CUDA_ARCHITECTURES= \ + -DVLLM_CPP_HIP=OFF \ + -DVLLM_CPP_HIP_ARCHITECTURES= \ + -DVLLM_CPP_LITERAL_STATIC=OFF \ + -DVLLM_CPP_METAL=ON \ + -DVLLM_CPP_MLX="$mlx" \ + -DMLX_ROOT="$mlx_root" \ + -DVLLM_CPP_TRITON=OFF \ + -DVLLM_CPP_VULKAN=OFF +cmake --build "$build_dir" --target server test_metal_backend -j 2 +"$build_dir/tests/test_metal_backend" + +release_dir="$build_dir/release" +stage_dir="$release_dir/stage" +metadata_dir="$release_dir/metadata" +archive="$release_dir/$artifact_id.tar.gz" +mkdir -p "$release_dir" +python3 scripts/package-server.py --build-dir "$build_dir" --stage-dir "$stage_dir" + +compiler=$(c++ --version | head -n 1) +toolchain="$(cmake --version | head -n 1); $(ninja --version)" +c_abi_version=$(sed -n 's/^#define VLLM_ABI_VERSION \([0-9][0-9]*\)$/\1/p' include/vllm.h) +python3 scripts/release_macos_metadata.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --output-dir "$metadata_dir" \ + --artifact-id "$artifact_id" \ + --channel "$channel" \ + --version "$VERSION" \ + --c-abi-version "$c_abi_version" \ + --source-commit "$SOURCE_SHA" \ + --source-clean \ + --abi-version "$(sw_vers -productVersion)" \ + --mlx-version "$mlx_version" \ + "${mlx_metadata_args[@]}" \ + --compiler "$compiler" \ + --toolchain "$toolchain" \ + --evidence-url "$EVIDENCE_URL" +python3 scripts/package-server.py \ + --build-dir "$build_dir" \ + --stage-dir "$stage_dir" \ + --metadata-dir "$metadata_dir" \ + --archive "$archive" +python3 scripts/validate-release-archive.py \ + --archive "$archive" \ + --checksum "$archive.sha256" \ + --provenance "$archive.provenance.json" \ + --repo-root . \ + --forbid-path "$PWD/$build_dir" diff --git a/scripts/check-arm-isa-build.py b/scripts/check-arm-isa-build.py new file mode 100755 index 000000000..7377daa57 --- /dev/null +++ b/scripts/check-arm-isa-build.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +"""Audit W4 aarch64 CPU tier isolation in CMake compile commands.""" + +from __future__ import annotations + +import argparse +import json +import shlex +import sys +from pathlib import Path + + +BASELINE_SOURCES = ( + "src/vt/cpu/cpu_isa_arm.cpp", + "src/vt/cpu/cpu_matmul_elem.cpp", + "src/vt/cpu/cpu_quant_dot.cpp", + "src/vt/cpu/cpu_quant_repack.cpp", +) +TIER_MARCH = { + "src/vt/cpu/cpu_quant_dot_sdot.cpp": "-march=armv8.2-a+dotprod+fp16", + "src/vt/cpu/cpu_quant_dot_arm.cpp": "-march=armv8.2-a+i8mm+dotprod", + "src/vt/cpu/cpu_quant_repack_arm.cpp": "-march=armv8.2-a+i8mm+dotprod", +} + + +def relative_source(value: str) -> str | None: + normalized = value.replace("\\", "/") + marker = "/src/" + if marker not in normalized: + return None + return "src/" + normalized.split(marker, 1)[1] + + +def command_arguments(entry: dict[str, object]) -> list[str]: + arguments = entry.get("arguments") + if isinstance(arguments, list): + return [str(value) for value in arguments] + return shlex.split(str(entry.get("command", ""))) + + +def march_flags(arguments: list[str]) -> list[str]: + return [argument for argument in arguments if argument.startswith("-march=")] + + +def validate_compile_commands(path: Path) -> list[str]: + entries = json.loads(path.read_text(encoding="utf-8")) + commands: dict[str, list[str]] = {} + errors: list[str] = [] + for entry in entries: + arguments = command_arguments(entry) + source = relative_source(str(entry.get("file", ""))) + label = source or str(entry.get("file", "")) + if "-march=native" in arguments: + errors.append(f"{label}: forbidden -march=native") + if source is not None: + commands[source] = arguments + + for source in BASELINE_SOURCES: + if source not in commands: + errors.append(f"missing Arm compile command for {source}") + continue + leaked = march_flags(commands[source]) + if leaked: + errors.append(f"{source}: baseline source leaks architecture flags {leaked}") + + for source, required in TIER_MARCH.items(): + if source not in commands: + errors.append(f"missing Arm compile command for {source}") + continue + actual = march_flags(commands[source]) + if actual != [required]: + errors.append(f"{source}: march flags {actual} != required {[required]}") + return errors + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--compile-commands", type=Path, required=True) + args = parser.parse_args() + + errors = validate_compile_commands(args.compile_commands) + if errors: + print("Arm ISA build audit FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("Arm ISA build audit: portable baseline and exact DotProd/i8mm tiers OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-cpu-isa-build.py b/scripts/check-cpu-isa-build.py new file mode 100755 index 000000000..8e574827c --- /dev/null +++ b/scripts/check-cpu-isa-build.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Audit W3 x86 CPU tier isolation in CMake compile commands.""" + +from __future__ import annotations + +import argparse +import json +import shlex +import sys +from pathlib import Path + + +BASELINE_SOURCES = ( + "src/vt/cpu/cpu_isa_x86.cpp", + "src/vt/cpu/cpu_matmul_elem.cpp", +) +TIER_FLAGS = { + "src/vt/cpu/cpu_matmul_elem_avx2.cpp": frozenset(("-mavx2", "-mf16c")), + "src/vt/cpu/cpu_matmul_elem_avx512.cpp": frozenset( + ("-mavx512f", "-mavx512bw", "-mavx512vl", "-mf16c") + ), +} + + +def relative_source(value: str) -> str | None: + normalized = value.replace("\\", "/") + marker = "/src/" + if marker not in normalized: + return None + return "src/" + normalized.split(marker, 1)[1] + + +def command_arguments(entry: dict[str, object]) -> list[str]: + arguments = entry.get("arguments") + if isinstance(arguments, list): + return [str(value) for value in arguments] + return shlex.split(str(entry.get("command", ""))) + + +def explicit_isa_flags(arguments: list[str]) -> frozenset[str]: + return frozenset( + argument + for argument in arguments + if argument.startswith("-mavx") or argument == "-mf16c" + ) + + +def validate_compile_commands(path: Path) -> list[str]: + entries = json.loads(path.read_text(encoding="utf-8")) + commands: dict[str, list[str]] = {} + errors: list[str] = [] + for entry in entries: + arguments = command_arguments(entry) + source = relative_source(str(entry.get("file", ""))) + label = source or str(entry.get("file", "")) + if "-march=native" in arguments: + errors.append(f"{label}: forbidden -march=native") + if source is not None: + commands[source] = arguments + + for source in BASELINE_SOURCES: + if source not in commands: + errors.append(f"missing CPU compile command for {source}") + continue + leaked = sorted(explicit_isa_flags(commands[source])) + if leaked: + errors.append(f"{source}: baseline dispatcher leaks ISA flags {leaked}") + + for source, required in TIER_FLAGS.items(): + if source not in commands: + errors.append(f"missing CPU compile command for {source}") + continue + actual = explicit_isa_flags(commands[source]) + missing = sorted(required.difference(actual)) + extra = sorted(actual.difference(required)) + if missing: + errors.append(f"{source}: missing required flags {missing}") + if extra: + errors.append(f"{source}: undeclared ISA flags {extra}") + return errors + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--compile-commands", type=Path, required=True) + args = parser.parse_args() + + errors = validate_compile_commands(args.compile_commands) + if errors: + print("CPU ISA build audit FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("CPU ISA build audit: portable baseline and exact x86 tiers OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-cuda-fat-gencode.py b/scripts/check-cuda-fat-gencode.py new file mode 100644 index 000000000..5e8688a95 --- /dev/null +++ b/scripts/check-cuda-fat-gencode.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +"""Audit W1 per-source CUDA gencode and the linked ten-SM archive.""" + +from __future__ import annotations + +import argparse +import json +import re +import subprocess +import sys +from pathlib import Path + + +ALL_SMS = ("80", "86", "87", "89", "90a", "100a", "103a", "110", "120a", "121a") +SM12X = ("120a", "121a") +FA2_SMS = ("80", "86", "87", "89", "120a", "121a") + +REQUIRED_SOURCES = ( + "src/vt/cuda/cuda_matmul_nvfp4.cu", + "src/vt/cuda/cuda_nvfp4_sm12x.cu", + "src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu", + "src/vt/cuda/cuda_matmul_fp8_cutlass.cu", + "src/vt/cuda/cuda_matmul_nvfp4_sm100.cu", + "src/vt/cuda/cuda_scaled_mm_c3x_sm90.cu", + "src/vt/cuda/cuda_scaled_mm_c3x_sm100.cu", + "src/vt/cuda/cuda_moe_marlin.cu", + "src/vt/cuda/cuda_flash_attn_fa2.cu", +) + +ARCH_RE = re.compile(r"arch=compute_([0-9]+[af]?),code=sm_([0-9]+[af]?)") +ARCHIVE_RE = re.compile(r"\bsm_([0-9]+[af]?)\b") +PROJECT_ROOT = Path(__file__).resolve().parent.parent + + +def relative_source(value: str, directory: str = "") -> str | None: + source = Path(value) + if not source.is_absolute(): + source = Path(directory) / source + try: + relative = source.resolve().relative_to(PROJECT_ROOT) + except ValueError: + return None + normalized = relative.as_posix() + return normalized if normalized.startswith("src/") else None + + +def expected_sms(source: str) -> tuple[str, ...]: + name = Path(source).name + if name == "cuda_matmul_nvfp4_sm100.cu": + return ("100a",) + if name == "cuda_scaled_mm_c3x_sm90.cu": + return ("90a",) + if name == "cuda_scaled_mm_c3x_sm100.cu": + return ("100a",) + if "flash_attn/" in source or name == "cuda_flash_attn_fa2.cu": + return FA2_SMS + if ( + name == "cuda_nvfp4_sm12x.cu" + or name == "cuda_matmul_nvfp4_cutlass.cu" + or name.startswith("cuda_nvfp4_tactics_") + or name == "cuda_matmul_fp8_cutlass.cu" + or name in { + "cuda_moe_marlin.cu", + "cuda_marlin_repack.cu", + "cuda_marlin_dense.cu", + } + or "/marlin/" in source + ): + return SM12X + return ALL_SMS + + +def command_sms(command: str) -> tuple[str, ...]: + found: list[str] = [] + for compute, code in ARCH_RE.findall(command): + if compute != code: + continue + if compute not in found: + found.append(compute) + return tuple(found) + + +def validate_compile_commands(path: Path) -> list[str]: + entries = json.loads(path.read_text(encoding="utf-8")) + commands: dict[str, str] = {} + for entry in entries: + source = relative_source( + str(entry.get("file", "")), str(entry.get("directory", "")) + ) + if source is None or not source.endswith(".cu"): + continue + command = entry.get("command") + if command is None: + command = " ".join(str(value) for value in entry.get("arguments", [])) + commands[source] = str(command) + + errors: list[str] = [] + for required in REQUIRED_SOURCES: + if required not in commands: + errors.append(f"missing CUDA compile command for {required}") + for source, command in sorted(commands.items()): + actual = command_sms(command) + expected = expected_sms(source) + if set(actual) != set(expected) or len(actual) != len(expected): + errors.append( + f"{source}: gencode {list(actual)} != expected {list(expected)}" + ) + return errors + + +def validate_archive_listing(listing: str) -> list[str]: + actual = set(ARCHIVE_RE.findall(listing)) + missing = [sm for sm in ALL_SMS if sm not in actual] + extra = sorted(actual.difference(ALL_SMS)) + errors: list[str] = [] + if missing: + errors.append(f"archive is missing CUDA SMs: {','.join(missing)}") + if extra: + errors.append(f"archive has undeclared CUDA SMs: {','.join(extra)}") + return errors + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--compile-commands", type=Path, required=True) + source = parser.add_mutually_exclusive_group(required=True) + source.add_argument("--library", type=Path) + source.add_argument("--cuobjdump-list", type=Path) + args = parser.parse_args() + + errors = validate_compile_commands(args.compile_commands) + if args.library is not None: + result = subprocess.run( + ["cuobjdump", "--list-elf", str(args.library)], + text=True, + capture_output=True, + check=False, + ) + if result.returncode != 0: + errors.append(f"cuobjdump failed: {result.stderr.strip()}") + listing = "" + else: + listing = result.stdout + else: + listing = args.cuobjdump_list.read_text(encoding="utf-8") + errors.extend(validate_archive_listing(listing)) + + if errors: + print("CUDA fat gencode audit FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("CUDA fat gencode audit: all ten SMs and per-source intersections OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-pr-size.py b/scripts/check-pr-size.py index 25132fb5e..1cda204d7 100755 --- a/scripts/check-pr-size.py +++ b/scripts/check-pr-size.py @@ -207,6 +207,7 @@ # exist and NO change to this checker could ever satisfy its own evidence # rule. Mapped to the file CI actually runs. "scripts/check-device-leakage.py": "tests/scripts/test_device_leakage.py", + "scripts/check-release-workflow.py": "tests/scripts/test_release_pipeline.py", } DISABLED_CREATION_CHECKER = ( @@ -223,8 +224,14 @@ b"def exact_waiver(*args, **kwargs): return None\n" b"def validate_waiver_targets(*args, **kwargs): return None\n" ), + "scripts/check-arm-isa-build.py": DISABLED_CREATION_CHECKER, + "scripts/check-cpu-isa-build.py": DISABLED_CREATION_CHECKER, + "scripts/check-cuda-fat-gencode.py": DISABLED_CREATION_CHECKER, + "scripts/check-release-workflow.py": DISABLED_CREATION_CHECKER, + "scripts/validate-release-archive.py": DISABLED_CREATION_CHECKER, "scripts/check-pr-size.py": DISABLED_CREATION_CHECKER, "scripts/check-prompt-contract.py": DISABLED_CREATION_CHECKER, + "scripts/check-triton-aot-multiarch.py": DISABLED_CREATION_CHECKER, } SELF_CHECKER = "scripts/check-pr-size.py" EVIDENCE_TIMEOUT_SECONDS = 120 @@ -323,6 +330,7 @@ def classify_path(path: str) -> str: return "vendored_dependency" if path in { "release/manifest-v1.schema.json", + "release/release-matrix.json", "scripts/env-doc-allowlist.txt", }: return "configuration" diff --git a/scripts/check-public-doc-tables.py b/scripts/check-public-doc-tables.py index b0885af30..7256e3152 100755 --- a/scripts/check-public-doc-tables.py +++ b/scripts/check-public-doc-tables.py @@ -435,7 +435,13 @@ def features_errors(text: str) -> list[str]: # compaction onto current main reduced the live page below the inherited # cap. The cap remains byte-tight and within the <=25 convention, so that # reduction cannot become untracked growth headroom. - "chars": 243761, + # 243694 since 2026-08-09 (measured 243694): the W6 claim checkpoint + # replaces the longer container-only paragraph with the current binary and + # container dependency state. Re-pinned byte-tight after the structured + # state migration so the reduction cannot become untracked growth headroom. + # 243632 since 2026-08-09 (measured 243632): the W1 candidate records local + # gencode gates while keeping the real ten-SM archive audit pending. + "chars": 243632, "h2_sections": 11, "long_paragraphs": 82, "oversized_cells": 44, diff --git a/scripts/check-release-binary-contract.py b/scripts/check-release-binary-contract.py index b3aa5ff36..ab7aff893 100644 --- a/scripts/check-release-binary-contract.py +++ b/scripts/check-release-binary-contract.py @@ -55,71 +55,71 @@ ANCHORS = { ".agents/engine-matrix.md": "| `ENG-RELEASE-BINARIES` |", ".agents/roadmap_v1.md": "| REL | `ROAD-V1-RELEASE` |", - ".agents/NOW.md": "| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** |", - ".agents/coordination.md": "| `CLAIM-ENG-RELEASE-BINARIES-W5` | `ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) |", - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md": "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE", - "docs/STATUS.md": "#141 W5 19/19 ACTIVE ART∅", - "docs/BENCHMARKS.md": "| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`:", + ".agents/NOW.md": "| Release | **ACTIVE; required W1-W11/W13 implemented in #196** |", + ".agents/coordination.md": "**Server binary release W1-W13 (`ENG-RELEASE-BINARIES`, 2026-08-09,", + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md": "# W6 installed server package green", + "docs/STATUS.md": "#196 binary pipeline implemented; no published binaries", + "docs/BENCHMARKS.md": "| **Binary release matrix (ACTIVE; required W1-W11/W13 implemented in #196)** |", } LIFECYCLE_RECORD_MUTATIONS = ( ( ".agents/engine-matrix.md", - "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", - "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", + "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |", + "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |", "engine-matrix release lifecycle", ), ( ".agents/engine-matrix.md", - "install/archive/publish implementation and all real release evidence remain pending", - "install/archive/publish implementation and all real release evidence are complete", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, and tagged publication remain pending", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, and tagged publication are complete", "engine-matrix release lifecycle", ), ( ".agents/roadmap_v1.md", - "`ACTIVE` | W5 versioned manifest schema/generator", - "`DONE` | W5 versioned manifest schema/generator", + "`ACTIVE` | Required W1-W11/W13 implementation is complete", + "`DONE` | Required W1-W11/W13 implementation is complete", "roadmap release lifecycle", ), ( ".agents/roadmap_v1.md", - "no archive exists", - "archive exists", + "no published binary exists", + "published binaries exist", "roadmap release lifecycle", ), ( ".agents/coordination.md", - "| `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; no archive or real runtime/correctness/performance evidence |", - "| `DONE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; archive and runtime/correctness/performance evidence complete |", + "| `ACTIVE` | 2026-08-09 — required W1-W11/W13 implementation complete;", + "| `DONE` | 2026-08-09 — required W1-W11/W13 implementation complete;", "coordination release lifecycle", ), ( ".agents/coordination.md", - "Excludes W1-W4, W6-W13, archives, install/package/publish workflows and runtime artifacts", - "Includes W1-W13, archives, install/package/publish workflows and runtime artifacts", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, rebase/merge, and tagged publication pending", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, rebase/merge, and tagged publication complete", "coordination release lifecycle", ), ( ".agents/coordination.md", - "no archive or real runtime/correctness/performance evidence", - "release archive and runtime/correctness/performance evidence shipped", + "W12 optional/non-primary |", + "W12 required/primary |", "coordination release lifecycle", ), ( - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md", - "`ACTIVE`, never `DONE` or `GATING`.", - "`DONE`, with every release gate complete.", + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md", + "The row remains `ACTIVE`. W1-W4 and W7-W13 remain pending", + "The row is `DONE`. Every release gate is complete", "state release lifecycle", ), ) BENCHMARKS_RELEASE_ROW = ( - "| **Binary release matrix (ACTIVE; W5 implemented)** | `ENG-RELEASE-BINARIES`: versioned deterministic manifest for primary host-ABI fat-CUDA + adaptive-CPU static-core bundles, optional per-SM diagnostics and experimental literal-static musl CPU | **W5 GREEN:** " - "19/19 + ten review mutations killed; contract 30/30; PR-size classes/budget GREEN. Synthetic fixtures only. **PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates | n/a |" + "| **Binary release matrix (ACTIVE; required W1-W11/W13 implemented in #196)** | Eight primary CPU/CUDA/Vulkan/Metal/MLX host tuples | " + "Adaptive x86 tiers, Vulkan 35/35 + cross-device 11/11, and metadata/mutation gates green. **PENDING:** hosted full matrix, matching hardware, tagged publish | n/a |" ) STATUS_RELEASE_FRAGMENTS = ( - "Subset; #141 W5 19/19 ACTIVE ART∅", + "Subset; #196 binary pipeline implemented; no published binaries", ) BACKEND_POLICY_PROSE = { @@ -273,14 +273,14 @@ PUBLIC_PENDING_MUTATIONS = ( ( "docs/BENCHMARKS.md", - "**PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/correctness/performance gates", + "**PENDING:** hosted full matrix, matching hardware, tagged publish", "**SHIPPED:** archive, runtime, correctness, and performance evidence complete", "docs/BENCHMARKS.md release row", ), ( "docs/STATUS.md", - "Subset; #141 W5 19/19 ACTIVE ART∅", - "Supported; #141: RELEASE DONE/ARTIFACTS✓", + "Subset; #196 binary pipeline implemented; no published binaries", + "Supported; #196: RELEASE DONE/ARTIFACTS✓", "docs/STATUS.md release row", ), ) @@ -400,7 +400,10 @@ "EXACT_MACHINE_FIELDS": { "lifecycle": "ACTIVE", "manifest_schema": "vllm.cpp.release-manifest.v1", + "delivery_pull_request": "196", + "delivery_mode": "single-pr-W1-W13", "work_W5_status": "implemented", + "work_W6_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -414,9 +417,11 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/completed/state-events/2026-08/" - "STATE-20260808T220000-002.md,docs/STATUS.md," + "STATE-20260809T160000-001.md,docs/STATUS.md," "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," - "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py," + "examples/CMakeLists.txt,scripts/package-server.py," + "tests/scripts/test_server_package.py" ), }, "EXPECTED_DEPS": {work: ",".join(deps) for work, deps in WORK_DEPS.items()}, @@ -569,7 +574,10 @@ EXACT_MACHINE_FIELDS = { "lifecycle": "ACTIVE", "manifest_schema": "vllm.cpp.release-manifest.v1", + "delivery_pull_request": "196", + "delivery_mode": "single-pr-W1-W13", "work_W5_status": "implemented", + "work_W6_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -583,9 +591,11 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/completed/state-events/2026-08/" - "STATE-20260808T220000-002.md,docs/STATUS.md," + "STATE-20260809T160000-001.md,docs/STATUS.md," "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," - "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py," + "examples/CMakeLists.txt,scripts/package-server.py," + "tests/scripts/test_server_package.py" ), } @@ -606,10 +616,10 @@ ) STATE_RELEASE_HEADING = ( - "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE" + "## Outcome" ) STATE_RELEASE_LIFECYCLE = ( - "The row advances from `SPIKE` to `ACTIVE`, never `DONE` or `GATING`." + "The row remains `ACTIVE`. W1-W4 and W7-W13 remain pending" ) @@ -1004,9 +1014,14 @@ def _release_lifecycle_errors(root: Path) -> list[str]: "engine-matrix release lifecycle", errors, ) - if engine is not None and (engine[7] != "`ACTIVE`" or "W5 schema" not in engine[4] or "install/archive/publish implementation and all real release evidence remain pending" not in engine[4]): + if engine is not None and ( + engine[7] != "`ACTIVE`" + or "Required W1-W11/W13 implementation is complete" not in engine[4] + or "hosted ten-SM completion, full eight-tuple dry run" not in engine[5] + or "tagged publication remain pending" not in engine[5] + ): errors.append( - "engine-matrix release lifecycle must be ACTIVE with W5 implemented and install/archive/publish plus real evidence pending" + "engine-matrix release lifecycle must be ACTIVE with required implementation complete and hosted publication pending" ) roadmap = _table_record( @@ -1017,29 +1032,40 @@ def _release_lifecycle_errors(root: Path) -> list[str]: "roadmap release lifecycle", errors, ) - if roadmap is not None and (roadmap[5] != "`ACTIVE`" or "W5 versioned manifest" not in roadmap[6] or "W1-W4 and W6-W13 remain pending" not in roadmap[6] or "no archive exists" not in roadmap[6]): + if roadmap is not None and ( + roadmap[5] != "`ACTIVE`" + or "Required W1-W11/W13 implementation is complete" not in roadmap[6] + or "full eight-tuple dry run" not in roadmap[6] + or "no published binary exists" not in roadmap[6] + ): errors.append( - "roadmap release lifecycle must be ACTIVE with W5 implemented while W1-W4/W6-W13 and archives remain pending" + "roadmap release lifecycle must be ACTIVE with required implementation complete and hosted publication pending" ) coordination = _table_record( root, ".agents/coordination.md", - "| `CLAIM-ENG-RELEASE-BINARIES-W5` |", + "| `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |", 8, "coordination release lifecycle", errors, ) - if coordination is not None and (coordination[6] != "`ACTIVE`" or "W5 implemented" not in coordination[5] or "Excludes W1-W4, W6-W13" not in coordination[5] or "no archive or real runtime/correctness/performance evidence" not in coordination[7]): + if coordination is not None and ( + coordination[6] != "`ACTIVE`" + or "Complete W1-W13 contract in one PR" not in coordination[5] + or "required W1-W11/W13 implementation complete" not in coordination[7] + or "tagged publication pending" not in coordination[7] + or "W12 optional/non-primary" not in coordination[7] + ): errors.append( - "coordination release lifecycle must keep W5 ACTIVE and implemented while later work and real artifact evidence remain pending" + "coordination release lifecycle must keep required implementation ACTIVE while hosted publication remains pending" ) - state_path = root / ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md" + state_path = root / ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md" if not state_path.is_file(): errors.append( "state release lifecycle record " - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md is missing" + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md is missing" ) else: state_text = state_path.read_text(encoding="utf-8") @@ -1055,7 +1081,7 @@ def _release_lifecycle_errors(root: Path) -> list[str]: end = state_text.find("\n## ", start) section = state_text[start:] if end < 0 else state_text[start:end] if STATE_RELEASE_LIFECYCLE not in _normalize_prose(section): - errors.append("state release lifecycle must say W5 advances the row only to ACTIVE, never DONE or GATING") + errors.append("state release lifecycle must keep W6 ACTIVE while W1-W4/W7-W13 remain pending") return errors @@ -1260,9 +1286,9 @@ def contract_errors(root: Path) -> list[str]: if fields.get(key) != expected: errors.append(_field_error(key, fields.get(key), expected)) - if "Status: accepted contract with W5 manifest tooling implemented for\n`ENG-RELEASE-BINARIES`." not in text: + if "Status: accepted contract with required W1-W11/W13 implementation complete for\n`ENG-RELEASE-BINARIES`" not in text: errors.append( - "release spec identity/status line must name the accepted W5 ENG-RELEASE-BINARIES contract" + "release spec identity/status line must name the accepted required implementation state" ) header = "| Work | Deps | Deliverable | Exit gate |" @@ -1328,8 +1354,8 @@ def contract_errors(root: Path) -> list[str]: encoding="utf-8" ): errors.append( - "docs/BENCHMARKS.md release row must stay PENDING with no archive, " - "runtime, correctness, or performance evidence" + "docs/BENCHMARKS.md release row must keep hosted validation and " + "publication pending" ) status = root / "docs/STATUS.md" status_text = status.read_text(encoding="utf-8") if status.is_file() else "" @@ -1339,8 +1365,7 @@ def contract_errors(root: Path) -> list[str]: ) if not all(fragment in status_row for fragment in STATUS_RELEASE_FRAGMENTS): errors.append( - "docs/STATUS.md release row must stay ACTIVE/W5 with no artifacts or " - "runtime claim" + "docs/STATUS.md release row must keep the binary pipeline unpublished" ) preflight = root / PREFLIGHT_PATH ci = root / CI_PATH diff --git a/scripts/check-release-workflow.py b/scripts/check-release-workflow.py new file mode 100755 index 000000000..5fc80f636 --- /dev/null +++ b/scripts/check-release-workflow.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Static least-privilege and immutable-handoff gate for W8 release.yml.""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +WORKFLOW = ROOT / ".github/workflows/release.yml" + + +def job_block(text: str, name: str) -> str: + match = re.search( + rf"(?ms)^ {re.escape(name)}:\n(.*?)(?=^ [a-zA-Z0-9_-]+:\n|\Z)", text + ) + return match.group(1) if match else "" + + +def validate(text: str) -> list[str]: + errors: list[str] = [] + required_global = ( + " workflow_dispatch: {}", + " push:\n tags: ['v*']", + "permissions:\n contents: read", + ) + for fragment in required_global: + if fragment not in text: + errors.append(f"workflow is missing required global contract: {fragment!r}") + if re.search(r"(?m)^\s*pull_request:", text): + errors.append("release workflow must not run for pull requests") + if "continue-on-error" in text: + errors.append("release workflow may not continue after an error") + + primary_build_jobs = ( + "cpu_x86", + "cpu_arm64", + "cpu_musl", + "cuda_x86", + "cuda_arm64", + "vulkan_x86", + "metal_arm64", + "mlx_arm64", + ) + read_only_jobs = ( + "plan", + *primary_build_jobs, + "build", + "verify", + ) + blocks = { + name: job_block(text, name) + for name in (*read_only_jobs, "attest", "publish") + } + for name, block in blocks.items(): + if not block: + errors.append(f"release workflow is missing {name} job") + for name in read_only_jobs: + block = blocks[name] + if " permissions:\n contents: read" not in block: + errors.append(f"{name} job must have contents: read only") + permission_block = re.search(r"(?m)^ permissions:\n((?: [^\n]*\n)+)", block) + if permission_block and "write" in permission_block.group(1): + errors.append(f"{name} job unexpectedly has write permission") + + build = blocks["build"] + expected_needs = " needs: [plan, " + ", ".join(primary_build_jobs) + "]" + if expected_needs not in build: + errors.append("handoff build job must depend on every primary release tuple") + for name in primary_build_jobs: + reference = f"${{{{ needs.{name}.outputs.artifact_id }}}}" + if reference not in build: + errors.append(f"handoff build job does not consume immutable {name} output") + + attest = blocks["attest"] + for permission in ( + " contents: read", + " id-token: write", + " attestations: write", + " artifact-metadata: write", + ): + if permission not in attest: + errors.append(f"attest job is missing {permission.strip()}") + if "uses: actions/attest@v4" not in attest: + errors.append("attest job must use actions/attest@v4") + tag_gate = "startsWith(github.ref, 'refs/tags/v')" + if tag_gate not in attest or "needs.plan.outputs.publish == 'true'" not in attest: + errors.append("attest job must require an exact tag and approved publish plan") + + publish = blocks["publish"] + if " permissions:\n contents: write" not in publish: + errors.append("publish job alone must receive contents: write") + if any(permission in publish for permission in ("id-token: write", "attestations: write")): + errors.append("publish job must not receive attestation authority") + if " environment: release" not in publish: + errors.append("publish job must use the protected release environment") + if " needs: [plan, verify, attest]" not in publish: + errors.append("publish job must consume only plan plus verified and attested handoffs") + if "uses: actions/checkout@v4" not in publish: + errors.append("publish job must check out the exact tagged publisher implementation") + if tag_gate not in publish or "needs.plan.outputs.publish == 'true'" not in publish: + errors.append("publish job must require an exact tag and approved publish plan") + if "scripts/release_pipeline.py publish" not in publish: + errors.append("publish job must use the byte-bound release publisher") + if "verified/assets" not in publish or "verified/release-index.json" not in publish: + errors.append("publish job must release verified assets and generated indexes") + if "gh release create" in publish: + errors.append("release workflow must not bypass the byte-bound publisher") + + required_handoff = ( + "name: release-plan-${{ github.sha }}", + "name: release-unverified-${{ github.sha }}", + "name: release-verified-${{ github.sha }}", + "artifact-ids: ${{ needs.plan.outputs.artifact_id }}", + "artifact-ids: ${{ needs.build.outputs.artifact_id }}", + "artifact-ids: ${{ needs.verify.outputs.artifact_id }}", + "overwrite: false", + "if-no-files-found: error", + "python3 scripts/release_index.py", + ) + for fragment in required_handoff: + if fragment not in text: + errors.append(f"immutable artifact handoff is missing {fragment!r}") + uploads = text.count("uses: actions/upload-artifact@v4") + downloads = text.count("uses: actions/download-artifact@v4") + if uploads < 3: + errors.append("release workflow requires immutable plan, asset, and verified uploads") + if text.count("overwrite: false") != uploads: + errors.append("every artifact upload must refuse overwrite") + if text.count("if-no-files-found: error") != uploads: + errors.append("every artifact upload must fail when its explicit file is missing") + if downloads < 5 or text.count("artifact-ids:") != downloads: + errors.append("every cross-job handoff must use an exact immutable artifact ID") + if re.search(r"(?m)^\s+path:\s*[^\n]*[*?]", text): + errors.append("release workflow artifact paths must not use wildcards") + if re.search(r"gh release (?:create|upload)[^\n]*[*?]", text): + errors.append("publish command must enumerate release assets without wildcards") + return errors + + +def main() -> int: + errors = validate(WORKFLOW.read_text(encoding="utf-8")) + if errors: + print("release workflow policy FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("release workflow policy: least privilege and immutable handoff OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-triton-aot-multiarch.py b/scripts/check-triton-aot-multiarch.py new file mode 100644 index 000000000..83eef1a62 --- /dev/null +++ b/scripts/check-triton-aot-multiarch.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""Audit W2 Triton AOT tree identities and collision-free archive symbols.""" + +from __future__ import annotations + +import argparse +import re +import struct +import subprocess +import sys +from pathlib import Path + + +TREE_FLAGS = { + "sm_80": 0x00500550, + "sm_86": 0x00560556, + "sm_89": 0x00590559, + "sm_90a": 0x005A0D5A, + "sm_100a": 0x0600640A, + "sm_121a": 0x0600790A, +} +CUBIN_ARRAY_RE = re.compile( + r"unsigned\s+char\s+CUBIN_NAME\s*\[[^]]+\]\s*=\s*\{(.*?)\};", + re.DOTALL, +) +BYTE_RE = re.compile(r"0x([0-9a-fA-F]{2})") +SYMBOL_RE = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]*)\b") + + +def manifest_bases(path: Path) -> set[str]: + bases: set[str] = set() + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("base "): + bases.add(line.split(maxsplit=2)[1]) + return bases + + +def cubin_flags(path: Path) -> int | None: + match = CUBIN_ARRAY_RE.search(path.read_text(encoding="utf-8")) + if match is None: + return None + data = bytes(int(value, 16) for value in BYTE_RE.findall(match.group(1))) + if len(data) < 52 or data[:4] != b"\x7fELF": + return None + return struct.unpack_from(" tuple[list[str], set[str]]: + errors: list[str] = [] + actual_trees = {path.name for path in root.iterdir() if path.is_dir()} + expected_trees = set(TREE_FLAGS) + if actual_trees != expected_trees: + errors.append( + "Triton AOT tree set " + f"{sorted(actual_trees)} != expected {sorted(expected_trees)}" + ) + + canonical_bases: set[str] | None = None + for tree, expected_flags in TREE_FLAGS.items(): + directory = root / tree + manifest = directory / "MANIFEST" + if not manifest.is_file(): + errors.append(f"{tree}: missing MANIFEST") + continue + lines = manifest.read_text(encoding="utf-8").splitlines() + if f"arch {tree}" not in lines: + errors.append(f"{tree}: MANIFEST arch mismatch") + bases = manifest_bases(manifest) + if not bases: + errors.append(f"{tree}: MANIFEST has no base declarations") + if canonical_bases is None: + canonical_bases = bases + elif bases != canonical_bases: + errors.append(f"{tree}: base set differs from the other trees") + + cubins = sorted(directory.glob("*.*.c")) + if not cubins: + errors.append(f"{tree}: no embedded cubin launchers") + for source in cubins: + actual_flags = cubin_flags(source) + if actual_flags != expected_flags: + shown = "unreadable" if actual_flags is None else hex(actual_flags) + errors.append( + f"{tree}/{source.name}: wrong cubin ELF flags {shown}; " + f"expected {hex(expected_flags)}" + ) + return errors, canonical_bases or set() + + +def validate_symbols(nm_listing: str, bases: set[str]) -> list[str]: + symbols = set(SYMBOL_RE.findall(nm_listing)) + expected = set() + for tree in TREE_FLAGS: + for base in bases: + expected.add(f"vt_aot_{tree}_{base}_default") + expected.add(f"vt_aot_{tree}_load_{base}") + missing = sorted(expected.difference(symbols)) + unnamespaced = sorted( + symbol + for base in bases + for symbol in (f"{base}_default", f"load_{base}") + if symbol in symbols + ) + errors: list[str] = [] + if missing: + errors.append(f"AOT symbol namespace is missing: {','.join(missing)}") + if unnamespaced: + errors.append(f"AOT symbol namespace leaked: {','.join(unnamespaced)}") + return errors + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--vendored-root", type=Path, required=True) + symbols = parser.add_mutually_exclusive_group(required=True) + symbols.add_argument("--library", type=Path) + symbols.add_argument("--nm-list", type=Path) + args = parser.parse_args() + + errors, bases = validate_trees(args.vendored_root) + if args.library is not None: + result = subprocess.run( + ["nm", "--defined-only", "--extern-only", str(args.library)], + text=True, + capture_output=True, + check=False, + ) + if result.returncode != 0: + errors.append(f"nm failed: {result.stderr.strip()}") + listing = "" + else: + listing = result.stdout + else: + listing = args.nm_list.read_text(encoding="utf-8") + errors.extend(validate_symbols(listing, bases)) + + if errors: + print("Triton AOT multi-arch audit FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("Triton AOT multi-arch audit: six exact trees and namespaces OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index 0d8700cd8..5abe69189 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -29,9 +29,6 @@ VT_BF16_GEMM_OUT VT_BF16_RESIDUAL VT_CONV_TILED VT_CONV_UPDATE_FAST -VT_CPU_MATMUL_TIER -VT_CPU_QUANT_MMLA -VT_CPU_QUANT_REPACK VT_CUTLASS_NOPOOL VT_DECODE_GRAPH_STATS VT_DEEPSEEK_CUDAGRAPH diff --git a/scripts/package-server.py b/scripts/package-server.py new file mode 100644 index 000000000..78ea91bd4 --- /dev/null +++ b/scripts/package-server.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +"""Stage the installed server component and optionally archive it reproducibly.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import os +import shutil +import subprocess +import tarfile +import tempfile +from pathlib import Path + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--build-dir", type=Path, required=True) + parser.add_argument("--stage-dir", type=Path, required=True) + parser.add_argument("--archive", type=Path) + parser.add_argument("--metadata-dir", type=Path) + parser.add_argument("--config", default="") + return parser.parse_args() + + +def source_date_epoch() -> int: + raw = os.environ.get("SOURCE_DATE_EPOCH", "0") + try: + epoch = int(raw) + except ValueError as exc: + raise SystemExit("SOURCE_DATE_EPOCH must be a non-negative integer") from exc + if epoch < 0: + raise SystemExit("SOURCE_DATE_EPOCH must be a non-negative integer") + return epoch + + +def require_build_output(path: Path, build_dir: Path, label: str) -> None: + try: + relative = path.relative_to(build_dir) + except ValueError as exc: + raise SystemExit(f"{label} must be inside the configured build directory") from exc + if relative == Path("."): + raise SystemExit(f"{label} must not replace the configured build directory") + + +def install_component(build_dir: Path, prefix: Path, config: str) -> None: + command = [ + "cmake", + "--install", + str(build_dir), + "--prefix", + str(prefix), + "--component", + "vllm-server", + ] + if config: + command.extend(("--config", config)) + subprocess.run(command, check=True) + + +def normalized_info(tar: tarfile.TarFile, path: Path, arcname: str, epoch: int) -> tarfile.TarInfo: + info = tar.gettarinfo(str(path), arcname=arcname) + info.uid = 0 + info.gid = 0 + info.uname = "root" + info.gname = "root" + info.mtime = epoch + if info.isdir(): + info.mode = 0o755 + elif info.isfile(): + info.mode = 0o755 if path.stat().st_mode & 0o111 else 0o644 + return info + + +def write_archive(stage_dir: Path, archive: Path, epoch: int) -> None: + archive.parent.mkdir(parents=True, exist_ok=True) + descriptor, temporary_name = tempfile.mkstemp( + dir=archive.parent, prefix=f".{archive.name}.", suffix=".tmp" + ) + os.close(descriptor) + temporary = Path(temporary_name) + try: + with temporary.open("wb") as raw: + with gzip.GzipFile(filename="", mode="wb", fileobj=raw, mtime=epoch) as compressed: + with tarfile.open(fileobj=compressed, mode="w", format=tarfile.GNU_FORMAT) as bundle: + for path in sorted(stage_dir.rglob("*"), key=lambda item: item.as_posix()): + arcname = path.relative_to(stage_dir).as_posix() + info = normalized_info(bundle, path, arcname, epoch) + if info.isfile(): + with path.open("rb") as source: + bundle.addfile(info, source) + else: + bundle.addfile(info) + os.replace(temporary, archive) + finally: + temporary.unlink(missing_ok=True) + + +def install_metadata(metadata_dir: Path, stage_dir: Path) -> None: + required = { + "THIRD_PARTY_NOTICES", + "VERSION", + "release-manifest.json", + "sbom.spdx.json", + } + files = { + path.relative_to(metadata_dir).as_posix(): path + for path in metadata_dir.rglob("*") + if path.is_file() + } + missing = sorted(required - files.keys()) + if missing: + raise SystemExit(f"release metadata is missing required files: {missing}") + licenses = [name for name in files if name.startswith("share/licenses/")] + if not licenses: + raise SystemExit("release metadata must include share/licenses entries") + allowed = required | set(licenses) + extra = sorted(set(files) - allowed) + if extra: + raise SystemExit(f"release metadata contains undeclared files: {extra}") + for relative, source in files.items(): + destination = stage_dir / relative + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, destination) + + +def write_archive_sidecars(archive: Path, stage_dir: Path) -> None: + hasher = hashlib.sha256() + with archive.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + hasher.update(chunk) + digest = hasher.hexdigest() + Path(f"{archive}.sha256").write_text( + f"{digest} {archive.name}\n", encoding="utf-8" + ) + manifest = json.loads((stage_dir / "release-manifest.json").read_text(encoding="utf-8")) + provenance = { + "_type": "https://in-toto.io/Statement/v1", + "predicateType": "https://slsa.dev/provenance/v1", + "subject": [{"name": archive.name, "digest": {"sha256": digest}}], + "predicate": { + "buildDefinition": { + "externalParameters": { + "artifact_id": manifest["artifact"]["id"], + "source_commit": manifest["build"]["source_commit"], + } + } + }, + } + Path(f"{archive}.provenance.json").write_text( + json.dumps(provenance, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + +def main() -> int: + args = parse_args() + build_dir = args.build_dir.resolve() + stage_dir = args.stage_dir.resolve() + if not (build_dir / "CMakeCache.txt").is_file(): + raise SystemExit("--build-dir must name a configured CMake build") + require_build_output(stage_dir, build_dir, "--stage-dir") + if args.archive is not None: + require_build_output(args.archive.resolve(), build_dir, "--archive") + metadata_dir = args.metadata_dir.resolve() if args.metadata_dir is not None else None + if metadata_dir is not None and not metadata_dir.is_dir(): + raise SystemExit("--metadata-dir must name a prepared metadata directory") + stage_dir.parent.mkdir(parents=True, exist_ok=True) + + with tempfile.TemporaryDirectory(prefix=".vllm-server-stage-", dir=stage_dir.parent) as raw: + fresh_stage = Path(raw) + install_component(build_dir, fresh_stage, args.config) + server_name = "vllm-server.exe" if os.name == "nt" else "vllm-server" + server = fresh_stage / "bin" / server_name + if not server.is_file(): + raise SystemExit(f"installed server component is missing {server.relative_to(fresh_stage)}") + if metadata_dir is not None: + install_metadata(metadata_dir, fresh_stage) + if args.archive is not None: + write_archive(fresh_stage, args.archive.resolve(), source_date_epoch()) + if metadata_dir is not None: + write_archive_sidecars(args.archive.resolve(), fresh_stage) + + if stage_dir.exists(): + shutil.rmtree(stage_dir) + shutil.copytree(fresh_stage, stage_dir, symlinks=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/release_accelerator_metadata.py b/scripts/release_accelerator_metadata.py new file mode 100755 index 000000000..38cb82865 --- /dev/null +++ b/scripts/release_accelerator_metadata.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Generate Linux CUDA/Vulkan release metadata from staged bytes and build facts.""" + +from __future__ import annotations + +import argparse +import json +import shutil +import sys +from pathlib import Path +from typing import Any + + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +import release_manifest # noqa: E402 +import release_metadata # noqa: E402 + + +ARTIFACTS = { + "linux-x86_64-glibc-cuda-fat": ("x86_64", "cuda"), + "linux-aarch64-glibc-cuda-fat": ("aarch64", "cuda"), + "linux-x86_64-glibc-vulkan": ("x86_64", "vulkan"), +} + + +def external_dependency(name: str, kind: str, version: str) -> dict[str, Any]: + return { + "bundled": False, + "kind": kind, + "linkage": "external", + "name": name, + "role": "external-runtime", + "version": version, + } + + +def dependency_rows(server: Path, backend: str, abi_version: str) -> list[dict[str, Any]]: + rows = release_metadata.dependencies(server, "glibc", abi_version) + if backend == "cuda": + rows.append(external_dependency("nvidia-driver", "driver", ">=580")) + else: + rows.extend( + ( + external_dependency("vulkan-loader", "library", ">=1.3"), + external_dependency("vulkan-icd", "library", "vendor-matched"), + external_dependency("vulkan-driver", "driver", "vendor-matched"), + ) + ) + return rows + + +def cuda_evidence(url: str) -> dict[str, Any]: + rows = [] + for sm in release_manifest.PRIMARY_CUDA_SMS: + available = release_manifest.AOT_AVAILABILITY[sm] + rows.append( + { + "aot_available": available, + "aot_evidence": ( + release_metadata.passed( + f"check exact embedded vt_aot_sm_{sm}_ namespace", url + ) + if available + else release_metadata.evidence( + "not-applicable", + "", + "", + "", + f"no complete Triton AOT tree exists for sm_{sm}", + ) + ), + "portable_fallback": not available, + "runtime_evidence": release_metadata.absent( + f"preview archive has no matching sm_{sm} runtime gate" + ), + "sm": sm, + } + ) + return { + "compiled_sms": list(release_manifest.PRIMARY_CUDA_SMS), + "sm_evidence": rows, + } + + +def prepare_accelerator_metadata(args: argparse.Namespace) -> dict[str, Any]: + if args.artifact_id not in ARTIFACTS: + raise ValueError(f"unsupported Linux accelerator artifact {args.artifact_id!r}") + arch, backend = ARTIFACTS[args.artifact_id] + if args.channel != "preview": + raise ValueError("Linux accelerator bundles remain preview until matching runtime evidence exists") + server = args.stage_dir / "bin/vllm-server" + if not server.is_file(): + raise ValueError("staged bin/vllm-server is missing") + flags = release_metadata.backend_flags( + release_metadata.parse_cache(args.build_dir / "CMakeCache.txt") + ) + expected_sms = list(release_manifest.PRIMARY_CUDA_SMS) if backend == "cuda" else [] + if flags["VLLM_CPP_CUDA"] is not (backend == "cuda"): + raise ValueError("configured CUDA flag does not match artifact backend") + if flags["VLLM_CPP_VULKAN"] is not (backend == "vulkan"): + raise ValueError("configured Vulkan flag does not match artifact backend") + if flags["VLLM_CPP_CUDA_ARCHITECTURES"] != expected_sms: + raise ValueError("CUDA release architecture cache must equal the exact ten-SM policy") + if flags["VLLM_CPP_TRITON"] is not (backend == "cuda"): + raise ValueError("CUDA release requires Triton and non-CUDA release forbids it") + dependencies = dependency_rows(server, backend, args.abi_version) + build_audit = ( + "check-cuda-fat-gencode.py && check-triton-aot-multiarch.py" + if backend == "cuda" + else "test_vulkan_backend && test_backend_cross_device" + ) + facts: dict[str, Any] = { + "artifact": { + "c_abi_version": args.c_abi_version, + "channel": args.channel, + "id": args.artifact_id, + "kind": "primary", + "static_boundary": "static-core", + "version": args.version, + }, + "backend": { + "flags": flags, + "gpu_driver_boundary": "external-host-never-bundled", + "name": backend, + }, + "build": { + "compiler": args.compiler, + "resolved_cmake_options": flags, + "source_clean": args.source_clean, + "source_commit": args.source_commit, + "test_commands": [build_audit, "python3 scripts/validate-release-archive.py"], + "toolchain": args.toolchain, + }, + "dependencies": dependencies, + "evidence": { + "archive_smoke": release_metadata.passed( + "extracted vllm-server --help && --version", args.evidence_url + ), + "build": release_metadata.passed("cmake --build --target server", args.evidence_url), + "correctness": release_metadata.absent( + "preview accelerator tuple has no matching-hardware model correctness gate" + ), + "dependency_audit": release_metadata.passed( + "readelf -dW && ldd/lddtree", args.evidence_url + ), + "performance": release_metadata.absent("preview artifact makes no performance claim"), + "runtime": release_metadata.absent( + "preview accelerator tuple has no matching-hardware runtime gate" + ), + }, + "host": { + "abi": "glibc", + "abi_version": args.abi_version, + "arch": arch, + "os": "linux", + }, + "supply_chain": { + "archive_checksum": release_metadata.passed("sha256sum ", args.evidence_url), + "licenses": release_metadata.passed("validate notices and licenses", args.evidence_url), + "provenance": release_metadata.passed("validate detached SLSA subject", args.evidence_url), + "sbom": release_metadata.passed("validate SPDX-2.3 inventory", args.evidence_url), + }, + } + if backend == "cuda": + facts["cuda"] = cuda_evidence(args.evidence_url) + schema = release_manifest.load_schema(args.repo_root / "release/manifest-v1.schema.json") + manifest = release_manifest.generate_manifest(facts, args.repo_root, schema) + output = args.output_dir + output.mkdir(parents=True, exist_ok=True) + (output / "release-manifest.json").write_text( + release_metadata.canonical_json(manifest), encoding="utf-8" + ) + version_values = { + "version": args.version, + "commit": args.source_commit, + "artifact_id": args.artifact_id, + "backend": backend, + "host_os": "linux", + "host_arch": arch, + "host_abi": "glibc", + "source_clean": "true" if args.source_clean else "false", + "c_abi_version": str(args.c_abi_version), + } + (output / "VERSION").write_text( + "".join(f"{key}={value}\n" for key, value in version_values.items()), + encoding="utf-8", + ) + (output / "sbom.spdx.json").write_text( + release_metadata.canonical_json( + release_metadata.spdx_document( + args.artifact_id, + args.version, + args.source_commit, + server, + dependencies, + ) + ), + encoding="utf-8", + ) + notices = ["vllm.cpp release dependency notices", ""] + notices.extend( + f"- {row['name']} {row['version']} ({row['linkage']})" for row in dependencies + ) + (output / "THIRD_PARTY_NOTICES").write_text("\n".join(notices) + "\n", encoding="utf-8") + license_dir = output / "share/licenses/vllm.cpp" + license_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(args.repo_root / "LICENSE", license_dir / "LICENSE") + return manifest + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--repo-root", type=Path, default=SCRIPT_DIR.parent) + parser.add_argument("--build-dir", type=Path, required=True) + parser.add_argument("--stage-dir", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--artifact-id", required=True) + parser.add_argument("--channel", choices=("preview",), required=True) + parser.add_argument("--version", required=True) + parser.add_argument("--c-abi-version", type=int, required=True) + parser.add_argument("--source-commit", required=True) + parser.add_argument("--source-clean", action="store_true") + parser.add_argument("--abi-version", required=True) + parser.add_argument("--compiler", required=True) + parser.add_argument("--toolchain", required=True) + parser.add_argument("--evidence-url", required=True) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + prepare_accelerator_metadata(args) + except (OSError, json.JSONDecodeError, KeyError, ValueError, release_manifest.ManifestError) as exc: + print(f"accelerator release metadata error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/release_index.py b/scripts/release_index.py new file mode 100755 index 000000000..148a9650d --- /dev/null +++ b/scripts/release_index.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""Generate W13 JSON/Markdown indexes exclusively from verified release bytes.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +import tarfile +from pathlib import Path, PurePosixPath +from typing import Any + + +SCHEMA = "vllm.cpp.release-index.v1" + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def read_manifest(archive: Path) -> dict[str, Any]: + with tarfile.open(archive, "r:gz") as bundle: + members = [member for member in bundle.getmembers() if member.name == "release-manifest.json"] + if len(members) != 1 or not members[0].isfile() or members[0].size > 4 * 1024 * 1024: + raise ValueError(f"{archive.name} must contain one bounded release-manifest.json") + if PurePosixPath(members[0].name).is_absolute(): + raise ValueError(f"{archive.name} has an unsafe manifest member") + handle = bundle.extractfile(members[0]) + if handle is None: + raise ValueError(f"{archive.name} manifest cannot be read") + value = json.loads(handle.read().decode("utf-8")) + if not isinstance(value, dict): + raise ValueError(f"{archive.name} manifest must be an object") + return value + + +def generate_index( + assets_dir: Path, + handoff: dict[str, Any], + json_output: Path, + markdown_output: Path, + retention_days: int = 7, +) -> dict[str, Any]: + if handoff.get("verified") is not True: + raise ValueError("release index requires a verified handoff") + retention = handoff.get("retention") + if retention != { + "ci_artifacts_days": retention_days, + "github_release": "maintainer-deletion-only", + }: + raise ValueError("release index retention does not match the verified handoff") + files = handoff.get("files") + if not isinstance(files, list) or not files: + raise ValueError("verified handoff has no release files") + by_name: dict[str, dict[str, Any]] = {} + for item in files: + if not isinstance(item, dict) or not isinstance(item.get("name"), str): + raise ValueError("verified handoff file inventory is malformed") + if item["name"] in by_name or Path(item["name"]).name != item["name"]: + raise ValueError("verified handoff has a duplicate or unsafe file name") + path = assets_dir / item["name"] + if not path.is_file() or path.is_symlink(): + raise ValueError(f"verified release file is missing: {item['name']}") + if sha256(path) != item.get("sha256") or path.stat().st_size != item.get("size"): + raise ValueError(f"verified release file drifted: {item['name']}") + by_name[item["name"]] = item + actual_names = { + path.name for path in assets_dir.iterdir() if path.is_file() or path.is_symlink() + } + if actual_names != set(by_name): + raise ValueError("release assets do not exactly match the verified handoff") + archives = sorted(name for name in by_name if name.endswith(".tar.gz")) + rows: list[dict[str, Any]] = [] + for archive_name in archives: + artifact_id = archive_name.removesuffix(".tar.gz") + required_names = { + archive_name, + f"{archive_name}.sha256", + f"{archive_name}.provenance.json", + } + if not required_names <= by_name.keys(): + raise ValueError(f"verified asset triplet is incomplete for {artifact_id}") + manifest = read_manifest(assets_dir / archive_name) + if manifest.get("artifact", {}).get("id") != artifact_id: + raise ValueError(f"archive manifest identity mismatch for {archive_name}") + if manifest.get("build", {}).get("source_commit") != handoff.get("source_sha"): + raise ValueError(f"archive source identity mismatch for {archive_name}") + external = [ + dependency["name"] + for dependency in manifest.get("dependencies", []) + if isinstance(dependency, dict) and dependency.get("role") == "external-runtime" + ] + limitations: list[str] = [] + channel = manifest["artifact"]["channel"] + if channel != "stable": + limitations.append(f"{channel}: see per-gate evidence in the embedded manifest") + if external: + limitations.append("external runtime: " + ", ".join(external)) + if manifest["artifact"]["static_boundary"] == "literal-static": + limitations.append("experimental CPU-only literal-static feasibility lane") + rows.append( + { + "archive": archive_name, + "backend": manifest["backend"]["name"], + "channel": channel, + "checksum": f"{archive_name}.sha256", + "cpu_tiers": [ + tier["name"] for tier in manifest.get("cpu", {}).get("compiled_tiers", []) + ], + "driver_boundary": manifest["backend"]["gpu_driver_boundary"], + "host": manifest["host"], + "id": artifact_id, + "limitations": limitations, + "provenance": f"{archive_name}.provenance.json", + "sha256": by_name[archive_name]["sha256"], + "size": by_name[archive_name]["size"], + "sms": manifest.get("cuda", {}).get("compiled_sms", []), + "static_boundary": manifest["artifact"]["static_boundary"], + } + ) + index = { + "artifacts": rows, + "release_tag": handoff.get("release_tag"), + "retention": { + "ci_artifacts_days": retention_days, + "github_release": retention["github_release"], + }, + "schema": SCHEMA, + "source_sha": handoff.get("source_sha"), + } + json_output.parent.mkdir(parents=True, exist_ok=True) + json_output.write_text(json.dumps(index, indent=2, sort_keys=True) + "\n", encoding="utf-8") + lines = [ + f"# vllm.cpp {index['release_tag']} binary index", + "", + f"Source: `{index['source_sha']}`", + "", + "| Artifact | Channel | Host ABI | Backend | CPU tiers / CUDA SMs | Boundary and limitations |", + "|---|---|---|---|---|---|", + ] + for row in rows: + host = row["host"] + capabilities = ", ".join(row["cpu_tiers"] or row["sms"] or ["platform-native"]) + boundary = "; ".join([row["static_boundary"], row["driver_boundary"], *row["limitations"]]) + lines.append( + f"| [{row['id']}](./{row['archive']}) ([sha256](./{row['checksum']})) " + f"| {row['channel']} | {host['os']}/{host['arch']}/{host['abi']} {host['abi_version']} " + f"| {row['backend']} | {capabilities} | {boundary} |" + ) + lines.extend( + ( + "", + f"CI handoff artifacts are retained for {retention_days} days. Published GitHub release assets follow the {retention['github_release']} policy.", + "", + ) + ) + markdown_output.parent.mkdir(parents=True, exist_ok=True) + markdown_output.write_text("\n".join(lines), encoding="utf-8") + return index + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--assets-dir", type=Path, required=True) + parser.add_argument("--handoff", type=Path, required=True) + parser.add_argument("--json-output", type=Path, required=True) + parser.add_argument("--markdown-output", type=Path, required=True) + parser.add_argument("--retention-days", type=int, default=7) + args = parser.parse_args() + try: + handoff = json.loads(args.handoff.read_text(encoding="utf-8")) + generate_index( + args.assets_dir, + handoff, + args.json_output, + args.markdown_output, + args.retention_days, + ) + except (OSError, json.JSONDecodeError, tarfile.TarError, ValueError) as exc: + print(f"release index error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/release_macos_metadata.py b/scripts/release_macos_metadata.py new file mode 100755 index 000000000..d47ab0966 --- /dev/null +++ b/scripts/release_macos_metadata.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +"""Generate native macOS Metal/MLX release metadata from staged bytes.""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any + + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +import release_manifest # noqa: E402 +import release_metadata # noqa: E402 + + +ARTIFACTS = { + "macos-arm64-metal": ("metal", "stable"), + "macos-arm64-metal-mlx": ("mlx", "preview"), +} + + +def otool_dependencies(server: Path) -> list[str]: + result = subprocess.run( + ["otool", "-L", str(server)], text=True, capture_output=True, check=False + ) + if result.returncode != 0: + raise ValueError(f"otool dependency inspection failed: {result.stderr.strip()}") + return [line.strip().split(" (", 1)[0] for line in result.stdout.splitlines()[1:] if line.strip()] + + +def dependency_name(install_name: str) -> tuple[str, str]: + for component in install_name.split("/"): + if component.endswith(".framework"): + return component, "framework" + return Path(install_name).name, "library" + + +def dependency_rows( + binaries: list[Path], backend: str, abi_version: str, mlx_version: str +) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + install_names = sorted( + {name for binary in binaries for name in otool_dependencies(binary)} + ) + for install_name in install_names: + name, kind = dependency_name(install_name) + if name == "libmlx.dylib": + rows.append( + { + "bundled": True, + "kind": "library", + "linkage": "dynamic", + "name": name, + "role": "runtime", + "version": mlx_version, + } + ) + elif kind == "framework": + rows.append( + { + "bundled": False, + "kind": kind, + "linkage": "external", + "name": name, + "role": "external-runtime", + "version": abi_version, + } + ) + else: + rows.append( + { + "bundled": False, + "kind": kind, + "linkage": "dynamic", + "name": name, + "role": "runtime", + "version": abi_version, + } + ) + if backend == "mlx": + rows.append( + { + "bundled": True, + "kind": "library", + "linkage": "dynamic", + "name": "mlx.metallib", + "role": "runtime", + "version": mlx_version, + } + ) + names = [row["name"] for row in rows] + if len(names) != len(set(names)): + raise ValueError("Mach-O dependency inventory contains duplicate names") + return rows + + +def prepare_macos_metadata(args: argparse.Namespace) -> dict[str, Any]: + if args.artifact_id not in ARTIFACTS: + raise ValueError(f"unsupported macOS artifact {args.artifact_id!r}") + backend, required_channel = ARTIFACTS[args.artifact_id] + if args.channel != required_channel: + raise ValueError(f"{args.artifact_id} requires channel {required_channel}") + server = args.stage_dir / "bin/vllm-server" + if not server.is_file(): + raise ValueError("staged bin/vllm-server is missing") + flags = release_metadata.backend_flags( + release_metadata.parse_cache(args.build_dir / "CMakeCache.txt") + ) + mlx = backend == "mlx" + if flags["VLLM_CPP_METAL"] is not True or flags["VLLM_CPP_MLX"] is not mlx: + raise ValueError("configured Metal/MLX flags do not match the artifact") + if mlx: + if not args.mlx_version: + raise ValueError("MLX preview requires an exact MLX version") + for relative in ( + "lib/libmlx.dylib", + "lib/mlx.metallib", + ): + if not (args.stage_dir / relative).is_file(): + raise ValueError(f"MLX staged runtime is missing {relative}") + if args.mlx_license is None or not args.mlx_license.is_file(): + raise ValueError("MLX preview requires the exact installed distribution license") + elif flags["MLX_ROOT"] != "": + raise ValueError("native Metal artifact must not record an MLX root") + binaries = [server] + if mlx: + binaries.append(args.stage_dir / "lib/libmlx.dylib") + dependencies = dependency_rows(binaries, backend, args.abi_version, args.mlx_version) + runtime_command = "test_metal_backend" + if mlx: + runtime_command += " (MLX provider enabled)" + facts: dict[str, Any] = { + "artifact": { + "c_abi_version": args.c_abi_version, + "channel": args.channel, + "id": args.artifact_id, + "kind": "primary", + "static_boundary": "static-core", + "version": args.version, + }, + "backend": { + "flags": flags, + "gpu_driver_boundary": "external-host-never-bundled", + "name": backend, + }, + "build": { + "compiler": args.compiler, + "resolved_cmake_options": flags, + "source_clean": args.source_clean, + "source_commit": args.source_commit, + "test_commands": [runtime_command, "python3 scripts/validate-release-archive.py"], + "toolchain": args.toolchain, + }, + "dependencies": dependencies, + "evidence": { + "archive_smoke": release_metadata.passed( + "extracted vllm-server --help && --version", args.evidence_url + ), + "build": release_metadata.passed("cmake --build --target server", args.evidence_url), + "correctness": release_metadata.passed(runtime_command, args.evidence_url), + "dependency_audit": release_metadata.passed("file && otool -L && otool -l", args.evidence_url), + "performance": release_metadata.absent("release packaging makes no performance claim"), + "runtime": release_metadata.passed(runtime_command, args.evidence_url), + }, + "host": { + "abi": "macos", + "abi_version": args.abi_version, + "arch": "aarch64", + "os": "macos", + }, + "supply_chain": { + "archive_checksum": release_metadata.passed("shasum -a 256 ", args.evidence_url), + "licenses": release_metadata.passed("validate vllm.cpp and MLX licenses", args.evidence_url), + "provenance": release_metadata.passed("validate detached SLSA subject", args.evidence_url), + "sbom": release_metadata.passed("validate SPDX-2.3 inventory", args.evidence_url), + }, + } + schema = release_manifest.load_schema(args.repo_root / "release/manifest-v1.schema.json") + manifest = release_manifest.generate_manifest(facts, args.repo_root, schema) + output = args.output_dir + output.mkdir(parents=True, exist_ok=True) + (output / "release-manifest.json").write_text( + release_metadata.canonical_json(manifest), encoding="utf-8" + ) + version_values = { + "version": args.version, + "commit": args.source_commit, + "artifact_id": args.artifact_id, + "backend": backend, + "host_os": "macos", + "host_arch": "aarch64", + "host_abi": "macos", + "source_clean": "true" if args.source_clean else "false", + "c_abi_version": str(args.c_abi_version), + } + (output / "VERSION").write_text( + "".join(f"{key}={value}\n" for key, value in version_values.items()), + encoding="utf-8", + ) + bundled_files: list[tuple[str, Path, str]] = [] + if mlx: + bundled_files = [ + ("lib/libmlx.dylib", args.stage_dir / "lib/libmlx.dylib", "MIT"), + ("lib/mlx.metallib", args.stage_dir / "lib/mlx.metallib", "MIT"), + ] + (output / "sbom.spdx.json").write_text( + release_metadata.canonical_json( + release_metadata.spdx_document( + args.artifact_id, + args.version, + args.source_commit, + server, + dependencies, + bundled_files, + ) + ), + encoding="utf-8", + ) + notices = ["vllm.cpp release dependency notices", ""] + notices.extend( + f"- {row['name']} {row['version']} ({row['linkage']})" for row in dependencies + ) + (output / "THIRD_PARTY_NOTICES").write_text("\n".join(notices) + "\n", encoding="utf-8") + license_dir = output / "share/licenses/vllm.cpp" + license_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(args.repo_root / "LICENSE", license_dir / "LICENSE") + if mlx: + mlx_license_dir = output / "share/licenses/MLX" + mlx_license_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(args.mlx_license, mlx_license_dir / "LICENSE") + return manifest + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--repo-root", type=Path, default=SCRIPT_DIR.parent) + parser.add_argument("--build-dir", type=Path, required=True) + parser.add_argument("--stage-dir", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--artifact-id", required=True) + parser.add_argument("--channel", choices=("stable", "preview"), required=True) + parser.add_argument("--version", required=True) + parser.add_argument("--c-abi-version", type=int, required=True) + parser.add_argument("--source-commit", required=True) + parser.add_argument("--source-clean", action="store_true") + parser.add_argument("--abi-version", required=True) + parser.add_argument("--mlx-version", default="") + parser.add_argument("--mlx-license", type=Path) + parser.add_argument("--compiler", required=True) + parser.add_argument("--toolchain", required=True) + parser.add_argument("--evidence-url", required=True) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + prepare_macos_metadata(args) + except (OSError, json.JSONDecodeError, KeyError, ValueError, release_manifest.ManifestError) as exc: + print(f"macOS release metadata error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/release_manifest.py b/scripts/release_manifest.py index e0406b2e8..57ff0346e 100644 --- a/scripts/release_manifest.py +++ b/scripts/release_manifest.py @@ -57,8 +57,11 @@ "bits": { "portable-sse2": {"sse2"}, "sse2-f16c": {"sse2", "avx", "f16c", "osxsave"}, - "avx2-f16c": {"avx", "avx2", "f16c", "osxsave"}, - "avx512f": {"avx", "avx512f", "osxsave"}, + "avx2-f16c": {"sse2", "avx", "avx2", "f16c", "osxsave"}, + "avx512f": { + "sse2", "avx", "f16c", "avx2", "avx512f", "avx512bw", + "avx512vl", "osxsave", + }, }, "os_state": { "portable-sse2": set(), @@ -72,20 +75,35 @@ }, "aarch64": { "baseline": "portable-neon", - "tiers": ("portable-neon", "i8mm"), + "tiers": ("portable-neon", "dotprod", "i8mm"), "kernel_families": { "portable-neon": {"matmul-elem-f32-bf16-f16"}, + "dotprod": {"quant-dot-q8_0-q8_0"}, "i8mm": { "quant-dot-q4_0-q8_0-q4_K-q6_K", "quant-repack-q8_0", }, }, - "bits": {"portable-neon": {"neon"}, "i8mm": {"i8mm"}}, + "bits": { + "portable-neon": {"neon"}, + "dotprod": {"neon", "dotprod"}, + "i8mm": {"neon", "dotprod", "i8mm"}, + }, "os_state": { "portable-neon": {"linux": set(), "macos": set()}, + "dotprod": { + "linux": {"getauxval:AT_HWCAP:HWCAP_ASIMDDP"}, + "macos": {"sysctl:hw.optional.arm.FEAT_DotProd"}, + }, "i8mm": { - "linux": {"getauxval:AT_HWCAP2:HWCAP2_I8MM"}, - "macos": {"sysctl:hw.optional.arm.FEAT_I8MM"}, + "linux": { + "getauxval:AT_HWCAP:HWCAP_ASIMDDP", + "getauxval:AT_HWCAP2:HWCAP2_I8MM", + }, + "macos": { + "sysctl:hw.optional.arm.FEAT_DotProd", + "sysctl:hw.optional.arm.FEAT_I8MM", + }, }, }, }, @@ -312,6 +330,10 @@ def _artifact_policy(manifest: dict[str, Any]) -> list[str]: return [] artifact_id = artifact.get("id") name = backend.get("name") + if not isinstance(artifact.get("c_abi_version"), int) or isinstance( + artifact.get("c_abi_version"), bool + ) or artifact.get("c_abi_version", 0) <= 0: + return ["$.artifact.c_abi_version: must be a positive integer"] policies = { "linux-x86_64-glibc-cpu": ("linux", "x86_64", "glibc", "cpu", "static-core", {"preview", "stable"}), "linux-aarch64-glibc-cpu": ("linux", "aarch64", "glibc", "cpu", "static-core", {"preview", "stable"}), @@ -347,8 +369,11 @@ def _artifact_policy(manifest: dict[str, Any]) -> list[str]: def _backend_policy(manifest: dict[str, Any], repo_root: Path) -> list[str]: backend = manifest.get("backend", {}) + artifact = manifest.get("artifact", {}) if not isinstance(backend, dict): return [] + if not isinstance(artifact, dict): + artifact = {} name = backend.get("name") flags = backend.get("flags", {}) if not isinstance(flags, dict): @@ -364,6 +389,11 @@ def _backend_policy(manifest: dict[str, Any], repo_root: Path) -> list[str]: for flag, expected in expected_switches.items(): if flags.get(flag) is not expected: errors.append(f"$.backend.flags.{flag}: inconsistent backend flags for {name}") + expects_literal_static = artifact.get("static_boundary") == "literal-static" + if flags.get("VLLM_CPP_LITERAL_STATIC") is not expects_literal_static: + errors.append( + "$.backend.flags.VLLM_CPP_LITERAL_STATIC: must agree with the artifact static boundary" + ) if flags.get("VLLM_CPP_SERVER") is not True or flags.get("VLLM_CPP_BUILD_EXAMPLES") is not True: errors.append("$.backend.flags: release backend flags must build server and examples") if flags.get("VLLM_CPP_HIP_ARCHITECTURES") != []: diff --git a/scripts/release_metadata.py b/scripts/release_metadata.py new file mode 100755 index 000000000..327d13ffc --- /dev/null +++ b/scripts/release_metadata.py @@ -0,0 +1,379 @@ +#!/usr/bin/env python3 +"""Generate W9 CPU release manifest, VERSION, notices, licenses, and SPDX.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any + + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +import release_manifest # noqa: E402 + + +ARTIFACT_HOST = { + "linux-x86_64-glibc-cpu": ("linux", "x86_64", "glibc", "static-core"), + "linux-aarch64-glibc-cpu": ("linux", "aarch64", "glibc", "static-core"), + "linux-x86_64-musl-cpu-static": ( + "linux", + "x86_64", + "musl", + "literal-static", + ), +} +REQUIRED_FLAGS = ( + "MLX_ROOT", + "VLLM_CPP_BUILD_EXAMPLES", + "VLLM_CPP_BUILD_TESTS", + "VLLM_CPP_CUDA", + "VLLM_CPP_CUDA_ARCHITECTURES", + "VLLM_CPP_HIP", + "VLLM_CPP_HIP_ARCHITECTURES", + "VLLM_CPP_LITERAL_STATIC", + "VLLM_CPP_METAL", + "VLLM_CPP_MLX", + "VLLM_CPP_SERVER", + "VLLM_CPP_TRITON", + "VLLM_CPP_VULKAN", +) + + +def canonical_json(value: Any) -> str: + return json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n" + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def evidence(state: str, command: str, result: str, url: str, reason: str = "") -> dict[str, str]: + return { + "command": command, + "reason": reason, + "result": result, + "state": state, + "url": url, + } + + +def passed(command: str, url: str) -> dict[str, str]: + return evidence("passed", command, "exit 0", url) + + +def absent(reason: str) -> dict[str, str]: + return evidence("absent", "", "", "", reason) + + +def parse_cache(path: Path) -> dict[str, str]: + values: dict[str, str] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + if not line or line.startswith(("//", "#")) or "=" not in line or ":" not in line.split("=", 1)[0]: + continue + key_type, value = line.split("=", 1) + key, _ = key_type.split(":", 1) + values[key] = value + return values + + +def cmake_bool(value: str, name: str) -> bool: + if value in {"ON", "TRUE", "1", "YES"}: + return True + if value in {"OFF", "FALSE", "0", "NO", ""}: + return False + raise ValueError(f"release CMake flag {name} must resolve explicitly ON or OFF, got {value!r}") + + +def backend_flags(cache: dict[str, str]) -> dict[str, Any]: + missing = [name for name in REQUIRED_FLAGS if name not in cache] + if missing: + raise ValueError(f"CMake cache is missing release flags: {missing}") + flags: dict[str, Any] = {} + for name in REQUIRED_FLAGS: + value = cache[name] + if name == "MLX_ROOT": + flags[name] = value + elif name.endswith("ARCHITECTURES"): + flags[name] = [item for item in value.split(";") if item] + else: + flags[name] = cmake_bool(value, name) + return flags + + +def parse_needed(server: Path) -> list[str]: + result = subprocess.run( + ["readelf", "-dW", str(server)], text=True, capture_output=True, check=False + ) + if result.returncode != 0: + raise ValueError(f"readelf dependency inspection failed: {result.stderr.strip()}") + return re.findall(r"\(NEEDED\).*?\[([^]]+)\]", result.stdout) + + +def dependencies(server: Path, abi: str, abi_version: str) -> list[dict[str, Any]]: + if abi == "musl": + return [ + { + "bundled": True, + "kind": "library", + "linkage": "static", + "name": "musl-libc", + "role": "build-time", + "version": abi_version, + } + ] + needed = parse_needed(server) + if not needed: + raise ValueError("glibc release server has no recorded dynamic dependencies") + return [ + { + "bundled": False, + "kind": "library", + "linkage": "dynamic", + "name": name, + "role": "runtime", + "version": "system", + } + for name in needed + ] + + +def load_tier_report(path: Path, arch: str, stable: bool) -> tuple[list[dict[str, Any]], str, list[str]]: + report = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(report, dict) or report.get("schema") != "vllm.cpp.cpu-tier-report.v1": + raise ValueError("CPU tier report has the wrong schema") + policy = release_manifest.CPU_TIER_POLICY[arch] + rows = report.get("tiers") + if not isinstance(rows, dict) or list(rows) != list(policy["tiers"]): + raise ValueError("CPU tier report must contain every compiled tier in policy order") + selected = report.get("selected_tier") + if selected not in rows: + raise ValueError("CPU tier report selected_tier is not compiled") + commands = report.get("commands") + if not isinstance(commands, list) or not commands or any(not isinstance(item, str) or not item for item in commands): + raise ValueError("CPU tier report commands must be a non-empty string array") + compiled: list[dict[str, Any]] = [] + for name in policy["tiers"]: + execution = rows[name] + if not isinstance(execution, dict): + raise ValueError(f"CPU tier report {name} evidence must be an object") + if stable and execution.get("state") != "passed": + raise ValueError(f"stable CPU bundle requires passed execution evidence for {name}") + state_policy = policy["os_state"][name] + os_state = state_policy["linux"] if isinstance(state_policy, dict) else state_policy + compiled.append( + { + "execution_evidence": execution, + "kernel_families": sorted(policy["kernel_families"][name]), + "name": name, + "required_cpu_bits": sorted(policy["bits"][name]), + "required_os_state": sorted(os_state), + } + ) + return compiled, selected, commands + + +def spdx_document( + artifact_id: str, + version: str, + source_commit: str, + server: Path, + dependency_rows: list[dict[str, Any]], + bundled_files: list[tuple[str, Path, str]] | None = None, +) -> dict[str, Any]: + binary_digest = sha256(server) + packages = [ + { + "SPDXID": "SPDXRef-Package-vllm-cpp", + "downloadLocation": "NOASSERTION", + "filesAnalyzed": True, + "licenseConcluded": "Apache-2.0", + "licenseDeclared": "Apache-2.0", + "name": "vllm.cpp", + "versionInfo": version, + } + ] + for index, dependency in enumerate(dependency_rows): + packages.append( + { + "SPDXID": f"SPDXRef-Dependency-{index}", + "downloadLocation": "NOASSERTION", + "filesAnalyzed": False, + "licenseConcluded": "NOASSERTION", + "licenseDeclared": "NOASSERTION", + "name": dependency["name"], + "versionInfo": dependency["version"], + } + ) + files = [ + { + "SPDXID": "SPDXRef-File-vllm-server", + "checksums": [{"algorithm": "SHA256", "checksumValue": binary_digest}], + "copyrightText": "NOASSERTION", + "fileName": "./bin/vllm-server", + "licenseConcluded": "Apache-2.0", + } + ] + for index, (relative, path, license_id) in enumerate(bundled_files or (), 1): + files.append( + { + "SPDXID": f"SPDXRef-Bundled-File-{index}", + "checksums": [{"algorithm": "SHA256", "checksumValue": sha256(path)}], + "copyrightText": "NOASSERTION", + "fileName": f"./{relative}", + "licenseConcluded": license_id, + } + ) + return { + "SPDXID": "SPDXRef-DOCUMENT", + "creationInfo": { + "created": "1970-01-01T00:00:00Z", + "creators": ["Organization: vllm.cpp"], + }, + "dataLicense": "CC0-1.0", + "documentNamespace": f"https://github.com/mudler/vllm.cpp/spdx/{source_commit}/{artifact_id}", + "files": files, + "name": artifact_id, + "packages": packages, + "spdxVersion": "SPDX-2.3", + } + + +def prepare_cpu_metadata(args: argparse.Namespace) -> dict[str, Any]: + if args.artifact_id not in ARTIFACT_HOST: + raise ValueError(f"unsupported W9 CPU artifact {args.artifact_id!r}") + os_name, arch, abi, static_boundary = ARTIFACT_HOST[args.artifact_id] + stable = args.channel == "stable" + server = args.stage_dir / "bin/vllm-server" + if not server.is_file(): + raise ValueError("staged bin/vllm-server is missing") + flags = backend_flags(parse_cache(args.build_dir / "CMakeCache.txt")) + compiled_tiers, selected_tier, test_commands = load_tier_report( + args.tier_report, arch, stable + ) + dependency_rows = dependencies(server, abi, args.abi_version) + gate_command = " && ".join(test_commands) + facts: dict[str, Any] = { + "artifact": { + "c_abi_version": args.c_abi_version, + "channel": args.channel, + "id": args.artifact_id, + "kind": "primary", + "static_boundary": static_boundary, + "version": args.version, + }, + "backend": { + "flags": flags, + "gpu_driver_boundary": "not-applicable", + "name": "cpu", + }, + "build": { + "compiler": args.compiler, + "resolved_cmake_options": flags, + "source_clean": args.source_clean, + "source_commit": args.source_commit, + "test_commands": [*test_commands, "python3 scripts/validate-release-archive.py"], + "toolchain": args.toolchain, + }, + "cpu": { + "baseline": release_manifest.CPU_TIER_POLICY[arch]["baseline"], + "compiled_tiers": compiled_tiers, + "selected_tier": selected_tier, + }, + "dependencies": dependency_rows, + "evidence": { + "archive_smoke": passed("extracted vllm-server --help && --version", args.evidence_url), + "build": passed("cmake --build --target server", args.evidence_url), + "correctness": passed(gate_command, args.evidence_url), + "dependency_audit": passed("readelf -dW && ldd/lddtree", args.evidence_url), + "performance": absent("release packaging does not imply a performance claim"), + "runtime": passed(gate_command, args.evidence_url), + }, + "host": { + "abi": abi, + "abi_version": args.abi_version, + "arch": arch, + "os": os_name, + }, + "supply_chain": { + "archive_checksum": passed("sha256sum ", args.evidence_url), + "licenses": passed("validate THIRD_PARTY_NOTICES and share/licenses", args.evidence_url), + "provenance": passed("validate detached in-toto SLSA subject digest", args.evidence_url), + "sbom": passed("validate SPDX-2.3 server checksum and dependencies", args.evidence_url), + }, + } + schema = release_manifest.load_schema(args.repo_root / "release/manifest-v1.schema.json") + manifest = release_manifest.generate_manifest(facts, args.repo_root, schema) + output = args.output_dir + output.mkdir(parents=True, exist_ok=True) + (output / "release-manifest.json").write_text(canonical_json(manifest), encoding="utf-8") + version_values = { + "version": args.version, + "commit": args.source_commit, + "artifact_id": args.artifact_id, + "backend": "cpu", + "host_os": os_name, + "host_arch": arch, + "host_abi": abi, + "source_clean": "true" if args.source_clean else "false", + "c_abi_version": str(args.c_abi_version), + } + (output / "VERSION").write_text( + "".join(f"{key}={value}\n" for key, value in version_values.items()), + encoding="utf-8", + ) + (output / "sbom.spdx.json").write_text( + canonical_json(spdx_document(args.artifact_id, args.version, args.source_commit, server, dependency_rows)), + encoding="utf-8", + ) + notices = ["vllm.cpp release dependency notices", ""] + notices.extend(f"- {row['name']} {row['version']} ({row['linkage']})" for row in dependency_rows) + (output / "THIRD_PARTY_NOTICES").write_text("\n".join(notices) + "\n", encoding="utf-8") + license_dir = output / "share/licenses/vllm.cpp" + license_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(args.repo_root / "LICENSE", license_dir / "LICENSE") + return manifest + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--repo-root", type=Path, default=SCRIPT_DIR.parent) + parser.add_argument("--build-dir", type=Path, required=True) + parser.add_argument("--stage-dir", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--tier-report", type=Path, required=True) + parser.add_argument("--artifact-id", required=True) + parser.add_argument("--channel", choices=("stable", "preview", "experimental-preview"), required=True) + parser.add_argument("--version", required=True) + parser.add_argument("--c-abi-version", type=int, required=True) + parser.add_argument("--source-commit", required=True) + parser.add_argument("--source-clean", action="store_true") + parser.add_argument("--abi-version", required=True) + parser.add_argument("--compiler", required=True) + parser.add_argument("--toolchain", required=True) + parser.add_argument("--evidence-url", required=True) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + prepare_cpu_metadata(args) + except (OSError, json.JSONDecodeError, KeyError, ValueError, release_manifest.ManifestError) as exc: + print(f"release metadata error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/release_pipeline.py b/scripts/release_pipeline.py new file mode 100755 index 000000000..b9e81dedb --- /dev/null +++ b/scripts/release_pipeline.py @@ -0,0 +1,366 @@ +#!/usr/bin/env python3 +"""Plan and verify immutable W8 release workflow handoffs.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import subprocess +import sys +from pathlib import Path +from typing import Any + + +PLAN_SCHEMA = "vllm.cpp.release-plan.v1" +HANDOFF_SCHEMA = "vllm.cpp.release-handoff.v1" +MATRIX_SCHEMA = "vllm.cpp.release-matrix.v1" +RELEASE_INDEX_SCHEMA = "vllm.cpp.release-index.v1" +CHANNELS = {"stable", "preview", "experimental-preview"} +RELEASE_TAG = re.compile(r"v[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?") + + +def canonical_json(value: Any) -> str: + return json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n" + + +def write_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(canonical_json(value), encoding="utf-8") + + +def read_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as handle: + value = json.load(handle) + if not isinstance(value, dict): + raise ValueError(f"{path} must contain a JSON object") + return value + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def validate_matrix(matrix: dict[str, Any]) -> list[dict[str, Any]]: + if set(matrix) != {"artifacts", "release_ready", "retention", "schema"}: + raise ValueError("release matrix has unknown or missing fields") + if matrix.get("schema") != MATRIX_SCHEMA: + raise ValueError(f"release matrix schema must be {MATRIX_SCHEMA}") + if type(matrix.get("release_ready")) is not bool: + raise ValueError("release matrix release_ready must be boolean") + if matrix.get("retention") != { + "ci_artifacts_days": 7, + "github_release": "maintainer-deletion-only", + }: + raise ValueError("release matrix retention policy is invalid") + artifacts = matrix.get("artifacts") + if not isinstance(artifacts, list) or not artifacts: + raise ValueError("release matrix artifacts must be a non-empty array") + normalized: list[dict[str, Any]] = [] + for index, item in enumerate(artifacts): + if not isinstance(item, dict) or set(item) != {"id", "channel", "required"}: + raise ValueError(f"release matrix artifact {index} has unknown or missing fields") + artifact_id = item.get("id") + if not isinstance(artifact_id, str) or re.fullmatch(r"[a-z0-9][a-z0-9_.-]+", artifact_id) is None: + raise ValueError(f"release matrix artifact {index} has unsafe id") + if item.get("channel") not in CHANNELS or type(item.get("required")) is not bool: + raise ValueError(f"release matrix artifact {artifact_id} has invalid policy") + normalized.append(dict(item)) + ids = [item["id"] for item in normalized] + if len(ids) != len(set(ids)): + raise ValueError("release matrix artifact ids must be unique") + return normalized + + +def make_plan( + event: str, ref: str, source_sha: str, version: str, matrix: dict[str, Any] +) -> dict[str, Any]: + if re.fullmatch(r"[0-9a-f]{40}", source_sha) is None: + raise ValueError("source SHA must be a full lowercase 40-hex commit") + if re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?", version) is None: + raise ValueError("version must be a semantic version") + artifacts = validate_matrix(matrix) + if event == "workflow_dispatch": + if not ref.startswith("refs/heads/"): + raise ValueError("manual dry runs must resolve a branch ref") + release_tag = f"dry-run-{source_sha[:12]}" + publish = False + elif event == "push": + release_tag = f"v{version}" + if ref != f"refs/tags/{release_tag}": + raise ValueError(f"release tag must exactly equal {release_tag}") + publish = matrix["release_ready"] + else: + raise ValueError(f"unsupported release event {event!r}") + return { + "artifacts": artifacts, + "event": event, + "publish": publish, + "release_tag": release_tag, + "retention": matrix["retention"], + "schema": PLAN_SCHEMA, + "source_sha": source_sha, + "version": version, + } + + +def inventory_assets(plan: dict[str, Any], assets_dir: Path) -> list[dict[str, Any]]: + if not assets_dir.is_dir(): + raise ValueError(f"asset directory does not exist: {assets_dir}") + artifacts = plan.get("artifacts") + if not isinstance(artifacts, list): + raise ValueError("plan artifacts are invalid") + expected: dict[str, str] = {} + required_sets: dict[str, set[str]] = {} + for item in artifacts: + artifact_id = item["id"] + archive = f"{artifact_id}.tar.gz" + names = { + archive, + f"{archive}.sha256", + f"{archive}.provenance.json", + } + required_sets[artifact_id] = names + expected.update({name: artifact_id for name in names}) + actual_paths = sorted(assets_dir.iterdir(), key=lambda path: path.name) + for path in actual_paths: + if path.is_symlink() or not path.is_file(): + raise ValueError(f"release asset must be a regular file: {path.name}") + if path.name not in expected: + raise ValueError(f"release asset is not declared by the matrix: {path.name}") + actual = {path.name for path in actual_paths} + for item in artifacts: + names = required_sets[item["id"]] + present = names & actual + if present and present != names: + raise ValueError(f"release asset triplet is incomplete for {item['id']}") + if plan.get("publish") is True and item["required"] and present != names: + raise ValueError(f"publish-ready handoff is missing required artifact {item['id']}") + return [ + { + "artifact_id": expected[path.name], + "name": path.name, + "sha256": file_sha256(path), + "size": path.stat().st_size, + } + for path in actual_paths + ] + + +def handoff_value(plan_path: Path, assets_dir: Path) -> dict[str, Any]: + plan = read_json(plan_path) + if plan.get("schema") != PLAN_SCHEMA: + raise ValueError("handoff input is not a release plan") + return { + "artifacts": plan.get("artifacts"), + "files": inventory_assets(plan, assets_dir), + "plan_sha256": file_sha256(plan_path), + "publish": plan.get("publish"), + "release_tag": plan.get("release_tag"), + "retention": plan.get("retention"), + "schema": HANDOFF_SCHEMA, + "source_sha": plan.get("source_sha"), + } + + +def make_handoff(plan_path: Path, assets_dir: Path, output: Path) -> None: + write_json(output, handoff_value(plan_path, assets_dir)) + + +def verify_handoff( + plan_path: Path, + handoff_path: Path, + assets_dir: Path, + output: Path, + expected_sha: str, +) -> None: + plan = read_json(plan_path) + handoff = read_json(handoff_path) + expected = handoff_value(plan_path, assets_dir) + expected["source_sha"] = expected_sha + if handoff != expected or plan.get("source_sha") != expected_sha: + raise ValueError("release handoff does not match the immutable plan and workflow SHA") + write_json(output, {**handoff, "verified": True}) + + +def publish_release( + handoff_path: Path, + assets_dir: Path, + index_json: Path, + index_markdown: Path, + tag: str, +) -> None: + """Publish only the regular files authenticated by a verified handoff.""" + handoff = read_json(handoff_path) + if handoff.get("verified") is not True or handoff.get("publish") is not True: + raise ValueError("release publication requires a verified publish handoff") + if RELEASE_TAG.fullmatch(tag) is None or handoff.get("release_tag") != tag: + raise ValueError("release tag does not match the verified handoff") + if not assets_dir.is_dir(): + raise ValueError(f"asset directory does not exist: {assets_dir}") + files = handoff.get("files") + if not isinstance(files, list) or not files: + raise ValueError("verified handoff has no release files") + + expected: dict[str, dict[str, Any]] = {} + for item in files: + if not isinstance(item, dict) or set(item) < {"name", "sha256", "size"}: + raise ValueError("verified handoff file inventory is malformed") + name = item["name"] + if ( + not isinstance(name, str) + or not name + or Path(name).name != name + or name in expected + ): + raise ValueError("verified handoff contains an unsafe or duplicate file name") + expected[name] = item + + actual_paths = sorted(assets_dir.iterdir(), key=lambda path: path.name) + actual_names = {path.name for path in actual_paths} + if actual_names != set(expected): + raise ValueError("release assets do not exactly match the verified handoff") + for path in actual_paths: + item = expected[path.name] + if path.is_symlink() or not path.is_file(): + raise ValueError(f"release asset must be a regular file: {path.name}") + if file_sha256(path) != item.get("sha256") or path.stat().st_size != item.get("size"): + raise ValueError(f"release asset drifted after verification: {path.name}") + for path in (index_json, index_markdown): + if path.is_symlink() or not path.is_file(): + raise ValueError(f"release index must be a regular file: {path}") + index = read_json(index_json) + if ( + index.get("schema") != RELEASE_INDEX_SCHEMA + or index.get("release_tag") != tag + or index.get("source_sha") != handoff.get("source_sha") + ): + raise ValueError("release index identity does not match the verified handoff") + index_rows = index.get("artifacts") + if not isinstance(index_rows, list): + raise ValueError("release index artifacts must be an array") + expected_archives = { + name: item for name, item in expected.items() if name.endswith(".tar.gz") + } + indexed_archives: set[str] = set() + for row in index_rows: + if not isinstance(row, dict): + raise ValueError("release index artifact row is malformed") + archive = row.get("archive") + artifact_id = row.get("id") + if ( + not isinstance(archive, str) + or not isinstance(artifact_id, str) + or archive != f"{artifact_id}.tar.gz" + or archive not in expected_archives + or archive in indexed_archives + or row.get("sha256") != expected_archives[archive].get("sha256") + ): + raise ValueError("release index does not match the verified archive inventory") + indexed_archives.add(archive) + if indexed_archives != set(expected_archives): + raise ValueError("release index does not enumerate every verified archive") + markdown = index_markdown.read_text(encoding="utf-8") + required_markdown = [tag, str(handoff.get("source_sha", "")), *indexed_archives] + if any(not value or value not in markdown for value in required_markdown): + raise ValueError("release Markdown index does not match the verified handoff") + + subprocess.run( + [ + "gh", + "release", + "create", + tag, + *(str(path) for path in actual_paths), + str(index_json), + str(index_markdown), + "--verify-tag", + "--title", + tag, + "--notes-file", + str(index_markdown), + ], + check=True, + ) + + +def write_outputs(path: Path, values: dict[str, str | bool]) -> None: + with path.open("a", encoding="utf-8") as handle: + for key, value in values.items(): + rendered = str(value).lower() if isinstance(value, bool) else value + if "\n" in rendered: + raise ValueError("workflow outputs must be single-line values") + handle.write(f"{key}={rendered}\n") + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser() + commands = parser.add_subparsers(dest="command", required=True) + plan = commands.add_parser("plan") + plan.add_argument("--event", required=True) + plan.add_argument("--ref", required=True) + plan.add_argument("--sha", required=True) + plan.add_argument("--version", required=True) + plan.add_argument("--matrix", type=Path, required=True) + plan.add_argument("--output", type=Path, required=True) + plan.add_argument("--github-output", type=Path) + handoff = commands.add_parser("handoff") + handoff.add_argument("--plan", type=Path, required=True) + handoff.add_argument("--assets-dir", type=Path, required=True) + handoff.add_argument("--output", type=Path, required=True) + verify = commands.add_parser("verify") + verify.add_argument("--plan", type=Path, required=True) + verify.add_argument("--handoff", type=Path, required=True) + verify.add_argument("--assets-dir", type=Path, required=True) + verify.add_argument("--output", type=Path, required=True) + verify.add_argument("--sha", required=True) + publish = commands.add_parser("publish") + publish.add_argument("--handoff", type=Path, required=True) + publish.add_argument("--assets-dir", type=Path, required=True) + publish.add_argument("--index-json", type=Path, required=True) + publish.add_argument("--index-markdown", type=Path, required=True) + publish.add_argument("--tag", required=True) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + if args.command == "plan": + plan = make_plan(args.event, args.ref, args.sha, args.version, read_json(args.matrix)) + write_json(args.output, plan) + if args.github_output: + write_outputs( + args.github_output, + { + "artifact_name": f"release-plan-{args.sha}", + "publish": plan["publish"], + "release_tag": plan["release_tag"], + "version": plan["version"], + }, + ) + elif args.command == "handoff": + make_handoff(args.plan, args.assets_dir, args.output) + elif args.command == "verify": + verify_handoff(args.plan, args.handoff, args.assets_dir, args.output, args.sha) + else: + publish_release( + args.handoff, + args.assets_dir, + args.index_json, + args.index_markdown, + args.tag, + ) + except (OSError, json.JSONDecodeError, subprocess.CalledProcessError, ValueError) as exc: + print(f"release pipeline error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/run-cpu-release-gates.py b/scripts/run-cpu-release-gates.py new file mode 100755 index 000000000..64504e47d --- /dev/null +++ b/scripts/run-cpu-release-gates.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Execute every adaptive CPU tier plus a feature-poor refusal gate.""" + +from __future__ import annotations + +import argparse +import json +import os +import shlex +import subprocess +import sys +from pathlib import Path + + +TIER_ENV = ( + "VT_CPU_MATMUL_TIER", + "VT_CPU_Q8_DOT", + "VT_CPU_QUANT_MMLA", + "VT_CPU_QUANT_REPACK", +) + + +class GateError(RuntimeError): + pass + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--arch", choices=("x86_64", "aarch64"), required=True) + parser.add_argument("--tests-dir", type=Path, required=True) + parser.add_argument("--poor-emulator", type=Path, required=True) + parser.add_argument("--rich-emulator", type=Path) + parser.add_argument("--rich-cpu", default="max") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--evidence-url", required=True) + return parser.parse_args() + + +def executable(tests_dir: Path, name: str) -> Path: + path = (tests_dir / name).resolve() + if not path.is_file() or not os.access(path, os.X_OK): + raise GateError(f"missing test executable: {path}") + return path + + +def display_command(prefix: list[str], env_values: dict[str, str], binary: Path) -> str: + assignments = " ".join(f"{name}={shlex.quote(value)}" for name, value in env_values.items()) + command = shlex.join([*prefix, str(binary)]) + return f"{assignments} {command}".strip() + + +def run_command( + prefix: list[str], env_values: dict[str, str], binary: Path, expect_failure: bool = False +) -> tuple[str, str]: + command = display_command(prefix, env_values, binary) + environment = os.environ.copy() + for name in TIER_ENV: + environment.pop(name, None) + environment.update(env_values) + result = subprocess.run( + [*prefix, str(binary)], + env=environment, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + check=False, + ) + output = result.stdout.strip()[-4000:] + if expect_failure: + if result.returncode == 0: + raise GateError(f"feature-poor host unexpectedly accepted: {command}") + return f"expect-refusal {command}", f"exit {result.returncode}: {output}" + if result.returncode != 0: + raise GateError(f"CPU release gate failed ({result.returncode}): {command}\n{output}") + return command, f"exit 0: {output}" + + +def evidence(commands: list[str], results: list[str], url: str) -> dict[str, str]: + return { + "command": " && ".join(commands), + "reason": "", + "result": " | ".join(results), + "state": "passed", + "url": url, + } + + +def run_group( + prefix: list[str], rows: list[tuple[dict[str, str], Path]], url: str +) -> tuple[dict[str, str], list[str]]: + commands: list[str] = [] + results: list[str] = [] + for env_values, binary in rows: + command, result = run_command(prefix, env_values, binary) + commands.append(command) + results.append(result) + return evidence(commands, results, url), commands + + +def gate(args: argparse.Namespace) -> dict[str, object]: + tests_dir = args.tests_dir.resolve() + matmul = executable(tests_dir, "test_ops_matmul_elem") + rich_prefix = ( + [str(args.rich_emulator.resolve()), "-cpu", args.rich_cpu] + if args.rich_emulator is not None + else [] + ) + poor_model = "Nehalem" if args.arch == "x86_64" else "cortex-a53" + poor_prefix = [str(args.poor_emulator.resolve()), "-cpu", poor_model] + tiers: dict[str, dict[str, str]] = {} + commands: list[str] = [] + + if args.arch == "x86_64": + definitions = ( + ("portable-sse2", [({"VT_CPU_MATMUL_TIER": "portable"}, matmul), ({"VT_CPU_MATMUL_TIER": "sse2"}, matmul)]), + ("sse2-f16c", [({"VT_CPU_MATMUL_TIER": "sse2+f16c"}, matmul)]), + ("avx2-f16c", [({"VT_CPU_MATMUL_TIER": "avx2"}, matmul)]), + ("avx512f", [({"VT_CPU_MATMUL_TIER": "avx512"}, matmul)]), + ) + for name, rows in definitions: + tiers[name], ran = run_group(rich_prefix, rows, args.evidence_url) + commands.extend(ran) + for env_values in ( + {"VT_CPU_MATMUL_TIER": "portable"}, + {"VT_CPU_MATMUL_TIER": "sse2"}, + ): + command, _ = run_command(poor_prefix, env_values, matmul) + commands.append(command) + for env_values in ( + {"VT_CPU_MATMUL_TIER": "sse2+f16c"}, + {"VT_CPU_MATMUL_TIER": "avx2"}, + ): + command, _ = run_command(poor_prefix, env_values, matmul, expect_failure=True) + commands.append(command) + selected_tier = "avx512f" + else: + cpu_isa = executable(tests_dir, "test_cpu_isa_arm") + quant_dot = executable(tests_dir, "test_ops_quant_dot") + quant_repack = executable(tests_dir, "test_ops_quant_repack") + definitions = ( + ( + "portable-neon", + [ + ({}, cpu_isa), + ({"VT_CPU_MATMUL_TIER": "portable"}, matmul), + ({"VT_CPU_MATMUL_TIER": "neon"}, matmul), + ({"VT_CPU_Q8_DOT": "portable", "VT_CPU_QUANT_MMLA": "portable"}, quant_dot), + ({"VT_CPU_QUANT_REPACK": "portable"}, quant_repack), + ], + ), + ("dotprod", [({"VT_CPU_Q8_DOT": "sdot"}, quant_dot)]), + ( + "i8mm", + [ + ({"VT_CPU_QUANT_MMLA": "i8mm"}, quant_dot), + ({"VT_CPU_QUANT_REPACK": "i8mm"}, quant_repack), + ], + ), + ) + for name, rows in definitions: + tiers[name], ran = run_group(rich_prefix, rows, args.evidence_url) + commands.extend(ran) + poor_pass = ( + ({}, cpu_isa), + ({"VT_CPU_MATMUL_TIER": "portable"}, matmul), + ({"VT_CPU_MATMUL_TIER": "neon"}, matmul), + ({"VT_CPU_Q8_DOT": "portable", "VT_CPU_QUANT_MMLA": "portable"}, quant_dot), + ({"VT_CPU_QUANT_REPACK": "portable"}, quant_repack), + ) + for env_values, binary in poor_pass: + command, _ = run_command(poor_prefix, env_values, binary) + commands.append(command) + poor_refuse = ( + ({"VT_CPU_Q8_DOT": "sdot"}, quant_dot), + ({"VT_CPU_QUANT_MMLA": "i8mm"}, quant_dot), + ({"VT_CPU_QUANT_REPACK": "i8mm"}, quant_repack), + ) + for env_values, binary in poor_refuse: + command, _ = run_command(poor_prefix, env_values, binary, expect_failure=True) + commands.append(command) + selected_tier = "i8mm" + + return { + "commands": commands, + "schema": "vllm.cpp.cpu-tier-report.v1", + "selected_tier": selected_tier, + "tiers": tiers, + } + + +def main() -> int: + args = parse_args() + try: + report = gate(args) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2, sort_keys=False) + "\n", encoding="utf-8") + except (GateError, OSError) as exc: + print(f"CPU release gate error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/validate-release-archive.py b/scripts/validate-release-archive.py new file mode 100755 index 000000000..e5b56f8ea --- /dev/null +++ b/scripts/validate-release-archive.py @@ -0,0 +1,660 @@ +#!/usr/bin/env python3 +"""Validate W7 release bytes after safe extraction, never from a build tree.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile +from pathlib import Path, PurePosixPath +from typing import Any + + +SCRIPT_DIR = Path(__file__).resolve().parent +sys.path.insert(0, str(SCRIPT_DIR)) +import release_manifest # noqa: E402 + + +PRIMARY_CUDA_SMS = release_manifest.PRIMARY_CUDA_SMS +AOT_SMS = tuple(sm for sm in PRIMARY_CUDA_SMS if release_manifest.AOT_AVAILABILITY[sm]) +REQUIRED_FILES = { + "VERSION", + "THIRD_PARTY_NOTICES", + "bin/vllm-server", + "release-manifest.json", + "sbom.spdx.json", +} +VERSION_FIELDS = { + "version", + "commit", + "artifact_id", + "backend", + "host_os", + "host_arch", + "host_abi", + "source_clean", + "c_abi_version", +} +FORBIDDEN_SUFFIXES = { + ".a", ".c", ".cc", ".cmake", ".cpp", ".cu", ".h", ".hpp", ".o", ".obj", ".py", ".pyc" +} +SECRET_NAMES = { + ".env", "credentials", "id_dsa", "id_ed25519", "id_rsa", "known_hosts" +} + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def load_json(path: Path, label: str) -> dict[str, Any]: + try: + with path.open(encoding="utf-8") as handle: + value = json.load(handle, object_pairs_hook=release_manifest._reject_duplicate_keys) + except (OSError, json.JSONDecodeError, release_manifest.ManifestError) as exc: + raise ValueError(f"{label} is not valid JSON: {exc}") from exc + if not isinstance(value, dict): + raise ValueError(f"{label} must be a JSON object") + return value + + +def validate_checksum(archive: Path, checksum_path: Path) -> tuple[str, list[str]]: + actual = sha256(archive) + errors: list[str] = [] + try: + lines = checksum_path.read_text(encoding="utf-8").splitlines() + except OSError as exc: + return actual, [f"checksum sidecar cannot be read: {exc}"] + expected_line = f"{actual} {archive.name}" + if lines != [expected_line]: + errors.append("checksum sidecar must contain the exact final archive SHA256 and filename") + return actual, errors + + +def validate_provenance( + path: Path, archive: Path, digest: str, manifest: dict[str, Any] | None = None +) -> list[str]: + try: + statement = load_json(path, "provenance sidecar") + except ValueError as exc: + return [str(exc)] + errors: list[str] = [] + if statement.get("_type") != "https://in-toto.io/Statement/v1": + errors.append("provenance _type must be the in-toto Statement v1 URI") + if statement.get("predicateType") != "https://slsa.dev/provenance/v1": + errors.append("provenance predicateType must be SLSA provenance v1") + subject = statement.get("subject") + expected_subject = [{"name": archive.name, "digest": {"sha256": digest}}] + if subject != expected_subject: + errors.append("provenance subject must name and digest the exact final archive") + if manifest is not None: + parameters = ( + statement.get("predicate", {}) + .get("buildDefinition", {}) + .get("externalParameters", {}) + ) + expected = { + "artifact_id": manifest.get("artifact", {}).get("id"), + "source_commit": manifest.get("build", {}).get("source_commit"), + } + if parameters != expected: + errors.append("provenance build parameters disagree with the release manifest") + return errors + + +def safe_extract(archive: Path, destination: Path) -> list[str]: + errors: list[str] = [] + try: + with tarfile.open(archive, "r:*") as bundle: + members = bundle.getmembers() + for member in members: + name = PurePosixPath(member.name) + if name.is_absolute() or ".." in name.parts or member.name in {"", "."}: + errors.append(f"unsafe archive path: {member.name!r}") + continue + if member.islnk(): + errors.append(f"hard links are not permitted in release archives: {member.name}") + continue + if member.issym(): + target = PurePosixPath(member.linkname) + if target.is_absolute() or ".." in target.parts or name.parts[0] != "lib": + errors.append(f"unsafe archive symlink: {member.name} -> {member.linkname}") + continue + if not (member.isdir() or member.isfile() or member.issym()): + errors.append(f"unsupported archive member type: {member.name}") + if errors: + return errors + bundle.extractall(destination, filter="data") + except (OSError, tarfile.TarError) as exc: + return [f"archive cannot be extracted: {exc}"] + return errors + + +def allowed_file(relative: str) -> bool: + if relative in REQUIRED_FILES: + return True + return relative.startswith("lib/") or relative.startswith("share/licenses/") + + +def scan_file(path: Path, needles: list[bytes]) -> tuple[set[bytes], bool]: + """Scan large release binaries without reading them wholly into memory.""" + found: set[bytes] = set() + aws_key = False + overlap = max([24, *(len(needle) for needle in needles)], default=24) - 1 + tail = b"" + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + data = tail + chunk + found.update(needle for needle in needles if needle and needle in data) + aws_key = aws_key or re.search(rb"AKIA[0-9A-Z]{16}", data) is not None + tail = data[-overlap:] if overlap else b"" + return found, aws_key + + +def validate_contents(root: Path, forbidden_paths: list[str]) -> list[str]: + errors: list[str] = [] + files = {path.relative_to(root).as_posix(): path for path in root.rglob("*") if not path.is_dir()} + missing = sorted(REQUIRED_FILES - files.keys()) + if missing: + errors.append(f"archive is missing required files: {missing}") + for relative, path in sorted(files.items()): + if not allowed_file(relative): + errors.append(f"archive contains undeclared path: {relative}") + if path.suffix.lower() in FORBIDDEN_SUFFIXES: + errors.append(f"archive contains source/object file: {relative}") + if path.name.lower() in SECRET_NAMES or path.suffix.lower() in {".key", ".pem"}: + errors.append(f"archive contains credential-like path: {relative}") + if path.is_symlink(): + continue + encoded_paths = [os.fsencode(forbidden) for forbidden in forbidden_paths if forbidden] + found_paths, aws_key = scan_file(path, encoded_paths) + for forbidden in forbidden_paths: + if forbidden and os.fsencode(forbidden) in found_paths: + errors.append(f"archive file {relative} embeds forbidden build path {forbidden!r}") + if aws_key: + errors.append(f"archive file {relative} embeds an AWS access-key-shaped credential") + server = root / "bin/vllm-server" + if server.exists() and not os.access(server, os.X_OK): + errors.append("bin/vllm-server is not executable") + license_files = [path for path in files if path.startswith("share/licenses/")] + if not license_files: + errors.append("archive has no license files under share/licenses/") + return errors + + +def parse_version(path: Path) -> tuple[dict[str, str], list[str]]: + values: dict[str, str] = {} + errors: list[str] = [] + try: + lines = path.read_text(encoding="utf-8").splitlines() + except OSError as exc: + return values, [f"VERSION cannot be read: {exc}"] + for number, line in enumerate(lines, 1): + if not line or "=" not in line: + errors.append(f"VERSION line {number} must be non-empty key=value") + continue + key, value = line.split("=", 1) + if key in values: + errors.append(f"VERSION duplicates field {key!r}") + values[key] = value + if set(values) != VERSION_FIELDS: + errors.append(f"VERSION fields {sorted(values)} != required {sorted(VERSION_FIELDS)}") + if values.get("source_clean") not in {"true", "false"}: + errors.append("VERSION source_clean must be true or false") + if not values.get("c_abi_version", "").isdigit(): + errors.append("VERSION c_abi_version must be a decimal integer") + return values, errors + + +def validate_version(values: dict[str, str], manifest: dict[str, Any]) -> list[str]: + artifact = manifest.get("artifact", {}) + build = manifest.get("build", {}) + backend = manifest.get("backend", {}) + host = manifest.get("host", {}) + expected = { + "version": artifact.get("version"), + "commit": build.get("source_commit"), + "artifact_id": artifact.get("id"), + "backend": backend.get("name"), + "host_os": host.get("os"), + "host_arch": host.get("arch"), + "host_abi": host.get("abi"), + "source_clean": "true" if build.get("source_clean") is True else "false", + "c_abi_version": str(artifact.get("c_abi_version", "")), + } + return [ + f"VERSION {key}={values.get(key)!r} disagrees with manifest value {value!r}" + for key, value in expected.items() + if values.get(key) != value + ] + + +def validate_sbom(path: Path, root: Path, manifest: dict[str, Any]) -> list[str]: + try: + sbom = load_json(path, "SPDX SBOM") + except ValueError as exc: + return [str(exc)] + errors: list[str] = [] + if sbom.get("spdxVersion") != "SPDX-2.3": + errors.append("SBOM spdxVersion must be SPDX-2.3") + if sbom.get("SPDXID") != "SPDXRef-DOCUMENT": + errors.append("SBOM document SPDXID must be SPDXRef-DOCUMENT") + if sbom.get("dataLicense") != "CC0-1.0": + errors.append("SBOM dataLicense must be CC0-1.0") + if sbom.get("name") != manifest.get("artifact", {}).get("id"): + errors.append("SBOM name disagrees with manifest artifact id") + files = sbom.get("files") + if not isinstance(files, list): + return errors + ["SBOM files must be an array"] + by_name = {item.get("fileName"): item for item in files if isinstance(item, dict)} + shipped = [root / "bin/vllm-server"] + lib_dir = root / "lib" + if lib_dir.is_dir(): + shipped.extend(path for path in lib_dir.rglob("*") if path.is_file() and not path.is_symlink()) + for shipped_path in shipped: + relative = "./" + shipped_path.relative_to(root).as_posix() + item = by_name.get(relative) + if not isinstance(item, dict): + errors.append(f"SBOM must inventory {relative}") + continue + recorded = { + checksum.get("checksumValue") + for checksum in item.get("checksums", []) + if isinstance(checksum, dict) and checksum.get("algorithm") == "SHA256" + } + if recorded != {sha256(shipped_path)}: + errors.append(f"SBOM SHA256 does not match extracted bytes for {relative}") + return errors + + +def parse_elf_needed(dynamic_output: str) -> list[str]: + return re.findall(r"\(NEEDED\).*?\[([^]]+)\]", dynamic_output) + + +def parse_elf_rpaths(dynamic_output: str) -> list[str]: + values = re.findall(r"\((?:RPATH|RUNPATH)\).*?\[([^]]*)\]", dynamic_output) + return [entry for value in values for entry in value.split(":") if entry] + + +def parse_elf_interpreter(program_output: str) -> str: + match = re.search(r"Requesting program interpreter:\s*([^]]+)\]", program_output) + return match.group(1) if match else "" + + +def validate_linux_dynamic( + manifest: dict[str, Any], + needed: list[str], + rpaths: list[str], + interpreter: str, + forbidden_paths: list[str], +) -> list[str]: + errors: list[str] = [] + declared = { + item.get("name") + for item in manifest.get("dependencies", []) + if ( + isinstance(item, dict) + and item.get("linkage") == "dynamic" + and not str(item.get("name", "")).endswith(".metallib") + ) + } + literal_static = manifest.get("artifact", {}).get("static_boundary") == "literal-static" + if literal_static and (needed or rpaths or interpreter): + errors.append("literal-static artifact has an ELF interpreter, dependency, or RPATH") + for dependency in needed: + if dependency not in declared: + errors.append(f"undeclared ELF dependency: {dependency}") + for dependency in sorted(declared - set(needed)): + errors.append(f"declared dynamic dependency is not linked: {dependency}") + for entry in rpaths: + if entry.startswith("/"): + errors.append(f"absolute ELF RPATH/RUNPATH is forbidden: {entry}") + if any(path and path in entry for path in forbidden_paths): + errors.append(f"ELF RPATH/RUNPATH contains a forbidden build path: {entry}") + if not (entry.startswith("$ORIGIN") or entry.startswith("${ORIGIN}")): + errors.append(f"ELF RPATH/RUNPATH must be relative to the extracted bundle: {entry}") + host = manifest.get("host", {}) + if not literal_static and host.get("abi") == "glibc" and interpreter and "ld-linux" not in interpreter: + errors.append(f"glibc artifact has unexpected ELF interpreter: {interpreter}") + if not literal_static and host.get("abi") == "musl" and "ld-musl" not in interpreter: + errors.append(f"musl artifact has unexpected ELF interpreter: {interpreter!r}") + return errors + + +def macho_dependency_name(install_name: str) -> str: + for component in install_name.split("/"): + if component.endswith(".framework"): + return component + return Path(install_name).name + + +def validate_macho_dynamic( + manifest: dict[str, Any], + dependencies: list[str], + rpaths: list[str], + forbidden_paths: list[str], +) -> list[str]: + errors: list[str] = [] + declared_dynamic = { + item.get("name") + for item in manifest.get("dependencies", []) + if ( + isinstance(item, dict) + and item.get("linkage") == "dynamic" + and not str(item.get("name", "")).endswith(".metallib") + ) + } + declared_external = { + item.get("name") + for item in manifest.get("dependencies", []) + if isinstance(item, dict) and item.get("linkage") == "external" + } + actual_dynamic: set[str] = set() + actual_external: set[str] = set() + for install_name in dependencies: + name = macho_dependency_name(install_name) + if name.endswith(".framework"): + actual_external.add(name) + else: + actual_dynamic.add(name) + allowed = install_name.startswith(("/usr/lib/", "/System/Library/", "@rpath/", "@loader_path/")) + if not allowed: + errors.append(f"forbidden Mach-O install name: {install_name}") + if any(path and path in install_name for path in forbidden_paths): + errors.append(f"Mach-O install name contains forbidden build path: {install_name}") + for name in sorted(actual_dynamic - declared_dynamic): + errors.append(f"undeclared Mach-O dependency: {name}") + for name in sorted(declared_dynamic - actual_dynamic): + errors.append(f"declared dynamic dependency is not linked: {name}") + for name in sorted(actual_external - declared_external): + errors.append(f"undeclared Mach-O framework dependency: {name}") + for name in sorted(declared_external - actual_external): + errors.append(f"declared external framework is not linked: {name}") + for entry in rpaths: + if not entry.startswith("@loader_path/"): + errors.append(f"Mach-O RPATH must be relative to the extracted bundle: {entry}") + if any(path and path in entry for path in forbidden_paths): + errors.append(f"Mach-O RPATH contains a forbidden build path: {entry}") + return errors + + +def parse_otool_dependencies(output: str) -> list[str]: + lines = output.splitlines()[1:] + return [line.strip().split(" (", 1)[0] for line in lines if line.strip()] + + +def parse_otool_rpaths(output: str) -> list[str]: + lines = output.splitlines() + rpaths: list[str] = [] + for index, line in enumerate(lines): + if line.strip() == "cmd LC_RPATH": + for candidate in lines[index + 1:index + 5]: + match = re.match(r"\s*path\s+(\S+)\s+\(offset", candidate) + if match: + rpaths.append(match.group(1)) + break + return rpaths + + +def validate_macho_install_id(install_id: str, forbidden_paths: list[str]) -> list[str]: + if not install_id.startswith(("@rpath/", "@loader_path/")): + return [f"bundled Mach-O install ID must be relative: {install_id}"] + if any(path and path in install_id for path in forbidden_paths): + return [f"bundled Mach-O install ID contains a forbidden build path: {install_id}"] + return [] + + +def validate_cuda_inventory( + manifest: dict[str, Any], images: list[str], symbols: list[str] +) -> list[str]: + if manifest.get("backend", {}).get("name") != "cuda": + return [] + errors: list[str] = [] + image_sms = {match.group(1) for image in images if (match := re.search(r"sm_([0-9]+a?)", image))} + for sm in PRIMARY_CUDA_SMS: + if sm not in image_sms: + errors.append(f"CUDA archive is missing sm_{sm} device code") + for sm in sorted(image_sms - set(PRIMARY_CUDA_SMS)): + errors.append(f"CUDA archive contains undeclared sm_{sm} device code") + symbol_text = "\n".join(symbols) + for sm in AOT_SMS: + if f"vt_aot_sm_{sm}_" not in symbol_text: + errors.append(f"CUDA archive is missing exact AOT namespace for sm_{sm}") + for sm in set(PRIMARY_CUDA_SMS) - set(AOT_SMS): + if f"vt_aot_sm_{sm}_" in symbol_text: + errors.append(f"CUDA archive fabricates unavailable AOT namespace for sm_{sm}") + return errors + + +def run(command: list[str]) -> tuple[int, str]: + result = subprocess.run(command, text=True, capture_output=True, check=False) + return result.returncode, result.stdout + result.stderr + + +def inspect_linux( + server: Path, + manifest: dict[str, Any], + forbidden_paths: list[str], + skip_version_smoke: bool, +) -> list[str]: + errors: list[str] = [] + for tool in ("file", "readelf", "ldd"): + if shutil.which(tool) is None: + errors.append(f"required Linux archive inspector is unavailable: {tool}") + if errors: + return errors + file_rc, file_output = run(["file", "-b", str(server)]) + if file_rc != 0 or "ELF" not in file_output: + errors.append(f"server is not an inspectable ELF executable: {file_output.strip()}") + expected_machine = "x86-64" if manifest.get("host", {}).get("arch") == "x86_64" else "aarch64" + if expected_machine.lower() not in file_output.lower(): + errors.append(f"ELF host architecture does not match manifest: {file_output.strip()}") + dynamic_rc, dynamic = run(["readelf", "-dW", str(server)]) + program_rc, program = run(["readelf", "-lW", str(server)]) + if dynamic_rc != 0 or program_rc != 0: + errors.append("readelf could not inspect the extracted server") + else: + errors.extend( + validate_linux_dynamic( + manifest, + parse_elf_needed(dynamic), + parse_elf_rpaths(dynamic), + parse_elf_interpreter(program), + forbidden_paths, + ) + ) + ldd_rc, ldd_output = run(["ldd", str(server)]) + literal_static = manifest.get("artifact", {}).get("static_boundary") == "literal-static" + static_ldd = "not a dynamic executable" in ldd_output or "statically linked" in ldd_output + if literal_static: + if not static_ldd: + errors.append(f"literal-static server has a dynamic ldd result: {ldd_output.strip()}") + elif ldd_rc != 0 or "not found" in ldd_output: + errors.append(f"ldd found a missing dependency: {ldd_output.strip()}") + for forbidden in forbidden_paths: + if forbidden and forbidden in ldd_output: + errors.append(f"ldd resolves a dependency through forbidden path {forbidden!r}") + help_rc, help_output = run([str(server), "--help"]) + if help_rc != 0 or "usage" not in help_output.lower(): + errors.append("extracted vllm-server --help smoke failed") + if not skip_version_smoke: + version_rc, version_output = run([str(server), "--version"]) + expected_version = manifest.get("artifact", {}).get("version", "") + version_record = parse_version(server.parent.parent / "VERSION")[0] + expected_abi = version_record.get("c_abi_version", "") + if ( + version_rc != 0 + or f"vllm.cpp {expected_version}" not in version_output + or f"c-abi={expected_abi}" not in version_output + ): + errors.append("extracted vllm-server --version disagrees with VERSION/manifest") + if manifest.get("backend", {}).get("name") == "cuda": + if shutil.which("cuobjdump") is None or shutil.which("nm") is None: + errors.append("CUDA archive validation requires cuobjdump and nm") + else: + cuda_rc, cuda_output = run(["cuobjdump", "--list-elf", str(server)]) + nm_rc, nm_output = run(["nm", "-a", str(server)]) + if cuda_rc != 0 or nm_rc != 0: + errors.append("CUDA archive inspectors could not read the extracted server") + else: + errors.extend(validate_cuda_inventory(manifest, cuda_output.splitlines(), nm_output.splitlines())) + return errors + + +def inspect_macos( + root: Path, + server: Path, + manifest: dict[str, Any], + forbidden_paths: list[str], + skip_version_smoke: bool, +) -> list[str]: + errors: list[str] = [] + for tool in ("file", "otool"): + if shutil.which(tool) is None: + errors.append(f"required macOS archive inspector is unavailable: {tool}") + if errors: + return errors + file_rc, file_output = run(["file", "-b", str(server)]) + if file_rc != 0 or "Mach-O" not in file_output or "arm64" not in file_output: + errors.append(f"server is not a Mach-O arm64 executable: {file_output.strip()}") + binaries = [server] + if manifest.get("backend", {}).get("name") == "mlx": + binaries.append(root / "lib/libmlx.dylib") + dependencies: set[str] = set() + rpaths: set[str] = set() + for binary in binaries: + binary_rc, binary_output = run(["file", "-b", str(binary)]) + deps_rc, deps_output = run(["otool", "-L", str(binary)]) + load_rc, load_output = run(["otool", "-l", str(binary)]) + if binary_rc != 0 or "Mach-O" not in binary_output or "arm64" not in binary_output: + errors.append(f"bundled executable/library is not Mach-O arm64: {binary}") + if deps_rc != 0 or load_rc != 0: + errors.append(f"otool could not inspect extracted Mach-O file: {binary}") + continue + dependencies.update(parse_otool_dependencies(deps_output)) + rpaths.update(parse_otool_rpaths(load_output)) + if binary.suffix == ".dylib": + id_rc, id_output = run(["otool", "-D", str(binary)]) + install_ids = [line.strip() for line in id_output.splitlines()[1:] if line.strip()] + if id_rc != 0 or len(install_ids) != 1: + errors.append(f"bundled dylib has no unique install ID: {binary}") + else: + errors.extend(validate_macho_install_id(install_ids[0], forbidden_paths)) + errors.extend( + validate_macho_dynamic( + manifest, + sorted(dependencies), + sorted(rpaths), + forbidden_paths, + ) + ) + if manifest.get("backend", {}).get("name") == "mlx": + for relative in ("lib/libmlx.dylib", "lib/mlx.metallib"): + if not (root / relative).is_file(): + errors.append(f"MLX archive is missing bundled {relative}") + help_rc, help_output = run([str(server), "--help"]) + if help_rc != 0 or "usage" not in help_output.lower(): + errors.append("extracted vllm-server --help smoke failed") + if not skip_version_smoke: + version_rc, version_output = run([str(server), "--version"]) + expected_version = manifest.get("artifact", {}).get("version", "") + version_record = parse_version(root / "VERSION")[0] + expected_abi = version_record.get("c_abi_version", "") + if ( + version_rc != 0 + or f"vllm.cpp {expected_version}" not in version_output + or f"c-abi={expected_abi}" not in version_output + ): + errors.append("extracted vllm-server --version disagrees with VERSION/manifest") + return errors + + +def validate_release(args: argparse.Namespace) -> list[str]: + archive = args.archive.resolve() + digest, errors = validate_checksum(archive, args.checksum.resolve()) + errors.extend(validate_provenance(args.provenance.resolve(), archive, digest)) + if errors: + return errors + with tempfile.TemporaryDirectory(prefix="vllm-release-validate-") as temporary: + extracted = Path(temporary) / "extracted" + extracted.mkdir() + errors.extend(safe_extract(archive, extracted)) + if errors: + return errors + forbidden = [str(Path(value).resolve()) for value in args.forbid_path] + errors.extend(validate_contents(extracted, forbidden)) + manifest_path = extracted / "release-manifest.json" + if not manifest_path.is_file(): + return errors + try: + manifest = load_json(manifest_path, "release manifest") + schema = release_manifest.load_schema(args.schema.resolve()) + except (ValueError, OSError, release_manifest.ManifestError) as exc: + return errors + [str(exc)] + errors.extend(release_manifest.validate_manifest(manifest, schema, args.repo_root.resolve())) + values, version_errors = parse_version(extracted / "VERSION") + errors.extend(version_errors) + errors.extend(validate_version(values, manifest)) + errors.extend(validate_sbom(extracted / "sbom.spdx.json", extracted, manifest)) + errors.extend(validate_provenance(args.provenance.resolve(), archive, digest, manifest)) + if manifest.get("host", {}).get("os") == "linux": + errors.extend( + inspect_linux( + extracted / "bin/vllm-server", + manifest, + forbidden, + args.skip_version_smoke, + ) + ) + elif manifest.get("host", {}).get("os") == "macos": + errors.extend( + inspect_macos( + extracted, + extracted / "bin/vllm-server", + manifest, + forbidden, + args.skip_version_smoke, + ) + ) + else: + errors.append("release archive has an unsupported host OS") + return errors + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--archive", type=Path, required=True) + parser.add_argument("--checksum", type=Path, required=True) + parser.add_argument("--provenance", type=Path, required=True) + parser.add_argument("--repo-root", type=Path, default=SCRIPT_DIR.parent) + parser.add_argument("--schema", type=Path, default=SCRIPT_DIR.parent / "release/manifest-v1.schema.json") + parser.add_argument("--forbid-path", action="append", default=[]) + parser.add_argument("--skip-version-smoke", action="store_true") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + errors = validate_release(args) + if errors: + print("release archive validation FAILED:", file=sys.stderr) + for error in errors: + print(f" - {error}", file=sys.stderr) + return 1 + print("release archive validation: extracted bytes, dependencies, metadata, and supply chain OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index 59c9749ee..e615d39f1 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -55,6 +55,7 @@ #include #endif +#include "vllm.h" #include "vllm/config/device.h" #include "vllm/config/kv_transfer.h" #include "vllm/config/scheduler.h" @@ -251,7 +252,8 @@ struct Args { " [--tool-call-parser |auto|none]\n" " [--reasoning-parser |auto|none]\n" " [--kv-transfer-config '']\n" - " [--speculative-config '']\n"; + " [--speculative-config '']\n" + " [--version]\n"; std::exit(code); } @@ -399,6 +401,10 @@ Args ParseArgs(int argc, char** argv) { a.kv_transfer_config = NextArg(argc, argv, i, argv[0]); } else if (flag == "--speculative-config") { a.speculative_config = NextArg(argc, argv, i, argv[0]); + } else if (flag == "--version") { + std::cout << "vllm.cpp " << vllm::Version() + << " c-abi=" << VLLM_ABI_VERSION << "\n"; + std::exit(0); } else if (flag == "-h" || flag == "--help") { Usage(argv[0], 0); } else { diff --git a/src/vt/cpu/cpu_isa_arm.cpp b/src/vt/cpu/cpu_isa_arm.cpp new file mode 100644 index 000000000..20f99415a --- /dev/null +++ b/src/vt/cpu/cpu_isa_arm.cpp @@ -0,0 +1,157 @@ +#include "vt/cpu/cpu_isa_arm.h" + +#include + +#if defined(__aarch64__) +#if defined(__linux__) +#include +#include +#elif defined(__APPLE__) +#include +#endif +#endif + +#if defined(__linux__) && !defined(HWCAP_ASIMD) +#define HWCAP_ASIMD (1UL << 1) +#endif +#if defined(__linux__) && !defined(HWCAP_ASIMDDP) +#define HWCAP_ASIMDDP (1UL << 20) +#endif +#if defined(__linux__) && !defined(HWCAP2_I8MM) +#define HWCAP2_I8MM (1UL << 13) +#endif + +namespace vt::cpu { +namespace { + +constexpr std::array kInventory{{ + {ArmIsaTier::kPortable, "portable", "scalar", "none", "none", "none"}, + {ArmIsaTier::kNeon, "neon", "elementwise-gemm", "neon", + "getauxval:AT_HWCAP:HWCAP_ASIMD", "aarch64-baseline:neon"}, + {ArmIsaTier::kDotProd, "dotprod", "q8-dot", "neon,dotprod", + "getauxval:AT_HWCAP:HWCAP_ASIMDDP", + "sysctl:hw.optional.arm.FEAT_DotProd"}, + {ArmIsaTier::kI8mm, "i8mm", "quant-dot,quant-repack", + "neon,dotprod,i8mm", "getauxval:AT_HWCAP2:HWCAP2_I8MM", + "sysctl:hw.optional.arm.FEAT_I8MM"}, +}}; + +#if defined(__aarch64__) && defined(__APPLE__) +bool DarwinFeature(const char* name) { + int value = 0; + size_t size = sizeof(value); + return sysctlbyname(name, &value, &size, nullptr, 0) == 0 && value != 0; +} +#endif + +} // namespace + +const std::array& ArmIsaTierInventory() { + return kInventory; +} + +const char* ArmIsaTierName(ArmIsaTier tier) { + for (const auto& item : kInventory) { + if (item.tier == tier) return item.name.data(); + } + return "unknown"; +} + +ArmIsaCaps DetectArmIsaCaps() { + ArmIsaCaps caps{}; +#if defined(__aarch64__) && defined(__linux__) + const unsigned long hwcap = getauxval(AT_HWCAP); + const unsigned long hwcap2 = getauxval(AT_HWCAP2); + caps.neon = (hwcap & HWCAP_ASIMD) != 0; + caps.dotprod = (hwcap & HWCAP_ASIMDDP) != 0; + caps.i8mm = (hwcap2 & HWCAP2_I8MM) != 0; +#elif defined(__aarch64__) && defined(__APPLE__) + caps.neon = true; + caps.dotprod = DarwinFeature("hw.optional.arm.FEAT_DotProd"); + caps.i8mm = DarwinFeature("hw.optional.arm.FEAT_I8MM"); +#endif + return caps; +} + +bool ArmIsaTierSupported(const ArmIsaCaps& caps, ArmIsaTier tier) { + switch (tier) { + case ArmIsaTier::kPortable: return true; + case ArmIsaTier::kNeon: return caps.neon; + case ArmIsaTier::kDotProd: return caps.neon && caps.dotprod; + case ArmIsaTier::kI8mm: + return caps.neon && caps.dotprod && caps.i8mm; + } + return false; +} + +bool SelectArmIsaTier(const ArmIsaCaps& caps, std::string_view forced, + ArmIsaTier* selected, std::string* error) { + ArmIsaTier requested = ArmIsaTier::kPortable; + if (forced.empty()) { + for (auto it = kInventory.rbegin(); it != kInventory.rend(); ++it) { + if (ArmIsaTierSupported(caps, it->tier)) { + *selected = it->tier; + return true; + } + } + } else { + bool found = false; + for (const auto& item : kInventory) { + if (item.name == forced) { + requested = item.tier; + found = true; + break; + } + } + if (!found) { + if (error != nullptr) *error = "unknown Arm ISA tier '" + std::string(forced) + "'"; + return false; + } + if (ArmIsaTierSupported(caps, requested)) { + *selected = requested; + return true; + } + } + + if (error != nullptr) { + const auto& requirement = kInventory[static_cast(requested)]; + std::ostringstream message; + message << "unsupported Arm ISA tier '" << requirement.name + << "': requires CPU features " << requirement.cpu_features; + *error = message.str(); + } + return false; +} + +bool ResolveArmIsaToggle(const ArmIsaCaps& caps, ArmIsaTier tier, + std::string_view value, bool* enabled, + std::string* error) { + if (value.empty() || value == "auto") { + *enabled = ArmIsaTierSupported(caps, tier); + return true; + } + if (value == "portable" || value == "0" || value == "off" || + value == "false") { + *enabled = false; + return true; + } + if (value != ArmIsaTierName(tier) && value != "1" && value != "on" && + value != "true") { + if (error != nullptr) { + *error = "unknown force value '" + std::string(value) + + "' for Arm ISA tier '" + ArmIsaTierName(tier) + "'"; + } + return false; + } + if (!ArmIsaTierSupported(caps, tier)) { + if (error != nullptr) { + *error = "unsupported forced Arm ISA tier '" + + std::string(ArmIsaTierName(tier)) + "'"; + } + return false; + } + *enabled = true; + return true; +} + +} // namespace vt::cpu diff --git a/src/vt/cpu/cpu_isa_arm.h b/src/vt/cpu/cpu_isa_arm.h new file mode 100644 index 000000000..3b0041a8e --- /dev/null +++ b/src/vt/cpu/cpu_isa_arm.h @@ -0,0 +1,36 @@ +#pragma once + +#include +#include +#include + +namespace vt::cpu { + +enum class ArmIsaTier { kPortable, kNeon, kDotProd, kI8mm }; + +struct ArmIsaCaps { + bool neon = false; + bool dotprod = false; + bool i8mm = false; +}; + +struct ArmIsaTierRequirement { + ArmIsaTier tier; + std::string_view name; + std::string_view kernel_families; + std::string_view cpu_features; + std::string_view linux_probe; + std::string_view darwin_probe; +}; + +const std::array& ArmIsaTierInventory(); +const char* ArmIsaTierName(ArmIsaTier tier); +ArmIsaCaps DetectArmIsaCaps(); +bool ArmIsaTierSupported(const ArmIsaCaps& caps, ArmIsaTier tier); +bool SelectArmIsaTier(const ArmIsaCaps& caps, std::string_view forced, + ArmIsaTier* selected, std::string* error); +bool ResolveArmIsaToggle(const ArmIsaCaps& caps, ArmIsaTier tier, + std::string_view value, bool* enabled, + std::string* error); + +} // namespace vt::cpu diff --git a/src/vt/cpu/cpu_isa_x86.cpp b/src/vt/cpu/cpu_isa_x86.cpp new file mode 100644 index 000000000..255c385e1 --- /dev/null +++ b/src/vt/cpu/cpu_isa_x86.cpp @@ -0,0 +1,159 @@ +#include "vt/cpu/cpu_isa_x86.h" + +#include + +#if defined(__x86_64__) || defined(_M_X64) +#if defined(_MSC_VER) +#include +#else +#include +#endif +#endif + +namespace vt::cpu { +namespace { + +constexpr uint64_t kAvxXcr0 = (1ULL << 1) | (1ULL << 2); +constexpr uint64_t kAvx512Xcr0 = + kAvxXcr0 | (1ULL << 5) | (1ULL << 6) | (1ULL << 7); + +constexpr std::array kInventory{{ + {X86IsaTier::kPortable, "portable", "elementwise-gemm", "none", "none", 0}, + {X86IsaTier::kSse2, "sse2", "elementwise-gemm", "sse2", "xmm", 0}, + {X86IsaTier::kSse2F16c, "sse2+f16c", "elementwise-gemm", + "sse2,avx,f16c,osxsave", "xcr0:xmm,ymm", kAvxXcr0}, + {X86IsaTier::kAvx2, "avx2", "elementwise-gemm", + "sse2,avx,f16c,avx2,osxsave", "xcr0:xmm,ymm", kAvxXcr0}, + {X86IsaTier::kAvx512, "avx512", "elementwise-gemm", + "sse2,avx,f16c,avx2,avx512f,avx512bw,avx512vl,osxsave", + "xcr0:xmm,ymm,opmask,zmm_hi256,hi16_zmm", kAvx512Xcr0}, +}}; + +#if defined(__x86_64__) || defined(_M_X64) +uint64_t ReadXcr0() { +#if defined(_MSC_VER) + return _xgetbv(0); +#else + uint32_t eax = 0; + uint32_t edx = 0; + __asm__ volatile("xgetbv" : "=a"(eax), "=d"(edx) : "c"(0)); + return (static_cast(edx) << 32) | eax; +#endif +} + +void ReadCpuid(uint32_t leaf, uint32_t subleaf, uint32_t* eax, uint32_t* ebx, + uint32_t* ecx, uint32_t* edx) { +#if defined(_MSC_VER) + int regs[4]{}; + __cpuidex(regs, static_cast(leaf), static_cast(subleaf)); + *eax = static_cast(regs[0]); + *ebx = static_cast(regs[1]); + *ecx = static_cast(regs[2]); + *edx = static_cast(regs[3]); +#else + __cpuid_count(leaf, subleaf, *eax, *ebx, *ecx, *edx); +#endif +} +#endif + +} // namespace + +const std::array& X86IsaTierInventory() { + return kInventory; +} + +const char* X86IsaTierName(X86IsaTier tier) { + for (const auto& item : kInventory) { + if (item.tier == tier) return item.name.data(); + } + return "unknown"; +} + +X86IsaCaps DetectX86IsaCaps() { + X86IsaCaps caps{}; +#if defined(__x86_64__) || defined(_M_X64) + uint32_t eax = 0; + uint32_t ebx = 0; + uint32_t ecx = 0; + uint32_t edx = 0; + ReadCpuid(0, 0, &eax, &ebx, &ecx, &edx); + const uint32_t max_leaf = eax; + if (max_leaf >= 1) { + ReadCpuid(1, 0, &eax, &ebx, &ecx, &edx); + caps.sse2 = (edx & (1U << 26)) != 0; + caps.osxsave = (ecx & (1U << 27)) != 0; + caps.avx = (ecx & (1U << 28)) != 0; + caps.f16c = (ecx & (1U << 29)) != 0; + if (caps.osxsave) caps.xcr0 = ReadXcr0(); + } + if (max_leaf >= 7) { + ReadCpuid(7, 0, &eax, &ebx, &ecx, &edx); + caps.avx2 = (ebx & (1U << 5)) != 0; + caps.avx512f = (ebx & (1U << 16)) != 0; + caps.avx512bw = (ebx & (1U << 30)) != 0; + caps.avx512vl = (ebx & (1U << 31)) != 0; + } +#endif + return caps; +} + +bool X86IsaTierSupported(const X86IsaCaps& caps, X86IsaTier tier) { + const bool avx_state = caps.osxsave && (caps.xcr0 & kAvxXcr0) == kAvxXcr0; + const bool avx512_state = + caps.osxsave && (caps.xcr0 & kAvx512Xcr0) == kAvx512Xcr0; + switch (tier) { + case X86IsaTier::kPortable: + return true; + case X86IsaTier::kSse2: + return caps.sse2; + case X86IsaTier::kSse2F16c: + return caps.sse2 && caps.avx && caps.f16c && avx_state; + case X86IsaTier::kAvx2: + return caps.sse2 && caps.avx && caps.f16c && caps.avx2 && avx_state; + case X86IsaTier::kAvx512: + return caps.sse2 && caps.avx && caps.f16c && caps.avx2 && caps.avx512f && + caps.avx512bw && caps.avx512vl && avx512_state; + } + return false; +} + +bool SelectX86IsaTier(const X86IsaCaps& caps, std::string_view forced, + X86IsaTier* selected, std::string* error) { + X86IsaTier requested = X86IsaTier::kPortable; + if (forced.empty()) { + for (auto it = kInventory.rbegin(); it != kInventory.rend(); ++it) { + if (X86IsaTierSupported(caps, it->tier)) { + *selected = it->tier; + return true; + } + } + } else { + bool found = false; + for (const auto& item : kInventory) { + if (item.name == forced) { + requested = item.tier; + found = true; + break; + } + } + if (!found) { + if (error != nullptr) *error = "unknown x86 ISA tier '" + std::string(forced) + "'"; + return false; + } + if (X86IsaTierSupported(caps, requested)) { + *selected = requested; + return true; + } + } + + if (error != nullptr) { + const auto& requirement = kInventory[static_cast(requested)]; + std::ostringstream message; + message << "unsupported x86 ISA tier '" << requirement.name << "': requires CPU " + << requirement.cpu_features << " and OS state " << requirement.os_state; + *error = message.str(); + } + return false; +} + +} // namespace vt::cpu diff --git a/src/vt/cpu/cpu_isa_x86.h b/src/vt/cpu/cpu_isa_x86.h new file mode 100644 index 000000000..c078cfce0 --- /dev/null +++ b/src/vt/cpu/cpu_isa_x86.h @@ -0,0 +1,40 @@ +#pragma once + +#include +#include +#include +#include + +namespace vt::cpu { + +enum class X86IsaTier { kPortable, kSse2, kSse2F16c, kAvx2, kAvx512 }; + +struct X86IsaCaps { + bool sse2 = false; + bool f16c = false; + bool avx = false; + bool osxsave = false; + bool avx2 = false; + bool avx512f = false; + bool avx512bw = false; + bool avx512vl = false; + uint64_t xcr0 = 0; +}; + +struct X86IsaTierRequirement { + X86IsaTier tier; + std::string_view name; + std::string_view kernel_family; + std::string_view cpu_features; + std::string_view os_state; + uint64_t xcr0_mask; +}; + +const std::array& X86IsaTierInventory(); +const char* X86IsaTierName(X86IsaTier tier); +X86IsaCaps DetectX86IsaCaps(); +bool X86IsaTierSupported(const X86IsaCaps& caps, X86IsaTier tier); +bool SelectX86IsaTier(const X86IsaCaps& caps, std::string_view forced, + X86IsaTier* selected, std::string* error); + +} // namespace vt::cpu diff --git a/src/vt/cpu/cpu_matmul_elem.cpp b/src/vt/cpu/cpu_matmul_elem.cpp index 0767fa427..8da308b9a 100644 --- a/src/vt/cpu/cpu_matmul_elem.cpp +++ b/src/vt/cpu/cpu_matmul_elem.cpp @@ -2,6 +2,8 @@ // anchors and the recorded "vectorize across OUTPUT columns, not along K" // deviation that keeps every result bit-identical to the scalar reference. #include "cpu_matmul_elem.h" +#include "vt/cpu/cpu_isa_arm.h" +#include "vt/cpu/cpu_isa_x86.h" #include "vt/quant.h" #include @@ -563,6 +565,18 @@ ElemGemmTierTable BuildTier() { return t; } #if defined(__aarch64__) + const ArmIsaCaps caps = DetectArmIsaCaps(); + const std::string arm_forced = + forced.empty() + ? (ArmIsaTierSupported(caps, ArmIsaTier::kNeon) ? "neon" : "portable") + : forced; + ArmIsaTier selected{}; + std::string selection_error; + VT_CHECK(SelectArmIsaTier(caps, arm_forced, &selected, &selection_error), + selection_error); + VT_CHECK(selected == ArmIsaTier::kPortable || selected == ArmIsaTier::kNeon, + "VT_CPU_MATMUL_TIER on Arm must be portable or neon"); + if (selected == ArmIsaTier::kPortable) return t; t.bt[kF32] = &Bt16Neon; t.bt[kF16] = &Bt16Neon; t.bt[kBF16] = &Bt16Neon; @@ -578,6 +592,12 @@ ElemGemmTierTable BuildTier() { t.mr = kMrNeon; t.name = "neon"; #elif defined(__x86_64__) || defined(_M_X64) + X86IsaTier selected{}; + std::string selection_error; + VT_CHECK(SelectX86IsaTier(DetectX86IsaCaps(), forced, &selected, + &selection_error), + selection_error); + if (selected == X86IsaTier::kPortable) return t; t.bt[kF32] = &Bt16Sse2; t.bt[kBF16] = &Bt16Sse2; t.nk[kF32] = &Nk16Sse2; @@ -588,27 +608,21 @@ ElemGemmTierTable BuildTier() { t.nkm[kBF16] = &NkM2Sse2; t.mr = kMrSse2; t.name = "sse2"; - if (__builtin_cpu_supports("f16c")) { + if (selected == X86IsaTier::kSse2) return t; + if (selected == X86IsaTier::kSse2F16c || + selected == X86IsaTier::kAvx2 || + selected == X86IsaTier::kAvx512) { t.bt[kF16] = &Bt16F16c; t.nk[kF16] = &Nk16F16c; t.btm[kF16] = &BtM2F16c; t.nkm[kF16] = &NkM2F16c; t.name = "sse2+f16c"; - } else { - // No f16 M-blocked kernel without F16C: fall back to the 1-row path for - // f16 only, which the caller selects on a null btm entry. - t.btm[kF16] = nullptr; - } - // Wider tiers replace the SSE2 table wholesale when the CPU has them. AVX2 - // needs F16C for its f16 widening (_mm256_cvtph_ps); every x86-64 CPU with - // AVX2 also has F16C, but the probe is explicit rather than assumed. - if ((forced.empty() || forced == "avx2") && __builtin_cpu_supports("avx2") && - __builtin_cpu_supports("f16c")) { + } + if (selected == X86IsaTier::kSse2F16c) return t; + if (selected == X86IsaTier::kAvx2 || selected == X86IsaTier::kAvx512) { FillAvx2Tier(&t); } - // AVX-512F last so it wins when present, unless a narrower tier was forced - // for a same-binary A/B. - if ((forced.empty() || forced == "avx512") && __builtin_cpu_supports("avx512f")) { + if (selected == X86IsaTier::kAvx512) { FillAvx512Tier(&t); } #endif diff --git a/src/vt/cpu/cpu_matmul_elem.h b/src/vt/cpu/cpu_matmul_elem.h index cff68b68b..ded863415 100644 --- a/src/vt/cpu/cpu_matmul_elem.h +++ b/src/vt/cpu/cpu_matmul_elem.h @@ -95,6 +95,8 @@ const ElemGemmTierTable& ElemGemmTier(); // VT_CPU_MATMUL_TIER selects the tier for a SAME-BINARY A/B: // "ref" — the historical one-accumulator scalar chunk kernel // "portable" — tier 0 only (no arch SIMD) +// "sse2", "sse2+f16c", "avx2", "avx512" — exact x86 tier; an unavailable +// tier fails closed after CPUID + XCR0 validation // unset — the best tier this CPU probes into (production default) const char* ElemGemmTierName(); diff --git a/src/vt/cpu/cpu_quant_dot_arm.cpp b/src/vt/cpu/cpu_quant_dot_arm.cpp index 6a0dd68f6..4f8012346 100644 --- a/src/vt/cpu/cpu_quant_dot_arm.cpp +++ b/src/vt/cpu/cpu_quant_dot_arm.cpp @@ -35,54 +35,26 @@ // `vmmlaq_s32`; the whole body is additionally `#if defined(__aarch64__) && // defined(__ARM_FEATURE_MATMUL_INT8)` so a build without the feature (or any // non-Arm build) links the empty stubs and the portable tier runs everywhere. -// At RUNTIME the mmla kernels are handed out ONLY when `getauxval(AT_HWCAP2) & -// HWCAP2_I8MM` — selecting an mmla kernel on a CPU that lacks i8mm would be an -// illegal-instruction crash. `VT_CPU_QUANT_MMLA=0|off|false` forces the tier -// off for a same-binary A/B (the portable nrc==1 path then serves every shape). +// At RUNTIME the mmla kernels are handed out only after the shared Linux +// HWCAP/HWCAP2 or Darwin sysctl detector proves the i8mm tier. The +// VT_CPU_QUANT_MMLA switch can force either portable or i8mm; a forced i8mm on +// unsupported hardware fails before an illegal instruction can execute. #include "vt/quant.h" #if defined(__aarch64__) && defined(__ARM_FEATURE_MATMUL_INT8) #include -#if defined(__linux__) -#include -#include -#elif defined(__APPLE__) -#include -#endif - #include #include +#include #include "cpu_quant_blocks.h" - -#if defined(__linux__) && !defined(HWCAP2_I8MM) -#define HWCAP2_I8MM (1 << 13) -#endif +#include "vt/cpu/cpu_isa_arm.h" namespace vt::cpu { namespace { -// Runtime i8mm detection is OS-specific: Linux exposes it through the aux -// vector, Darwin through sysctl. The compile-time __ARM_FEATURE_MATMUL_INT8 -// gate above only says the compiler MAY emit i8mm - the running core still -// has to confirm it. -bool CpuHasI8mm() { -#if defined(__linux__) - return (getauxval(AT_HWCAP2) & HWCAP2_I8MM) != 0; -#elif defined(__APPLE__) - int v = 0; - size_t sz = sizeof(v); - if (sysctlbyname("hw.optional.arm.FEAT_I8MM", &v, &sz, nullptr, 0) != 0) { - return false; - } - return v != 0; -#else - return false; -#endif -} - // GGML_CPU_FP16_TO_FP32 on this target is an IEEE-exact fp16->fp32 widen; so is // vt::F16ToF32. Using the latter keeps the scale bit-identical to the portable // tier's `F16ToF32(x.d)`. @@ -526,11 +498,12 @@ void VecDotMmlaQ6_K(int n, float* s, size_t bs, const void* vx, size_t bx, bool QuantMmlaActive() { static const bool v = [] { const char* e = std::getenv("VT_CPU_QUANT_MMLA"); - if (e != nullptr && - (std::strcmp(e, "0") == 0 || std::strcmp(e, "off") == 0 || std::strcmp(e, "false") == 0)) { - return false; - } - return CpuHasI8mm(); + bool enabled = false; + std::string error; + VT_CHECK(ResolveArmIsaToggle(DetectArmIsaCaps(), ArmIsaTier::kI8mm, + e == nullptr ? "auto" : e, &enabled, &error), + error); + return enabled; }(); return v; } diff --git a/src/vt/cpu/cpu_quant_dot_sdot.cpp b/src/vt/cpu/cpu_quant_dot_sdot.cpp index 799e8cae5..d3529935c 100644 --- a/src/vt/cpu/cpu_quant_dot_sdot.cpp +++ b/src/vt/cpu/cpu_quant_dot_sdot.cpp @@ -7,38 +7,28 @@ // measurable AAPCS64 schedule over the exact same arithmetic. #include "vt/quant.h" -#if defined(VT_CPU_A76_Q8_DOT) && defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD) +#if defined(VT_CPU_ARM_Q8_DOT) && defined(__aarch64__) && defined(__ARM_FEATURE_DOTPROD) #include -#if defined(__linux__) -#include -#include -#endif - #include #include #include #include #include "cpu_quant_blocks.h" +#include "vt/cpu/cpu_isa_arm.h" -#if defined(__linux__) && !defined(HWCAP_ASIMDDP) -#define HWCAP_ASIMDDP (1 << 20) -#endif - +#if defined(VT_CPU_A76_Q8_DOT) extern "C" void vt_cpu_q8_dot_a76_asm(int n, float* s, size_t bs, const void* x, size_t bx, const void* y, size_t by, int nrc); +#endif namespace vt::cpu { namespace { bool CpuHasDotProd() { -#if defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_ASIMDDP) != 0; -#else - return true; -#endif + return ArmIsaTierSupported(DetectArmIsaCaps(), ArmIsaTier::kDotProd); } bool CpuIsCortexA76() { @@ -103,22 +93,34 @@ VecDotFn QuantQ8SdotVecDot() { } VecDotFn QuantQ8A76AsmVecDot() { +#if defined(VT_CPU_A76_Q8_DOT) return CpuHasDotProd() ? &vt_cpu_q8_dot_a76_asm : nullptr; +#else + return nullptr; +#endif } VecDotFn SelectQuantQ8VecDot(VecDotFn portable) { const char* value = std::getenv("VT_CPU_Q8_DOT"); if (value == nullptr || std::strcmp(value, "auto") == 0) { - return QuantQ8A76AsmActive() ? QuantQ8A76AsmVecDot() : portable; + if (QuantQ8A76AsmActive()) return QuantQ8A76AsmVecDot(); + const VecDotFn sdot = QuantQ8SdotVecDot(); + return sdot != nullptr ? sdot : portable; } if (std::strcmp(value, "portable") == 0) { return portable; } if (std::strcmp(value, "sdot") == 0) { - return QuantQ8SdotVecDot() != nullptr ? QuantQ8SdotVecDot() : portable; + const VecDotFn selected = QuantQ8SdotVecDot(); + VT_CHECK(selected != nullptr, + "VT_CPU_Q8_DOT=sdot requires NEON DotProd OS capability"); + return selected; } if (std::strcmp(value, "a76-asm") == 0) { - return QuantQ8A76AsmVecDot() != nullptr ? QuantQ8A76AsmVecDot() : portable; + const VecDotFn selected = QuantQ8A76AsmVecDot(); + VT_CHECK(selected != nullptr, + "VT_CPU_Q8_DOT=a76-asm requires the Linux A76 assembly tier"); + return selected; } VT_CHECK(false, "VT_CPU_Q8_DOT must be auto, portable, sdot, or a76-asm"); return portable; diff --git a/src/vt/cpu/cpu_quant_repack_arm.cpp b/src/vt/cpu/cpu_quant_repack_arm.cpp index 9b4558f5e..5bcd2c6b4 100644 --- a/src/vt/cpu/cpu_quant_repack_arm.cpp +++ b/src/vt/cpu/cpu_quant_repack_arm.cpp @@ -17,57 +17,30 @@ // // COMPILE/RUNTIME GATING mirrors the mmla tier: compiled with +i8mm for one TU // (CMakeLists), body guarded by `__aarch64__ && __ARM_FEATURE_MATMUL_INT8`, and -// handed out at runtime only when HWCAP2_I8MM is set and VT_CPU_QUANT_REPACK is -// not disabled. Off i8mm aarch64 the stubs make QuantRepackActive() false, so -// the loader never repacks and this code is never reached. +// handed out at runtime only after the shared Linux HWCAP/HWCAP2 or Darwin +// sysctl detector proves i8mm+DotProd. VT_CPU_QUANT_REPACK can force portable +// or i8mm; forcing i8mm on unsupported hardware fails before these instructions +// can execute. Off i8mm aarch64 the loader never repacks. #include "vt/quant.h" #if defined(__aarch64__) && defined(__ARM_FEATURE_MATMUL_INT8) #include -#if defined(__linux__) -#include -#include -#elif defined(__APPLE__) -#include -#endif - #include #include +#include #include #include "cpu_quant_blocks.h" #include "cpu_quant_repack.h" #include "cpu_threadpool.h" +#include "vt/cpu/cpu_isa_arm.h" #include "vt/tensor.h" -#ifndef HWCAP2_I8MM -#define HWCAP2_I8MM (1 << 13) -#endif - namespace vt::cpu { namespace { -// Runtime i8mm detection is OS-specific: Linux exposes it through the aux -// vector, Darwin through sysctl. The compile-time __ARM_FEATURE_MATMUL_INT8 -// gate above only says the compiler MAY emit i8mm - the running core still -// has to confirm it. -bool CpuHasI8mm() { -#if defined(__linux__) - return (getauxval(AT_HWCAP2) & HWCAP2_I8MM) != 0; -#elif defined(__APPLE__) - int v = 0; - size_t sz = sizeof(v); - if (sysctlbyname("hw.optional.arm.FEAT_I8MM", &v, &sz, nullptr, 0) != 0) { - return false; - } - return v != 0; -#else - return false; -#endif -} - } // namespace namespace { @@ -199,11 +172,12 @@ void GemvRowQ8_0(float* s_row, const BlockQ8_0x4* wgroups, bool QuantRepackActive() { static const bool v = [] { const char* e = std::getenv("VT_CPU_QUANT_REPACK"); - if (e != nullptr && (std::strcmp(e, "0") == 0 || std::strcmp(e, "off") == 0 || - std::strcmp(e, "false") == 0)) { - return false; - } - return CpuHasI8mm(); + bool enabled = false; + std::string error; + VT_CHECK(ResolveArmIsaToggle(DetectArmIsaCaps(), ArmIsaTier::kI8mm, + e == nullptr ? "auto" : e, &enabled, &error), + error); + return enabled; }(); return v; } diff --git a/src/vt/cuda/cuda_gdn.cu b/src/vt/cuda/cuda_gdn.cu index 9d4706323..ae4d83f67 100644 --- a/src/vt/cuda/cuda_gdn.cu +++ b/src/vt/cuda/cuda_gdn.cu @@ -19,6 +19,7 @@ #include #include +#include #include #include #include @@ -40,6 +41,7 @@ #include "vt/cuda/rmsnorm_gated_fast.h" #include "vt/cuda/tile/cp_async.cuh" #include "vt/cuda/tile/tma_pipeline.cuh" +#include "vt/cuda/triton_aot_arch_dispatch.h" #include "vt/ops.h" #ifdef VLLM_CPP_TRITON @@ -51,46 +53,96 @@ // runtime. One spec per gate-model GDN shape: h48 (Qwen3.6-27B), h32 (35B). // gdn_deltah_hNN_default(stream, k, v, w, v_new, g, gk, h, h0, ht, cu_seqlens, // chunk_offsets, T, NH) launches the FLA chunk_gated_delta_rule_fwd_kernel. -extern "C" { -#include "gdn_deltah_h48.h" -#include "gdn_deltah_h32.h" -// GDN chunk_o (the output kernel): gdn_chunko_hNN_default, and optionally -// gdn_chunko_bf16_hNN_default when the vendored artifacts are present. -// See triton_kernels/chunk_o.py. o(=out) is f32 for the default GDN out dtype; -// bf16 mirrors the vLLM-faithful recurrence-output dtype once regenerated. -#include "gdn_chunko_h48.h" -#include "gdn_chunko_h32.h" -#ifdef VLLM_CPP_TRITON_CHUNKO_BF16 -#include "gdn_chunko_bf16_h48.h" -#include "gdn_chunko_bf16_h32.h" -#endif -// GDN WU pipeline (kkt -> solve_tril -> recompute_w_u), 3 stable dispatchers each. -// See triton_kernels/{chunk_scaled_dot_kkt,solve_tril,wy_fast}.py. -#include "gdn_kkt_h48.h" -#include "gdn_kkt_h32.h" -#include "gdn_tril_h48.h" -#include "gdn_tril_h32.h" -#include "gdn_wu_h48.h" -#include "gdn_wu_h32.h" -// GDN packed pure-decode recurrence (fused_recurrent_gated_delta_rule packed -// decode): gdn_decode_h48_default / gdn_decode_h32_default. See -// triton_kernels/fused_recurrent_packed_decode.py. The MEASURED codegen-bound -// lever (dgx phase1 2026-07-16): REG:205/0-spill under Triton vs REG:255+STACK:48 -// (spills) as hand-CUDA at BK=128. -#include "gdn_decode_h48.h" -#include "gdn_decode_h32.h" -// KDA chunk-prefill family (Kimi-Linear; H=32). The FLA chunk_kda_with_fused_gate -// forward: kda_gate_cumsum -> kkt(inter+intra) -> solve_tril (REUSES gdn_tril_h32) -// -> recompute_w_u (kda_wu) -> chunk_delta_h (kda_deltah_h32) -> chunk_gla_o -// (kda_gla_o). See triton_kernels/{kda_gate_cumsum,chunk_kda_kkt,recompute_w_u_kda, -// chunk_gla_o}.py + chunk_delta_h.py (kda_deltah pin) + .agents/specs/kimi-linear.md §17. -#include "kda_gate_cumsum.h" -#include "kda_kkt_inter.h" -#include "kda_kkt_intra.h" -#include "kda_wu.h" -#include "kda_deltah_h32.h" -#include "kda_gla_o.h" -} +#include "triton_aot_multiarch.h" + +#define VT_AOT_SYMBOL_IMPL(ARCH, NAME) vt_aot_##ARCH##_##NAME##_default +#define VT_AOT_SYMBOL(ARCH, NAME) VT_AOT_SYMBOL_IMPL(ARCH, NAME) +#define VT_AOT_LOAD_SYMBOL_IMPL(ARCH, NAME) vt_aot_##ARCH##_load_##NAME +#define VT_AOT_LOAD_SYMBOL(ARCH, NAME) VT_AOT_LOAD_SYMBOL_IMPL(ARCH, NAME) +#define VT_TRITON_AOT_CALL(NAME, ...) \ + vt::cuda::DispatchTritonAot( \ + vt::cuda::GetDeviceCaps().sm_major, \ + vt::cuda::GetDeviceCaps().sm_minor, CUDA_ERROR_NOT_SUPPORTED, \ + std::array{&VT_AOT_SYMBOL(sm_80, NAME), \ + &VT_AOT_SYMBOL(sm_86, NAME), \ + &VT_AOT_SYMBOL(sm_89, NAME), \ + &VT_AOT_SYMBOL(sm_90a, NAME), \ + &VT_AOT_SYMBOL(sm_100a, NAME), \ + &VT_AOT_SYMBOL(sm_121a, NAME)}, \ + __VA_ARGS__) +#define VT_TRITON_AOT_LOAD(NAME) \ + vt::cuda::DispatchTritonAotVoid( \ + vt::cuda::GetDeviceCaps().sm_major, \ + vt::cuda::GetDeviceCaps().sm_minor, \ + std::array{&VT_AOT_LOAD_SYMBOL(sm_80, NAME), \ + &VT_AOT_LOAD_SYMBOL(sm_86, NAME), \ + &VT_AOT_LOAD_SYMBOL(sm_89, NAME), \ + &VT_AOT_LOAD_SYMBOL(sm_90a, NAME), \ + &VT_AOT_LOAD_SYMBOL(sm_100a, NAME), \ + &VT_AOT_LOAD_SYMBOL(sm_121a, NAME)}) + +#define gdn_deltah_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_deltah_h48, __VA_ARGS__) +#define gdn_deltah_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_deltah_h32, __VA_ARGS__) +#define gdn_chunko_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_chunko_h48, __VA_ARGS__) +#define gdn_chunko_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_chunko_h32, __VA_ARGS__) +#define gdn_chunko_bf16_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_chunko_bf16_h48, __VA_ARGS__) +#define gdn_chunko_bf16_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_chunko_bf16_h32, __VA_ARGS__) +#define gdn_kkt_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_kkt_h48, __VA_ARGS__) +#define gdn_kkt_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_kkt_h32, __VA_ARGS__) +#define gdn_tril_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_tril_h48, __VA_ARGS__) +#define gdn_tril_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_tril_h32, __VA_ARGS__) +#define gdn_wu_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_wu_h48, __VA_ARGS__) +#define gdn_wu_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_wu_h32, __VA_ARGS__) +#define gdn_decode_h48_default(...) \ + VT_TRITON_AOT_CALL(gdn_decode_h48, __VA_ARGS__) +#define gdn_decode_h32_default(...) \ + VT_TRITON_AOT_CALL(gdn_decode_h32, __VA_ARGS__) +#define kda_gate_cumsum_default(...) \ + VT_TRITON_AOT_CALL(kda_gate_cumsum, __VA_ARGS__) +#define kda_kkt_inter_default(...) \ + VT_TRITON_AOT_CALL(kda_kkt_inter, __VA_ARGS__) +#define kda_kkt_intra_default(...) \ + VT_TRITON_AOT_CALL(kda_kkt_intra, __VA_ARGS__) +#define kda_wu_default(...) VT_TRITON_AOT_CALL(kda_wu, __VA_ARGS__) +#define kda_deltah_h32_default(...) \ + VT_TRITON_AOT_CALL(kda_deltah_h32, __VA_ARGS__) +#define kda_gla_o_default(...) VT_TRITON_AOT_CALL(kda_gla_o, __VA_ARGS__) + +#define VT_DEFINE_TRITON_AOT_LOADER(NAME) \ + static void load_##NAME() { (void)VT_TRITON_AOT_LOAD(NAME); } +VT_DEFINE_TRITON_AOT_LOADER(gdn_deltah_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_deltah_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_chunko_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_chunko_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_chunko_bf16_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_chunko_bf16_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_kkt_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_kkt_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_tril_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_tril_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_wu_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_wu_h32) +VT_DEFINE_TRITON_AOT_LOADER(gdn_decode_h48) +VT_DEFINE_TRITON_AOT_LOADER(gdn_decode_h32) +VT_DEFINE_TRITON_AOT_LOADER(kda_gate_cumsum) +VT_DEFINE_TRITON_AOT_LOADER(kda_kkt_inter) +VT_DEFINE_TRITON_AOT_LOADER(kda_kkt_intra) +VT_DEFINE_TRITON_AOT_LOADER(kda_wu) +VT_DEFINE_TRITON_AOT_LOADER(kda_deltah_h32) +VT_DEFINE_TRITON_AOT_LOADER(kda_gla_o) +#undef VT_DEFINE_TRITON_AOT_LOADER #endif namespace vt::cuda { @@ -4476,6 +4528,16 @@ static bool GdnTritonEnvOn(const char* n) { return e == nullptr || e[0] != '0'; } +int CurrentTritonAotTreeIndex() { + const DeviceCaps& caps = GetDeviceCaps(); + if (!caps.valid) return -1; + return TritonAotTreeIndex(caps.sm_major, caps.sm_minor); +} + +bool TritonAotAvailableOnCurrentDevice() { + return CurrentTritonAotTreeIndex() >= 0; +} + // Persistent per-stream scratch for the Triton-AOT GDN build. This mirrors the // caching allocator vLLM gets through PyTorch: buffers grow to the largest shape // seen on a stream and are reused by subsequent GDN layers/steps. Reuse is safe @@ -4620,29 +4682,29 @@ T* EnsureGdnScratch(GdnScratchBuf& buf, size_t count, cudaStream_t s, const char // so concurrent first use from separate queues cannot double-load or race those // globals. CUDA primary-context modules are process-wide for the active device. struct GdnAotModuleOnce { - std::once_flag deltah_h48; - std::once_flag deltah_h32; - std::once_flag chunko_h48; - std::once_flag chunko_h32; + std::array deltah_h48; + std::array deltah_h32; + std::array chunko_h48; + std::array chunko_h32; #ifdef VLLM_CPP_TRITON_CHUNKO_BF16 - std::once_flag chunko_bf16_h48; - std::once_flag chunko_bf16_h32; + std::array chunko_bf16_h48; + std::array chunko_bf16_h32; #endif - std::once_flag kkt_h48; - std::once_flag tril_h48; - std::once_flag wu_h48; - std::once_flag kkt_h32; - std::once_flag tril_h32; - std::once_flag wu_h32; - std::once_flag decode_h48; - std::once_flag decode_h32; + std::array kkt_h48; + std::array tril_h48; + std::array wu_h48; + std::array kkt_h32; + std::array tril_h32; + std::array wu_h32; + std::array decode_h48; + std::array decode_h32; // KDA chunk-prefill family (Kimi-Linear; H=32). - std::once_flag kda_gate_cumsum; - std::once_flag kda_kkt_inter; - std::once_flag kda_kkt_intra; - std::once_flag kda_wu; - std::once_flag kda_deltah_h32; - std::once_flag kda_gla_o; + std::array kda_gate_cumsum; + std::array kda_kkt_inter; + std::array kda_kkt_intra; + std::array kda_wu; + std::array kda_deltah_h32; + std::array kda_gla_o; }; GdnAotModuleOnce& GdnAotModules() { @@ -4652,43 +4714,47 @@ GdnAotModuleOnce& GdnAotModules() { void EnsureGdnDeltaHLoaded(int64_t hv_n) { auto& modules = GdnAotModules(); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); if (hv_n == 48) { - std::call_once(modules.deltah_h48, load_gdn_deltah_h48); + std::call_once(modules.deltah_h48[tree], load_gdn_deltah_h48); } else { - std::call_once(modules.deltah_h32, load_gdn_deltah_h32); + std::call_once(modules.deltah_h32[tree], load_gdn_deltah_h32); } } void EnsureGdnChunkOF32Loaded(int64_t hv_n) { auto& modules = GdnAotModules(); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); if (hv_n == 48) { - std::call_once(modules.chunko_h48, load_gdn_chunko_h48); + std::call_once(modules.chunko_h48[tree], load_gdn_chunko_h48); } else { - std::call_once(modules.chunko_h32, load_gdn_chunko_h32); + std::call_once(modules.chunko_h32[tree], load_gdn_chunko_h32); } } #ifdef VLLM_CPP_TRITON_CHUNKO_BF16 void EnsureGdnChunkOBF16Loaded(int64_t hv_n) { auto& modules = GdnAotModules(); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); if (hv_n == 48) { - std::call_once(modules.chunko_bf16_h48, load_gdn_chunko_bf16_h48); + std::call_once(modules.chunko_bf16_h48[tree], load_gdn_chunko_bf16_h48); } else { - std::call_once(modules.chunko_bf16_h32, load_gdn_chunko_bf16_h32); + std::call_once(modules.chunko_bf16_h32[tree], load_gdn_chunko_bf16_h32); } } #endif void EnsureGdnWULoaded(int64_t hv_n) { auto& modules = GdnAotModules(); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); if (hv_n == 48) { - std::call_once(modules.kkt_h48, load_gdn_kkt_h48); - std::call_once(modules.tril_h48, load_gdn_tril_h48); - std::call_once(modules.wu_h48, load_gdn_wu_h48); + std::call_once(modules.kkt_h48[tree], load_gdn_kkt_h48); + std::call_once(modules.tril_h48[tree], load_gdn_tril_h48); + std::call_once(modules.wu_h48[tree], load_gdn_wu_h48); } else { - std::call_once(modules.kkt_h32, load_gdn_kkt_h32); - std::call_once(modules.tril_h32, load_gdn_tril_h32); - std::call_once(modules.wu_h32, load_gdn_wu_h32); + std::call_once(modules.kkt_h32[tree], load_gdn_kkt_h32); + std::call_once(modules.tril_h32[tree], load_gdn_tril_h32); + std::call_once(modules.wu_h32[tree], load_gdn_wu_h32); } } @@ -4696,14 +4762,16 @@ void EnsureGdnWULoaded(int64_t hv_n) { // select packed decode; Hv=32 serves the dense 4B model. void EnsureGdnPackedDecodeLoaded(int64_t hv_n) { auto& modules = GdnAotModules(); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); if (hv_n == 48) { - std::call_once(modules.decode_h48, load_gdn_decode_h48); + std::call_once(modules.decode_h48[tree], load_gdn_decode_h48); } else { - std::call_once(modules.decode_h32, load_gdn_decode_h32); + std::call_once(modules.decode_h32[tree], load_gdn_decode_h32); } } void EnsureAllGdnAotModulesLoaded() { + if (!TritonAotAvailableOnCurrentDevice()) return; for (const int64_t hv_n : {48, 32}) { EnsureGdnDeltaHLoaded(hv_n); EnsureGdnChunkOF32Loaded(hv_n); @@ -4738,6 +4806,7 @@ bool TryTritonPackedDecode(cudaStream_t stream, Tensor& out, const Tensor& dt_bias, Tensor& state, const Tensor& state_idx, const GdnArgs& args) { // Default ON: fire unless VT_GDN_PACKED_DECODE_TRITON leads with '0' (rollback). + if (!TritonAotAvailableOnCurrentDevice()) return false; if (!GdnPackedDecodeTritonFlagIsOn( std::getenv("VT_GDN_PACKED_DECODE_TRITON"))) return false; @@ -4803,6 +4872,7 @@ bool TryTritonDeltaH(cudaStream_t s, float* state, __nv_bfloat16* hstate, __nv_b const __nv_bfloat16* k, const __nv_bfloat16* u, const __nv_bfloat16* w, const float* gcum, const int32_t* qsl, const int32_t* boh, int64_t hk_n, int64_t dk, int64_t hv_n, int64_t dv, int64_t n_seq, int64_t t_tot) { + if (!TritonAotAvailableOnCurrentDevice()) return false; if (!GdnTritonEnvOn("VT_GDN_DELTAH_TRITON")) return false; // default ON (see GdnTritonEnvOn); =0 restores hand path if (dk != 128 || dv != 128 || hk_n != 16) return false; if (hv_n != 48 && hv_n != 32) return false; @@ -4836,6 +4906,7 @@ bool TryTritonChunkO(cudaStream_t s, Tout* out, const __nv_bfloat16* q, const __ int64_t dk, int64_t hv_n, int64_t dv, int64_t nt_tot, int64_t t_tot) { static_assert(std::is_same::value || std::is_same::value, "Triton chunk_o supports f32 or bf16 output"); + if (!TritonAotAvailableOnCurrentDevice()) return false; if (!GdnTritonEnvOn("VT_GDN_CHUNKO_TRITON")) return false; if (dk != 128 || dv != 128 || hk_n != 16) return false; if (hv_n != 48 && hv_n != 32) return false; @@ -4898,6 +4969,7 @@ bool TryTritonWU(cudaStream_t s, __nv_bfloat16* u, __nv_bfloat16* w, const __nv_ const __nv_bfloat16* v, const float* beta, const float* gcum, const int32_t* qsl, const int32_t* cidx, int64_t hk_n, int64_t dk, int64_t hv_n, int64_t dv, int64_t nt_tot, int64_t t_tot, GdnWuScratch* scratch) { + if (!TritonAotAvailableOnCurrentDevice()) return false; if (!GdnTritonEnvOn("VT_GDN_WU_TRITON")) return false; if (dk != 128 || dv != 128 || hk_n != 16) return false; if (hv_n != 48 && hv_n != 32) return false; @@ -4976,13 +5048,14 @@ bool TryTritonWU(cudaStream_t s, __nv_bfloat16* u, __nv_bfloat16* w, const __nv_ #ifdef VLLM_CPP_TRITON void EnsureKdaChunkLoaded() { auto& m = GdnAotModules(); - std::call_once(m.kda_gate_cumsum, load_kda_gate_cumsum); - std::call_once(m.kda_kkt_inter, load_kda_kkt_inter); - std::call_once(m.kda_kkt_intra, load_kda_kkt_intra); - std::call_once(m.tril_h32, load_gdn_tril_h32); // REUSE gdn_tril_h32 (byte-identical sig) - std::call_once(m.kda_wu, load_kda_wu); - std::call_once(m.kda_deltah_h32, load_kda_deltah_h32); - std::call_once(m.kda_gla_o, load_kda_gla_o); + const std::size_t tree = static_cast(CurrentTritonAotTreeIndex()); + std::call_once(m.kda_gate_cumsum[tree], load_kda_gate_cumsum); + std::call_once(m.kda_kkt_inter[tree], load_kda_kkt_inter); + std::call_once(m.kda_kkt_intra[tree], load_kda_kkt_intra); + std::call_once(m.tril_h32[tree], load_gdn_tril_h32); // REUSE gdn_tril_h32 (byte-identical sig) + std::call_once(m.kda_wu[tree], load_kda_wu); + std::call_once(m.kda_deltah_h32[tree], load_kda_deltah_h32); + std::call_once(m.kda_gla_o[tree], load_kda_gla_o); } template @@ -5162,7 +5235,8 @@ void KdaChunkPrefillKernelCuda(Queue& q, Tensor& out, const Tensor& q_in, const const float baked_scale = 1.0f / std::sqrt(static_cast(dk)); const bool geom_ok = (hk_n == 32 && hv_n == 32 && dk == 128 && dv == 128 && has_bias && std::fabs(args.scale - baked_scale) <= 1e-6f * baked_scale && T > 1); - if (geom_ok && GdnTritonEnvOn("VT_KDA_CHUNK_TRITON")) { + if (geom_ok && TritonAotAvailableOnCurrentDevice() && + GdnTritonEnvOn("VT_KDA_CHUNK_TRITON")) { LaunchKdaChunkPrefill(s, out, q_in, k, v, g_raw, beta, a_log, dt_bias, state, qsl, args); return; } diff --git a/src/vt/cuda/cuda_matmul_nvfp4.cu b/src/vt/cuda/cuda_matmul_nvfp4.cu index d3fb00741..17e28ea00 100644 --- a/src/vt/cuda/cuda_matmul_nvfp4.cu +++ b/src/vt/cuda/cuda_matmul_nvfp4.cu @@ -56,6 +56,7 @@ #include "vt/cuda/cuda_arch_tactics.h" #include "vt/cuda/cuda_device_caps.h" +#include "vt/cuda/cuda_nvfp4_sm12x.h" #include "vt/cuda/fp4_quant_fast.h" #include "vt/cuda/graph_safe_scratch.h" #include "vt/ops.h" @@ -119,20 +120,6 @@ bool Fp4VecEnabled() { // -Werror rejects the TU (#177-D "declared but never referenced"). On sm_12x // (VT_FP4_MMA_SM120A defined) they are compiled exactly as before — byte // identical. See .agents/specs/cuda-arch-additivity.md §W9 (BACKEND-CUDA-SM090). -#if defined(VT_FP4_MMA_SM120A) -bool FusedFp4VectorEnabled() { - static const bool on = [] { - const char* e = std::getenv("VT_FP4_FUSED_VEC"); - return e != nullptr && e[0] == '1'; - }(); - return on; -} - -bool PointerAligned(const void* pointer, uintptr_t alignment) { - return (reinterpret_cast(pointer) & (alignment - 1)) == 0; -} -#endif // VT_FP4_MMA_SM120A - // Decode-specialized MoE grouped GEMM path toggle (M2.9; A/B, default ON). Set // VT_MOE_DECODE=0 to force the prefill-tuned BM=64 WMMA tile for small-M decode // (the pre-M2.9 behavior) for same-binary A/B. See LaunchGrouped for the gate. @@ -2233,182 +2220,9 @@ void SigmoidGateFp4QuantKernelCuda(Queue& q, Tensor& out_packed, Tensor& out_sca Check(cudaGetLastError(), "sigmoid_gate_fp4_quant kernel launch"); } -// W3-I1 specialization of vLLM@e24d1b24 -// csrc/libtorch_stable/quantization/fp4/ -// activation_nvfp4_quant_fusion_kernels.cu:30-116,120-163 -// csrc/libtorch_stable/quantization/fp4/nvfp4_utils.cuh:118-329 -// csrc/libtorch_stable/cuda_vec_utils.cuh:123-175,264-288 -// -// The upstream compiled body consumes 16 BF16 values per thread with two -// 256-bit gate/up loads, computes/rounds packed BF16 pairs, emits one E4M3 scale -// byte and one 64-bit E2M1 payload, and launches over actual rows/groups. vLLM's -// generated graph pre-zeroes padded scale storage outside the custom body. Our -// pooled DBuf storage is dirty, so the launcher below records an explicit -// cudaMemsetAsync before this kernel. That lifecycle is intentionally visible -// to graph tracing and remains a separately measurable W3-I2 opportunity. -#if defined(VT_FP4_MMA_SM120A) -struct alignas(32) PackedBf16x16 { - uint32_t words[8]; -}; - -struct PackedFp4x16 { - uint32_t lo; - uint32_t hi; -}; - -__device__ __forceinline__ uint32_t CanonicalizeFp4NegativeZero( - uint32_t packed) { - // Blackwell's packed E2M1 conversion preserves the sign of values that - // round to zero. The established vt exact-mode reference deliberately - // canonicalizes both signed zeros to nibble 0x0. Detect a non-zero - // magnitude independently in every nibble, then retain its sign bit only - // when at least one of the three magnitude bits is set. - constexpr uint32_t kMagnitude = 0x77777777u; - constexpr uint32_t kSign = 0x88888888u; - const uint32_t magnitude = packed & kMagnitude; - const uint32_t nonzero_sign = - ((magnitude << 1) | (magnitude << 2) | (magnitude << 3)) & kSign; - return packed & (kMagnitude | nonzero_sign); -} - -__device__ __forceinline__ void LoadBf16x16Cg(PackedBf16x16& value, - const __nv_bfloat16* pointer) { - asm volatile( - "ld.global.cg.v8.u32 {%0,%1,%2,%3,%4,%5,%6,%7}, [%8];\n" - : "=r"(value.words[0]), "=r"(value.words[1]), "=r"(value.words[2]), - "=r"(value.words[3]), "=r"(value.words[4]), "=r"(value.words[5]), - "=r"(value.words[6]), "=r"(value.words[7]) - : "l"(pointer)); -} - -__device__ __forceinline__ __nv_bfloat162& Bf16Pair(PackedBf16x16& value, - int index) { - return reinterpret_cast<__nv_bfloat162*>(value.words)[index]; -} - -__device__ __forceinline__ PackedFp4x16 PackFp4x16(float2 (&values)[8]) { - PackedFp4x16 packed; - asm volatile( - "{\n" - ".reg .b8 b0;\n" - ".reg .b8 b1;\n" - ".reg .b8 b2;\n" - ".reg .b8 b3;\n" - ".reg .b8 b4;\n" - ".reg .b8 b5;\n" - ".reg .b8 b6;\n" - ".reg .b8 b7;\n" - "cvt.rn.satfinite.e2m1x2.f32 b0, %3, %2;\n" - "cvt.rn.satfinite.e2m1x2.f32 b1, %5, %4;\n" - "cvt.rn.satfinite.e2m1x2.f32 b2, %7, %6;\n" - "cvt.rn.satfinite.e2m1x2.f32 b3, %9, %8;\n" - "cvt.rn.satfinite.e2m1x2.f32 b4, %11, %10;\n" - "cvt.rn.satfinite.e2m1x2.f32 b5, %13, %12;\n" - "cvt.rn.satfinite.e2m1x2.f32 b6, %15, %14;\n" - "cvt.rn.satfinite.e2m1x2.f32 b7, %17, %16;\n" - "mov.b32 %0, {b0, b1, b2, b3};\n" - "mov.b32 %1, {b4, b5, b6, b7};\n" - "}\n" - : "=r"(packed.lo), "=r"(packed.hi) - : "f"(values[0].x), "f"(values[0].y), "f"(values[1].x), - "f"(values[1].y), "f"(values[2].x), "f"(values[2].y), - "f"(values[3].x), "f"(values[3].y), "f"(values[4].x), - "f"(values[4].y), "f"(values[5].x), "f"(values[5].y), - "f"(values[6].x), "f"(values[6].y), "f"(values[7].x), - "f"(values[7].y)); - packed.lo = CanonicalizeFp4NegativeZero(packed.lo); - packed.hi = CanonicalizeFp4NegativeZero(packed.hi); - return packed; -} - -__global__ __launch_bounds__(512) void SiluAndMulFp4QuantPackedBf16Kernel( - uint8_t* __restrict__ packed, uint8_t* __restrict__ scale, - const __nv_bfloat16* __restrict__ gate_up, float input_global_scale, - int32_t m_rows, int32_t i_dim, int32_t scale_cols, bool approx_recip) { - const int32_t groups = i_dim / 16; - const int32_t group = - static_cast(blockIdx.y * blockDim.x + threadIdx.x); - if (group >= groups) return; - - for (int32_t row = static_cast(blockIdx.x); row < m_rows; - row += static_cast(gridDim.x)) { - const int64_t input_row_base = static_cast(row) * 2 * i_dim; - const int64_t group_base = static_cast(group) * 16; - PackedBf16x16 gate; - PackedBf16x16 up; - LoadBf16x16Cg(gate, gate_up + input_row_base + group_base); - LoadBf16x16Cg(up, gate_up + input_row_base + i_dim + group_base); - - PackedBf16x16 activation; -#pragma unroll - for (int pair = 0; pair < 8; ++pair) { - const float2 gate_pair = __bfloat1622float2(Bf16Pair(gate, pair)); - const float2 up_pair = __bfloat1622float2(Bf16Pair(up, pair)); - const float lo = - (gate_pair.x / (1.0f + expf(-gate_pair.x))) * up_pair.x; - const float hi = - (gate_pair.y / (1.0f + expf(-gate_pair.y))) * up_pair.y; - Bf16Pair(activation, pair) = __floats2bfloat162_rn(lo, hi); - } - - __nv_bfloat162 local_max = __habs2(Bf16Pair(activation, 0)); -#pragma unroll - for (int pair = 1; pair < 8; ++pair) { - local_max = __hmax2(local_max, __habs2(Bf16Pair(activation, pair))); - } - const float2 max_pair = __bfloat1622float2(local_max); - const float vmax = fmaxf(max_pair.x, max_pair.y); - const float inverse_six = - approx_recip ? ReciprocalApproximateFtz(6.0f) : (1.0f / 6.0f); - float sf = input_global_scale * (vmax * inverse_six); - sf = fminf(fmaxf(sf, -448.0f), 448.0f); - const uint8_t sf8 = F32ToFp8Dev(sf); - scale[CutlassScaleOffset(row, group, scale_cols)] = sf8; - - const float sf_value = F8E4M3ToF32Dev(sf8); - float output_scale = 0.0f; - if (sf_value != 0.0f) { - output_scale = - approx_recip - ? ReciprocalApproximateFtz( - sf_value * ReciprocalApproximateFtz(input_global_scale)) - : input_global_scale / sf_value; - } - float2 values[8]; -#pragma unroll - for (int pair = 0; pair < 8; ++pair) { - values[pair] = __bfloat1622float2(Bf16Pair(activation, pair)); - values[pair].x *= output_scale; - values[pair].y *= output_scale; - } - const PackedFp4x16 fp4 = PackFp4x16(values); - const uint64_t packed64 = - (static_cast(fp4.hi) << 32) | fp4.lo; - const int64_t output_byte = - (static_cast(row) * i_dim + group_base) / 2; - *reinterpret_cast(packed + output_byte) = packed64; - } -} - -int PackedFusedFp4ResidentBlocks() { - static const int blocks = [] { - int device = 0; - int multiprocessors = 0; - int blocks_per_multiprocessor = 0; - Check(cudaGetDevice(&device), "packed fused quant get device"); - Check(cudaDeviceGetAttribute(&multiprocessors, cudaDevAttrMultiProcessorCount, - device), - "packed fused quant multiprocessor count"); - Check(cudaOccupancyMaxActiveBlocksPerMultiprocessor( - &blocks_per_multiprocessor, - SiluAndMulFp4QuantPackedBf16Kernel, 512, 0), - "packed fused quant occupancy"); - return std::max(1, multiprocessors * blocks_per_multiprocessor); - }(); - return blocks; -} -#endif - +// The sm12x-only packed BF16 producer is compiled in +// cuda_nvfp4_sm12x.cu. This common TU retains the portable producer below for +// every release SM and calls the native producer through an exact-arch seam. // Exact one-input custom-op form used by vLLM's ActivationQuantFusionPass. // Upstream: vllm@e24d1b24 // csrc/libtorch_stable/quantization/fp4/ @@ -2544,26 +2358,10 @@ void SiluAndMulFp4QuantKernelCuda(Queue& q, Tensor& out_packed, auto* sc = out_scale.Ptr(); const bool approx = NativeFp4MmaEnabled(); #if defined(VT_FP4_MMA_SM120A) - const bool packed_eligible = - FusedFp4VectorEnabled() && swizzled && gate_up.dtype == DType::kBF16 && - m <= std::numeric_limits::max() && - i <= std::numeric_limits::max() && - scale_cols <= std::numeric_limits::max() && - PointerAligned(gate_up.data, 32) && PointerAligned(pk, 8); - if (packed_eligible) { - Check(cudaMemsetAsync(sc, 0, static_cast(out_scale.Numel()), s), - "packed fused quant zero scale"); - const int groups = static_cast(i / 16); - const int block = std::min(groups, 512); - const int grid_y = (groups + block - 1) / block; - const int grid_x = std::min( - static_cast(m), - std::max(1, PackedFusedFp4ResidentBlocks() / grid_y)); - SiluAndMulFp4QuantPackedBf16Kernel<<>>( - pk, sc, gate_up.Ptr<__nv_bfloat16>(), input_global_scale_inv, - static_cast(m), static_cast(i), - static_cast(scale_cols), approx); - Check(cudaGetLastError(), "packed silu_and_mul_fp4_quant kernel launch"); + if (swizzled && gate_up.dtype == DType::kBF16 && + TryLaunchSiluAndMulFp4QuantPackedSm12x( + s, pk, sc, gate_up.data, input_global_scale_inv, m, i, scale_cols, + out_scale.Numel(), approx)) { return; } #endif @@ -2815,170 +2613,6 @@ __global__ void MatmulNvfp4Fp4Wmma(Tout* out, const uint8_t* a_packed, const uin #endif } -// --- NATIVE block-scaled fp4xfp4 MMA (Blackwell sm120a mxf4nvf4 tensor cores) -- -// Mirrors vllm's cutlass_scaled_fp4_mm_sm120a: the fp4 operands and their fp8 -// (e4m3) group-16 block scales are fed DIRECTLY to the block-scaled tensor-core -// MMA (no bf16 dequant), fp32 accumulate, then x alpha. This is the true W4A4 -// GEMM the 27B oracle golden was captured with (native != bf16-dequant on -// near-ties). Only compiled when the TU is built for the architecture-specific -// sm_120a/sm_121a target (__CUDA_ARCH_SPECIFIC__); the host gates the launch on -// the VT_FP4_MMA_SM120A build define so non-a builds keep the bf16 path. -// -// The per-thread operand/scale layout was validated device-vs-CPU bit-exact on -// GB10 sm_121a (m16n8k64 e2m1, scale_vec::4X ue4m3): -// lane: g=lane/4 (0..7), t=lane%4 (0..3) -// A frags a0(row g, k=t*8+j) a1(row g+8,..) a2(row g,32+t*8+j) a3(row g+8,..) -// B frags b0(n g, k=t*8+j) b1(n g,32+t*8+j); nibble j -> k-elem j -// D d0(g,t*2) d1(g,t*2+1) d2(g+8,t*2) d3(g+8,t*2+1) -// A scale: row r held in lane (r%8)*4 + (r>=8?1:0), byte b = k-block b -// B scale: col n held in lane n*4, byte b = k-block b -// Consumed only by the MatmulNvfp4Fp4Native template below, which is instantiated -// exclusively from the native fp4 tactic (under VT_FP4_MMA_SM120A). Gate on BOTH -// so a cross-family single-arch build where __CUDA_ARCH_SPECIFIC__ is defined -// (e.g. sm_90a) but the native path is OFF does not leave this device helper -// referenced nowhere (-Werror #177-D). On sm_12x both are defined -> byte -// identical. See .agents/specs/cuda-arch-additivity.md §W9 (BACKEND-CUDA-SM090). -#if defined(__CUDA_ARCH_SPECIFIC__) && defined(VT_FP4_MMA_SM120A) -__device__ __forceinline__ uint8_t GetNib(const uint8_t* p, int64_t row, - int64_t col, int64_t k) { - const uint8_t byte = p[row * (k / 2) + col / 2]; - return (col & 1) ? static_cast(byte >> 4) : static_cast(byte & 0xFu); -} -#endif - -template -__global__ void MatmulNvfp4Fp4Native(Tout* out, const uint8_t* a_packed, const uint8_t* a_scale, - const uint8_t* b_packed, const uint8_t* b_scale, float alpha, - int64_t M, int64_t N, int64_t K) { - const int lane = static_cast(threadIdx.x); - const int g = lane / 4, t = lane % 4; - const int64_t m0 = static_cast(blockIdx.y) * 16; - const int64_t n0 = static_cast(blockIdx.x) * 8; - const int64_t groups = K / 16; - float d0 = 0.0f, d1 = 0.0f, d2 = 0.0f, d3 = 0.0f; -#if defined(__CUDA_ARCH_SPECIFIC__) - for (int64_t k0 = 0; k0 < K; k0 += 64) { - const int64_t rA = m0 + g, rA8 = m0 + g + 8, rB = n0 + g; - uint32_t a0 = 0, a1 = 0, a2 = 0, a3 = 0, b0 = 0, b1 = 0; -#pragma unroll - for (int j = 0; j < 8; ++j) { - const int64_t ka = k0 + t * 8 + j, kb = k0 + 32 + t * 8 + j; - if (rA < M) { - if (ka < K) a0 |= static_cast(GetNib(a_packed, rA, ka, K)) << (4 * j); - if (kb < K) a2 |= static_cast(GetNib(a_packed, rA, kb, K)) << (4 * j); - } - if (rA8 < M) { - if (ka < K) a1 |= static_cast(GetNib(a_packed, rA8, ka, K)) << (4 * j); - if (kb < K) a3 |= static_cast(GetNib(a_packed, rA8, kb, K)) << (4 * j); - } - if (rB < N) { - if (ka < K) b0 |= static_cast(GetNib(b_packed, rB, ka, K)) << (4 * j); - if (kb < K) b1 |= static_cast(GetNib(b_packed, rB, kb, K)) << (4 * j); - } - } - // 1.0 in fp8-e4m3 = 0x38: harmless default for unused/tail scale bytes. - uint32_t sfa = 0x38383838u, sfb = 0x38383838u; - const int64_t scaleRowA = (t == 0) ? (m0 + g) : (t == 1 ? (m0 + g + 8) : -1); - if (scaleRowA >= 0 && scaleRowA < M) { - uint32_t v = 0; -#pragma unroll - for (int b = 0; b < 4; ++b) { - const int64_t gc = k0 / 16 + b; - const uint8_t sv = (gc < groups) ? a_scale[scaleRowA * groups + gc] : 0x38u; - v |= static_cast(sv) << (8 * b); - } - sfa = v; - } - if (t == 0) { - const int64_t cB = n0 + g; - if (cB < N) { - uint32_t v = 0; -#pragma unroll - for (int b = 0; b < 4; ++b) { - const int64_t gc = k0 / 16 + b; - const uint8_t sv = (gc < groups) ? b_scale[cB * groups + gc] : 0x38u; - v |= static_cast(sv) << (8 * b); - } - sfb = v; - } - } - asm volatile( - "mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X." - "f32.e2m1.e2m1.f32.ue4m3 " - "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%10,%11,%12,%13}, " - "%14, {%15, %16}, %17, {%18, %19};\n" - : "=f"(d0), "=f"(d1), "=f"(d2), "=f"(d3) - : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1), - "f"(d0), "f"(d1), "f"(d2), "f"(d3), - "r"(sfa), "n"(0), "n"(0), "r"(sfb), "n"(0), "n"(0)); - } -#else - (void)a_packed; (void)a_scale; (void)b_packed; (void)b_scale; (void)groups; (void)t; (void)g; -#endif - const int64_t r = m0 + g, r8 = m0 + g + 8, c0 = n0 + t * 2, c1 = n0 + t * 2 + 1; - if (r < M && c0 < N) Store(out, r * N + c0, alpha * d0); - if (r < M && c1 < N) Store(out, r * N + c1, alpha * d1); - if (r8 < M && c0 < N) Store(out, r8 * N + c0, alpha * d2); - if (r8 < M && c1 < N) Store(out, r8 * N + c1, alpha * d3); -} - -// ── RUNTIME SM-DISPATCH TACTIC — the ONE tactic registered today ───────────── -// BACKEND-CUDA-ARCH-ADDITIVITY seam-gap #2 (.agents/specs/cuda-arch-additivity.md). -// The native block-scaled fp4xfp4 path used to be reachable only through a bare -// `#if defined(VT_FP4_MMA_SM120A)` inside LaunchFp4Fp4 — a COMPILE-time arch -// assumption with no runtime check, so a fat binary that also targeted another -// architecture would have entered this path on a device whose tensor cores do -// not implement `mma.sync ... kind::mxf4nvf4`. It is now a registered tactic -// with an explicit capability predicate, selected at launch time. -// -// ADDING AN ARCHITECTURE: write its kernel body in its OWN TU, give it a -// `supports`/`launch` pair over the same Nvfp4Fp4MmaArgs, and call -// RegisterArchTactic() from a static registrar there. Nothing in THIS file, and -// nothing in LaunchFp4Fp4, changes. (Hopper wgmma / sm_100 tcgen05 bodies are a -// separate, hardware-blocked kernel campaign — see the spec's Risks section.) -#if defined(VT_FP4_MMA_SM120A) -bool Sm12xFp4MmaSupports(const DeviceCaps& caps) { - // `mma.sync ... kind::mxf4nvf4` is consumer-Blackwell (sm_120/sm_121) only and - // is emitted only for the architecture-SPECIFIC 'a' target that the CMake - // `fp4-mma` FEATURE-TABLE row gates VT_FP4_MMA_SM120A on. The env A/B toggle - // stays INSIDE the predicate so VT_NVFP4_FP4_NATIVE keeps exactly its previous - // same-binary meaning (default OFF -> tactic not selected -> portable path). - return caps.valid && caps.sm_major == 12 && NativeFp4MmaEnabled(); -} - -bool Sm12xFp4MmaLaunch(const DeviceCaps&, void* args_v) { - const auto& a = *static_cast(args_v); - auto* s = static_cast(a.stream); - const dim3 grid(static_cast((a.n + 7) / 8), static_cast((a.m + 15) / 16)); - switch (a.out_dtype) { - case DType::kF32: - MatmulNvfp4Fp4Native<<>>(static_cast(a.out), a.a_packed, - a.a_scale, a.b_packed, a.b_scale, a.alpha, - a.m, a.n, a.k); - break; - case DType::kBF16: - MatmulNvfp4Fp4Native<__nv_bfloat16><<>>( - static_cast<__nv_bfloat16*>(a.out), a.a_packed, a.a_scale, a.b_packed, a.b_scale, a.alpha, - a.m, a.n, a.k); - break; - default: - return false; // declined -> launcher keeps its portable path - } - Check(cudaGetLastError(), "matmul_nvfp4_fp4 kernel launch (native sm120a)"); - return true; -} -#endif // VT_FP4_MMA_SM120A - -// Table fill only — no CUDA API calls before main() (cuda_ops.cu discipline). -struct Fp4MmaTacticRegistrar { - Fp4MmaTacticRegistrar() { -#if defined(VT_FP4_MMA_SM120A) - RegisterArchTactic( - TacticFamily::kNvfp4Fp4Mma, - ArchTactic{"nvfp4-fp4-mma/sm12x", &Sm12xFp4MmaSupports, &Sm12xFp4MmaLaunch}); -#endif - } -} fp4_mma_tactic_registrar; template void LaunchFp4Fp4(cudaStream_t s, const DeviceCaps& caps, Tensor& out, const Tensor& a_packed, diff --git a/src/vt/cuda/cuda_nvfp4_sm12x.cu b/src/vt/cuda/cuda_nvfp4_sm12x.cu new file mode 100644 index 000000000..c500a5238 --- /dev/null +++ b/src/vt/cuda/cuda_nvfp4_sm12x.cu @@ -0,0 +1,422 @@ +// Architecture-specific NVFP4 bodies split from cuda_matmul_nvfp4.cu for the +// W1 heterogeneous fat binary. This TU is gencode'd only for sm_120a/sm_121a; +// the portable NVFP4 kernels remain in the common TU for all release SMs. +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "vt/cuda/cuda_arch_tactics.h" +#include "vt/cuda/cuda_device_caps.h" +#include "vt/cuda/cuda_nvfp4_sm12x.h" + +namespace vt::cuda { +namespace { + +void Check(cudaError_t err, const char* what) { + if (err != cudaSuccess) { + throw std::runtime_error(std::string("vt cuda: nvfp4 sm12x: ") + what + + ": " + cudaGetErrorString(err)); + } +} + +bool NativeFp4MmaEnabled() { + const char* value = std::getenv("VT_NVFP4_FP4_NATIVE"); + return value != nullptr && value[0] == '1'; +} + +bool FusedFp4VectorEnabled() { + const char* value = std::getenv("VT_FP4_FUSED_VEC"); + return value != nullptr && value[0] == '1'; +} + +bool PointerAligned(const void* pointer, uintptr_t alignment) { + return (reinterpret_cast(pointer) & (alignment - 1)) == 0; +} + +__device__ __forceinline__ float F8E4M3ToF32Dev(uint8_t byte) { + const uint32_t sign = static_cast(byte >> 7) & 0x1u; + const uint32_t exp = static_cast(byte >> 3) & 0xFu; + const uint32_t mant = static_cast(byte) & 0x7u; + const float sign_mul = sign ? -1.0f : 1.0f; + if (exp == 0xFu && mant == 0x7u) return CUDART_NAN_F; + if (exp == 0u) return sign_mul * (static_cast(mant) * (1.0f / 512.0f)); + return sign_mul * + ldexpf(1.0f + static_cast(mant) * (1.0f / 8.0f), + static_cast(exp) - 7); +} + +__device__ __forceinline__ uint8_t F32ToFp8Dev(float value) { + return static_cast( + __nv_cvt_float_to_fp8(value, __NV_SATFINITE, __NV_E4M3)); +} + +__device__ __forceinline__ float ReciprocalApproximateFtz(float value) { + float reciprocal; + asm volatile("rcp.approx.ftz.f32 %0, %1;" + : "=f"(reciprocal) + : "f"(value)); + return reciprocal; +} + +__device__ __forceinline__ int64_t CutlassScaleOffset( + int64_t row, int64_t column, int64_t padded_cols) { + const int64_t m_tile = row / 128; + const int64_t outer_m = row % 32; + const int64_t inner_m = (row % 128) / 32; + const int64_t k_tile = column / 4; + const int64_t inner_k = column % 4; + return ((((m_tile * (padded_cols / 4) + k_tile) * 32 + outer_m) * 4 + + inner_m) * + 4 + + inner_k); +} + +struct alignas(32) PackedBf16x16 { + uint32_t words[8]; +}; + +struct PackedFp4x16 { + uint32_t lo; + uint32_t hi; +}; + +__device__ __forceinline__ uint32_t CanonicalizeFp4NegativeZero( + uint32_t packed) { + constexpr uint32_t kMagnitude = 0x77777777u; + constexpr uint32_t kSign = 0x88888888u; + const uint32_t magnitude = packed & kMagnitude; + const uint32_t nonzero_sign = + ((magnitude << 1) | (magnitude << 2) | (magnitude << 3)) & kSign; + return packed & (kMagnitude | nonzero_sign); +} + +__device__ __forceinline__ void LoadBf16x16Cg( + PackedBf16x16& value, const __nv_bfloat16* pointer) { + asm volatile( + "ld.global.cg.v8.u32 {%0,%1,%2,%3,%4,%5,%6,%7}, [%8];\n" + : "=r"(value.words[0]), "=r"(value.words[1]), + "=r"(value.words[2]), "=r"(value.words[3]), + "=r"(value.words[4]), "=r"(value.words[5]), + "=r"(value.words[6]), "=r"(value.words[7]) + : "l"(pointer)); +} + +__device__ __forceinline__ __nv_bfloat162& Bf16Pair( + PackedBf16x16& value, int index) { + return reinterpret_cast<__nv_bfloat162*>(value.words)[index]; +} + +__device__ __forceinline__ PackedFp4x16 PackFp4x16(float2 (&values)[8]) { + PackedFp4x16 packed; + asm volatile( + "{\n" + ".reg .b8 b0; .reg .b8 b1; .reg .b8 b2; .reg .b8 b3;\n" + ".reg .b8 b4; .reg .b8 b5; .reg .b8 b6; .reg .b8 b7;\n" + "cvt.rn.satfinite.e2m1x2.f32 b0, %3, %2;\n" + "cvt.rn.satfinite.e2m1x2.f32 b1, %5, %4;\n" + "cvt.rn.satfinite.e2m1x2.f32 b2, %7, %6;\n" + "cvt.rn.satfinite.e2m1x2.f32 b3, %9, %8;\n" + "cvt.rn.satfinite.e2m1x2.f32 b4, %11, %10;\n" + "cvt.rn.satfinite.e2m1x2.f32 b5, %13, %12;\n" + "cvt.rn.satfinite.e2m1x2.f32 b6, %15, %14;\n" + "cvt.rn.satfinite.e2m1x2.f32 b7, %17, %16;\n" + "mov.b32 %0, {b0, b1, b2, b3};\n" + "mov.b32 %1, {b4, b5, b6, b7};\n" + "}\n" + : "=r"(packed.lo), "=r"(packed.hi) + : "f"(values[0].x), "f"(values[0].y), "f"(values[1].x), + "f"(values[1].y), "f"(values[2].x), "f"(values[2].y), + "f"(values[3].x), "f"(values[3].y), "f"(values[4].x), + "f"(values[4].y), "f"(values[5].x), "f"(values[5].y), + "f"(values[6].x), "f"(values[6].y), "f"(values[7].x), + "f"(values[7].y)); + packed.lo = CanonicalizeFp4NegativeZero(packed.lo); + packed.hi = CanonicalizeFp4NegativeZero(packed.hi); + return packed; +} + +__global__ __launch_bounds__(512) void SiluAndMulFp4QuantPackedBf16Kernel( + uint8_t* __restrict__ packed, uint8_t* __restrict__ scale, + const __nv_bfloat16* __restrict__ gate_up, float input_global_scale, + int32_t m_rows, int32_t i_dim, int32_t scale_cols, bool approx_recip) { + const int32_t groups = i_dim / 16; + const int32_t group = + static_cast(blockIdx.y * blockDim.x + threadIdx.x); + if (group >= groups) return; + for (int32_t row = static_cast(blockIdx.x); row < m_rows; + row += static_cast(gridDim.x)) { + const int64_t input_row_base = static_cast(row) * 2 * i_dim; + const int64_t group_base = static_cast(group) * 16; + PackedBf16x16 gate; + PackedBf16x16 up; + LoadBf16x16Cg(gate, gate_up + input_row_base + group_base); + LoadBf16x16Cg(up, gate_up + input_row_base + i_dim + group_base); + PackedBf16x16 activation; +#pragma unroll + for (int pair = 0; pair < 8; ++pair) { + const float2 gate_pair = __bfloat1622float2(Bf16Pair(gate, pair)); + const float2 up_pair = __bfloat1622float2(Bf16Pair(up, pair)); + const float lo = + (gate_pair.x / (1.0f + expf(-gate_pair.x))) * up_pair.x; + const float hi = + (gate_pair.y / (1.0f + expf(-gate_pair.y))) * up_pair.y; + Bf16Pair(activation, pair) = __floats2bfloat162_rn(lo, hi); + } + __nv_bfloat162 local_max = __habs2(Bf16Pair(activation, 0)); +#pragma unroll + for (int pair = 1; pair < 8; ++pair) { + local_max = __hmax2(local_max, __habs2(Bf16Pair(activation, pair))); + } + const float2 max_pair = __bfloat1622float2(local_max); + const float vmax = fmaxf(max_pair.x, max_pair.y); + const float inverse_six = + approx_recip ? ReciprocalApproximateFtz(6.0f) : (1.0f / 6.0f); + float sf = input_global_scale * (vmax * inverse_six); + sf = fminf(fmaxf(sf, -448.0f), 448.0f); + const uint8_t sf8 = F32ToFp8Dev(sf); + scale[CutlassScaleOffset(row, group, scale_cols)] = sf8; + const float sf_value = F8E4M3ToF32Dev(sf8); + float output_scale = 0.0f; + if (sf_value != 0.0f) { + output_scale = + approx_recip + ? ReciprocalApproximateFtz( + sf_value * ReciprocalApproximateFtz(input_global_scale)) + : input_global_scale / sf_value; + } + float2 values[8]; +#pragma unroll + for (int pair = 0; pair < 8; ++pair) { + values[pair] = __bfloat1622float2(Bf16Pair(activation, pair)); + values[pair].x *= output_scale; + values[pair].y *= output_scale; + } + const PackedFp4x16 fp4 = PackFp4x16(values); + const uint64_t packed64 = + (static_cast(fp4.hi) << 32) | fp4.lo; + const int64_t output_byte = + (static_cast(row) * i_dim + group_base) / 2; + *reinterpret_cast(packed + output_byte) = packed64; + } +} + +int PackedFusedFp4ResidentBlocks() { + static const int blocks = [] { + int device = 0; + int multiprocessors = 0; + int blocks_per_multiprocessor = 0; + Check(cudaGetDevice(&device), "packed producer get device"); + Check(cudaDeviceGetAttribute(&multiprocessors, + cudaDevAttrMultiProcessorCount, device), + "packed producer multiprocessor count"); + Check(cudaOccupancyMaxActiveBlocksPerMultiprocessor( + &blocks_per_multiprocessor, + SiluAndMulFp4QuantPackedBf16Kernel, 512, 0), + "packed producer occupancy"); + return std::max(1, multiprocessors * blocks_per_multiprocessor); + }(); + return blocks; +} + +__device__ __forceinline__ uint8_t GetNib( + const uint8_t* pointer, int64_t row, int64_t column, int64_t k) { + const uint8_t byte = pointer[row * (k / 2) + column / 2]; + return (column & 1) ? static_cast(byte >> 4) + : static_cast(byte & 0xFu); +} + +__device__ inline void Store(float* pointer, int64_t index, float value) { + pointer[index] = value; +} +__device__ inline void Store(__nv_bfloat16* pointer, int64_t index, + float value) { + pointer[index] = __float2bfloat16(value); +} + +template +__global__ void MatmulNvfp4Fp4Native( + Tout* out, const uint8_t* a_packed, const uint8_t* a_scale, + const uint8_t* b_packed, const uint8_t* b_scale, float alpha, + int64_t m_rows, int64_t n_cols, int64_t k_dim) { + const int lane = static_cast(threadIdx.x); + const int g = lane / 4; + const int t = lane % 4; + const int64_t m0 = static_cast(blockIdx.y) * 16; + const int64_t n0 = static_cast(blockIdx.x) * 8; + const int64_t groups = k_dim / 16; + float d0 = 0.0f; + float d1 = 0.0f; + float d2 = 0.0f; + float d3 = 0.0f; + for (int64_t k0 = 0; k0 < k_dim; k0 += 64) { + const int64_t row_a = m0 + g; + const int64_t row_a8 = row_a + 8; + const int64_t row_b = n0 + g; + uint32_t a0 = 0, a1 = 0, a2 = 0, a3 = 0, b0 = 0, b1 = 0; +#pragma unroll + for (int j = 0; j < 8; ++j) { + const int64_t ka = k0 + t * 8 + j; + const int64_t kb = k0 + 32 + t * 8 + j; + if (row_a < m_rows) { + if (ka < k_dim) { + a0 |= static_cast(GetNib(a_packed, row_a, ka, k_dim)) + << (4 * j); + } + if (kb < k_dim) { + a2 |= static_cast(GetNib(a_packed, row_a, kb, k_dim)) + << (4 * j); + } + } + if (row_a8 < m_rows) { + if (ka < k_dim) { + a1 |= static_cast(GetNib(a_packed, row_a8, ka, k_dim)) + << (4 * j); + } + if (kb < k_dim) { + a3 |= static_cast(GetNib(a_packed, row_a8, kb, k_dim)) + << (4 * j); + } + } + if (row_b < n_cols) { + if (ka < k_dim) { + b0 |= static_cast(GetNib(b_packed, row_b, ka, k_dim)) + << (4 * j); + } + if (kb < k_dim) { + b1 |= static_cast(GetNib(b_packed, row_b, kb, k_dim)) + << (4 * j); + } + } + } + uint32_t scale_a = 0x38383838u; + uint32_t scale_b = 0x38383838u; + const int64_t scale_row_a = + (t == 0) ? (m0 + g) : (t == 1 ? (m0 + g + 8) : -1); + if (scale_row_a >= 0 && scale_row_a < m_rows) { + uint32_t value = 0; +#pragma unroll + for (int block = 0; block < 4; ++block) { + const int64_t group = k0 / 16 + block; + const uint8_t scale = + group < groups ? a_scale[scale_row_a * groups + group] : 0x38u; + value |= static_cast(scale) << (8 * block); + } + scale_a = value; + } + if (t == 0 && row_b < n_cols) { + uint32_t value = 0; +#pragma unroll + for (int block = 0; block < 4; ++block) { + const int64_t group = k0 / 16 + block; + const uint8_t scale = + group < groups ? b_scale[row_b * groups + group] : 0x38u; + value |= static_cast(scale) << (8 * block); + } + scale_b = value; + } + asm volatile( + "mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X." + "f32.e2m1.e2m1.f32.ue4m3 " + "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%10,%11,%12,%13}, " + "%14, {%15, %16}, %17, {%18, %19};\n" + : "=f"(d0), "=f"(d1), "=f"(d2), "=f"(d3) + : "r"(a0), "r"(a1), "r"(a2), "r"(a3), "r"(b0), "r"(b1), + "f"(d0), "f"(d1), "f"(d2), "f"(d3), "r"(scale_a), + "n"(0), "n"(0), "r"(scale_b), "n"(0), "n"(0)); + } + const int64_t row = m0 + g; + const int64_t row8 = row + 8; + const int64_t col0 = n0 + t * 2; + const int64_t col1 = col0 + 1; + if (row < m_rows && col0 < n_cols) { + Store(out, row * n_cols + col0, alpha * d0); + } + if (row < m_rows && col1 < n_cols) { + Store(out, row * n_cols + col1, alpha * d1); + } + if (row8 < m_rows && col0 < n_cols) { + Store(out, row8 * n_cols + col0, alpha * d2); + } + if (row8 < m_rows && col1 < n_cols) { + Store(out, row8 * n_cols + col1, alpha * d3); + } +} + +bool Sm12xFp4MmaSupports(const DeviceCaps& caps) { + return caps.valid && caps.sm_major == 12 && NativeFp4MmaEnabled(); +} + +bool Sm12xFp4MmaLaunch(const DeviceCaps&, void* args_value) { + const auto& args = *static_cast(args_value); + auto* stream = static_cast(args.stream); + const dim3 grid(static_cast((args.n + 7) / 8), + static_cast((args.m + 15) / 16)); + switch (args.out_dtype) { + case DType::kF32: + MatmulNvfp4Fp4Native<<>>( + static_cast(args.out), args.a_packed, args.a_scale, + args.b_packed, args.b_scale, args.alpha, args.m, args.n, args.k); + break; + case DType::kBF16: + MatmulNvfp4Fp4Native<__nv_bfloat16><<>>( + static_cast<__nv_bfloat16*>(args.out), args.a_packed, args.a_scale, + args.b_packed, args.b_scale, args.alpha, args.m, args.n, args.k); + break; + default: + return false; + } + Check(cudaGetLastError(), "native fp4 MMA launch"); + return true; +} + +struct Fp4MmaTacticRegistrar { + Fp4MmaTacticRegistrar() { + RegisterArchTactic( + TacticFamily::kNvfp4Fp4Mma, + ArchTactic{"nvfp4-fp4-mma/sm12x", &Sm12xFp4MmaSupports, + &Sm12xFp4MmaLaunch}); + } +} fp4_mma_tactic_registrar; + +} // namespace + +bool TryLaunchSiluAndMulFp4QuantPackedSm12x( + void* stream_value, uint8_t* packed, uint8_t* scale, const void* gate_up, + float input_global_scale, int64_t m_rows, int64_t i_dim, + int64_t scale_cols, int64_t scale_numel, bool approx_recip) { + const DeviceCaps& caps = GetDeviceCaps(); + const bool eligible = + FusedFp4VectorEnabled() && caps.valid && caps.sm_major == 12 && + m_rows <= std::numeric_limits::max() && + i_dim <= std::numeric_limits::max() && + scale_cols <= std::numeric_limits::max() && + PointerAligned(gate_up, 32) && PointerAligned(packed, 8); + if (!eligible) return false; + auto* stream = static_cast(stream_value); + Check(cudaMemsetAsync(scale, 0, static_cast(scale_numel), stream), + "packed producer zero scale"); + const int groups = static_cast(i_dim / 16); + const int block = std::min(groups, 512); + const int grid_y = (groups + block - 1) / block; + const int grid_x = std::min( + static_cast(m_rows), + std::max(1, PackedFusedFp4ResidentBlocks() / grid_y)); + SiluAndMulFp4QuantPackedBf16Kernel<<>>( + packed, scale, static_cast(gate_up), + input_global_scale, static_cast(m_rows), + static_cast(i_dim), static_cast(scale_cols), + approx_recip); + Check(cudaGetLastError(), "packed producer launch"); + return true; +} + +} // namespace vt::cuda diff --git a/src/vt/cuda/cuda_nvfp4_sm12x.h b/src/vt/cuda/cuda_nvfp4_sm12x.h new file mode 100644 index 000000000..cec2abe88 --- /dev/null +++ b/src/vt/cuda/cuda_nvfp4_sm12x.h @@ -0,0 +1,15 @@ +#pragma once + +#include + +namespace vt::cuda { + +// Launch the optional sm12x packed BF16 SiLU->NVFP4 producer. Returns false +// when the device, shape, layout, alignment, or A/B flag selects the portable +// producer retained in cuda_matmul_nvfp4.cu. +bool TryLaunchSiluAndMulFp4QuantPackedSm12x( + void* stream, uint8_t* packed, uint8_t* scale, const void* gate_up, + float input_global_scale, int64_t m_rows, int64_t i_dim, + int64_t scale_cols, int64_t scale_numel, bool approx_recip); + +} // namespace vt::cuda diff --git a/src/vt/cuda/triton_aot_arch_dispatch.h b/src/vt/cuda/triton_aot_arch_dispatch.h new file mode 100644 index 000000000..eaba11bbd --- /dev/null +++ b/src/vt/cuda/triton_aot_arch_dispatch.h @@ -0,0 +1,42 @@ +// Exact runtime selector for the six vendored Triton AOT CUDA trees shipped in +// the release fat binary. The other four release SMs use the portable CUDA +// implementation; a cubin is never tried on a merely similar architecture. +#pragma once + +#include +#include +#include + +namespace vt::cuda { + +inline constexpr int TritonAotTreeIndex(int major, int minor) { + if (major == 8 && minor == 0) return 0; + if (major == 8 && minor == 6) return 1; + if (major == 8 && minor == 9) return 2; + if (major == 9 && minor == 0) return 3; + if (major == 10 && minor == 0) return 4; + if (major == 12 && minor == 1) return 5; + return -1; +} + +template +Result DispatchTritonAot(int major, int minor, Result fallback, + const std::array& trees, + Args&&... args) { + const int index = TritonAotTreeIndex(major, minor); + if (index < 0) return fallback; + return trees[static_cast(index)]( + std::forward(args)...); +} + +template +bool DispatchTritonAotVoid(int major, int minor, + const std::array& trees, + Args&&... args) { + const int index = TritonAotTreeIndex(major, minor); + if (index < 0) return false; + trees[static_cast(index)](std::forward(args)...); + return true; +} + +} // namespace vt::cuda diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index acb7a4ddc..e0bfe4122 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -739,6 +739,14 @@ vllm_cpp_add_test(test_backend_cross_device vt/test_backend_cross_device.cpp) # of that skeleton carrying a DECISION is gated on a CPU-only CI runner with no # AMD GPU. The rest of the ROCm skeleton is under VLLM_CPP_HIP below. vllm_cpp_add_test(test_rocm_arch vt/test_rocm_arch.cpp) +vllm_cpp_add_test(test_triton_aot_arch_dispatch + vt/test_triton_aot_arch_dispatch.cpp) +target_include_directories(test_triton_aot_arch_dispatch PRIVATE + ${CMAKE_SOURCE_DIR}/src) +vllm_cpp_add_test(test_cpu_isa_x86 vt/test_cpu_isa_x86.cpp) +target_include_directories(test_cpu_isa_x86 PRIVATE ${CMAKE_SOURCE_DIR}/src) +vllm_cpp_add_test(test_cpu_isa_arm vt/test_cpu_isa_arm.cpp) +target_include_directories(test_cpu_isa_arm PRIVATE ${CMAKE_SOURCE_DIR}/src) if(VLLM_CPP_HIP) # Plain C++ (no HIP header): every assertion goes through the vt:: seam. Each # case no-ops when the build has HIP but the box has no AMD GPU. diff --git a/tests/scripts/fixtures/release_manifest/v1/cpu-input.json b/tests/scripts/fixtures/release_manifest/v1/cpu-input.json index 4ff652d0b..895b8323e 100644 --- a/tests/scripts/fixtures/release_manifest/v1/cpu-input.json +++ b/tests/scripts/fixtures/release_manifest/v1/cpu-input.json @@ -1,5 +1,6 @@ { "artifact": { + "c_abi_version": 17, "channel": "preview", "id": "linux-x86_64-glibc-cpu", "kind": "primary", @@ -15,6 +16,7 @@ "VLLM_CPP_CUDA_ARCHITECTURES": [], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -34,6 +36,7 @@ "VLLM_CPP_CUDA_ARCHITECTURES": [], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -107,7 +110,8 @@ "avx", "avx2", "f16c", - "osxsave" + "osxsave", + "sse2" ], "required_os_state": [ "xcr0:xmm", @@ -128,8 +132,13 @@ "name": "avx512f", "required_cpu_bits": [ "avx", + "avx2", + "avx512bw", "avx512f", - "osxsave" + "avx512vl", + "f16c", + "osxsave", + "sse2" ], "required_os_state": [ "xcr0:xmm", diff --git a/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json b/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json index afdc5b43a..0dad3636b 100644 --- a/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json +++ b/tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json @@ -1,5 +1,6 @@ { "artifact": { + "c_abi_version": 17, "channel": "preview", "id": "linux-x86_64-glibc-cpu", "kind": "primary", @@ -15,6 +16,7 @@ "VLLM_CPP_CUDA_ARCHITECTURES": [], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -35,6 +37,7 @@ "VLLM_CPP_CUDA_ARCHITECTURES": [], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -108,7 +111,8 @@ "avx", "avx2", "f16c", - "osxsave" + "osxsave", + "sse2" ], "required_os_state": [ "xcr0:xmm", @@ -129,8 +133,13 @@ "name": "avx512f", "required_cpu_bits": [ "avx", + "avx2", + "avx512bw", "avx512f", - "osxsave" + "avx512vl", + "f16c", + "osxsave", + "sse2" ], "required_os_state": [ "xcr0:xmm", diff --git a/tests/scripts/fixtures/release_manifest/v1/cuda-input.json b/tests/scripts/fixtures/release_manifest/v1/cuda-input.json index 613a6a1ab..62115230c 100644 --- a/tests/scripts/fixtures/release_manifest/v1/cuda-input.json +++ b/tests/scripts/fixtures/release_manifest/v1/cuda-input.json @@ -1,5 +1,6 @@ { "artifact": { + "c_abi_version": 17, "channel": "preview", "id": "linux-x86_64-glibc-cuda-fat", "kind": "primary", @@ -26,6 +27,7 @@ ], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -56,6 +58,7 @@ ], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, diff --git a/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json b/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json index 07594ade8..349554ce8 100644 --- a/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json +++ b/tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json @@ -1,5 +1,6 @@ { "artifact": { + "c_abi_version": 17, "channel": "preview", "id": "linux-x86_64-glibc-cuda-fat", "kind": "primary", @@ -26,6 +27,7 @@ ], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, @@ -115,6 +117,7 @@ ], "VLLM_CPP_HIP": false, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": false, "VLLM_CPP_METAL": false, "VLLM_CPP_MLX": false, "VLLM_CPP_SERVER": true, diff --git a/tests/scripts/test_check_arm_isa_build.py b/tests/scripts/test_check_arm_isa_build.py new file mode 100755 index 000000000..78063d340 --- /dev/null +++ b/tests/scripts/test_check_arm_isa_build.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Mutation tests for the W4 aarch64 ISA compile-command audit.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-arm-isa-build.py" + + +def entry(source: str, flags: str = "") -> dict[str, str]: + return { + "file": str(ROOT / source), + "command": f"c++ {flags} -c {source}", + } + + +def valid_commands() -> list[dict[str, str]]: + return [ + entry("src/vt/cpu/cpu_isa_arm.cpp"), + entry("src/vt/cpu/cpu_matmul_elem.cpp"), + entry("src/vt/cpu/cpu_quant_dot.cpp"), + entry("src/vt/cpu/cpu_quant_repack.cpp"), + entry( + "src/vt/cpu/cpu_quant_dot_sdot.cpp", + "-march=armv8.2-a+dotprod+fp16", + ), + entry( + "src/vt/cpu/cpu_quant_dot_arm.cpp", + "-march=armv8.2-a+i8mm+dotprod", + ), + entry( + "src/vt/cpu/cpu_quant_repack_arm.cpp", + "-march=armv8.2-a+i8mm+dotprod", + ), + ] + + +class ArmIsaBuildContract(unittest.TestCase): + def run_checker( + self, entries: list[dict[str, str]] + ) -> subprocess.CompletedProcess[str]: + with tempfile.TemporaryDirectory() as temporary: + commands = Path(temporary) / "compile_commands.json" + commands.write_text(json.dumps(entries), encoding="utf-8") + return subprocess.run( + [ + sys.executable, + str(CHECKER), + "--compile-commands", + str(commands), + ], + text=True, + capture_output=True, + check=False, + ) + + def test_exact_per_source_flags_pass(self) -> None: + result = self.run_checker(valid_commands()) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_march_native_anywhere_fails(self) -> None: + entries = valid_commands() + entries.append(entry("src/vt/cpu/cpu_ops.cpp", "-march=native")) + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("-march=native", result.stdout + result.stderr) + + def test_baseline_source_rejects_arch_extension_leak(self) -> None: + entries = valid_commands() + entries[1] = entry("src/vt/cpu/cpu_matmul_elem.cpp", "-march=armv8.2-a+i8mm") + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("cpu_matmul_elem.cpp", result.stdout + result.stderr) + + def test_dotprod_source_requires_exact_tier(self) -> None: + entries = valid_commands() + entries[4] = entry("src/vt/cpu/cpu_quant_dot_sdot.cpp", "-march=armv8.2-a") + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("dotprod+fp16", result.stdout + result.stderr) + + def test_i8mm_sources_require_exact_tier(self) -> None: + for index in (5, 6): + with self.subTest(index=index): + entries = valid_commands() + source = entries[index]["file"] + entries[index] = { + "file": source, + "command": f"c++ -march=armv8.2-a+dotprod -c {source}", + } + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("i8mm+dotprod", result.stdout + result.stderr) + + def test_required_tier_source_must_be_compiled(self) -> None: + result = self.run_checker(valid_commands()[:-1]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("missing", result.stdout + result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_check_cpu_isa_build.py b/tests/scripts/test_check_cpu_isa_build.py new file mode 100755 index 000000000..0cdfa9995 --- /dev/null +++ b/tests/scripts/test_check_cpu_isa_build.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Mutation tests for the W3 x86 ISA compile-command audit.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-cpu-isa-build.py" + + +def entry(source: str, flags: str = "") -> dict[str, str]: + return { + "file": str(ROOT / source), + "command": f"c++ {flags} -c {source}", + } + + +def valid_commands() -> list[dict[str, str]]: + return [ + entry("src/vt/cpu/cpu_isa_x86.cpp"), + entry("src/vt/cpu/cpu_matmul_elem.cpp"), + entry("src/vt/cpu/cpu_matmul_elem_avx2.cpp", "-mavx2 -mf16c"), + entry( + "src/vt/cpu/cpu_matmul_elem_avx512.cpp", + "-mavx512f -mavx512bw -mavx512vl -mf16c", + ), + ] + + +class CpuIsaBuildContract(unittest.TestCase): + def run_checker( + self, entries: list[dict[str, str]] + ) -> subprocess.CompletedProcess[str]: + with tempfile.TemporaryDirectory() as temporary: + commands = Path(temporary) / "compile_commands.json" + commands.write_text(json.dumps(entries), encoding="utf-8") + return subprocess.run( + [ + sys.executable, + str(CHECKER), + "--compile-commands", + str(commands), + ], + text=True, + capture_output=True, + check=False, + ) + + def test_exact_per_source_flags_pass(self) -> None: + result = self.run_checker(valid_commands()) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_march_native_anywhere_fails(self) -> None: + entries = valid_commands() + entries.append(entry("src/vt/cpu/cpu_ops.cpp", "-march=native")) + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("-march=native", result.stdout + result.stderr) + + def test_baseline_dispatch_source_rejects_avx_leak(self) -> None: + entries = valid_commands() + entries[1] = entry("src/vt/cpu/cpu_matmul_elem.cpp", "-mavx2") + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("cpu_matmul_elem.cpp", result.stdout + result.stderr) + + def test_avx2_source_requires_f16c(self) -> None: + entries = valid_commands() + entries[2] = entry("src/vt/cpu/cpu_matmul_elem_avx2.cpp", "-mavx2") + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("-mf16c", result.stdout + result.stderr) + + def test_avx512_source_requires_exact_feature_set(self) -> None: + entries = valid_commands() + entries[3] = entry( + "src/vt/cpu/cpu_matmul_elem_avx512.cpp", "-mavx512f -mf16c" + ) + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + output = result.stdout + result.stderr + self.assertIn("-mavx512bw", output) + self.assertIn("-mavx512vl", output) + + def test_required_tier_source_must_be_compiled(self) -> None: + result = self.run_checker(valid_commands()[:-1]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("missing", result.stdout + result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_check_cuda_fat_gencode.py b/tests/scripts/test_check_cuda_fat_gencode.py new file mode 100644 index 000000000..8f7273071 --- /dev/null +++ b/tests/scripts/test_check_cuda_fat_gencode.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +"""Mutation tests for the W1 ten-SM fat-CUDA compile audit.""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-cuda-fat-gencode.py" +ALL = ("80", "86", "87", "89", "90a", "100a", "103a", "110", "120a", "121a") + + +def command(source: str, sms: tuple[str, ...]) -> dict[str, str]: + flags = " ".join( + f"-gencode arch=compute_{sm},code=sm_{sm}" for sm in sms + ) + return {"file": str(ROOT / source), "command": f"nvcc {flags} -c {source}"} + + +def valid_commands() -> list[dict[str, str]]: + return [ + command("src/vt/cuda/cuda_matmul_nvfp4.cu", ALL), + command("src/vt/cuda/cuda_nvfp4_sm12x.cu", ("120a", "121a")), + command("src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu", ("120a", "121a")), + command("src/vt/cuda/cuda_matmul_fp8_cutlass.cu", ("120a", "121a")), + command("src/vt/cuda/cuda_matmul_nvfp4_sm100.cu", ("100a",)), + command("src/vt/cuda/cuda_scaled_mm_c3x_sm90.cu", ("90a",)), + command("src/vt/cuda/cuda_scaled_mm_c3x_sm100.cu", ("100a",)), + command("src/vt/cuda/cuda_moe_marlin.cu", ("120a", "121a")), + command( + "src/vt/cuda/cuda_flash_attn_fa2.cu", + ("80", "86", "87", "89", "120a", "121a"), + ), + ] + + +class FatGencodeContract(unittest.TestCase): + def run_checker( + self, entries: list[dict[str, str]], archive_sms: tuple[str, ...] = ALL + ) -> subprocess.CompletedProcess[str]: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + commands = root / "compile_commands.json" + commands.write_text(json.dumps(entries), encoding="utf-8") + listing = root / "cuobjdump.txt" + listing.write_text( + "\n".join(f"Fatbin elf code: sm_{sm}" for sm in archive_sms), + encoding="utf-8", + ) + return subprocess.run( + [ + sys.executable, + str(CHECKER), + "--compile-commands", + str(commands), + "--cuobjdump-list", + str(listing), + ], + text=True, + capture_output=True, + check=False, + ) + + def test_exact_source_and_archive_matrix_passes(self) -> None: + entries = valid_commands() + entries.append( + { + "file": "/tmp/cutlass/tools/profiler/src/cublas_helpers.cu", + "command": "nvcc -c cublas_helpers.cu", + } + ) + result = self.run_checker(entries) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_portable_source_missing_one_sm_fails(self) -> None: + entries = valid_commands() + entries[0] = command("src/vt/cuda/cuda_matmul_nvfp4.cu", ALL[:-1]) + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("cuda_matmul_nvfp4.cu", result.stdout + result.stderr) + + def test_sm12x_source_rejects_incompatible_sm(self) -> None: + entries = valid_commands() + entries[1] = command( + "src/vt/cuda/cuda_nvfp4_sm12x.cu", ("90a", "120a", "121a") + ) + result = self.run_checker(entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("cuda_nvfp4_sm12x.cu", result.stdout + result.stderr) + + def test_archive_missing_one_sm_fails(self) -> None: + result = self.run_checker(valid_commands(), ALL[:-1]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("archive", result.stdout + result.stderr) + + def test_archive_rejects_undeclared_sm(self) -> None: + result = self.run_checker(valid_commands(), ALL + ("999",)) + self.assertNotEqual(result.returncode, 0) + self.assertIn("undeclared", result.stdout + result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_check_pr_size.py b/tests/scripts/test_check_pr_size.py index fab8cebfe..5253a3498 100755 --- a/tests/scripts/test_check_pr_size.py +++ b/tests/scripts/test_check_pr_size.py @@ -87,6 +87,7 @@ def test_each_mutable_surface_has_an_explicit_class(self) -> None: ".github/workflows/ci.yml": "ci", "src/vt/vulkan/vulkan_spirv.cpp": "generated", "release/manifest-v1.schema.json": "configuration", + "release/release-matrix.json": "configuration", "scripts/env-doc-allowlist.txt": "configuration", "tests/scripts/fixtures/release_manifest/v1/cpu-input.json": "asset", } @@ -273,9 +274,15 @@ def test_checker_change_requires_its_recognized_mutation_test(self) -> None: def test_every_created_checker_has_closed_bootstrap_evidence(self) -> None: expected = { + "scripts/check-arm-isa-build.py", "scripts/check-commit-trailers.py", + "scripts/check-cpu-isa-build.py", + "scripts/check-cuda-fat-gencode.py", + "scripts/check-release-workflow.py", + "scripts/validate-release-archive.py", "scripts/check-pr-size.py", "scripts/check-prompt-contract.py", + "scripts/check-triton-aot-multiarch.py", } self.assertEqual(set(checker.CREATION_MUTATIONS), expected) for path, mutation in checker.CREATION_MUTATIONS.items(): diff --git a/tests/scripts/test_check_public_doc_tables.py b/tests/scripts/test_check_public_doc_tables.py index 56738ffa2..0bea6e938 100644 --- a/tests/scripts/test_check_public_doc_tables.py +++ b/tests/scripts/test_check_public_doc_tables.py @@ -451,6 +451,10 @@ def test_the_live_page_keeps_the_character_ratchet_tight(self) -> None: self.assertGreaterEqual(slack, 0) self.assertLessEqual(slack, 25) + def test_the_rebased_character_ratchet_is_byte_tight(self) -> None: + text = doc_tables.STATUS.read_text(encoding="utf-8") + self.assertEqual(doc_tables.STATUS_RATCHET["chars"], len(text)) + def test_the_ratchet_carries_no_hidden_headroom(self) -> None: # A ratchet parked well above the page it guards is not a ratchet: it # silently licenses regrowth up to the old number. The rule is "lower it diff --git a/tests/scripts/test_check_release_binary_contract.py b/tests/scripts/test_check_release_binary_contract.py index 57ebaee17..b40cbf3b7 100644 --- a/tests/scripts/test_check_release_binary_contract.py +++ b/tests/scripts/test_check_release_binary_contract.py @@ -29,7 +29,7 @@ ".agents/roadmap_v1.md", ".agents/NOW.md", ".agents/coordination.md", - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md", + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md", "docs/STATUS.md", "docs/BENCHMARKS.md", ".github/workflows/ci.yml", @@ -127,78 +127,70 @@ RECORD_ANCHORS = { ".agents/engine-matrix.md": "| `ENG-RELEASE-BINARIES` |", ".agents/roadmap_v1.md": "| REL | `ROAD-V1-RELEASE` |", - ".agents/NOW.md": ( - "| Release | **ACTIVE; W5 19/19+10/10; contract 30/30** |" - ), + ".agents/NOW.md": "| Release | **ACTIVE; required W1-W11/W13 implemented in #196** |", ".agents/coordination.md": ( - "| `CLAIM-ENG-RELEASE-BINARIES-W5` | " - "`ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) |" + "**Server binary release W1-W13 (`ENG-RELEASE-BINARIES`, 2026-08-09," ), - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md": ( - "## 2026-08-08 — Release manifest W5 implemented; release row is ACTIVE" + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md": ( + "# W6 installed server package green" ), - "docs/STATUS.md": "#141 W5 19/19 ACTIVE ART∅", + "docs/STATUS.md": "#196 binary pipeline implemented; no published binaries", "docs/BENCHMARKS.md": ( - "| **Binary release matrix (ACTIVE; W5 implemented)** | " - "`ENG-RELEASE-BINARIES`:" + "| **Binary release matrix (ACTIVE; required W1-W11/W13 implemented in #196)** |" ), } LIFECYCLE_RECORD_MUTATIONS = ( ( ".agents/engine-matrix.md", - "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", - "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W5` |", + "`ACTIVE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |", + "`DONE` | `CLAIM-ENG-RELEASE-BINARIES-W1-W13` |", "engine-matrix release lifecycle", ), ( ".agents/engine-matrix.md", - "install/archive/publish implementation and all real release evidence " - "remain pending", - "install/archive/publish implementation and all real release evidence " - "are complete", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware " + "gates, and tagged publication remain pending", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware " + "gates, and tagged publication are complete", "engine-matrix release lifecycle", ), ( ".agents/roadmap_v1.md", - "`ACTIVE` | W5 versioned manifest schema/generator", - "`DONE` | W5 versioned manifest schema/generator", + "`ACTIVE` | Required W1-W11/W13 implementation is complete", + "`DONE` | Required W1-W11/W13 implementation is complete", "roadmap release lifecycle", ), ( ".agents/roadmap_v1.md", - "no archive exists", - "archive exists", + "no published binary exists", + "published binaries exist", "roadmap release lifecycle", ), ( ".agents/coordination.md", - "| `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals " - "10/10 killed; accepted release suite 30/30; no archive or real " - "runtime/correctness/performance evidence |", - "| `DONE` | 2026-08-08 — W5 19/19; fresh-review production removals " - "10/10 killed; accepted release suite 30/30; " - "archive and runtime/correctness/performance evidence complete |", + "| `ACTIVE` | 2026-08-09 — required W1-W11/W13 implementation complete;", + "| `DONE` | 2026-08-09 — required W1-W11/W13 implementation complete;", "coordination release lifecycle", ), ( ".agents/coordination.md", - "Excludes W1-W4, W6-W13, archives, install/package/publish workflows and " - "runtime artifacts", - "Includes W1-W13, archives, install/package/publish workflows and " - "runtime artifacts", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware " + "gates, rebase/merge, and tagged publication pending", + "hosted ten-SM completion, full eight-tuple dry run, matching-hardware " + "gates, rebase/merge, and tagged publication complete", "coordination release lifecycle", ), ( ".agents/coordination.md", - "no archive or real runtime/correctness/performance evidence", - "release archive and runtime/correctness/performance evidence shipped", + "W12 optional/non-primary |", + "W12 required/primary |", "coordination release lifecycle", ), ( - ".agents/completed/state-events/2026-08/STATE-20260808T220000-002.md", - "`ACTIVE`, never `DONE` or `GATING`.", - "`DONE`, with every release gate complete.", + ".agents/completed/state-events/2026-08/STATE-20260809T160000-001.md", + "The row remains `ACTIVE`. W1-W4 and W7-W13 remain pending", + "The row is `DONE`. Every release gate is complete", "state release lifecycle", ), ) @@ -238,16 +230,15 @@ PUBLIC_PENDING_MUTATIONS = ( ( "docs/BENCHMARKS.md", - "**PENDING:** W1-W4/W6-W13 and all archive/staged-smoke/runtime/" - "correctness/performance gates", + "**PENDING:** hosted full matrix, matching hardware, tagged publish", "**SHIPPED:** archive, runtime, correctness, and performance evidence " "complete", "docs/BENCHMARKS.md release row", ), ( "docs/STATUS.md", - "Subset; #141 W5 19/19 ACTIVE ART∅", - "Supported; #141: RELEASE DONE/ARTIFACTS✓", + "Subset; #196 binary pipeline implemented; no published binaries", + "Supported; #196: RELEASE DONE/ARTIFACTS✓", "docs/STATUS.md release row", ), ) @@ -287,7 +278,10 @@ EXACT_MACHINE_FIELDS = { "lifecycle": "ACTIVE", "manifest_schema": "vllm.cpp.release-manifest.v1", + "delivery_pull_request": "196", + "delivery_mode": "single-pr-W1-W13", "work_W5_status": "implemented", + "work_W6_status": "implemented", "work_W12_policy": "optional-non-blocking", "archive_claims": "pending", "runtime_claims": "pending", @@ -301,9 +295,11 @@ "required_anchor_paths": ( ".agents/engine-matrix.md,.agents/roadmap_v1.md,.agents/NOW.md," ".agents/coordination.md,.agents/completed/state-events/2026-08/" - "STATE-20260808T220000-002.md,docs/STATUS.md," + "STATE-20260809T160000-001.md,docs/STATUS.md," "docs/BENCHMARKS.md,docs/FEATURES.md,release/manifest-v1.schema.json," - "scripts/release_manifest.py,tests/scripts/test_release_manifest.py" + "scripts/release_manifest.py,tests/scripts/test_release_manifest.py," + "examples/CMakeLists.txt,scripts/package-server.py," + "tests/scripts/test_server_package.py" ), } diff --git a/tests/scripts/test_check_triton_aot_multiarch.py b/tests/scripts/test_check_triton_aot_multiarch.py new file mode 100644 index 000000000..a2c52ddc9 --- /dev/null +++ b/tests/scripts/test_check_triton_aot_multiarch.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Mutation tests for W2 multi-SM Triton AOT artifact auditing.""" + +from __future__ import annotations + +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-triton-aot-multiarch.py" +FLAGS = { + "sm_80": 0x00500550, + "sm_86": 0x00560556, + "sm_89": 0x00590559, + "sm_90a": 0x005A0D5A, + "sm_100a": 0x0600640A, + "sm_121a": 0x0600790A, +} + + +def cubin_source(flags: int) -> str: + data = bytearray(64) + data[:7] = b"\x7fELF\x02\x01\x01" + struct.pack_into(" tuple[Path, Path]: + vendored = root / "vendored" + nm = root / "nm.txt" + symbols = [] + for arch, flags in FLAGS.items(): + tree = vendored / arch + tree.mkdir(parents=True) + (tree / "MANIFEST").write_text( + f"arch {arch}\nbase toy py=toy.py kernel=toy\n", + encoding="utf-8", + ) + (tree / "toy.hash.c").write_text(cubin_source(flags), encoding="utf-8") + (tree / "toy.c").write_text("int toy_default(void);\n", encoding="utf-8") + symbols.append(f"00000000 T vt_aot_{arch}_toy_default") + symbols.append(f"00000000 T vt_aot_{arch}_load_toy") + nm.write_text("\n".join(symbols) + "\n", encoding="utf-8") + return vendored, nm + + def run_checker(self, mutate=None) -> subprocess.CompletedProcess[str]: + with tempfile.TemporaryDirectory() as temporary: + vendored, nm = self.make_fixture(Path(temporary)) + if mutate is not None: + mutate(vendored, nm) + return subprocess.run( + [ + sys.executable, + str(CHECKER), + "--vendored-root", + str(vendored), + "--nm-list", + str(nm), + ], + text=True, + capture_output=True, + check=False, + ) + + def test_exact_six_trees_and_namespaced_symbols_pass(self) -> None: + result = self.run_checker() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_wrong_tree_cubin_fails(self) -> None: + def mutate(vendored: Path, _nm: Path) -> None: + (vendored / "sm_80" / "toy.hash.c").write_text( + cubin_source(FLAGS["sm_86"]), encoding="utf-8" + ) + + result = self.run_checker(mutate) + self.assertNotEqual(result.returncode, 0) + self.assertIn("wrong cubin", result.stdout + result.stderr) + + def test_missing_tree_fails(self) -> None: + def mutate(vendored: Path, _nm: Path) -> None: + for path in (vendored / "sm_89").iterdir(): + path.unlink() + (vendored / "sm_89").rmdir() + + result = self.run_checker(mutate) + self.assertNotEqual(result.returncode, 0) + self.assertIn("tree set", result.stdout + result.stderr) + + def test_unnamespaced_or_missing_symbol_fails(self) -> None: + def mutate(_vendored: Path, nm: Path) -> None: + text = nm.read_text(encoding="utf-8") + text = text.replace("vt_aot_sm_80_toy_default", "toy_default") + nm.write_text(text, encoding="utf-8") + + result = self.run_checker(mutate) + self.assertNotEqual(result.returncode, 0) + self.assertIn("namespace", result.stdout + result.stderr) + + def test_missing_namespaced_loader_fails(self) -> None: + def mutate(_vendored: Path, nm: Path) -> None: + text = nm.read_text(encoding="utf-8") + text = text.replace("00000000 T vt_aot_sm_121a_load_toy\n", "") + nm.write_text(text, encoding="utf-8") + + result = self.run_checker(mutate) + self.assertNotEqual(result.returncode, 0) + self.assertIn("namespace is missing", result.stdout + result.stderr) + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_cpu_release_gates.py b/tests/scripts/test_cpu_release_gates.py new file mode 100755 index 000000000..d5a28760d --- /dev/null +++ b/tests/scripts/test_cpu_release_gates.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Executable contract for W9 CPU release-tier evidence.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts/run-cpu-release-gates.py" + + +FAKE_TEST = """#!/bin/sh +if [ "${FAKE_POOR:-0}" = 1 ]; then + case "${VT_CPU_MATMUL_TIER:-}${VT_CPU_Q8_DOT:-}${VT_CPU_QUANT_MMLA:-}${VT_CPU_QUANT_REPACK:-}" in + *f16c*|*avx2*|*avx512*|*sdot*|*i8mm*) exit 17 ;; + esac +fi +echo "selected ${VT_CPU_MATMUL_TIER:-${VT_CPU_Q8_DOT:-${VT_CPU_QUANT_MMLA:-${VT_CPU_QUANT_REPACK:-portable}}}}" +""" + + +FAKE_EMULATOR = """#!/bin/sh +export FAKE_POOR=1 +test "$1" = -cpu +shift 2 +exec "$@" +""" + + +class CpuReleaseGatesContract(unittest.TestCase): + def fixture(self, scratch: Path) -> tuple[Path, Path]: + tests = scratch / "tests" + tests.mkdir(parents=True) + for name in ( + "test_cpu_isa_arm", + "test_ops_matmul_elem", + "test_ops_quant_dot", + "test_ops_quant_repack", + ): + path = tests / name + path.write_text(FAKE_TEST, encoding="utf-8") + path.chmod(0o755) + emulator = scratch / "fake-emulator" + emulator.write_text(FAKE_EMULATOR, encoding="utf-8") + emulator.chmod(0o755) + return tests, emulator + + def invoke(self, tests: Path, emulator: Path, arch: str, output: Path): + return subprocess.run( + [ + sys.executable, + str(TOOL), + "--arch", + arch, + "--tests-dir", + str(tests), + "--poor-emulator", + str(emulator), + "--output", + str(output), + "--evidence-url", + "https://github.com/mudler/vllm.cpp/actions/runs/1", + ], + text=True, + capture_output=True, + env={**os.environ, "VLLM_CPP_RELEASE_GATE_TEST": "1"}, + check=False, + ) + + def test_x86_executes_all_tiers_and_poor_host_refusal(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + tests, emulator = self.fixture(Path(temporary)) + output = Path(temporary) / "report.json" + result = self.invoke(tests, emulator, "x86_64", output) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + report = json.loads(output.read_text()) + self.assertEqual( + list(report["tiers"]), + ["portable-sse2", "sse2-f16c", "avx2-f16c", "avx512f"], + ) + self.assertEqual({row["state"] for row in report["tiers"].values()}, {"passed"}) + self.assertTrue(any("Nehalem" in command for command in report["commands"])) + self.assertTrue(any("expect-refusal" in command for command in report["commands"])) + + def test_arm_executes_dotprod_i8mm_and_poor_host_refusal(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + tests, emulator = self.fixture(Path(temporary)) + output = Path(temporary) / "report.json" + result = self.invoke(tests, emulator, "aarch64", output) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + report = json.loads(output.read_text()) + self.assertEqual(list(report["tiers"]), ["portable-neon", "dotprod", "i8mm"]) + self.assertTrue(any("cortex-a53" in command for command in report["commands"])) + self.assertIn("VT_CPU_Q8_DOT=sdot", report["tiers"]["dotprod"]["command"]) + self.assertIn("VT_CPU_QUANT_MMLA=i8mm", report["tiers"]["i8mm"]["command"]) + + def test_missing_binary_or_accepted_unsupported_tier_is_fatal(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + tests, emulator = self.fixture(Path(temporary)) + (tests / "test_ops_matmul_elem").unlink() + result = self.invoke(tests, emulator, "x86_64", Path(temporary) / "report.json") + self.assertNotEqual(result.returncode, 0) + self.assertIn("missing test executable", result.stderr) + with tempfile.TemporaryDirectory() as temporary: + tests, emulator = self.fixture(Path(temporary)) + emulator.write_text("#!/bin/sh\nshift 2\nexec \"$@\"\n", encoding="utf-8") + emulator.chmod(0o755) + result = self.invoke(tests, emulator, "aarch64", Path(temporary) / "report.json") + self.assertNotEqual(result.returncode, 0) + self.assertIn("unexpectedly accepted", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_accelerator_metadata.py b/tests/scripts/test_release_accelerator_metadata.py new file mode 100755 index 000000000..c8819478e --- /dev/null +++ b/tests/scripts/test_release_accelerator_metadata.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""W10/W11 Linux accelerator release metadata contract.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import shutil +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts/release_accelerator_metadata.py" +SHA = "0123456789abcdef0123456789abcdef01234567" +SMS = ["80", "86", "87", "89", "90a", "100a", "103a", "110", "120a", "121a"] + + +def load(): + spec = importlib.util.spec_from_file_location("release_accelerator_metadata", TOOL) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {TOOL}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class AcceleratorMetadataContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.tool = load() + + def fixture(self, scratch: Path, artifact_id: str): + build = scratch / "build" + stage = scratch / "stage" + output = scratch / "metadata" + build.mkdir() + (stage / "bin").mkdir(parents=True) + shutil.copy2("/bin/true", stage / "bin/vllm-server") + cuda = "cuda" in artifact_id + vulkan = "vulkan" in artifact_id + cache = { + "MLX_ROOT": ("PATH", ""), + "VLLM_CPP_BUILD_EXAMPLES": ("BOOL", "ON"), + "VLLM_CPP_BUILD_TESTS": ("BOOL", "ON"), + "VLLM_CPP_CUDA": ("BOOL", "ON" if cuda else "OFF"), + "VLLM_CPP_CUDA_ARCHITECTURES": ("STRING", ";".join(SMS) if cuda else ""), + "VLLM_CPP_HIP": ("BOOL", "OFF"), + "VLLM_CPP_HIP_ARCHITECTURES": ("STRING", ""), + "VLLM_CPP_LITERAL_STATIC": ("BOOL", "OFF"), + "VLLM_CPP_METAL": ("BOOL", "OFF"), + "VLLM_CPP_MLX": ("BOOL", "OFF"), + "VLLM_CPP_SERVER": ("BOOL", "ON"), + "VLLM_CPP_TRITON": ("BOOL", "ON" if cuda else "OFF"), + "VLLM_CPP_VULKAN": ("BOOL", "ON" if vulkan else "OFF"), + } + (build / "CMakeCache.txt").write_text( + "".join(f"{key}:{kind}={value}\n" for key, (kind, value) in cache.items()), + encoding="utf-8", + ) + return argparse.Namespace( + abi_version="2.39", + artifact_id=artifact_id, + build_dir=build, + c_abi_version=17, + channel="preview", + compiler="GNU C++ 13.2.0", + evidence_url="https://github.com/mudler/vllm.cpp/actions/runs/1", + output_dir=output, + repo_root=ROOT, + source_clean=True, + source_commit=SHA, + stage_dir=stage, + toolchain="cmake-3.30+ninja-1.12", + version="0.0.1", + ) + + def test_cuda_manifest_carries_all_sms_aot_and_external_driver(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args = self.fixture(Path(temporary), "linux-x86_64-glibc-cuda-fat") + manifest = self.tool.prepare_accelerator_metadata(args) + self.assertEqual(manifest["cuda"]["compiled_sms"], SMS) + self.assertEqual( + [row["aot_available"] for row in manifest["cuda"]["sm_evidence"]], + [True, True, False, True, True, True, False, False, False, True], + ) + driver = next(row for row in manifest["dependencies"] if row["name"] == "nvidia-driver") + self.assertEqual((driver["linkage"], driver["bundled"]), ("external", False)) + + def test_vulkan_manifest_has_all_three_external_runtime_boundaries(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args = self.fixture(Path(temporary), "linux-x86_64-glibc-vulkan") + manifest = self.tool.prepare_accelerator_metadata(args) + names = {row["name"] for row in manifest["dependencies"]} + self.assertTrue({"vulkan-loader", "vulkan-icd", "vulkan-driver"} <= names) + self.assertNotIn("cuda", manifest) + + def test_cuda_refuses_partial_sm_or_disabled_triton_cache(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args = self.fixture(Path(temporary), "linux-x86_64-glibc-cuda-fat") + cache = (args.build_dir / "CMakeCache.txt").read_text() + for before, after in ( + ("80;86;87;89;90a;100a;103a;110;120a;121a", "80;86"), + ("VLLM_CPP_TRITON:BOOL=ON", "VLLM_CPP_TRITON:BOOL=OFF"), + ): + (args.build_dir / "CMakeCache.txt").write_text(cache.replace(before, after), encoding="utf-8") + with self.assertRaises(ValueError): + self.tool.prepare_accelerator_metadata(args) + (args.build_dir / "CMakeCache.txt").write_text(cache, encoding="utf-8") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_archive.py b/tests/scripts/test_release_archive.py new file mode 100755 index 000000000..6160b864d --- /dev/null +++ b/tests/scripts/test_release_archive.py @@ -0,0 +1,335 @@ +#!/usr/bin/env python3 +"""Executable W7 contract for extracted release archives and sidecars.""" + +from __future__ import annotations + +import hashlib +import importlib.util +import json +import platform +import shutil +import subprocess +import sys +import tarfile +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts" / "validate-release-archive.py" +FIXTURE = ROOT / "tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json" +CUDA_FIXTURE = ROOT / "tests/scripts/fixtures/release_manifest/v1/cuda-manifest.json" + + +def load_tool(): + spec = importlib.util.spec_from_file_location("validate_release_archive", TOOL) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {TOOL}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@unittest.skipUnless(platform.system() == "Linux" and platform.machine() == "x86_64", "Linux x86_64 W7 fixture") +class ReleaseArchiveContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.tool = load_tool() + + def manifest(self, executable: Path) -> dict[str, object]: + manifest = json.loads(FIXTURE.read_text(encoding="utf-8")) + dynamic = subprocess.run( + ["readelf", "-dW", str(executable)], + check=True, + text=True, + capture_output=True, + ).stdout + needed = self.tool.parse_elf_needed(dynamic) + manifest["dependencies"] = [ + { + "name": name, + "version": "system", + "kind": "library", + "linkage": "dynamic", + "bundled": False, + "role": "runtime", + } + for name in needed + ] + return manifest + + def make_release(self, scratch: Path, mutate=None) -> tuple[Path, Path, Path]: + stage = scratch / "stage" + (stage / "bin").mkdir(parents=True) + executable = stage / "bin/vllm-server" + shutil.copy2("/bin/true", executable) + executable.chmod(0o755) + manifest = self.manifest(executable) + version = { + "version": manifest["artifact"]["version"], + "commit": manifest["build"]["source_commit"], + "artifact_id": manifest["artifact"]["id"], + "backend": manifest["backend"]["name"], + "host_os": manifest["host"]["os"], + "host_arch": manifest["host"]["arch"], + "host_abi": manifest["host"]["abi"], + "source_clean": "true", + "c_abi_version": "17", + } + binary_digest = hashlib.sha256(executable.read_bytes()).hexdigest() + sbom = { + "spdxVersion": "SPDX-2.3", + "dataLicense": "CC0-1.0", + "SPDXID": "SPDXRef-DOCUMENT", + "name": manifest["artifact"]["id"], + "documentNamespace": "https://github.com/mudler/vllm.cpp/spdx/fixture", + "creationInfo": { + "created": "1970-01-01T00:00:00Z", + "creators": ["Organization: vllm.cpp"], + }, + "files": [ + { + "fileName": "./bin/vllm-server", + "SPDXID": "SPDXRef-File-vllm-server", + "checksums": [{"algorithm": "SHA256", "checksumValue": binary_digest}], + "licenseConcluded": "NOASSERTION", + "copyrightText": "NOASSERTION", + } + ], + } + (stage / "release-manifest.json").write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + (stage / "VERSION").write_text( + "".join(f"{key}={value}\n" for key, value in version.items()), encoding="utf-8" + ) + (stage / "sbom.spdx.json").write_text( + json.dumps(sbom, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + (stage / "THIRD_PARTY_NOTICES").write_text( + "vllm.cpp release dependency notices\n", encoding="utf-8" + ) + licenses = stage / "share/licenses/vllm.cpp" + licenses.mkdir(parents=True) + shutil.copy2(ROOT / "LICENSE", licenses / "LICENSE") + if mutate is not None: + mutate(stage, manifest, sbom) + archive = scratch / "vllm.cpp-fixture.tar.gz" + with tarfile.open(archive, "w:gz") as bundle: + for path in sorted(stage.rglob("*")): + bundle.add(path, arcname=path.relative_to(stage)) + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + checksum = Path(f"{archive}.sha256") + checksum.write_text(f"{digest} {archive.name}\n", encoding="utf-8") + provenance = Path(f"{archive}.provenance.json") + provenance.write_text( + json.dumps( + { + "_type": "https://in-toto.io/Statement/v1", + "predicateType": "https://slsa.dev/provenance/v1", + "subject": [{"name": archive.name, "digest": {"sha256": digest}}], + "predicate": { + "buildDefinition": { + "externalParameters": { + "artifact_id": manifest["artifact"]["id"], + "source_commit": manifest["build"]["source_commit"], + } + } + }, + }, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + return archive, checksum, provenance + + def run_validator(self, archive: Path, checksum: Path, provenance: Path): + return subprocess.run( + [ + sys.executable, + str(TOOL), + "--archive", + str(archive), + "--checksum", + str(checksum), + "--provenance", + str(provenance), + "--repo-root", + str(ROOT), + "--skip-version-smoke", + "--forbid-path", + str(archive.parent / "build-secret"), + ], + text=True, + capture_output=True, + check=False, + ) + + def test_valid_extracted_cpu_archive_passes(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + paths = self.make_release(Path(temporary)) + result = self.run_validator(*paths) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_extra_source_object_and_missing_metadata_fail(self) -> None: + mutations = ( + lambda stage, manifest, sbom: (stage / "source.cpp").write_text("int x;"), + lambda stage, manifest, sbom: (stage / "object.o").write_bytes(b"ELF"), + lambda stage, manifest, sbom: (stage / "VERSION").unlink(), + ) + for mutation in mutations: + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as temporary: + paths = self.make_release(Path(temporary), mutation) + result = self.run_validator(*paths) + self.assertNotEqual(result.returncode, 0) + + def test_wrong_checksum_and_provenance_subject_fail(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + archive, checksum, provenance = self.make_release(Path(temporary)) + checksum.write_text(f"{'0' * 64} {archive.name}\n", encoding="utf-8") + self.assertNotEqual(self.run_validator(archive, checksum, provenance).returncode, 0) + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + checksum.write_text(f"{digest} {archive.name}\n", encoding="utf-8") + statement = json.loads(provenance.read_text()) + statement["subject"][0]["digest"]["sha256"] = "f" * 64 + provenance.write_text(json.dumps(statement), encoding="utf-8") + self.assertNotEqual(self.run_validator(archive, checksum, provenance).returncode, 0) + + def test_sbom_binary_digest_and_version_manifest_agreement_are_live(self) -> None: + def bad_sbom(stage, manifest, sbom): + sbom["files"][0]["checksums"][0]["checksumValue"] = "0" * 64 + (stage / "sbom.spdx.json").write_text(json.dumps(sbom), encoding="utf-8") + + def bad_version(stage, manifest, sbom): + path = stage / "VERSION" + path.write_text(path.read_text().replace("version=0.1.0-test", "version=9.9.9")) + + for mutation in (bad_sbom, bad_version): + with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as temporary: + paths = self.make_release(Path(temporary), mutation) + result = self.run_validator(*paths) + self.assertNotEqual(result.returncode, 0) + + def test_build_path_and_undeclared_dynamic_dependency_fail(self) -> None: + manifest = json.loads(FIXTURE.read_text(encoding="utf-8")) + errors = self.tool.validate_linux_dynamic( + manifest, + ["libc.so.6", "libundeclared.so.1"], + ["/tmp/build-secret/lib"], + "/lib64/ld-linux-x86-64.so.2", + ["/tmp/build-secret"], + ) + self.assertTrue(any("undeclared" in error for error in errors), errors) + self.assertTrue(any("RPATH" in error for error in errors), errors) + self.assertEqual( + self.tool.validate_macho_install_id("@rpath/libmlx.dylib", ["/tmp/build"]), + [], + ) + self.assertTrue( + self.tool.validate_macho_install_id("/tmp/build/libmlx.dylib", ["/tmp/build"]) + ) + + def test_literal_static_policy_rejects_any_dynamic_boundary(self) -> None: + manifest = json.loads(FIXTURE.read_text(encoding="utf-8")) + manifest["artifact"]["static_boundary"] = "literal-static" + manifest["host"]["abi"] = "musl" + manifest["dependencies"] = [] + self.assertEqual(self.tool.validate_linux_dynamic(manifest, [], [], "", []), []) + errors = self.tool.validate_linux_dynamic( + manifest, ["libc.so"], ["$ORIGIN/lib"], "/lib/ld-musl-x86_64.so.1", [] + ) + self.assertTrue(any("literal-static" in error for error in errors), errors) + + def test_large_file_scanner_catches_a_credential_across_chunk_boundary(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "large-binary" + path.write_bytes(b"x" * (1024 * 1024 - 8) + b"AKIA0123456789ABCDEF") + found, aws_key = self.tool.scan_file(path, [b"not-present"]) + self.assertEqual(found, set()) + self.assertTrue(aws_key) + + def test_cuda_inventory_requires_all_sms_and_exact_aot_symbols(self) -> None: + manifest = json.loads(CUDA_FIXTURE.read_text(encoding="utf-8")) + images = [f"sm_{sm}" for sm in self.tool.PRIMARY_CUDA_SMS] + symbols = [f"vt_aot_sm_{sm}_gdn_default" for sm in self.tool.AOT_SMS] + self.assertEqual(self.tool.validate_cuda_inventory(manifest, images, symbols), []) + for removed in self.tool.PRIMARY_CUDA_SMS: + with self.subTest(missing_sm=removed): + errors = self.tool.validate_cuda_inventory( + manifest, [image for image in images if image != f"sm_{removed}"], symbols + ) + self.assertTrue(any(removed in error for error in errors), errors) + errors = self.tool.validate_cuda_inventory( + manifest, + images, + [symbol.replace("sm_80", "sm_87") for symbol in symbols], + ) + self.assertTrue(any("AOT" in error for error in errors), errors) + + def test_macho_dependencies_install_names_and_rpaths_fail_closed(self) -> None: + manifest = json.loads(FIXTURE.read_text(encoding="utf-8")) + manifest["host"].update({"os": "macos", "arch": "aarch64", "abi": "macos"}) + manifest["backend"]["name"] = "mlx" + manifest["dependencies"] = [ + {"name": "libmlx.dylib", "linkage": "dynamic"}, + {"name": "libc++.1.dylib", "linkage": "dynamic"}, + {"name": "libSystem.B.dylib", "linkage": "dynamic"}, + {"name": "Metal.framework", "linkage": "external"}, + {"name": "Foundation.framework", "linkage": "external"}, + ] + dependencies = [ + "@rpath/libmlx.dylib", + "/usr/lib/libc++.1.dylib", + "/usr/lib/libSystem.B.dylib", + "/System/Library/Frameworks/Metal.framework/Versions/A/Metal", + "/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation", + ] + self.assertEqual( + self.tool.validate_macho_dynamic( + manifest, dependencies, ["@loader_path/../lib"], ["/tmp/build-secret"] + ), + [], + ) + errors = self.tool.validate_macho_dynamic( + manifest, + [*dependencies, "/tmp/build-secret/libbad.dylib"], + ["/tmp/build-secret"], + ["/tmp/build-secret"], + ) + self.assertTrue(any("install name" in error for error in errors), errors) + self.assertTrue(any("RPATH" in error for error in errors), errors) + + def test_tar_traversal_is_rejected_before_extraction(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + scratch = Path(temporary) + archive = scratch / "bad.tar.gz" + source = scratch / "payload" + source.write_text("escape", encoding="utf-8") + with tarfile.open(archive, "w:gz") as bundle: + bundle.add(source, arcname="../escape") + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + checksum = Path(f"{archive}.sha256") + checksum.write_text(f"{digest} {archive.name}\n", encoding="utf-8") + provenance = Path(f"{archive}.provenance.json") + provenance.write_text( + json.dumps( + { + "_type": "https://in-toto.io/Statement/v1", + "predicateType": "https://slsa.dev/provenance/v1", + "subject": [ + {"name": archive.name, "digest": {"sha256": digest}} + ], + } + ), + encoding="utf-8", + ) + result = self.run_validator(archive, checksum, provenance) + self.assertNotEqual(result.returncode, 0) + self.assertIn("unsafe archive path", result.stdout + result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_index.py b/tests/scripts/test_release_index.py new file mode 100755 index 000000000..2d62d2af0 --- /dev/null +++ b/tests/scripts/test_release_index.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""W13 generated release index contract.""" + +from __future__ import annotations + +import importlib.util +import json +import tarfile +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts/release_index.py" +FIXTURE = ROOT / "tests/scripts/fixtures/release_manifest/v1/cpu-manifest.json" + + +def load(): + spec = importlib.util.spec_from_file_location("release_index", TOOL) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {TOOL}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class ReleaseIndexContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.tool = load() + + def assets(self, scratch: Path) -> tuple[Path, dict]: + assets = scratch / "assets" + assets.mkdir() + manifest = json.loads(FIXTURE.read_text(encoding="utf-8")) + artifact_id = manifest["artifact"]["id"] + archive = assets / f"{artifact_id}.tar.gz" + manifest_path = scratch / "release-manifest.json" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + with tarfile.open(archive, "w:gz") as bundle: + bundle.add(manifest_path, arcname="release-manifest.json") + digest = self.tool.sha256(archive) + (assets / f"{archive.name}.sha256").write_text(f"{digest} {archive.name}\n") + (assets / f"{archive.name}.provenance.json").write_text("{}\n") + handoff = { + "files": [ + { + "artifact_id": artifact_id, + "name": path.name, + "sha256": self.tool.sha256(path), + "size": path.stat().st_size, + } + for path in sorted(assets.iterdir()) + ], + "release_tag": "v0.0.1", + "retention": { + "ci_artifacts_days": 7, + "github_release": "maintainer-deletion-only", + }, + "source_sha": manifest["build"]["source_commit"], + "verified": True, + } + return assets, handoff + + def test_index_is_derived_from_archive_manifests_and_byte_inventory(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + assets, handoff = self.assets(root) + json_out = root / "release-index.json" + markdown_out = root / "RELEASE_INDEX.md" + self.tool.generate_index(assets, handoff, json_out, markdown_out) + index = json.loads(json_out.read_text()) + self.assertEqual(index["schema"], "vllm.cpp.release-index.v1") + self.assertEqual(index["artifacts"][0]["id"], "linux-x86_64-glibc-cpu") + self.assertEqual(index["artifacts"][0]["cpu_tiers"], [ + "portable-sse2", "sse2-f16c", "avx2-f16c", "avx512f" + ]) + self.assertIn("linux-x86_64-glibc-cpu", markdown_out.read_text()) + + def test_manifest_identity_or_handoff_digest_drift_is_fatal(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + assets, handoff = self.assets(root) + handoff["files"][0]["sha256"] = "0" * 64 + with self.assertRaises(ValueError): + self.tool.generate_index(assets, handoff, root / "i.json", root / "i.md") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_macos_metadata.py b/tests/scripts/test_release_macos_metadata.py new file mode 100755 index 000000000..8225f9ffa --- /dev/null +++ b/tests/scripts/test_release_macos_metadata.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""W11 Metal and MLX release metadata contract.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import shutil +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[2] +TOOL = ROOT / "scripts/release_macos_metadata.py" +SHA = "0123456789abcdef0123456789abcdef01234567" + + +def load(): + spec = importlib.util.spec_from_file_location("release_macos_metadata", TOOL) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {TOOL}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class MacosMetadataContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.tool = load() + + def fixture(self, scratch: Path, mlx: bool): + build = scratch / "build" + stage = scratch / "stage" + output = scratch / "metadata" + build.mkdir() + (stage / "bin").mkdir(parents=True) + shutil.copy2("/bin/true", stage / "bin/vllm-server") + mlx_root = "/opt/mlx" if mlx else "" + if mlx: + (stage / "lib").mkdir() + (stage / "lib/libmlx.dylib").write_bytes(b"mlx") + (stage / "lib/mlx.metallib").write_bytes(b"metal") + license_dir = stage / "share/licenses/MLX" + license_dir.mkdir(parents=True) + (license_dir / "LICENSE").write_text("MIT\n", encoding="utf-8") + cache = { + "MLX_ROOT": ("PATH", mlx_root), + "VLLM_CPP_BUILD_EXAMPLES": ("BOOL", "ON"), + "VLLM_CPP_BUILD_TESTS": ("BOOL", "ON"), + "VLLM_CPP_CUDA": ("BOOL", "OFF"), + "VLLM_CPP_CUDA_ARCHITECTURES": ("STRING", ""), + "VLLM_CPP_HIP": ("BOOL", "OFF"), + "VLLM_CPP_HIP_ARCHITECTURES": ("STRING", ""), + "VLLM_CPP_LITERAL_STATIC": ("BOOL", "OFF"), + "VLLM_CPP_METAL": ("BOOL", "ON"), + "VLLM_CPP_MLX": ("BOOL", "ON" if mlx else "OFF"), + "VLLM_CPP_SERVER": ("BOOL", "ON"), + "VLLM_CPP_TRITON": ("BOOL", "OFF"), + "VLLM_CPP_VULKAN": ("BOOL", "OFF"), + } + (build / "CMakeCache.txt").write_text( + "".join(f"{key}:{kind}={value}\n" for key, (kind, value) in cache.items()), + encoding="utf-8", + ) + return argparse.Namespace( + abi_version="15.0", + artifact_id="macos-arm64-metal-mlx" if mlx else "macos-arm64-metal", + build_dir=build, + c_abi_version=17, + channel="preview" if mlx else "stable", + compiler="AppleClang 16", + evidence_url="https://github.com/mudler/vllm.cpp/actions/runs/1", + mlx_version="0.32.0" if mlx else "", + mlx_license=(stage / "share/licenses/MLX/LICENSE") if mlx else None, + output_dir=output, + repo_root=ROOT, + source_clean=True, + source_commit=SHA, + stage_dir=stage, + toolchain="cmake-3.30+ninja-1.12", + version="0.0.1", + ) + + def otool(self, mlx: bool): + names = [ + "/System/Library/Frameworks/Metal.framework/Versions/A/Metal", + "/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation", + "/usr/lib/libc++.1.dylib", + "/usr/lib/libSystem.B.dylib", + ] + if mlx: + names.insert(0, "@rpath/libmlx.dylib") + return names + + def test_native_metal_stable_has_runtime_correctness_and_frameworks(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args = self.fixture(Path(temporary), False) + with mock.patch.object(self.tool, "otool_dependencies", return_value=self.otool(False)): + manifest = self.tool.prepare_macos_metadata(args) + self.assertEqual(manifest["artifact"]["channel"], "stable") + self.assertEqual(manifest["evidence"]["runtime"]["state"], "passed") + self.assertEqual(manifest["evidence"]["correctness"]["state"], "passed") + names = {row["name"] for row in manifest["dependencies"]} + self.assertTrue({"Metal.framework", "Foundation.framework"} <= names) + + def test_mlx_preview_requires_bundled_versioned_runtime_and_license(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args = self.fixture(Path(temporary), True) + with mock.patch.object(self.tool, "otool_dependencies", return_value=self.otool(True)): + manifest = self.tool.prepare_macos_metadata(args) + dependencies = {row["name"]: row for row in manifest["dependencies"]} + for name in ("libmlx.dylib", "mlx.metallib"): + self.assertTrue(dependencies[name]["bundled"]) + self.assertEqual(dependencies[name]["version"], "0.32.0") + (args.stage_dir / "lib/mlx.metallib").unlink() + with mock.patch.object(self.tool, "otool_dependencies", return_value=self.otool(True)): + with self.assertRaises(ValueError): + self.tool.prepare_macos_metadata(args) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_manifest.py b/tests/scripts/test_release_manifest.py index 1e317b64c..332547ea2 100644 --- a/tests/scripts/test_release_manifest.py +++ b/tests/scripts/test_release_manifest.py @@ -69,6 +69,7 @@ def flags(backend: str, sms: list[str] | None = None) -> dict[str, object]: "VLLM_CPP_CUDA_ARCHITECTURES": list(sms or []), "VLLM_CPP_HIP": False, "VLLM_CPP_HIP_ARCHITECTURES": [], + "VLLM_CPP_LITERAL_STATIC": False, "VLLM_CPP_METAL": backend in {"metal", "mlx"}, "VLLM_CPP_MLX": backend == "mlx", "VLLM_CPP_SERVER": True, @@ -117,14 +118,23 @@ def cpu_facts() -> dict[str, object]: { "name": "avx2-f16c", "kernel_families": ["matmul-elem-f32-bf16-f16"], - "required_cpu_bits": ["avx", "avx2", "f16c", "osxsave"], + "required_cpu_bits": ["avx", "avx2", "f16c", "osxsave", "sse2"], "required_os_state": ["xcr0:xmm", "xcr0:ymm"], "execution_evidence": evidence("absent"), }, { "name": "avx512f", "kernel_families": ["matmul-elem-f32-bf16"], - "required_cpu_bits": ["avx", "avx512f", "osxsave"], + "required_cpu_bits": [ + "avx", + "avx2", + "avx512bw", + "avx512f", + "avx512vl", + "f16c", + "osxsave", + "sse2", + ], "required_os_state": [ "xcr0:xmm", "xcr0:ymm", @@ -139,6 +149,7 @@ def cpu_facts() -> dict[str, object]: "artifact": { "id": "linux-x86_64-glibc-cpu", "version": "0.1.0-test", + "c_abi_version": 17, "channel": "preview", "kind": "primary", "static_boundary": "static-core", @@ -200,6 +211,7 @@ def cuda_facts() -> dict[str, object]: "artifact": { "id": "linux-x86_64-glibc-cuda-fat", "version": "0.1.0-test", + "c_abi_version": 17, "channel": "preview", "kind": "primary", "static_boundary": "static-core", @@ -399,6 +411,18 @@ def test_schema_type_enum_and_const_semantics_are_independently_live(self) -> No manifest["schema_version"] = True self.assert_invalid(manifest, "schema_version") + def test_artifact_c_abi_version_is_mandatory_and_numeric(self) -> None: + manifest = self.generated(cpu_facts()) + self.assertEqual(manifest["artifact"]["c_abi_version"], 17) + for value in (None, True, 0, -1, "17"): + with self.subTest(value=value): + mutant = copy.deepcopy(manifest) + if value is None: + del mutant["artifact"]["c_abi_version"] + else: + mutant["artifact"]["c_abi_version"] = value + self.assert_invalid(mutant, "c_abi_version") + def test_boolean_is_neither_an_integer_type_nor_integer_json_constant(self) -> None: self.assertFalse( self.tool._type_matches(True, "integer"), @@ -521,6 +545,35 @@ def test_every_current_x86_tier_and_its_contract_are_mandatory(self) -> None: mutant["cpu"]["compiled_tiers"][1]["required_os_state"] = [] self.assert_invalid(mutant, "OS-state") + def test_literal_static_flag_is_exactly_scoped_to_the_musl_cpu_tuple(self) -> None: + glibc = self.generated(cpu_facts()) + glibc["backend"]["flags"]["VLLM_CPP_LITERAL_STATIC"] = True + glibc["build"]["resolved_cmake_options"]["VLLM_CPP_LITERAL_STATIC"] = True + self.assert_invalid(glibc, "must agree with the artifact static boundary") + + facts = cpu_facts() + facts["artifact"].update( + { + "id": "linux-x86_64-musl-cpu-static", + "channel": "experimental-preview", + "static_boundary": "literal-static", + } + ) + facts["host"].update({"abi": "musl", "abi_version": "1.2.5"}) + facts["backend"]["flags"]["VLLM_CPP_LITERAL_STATIC"] = True + facts["build"]["resolved_cmake_options"]["VLLM_CPP_LITERAL_STATIC"] = True + facts["dependencies"] = [ + { + "name": "musl-libc", + "version": "1.2.5", + "kind": "library", + "linkage": "static", + "bundled": True, + "role": "build-time", + } + ] + self.generated(facts) + def test_cpu_tier_inventory_is_exact_on_x86_and_aarch64(self) -> None: manifest = self.generated(cpu_facts()) for index, tier in enumerate(manifest["cpu"]["compiled_tiers"]): @@ -559,19 +612,22 @@ def test_cpu_tier_inventory_is_exact_on_x86_and_aarch64(self) -> None: arm_kernel_families = { "portable-neon": ["matmul-elem-f32-bf16-f16"], + "dotprod": ["quant-dot-q8_0-q8_0"], "i8mm": ["quant-dot-q4_0-q8_0-q4_K-q6_K", "quant-repack-q8_0"], } - for os_name, abi, artifact_id, probe in ( + for os_name, abi, artifact_id, dotprod_probe, i8mm_probe in ( ( "linux", "glibc", "linux-aarch64-glibc-cpu", + "getauxval:AT_HWCAP:HWCAP_ASIMDDP", "getauxval:AT_HWCAP2:HWCAP2_I8MM", ), ( "macos", "macos", "macos-arm64-cpu-policy-probe", + "sysctl:hw.optional.arm.FEAT_DotProd", "sysctl:hw.optional.arm.FEAT_I8MM", ), ): @@ -589,11 +645,17 @@ def test_cpu_tier_inventory_is_exact_on_x86_and_aarch64(self) -> None: "required_cpu_bits": ["neon"], "required_os_state": [], }, + { + "name": "dotprod", + "kernel_families": arm_kernel_families["dotprod"], + "required_cpu_bits": ["dotprod", "neon"], + "required_os_state": [dotprod_probe], + }, { "name": "i8mm", "kernel_families": arm_kernel_families["i8mm"], - "required_cpu_bits": ["i8mm"], - "required_os_state": [probe], + "required_cpu_bits": ["dotprod", "i8mm", "neon"], + "required_os_state": [dotprod_probe, i8mm_probe], }, ], }, @@ -601,7 +663,7 @@ def test_cpu_tier_inventory_is_exact_on_x86_and_aarch64(self) -> None: with self.subTest(aarch64_os=os_name): self.assertEqual(self.tool._cpu_policy(arm), []) wrong_probe = copy.deepcopy(arm) - wrong_probe["cpu"]["compiled_tiers"][1]["required_os_state"] = [ + wrong_probe["cpu"]["compiled_tiers"][2]["required_os_state"] = [ "fabricated:os-probe" ] self.assertTrue( @@ -766,6 +828,8 @@ def test_backend_dependency_and_static_boundaries_are_fail_closed(self) -> None: "static_boundary": "literal-static", }) musl["host"].update({"abi": "musl", "abi_version": "1.2.5"}) + musl["backend"]["flags"]["VLLM_CPP_LITERAL_STATIC"] = True + musl["build"]["resolved_cmake_options"]["VLLM_CPP_LITERAL_STATIC"] = True musl["dependencies"] = [{ "name": "musl", "version": "1.2.5", "kind": "library", "linkage": "static", "bundled": True, "role": "runtime", diff --git a/tests/scripts/test_release_metadata.py b/tests/scripts/test_release_metadata.py new file mode 100755 index 000000000..810861b88 --- /dev/null +++ b/tests/scripts/test_release_metadata.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""W9 CPU release metadata and packaged-sidecar contract.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +METADATA = ROOT / "scripts/release_metadata.py" +PACKAGE = ROOT / "scripts/package-server.py" +VALIDATOR = ROOT / "scripts/validate-release-archive.py" +SHA = "0123456789abcdef0123456789abcdef01234567" + + +def load(path: Path, name: str): + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def passed(command: str) -> dict[str, str]: + return { + "command": command, + "reason": "", + "result": "exit 0", + "state": "passed", + "url": "https://github.com/mudler/vllm.cpp/actions/runs/1", + } + + +class ReleaseMetadataContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.metadata = load(METADATA, "release_metadata") + cls.package = load(PACKAGE, "package_server") + + def fixture(self, scratch: Path): + build = scratch / "build" + stage = scratch / "stage" + metadata = scratch / "metadata" + build.mkdir(parents=True) + (stage / "bin").mkdir(parents=True) + shutil.copy2("/bin/true", stage / "bin/vllm-server") + (stage / "bin/vllm-server").chmod(0o755) + cache = { + "MLX_ROOT": ("PATH", ""), + "VLLM_CPP_BUILD_EXAMPLES": ("BOOL", "ON"), + "VLLM_CPP_BUILD_TESTS": ("BOOL", "ON"), + "VLLM_CPP_CUDA": ("BOOL", "OFF"), + "VLLM_CPP_CUDA_ARCHITECTURES": ("STRING", ""), + "VLLM_CPP_HIP": ("BOOL", "OFF"), + "VLLM_CPP_HIP_ARCHITECTURES": ("STRING", ""), + "VLLM_CPP_LITERAL_STATIC": ("BOOL", "OFF"), + "VLLM_CPP_METAL": ("STRING", "OFF"), + "VLLM_CPP_MLX": ("BOOL", "OFF"), + "VLLM_CPP_SERVER": ("BOOL", "ON"), + "VLLM_CPP_TRITON": ("STRING", "OFF"), + "VLLM_CPP_VULKAN": ("STRING", "OFF"), + } + (build / "CMakeCache.txt").write_text( + "".join(f"{key}:{kind}={value}\n" for key, (kind, value) in cache.items()), + encoding="utf-8", + ) + policy = self.metadata.release_manifest.CPU_TIER_POLICY["x86_64"] + report = { + "commands": [f"VT_CPU_MATMUL_TIER={name} test_ops_matmul_elem" for name in policy["tiers"]], + "schema": "vllm.cpp.cpu-tier-report.v1", + "selected_tier": "avx512f", + "tiers": {name: passed(f"force {name}") for name in policy["tiers"]}, + } + report_path = scratch / "tier-report.json" + report_path.write_text(json.dumps(report), encoding="utf-8") + args = argparse.Namespace( + abi_version="2.31", + artifact_id="linux-x86_64-glibc-cpu", + build_dir=build, + c_abi_version=17, + channel="stable", + compiler="GNU C++ 13.2.0", + evidence_url="https://github.com/mudler/vllm.cpp/actions/runs/1", + output_dir=metadata, + repo_root=ROOT, + source_clean=True, + source_commit=SHA, + stage_dir=stage, + tier_report=report_path, + toolchain="cmake-3.30+ninja-1.12", + version="0.0.1", + ) + return args, report + + def test_stable_cpu_metadata_is_schema_valid_and_byte_bound(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args, _ = self.fixture(Path(temporary)) + manifest = self.metadata.prepare_cpu_metadata(args) + self.assertEqual(manifest["artifact"]["c_abi_version"], 17) + self.assertEqual(manifest["artifact"]["channel"], "stable") + self.assertEqual( + [tier["execution_evidence"]["state"] for tier in manifest["cpu"]["compiled_tiers"]], + ["passed"] * 4, + ) + sbom = json.loads((args.output_dir / "sbom.spdx.json").read_text()) + self.assertEqual(sbom["spdxVersion"], "SPDX-2.3") + self.assertTrue((args.output_dir / "share/licenses/vllm.cpp/LICENSE").is_file()) + + def test_stable_metadata_refuses_a_missing_or_unexecuted_tier(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + args, report = self.fixture(Path(temporary)) + del report["tiers"]["avx512f"] + args.tier_report.write_text(json.dumps(report), encoding="utf-8") + with self.assertRaises(ValueError): + self.metadata.prepare_cpu_metadata(args) + args, report = self.fixture(Path(temporary) / "second") + report["tiers"]["avx512f"] = { + "command": "", + "reason": "runner lacks AVX-512", + "result": "", + "state": "absent", + "url": "", + } + args.tier_report.write_text(json.dumps(report), encoding="utf-8") + with self.assertRaises(ValueError): + self.metadata.prepare_cpu_metadata(args) + + def test_metadata_is_installed_archived_and_validated_with_final_sidecars(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + scratch = Path(temporary) + args, _ = self.fixture(scratch) + self.metadata.prepare_cpu_metadata(args) + packaged = scratch / "packaged" + shutil.copytree(args.stage_dir, packaged) + self.package.install_metadata(args.output_dir, packaged) + archive = scratch / "vllm.cpp-0.0.1-cpu-linux-x86_64.tar.gz" + self.package.write_archive(packaged, archive, 0) + self.package.write_archive_sidecars(archive, packaged) + result = subprocess.run( + [ + sys.executable, + str(VALIDATOR), + "--archive", + str(archive), + "--checksum", + f"{archive}.sha256", + "--provenance", + f"{archive}.provenance.json", + "--repo-root", + str(ROOT), + "--skip-version-smoke", + ], + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + first = Path(f"{archive}.sha256").read_text() + self.package.write_archive(packaged, archive, 0) + self.package.write_archive_sidecars(archive, packaged) + self.assertEqual(first, Path(f"{archive}.sha256").read_text()) + + def test_metadata_install_rejects_unallowlisted_payload(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + scratch = Path(temporary) + args, _ = self.fixture(scratch) + self.metadata.prepare_cpu_metadata(args) + (args.output_dir / "model.safetensors").write_bytes(b"forbidden") + with self.assertRaises(SystemExit): + self.package.install_metadata(args.output_dir, scratch / "stage-copy") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_release_pipeline.py b/tests/scripts/test_release_pipeline.py new file mode 100755 index 000000000..4d5bcd52c --- /dev/null +++ b/tests/scripts/test_release_pipeline.py @@ -0,0 +1,303 @@ +#!/usr/bin/env python3 +"""W8 dry-run/tag planning, immutable handoff, and permission mutations.""" + +from __future__ import annotations + +import copy +import importlib.util +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[2] +PIPELINE = ROOT / "scripts/release_pipeline.py" +CHECKER = ROOT / "scripts/check-release-workflow.py" +WORKFLOW = ROOT / ".github/workflows/release.yml" +MATRIX = ROOT / "release/release-matrix.json" +SHA = "0123456789abcdef0123456789abcdef01234567" + + +def load(path: Path, name: str): + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class ReleasePipelineContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.pipeline = load(PIPELINE, "release_pipeline") + cls.checker = load(CHECKER, "check_release_workflow") + + def plan(self, event: str, ref: str, release_ready: bool = False): + matrix = json.loads(MATRIX.read_text(encoding="utf-8")) + matrix["release_ready"] = release_ready + return self.pipeline.make_plan(event, ref, SHA, "0.0.1", matrix) + + def test_manual_dispatch_is_always_a_non_publishing_dry_run(self) -> None: + plan = self.plan("workflow_dispatch", "refs/heads/main", release_ready=True) + self.assertFalse(plan["publish"]) + self.assertEqual(plan["release_tag"], f"dry-run-{SHA[:12]}") + self.assertEqual(plan["source_sha"], SHA) + + def test_tag_publish_requires_exact_version_and_ready_matrix(self) -> None: + self.assertFalse(self.plan("push", "refs/tags/v0.0.1")["publish"]) + self.assertTrue(self.plan("push", "refs/tags/v0.0.1", release_ready=True)["publish"]) + for ref in ("refs/tags/v0.0.2", "refs/heads/v0.0.1", "refs/tags/0.0.1"): + with self.subTest(ref=ref): + with self.assertRaises(ValueError): + self.plan("push", ref, release_ready=True) + + def test_only_explicit_matrix_artifacts_enter_the_plan(self) -> None: + plan = self.plan("workflow_dispatch", "refs/heads/topic") + ids = [item["id"] for item in plan["artifacts"]] + self.assertEqual(len(ids), len(set(ids))) + self.assertIn("linux-x86_64-glibc-cpu", ids) + self.assertIn("linux-aarch64-glibc-cuda-fat", ids) + self.assertNotIn("rocm", " ".join(ids)) + self.assertEqual(plan["retention"]["ci_artifacts_days"], 7) + + def test_matrix_retention_policy_is_exact(self) -> None: + matrix = json.loads(MATRIX.read_text(encoding="utf-8")) + matrix["retention"]["ci_artifacts_days"] = 90 + with self.assertRaises(ValueError): + self.pipeline.validate_matrix(matrix) + + def test_publish_matrix_contains_all_eight_primary_bundles(self) -> None: + matrix = json.loads(MATRIX.read_text(encoding="utf-8")) + artifacts = self.pipeline.validate_matrix(matrix) + self.assertTrue(matrix["release_ready"]) + self.assertEqual( + {item["id"]: item["channel"] for item in artifacts}, + { + "linux-x86_64-glibc-cpu": "stable", + "linux-aarch64-glibc-cpu": "stable", + "linux-x86_64-musl-cpu-static": "experimental-preview", + "linux-x86_64-glibc-cuda-fat": "preview", + "linux-aarch64-glibc-cuda-fat": "preview", + "linux-x86_64-glibc-vulkan": "preview", + "macos-arm64-metal": "stable", + "macos-arm64-metal-mlx": "preview", + }, + ) + self.assertTrue(all(item["required"] is True for item in artifacts)) + + def test_handoff_digest_and_source_sha_are_immutable(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + plan_path = root / "release-plan.json" + handoff_path = root / "release-handoff.json" + verified_path = root / "verified-handoff.json" + assets = root / "assets" + assets.mkdir() + archive = assets / "linux-x86_64-glibc-cpu.tar.gz" + archive.write_bytes(b"release bytes") + digest = self.pipeline.file_sha256(archive) + (assets / f"{archive.name}.sha256").write_text( + f"{digest} {archive.name}\n", encoding="utf-8" + ) + (assets / f"{archive.name}.provenance.json").write_text( + json.dumps({"subject": [{"name": archive.name, "digest": {"sha256": digest}}]}), + encoding="utf-8", + ) + self.pipeline.write_json(plan_path, self.plan("workflow_dispatch", "refs/heads/main")) + self.pipeline.make_handoff(plan_path, assets, handoff_path) + self.pipeline.verify_handoff(plan_path, handoff_path, assets, verified_path, SHA) + verified = json.loads(verified_path.read_text()) + self.assertTrue(verified["verified"]) + self.assertEqual([item["name"] for item in verified["files"]], sorted(path.name for path in assets.iterdir())) + mutant = json.loads(handoff_path.read_text()) + mutant["source_sha"] = "f" * 40 + self.pipeline.write_json(handoff_path, mutant) + with self.assertRaises(ValueError): + self.pipeline.verify_handoff(plan_path, handoff_path, assets, verified_path, SHA) + self.pipeline.make_handoff(plan_path, assets, handoff_path) + archive.write_bytes(b"mutated") + with self.assertRaises(ValueError): + self.pipeline.verify_handoff(plan_path, handoff_path, assets, verified_path, SHA) + + def test_publish_ready_plan_requires_every_required_asset_triplet(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + plan = self.plan("push", "refs/tags/v0.0.1", release_ready=True) + plan_path = root / "plan.json" + self.pipeline.write_json(plan_path, plan) + assets = root / "assets" + assets.mkdir() + with self.assertRaises(ValueError): + self.pipeline.make_handoff(plan_path, assets, root / "handoff.json") + + def test_publish_enumerates_only_verified_assets_without_shell_globs(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + assets = root / "assets" + assets.mkdir() + archive = assets / "linux-x86_64-glibc-cpu.tar.gz" + archive.write_bytes(b"release bytes") + digest = self.pipeline.file_sha256(archive) + for suffix, content in ( + (".sha256", f"{digest} {archive.name}\n"), + (".provenance.json", "{}\n"), + ): + (assets / f"{archive.name}{suffix}").write_text(content) + handoff = { + "files": [ + {"name": path.name, "sha256": self.pipeline.file_sha256(path), "size": path.stat().st_size} + for path in sorted(assets.iterdir()) + ], + "publish": True, + "release_tag": "v0.0.1", + "source_sha": SHA, + "verified": True, + } + handoff_path = root / "verified-handoff.json" + index_json = root / "release-index.json" + index_md = root / "RELEASE_INDEX.md" + self.pipeline.write_json(handoff_path, handoff) + self.pipeline.write_json(index_json, { + "artifacts": [{ + "archive": archive.name, + "id": archive.name.removesuffix(".tar.gz"), + "sha256": digest, + }], + "release_tag": "v0.0.1", + "schema": "vllm.cpp.release-index.v1", + "source_sha": SHA, + }) + index_md.write_text( + f"# release v0.0.1\n\nSource: `{SHA}`\n\n{archive.name}\n" + ) + with mock.patch.object(self.pipeline.subprocess, "run") as run: + self.pipeline.publish_release( + handoff_path, assets, index_json, index_md, "v0.0.1" + ) + argv = run.call_args.args[0] + self.assertEqual(argv[:4], ["gh", "release", "create", "v0.0.1"]) + self.assertFalse(any("*" in value or "?" in value for value in argv)) + self.assertIn(str(archive), argv) + run.assert_called_once() + + index = json.loads(index_json.read_text()) + index["source_sha"] = "f" * 40 + self.pipeline.write_json(index_json, index) + with self.assertRaises(ValueError): + self.pipeline.publish_release( + handoff_path, assets, index_json, index_md, "v0.0.1" + ) + + def test_publish_rejects_unverified_drift_and_extra_assets(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + assets = root / "assets" + assets.mkdir() + archive = assets / "asset.tar.gz" + archive.write_bytes(b"bytes") + handoff = { + "files": [{ + "name": archive.name, + "sha256": self.pipeline.file_sha256(archive), + "size": archive.stat().st_size, + }], + "publish": True, + "release_tag": "v0.0.1", + "verified": True, + } + handoff_path = root / "verified-handoff.json" + index_json = root / "release-index.json" + index_md = root / "RELEASE_INDEX.md" + self.pipeline.write_json(handoff_path, handoff) + index_json.write_text("{}\n") + index_md.write_text("# release\n") + (assets / "unexpected").write_text("no") + with self.assertRaises(ValueError): + self.pipeline.publish_release( + handoff_path, assets, index_json, index_md, "v0.0.1" + ) + (assets / "unexpected").unlink() + archive.write_bytes(b"drift") + with self.assertRaises(ValueError): + self.pipeline.publish_release( + handoff_path, assets, index_json, index_md, "v0.0.1" + ) + + def test_cli_dry_run_never_calls_github_or_creates_a_release(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + output = Path(temporary) / "plan.json" + result = subprocess.run( + [ + sys.executable, + str(PIPELINE), + "plan", + "--event", + "workflow_dispatch", + "--ref", + "refs/heads/main", + "--sha", + SHA, + "--version", + "0.0.1", + "--matrix", + str(MATRIX), + "--output", + str(output), + ], + text=True, + capture_output=True, + check=False, + env={"PATH": "/nonexistent"}, + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertFalse(json.loads(output.read_text())["publish"]) + + def test_workflow_has_exact_least_privilege_stage_boundaries(self) -> None: + errors = self.checker.validate(WORKFLOW.read_text(encoding="utf-8")) + self.assertEqual(errors, []) + + def test_security_critical_workflow_mutations_fail(self) -> None: + original = WORKFLOW.read_text(encoding="utf-8") + mutations = { + "pull request trigger": (" workflow_dispatch: {}", " pull_request: {}\n workflow_dispatch: {}"), + "global write": ("permissions:\n contents: read", "permissions:\n contents: write"), + "mutable upload": ("overwrite: false", "overwrite: true"), + "primary tuple omitted from handoff": ( + "needs: [plan, cpu_x86, cpu_arm64, cpu_musl, cuda_x86, cuda_arm64, vulkan_x86, metal_arm64, mlx_arm64]", + "needs: [plan, cpu_x86, cpu_arm64, cpu_musl, cuda_x86, cuda_arm64, vulkan_x86, metal_arm64]", + ), + "name not SHA-bound": ("release-unverified-${{ github.sha }}", "release-unverified"), + "name download": ("artifact-ids: ${{ needs.build.outputs.artifact_id }}", "name: release-unverified"), + "attest no OIDC": (" id-token: write", " id-token: none"), + "attest no repository grant": (" attestations: write", " attestations: none"), + "attest no metadata grant": (" artifact-metadata: write", " artifact-metadata: none"), + "publish no environment": (" environment: release", " # environment removed"), + "publish broad dependency": (" needs: [plan, verify, attest]", " needs: [plan, build, verify, attest]"), + "publish no checkout": ( + " steps:\n - uses: actions/checkout@v4\n - name: Download only the verified handoff by ID", + " steps:\n - name: Download only the verified handoff by ID", + ), + "publish bypasses byte binding": ( + "python3 scripts/release_pipeline.py publish", + "python3 -c 'pass'", + ), + "publish not tag gated": ( + "startsWith(github.ref, 'refs/tags/v')", + "startsWith(github.ref, 'refs/heads/')", + ), + "continue on error": (" runs-on: ubuntu-latest", " continue-on-error: true\n runs-on: ubuntu-latest"), + } + for label, (before, after) in mutations.items(): + with self.subTest(label=label): + self.assertIn(before, original) + mutant = original.replace(before, after, 1) + self.assertTrue(self.checker.validate(mutant), label) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_server_package.py b/tests/scripts/test_server_package.py new file mode 100644 index 000000000..53940ef69 --- /dev/null +++ b/tests/scripts/test_server_package.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +"""End-to-end contract for the W6 installed server archive. + +This test catches a package target that copies a build-tree executable, keeps +the historical ``server`` filename, or leaves the extracted executable coupled +to a build-tree ``libvllm``. It configures the real CPU server, builds the real +archive target, extracts it into a second empty tree, and invokes that binary. +""" + +from __future__ import annotations + +import hashlib +import os +import platform +import shutil +import subprocess +import tarfile +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +BUILD_JOBS = os.environ.get("VLLM_CPP_PACKAGE_TEST_JOBS", "2") + + +def run(*command: str, env: dict[str, str] | None = None) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + command, + cwd=ROOT, + env=env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + check=False, + ) + if result.returncode != 0: + raise AssertionError( + f"command failed ({result.returncode}): {' '.join(command)}\n{result.stdout}" + ) + return result + + +class ServerPackageTest(unittest.TestCase): + def test_archive_is_deterministic_installed_and_runnable(self) -> None: + with tempfile.TemporaryDirectory(prefix="vllm-server-package-") as temporary: + scratch = Path(temporary) + configured_build = os.environ.get("VLLM_CPP_PACKAGE_TEST_BUILD_DIR") + if configured_build: + build = Path(configured_build).resolve() + self.assertTrue( + (build / "CMakeCache.txt").is_file(), + "VLLM_CPP_PACKAGE_TEST_BUILD_DIR must name a configured build", + ) + else: + build = scratch / "build" + run( + "cmake", + "-S", + str(ROOT), + "-B", + str(build), + "-DCMAKE_BUILD_TYPE=Release", + "-DVLLM_CPP_BUILD_TESTS=OFF", + "-DVLLM_CPP_BUILD_EXAMPLES=ON", + "-DVLLM_CPP_SERVER=ON", + "-DVLLM_CPP_CUDA=OFF", + "-DVLLM_CPP_METAL=OFF", + "-DVLLM_CPP_VULKAN=OFF", + "-DVLLM_CPP_HIP=OFF", + ) + run( + "cmake", + "--build", + str(build), + "--target", + "vllm-server-archive", + f"-j{BUILD_JOBS}", + ) + + archives = sorted((build / "release").glob("vllm.cpp-*-cpu-*.tar.gz")) + self.assertEqual(len(archives), 1, "the package target must emit one archive") + archive = archives[0] + first_digest = hashlib.sha256(archive.read_bytes()).hexdigest() + + # A second package invocation must reproduce the bytes, not merely a + # staging tree with equivalent contents. + archive.unlink() + run( + "cmake", + "--build", + str(build), + "--target", + "vllm-server-archive", + f"-j{BUILD_JOBS}", + ) + self.assertEqual(first_digest, hashlib.sha256(archive.read_bytes()).hexdigest()) + + extracted = scratch / "extracted" + extracted.mkdir() + with tarfile.open(archive, "r:gz") as bundle: + bundle.extractall(extracted, filter="data") + + executable = extracted / "bin" / ( + "vllm-server.exe" if platform.system() == "Windows" else "vllm-server" + ) + self.assertTrue(executable.is_file(), "archive must use the canonical server name") + self.assertTrue(os.access(executable, os.X_OK), "installed server must be executable") + self.assertTrue( + (build / "release" / "stage" / "bin" / executable.name).is_file(), + "the package target must retain its installed staging tree", + ) + + clean_env = os.environ.copy() + clean_env.pop("LD_LIBRARY_PATH", None) + clean_env.pop("DYLD_LIBRARY_PATH", None) + help_result = run(str(executable), "--help", env=clean_env) + self.assertIn("--max-num-seqs N", help_result.stdout) + self.assertIn("--max-num-batched-tokens N", help_result.stdout) + version_result = run(str(executable), "--version", env=clean_env) + self.assertRegex(version_result.stdout, r"^vllm\.cpp \d+\.\d+\.\d+(?:\+cuda)? c-abi=\d+\n$") + + if platform.system() == "Linux" and shutil.which("ldd"): + dependencies = run("ldd", str(executable), env=clean_env).stdout + self.assertNotIn("libvllm", dependencies) + + # W6 adds a server component without replacing the existing library + # installation contract. + install_prefix = scratch / "library-install" + run( + "cmake", + "--build", + str(build), + "--target", + "vllm_shared", + f"-j{BUILD_JOBS}", + ) + run("cmake", "--install", str(build), "--prefix", str(install_prefix)) + self.assertTrue((install_prefix / "include" / "vllm.h").is_file()) + self.assertTrue(any((install_prefix / "lib").glob("libvllm.a"))) + shared_libraries = [ + *install_prefix.glob("lib/libvllm.so*"), + *install_prefix.glob("lib/libvllm*.dylib"), + *install_prefix.glob("bin/vllm*.dll"), + ] + self.assertTrue(shared_libraries, "the existing shared-library install must remain") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/vt/test_cpu_isa_arm.cpp b/tests/vt/test_cpu_isa_arm.cpp new file mode 100644 index 000000000..4645f62a9 --- /dev/null +++ b/tests/vt/test_cpu_isa_arm.cpp @@ -0,0 +1,97 @@ +#include + +#include "doctest/doctest.h" +#include "vt/cpu/cpu_isa_arm.h" + +TEST_CASE("Arm ISA selection requires every CPU capability in a tier") { + using vt::cpu::ArmIsaTier; + using vt::cpu::SelectArmIsaTier; + ArmIsaTier selected{}; + std::string error; + + vt::cpu::ArmIsaCaps caps{}; + CHECK(SelectArmIsaTier(caps, "", &selected, &error)); + CHECK(selected == ArmIsaTier::kPortable); + + caps.neon = true; + CHECK(SelectArmIsaTier(caps, "", &selected, &error)); + CHECK(selected == ArmIsaTier::kNeon); + + caps.dotprod = true; + CHECK(SelectArmIsaTier(caps, "", &selected, &error)); + CHECK(selected == ArmIsaTier::kDotProd); + + caps.i8mm = true; + CHECK(SelectArmIsaTier(caps, "", &selected, &error)); + CHECK(selected == ArmIsaTier::kI8mm); + + caps.dotprod = false; + CHECK_FALSE(SelectArmIsaTier(caps, "i8mm", &selected, &error)); +} + +TEST_CASE("Arm forced tiers fail closed instead of silently narrowing") { + using vt::cpu::ArmIsaTier; + ArmIsaTier selected{}; + std::string error; + const vt::cpu::ArmIsaCaps baseline{.neon = true}; + + CHECK(vt::cpu::SelectArmIsaTier(baseline, "portable", &selected, &error)); + CHECK(selected == ArmIsaTier::kPortable); + CHECK(vt::cpu::SelectArmIsaTier(baseline, "neon", &selected, &error)); + CHECK(selected == ArmIsaTier::kNeon); + CHECK_FALSE(vt::cpu::SelectArmIsaTier(baseline, "dotprod", &selected, &error)); + CHECK_FALSE(vt::cpu::SelectArmIsaTier(baseline, "i8mm", &selected, &error)); + CHECK_FALSE(vt::cpu::SelectArmIsaTier(baseline, "sve2", &selected, &error)); + const bool has_reason = error.find("unsupported") != std::string::npos || + error.find("unknown") != std::string::npos; + CHECK(has_reason); +} + +TEST_CASE("independent Arm feature toggles force execution or fail closed") { + bool enabled = false; + std::string error; + const vt::cpu::ArmIsaCaps baseline{.neon = true}; + const vt::cpu::ArmIsaCaps rich{.neon = true, .dotprod = true, .i8mm = true}; + + CHECK(vt::cpu::ResolveArmIsaToggle( + baseline, vt::cpu::ArmIsaTier::kI8mm, "portable", &enabled, &error)); + CHECK_FALSE(enabled); + CHECK_FALSE(vt::cpu::ResolveArmIsaToggle( + baseline, vt::cpu::ArmIsaTier::kI8mm, "i8mm", &enabled, &error)); + CHECK(vt::cpu::ResolveArmIsaToggle( + rich, vt::cpu::ArmIsaTier::kI8mm, "i8mm", &enabled, &error)); + CHECK(enabled); + CHECK(vt::cpu::ResolveArmIsaToggle( + rich, vt::cpu::ArmIsaTier::kDotProd, "auto", &enabled, &error)); + CHECK(enabled); + CHECK_FALSE(vt::cpu::ResolveArmIsaToggle( + rich, vt::cpu::ArmIsaTier::kDotProd, "sve2", &enabled, &error)); +} + +TEST_CASE("Arm release inventory lists only kernels that exist") { + const auto inventory = vt::cpu::ArmIsaTierInventory(); + REQUIRE(inventory.size() == 4); + CHECK(inventory[0].name == "portable"); + CHECK(inventory[1].name == "neon"); + CHECK(inventory[2].name == "dotprod"); + CHECK(inventory[3].name == "i8mm"); + CHECK(inventory[2].kernel_families == "q8-dot"); + CHECK(inventory[2].linux_probe == "getauxval:AT_HWCAP:HWCAP_ASIMDDP"); + CHECK(inventory[2].darwin_probe == "sysctl:hw.optional.arm.FEAT_DotProd"); + CHECK(inventory[3].kernel_families == "quant-dot,quant-repack"); + CHECK(inventory[3].linux_probe == "getauxval:AT_HWCAP2:HWCAP2_I8MM"); + CHECK(inventory[3].darwin_probe == "sysctl:hw.optional.arm.FEAT_I8MM"); + for (const auto& tier : inventory) { + CHECK(tier.cpu_features.find("sve") == std::string_view::npos); + CHECK(tier.cpu_features.find("sme") == std::string_view::npos); + CHECK(tier.cpu_features.find("bf16") == std::string_view::npos); + } +} + +TEST_CASE("detected Arm host resolves to a supported tier") { + vt::cpu::ArmIsaTier selected{}; + std::string error; + const auto caps = vt::cpu::DetectArmIsaCaps(); + CHECK(vt::cpu::SelectArmIsaTier(caps, "", &selected, &error)); + CHECK(vt::cpu::ArmIsaTierSupported(caps, selected)); +} diff --git a/tests/vt/test_cpu_isa_x86.cpp b/tests/vt/test_cpu_isa_x86.cpp new file mode 100644 index 000000000..f4c0cdcc8 --- /dev/null +++ b/tests/vt/test_cpu_isa_x86.cpp @@ -0,0 +1,90 @@ +#include +#include + +#include "doctest/doctest.h" +#include "vt/cpu/cpu_isa_x86.h" + +namespace { + +vt::cpu::X86IsaCaps FullAvx512() { + return {.sse2 = true, + .f16c = true, + .avx = true, + .osxsave = true, + .avx2 = true, + .avx512f = true, + .avx512bw = true, + .avx512vl = true, + .xcr0 = 0xe6}; +} + +} // namespace + +TEST_CASE("x86 ISA selection requires CPU bits and exact OS-enabled state") { + using vt::cpu::SelectX86IsaTier; + using vt::cpu::X86IsaTier; + X86IsaTier selected{}; + std::string error; + + vt::cpu::X86IsaCaps caps{.sse2 = true}; + CHECK(SelectX86IsaTier(caps, "", &selected, &error)); + CHECK(selected == X86IsaTier::kSse2); + + caps = FullAvx512(); + caps.xcr0 = 0x6; + CHECK(SelectX86IsaTier(caps, "", &selected, &error)); + CHECK(selected == X86IsaTier::kAvx2); + CHECK_FALSE(SelectX86IsaTier(caps, "avx512", &selected, &error)); + + caps = FullAvx512(); + for (bool* bit : std::array{&caps.avx512f, &caps.avx512bw, &caps.avx512vl}) { + *bit = false; + CHECK_FALSE(SelectX86IsaTier(caps, "avx512", &selected, &error)); + *bit = true; + } + CHECK(SelectX86IsaTier(caps, "avx512", &selected, &error)); + CHECK(selected == X86IsaTier::kAvx512); +} + +TEST_CASE("x86 forced tiers fail closed instead of silently narrowing") { + using vt::cpu::SelectX86IsaTier; + using vt::cpu::X86IsaTier; + X86IsaTier selected{}; + std::string error; + const vt::cpu::X86IsaCaps baseline{.sse2 = true}; + + CHECK(SelectX86IsaTier(baseline, "portable", &selected, &error)); + CHECK(selected == X86IsaTier::kPortable); + CHECK(SelectX86IsaTier(baseline, "sse2", &selected, &error)); + CHECK(selected == X86IsaTier::kSse2); + CHECK_FALSE(SelectX86IsaTier(baseline, "sse2+f16c", &selected, &error)); + CHECK_FALSE(SelectX86IsaTier(baseline, "avx2", &selected, &error)); + CHECK_FALSE(SelectX86IsaTier(baseline, "avx512", &selected, &error)); + CHECK_FALSE(SelectX86IsaTier(baseline, "amx", &selected, &error)); + const bool has_reason = error.find("unsupported") != std::string::npos || + error.find("unknown") != std::string::npos; + CHECK(has_reason); +} + +TEST_CASE("x86 release inventory lists only compiled elementwise GEMM tiers") { + const auto inventory = vt::cpu::X86IsaTierInventory(); + REQUIRE(inventory.size() == 5); + CHECK(inventory[0].name == "portable"); + CHECK(inventory[1].name == "sse2"); + CHECK(inventory[2].name == "sse2+f16c"); + CHECK(inventory[3].name == "avx2"); + CHECK(inventory[4].name == "avx512"); + for (const auto& tier : inventory) { + CHECK(tier.kernel_family == "elementwise-gemm"); + CHECK(tier.cpu_features.find("vnni") == std::string_view::npos); + CHECK(tier.cpu_features.find("amx") == std::string_view::npos); + } +} + +TEST_CASE("detected host resolves to a supported tier") { + vt::cpu::X86IsaTier selected{}; + std::string error; + const auto caps = vt::cpu::DetectX86IsaCaps(); + CHECK(vt::cpu::SelectX86IsaTier(caps, "", &selected, &error)); + CHECK(vt::cpu::X86IsaTierSupported(caps, selected)); +} diff --git a/tests/vt/test_triton_aot_arch_dispatch.cpp b/tests/vt/test_triton_aot_arch_dispatch.cpp new file mode 100644 index 000000000..6e46b1eb3 --- /dev/null +++ b/tests/vt/test_triton_aot_arch_dispatch.cpp @@ -0,0 +1,66 @@ +#include +#include + +#include "doctest/doctest.h" +#include "vt/cuda/triton_aot_arch_dispatch.h" + +namespace { + +int Tree0(int value) { return value + 0; } +int Tree1(int value) { return value + 10; } +int Tree2(int value) { return value + 20; } +int Tree3(int value) { return value + 30; } +int Tree4(int value) { return value + 40; } +int Tree5(int value) { return value + 50; } + +int loaded_tree = -1; +void Load0() { loaded_tree = 0; } +void Load1() { loaded_tree = 1; } +void Load2() { loaded_tree = 2; } +void Load3() { loaded_tree = 3; } +void Load4() { loaded_tree = 4; } +void Load5() { loaded_tree = 5; } + +} // namespace + +TEST_CASE("Triton AOT selects only the exact six vendored SM trees") { + const std::array trees{&Tree0, &Tree1, &Tree2, &Tree3, &Tree4, &Tree5}; + CHECK(vt::cuda::TritonAotTreeIndex(8, 0) == 0); + CHECK(vt::cuda::TritonAotTreeIndex(8, 6) == 1); + CHECK(vt::cuda::TritonAotTreeIndex(8, 9) == 2); + CHECK(vt::cuda::TritonAotTreeIndex(9, 0) == 3); + CHECK(vt::cuda::TritonAotTreeIndex(10, 0) == 4); + CHECK(vt::cuda::TritonAotTreeIndex(12, 1) == 5); + + CHECK(vt::cuda::DispatchTritonAot(8, 0, -1, trees, 7) == 7); + CHECK(vt::cuda::DispatchTritonAot(8, 6, -1, trees, 7) == 17); + CHECK(vt::cuda::DispatchTritonAot(8, 9, -1, trees, 7) == 27); + CHECK(vt::cuda::DispatchTritonAot(9, 0, -1, trees, 7) == 37); + CHECK(vt::cuda::DispatchTritonAot(10, 0, -1, trees, 7) == 47); + CHECK(vt::cuda::DispatchTritonAot(12, 1, -1, trees, 7) == 57); +} + +TEST_CASE("Triton AOT falls back for the four release SMs without trees") { + const std::array trees{&Tree0, &Tree1, &Tree2, &Tree3, &Tree4, &Tree5}; + for (const auto [major, minor] : + std::array, 4>{{{8, 7}, {10, 3}, {11, 0}, {12, 0}}}) { + CHECK(vt::cuda::TritonAotTreeIndex(major, minor) == -1); + CHECK(vt::cuda::DispatchTritonAot(major, minor, -99, trees, 7) == -99); + } +} + +TEST_CASE("Triton AOT wrong-tree mutation is observable") { + const std::array swapped{&Tree1, &Tree0, &Tree2, &Tree3, &Tree4, &Tree5}; + CHECK(vt::cuda::DispatchTritonAot(8, 0, -1, swapped, 7) != 7); + CHECK(vt::cuda::DispatchTritonAot(8, 6, -1, swapped, 7) != 17); +} + +TEST_CASE("Triton AOT loader dispatch is exact and unavailable trees are no-op") { + const std::array loaders{&Load0, &Load1, &Load2, &Load3, &Load4, &Load5}; + loaded_tree = -1; + CHECK(vt::cuda::DispatchTritonAotVoid(9, 0, loaders)); + CHECK(loaded_tree == 3); + loaded_tree = -1; + CHECK_FALSE(vt::cuda::DispatchTritonAotVoid(12, 0, loaders)); + CHECK(loaded_tree == -1); +}