fix(release): flatten immutable artifact handoffs

.agents/coordination.md

@@ -1559,7 +1559,7 @@ conflict.
 |---|---|---|---|---|---|---|---|
 | `CLAIM-MOONCAKE-STORE` | `KV-MOONCAKE-STORE` (new, `SPIKE`) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-mooncake`; records-only — NO build, NO GPU, NO download, NO external dependency installed | `row/KV-MOONCAKE-STORE`, base `origin/main` `848d4a87` | The `MooncakeStoreConnector` spike, records-only in this commit: NEW [`.agents/specs/mooncake-store-connector.md`](specs/mooncake-store-connector.md), the NEW `KV-MOONCAKE-STORE` engine-matrix row + section/total counters, the `ENGINE_ROWS` 146→147 bump in `scripts/check-agent-record.py` with its justification comment, a PROSE-ONLY Mooncake-disposition correction inside the `KV-CONNECTORS` row (both NOT-SCHEDULED sentences; that row's `ANCHOR-BACKFILL` state, tier, anchors, evidence, spec link and owner are UNCHANGED, so it is not claimed here), the `ROAD-V1-D4` portfolio note + canonical-table link, the roadmap issue-table row for [#287](https://github.com/mudler/vllm.cpp/issues/287), and this claim. **NON-COLLISION:** touches NO `src/`, `include/`, `tests/`, `examples/` or CMake path; the only script touched is the record checker's own row-count constant. | `ACTIVE` | 2026-08-10 — spec committed. The `KV-CONNECTORS` spike's blanket "Mooncake NOT SCHEDULED" conflated TWO connectors: `MooncakeConnector` (P2P prefill/decode over the Transfer Engine — two nodes, fabric, proxy) KEEPS that verdict; `MooncakeStoreConnector` (shared KV object store, the LMCache analogue) is reopened because (a) Mooncake is NATIVE C++ — `mooncake::Client` in `client_service.h`, and the `MooncakeDistributedStore` vLLM imports is a pybind wrapper over it — so we LINK instead of reimplementing a wire (the inverse of the LMCache cost shape), and (b) its single-node `protocol: "tcp"` + `mooncake_master` config is gateable on one box with NO RDMA NIC. The landed W5 `KVConnector` seam needs NO change. NEXT: W0, the go/no-go link spike (build Mooncake from source, pin the revision, drive `mooncake::Client` from a standalone C++ TU against a local master over TCP) — a genuine stop point per spec §S1. Speed is recorded as an OPEN axis: the RDMA/GPUDirect path that motivates the connector is unmeasurable for want of a fabric on any box we own. PENDING developer authority: the Mooncake source build, push/PR. |
 | `CLAIM-SPEC-DSPARK` | `SPEC-DSPARK` (`ACTIVE`) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-spec-dspark`; CPU-only so far, NO build, NO GPU, NO download | `row/SPEC-DSPARK`, base `origin/main` `bc6e3d72`; NOT PUSHED, no PR yet (remote step PENDING developer authority) | The DSpark spike, records-only in this commit: NEW `.agents/specs/dspark-spec-decode.md`, the `SPEC-DSPARK` engine-matrix row + section/total counters, the feature-matrix §8 DSpark row, the superseded grounding-note header, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/FEATURES.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** touches NO `src/`, `include/`, `tests/`, `examples/` or CMake path. Implementation slices W1-W6 follow under this same claim. | `ACTIVE` | 2026-08-09 — spike committed. DSpark = the landed DFlash lane + Markov logit-bias head + sequential block sampling + anchor-as-first-prediction layout + `d2t` reduced vocab + method/config resolution + Speculators-format translation; upstream surface is 1613 lines over 5 files, 3 of them DFlash subclasses. Draft checkpoints exist for both gate models and for the 4B pair the upstream test uses; DeepSeek-V4 DSpark is out of scope (HW-blocked). NEXT: W1 config slice (CPU, RED = `speculative.cpp:44` rejects `"dspark"` today) and R1, prove the pinned oracle `555967922` actually RUNS DSpark (it forces the V2 runner). PENDING developer authority: checkpoint downloads, dgx GPU time, push/draft-PR. |
-| `CLAIM-ENG-RELEASE-BINARIES-W1-W13` | `ENG-RELEASE-BINARIES` (`ACTIVE`; complete release matrix) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; inline execution in the existing isolated worktree | `row/ENG-RELEASE-BINARIES`, base `origin/main` `81291a89`, draft PR #196 | Complete W1-W13 contract in one PR: fat-CUDA gencode and AOT, adaptive CPU tiers, staged validation and supply chain, least-privilege release automation, all declared bundles, generated index, and tagged-release audit. W5 is inherited from main; ROCm remains blocked by contract | `ACTIVE` | 2026-08-09 — required W1-W11/W13 implementation complete; local CPU/Vulkan/mutation gates green; hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, rebase/merge, and tagged publication pending; W12 optional/non-primary |
+| `CLAIM-ENG-RELEASE-BINARIES-W1-W13` | `ENG-RELEASE-BINARIES` (`ACTIVE`; complete release matrix) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; inline execution in the existing isolated worktree | `row/ENG-RELEASE-HANDOFF-FLAT`, base `origin/main` `24306364`; follow-up PR pending | Complete W1-W13 contract in one PR; the current follow-up completes the W8/W13 hosted handoff after run `31408404388` built and uploaded all eight required tuples but aggregation failed on nested `download-artifact` extraction. Owns the workflow-wide flat-download invariant, checker and red-first mutation, hosted dry-run verification, and exact release-readiness audit; no backend/package/manifest behavior change. ROCm remains blocked | `ACTIVE` | 2026-08-09 — required W1-W11/W13 implementation complete; hosted ten-SM completion, full eight-tuple dry run, matching-hardware gates, rebase/merge, and tagged publication pending. Update 2026-08-10: all eight bundles now build and upload, while aggregation still fails before verify because `plan/release-plan.json` is nested under the artifact name; W12 optional/non-primary |
 | `CLAIM-ENG-RELEASE-BINARIES-W5` | `ENG-RELEASE-BINARIES` (`ACTIVE`; W5 only) | Codex (GPT-5) | `/home/mudler/_git/vllm.cpp-release-binaries`; CPU/build-time metadata tooling only, no GPU/download/service work | `row/ENG-RELEASE-BINARIES`, base `b38f78a7`, claim commit `29107d0b`, draft PR #141 | W5 implemented: versioned manifest schema + deterministic generator/validator, canonical synthetic CPU/CUDA fixtures, fail-closed mutation tests, release checker/registration and required record/doc checkpoints. Excludes W1-W4, W6-W13, archives, install/package/publish workflows and runtime artifacts | `ACTIVE` | 2026-08-08 — W5 19/19; fresh-review production removals 10/10 killed; accepted release suite 30/30; no archive or real runtime/correctness/performance evidence |
 | `CLAIM-KIMI-LINEAR-W0` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — dedicated W0 spike) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a771cc029e6843be0`; CPU-only, records-only — NO build, NO GPU, NO download (two GPU jobs queued ahead; the W0 GPU golden capture is a SEPARATE later step) | branch `worktree-agent-a771cc029e6843be0`, base `origin/main` HEAD `10dd23ee` | The FULL dedicated W0 spike for `KimiLinearForCausalLM` (Kimi-Linear-48B-A3B) per the spike-first protocol, so W1 implementation can start immediately. Owns ONLY: NEW `.agents/specs/kimi-linear.md`, the `MODEL-TEXT-kimi-linear-*` matrix row (Spike link → the dedicated spec + note; STAYS `SPIKE` because it is actively claimed — protocol forbids a claimed row from being `READY`; checklist mark stays `📋`), this claim row + the narrative block below, the roadmap breadth note, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, the `.agents/NOW.md` live-claim row + stamp, and one `.agents/state.md` entry. **NON-COLLISION:** records-only — touches NO model/kernel/registry/loader source, NO CMake, NO test. Co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` (MLA half) + `CLAIM-KDA-KERNEL` (KDA host refs), exactly as the K3 row is co-owned by `CLAIM-KIMI-K3-SCOPE`/`CLAIM-KIMI-K3-W2-W5`. | `DONE` | 2026-08-05 — **W0 DEDICATED SPIKE LANDED (records-only, NOT pushed).** Authoritative `config.json` fetched (27 layers = 20 KDA + 7 NoPE-MLA; 256e/top-8/1-shared sigmoid `noaux_tc` `routed_scaling=2.446` `first_k_dense_replace=1`; `num_nextn_predict_layers=0` ⇒ no MTP); reuse-vs-new map with our `file:line`; HW-fit (FITS one GB10, 91.5 GiB / 0.77× pool — real e2e SACRED gate reachable); W0-W7 breakdown + GPU golden-capture recipe. NET-NEW = KDA device kernel (host refs landed, `CLAIM-KDA-KERNEL`), NoPE-MLA branch, hybrid schedule/het-KV, loader name-map. Record checkers green (`check-model-checklist`, `check-agent-record`, `check-doc-checkpoint`, `check-now-current`). |
 | `CLAIM-KIMI-LINEAR-W1` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` (stays SPIKE — W1 registry/config/loader scaffolding; forward REFUSES-by-name) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/agent-a11ae231a71e8694b`; CPU-only `build-cpu` (`-DVLLM_CPP_CUDA=OFF` Release); NO GPU, NO download — additive TU + one REGISTER line, forward is a `VT_CHECK(false)` stub | branch `worktree-agent-a11ae231a71e8694b`, base `origin/main` HEAD `053116df` | Kimi-Linear-48B-A3B W1 — registry + config + loader + KV-spec scaffolding so the W3-W6 forward can start. Owns ONLY: NEW `include/vllm/model_executor/models/kimi_linear.h`, NEW `src/vllm/model_executor/models/{kimi_linear_registry,kimi_linear_weights,kimi_linear}.cpp`, NEW `tests/vllm/models/test_kimi_linear_scaffold.cpp`, its two CMake registration lines (`CMakeLists.txt` source list + `tests/CMakeLists.txt`), the `KimiLinearForCausalLM` sorted-set + error-message + model-property inserts in `tests/vllm/models/test_model_registry.cpp`, the `MODEL-TEXT-kimi-linear-*` row cells + checklist Status (STAYS `SPIKE`/`📋` — the forward refuses, rollup UNCHANGED), this claim, `.agents/NOW.md`, `docs/STATUS.md`/`docs/BENCHMARKS.md`/`docs/FEATURES.md` one-liners, and one `.agents/state.md` entry. **NON-COLLISION:** additive TU + one REGISTER line ⇒ ZERO edit to any shared array; the forward is a `VT_CHECK(false)` refuse stub so NO production/device path changes; co-owns the Kimi-Linear row with `CLAIM-MLA-DEEPSEEK` + `CLAIM-KDA-KERNEL` + `CLAIM-KIMI-LINEAR-W0`; MUST NOT touch README, Metal/SACRED/apex/darwin, or any other model/kernel source. | `DONE` | 2026-08-05 — **W1 CPU SCAFFOLDING LANDED (foreground, NOT pushed).** Additive registry (`kimi_linear`/`KimiLinearForCausalLM`, `is_hybrid`, `supports_multimodal=false`) + `ParseKimiLinearParams` (20 KDA + 7 NoPE-MLA schedule, MLA 512/128/64/128 q_lora-null, 256e/top-8/1-shared sigmoid `noaux_tc` 2.446, `first_k_dense_replace=1`; asserts `mla_use_nope`/`q_lora==null`) + `EnumerateKimiLinearTensors` name-map VERIFIED vs the real HF safetensors index (MoE = `block_sparse_moe.*`, the correction of the DERIVED K3 `mlp.*`) + loader THROWS BY NAME on missing/mis-shaped tensors + het KV spec (MLA latent-576 + KDA mamba `12288×3`/`32×128×128`) + REFUSE-by-name forward. `test_kimi_linear_scaffold` **9/9·83** + `test_model_registry` **24/24** + `test_kimi_k3_scaffold` 6/6 unchanged; clean CPU Release build; `check-fusion-consistency`/`check-runner-routing-consistency` green (refuse stub skipped); record checkers rc=0. RESIDUAL (W3-W6): KDA device kernel + NoPE-MLA route + sigmoid-noaux MoE + het-KV born-on-runner forward + the e2e SACRED gate (spec §8 recipe). |

.agents/specs/release-binary-matrix.md

@@ -531,6 +531,55 @@ or replacing the runtime resolution smoke with `readelf` alone. The former
 weakens the no-build-path release invariant, while the latter stops proving
 that the extracted executable's declared dependencies resolve.
 
+## Hosted artifact-handoff completion
+
+The 2026-08-10 manual dry run at Actions run `31408404388` proved that all
+eight required platform jobs build, validate, package, and upload their exact
+bundle triplets. The aggregate `build` job `93565669335` then failed before
+handoff verification with
+`[Errno 2] No such file or directory: 'plan/release-plan.json'`.
+`actions/download-artifact@v4` had extracted the exact plan artifact below an
+additional artifact-name directory because that download did not set
+`merge-multiple: true`; the consumer intentionally reads the stable flat path
+`plan/release-plan.json`.
+
+The same latent layout mismatch applies to every later exact single-artifact
+handoff. `verify` reads both `plan/release-plan.json` and
+`unverified/release-handoff.json`; `attest` and `publish` read the verified
+handoff and assets from fixed paths. The asset-set download already opts into
+flat extraction and succeeded. Fixing only the first observed failure would
+therefore defer the same failure to verify or to the first real tag run.
+
+The approved completion is one invariant across the whole workflow:
+
+1. Every `actions/download-artifact@v4` step uses `merge-multiple: true`,
+   including exact single-artifact downloads. Artifact names stay immutable and
+   SHA-bound, and consumer paths stay stable and explicit.
+2. `scripts/check-release-workflow.py` fails unless the flattening invariant is
+   present on every download step. It continues requiring exact artifact names,
+   explicit paths, immutable handoffs, least-privilege permissions, and
+   wildcard-free publication.
+3. `tests/scripts/test_release_pipeline.py` first demonstrates a red mutation
+   by removing or falsifying one download's flattening flag, then proves the
+   repaired workflow green. Existing workflow and release-pipeline mutations
+   remain green.
+4. The focused workflow checker and mutation suite, full preflight, fresh
+   static plus scratch-mutation review, and operator rerun must pass before the
+   branch is pushed. A new manual dry run must then reach the `verify` job with
+   all eight tuples and an immutable verified handoff.
+5. Manual `workflow_dispatch` remains non-publishing by design: successful
+   build and verify may advance archive evidence, but cannot prove OIDC
+   attestation or GitHub Release publication. Those stages require a real
+   signed/authorized `v*` tag whose version matches `CMakeLists.txt`, followed
+   by an audit that every matrix archive, checksum, SBOM/provenance sidecar, and
+   generated release index was attached from the verified handoff.
+
+Rejected alternatives are duplicating artifact-name subdirectories throughout
+consumer paths or adding discovery/move scripts. Both repeat generated names
+outside their producing expressions and weaken the fixed-path handoff contract.
+Universal flat extraction is the action's native mechanism and keeps the
+workflow's exact-file publication boundary unchanged.
+
 ## Spike verdict
 
 The release program is feasible as backend-specific static-core bundles with a

.github/workflows/release.yml

@@ -372,6 +372,7 @@ jobs:
         with:
           artifact-ids: ${{ needs.plan.outputs.artifact_id }}
           path: plan
+          merge-multiple: true
       - name: Download exact validated CPU artifacts by immutable IDs
         uses: actions/download-artifact@v4
         with:
@@ -410,11 +411,13 @@ jobs:
         with:
           artifact-ids: ${{ needs.plan.outputs.artifact_id }}
           path: plan
+          merge-multiple: true
       - name: Download exact unverified handoff by ID
         uses: actions/download-artifact@v4
         with:
           artifact-ids: ${{ needs.build.outputs.artifact_id }}
           path: unverified
+          merge-multiple: true
       - name: Verify handoff against plan and workflow SHA
         run: |
           mkdir -p verified
@@ -457,6 +460,7 @@ jobs:
         with:
           artifact-ids: ${{ needs.verify.outputs.artifact_id }}
           path: verified
+          merge-multiple: true
       - name: Attest verified bytes
         uses: actions/attest@v4
         with:
@@ -475,6 +479,7 @@ jobs:
         with:
           artifact-ids: ${{ needs.verify.outputs.artifact_id }}
           path: verified
+          merge-multiple: true
       - name: Publish the exact verified release assets
         env:
           GH_TOKEN: ${{ github.token }}

scripts/check-release-workflow.py

@@ -19,6 +19,29 @@ def job_block(text: str, name: str) -> str:
     return match.group(1) if match else ""
 
 
+def action_steps(text: str, action: str) -> list[str]:
+    starts = [match.start() for match in re.finditer(r"(?m)^      - ", text)]
+    starts.append(len(text))
+    return [
+        text[start:end]
+        for start, end in zip(starts, starts[1:])
+        if re.search(rf"(?m)^        uses: {re.escape(action)}$", text[start:end])
+    ]
+
+
+def step_mapping_blocks(step: str, name: str) -> list[str]:
+    lines = step.splitlines()
+    blocks: list[str] = []
+    for index, line in enumerate(lines):
+        if line != f"        {name}:":
+            continue
+        end = index + 1
+        while end < len(lines) and not re.match(r"^        \S", lines[end]):
+            end += 1
+        blocks.append("\n".join(lines[index + 1 : end]))
+    return blocks
+
+
 def validate(text: str) -> list[str]:
     errors: list[str] = []
     required_global = (
@@ -124,7 +147,8 @@ def validate(text: str) -> list[str]:
         if fragment not in text:
             errors.append(f"immutable artifact handoff is missing {fragment!r}")
     uploads = text.count("uses: actions/upload-artifact@v4")
-    downloads = text.count("uses: actions/download-artifact@v4")
+    download_steps = action_steps(text, "actions/download-artifact@v4")
+    downloads = len(download_steps)
     if uploads < 3:
         errors.append("release workflow requires immutable plan, asset, and verified uploads")
     if text.count("overwrite: false") != uploads:
@@ -133,6 +157,19 @@ def validate(text: str) -> list[str]:
         errors.append("every artifact upload must fail when its explicit file is missing")
     if downloads < 5 or text.count("artifact-ids:") != downloads:
         errors.append("every cross-job handoff must use an exact immutable artifact ID")
+    flatten_values = []
+    for step in download_steps:
+        with_blocks = step_mapping_blocks(step, "with")
+        flatten_values.append(
+            re.findall(
+                r"(?m)^          merge-multiple:\s*(true|'true'|\"true\")\s*$",
+                with_blocks[0],
+            )
+            if len(with_blocks) == 1
+            else []
+        )
+    if any(values not in (["true"], ["'true'"], ['"true"']) for values in flatten_values):
+        errors.append("every artifact download must flatten into its declared path")
     if re.search(r"(?m)^\s+path:\s*[^\n]*[*?]", text):
         errors.append("release workflow artifact paths must not use wildcards")
     if re.search(r"gh release (?:create|upload)[^\n]*[*?]", text):