ci: give main a baseline lane that can actually finish (#274)

.agents/roadmap_v1.md

@@ -94,7 +94,9 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#243](https://github.com/mudler/vllm.cpp/issues/243) | — | `vllm-feature-gap-analysis.md` is a stale 2026-07-28 snapshot: 9 of 16 HIGH/MED gaps have since landed | bug |
 | [#250](https://github.com/mudler/vllm.cpp/issues/250) | — | `a5b52047` reached main without a task branch, and `check-role-discipline` cannot be waived | bug |
 | [#285](https://github.com/mudler/vllm.cpp/issues/285) | — | The operator lock refuses a second coordinator; it should only RECORD who is working where (spec `specs/operator-record.md`) | bug |
+| [#274](https://github.com/mudler/vllm.cpp/issues/274) | — | `main` is not verified by its own CI: the per-job `github.ref` concurrency groups cancel every long job on the next push, so the suite never completes (spec `specs/main-verifiability.md`) | bug |
 | [#296](https://github.com/mudler/vllm.cpp/issues/296) | — | Two limitations recorded when #285 landed: a stale TTL comment, and a publish-NAME pin `os.rename` escapes (spec `specs/operator-record.md`, "Follow-up") | bug |
+| [#408](https://github.com/mudler/vllm.cpp/issues/408) | — | 12 of 54 `tests/scripts` suites are executed by nothing, and `check-test-registration.py`'s fixed `REQUIRED_TESTS` cannot see the class (found while repairing #274) | bug |
 
 ## Top-level portfolio
 

.agents/verification.md

@@ -28,6 +28,37 @@ type before believing a surprising green.
 Tests that starve under `ctest -j` are re-run serially before being called a
 regression.
 
+## Is `main` green? — the baseline lane
+
+Before spending a build cycle proving a red check is not yours, ask:
+
+```sh
+scripts/main-baseline.py            # last fully green SHA, and what is failing now
+scripts/main-baseline.py --json
+```
+
+It reads the `schedule`/`workflow_dispatch` runs of `.github/workflows/ci.yml` —
+the only lane whose long jobs are not cancelled by the next push — and derives
+the verdict at read time. Nothing is stored, so there is no file to conflict on
+and no file that can be stale relative to the runs.
+
+Three things to know before you trust or dismiss a red check.
+
+- **A `push` run on `main` proves almost nothing.** Its expensive jobs share a
+  ref-keyed concurrency group, so the next push cancels them. Of 40 consecutive
+  runs measured for [#274](https://github.com/mudler/vllm.cpp/issues/274), 26
+  were `cancelled` and exactly one completed.
+- **A run's own conclusion is not the verdict.** `sanitize-cpu` is
+  `continue-on-error`, so a run reports `success` with the sanitizers red — run
+  `31448896841` at `5812b8b6` is exactly that. The tool reads per-job
+  conclusions and so should you.
+- **Staleness is visible, not silent.** Every line carries the run's date. If the
+  newest baseline is old, say so; never read an absent run as a pass, and never
+  read `REMOTE_UNVERIFIED` as one either.
+
+To pin a baseline on a SHA you care about right now, rather than waiting for the
+4-hourly cron: `gh workflow run ci.yml --ref main`.
+
 ## Reviewing
 
 Review happens only after the implementation's own gates pass, and only on an

.github/workflows/ci.yml

@@ -3,6 +3,31 @@ on:
   push:
     branches: [main]
   pull_request:
+  # THE BASELINE LANE (issue #274, spec .agents/specs/main-verifiability.md).
+  #
+  # The push lane below cannot answer "is main green?", by construction. Every
+  # expensive job carries a job-level group keyed on `github.ref`, which is the
+  # constant `refs/heads/main` for every push, so consecutive pushes cancel each
+  # other's long jobs. Measured over 40 consecutive main runs at 0eb049f7: 26
+  # cancelled, 12 failure, 1 success. The cancel instant equals the next push's
+  # start instant (run 31485402200 died 11:46:33, run 31488132224 started
+  # 11:46:32; 31482845117 died 11:05:12, 31485054749 started 11:05:11). At 55
+  # pushes/day and a 99-minute suite, that is structural, not bad luck.
+  #
+  # This lane runs the SAME jobs on a cadence, in its own concurrency groups so
+  # a push cannot cancel it, and publishes a verdict via `baseline-summary`.
+  # Every 4h: at 55 pushes/day a red baseline then names ~9 commits (three
+  # bisect steps), and successive runs still cannot overlap a 99-minute suite.
+  # Making every MERGE non-cancellable was rejected: ~55 overlapping runs/day,
+  # ~190 h/day of job time on a pool where run 31485402200 already sat queued 37
+  # minutes without starting a single job -- and at that rate the newest
+  # complete baseline would be several commits stale anyway.
+  schedule:
+    - cron: '17 */4 * * *'
+  # The hybrid half: pin a baseline on a SHA you care about right after merging
+  # it, without paying for a full run on all 55 of the day's pushes.
+  #   gh workflow run ci.yml --ref main
+  workflow_dispatch:
 
 # Workflow-level: dedupe PR pushes only. The group is keyed on the SHA for a
 # push, so two pushes to main never share a group and never cancel each other.
@@ -23,17 +48,24 @@ on:
 # gates written to cover them. The groups are removed; these two jobs must never
 # regain one. They are a checkout plus a Python script, so the cost of always
 # running them is negligible against a silently skipped gate.
+#
+# 2026-08-11 (#274): `github.event_name` joins the key. `cancel-in-progress` is
+# already false for every non-PR event, so this cancels nothing new -- it stops
+# a `workflow_dispatch` baseline fired on a SHA that was just pushed from
+# sharing group `ci-<sha>-<repo>` with that push's run and QUEUEING behind it.
+# For `push` and `pull_request` the added token is a constant, so their
+# partition into groups is exactly what it was.
 concurrency:
-  group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  group: ci-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
   cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   agent-record:
     # Tree-scoped: validates the record as it stands at HEAD, so only the newest
     # push to a ref is meaningful. Cancellable.
     concurrency:
-      group: ci-agent-record-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-agent-record-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -120,7 +152,29 @@ jobs:
         run: |
           python3 scripts/check-gemv-invocation-consistency.py
           python3 tests/scripts/test_check_gemv_invocation_consistency.py
+      - name: The main baseline lane and its reader stay honest
+        # Registered in TWO places, deliberately: here and in the `SUITES` array
+        # of `scripts/agent-preflight.sh`. It shipped in neither, so the 24 tests
+        # that guard the baseline lane ran on no machine -- the same class of
+        # defect as an unregistered CTest target, one layer up. #408 tracks
+        # whether a checker should catch the class; this closes the instance.
+        run: |
+          python3 tests/scripts/test_main_baseline.py
       - name: Protocol prose matches the checkers that enforce it
+        # PART tree-scoped, PART diff-scoped, and the two must be told apart on
+        # the baseline lane (#274). `documentation-checkpoint` and
+        # `commit-protocol-tag` opt OUT of `schedule`/`workflow_dispatch`
+        # wholesale because they are diff-scoped end to end; this job cannot,
+        # because most of it is tree-scoped and the baseline needs it. So the
+        # RANGE-SCOPED call is guarded here instead of the whole job.
+        #
+        # Without that guard this step DID kill the lane: a `schedule` payload
+        # has no `github.event.before`, `PUSH_BASE` renders empty, and under
+        # `set -eu` the walk aborts with
+        #   `--range: range must be exactly BASE..HEAD` (exit 2)
+        # so `agent-record` could never be green and `baseline-summary`, which
+        # `needs:` it, could never publish GREEN. Reproduced by replaying this
+        # body with EVENT_NAME=schedule, PUSH_BASE="".
         env:
           EVENT_NAME: ${{ github.event_name }}
           PR_BASE: ${{ github.event.pull_request.base.sha }}
@@ -141,7 +195,12 @@ jobs:
             base="$PUSH_BASE"
             head="$PUSH_HEAD"
           fi
-          python3 scripts/check-commit-trailers.py --range "$base..$head"
+          if [ -z "$base" ]; then
+            echo "no diff range on the $EVENT_NAME lane: the trailer walk is"
+            echo "diff-scoped and every commit's own push already ran it."
+          else
+            python3 scripts/check-commit-trailers.py --range "$base..$head"
+          fi
       - name: NOW.md stays a short one-Read resume surface
         run: |
           python3 scripts/check-now-current.py
@@ -159,6 +218,16 @@ jobs:
           python3 scripts/check-gate-commands.py --check
           python3 tests/scripts/test_check_gate_commands.py
       - name: Agent role machinery and role discipline
+        # Same split as the step above: the two suites are tree-scoped and run
+        # on every lane; `check-role-discipline.py --base/--head` is DIFF-scoped
+        # and is skipped where there is no range.
+        #
+        # It would not have aborted on its own -- passing `--base ""` makes
+        # `commits_in_range` fall back to the head commit alone
+        # (scripts/check-role-discipline.py:328-333), so the baseline lane would
+        # have reported a PASS covering one commit while looking like it covered
+        # a range. Skipping loudly beats passing vacuously. Recorded as risk 10
+        # in .agents/specs/main-verifiability.md.
         env:
           EVENT_NAME: ${{ github.event_name }}
           PR_BASE: ${{ github.event.pull_request.base.sha }}
@@ -176,8 +245,13 @@ jobs:
             base="$PUSH_BASE"
             head="$PUSH_HEAD"
           fi
-          python3 scripts/check-role-discipline.py \
-            --base "$base" --head "$head" "${pending_args[@]}"
+          if [ -z "$base" ]; then
+            echo "no diff range on the $EVENT_NAME lane: role discipline is"
+            echo "diff-scoped and every commit's own push already ran it."
+          else
+            python3 scripts/check-role-discipline.py \
+              --base "$base" --head "$head" "${pending_args[@]}"
+          fi
           python3 tests/scripts/test_agent_role.py
           python3 tests/scripts/test_agent_onboard.py
       - name: Claim view, helper queue and PR reviewability
@@ -216,6 +290,13 @@ jobs:
     # DIFF-scoped: deliberately carries NO concurrency group. Its verdict covers
     # this push's own `before..sha` range and nothing re-covers that range later,
     # so cancelling it would silently exempt those commits from the gate.
+    #
+    # Not on the baseline lane (#274): `schedule` and `workflow_dispatch`
+    # payloads have no `github.event.before`, so the range below would be empty
+    # and the gate vacuous. The push that landed each commit already ran this,
+    # which is exactly why it carries no group. The baseline's subject is the
+    # TREE at a SHA, and `scripts/main-baseline.py` names the jobs it covered.
+    if: github.event_name != 'schedule' && github.event_name != 'workflow_dispatch'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -252,6 +333,10 @@ jobs:
     # DIFF-scoped: deliberately carries NO concurrency group, same reasoning as
     # documentation-checkpoint above. It walks `before..sha` per push, so a
     # cancelled run means those commits are never checked for the trailer.
+    #
+    # Not on the baseline lane (#274), same reasoning: no `before` on a
+    # `schedule` or `workflow_dispatch` payload, so the walk has no range.
+    if: github.event_name != 'schedule' && github.event_name != 'workflow_dispatch'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -320,8 +405,8 @@ jobs:
     #
     # Tree-scoped (drives cmake at HEAD): cancellable.
     concurrency:
-      group: ci-cuda-arch-features-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-cuda-arch-features-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -346,8 +431,8 @@ jobs:
     permissions:
       contents: read
     concurrency:
-      group: ci-cuda-fat-build-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-cuda-fat-build-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     timeout-minutes: 180
     container: nvidia/cuda:13.3.0-devel-ubuntu24.04
@@ -418,8 +503,8 @@ jobs:
     #
     # Tree-scoped (checks HEAD's artifact): cancellable.
     concurrency:
-      group: ci-vulkan-spirv-freshness-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-vulkan-spirv-freshness-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -452,8 +537,8 @@ jobs:
     #
     # Tree-scoped: cancellable.
     concurrency:
-      group: ci-build-test-vulkan-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-build-test-vulkan-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -494,8 +579,8 @@ jobs:
     # Tree-scoped: the ratchet compares HEAD against the committed baseline, so
     # only the newest push matters. Cancellable.
     concurrency:
-      group: ci-device-leakage-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-device-leakage-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -506,8 +591,8 @@ jobs:
   build-test-cpu:
     # Tree-scoped and the most expensive lane in the workflow. Cancellable.
     concurrency:
-      group: ci-build-test-cpu-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-build-test-cpu-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -557,8 +642,8 @@ jobs:
     # every available rich tier; qemu's Cortex-A53 model supplies the required
     # feature-poor execution and illegal-tier refusal gate.
     concurrency:
-      group: ci-build-test-cpu-arm64-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-build-test-cpu-arm64-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-24.04-arm
     steps:
       - uses: actions/checkout@v4
@@ -641,8 +726,8 @@ jobs:
     # the two mutually-exclusive sanitizer legs of the SAME run would land in one
     # group and cancel each other, leaving only whichever started second.
     concurrency:
-      group: ci-sanitize-cpu-${{ matrix.lane }}-${{ github.ref }}-${{ github.repository }}
-      cancel-in-progress: true
+      group: ci-sanitize-cpu-${{ matrix.lane }}-${{ github.event_name }}-${{ github.ref }}-${{ github.repository }}
+      cancel-in-progress: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -665,3 +750,47 @@ jobs:
           # distinguish that cache from a leak and can see use-after-free.
           VT_POOL_BYPASS: "1"
         run: ctest --test-dir build-sanitize --output-on-failure
+  baseline-summary:
+    # THE PUBLISHED VERDICT (issue #274, spec .agents/specs/main-verifiability.md).
+    #
+    # Baseline lane ONLY. It can never run on a `pull_request` or a `push`, so it
+    # blocks no contributor and changes nothing about what a PR must pass.
+    #
+    # It fails when any covered job is red, `sanitize-cpu` INCLUDED. That job is
+    # `continue-on-error: true` for the push/PR lanes and stays that way -- its
+    # removal is the closing step of the hardening row, not this one. What this
+    # job refuses is a BASELINE that reports green while a job it covered is red,
+    # which would rebuild the very defect #274 was filed about.
+    #
+    # It reads per-job conclusions from the Actions API for its own run, NOT from
+    # `needs.<job>.result`: `continue-on-error` makes that context report
+    # `success` for a failed job, which is the trap the whole row is about. Run
+    # 31448896841 is the live proof -- run conclusion `success`, both sanitizer
+    # lanes `failure`.
+    #
+    # On the day this lands the first run is EXPECTED RED, on the six sanitizer
+    # failures of #274 finding #1 / #301. That is the correct first verdict. A
+    # baseline that hid them would certify nothing.
+    if: always() && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
+    needs:
+      - agent-record
+      - cuda-arch-features
+      - cuda-fat-build
+      - vulkan-spirv-freshness
+      - build-test-vulkan
+      - device-leakage
+      - build-test-cpu
+      - build-test-cpu-arm64
+      - sanitize-cpu
+    permissions:
+      contents: read
+      actions: read
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Publish the per-job baseline verdict for this SHA
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          python3 scripts/main-baseline.py \
+            --run-id "${{ github.run_id }}" --emit-summary

scripts/agent-preflight.sh

@@ -111,6 +111,7 @@ SUITES=(
   test_check_test_registration
   test_audit_live_rows
   test_check_gate_commands
+  test_main_baseline
 )
 
 failed=()