From 23a0d84cb3f97177779d13c89e258108ec781576 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 21:18:10 +0000 Subject: [PATCH 01/12] chore(policy): reserve internal-policy-optimization-1 Reserve the helper row before the policy audit begins. No repository content changes in this commit. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:GPT-5 [Codex] From 5107624d2e5594b1a50f841be6ed4fa21c1665e5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 21:48:08 +0000 Subject: [PATCH 02/12] docs(policy): design strict agent-policy consolidation Define the accepted contract-first redesign for policy authority, helper claims, live PR enforcement, expiring waivers, prompt contracts, consolidation, and bounded cutover. Keep the full structured-record migration as a separate follow-up PR. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Codex:GPT-5 [Codex] --- ...-07-internal-policy-optimization-design.md | 594 ++++++++++++++++++ 1 file changed, 594 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md diff --git a/docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md b/docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md new file mode 100644 index 000000000..38818af82 --- /dev/null +++ b/docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md @@ -0,0 +1,594 @@ +# Internal policy optimization 1 + +Status: accepted design for PR #128. This specification covers the policy and +agent-prompt layer only. Converting project records into structured sources is +a separate follow-up campaign. + +## Goal + +Make the repository's agent policy smaller, unambiguous, agent-agnostic, and +mechanically enforced. A green gate must mean either that the applicable rule +was satisfied or that a narrow, attributable, unexpired waiver exists. It must +never mean that prose was present, a remote snapshot was merely recent, or a +failure was silently treated as absence. + +The root `AGENTS.md` remains the agent-facing entrypoint and index. It stops +being a second full copy of the policy. + +## Scope + +This PR owns: + +- the authority and precedence of repository policy; +- a compact, structured policy-rule registry; +- the root agent index and the active policy/procedure file set; +- role, worktree, branch, task, PR, trailer, and waiver enforcement; +- the implementer, reviewer, and operator prompt contracts; +- bounded migration from the current protocol; +- mutation tests proving that the new gates fail for the defects they name. + +This PR does not convert matrices, state, ledgers, benchmark records, or public +status pages into structured data. That is the explicitly deferred +"structured records" PR. This PR may archive obsolete policy rationale and +legacy claim narratives, but it must not reinterpret feature evidence or +rewrite matrix lifecycle state. + +## Why the current system is not strict + +The current preflight is green, but several binding claims are false or +contradict one another. + +### Authority and context + +- `AGENTS.md:3-4` says to read `AGENTS.md` first, while `AGENTS.md:39` and + `.agents/workflow.md:59` say `NOW.md` is first. There is no single boot order. +- `AGENTS.md:28-34` requires a full interactive environment/preferences setup, + while `.agents/workflow.md:40-44` requires just-in-time, one-value onboarding + and explicitly forbids walking the whole template. +- `AGENTS.md:348` requires a state entry every working session, while + `AGENTS.md:168-171` exempts routine review, Git housekeeping, and protocol + discussion. +- `.agents/coordination.md:72-80` still defines hand-maintained claims, while + `.agents/specs/operator-helper-protocol.md:175-187` says PR-derived claims + replace that table and must never be hand-edited. +- `.agents/coordination.md:84-87` still requires `README.md` at every checkpoint, + contradicting the public-document contract at `AGENTS.md:175-202` and + `.agents/workflow.md:141-158`. +- `AGENTS.md:88-90` calls 16/16 token identity an exceptionless precondition, + while `.agents/NOW.md` explicitly recognizes a ratified distributional gate. + Exceptions exist in practice but are not represented as structured policy. +- `.agents/specs/operator-helper-protocol.md:3-5` says the design is not yet + enforced and `AGENTS.md` is untouched, while its own work table at lines + 256-261 says W0-W5 landed and the current `AGENTS.md` contains the protocol. + +The root `AGENTS.md` is 23,181 bytes. It is below Codex's default 32 KiB +combined project-instruction limit by itself, but leaves little room for global +or nested instructions and points to more than 70 KiB of additional binding +prose. Linked files are not automatically part of an agent's instruction +chain. Current official guidance recommends a concise, practical `AGENTS.md`, +task prompts with goal/context/constraints/completion criteria, and reusable +workflows outside the root instruction file: + +- +- + +The design remains agent-agnostic; these sources validate the general context +and prompt principles rather than defining tool-specific policy. + +### Enforcement gaps + +- `scripts/agent-role.py:220-291` writes a marker for a helper. It does not + create a worktree, create the exact branch, validate a known task, push a + reservation, or open a PR. It accepts helper mode in the primary checkout. +- `scripts/claim-view.py:42-45,113-145` checks only that a committed PR snapshot + is syntactically valid, references known matrix rows, and is no more than 14 + days old. On 2026-08-07 its view said `_none_` while six live `row/*` PRs + existed; preflight still passed. +- `scripts/ready-for-helper.py:50-96` does not test the documented unmerged-PR + dependency condition. It tests file existence, not that the spec is + committed; approximates runnable gates through an anchor heuristic; and + treats the absence of hardware words as proof of CPU gateability. +- `.github/workflows/ci.yml:180-214` searches for the string + `FOLLOWING_AGENTS_PROTOCOL` anywhere in a commit message. It is not a Git + trailer and the gate does not validate the `Assisted-by:` contract. +- `scripts/check-role-discipline.py:43-60` and + `scripts/check-pr-size.py:24-25` exempt broad directory classes. Material + feature or enforcement logic under `scripts/`, `docs/`, or `.agents/` can + bypass the path and size rules. +- `scripts/check-protocol-consistency.py` pins a few literal phrases but does not + check the semantics above. Contradictory prose can contain every required + phrase and remain green. +- The new `internal-policy-optimization-1` helper claim is itself unknown to the + matrix-row registry. Refreshing the current claim view would reject the PR + that repairs the claim system. + +### Live-context growth + +The tracked `.agents/` tree is about 131,000 lines. `state.md` is about 41,000 +lines, `benchmark-record.md` about 15,000, and `coordination.md` about 2,000. +This conflicts with the stated goal that the active `.agents/` surface contains +only current context. `NOW.md` records that line-number evidence anchors now +block record-era rollover. The full structural repair belongs to the follow-up +PR; this PR must avoid adding another narrative policy log and must archive +superseded policy material. + +## Authority model + +### Root index + +`AGENTS.md` remains the automatically discovered, agent-facing index. It is +hand-curated outside delimited generated blocks and has a hard budget of 12 KiB. +It contains only: + +1. policy precedence and the exact boot sequence; +2. the small T0 rule subset required under context pressure; +3. exact role/preflight/ready commands; +4. the task-based map to current procedures and records; +5. the commit and PR handoff summary. + +The T0 block is rendered from the structured policy registry. Manual edits to +that block fail `--check`. + +### Canonical rule registry + +`.agents/policy.csv` is the sole repository-policy authority. It is an RFC 4180 +CSV file with this header: + +```csv +rule_id,scope,trigger,requirement,enforcement,waiver_class,procedure +``` + +Contract: + +- one rule per physical line; multiline fields are forbidden; +- `rule_id` is stable and unique; +- `scope` and `trigger` say when the rule applies; +- `requirement` is one short, testable statement; +- `enforcement` names one or more real checker entrypoints; +- `waiver_class` is `never`, `expiring`, or `migration-only`; +- `procedure` links to the instructions for satisfying the rule; +- rationale, incidents, dated counts, and attempt history are forbidden; +- maximum 60 rules and 16 KiB. Growth beyond either budget fails and requires + consolidation or moving procedural detail. + +The registry is machine-readable and human-reviewable. There is no second +generated full policy-reference file. + +### Procedures + +The active cross-project procedure set becomes: + +- `.agents/workflow.md` — session, role, task, PR, and handoff procedure; +- `.agents/verification.md` — correctness, performance, oracle, trace, and + reproduction procedures, consolidating `gates.md` and + `benchmark-protocol.md`; +- `.agents/porting.md` — upstream grounding, design discipline, test porting, + and deviation procedure, consolidating `discipline.md` and `test-porting.md`. + +Procedures explain how to satisfy a rule. A normative paragraph must cite its +`POL-*` rule ID and cannot create a new independent obligation. Specialized +technical documents such as parity-lever, upstream-sync, backend, environment, +and matrix documents remain task-specific references rather than global policy +authorities. + +The active policy set has explicit size budgets. Proposed initial limits are +12 KiB for `AGENTS.md`, 16 KiB for `policy.csv`, 16 KiB for `workflow.md`, 24 +KiB for `verification.md`, 16 KiB for `porting.md`, and 4 KiB per runtime prompt. + +### Public documentation projections + +The redesign preserves and strengthens the current public-document obligations. +It does not require every document on every change. Each surface has one +purpose-specific policy rule and trigger: + +- `docs/STATUS.md` changes at every feature or iteration checkpoint and states + the current lifecycle, active gap, evidence, and next gate; +- `docs/BENCHMARKS.md` changes at every feature or iteration checkpoint with an + accepted result or explicit `PENDING`, `NOT APPLICABLE`, `FAILED`, or `VOID` + disposition and reproduction action; +- `docs/FEATURES.md` changes whenever a feature, model, backend, or quantization + support surface changes; +- `docs/USAGE.md` changes whenever a user-facing command, flag, API, endpoint, + configuration key, installation step, or runnable workflow changes; +- `README.md` remains the human-friendly landing page and changes only when a + user-visible headline, quickstart, installation path, positioning statement, + or headline benchmark changes; +- `.agents/NOW.md` changes when live claims, the current gate, next actions, or + a `state.md` append changes. A policy-only design that moves none of those + does not manufacture a NOW entry. + +The repository has no `NEWS` or changelog surface at this design point. The +word "news" is interpreted as `.agents/NOW.md`; adding a release-news surface +would be a separate product/documentation decision. + +The policy registry owns these triggers. A documentation checker classifies +the semantic change class from explicit path/rule mappings, not from the mere +presence of any file under `.agents/specs/`. This fixes the observed failure +where staging this governance design falsely demanded public feature and +benchmark updates. + +The public files are projections, not duplicate narratives: README links to +USAGE, STATUS, FEATURES, and BENCHMARKS; each detail lives in exactly one of +those destinations. + +### Precedence and boot order + +External system, developer, and user instructions retain their platform-defined +precedence. Within the repository: + +1. `AGENTS.md` selects the applicable rule IDs and references; +2. `policy.csv` defines the rule; +3. the linked procedure explains how to comply; +4. task records and evidence describe the current instance but do not amend the + rule. + +The one boot sequence is: + +1. resolve the existing worktree role; if undeclared, ask what work is being + done and materialize the answer; +2. read `NOW.md`; +3. read shared developer preferences and only the environment values needed by + the current gate; +4. read the claimed task/row and only its linked procedures/evidence; +5. run preflight before edits and before every commit/push transition. + +Untracked developer configuration must resolve from one shared location across +linked worktrees. A helper must not silently lose the primary checkout's +preferences or `.env`, and it must never infer another developer's values. + +## Consolidation and archival + +The following active policy sources are retired or folded: + +- `.agents/directives.md` -> rules in `policy.csv`, procedures in the three + procedure documents, historical rationale in `completed/`; +- `.agents/ai-coding-assistants.md` -> contribution rules in `policy.csv` and + commit procedure in `workflow.md`; +- `.agents/specs/operator-helper-protocol.md` -> current procedure in + `workflow.md`, design/incidents archived under `completed/`; +- `.agents/gates.md` + `.agents/benchmark-protocol.md` -> + `.agents/verification.md`; +- `.agents/discipline.md` + `.agents/test-porting.md` -> `.agents/porting.md`; +- the policy/procedure preamble of `.agents/coordination.md` -> `workflow.md`. + +Legacy/manual claim narratives needed to interpret pre-cutover owners move to +an era-stamped completed record. New claims are PR-derived. Feature lifecycle +rows, append-only evidence, and public projections are not structurally migrated +in this PR. + +Every move repairs repository links in the same change. No evidence is deleted. + +## Role and claim enforcement + +### Known tasks + +Feature/model/kernel/backend IDs continue to come from their matrices. A small +`.agents/governance-tasks.csv` admits non-feature tasks such as this policy PR. +It uses stable IDs and contains only current governance work. It is not the +general structured-record system deferred to the follow-up PR. + +### Helper start + +The supported helper entrypoint is a transactional orchestration command rather +than a marker-only `claim helper`: + +```text +scripts/agent-role.py start-helper --task [--headless] [--open-pr] +``` + +It must: + +1. validate the task against a matrix or governance registry; +2. refuse the primary checkout as the helper's final workspace; +3. create a linked worktree and exact `row/` branch from the selected base; +4. write the role marker in the linked worktree's Git directory; +5. when remote operations are authorized, push a reservation commit and open a + draft PR before implementation; +6. return the worktree path, branch, task, role, PR, and verification state. + +If a remote step fails, the command must either roll back safe local reservation +state or retain it as an explicit `INCOMPLETE_RESERVATION`. It must never print +success for a marker-only helper. Without remote authority the session may +prepare locally, but it cannot claim `PR_VERIFIED` or mark the task ready. + +Plain role resolution refuses: + +- helper markers in the primary checkout; +- branch/task mismatch; +- unknown tasks; +- missing or conflicting operator locks; +- a supposedly ready helper with no matching open PR. + +### Live claims + +The committed timestamp snapshot and its 14-day TTL are removed. Local, +network-independent preflight validates the marker/worktree/branch/task tuple. +The ready/integration gate and CI query live PR state and reject: + +- two open PRs claiming one task; +- an unknown task; +- a `row/*` PR whose head does not exactly match its task; +- a task marked ready without a draft/ready PR; +- a merged/closed PR still represented as live. + +Remote unavailability is `REMOTE_UNVERIFIED`, not "no claim" and not green for +integration. + +### Ready-for-helper + +The helper queue must actually prove its advertised conditions: + +- the spec is tracked and reachable from the chosen base commit; +- the gate command exists, is executable, and has a nonzero failure mutation; +- CPU gateability is explicit, otherwise exact hardware is declared; +- dependencies are parsed and merged/satisfied; +- live PR state contains no claim; +- the lifecycle state is pickable. + +Absence of a hardware keyword is not evidence of CPU gateability. + +## PR, path, and commit enforcement + +All new feature, policy, checker, documentation, and record changes arrive +through a PR. The operator integrates; it does not directly push new work to +`main`. Emergency repairs use an exact, expiring waiver. + +Path classification must not exempt whole mutable trees. Policy/checker work +under `scripts/`, `tests/scripts/`, `.agents/`, `docs/`, or `.github/` remains +reviewed and size-bounded. Generated evidence and append-only records may use +separate budgets, but they are explicit classes rather than blanket exemptions. + +New commits use real Git trailers: + +```text +Following-Agents-Protocol: true +AI-Assisted: true +Assisted-by: Codex:GPT-5 [Codex] +``` + +`AI-Assisted: false` omits `Assisted-by`. When true, at least one syntactically +valid `Assisted-by` is required. An AI agent never adds its own `Signed-off-by` +or `Co-authored-by`. The checker uses `git interpret-trailers` semantics rather +than substring search. + +The old magic line is accepted only for commits before the cutover. PR #128 is +squashed or amended so its final range satisfies the new contract. + +## Waivers and policy changes + +`.agents/waivers.csv` has this schema: + +```csv +waiver_id,rule_id,scope,owner,reason,evidence,expires +``` + +Rules: + +- `rule_id` must exist and permit the requested waiver class; +- `scope` identifies an exact task, PR, commit, path, gate, or hardware leg; +- `owner`, `reason`, and `evidence` are nonempty; +- `expires` is an ISO date and must be in the future; +- wildcard or repository-wide scope is forbidden; +- expired, unused, duplicated, or unknown waivers fail; +- migration-only waivers cannot be created after the cutover window; +- `waiver_class=never` cannot be waived. + +Permanent `REPORT-ONLY` enforcement is removed. Diagnostics may exist, but a +rule is either not applicable, satisfied, waived, pending a named external gate, +or failing. + +"Never weaken a checker" becomes a testable change-control rule. A checker +semantic change requires: + +1. a dedicated policy PR/task; +2. the affected rule IDs; +3. a before/after mutation that demonstrates the old and intended behavior; +4. synchronized registry and procedure updates; +5. no unrelated gate relaxation. + +This permits legitimate checker repairs without allowing a red transition to be +made green by assertion deletion or scope widening. + +## Agent-agnostic prompt contracts + +The runtime prompts remain plain Markdown and tool-neutral. Each is versioned, +under 4 KiB, and contains four sections: required inputs, role method, required +output, and stop conditions. Optional tool adapters may explain invocation but +cannot alter policy or acceptance criteria. + +### Common task envelope + +Every dispatched task supplies: + +- `Goal` — the outcome, not a preselected implementation; +- `Context` — exact task/row, files, sources, baseline, and dependencies; +- `Constraints` — boundaries, allowed actions, policy IDs, and prohibitions; +- `Done when` — observable acceptance criteria and exact gates; +- `Required evidence` — commands, mutations, traces, or comparisons owed; +- `Authority` — remote, hardware, dependency, download, and service permissions. + +A missing required field is `NEEDS_CONTEXT`; the agent does not guess a +materially different task. + +### Implementer + +The implementer uses test-first development where applicable, makes the minimum +in-scope change, runs focused gates, and then runs the required project gate. A +material disagreement with the brief returns `NEEDS_DECISION`; it does not +silently expand scope. Baseline red gates are reported and block ready status +unless covered by an applicable waiver. + +Required output: + +- status: `COMPLETE`, `BLOCKED`, `NEEDS_CONTEXT`, or `NEEDS_DECISION`; +- summary and changed files; +- commands and exit results; +- negative/mutation evidence for new tests; +- deviations and applicable waiver IDs; +- unresolved risks and omitted gates; +- commit SHA when committed. + +### Reviewer + +Review has three independent lenses: + +1. static contract review for missing requirements, architecture, security, + unsupported claims, absent tests, and scope drift; +2. targeted mutation/negative tests for the important changed claims; +3. relevant full-gate verification once focused checks are complete. + +Mutation supplements static review; it never replaces it. Each mutation runs +the smallest relevant test in a scratch copy. The reviewer does not mutate the +reviewed worktree and does not repair findings. + +Output starts with findings ordered by severity. Every finding includes +`file:line`, evidence, the violated requirement or policy ID, and required +remediation. There is no mandatory praise preamble and no dated anecdote in the +runtime prompt. + +### Operator + +The operator contract receives the task, implementer report, reviewer report, +and exact integration gates. It verifies rather than trusts reports, returns +findings to an implementer, and dispositions the PR according to policy. It +does not implement feature fixes inside the coordinating context. + +### Prompt enforcement + +`scripts/check-prompt-contract.py` parses structural fields and role boundaries. +It must not pin incidental sentences. Mutation tests delete each required +section, remove a role boundary, and weaken a required output to prove the gate +fails semantically. + +## Preflight and CI states + +One command must not blur local and remote evidence. + +- Local preflight: role/task/worktree/branch consistency, policy schema, + generated blocks, local gates, staged change, and committed range. No network. +- Ready gate: all local checks plus live PR identity, collision, and current CI + requirements. Remote failure is explicit and non-green. +- Integration gate: ready gate plus review disposition, commit trailers, + waivers, and base freshness. +- CI: re-runs repository checks from a clean checkout and queries the current PR + event rather than trusting committed remote snapshots. + +Hooks remain convenience backstops. They are never described as proof because +`--no-verify` can bypass them. + +## Migration and cutover + +1. Inventory every active normative statement and map it to a `POL-*` rule, + procedure, archived rationale, or deletion as duplicate. +2. Land `policy.csv`, schemas, render/check tooling, and mutation tests without + changing project feature claims. +3. Render the compact `AGENTS.md`; consolidate procedures and repair links. +4. Land prompt contracts and their checker. +5. Land role/task/claim/PR/trailer/waiver enforcement. +6. Register the governance task for PR #128 and make the PR pass its own rules. +7. Enumerate existing open-PR migration waivers with short expirations. +8. Declare one cutover commit. Historical commits remain grandfathered; every + new commit and PR is strict. +9. Archive superseded policy and pre-cutover claim rationale. + +No permanent legacy switch remains after cutover. An open PR that does not +finish within its migration waiver must rebase and comply. + +## Verification and mutation matrix + +The new test suite must prove failure for at least: + +| Surface | Required red mutation | +|---|---| +| policy CSV | missing column, duplicate/unknown ID, multiline cell, bad enum, unknown checker, over size/count budget | +| AGENTS index | stale generated T0, unknown link/rule, over 12 KiB, contradictory boot-order marker | +| procedure authority | normative paragraph without exactly one applicable rule ID, unknown rule ID, stale procedure back-reference | +| public documentation | missing STATUS/BENCHMARKS checkpoint, missing FEATURES support update, missing USAGE change, README churn without a headline trigger, stale NOW after live-state change, governance-only design misclassified as feature work | +| helper role | primary checkout, wrong branch, unknown task, marker-only success, missing worktree | +| live claim | missing PR, duplicate PR, wrong head, closed PR treated live, remote failure treated absent | +| helper queue | uncommitted spec, no failing gate mutation, undeclared hardware, unmet dependency, reserved row | +| PR/path size | oversized policy/checker change, material script change misclassified as exempt | +| trailers | substring-only legacy tag, missing protocol trailer, malformed AI declaration, missing assistance attribution | +| waivers | unknown/non-waivable rule, wildcard scope, expired/unused/duplicate waiver, missing evidence | +| prompts | missing envelope field, missing output field, removed stop condition, reviewer mutation replacing static review | +| migration | post-cutover legacy commit/PR, expired migration waiver, new report-only mode | + +Integration tests use throwaway Git repositories and fixture PR payloads. They +must not mutate the real checkout or require live GitHub access. One authorized +online smoke test verifies the GitHub query against PR #128 before the PR is +marked ready. + +Existing record, document, source, and unit gates remain green throughout. A +policy-only change does not fabricate a feature checkpoint or benchmark result. + +## Work breakdown for PR #128 + +The implementation plan may split commits, but each commit must satisfy the +same-change obligations applicable after its point in the migration. + +1. Policy inventory and CSV schema. +2. Policy parser, renderer, authority checker, and mutation tests. +3. Compact root index and procedure consolidation/archive. +4. Prompt contracts and semantic checker. +5. Governance task registry and helper materialization. +6. Live PR claim and ready-queue enforcement. +7. Trailer, path-classification, PR-size, and waiver enforcement. +8. Purpose-specific public-document trigger enforcement. +9. Preflight/CI integration and cutover migration. +10. Full self-hosting verification, link audit, final compactness audit, and PR + evidence. + +PR #128 remains one reviewable PR as requested. Reorder commits or trim +incidental cleanup if necessary, but do not split these acceptance criteria into +a series and do not leave prose claiming enforcement before its gate lands. + +## Risks and decisions + +- **Bootstrap recursion:** the new rules govern their own implementation. + Resolve with ordered commits and a single explicit cutover, not a permanent + bypass. +- **Remote dependence:** local work must remain possible, but remote absence + cannot prove a claim. Keep local and ready states separate. +- **CSV abuse:** long quoted prose would recreate the current problem. Forbid + multiline fields and enforce rule/file budgets. +- **False deduplication:** similar procedures may encode distinct constraints. + Inventory by stable rule semantics before archiving text. +- **Historical link breakage:** archive with link repair and link checking in + the same change. +- **Cross-agent capability differences:** the core prompt specifies outcomes + and evidence, while adapters only describe tool invocation. +- **Waiver normalization:** a waiver is visible debt, not success. Expiry and + exact scope are mandatory. + +## Deferred structured-record PR + +The follow-up PR will separately design and migrate structured sources for: + +- roadmap and area matrices; +- claims and dependencies; +- state and benchmark events; +- parity-ledger evidence with stable IDs instead of line anchors; +- `NOW.md` and public Markdown projections; +- era rollover and archival automation. + +Its migration must preserve every evidence item, replace fragile line-number +anchors with stable IDs, generate human-readable projections, and prove +byte/semantic equivalence before deleting any legacy source. None of those +record-format decisions are smuggled into PR #128. + +## Acceptance criteria + +PR #128 is complete when: + +- one compact policy CSV is authoritative and all normative statements in the + designated active policy surfaces map to it; +- `AGENTS.md` remains a compact index under its budget; +- active policy/procedure files are fewer and duplicates are archived; +- role, claim, PR, trailer, waiver, and prompt promises are executable; +- STATUS, BENCHMARKS, FEATURES, USAGE, README, and NOW update exactly on their + defined triggers without duplicating one another; +- all new gates have red-first mutation evidence; +- existing project gates remain green; +- current open PR migration is explicit and expiring; +- PR #128 passes the new local, ready, and integration gates on its own head; +- the structured-record migration remains a named, separate follow-up. From f8510465eab8cc4de9ddb64602e80b6ec08bd669 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 22:14:44 +0000 Subject: [PATCH 03/12] policy: add authoritative rule registry Add bounded CSV policy and waiver contracts with a shared parser, CLI validator, and red-first mutation coverage for exact exception scopes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/policy.csv | 58 +++++ .agents/waivers.csv | 1 + scripts/check-policy.py | 35 +++ scripts/policy_contract.py | 320 +++++++++++++++++++++++++ tests/scripts/test_policy_contract.py | 324 ++++++++++++++++++++++++++ 5 files changed, 738 insertions(+) create mode 100644 .agents/policy.csv create mode 100644 .agents/waivers.csv create mode 100644 scripts/check-policy.py create mode 100644 scripts/policy_contract.py create mode 100644 tests/scripts/test_policy_contract.py diff --git a/.agents/policy.csv b/.agents/policy.csv new file mode 100644 index 000000000..a91884f71 --- /dev/null +++ b/.agents/policy.csv @@ -0,0 +1,58 @@ +rule_id,scope,trigger,requirement,enforcement,waiver_class,procedure +POL-AUTH-REGISTRY,repository,any policy decision,Use policy.csv as the sole repository-policy authority.,scripts/check-policy.py,never,.agents/directives.md +POL-AUTH-PRECEDENCE,repository,repository instructions conflict,Apply AGENTS then the selected policy rule then its procedure then task evidence.,scripts/check-policy.py,never,.agents/workflow.md +POL-BOOT-ROLE,session,before task work,Resolve or declare the worktree role before reading task state or editing.,scripts/check-role-discipline.py; scripts/check-policy.py,never,.agents/workflow.md +POL-BOOT-NOW,session,after role resolution,Read NOW.md before deeper task records.,scripts/check-now-current.py; scripts/check-policy.py,never,.agents/workflow.md +POL-BOOT-TASK,session,after NOW,Read only the claimed task and its linked procedures and evidence.,scripts/check-policy.py,expiring,.agents/workflow.md +POL-CONFIG-SHARED,worktree,configuration is needed,Resolve developer preferences and env values from the shared checkout configuration.,scripts/check-policy.py,expiring,.agents/workflow.md +POL-CONFIG-JIT,session,an environment value is required,Request only the missing value needed by the current gate and never infer another developer's setup.,scripts/check-policy.py,never,.agents/workflow.md +POL-CONFIG-NO-WEAKEN,developer preferences,preferences alter operations,Preferences must not weaken correctness evidence attribution lifecycle testing or documentation rules.,scripts/check-policy.py,never,.agents/workflow.md +POL-ROLE-DECLARED,write session,before edits,Declare operator helper or read-only and keep the role state consistent with the worktree.,scripts/check-role-discipline.py; scripts/check-policy.py,never,.agents/specs/operator-helper-protocol.md +POL-HELPER-TASK,helper,start helper,Use a known matrix or governance task ID.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/specs/operator-helper-protocol.md +POL-HELPER-WORKTREE,helper,start helper,Work only in a linked worktree on the exact row task branch.,scripts/check-role-discipline.py; scripts/check-policy.py,never,.agents/specs/operator-helper-protocol.md +POL-HELPER-PR,helper,before implementation,Create a draft PR as the remote claim when remote operations are authorized.,scripts/check-policy.py,expiring,.agents/specs/operator-helper-protocol.md +POL-REMOTE-UNKNOWN,remote gate,remote state cannot be queried,Report REMOTE_UNVERIFIED and never treat unknown remote state as absence or success.,scripts/check-policy.py,never,.agents/specs/operator-helper-protocol.md +POL-OPERATOR-BOUNDARY,operator,feature work,Delegate implementation and keep main integration and GPU ownership in the operator role.,scripts/check-role-discipline.py; scripts/check-policy.py,expiring,.agents/workflow.md +POL-OPERATOR-VERIFY,operator,implementation returns,Run the claimed gate instead of trusting an implementer report.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md +POL-REVIEW-FRESH,review,implementation is ready for review,Use a fresh reviewer that performs static review and targeted scratch mutation.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md +POL-REVIEW-NO-REPAIR,review,a finding is found,Return findings to a fresh implementer and do not repair them in the coordinating session.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md +POL-PR-DISPOSITION,operator,a PR is verified or obsolete,Merge a verified PR in-session or close an obsolete PR with the reason recorded.,scripts/check-policy.py,expiring,.agents/workflow.md +POL-MIRROR-VLLM,feature port,vLLM defines behavior,Mirror every applicable vLLM mode instead of inventing product behavior.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/directives.md +POL-GROUND-CHAIN,parity investigation,comparing implementation,Ground conclusions in vLLM and the complete runtime dependency and generated-kernel chain.,scripts/check-policy.py,never,.agents/parity-lever-protocol.md +POL-TRACE-SAME-TOOL,performance comparison,before comparing throughput,Trace vLLM and this project on the same workload with the same tracing tool.,scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-GEMV-CONTRACT,GEMM or GEMV parity claim,claiming invocation parity,Match output dtype compute and scale type entry point algorithm policy and resolved template dtypes.,scripts/check-gemv-invocation-consistency.py; scripts/check-policy.py,never,.agents/parity-lever-protocol.md +POL-SEAM-FUSION,model implementation,model has fusible chains,Route through vt::FusedChain or declare an exact tracked exception.,scripts/check-fusion-consistency.py; scripts/check-policy.py,expiring,.agents/discipline.md +POL-SEAM-MERGED-GEMM,model implementation,model has mergeable MLP projections,Route through the shared merged-GEMM family or declare an exact tracked exception.,scripts/check-fusion-consistency.py; scripts/check-policy.py,expiring,.agents/discipline.md +POL-SEAM-RUNNER,model implementation,model supports decode,Route through the shared decode runner and on-device sampling or declare an exact tracked exception.,scripts/check-runner-routing-consistency.py; scripts/check-policy.py,expiring,.agents/discipline.md +POL-ORACLE,correctness or performance gate,running a comparison,Use the pinned oracle and an identical workload on both sides.,scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-PERF-EVERY-AXIS,performance gate,accepting a checkpoint,Match or beat the oracle on every required throughput latency and memory axis.,scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-CORRECTNESS-GATE,correctness gate,accepting a checkpoint,Pass the declared token-exact or ratified distributional correctness gate before performance acceptance.,scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-REPRODUCE,performance gate,accepting a measured result,Record the recipe and reproduce the result on an idle box with same-binary comparisons.,scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-NO-CEILING,performance investigation,a gap remains,Keep the gap open and identify the next concrete implementation difference.,scripts/check-policy.py,never,.agents/parity-lever-protocol.md +POL-PORT-TESTS,upstream port,porting code,Port the applicable upstream tests in the same change.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/discipline.md +POL-SPIKE-FIRST,task lifecycle,before READY or ACTIVE,Commit a complete spike spec before a row enters READY or ACTIVE.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/workflow.md +POL-EVIDENCE-PRESERVE,records,compacting or retiring records,Move evidence without deleting it.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/directives.md +POL-ROW-SYNC,checkpoint,feature lifecycle changes,Update the roadmap row and owning matrix row in the same change.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/workflow.md +POL-STATE-ORDER,state log,appending state,Append below the enforced marker in chronological anchor order.,scripts/check-state-order.py; scripts/check-policy.py,never,.agents/workflow.md +POL-NOW-COUPLING,live state,state is appended or live claims gates or actions change,Refresh NOW.md in the same change.,scripts/check-now-current.py; scripts/check-policy.py,never,.agents/workflow.md +POL-DOC-STATUS,public status,feature or iteration checkpoint,Update docs/STATUS.md with lifecycle gap evidence and next gate.,scripts/check-doc-checkpoint.py; scripts/check-policy.py,never,.agents/directives.md +POL-DOC-BENCHMARKS,public benchmarks,feature or iteration checkpoint,Update docs/BENCHMARKS.md with an accepted or explicit pending failed not-applicable or void result.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-DOC-FEATURES,public features,feature model backend or quantization support changes,Update the keyed docs/FEATURES.md row in place.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/directives.md +POL-DOC-USAGE,public usage,user commands flags APIs endpoints config install or runnable workflows change,Update docs/USAGE.md with the user-facing behavior.,scripts/check-policy.py,never,.agents/workflow.md +POL-DOC-README,landing page,a headline quickstart install positioning or headline benchmark changes,Update README.md only for that landing-page change.,scripts/check-readme-structure.py; scripts/check-policy.py,never,.agents/workflow.md +POL-KEYED-MERGE,keyed records,resolving concurrent changes,Take main's keyed record and reapply the edit instead of using a three-way merge.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/workflow.md +POL-TABLE-INVENTORY,project inventory,adding or changing an inventory item,Use a stable ID and record source anchor evidence spike lifecycle and owner.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/discipline.md +POL-PR-REQUIRED,repository change,changing feature policy checker docs or records,Deliver the change through a PR except under an exact expiring emergency waiver.,scripts/check-role-discipline.py; scripts/check-policy.py,expiring,.agents/workflow.md +POL-COMMIT-TRAILERS,commit,creating a post-cutover commit,Use real protocol and AI-assistance Git trailers.,scripts/check-policy.py,migration-only,.agents/ai-coding-assistants.md +POL-AI-ATTRIBUTION,AI-assisted commit,AI assistance was used,Add a syntactically valid Assisted-by trailer and do not add an AI Signed-off-by or Co-authored-by trailer.,scripts/check-policy.py,never,.agents/ai-coding-assistants.md +POL-PREFLIGHT,write transition,before edits commits or pushes,Run the applicable preflight and prevent a failed gate from being followed by a push.,scripts/check-policy.py,never,.agents/workflow.md +POL-CHECKER-CHANGE,checker semantics,changing enforcement behavior,Use a governance task with affected rule IDs and red-before after-mutation evidence plus synchronized policy procedure edits.,scripts/check-policy.py,never,.agents/workflow.md +POL-WAIVER-EXACT,waiver,creating or using an exception,Use one narrow owned evidenced unexpired waiver whose rule permits it.,scripts/check-policy.py,never,.agents/workflow.md +POL-NO-REPORT-ONLY,enforcement,a rule applies,Return satisfied waived pending-external or failing and never permanent report-only success.,scripts/check-policy.py,never,.agents/workflow.md +POL-PATH-CLASSIFICATION,change scope,classifying mutable files,Classify policy checker docs scripts tests and CI explicitly without blanket directory exemptions.,scripts/check-pr-size.py; scripts/check-policy.py,never,.agents/workflow.md +POL-PR-SIZE,pull request,before ready,Keep each explicit path class within its reviewed change budget.,scripts/check-pr-size.py; scripts/check-policy.py,expiring,.agents/workflow.md +POL-READY-SPEC,helper queue,advertising a task as ready,Prove the spec is committed and reachable and the declared gate has a failing mutation.,scripts/check-gate-commands.py; scripts/check-policy.py,never,.agents/coordination.md +POL-READY-HARDWARE,helper queue,advertising a task as ready,Declare CPU gateability or the exact required hardware.,scripts/check-gate-commands.py; scripts/check-policy.py,never,.agents/coordination.md +POL-READY-DEPS,helper queue,advertising a task as ready,Prove parsed dependencies are satisfied and no live claim exists.,scripts/check-policy.py,never,.agents/coordination.md +POL-PROMPT-ENVELOPE,delegated task,dispatching an agent,Supply goal context constraints done-when required-evidence and authority.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md +POL-PROMPT-BOUNDARIES,runtime prompt,changing an agent prompt,Keep prompts tool-neutral versioned bounded and explicit about method output and stop conditions.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md diff --git a/.agents/waivers.csv b/.agents/waivers.csv new file mode 100644 index 000000000..176df228f --- /dev/null +++ b/.agents/waivers.csv @@ -0,0 +1 @@ +waiver_id,rule_id,scope,owner,reason,evidence,expires diff --git a/scripts/check-policy.py b/scripts/check-policy.py new file mode 100644 index 000000000..933b16126 --- /dev/null +++ b/scripts/check-policy.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +"""Validate the authoritative repository policy and waiver registries.""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +from policy_contract import validate_policy + + +ROOT = Path(__file__).resolve().parents[1] + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--schema-only", + action="store_true", + help="validate registries and named paths during the ordered bootstrap", + ) + args = parser.parse_args() + errors = validate_policy(ROOT, schema_only=args.schema_only) + if errors: + print("policy contract FAILED:") + for error in errors: + print(f" - {error}") + return 1 + mode = "schema" if args.schema_only else "full" + print(f"OK: policy contract ({mode})") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/policy_contract.py b/scripts/policy_contract.py new file mode 100644 index 000000000..dcfa9b366 --- /dev/null +++ b/scripts/policy_contract.py @@ -0,0 +1,320 @@ +#!/usr/bin/env python3 +"""Shared parser for the repository policy and waiver registries.""" + +from __future__ import annotations + +import csv +import datetime as dt +import re +from dataclasses import dataclass +from pathlib import Path +from pathlib import PurePosixPath + + +POLICY_HEADER = ( + "rule_id", + "scope", + "trigger", + "requirement", + "enforcement", + "waiver_class", + "procedure", +) +WAIVER_HEADER = ( + "waiver_id", + "rule_id", + "scope", + "owner", + "reason", + "evidence", + "expires", +) +WAIVER_CLASSES = frozenset({"never", "expiring", "migration-only"}) +POLICY_MAX_BYTES = 16 * 1024 +POLICY_MAX_RULES = 60 +RULE_ID = re.compile(r"POL-[A-Z0-9]+(?:-[A-Z0-9]+)*\Z") +WAIVER_ID = re.compile(r"WAIVER-[A-Z0-9]+(?:-[A-Z0-9]+)*\Z") +SCOPE_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*\Z") +BROAD_SCOPE_VALUES = frozenset( + {"all", "any", "everything", "global", "repo", "repository"} +) + + +@dataclass(frozen=True) +class PolicyRule: + rule_id: str + scope: str + trigger: str + requirement: str + enforcement: str + waiver_class: str + procedure: str + + +@dataclass(frozen=True) +class Waiver: + waiver_id: str + rule_id: str + scope: str + owner: str + reason: str + evidence: str + expires: dt.date + + +def _read_rows(path: Path, header: tuple[str, ...]) -> tuple[list[list[str]], list[str]]: + errors: list[str] = [] + if not path.is_file(): + return [], [f"missing registry: {path.relative_to(path.parents[1])}"] + try: + text = path.read_text(encoding="utf-8") + except UnicodeDecodeError as exc: + return [], [f"{path.name}: not valid UTF-8: {exc}"] + + physical = text.splitlines() + if any(not line.strip() for line in physical[1:]): + errors.append(f"{path.name}: blank physical records are forbidden") + + try: + reader = csv.reader(text.splitlines(keepends=True), strict=True) + parsed = list(reader) + except csv.Error as exc: + return [], errors + [f"{path.name}: invalid RFC 4180 CSV: {exc}"] + + if not parsed: + return [], errors + [f"{path.name}: missing header"] + if tuple(parsed[0]) != header: + errors.append( + f"{path.name}: header must be exactly {','.join(header)}" + ) + + rows: list[list[str]] = [] + for number, row in enumerate(parsed[1:], start=2): + if len(row) != len(header): + errors.append( + f"{path.name}:{number}: expected {len(header)} columns; got {len(row)}" + ) + continue + if any("\n" in field or "\r" in field for field in row): + errors.append( + f"{path.name}:{number}: multiline fields are forbidden; one physical record is required" + ) + rows.append(row) + return rows, errors + + +def _repo_relative(value: str) -> bool: + path = PurePosixPath(value) + return ( + bool(value) + and not path.is_absolute() + and ".." not in path.parts + and "\\" not in value + ) + + +def _is_exact_scope(scope: str) -> bool: + """Return whether *scope* identifies one concrete waiver target.""" + + try: + kind, value = scope.split(":", 1) + except ValueError: + return False + if not value or value.casefold() in BROAD_SCOPE_VALUES: + return False + if kind == "pr": + return bool(re.fullmatch(r"[1-9][0-9]*", value)) + if kind == "commit": + return bool(re.fullmatch(r"(?:[0-9A-Fa-f]{40}|[0-9A-Fa-f]{64})", value)) + if kind == "path": + path = PurePosixPath(value) + return ( + _repo_relative(value) + and value != "." + and not value.endswith("/") + and path.as_posix() == value + ) + if kind in {"task", "gate", "hardware"}: + return bool(SCOPE_ID.fullmatch(value)) + return False + + +def _parse_policy(root: Path) -> tuple[dict[str, PolicyRule], list[str]]: + path = root / ".agents/policy.csv" + rows, errors = _read_rows(path, POLICY_HEADER) + if path.is_file() and path.stat().st_size > POLICY_MAX_BYTES: + errors.append( + f"policy.csv exceeds the {POLICY_MAX_BYTES}-byte budget ({path.stat().st_size} bytes)" + ) + if len(rows) > POLICY_MAX_RULES: + errors.append(f"policy.csv has more than {POLICY_MAX_RULES} rules ({len(rows)})") + + rules: dict[str, PolicyRule] = {} + for number, values in enumerate(rows, start=2): + row = dict(zip(POLICY_HEADER, values, strict=True)) + rule_id = row["rule_id"] + if not RULE_ID.fullmatch(rule_id): + errors.append(f"policy.csv:{number}: malformed rule_id {rule_id!r}") + if rule_id in rules: + errors.append(f"policy.csv:{number}: duplicate rule_id {rule_id!r}") + for field in ("scope", "trigger", "requirement", "enforcement", "procedure"): + if not row[field].strip(): + errors.append(f"policy.csv:{number}: {field} is empty") + if row["waiver_class"] not in WAIVER_CLASSES: + errors.append( + f"policy.csv:{number}: waiver_class {row['waiver_class']!r} is not one of " + + ", ".join(sorted(WAIVER_CLASSES)) + ) + + for checker_value in row["enforcement"].split(";"): + checker = checker_value.strip() + if ( + not _repo_relative(checker) + or not checker.startswith("scripts/check-") + or Path(checker).suffix not in {".py", ".sh"} + ): + errors.append( + f"policy.csv:{number}: enforcement {checker!r} is not a repo-relative checker entrypoint" + ) + elif not (root / checker).is_file(): + errors.append(f"policy.csv:{number}: unknown enforcement path {checker!r}") + + procedure = row["procedure"] + if not _repo_relative(procedure): + errors.append( + f"policy.csv:{number}: procedure {procedure!r} is not repo-relative" + ) + elif Path(procedure).suffix != ".md" or not (root / procedure).is_file(): + errors.append(f"policy.csv:{number}: unknown procedure path {procedure!r}") + + if rule_id not in rules: + rules[rule_id] = PolicyRule(**row) + + if not rows: + errors.append("policy.csv must contain at least one rule") + return rules, errors + + +def load_policy(root: Path) -> dict[str, PolicyRule]: + rules, errors = _parse_policy(root) + if errors: + raise ValueError("\n".join(errors)) + return rules + + +def load_waivers( + root: Path, + rules: dict[str, PolicyRule], + today: dt.date | None = None, +) -> list[Waiver]: + current = today or dt.date.today() + rows, errors = _read_rows(root / ".agents/waivers.csv", WAIVER_HEADER) + waivers: list[Waiver] = [] + seen_ids: set[str] = set() + seen_targets: set[tuple[str, str]] = set() + for number, values in enumerate(rows, start=2): + row = dict(zip(WAIVER_HEADER, values, strict=True)) + waiver_id = row["waiver_id"] + if not WAIVER_ID.fullmatch(waiver_id): + errors.append(f"waivers.csv:{number}: malformed waiver_id {waiver_id!r}") + if waiver_id in seen_ids: + errors.append(f"waivers.csv:{number}: duplicate waiver_id {waiver_id!r}") + seen_ids.add(waiver_id) + + rule = rules.get(row["rule_id"]) + if rule is None: + errors.append( + f"waivers.csv:{number}: unknown rule {row['rule_id']!r}" + ) + elif rule.waiver_class == "never": + errors.append( + f"waivers.csv:{number}: rule {row['rule_id']!r} cannot be waived" + ) + + scope = row["scope"] + if any(character in scope for character in "*?[]{}"): + errors.append(f"waivers.csv:{number}: wildcard scope {scope!r} is forbidden") + if not _is_exact_scope(scope): + errors.append( + f"waivers.csv:{number}: {scope!r} is not an exact scope " + "(task/PR/commit/path/gate/hardware required)" + ) + target = (row["rule_id"], scope) + if target in seen_targets: + errors.append( + f"waivers.csv:{number}: duplicate rule/scope {row['rule_id']!r} {scope!r}" + ) + seen_targets.add(target) + + for field in ("owner", "reason", "evidence"): + if not row[field].strip(): + errors.append(f"waivers.csv:{number}: {field} is empty") + + expiry: dt.date | None = None + try: + expiry = dt.date.fromisoformat(row["expires"]) + except ValueError: + errors.append( + f"waivers.csv:{number}: expires {row['expires']!r} is not a valid ISO date" + ) + if expiry is not None and expiry <= current: + errors.append( + f"waivers.csv:{number}: waiver {waiver_id!r} expired on {expiry.isoformat()}" + ) + + if waiver_id and not _waiver_is_referenced(root, waiver_id): + errors.append( + f"waivers.csv:{number}: unused waiver {waiver_id!r}; cite it in its evidence or consuming record" + ) + + if expiry is not None: + waivers.append(Waiver(**{**row, "expires": expiry})) + + if errors: + raise ValueError("\n".join(errors)) + return waivers + + +def _waiver_is_referenced(root: Path, waiver_id: str) -> bool: + """A waiver is live debt only when a non-code record names it explicitly.""" + + needle = re.compile( + rf"(? list[str]: + """Return every policy/waiver contract defect found under *root*. + + ``schema_only`` is the bootstrap mode. Task 1 has no generated prose to + check yet, so both modes intentionally enforce the complete registry + schema and path contract; later cutover checks extend only full mode. + """ + + del schema_only + rules, errors = _parse_policy(root) + if errors: + return errors + try: + load_waivers(root, rules) + except (ValueError, KeyError) as exc: + errors.extend(str(exc).splitlines()) + return errors diff --git a/tests/scripts/test_policy_contract.py b/tests/scripts/test_policy_contract.py new file mode 100644 index 000000000..801bc76fd --- /dev/null +++ b/tests/scripts/test_policy_contract.py @@ -0,0 +1,324 @@ +#!/usr/bin/env python3 +"""Mutation tests for the authoritative policy and waiver registries.""" + +from __future__ import annotations + +import dataclasses +import datetime as dt +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path + +from scripts.policy_contract import ( + PolicyRule, + Waiver, + load_policy, + load_waivers, + validate_policy, +) + + +POLICY_HEADER = "rule_id,scope,trigger,requirement,enforcement,waiver_class,procedure\n" +WAIVER_HEADER = "waiver_id,rule_id,scope,owner,reason,evidence,expires\n" +ROOT = Path(__file__).resolve().parents[2] + + +class PolicyFixture: + def __init__(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + (self.root / ".agents").mkdir() + (self.root / "scripts").mkdir() + (self.root / "scripts/check-policy.py").write_text("#!/usr/bin/env python3\n") + (self.root / ".agents/workflow.md").write_text("# Workflow\n") + self.write_policy( + "POL-TEST-001,all,always,Keep the contract testable.," + "scripts/check-policy.py,expiring,.agents/workflow.md\n" + ) + self.write_waivers("") + + def close(self) -> None: + self.tmp.cleanup() + + def write_policy(self, rows: str) -> None: + (self.root / ".agents/policy.csv").write_text(POLICY_HEADER + rows) + + def write_waivers(self, rows: str) -> None: + (self.root / ".agents/waivers.csv").write_text(WAIVER_HEADER + rows) + + +class LoadContracts(unittest.TestCase): + def setUp(self) -> None: + self.fx = PolicyFixture() + + def tearDown(self) -> None: + self.fx.close() + + def test_records_are_immutable_and_valid_registry_loads(self) -> None: + rules = load_policy(self.fx.root) + self.assertEqual(set(rules), {"POL-TEST-001"}) + self.assertEqual(rules["POL-TEST-001"].waiver_class, "expiring") + self.assertEqual(load_waivers(self.fx.root, rules, today=dt.date(2026, 8, 7)), []) + with self.assertRaises(dataclasses.FrozenInstanceError): + rules["POL-TEST-001"].scope = "changed" # type: ignore[misc] + self.assertTrue(dataclasses.is_dataclass(PolicyRule)) + self.assertTrue(dataclasses.is_dataclass(Waiver)) + + +class PolicyMutations(unittest.TestCase): + def setUp(self) -> None: + self.fx = PolicyFixture() + + def tearDown(self) -> None: + self.fx.close() + + def errors(self) -> list[str]: + return validate_policy(self.fx.root, schema_only=True) + + def test_valid_fixture_passes_schema_validation(self) -> None: + self.assertEqual(self.errors(), []) + + def test_exact_header_is_required(self) -> None: + path = self.fx.root / ".agents/policy.csv" + path.write_text(path.read_text().replace("rule_id,scope", "id,scope", 1)) + self.assertTrue(any("header" in error for error in self.errors())) + + def test_duplicate_and_malformed_rule_ids_are_rejected(self) -> None: + self.fx.write_policy( + "POL-TEST-001,all,always,First.,scripts/check-policy.py,never,.agents/workflow.md\n" + "POL-TEST-001,all,always,Second.,scripts/check-policy.py,never,.agents/workflow.md\n" + "test-2,all,always,Third.,scripts/check-policy.py,never,.agents/workflow.md\n" + ) + errors = self.errors() + self.assertTrue(any("duplicate rule_id" in error for error in errors), errors) + self.assertTrue(any("malformed rule_id" in error for error in errors), errors) + + def test_multiline_field_and_blank_physical_record_are_rejected(self) -> None: + self.fx.write_policy( + 'POL-TEST-001,all,always,"First physical line\nsecond physical line",' + "scripts/check-policy.py,never,.agents/workflow.md\n\n" + ) + errors = self.errors() + self.assertTrue(any("multiline" in error for error in errors), errors) + self.assertTrue(any("blank physical" in error for error in errors), errors) + + def test_empty_fields_and_bad_waiver_class_are_rejected_in_bootstrap(self) -> None: + self.fx.write_policy( + "POL-TEST-001,,always,,scripts/check-policy.py,forever,.agents/workflow.md\n" + ) + errors = self.errors() + self.assertTrue(any("scope is empty" in error for error in errors), errors) + self.assertTrue(any("requirement is empty" in error for error in errors), errors) + self.assertTrue(any("waiver_class" in error for error in errors), errors) + + def test_checker_and_procedure_paths_must_be_known_repo_files(self) -> None: + self.fx.write_policy( + "POL-TEST-001,all,always,Do it.,scripts/check-missing.py,never,.agents/missing.md\n" + ) + errors = self.errors() + self.assertTrue(any("unknown enforcement" in error for error in errors), errors) + self.assertTrue(any("unknown procedure" in error for error in errors), errors) + + def test_absolute_traversing_and_non_checker_paths_are_rejected(self) -> None: + (self.fx.root / "scripts/tool.py").write_text("pass\n") + self.fx.write_policy( + "POL-TEST-001,all,always,Do it.,scripts/tool.py;/tmp/check.py,never,../workflow.md\n" + ) + errors = self.errors() + self.assertTrue(any("checker entrypoint" in error for error in errors), errors) + self.assertTrue(any("repo-relative" in error for error in errors), errors) + + def test_policy_count_and_byte_budgets_are_enforced(self) -> None: + rows = "".join( + f"POL-TEST-{index:03d},all,always,Rule {index}.,scripts/check-policy.py,never,.agents/workflow.md\n" + for index in range(61) + ) + self.fx.write_policy(rows) + self.assertTrue(any("more than 60" in error for error in self.errors())) + self.fx.write_policy( + "POL-TEST-001,all,always," + ("x" * 17000) + ",scripts/check-policy.py,never,.agents/workflow.md\n" + ) + self.assertTrue(any("16384-byte" in error for error in self.errors())) + + def test_load_policy_refuses_an_invalid_registry(self) -> None: + self.fx.write_policy( + "POL-TEST-001,all,always,Do it.,scripts/check-missing.py,never,.agents/workflow.md\n" + ) + with self.assertRaises(ValueError): + load_policy(self.fx.root) + + +class WaiverMutations(unittest.TestCase): + TODAY = dt.date(2026, 8, 7) + + def setUp(self) -> None: + self.fx = PolicyFixture() + + def tearDown(self) -> None: + self.fx.close() + + def load_errors(self) -> str: + rules = load_policy(self.fx.root) + with self.assertRaises(ValueError) as raised: + load_waivers(self.fx.root, rules, today=self.TODAY) + return str(raised.exception) + + def mark_used(self, waiver_id: str) -> None: + (self.fx.root / ".agents/waiver-evidence.md").write_text( + f"Applied exception: {waiver_id}\n" + ) + + def test_exact_header_and_valid_future_waiver(self) -> None: + self.mark_used("WAIVER-TEST-001") + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,task:POLICY-1,maintainer," + "Bounded migration,docs/evidence.md,2026-08-08\n" + ) + waiver = load_waivers( + self.fx.root, load_policy(self.fx.root), today=self.TODAY + )[0] + self.assertEqual(waiver.expires, dt.date(2026, 8, 8)) + self.assertEqual(waiver.scope, "task:POLICY-1") + + path = self.fx.root / ".agents/waivers.csv" + path.write_text(path.read_text().replace("waiver_id,rule_id", "id,rule_id", 1)) + self.assertIn("header", self.load_errors()) + + def test_unknown_and_never_waivable_rules_are_rejected(self) -> None: + self.fx.write_policy( + "POL-TEST-001,all,always,Do it.,scripts/check-policy.py,never,.agents/workflow.md\n" + ) + self.fx.write_waivers( + "WAIVER-TEST-001,POL-MISSING,task:POLICY-1,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-002,POL-TEST-001,task:POLICY-2,owner,reason,evidence,2026-08-08\n" + ) + errors = self.load_errors() + self.assertIn("unknown rule", errors) + self.assertIn("cannot be waived", errors) + + def test_scope_must_be_narrow_exact_and_unique(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,path:*,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-002,POL-TEST-001,repository:all,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-003,POL-TEST-001,task:POLICY-3,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-004,POL-TEST-001,task:POLICY-3,owner,reason,evidence,2026-08-09\n" + ) + errors = self.load_errors() + self.assertIn("wildcard", errors) + self.assertIn("exact scope", errors) + self.assertIn("duplicate rule/scope", errors) + + def test_repository_wide_and_traversing_scopes_are_not_exact(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,pr:all,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-002,POL-TEST-001,commit:all,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-003,POL-TEST-001,hardware:all,owner,reason,evidence,2026-08-08\n" + "WAIVER-TEST-004,POL-TEST-001,path:foo/..,owner,reason,evidence,2026-08-08\n" + ) + errors = self.load_errors() + for scope in ("pr:all", "commit:all", "hardware:all", "path:foo/.."): + with self.subTest(scope=scope): + self.assertIn(repr(scope), errors) + self.assertIn("exact scope", errors) + + def test_each_supported_scope_kind_accepts_a_concrete_target(self) -> None: + waiver_ids = [f"WAIVER-TEST-{index:03d}" for index in range(1, 7)] + (self.fx.root / ".agents/waiver-evidence.md").write_text( + "Applied exceptions: " + " ".join(waiver_ids) + "\n" + ) + scopes = ( + "task:POLICY-1", + "pr:128", + f"commit:{'a' * 40}", + "path:.agents/workflow.md", + "gate:policy-schema", + "hardware:GB10-1", + ) + self.fx.write_waivers( + "".join( + f"{waiver_id},POL-TEST-001,{scope},owner,reason,evidence,2026-08-08\n" + for waiver_id, scope in zip(waiver_ids, scopes, strict=True) + ) + ) + waivers = load_waivers( + self.fx.root, load_policy(self.fx.root), today=self.TODAY + ) + self.assertEqual([waiver.scope for waiver in waivers], list(scopes)) + + def test_duplicate_ids_and_missing_attribution_are_rejected(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,task:POLICY-1,,reason,evidence,2026-08-08\n" + "WAIVER-TEST-001,POL-TEST-001,task:POLICY-2,owner,,evidence,2026-08-08\n" + "WAIVER-TEST-003,POL-TEST-001,task:POLICY-3,owner,reason,,2026-08-08\n" + ) + errors = self.load_errors() + self.assertIn("duplicate waiver_id", errors) + self.assertIn("owner is empty", errors) + self.assertIn("reason is empty", errors) + self.assertIn("evidence is empty", errors) + + def test_invalid_expired_and_today_expiry_are_rejected(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,task:POLICY-1,owner,reason,evidence,not-a-date\n" + "WAIVER-TEST-002,POL-TEST-001,task:POLICY-2,owner,reason,evidence,2026-08-06\n" + "WAIVER-TEST-003,POL-TEST-001,task:POLICY-3,owner,reason,evidence,2026-08-07\n" + ) + errors = self.load_errors() + self.assertIn("ISO date", errors) + self.assertIn("expired", errors) + + def test_unreferenced_waiver_is_rejected_as_unused(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-TEST-001,task:POLICY-1,owner,reason,evidence,2026-08-08\n" + ) + self.assertIn("unused waiver", self.load_errors()) + + def test_schema_only_does_not_suppress_waiver_defects(self) -> None: + self.fx.write_waivers( + "WAIVER-TEST-001,POL-MISSING,path:*,owner,reason,evidence,2026-08-06\n" + ) + errors = validate_policy(self.fx.root, schema_only=True) + self.assertTrue(any("unknown rule" in error for error in errors), errors) + self.assertTrue(any("wildcard" in error for error in errors), errors) + self.assertTrue(any("expired" in error for error in errors), errors) + + +class CommandLineContract(unittest.TestCase): + def setUp(self) -> None: + self.fx = PolicyFixture() + shutil.copy2(ROOT / "scripts/policy_contract.py", self.fx.root / "scripts") + shutil.copy2(ROOT / "scripts/check-policy.py", self.fx.root / "scripts") + + def tearDown(self) -> None: + self.fx.close() + + def run_checker(self, *arguments: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["python3", "scripts/check-policy.py", *arguments], + cwd=self.fx.root, + text=True, + capture_output=True, + check=False, + ) + + def test_cli_passes_valid_registry_and_fails_bootstrap_defect(self) -> None: + self.assertEqual(self.run_checker("--schema-only").returncode, 0) + path = self.fx.root / ".agents/policy.csv" + path.write_text(path.read_text().replace("rule_id", "id", 1)) + result = self.run_checker("--schema-only") + self.assertNotEqual(result.returncode, 0) + self.assertIn("header", result.stdout + result.stderr) + + +class RepositoryRegistry(unittest.TestCase): + def test_accepted_design_inventory_passes_schema(self) -> None: + rules = load_policy(ROOT) + self.assertLessEqual(len(rules), 60) + self.assertLessEqual((ROOT / ".agents/policy.csv").stat().st_size, 16 * 1024) + self.assertEqual(validate_policy(ROOT, schema_only=True), []) + + +if __name__ == "__main__": + unittest.main() From 2be41cbb23f9912858fb93d5af1a10ccaac469f9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 00:10:20 +0000 Subject: [PATCH 04/12] policy(docs): close public projection grammar Affected rules: POL-DOC-STATUS, POL-DOC-BENCHMARKS, POL-DOC-FEATURES, POL-DOC-USAGE, POL-DOC-README, POL-NOW-COUPLING. Parse projection requirements through a portable path grammar, reserve public target ownership independently of positive syntax, and bind exact semantic path classes with delimiter-safe left and right boundary controls. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/policy.csv | 12 +- scripts/check-doc-checkpoint.py | 582 +++++++++++++++--- scripts/check-protocol-consistency.py | 159 ++--- .../test_check_protocol_consistency.py | 329 ++++++++-- tests/scripts/test_doc_checkpoint.py | 334 +++++++--- 5 files changed, 1081 insertions(+), 335 deletions(-) diff --git a/.agents/policy.csv b/.agents/policy.csv index a91884f71..e7e3c056c 100644 --- a/.agents/policy.csv +++ b/.agents/policy.csv @@ -34,12 +34,12 @@ POL-SPIKE-FIRST,task lifecycle,before READY or ACTIVE,Commit a complete spike sp POL-EVIDENCE-PRESERVE,records,compacting or retiring records,Move evidence without deleting it.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/directives.md POL-ROW-SYNC,checkpoint,feature lifecycle changes,Update the roadmap row and owning matrix row in the same change.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/workflow.md POL-STATE-ORDER,state log,appending state,Append below the enforced marker in chronological anchor order.,scripts/check-state-order.py; scripts/check-policy.py,never,.agents/workflow.md -POL-NOW-COUPLING,live state,state is appended or live claims gates or actions change,Refresh NOW.md in the same change.,scripts/check-now-current.py; scripts/check-policy.py,never,.agents/workflow.md -POL-DOC-STATUS,public status,feature or iteration checkpoint,Update docs/STATUS.md with lifecycle gap evidence and next gate.,scripts/check-doc-checkpoint.py; scripts/check-policy.py,never,.agents/directives.md -POL-DOC-BENCHMARKS,public benchmarks,feature or iteration checkpoint,Update docs/BENCHMARKS.md with an accepted or explicit pending failed not-applicable or void result.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/benchmark-protocol.md -POL-DOC-FEATURES,public features,feature model backend or quantization support changes,Update the keyed docs/FEATURES.md row in place.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/directives.md -POL-DOC-USAGE,public usage,user commands flags APIs endpoints config install or runnable workflows change,Update docs/USAGE.md with the user-facing behavior.,scripts/check-policy.py,never,.agents/workflow.md -POL-DOC-README,landing page,a headline quickstart install positioning or headline benchmark changes,Update README.md only for that landing-page change.,scripts/check-readme-structure.py; scripts/check-policy.py,never,.agents/workflow.md +POL-NOW-COUPLING,.agents/NOW.md,live_state,Refresh .agents/NOW.md.,scripts/check-doc-checkpoint.py; scripts/check-now-current.py; scripts/check-policy.py,never,.agents/workflow.md +POL-DOC-STATUS,docs/STATUS.md,feature_checkpoint,Update docs/STATUS.md.,scripts/check-doc-checkpoint.py; scripts/check-policy.py,never,.agents/directives.md +POL-DOC-BENCHMARKS,docs/BENCHMARKS.md,feature_checkpoint,Update docs/BENCHMARKS.md.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/benchmark-protocol.md +POL-DOC-FEATURES,docs/FEATURES.md,feature_surface,Update docs/FEATURES.md.,scripts/check-doc-checkpoint.py; scripts/check-public-doc-tables.py; scripts/check-policy.py,never,.agents/directives.md +POL-DOC-USAGE,docs/USAGE.md,user_usage,Update docs/USAGE.md.,scripts/check-doc-checkpoint.py; scripts/check-policy.py,never,.agents/workflow.md +POL-DOC-README,README.md,landing_page,Update README.md.,scripts/check-doc-checkpoint.py; scripts/check-readme-structure.py; scripts/check-policy.py,never,.agents/workflow.md POL-KEYED-MERGE,keyed records,resolving concurrent changes,Take main's keyed record and reapply the edit instead of using a three-way merge.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/workflow.md POL-TABLE-INVENTORY,project inventory,adding or changing an inventory item,Use a stable ID and record source anchor evidence spike lifecycle and owner.,scripts/check-agent-record.py; scripts/check-policy.py,never,.agents/discipline.md POL-PR-REQUIRED,repository change,changing feature policy checker docs or records,Deliver the change through a PR except under an exact expiring emergency waiver.,scripts/check-role-discipline.py; scripts/check-policy.py,expiring,.agents/workflow.md diff --git a/scripts/check-doc-checkpoint.py b/scripts/check-doc-checkpoint.py index 3f3f85cf1..18eb2a1ea 100755 --- a/scripts/check-doc-checkpoint.py +++ b/scripts/check-doc-checkpoint.py @@ -1,41 +1,389 @@ #!/usr/bin/env python3 -"""Require public documentation updates for every feature checkpoint.""" +"""Enforce purpose-specific public-document projections for each change.""" from __future__ import annotations import argparse +import string import subprocess import sys +from dataclasses import dataclass from pathlib import Path - ROOT = Path(__file__).resolve().parents[1] -# The per-capability status surface is docs/STATUS.md, NOT README.md: pointing -# this obligation at the README is what drifted it from a landing page into a -# status log. README.md changes only when a user-visible headline shifts, which -# is a judgement call this checker deliberately does not force. -PUBLIC_CHECKPOINTS = ("docs/STATUS.md", "docs/BENCHMARKS.md") - -# The public feature surface is docs/FEATURES.md: the user-facing comparison of -# what we support against vLLM, SGLang and llama.cpp. It is NOT owed by every -# checkpoint (most commits move no feature row), so it has its own narrower -# trigger set: the area matrices that define feature/model/backend/quant state, -# and the model implementations themselves. AGENTS.md already requires those -# matrices to move in the same change as the code, so this only mirrors that -# obligation onto the public surface. -FEATURE_CHECKPOINT = "docs/FEATURES.md" -FEATURE_TRIGGER_PREFIXES = ("src/vllm/model_executor/models/",) -FEATURE_TRIGGER_FILES = { - ".agents/backend-matrix.md", - ".agents/feature-matrix.md", - ".agents/model-matrix.md", - ".agents/quantization-matrix.md", +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +from scripts.policy_contract import PolicyRule, load_policy + + +CHECKER_PATH = "scripts/check-doc-checkpoint.py" + + +@dataclass(frozen=True) +class PublicRuleContract: + surface: str + change_class: str + action: str + enforcement: tuple[str, ...] + + +@dataclass(frozen=True) +class PublicRuleBinding: + rule_id: str + surface: str + change_class: str + action: str + enforcement: tuple[str, ...] + + +# Public rules are a closed machine interface. Their procedures carry the +# explanatory detail; these cells contain no prose for a checker to interpret. +PUBLIC_RULE_CONTRACTS = { + "POL-DOC-STATUS": PublicRuleContract( + "docs/STATUS.md", + "feature_checkpoint", + "Update", + (CHECKER_PATH, "scripts/check-policy.py"), + ), + "POL-DOC-BENCHMARKS": PublicRuleContract( + "docs/BENCHMARKS.md", + "feature_checkpoint", + "Update", + ( + CHECKER_PATH, + "scripts/check-public-doc-tables.py", + "scripts/check-policy.py", + ), + ), + "POL-DOC-FEATURES": PublicRuleContract( + "docs/FEATURES.md", + "feature_surface", + "Update", + ( + CHECKER_PATH, + "scripts/check-public-doc-tables.py", + "scripts/check-policy.py", + ), + ), + "POL-DOC-USAGE": PublicRuleContract( + "docs/USAGE.md", + "user_usage", + "Update", + (CHECKER_PATH, "scripts/check-policy.py"), + ), + "POL-DOC-README": PublicRuleContract( + "README.md", + "landing_page", + "Update", + ( + CHECKER_PATH, + "scripts/check-readme-structure.py", + "scripts/check-policy.py", + ), + ), + "POL-NOW-COUPLING": PublicRuleContract( + ".agents/NOW.md", + "live_state", + "Refresh", + ( + CHECKER_PATH, + "scripts/check-now-current.py", + "scripts/check-policy.py", + ), + ), } +PUBLIC_RULE_IDS = tuple(PUBLIC_RULE_CONTRACTS) +PUBLIC_CHANGE_CLASSES = frozenset( + contract.change_class for contract in PUBLIC_RULE_CONTRACTS.values() +) +PUBLIC_SURFACES = frozenset( + contract.surface for contract in PUBLIC_RULE_CONTRACTS.values() +) +POSITIVE_ACTIONS = frozenset({"Update", "Refresh"}) +REPOSITORY_PATH_CHARACTERS = frozenset( + string.ascii_letters + string.digits + "._-" +) +REPOSITORY_PATH_LEXICAL_CHARACTERS = REPOSITORY_PATH_CHARACTERS | frozenset("/") + + +def parse_requirement(requirement: str) -> tuple[str, str]: + """Parse one positive action and repository-relative target exactly. + + The final period is requirement syntax, not part of the path. Validate it + separately so a greedy target cannot consume an extra period and evade a + target-ownership comparison. + """ + + if not requirement.endswith("."): + raise ValueError("must end with exactly one terminal period") + + body = requirement[:-1] + fields = body.split(" ") + if len(fields) != 2 or any(not field for field in fields): + raise ValueError("must contain one action and one path token") + action, target = fields + if action not in POSITIVE_ACTIONS: + raise ValueError("action must be Update or Refresh") + + if target.startswith("/"): + raise ValueError("target must be repository-relative") + segments = target.split("/") + if any(not segment for segment in segments): + raise ValueError("target path segments must be nonempty") + for segment in segments: + if segment in {".", ".."}: + raise ValueError("target must not contain dot path segments") + if segment.endswith("."): + raise ValueError("target path segments must not end with a period") + if any( + character not in REPOSITORY_PATH_CHARACTERS for character in segment + ): + raise ValueError("target contains a non-portable repository-path character") + return action, target + + +def _uses_repository_target_form(requirement: str) -> bool: + """Distinguish controlled path requirements from ordinary policy prose.""" + + for action in POSITIVE_ACTIONS: + prefix = f"{action} " + if not requirement.startswith(prefix): + continue + first_field = requirement[len(prefix) :].split(" ", 1)[0] + return any(marker in first_field for marker in ("/", "\\", ".")) + return False + + +def reserved_public_targets(requirement: str) -> set[str]: + """Find exact public-surface tokens without interpreting the sentence. + + Ownership is independent of the positive requirement grammar. A non-owner + cannot hide a reserved target behind another verb, an inserted word, or + malformed whitespace. Boundaries use the repository-path alphabet plus + ``/`` so path prefixes such as ``docs/STATUS.md.extra`` and nested paths do + not accidentally claim the exact ``docs/STATUS.md`` token. + """ + + found: set[str] = set() + for surface in PUBLIC_SURFACES: + start = 0 + while True: + index = requirement.find(surface, start) + if index == -1: + break + end = index + len(surface) + left_is_path = ( + index > 0 + and requirement[index - 1] in REPOSITORY_PATH_LEXICAL_CHARACTERS + ) + right_is_path = False + if end < len(requirement): + right = requirement[end] + if right != ".": + right_is_path = right in REPOSITORY_PATH_LEXICAL_CHARACTERS + else: + # A run of sentence periods delimits the exact token. A + # following path character makes it a longer path instead + # (for example docs/STATUS.md.extra). + cursor = end + while cursor < len(requirement) and requirement[cursor] == ".": + cursor += 1 + right_is_path = ( + cursor < len(requirement) + and requirement[cursor] + in REPOSITORY_PATH_LEXICAL_CHARACTERS + ) + if not left_is_path and not right_is_path: + found.add(surface) + start = index + 1 + return found + + +def public_namespace_errors(rules: dict[str, PolicyRule]) -> list[str]: + """Reject any rule that trespasses on the closed public-rule namespace.""" + + trigger_owners: dict[str, set[str]] = {} + scope_owners: dict[str, str] = {} + target_owners: dict[str, str] = {} + semantic_owners: dict[tuple[str, str, str, str], str] = {} + for rule_id, contract in PUBLIC_RULE_CONTRACTS.items(): + trigger_owners.setdefault(contract.change_class, set()).add(rule_id) + scope_owners[contract.surface] = rule_id + target_owners[contract.surface] = rule_id + semantic_owners[ + (contract.surface, contract.change_class, contract.action, contract.surface) + ] = rule_id -CHECKPOINT_PREFIXES = ( - ".agents/completed/", - ".agents/specs/", + errors: list[str] = [] + for rule_id, rule in sorted(rules.items()): + if rule_id.startswith("POL-DOC-") and rule_id not in PUBLIC_RULE_CONTRACTS: + errors.append( + f"{rule_id}: reserved POL-DOC-* namespace belongs to the public " + "document contract" + ) + + allowed_trigger_owners = trigger_owners.get(rule.trigger) + if allowed_trigger_owners is not None and rule_id not in allowed_trigger_owners: + errors.append( + f"{rule_id}: trigger {rule.trigger!r} is reserved for " + + ", ".join(sorted(allowed_trigger_owners)) + ) + + scope_owner = scope_owners.get(rule.scope) + if scope_owner is not None and rule_id != scope_owner: + errors.append( + f"{rule_id}: scope {rule.scope!r} is reserved for {scope_owner}" + ) + + claimed_targets = reserved_public_targets(rule.requirement) + for target in sorted(claimed_targets): + target_owner = target_owners[target] + if rule_id != target_owner: + errors.append( + f"{rule_id}: requirement target {target!r} is reserved for " + f"{target_owner}" + ) + + try: + action, target = parse_requirement(rule.requirement) + except ValueError as exc: + if _uses_repository_target_form(rule.requirement): + errors.append(f"{rule_id}: requirement is invalid: {exc}") + continue + target_owner = target_owners.get(target) + if ( + target_owner is not None + and rule_id != target_owner + and target not in claimed_targets + ): + errors.append( + f"{rule_id}: requirement target {target!r} is reserved for " + f"{target_owner}" + ) + + semantic_owner = semantic_owners.get( + (rule.scope, rule.trigger, action, target) + ) + if semantic_owner is not None and rule_id != semantic_owner: + errors.append( + f"{rule_id}: duplicate public semantic binding owned by " + f"{semantic_owner}" + ) + return errors + + +def parse_public_rule(rule: PolicyRule) -> PublicRuleBinding: + """Parse one public rule completely or reject it as non-canonical.""" + + contract = PUBLIC_RULE_CONTRACTS.get(rule.rule_id) + if contract is None: + raise ValueError(f"{rule.rule_id}: unknown public rule") + if rule.scope != contract.surface: + raise ValueError( + f"{rule.rule_id}: scope {rule.scope!r} must be {contract.surface!r}" + ) + if rule.trigger != contract.change_class: + raise ValueError( + f"{rule.rule_id}: trigger {rule.trigger!r} must be " + f"{contract.change_class!r}" + ) + + try: + action, target = parse_requirement(rule.requirement) + except ValueError as exc: + raise ValueError( + f"{rule.rule_id}: requirement must fully match " + f"{contract.action} {contract.surface}." + ) from exc + if action != contract.action or target != contract.surface: + raise ValueError( + f"{rule.rule_id}: requirement must be exactly " + f"{contract.action} {contract.surface}." + ) + + enforcement = tuple(item.strip() for item in rule.enforcement.split(";")) + if any(not item for item in enforcement) or enforcement != contract.enforcement: + raise ValueError( + f"{rule.rule_id}: enforcement must be exactly " + + "; ".join(contract.enforcement) + ) + return PublicRuleBinding( + rule_id=rule.rule_id, + surface=contract.surface, + change_class=contract.change_class, + action=contract.action, + enforcement=enforcement, + ) + + +def public_document_rules(root: Path = ROOT) -> dict[str, PolicyRule]: + """Load the exact public-projection rules from the policy authority.""" + + rules = load_policy(root) + namespace_errors = public_namespace_errors(rules) + if namespace_errors: + raise ValueError("\n".join(namespace_errors)) + missing = [rule_id for rule_id in PUBLIC_RULE_IDS if rule_id not in rules] + if missing: + raise ValueError("policy.csv is missing public rules: " + ", ".join(missing)) + return {rule_id: rules[rule_id] for rule_id in PUBLIC_RULE_IDS} + + +def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: + """Return parsed public-rule bindings keyed by stable rule ID.""" + + return { + rule_id: parse_public_rule(rule) + for rule_id, rule in public_document_rules(root).items() + } + + +# Governance paths are exact. Unlisted scripts and tests remain checkpoints, +# including technical runtime consistency checkers with similar names. +GOVERNANCE_FILES = frozenset( + { + ".agents/prompts/implementer.md", + ".agents/prompts/reviewer.md", + ".agents/governance-tasks.csv", + ".agents/policy.csv", + ".agents/waivers.csv", + "scripts/agent-role.py", + "scripts/check-doc-checkpoint.py", + "scripts/check-gate-commands.py", + "scripts/policy_contract.py", + "scripts/check-policy.py", + "scripts/check-pr-size.py", + "scripts/check-protocol-consistency.py", + "scripts/check-role-discipline.py", + "scripts/claim-view.py", + "tests/scripts/test_agent_role.py", + "tests/scripts/test_claim_view.py", + "tests/scripts/test_policy_contract.py", + "tests/scripts/test_doc_checkpoint.py", + "tests/scripts/test_check_protocol_consistency.py", + "docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md", + } +) +FEATURE_CHECKPOINT_FILES = frozenset( + { + "CMakeLists.txt", + ".agents/backend-matrix.md", + ".agents/coordination.md", + ".agents/engine-matrix.md", + ".agents/feature-matrix.md", + ".agents/kernel-matrix.md", + ".agents/model-matrix.md", + ".agents/parity-ledger.md", + ".agents/porting-inventory.md", + ".agents/quantization-matrix.md", + ".agents/roadmap_v1.md", + ".agents/state.md", + } +) +FEATURE_CHECKPOINT_PREFIXES = ( ".github/workflows/", + ".agents/specs/", "cmake/", "examples/", "include/", @@ -44,65 +392,132 @@ "tests/", "tools/", ) -CHECKPOINT_FILES = { - "CMakeLists.txt", - ".agents/backend-matrix.md", - ".agents/coordination.md", - ".agents/engine-matrix.md", - ".agents/feature-matrix.md", - ".agents/kernel-matrix.md", - ".agents/model-matrix.md", - ".agents/parity-ledger.md", - ".agents/porting-inventory.md", - ".agents/quantization-matrix.md", - ".agents/roadmap_v1.md", - ".agents/state.md", -} - - -def is_checkpoint_path(path: str) -> bool: - """Return whether a changed path advances a feature/iteration checkpoint.""" - return path in CHECKPOINT_FILES or path.startswith(CHECKPOINT_PREFIXES) - - -def is_feature_path(path: str) -> bool: - """Return whether a changed path can move a row in the public feature matrix.""" - return path in FEATURE_TRIGGER_FILES or path.startswith(FEATURE_TRIGGER_PREFIXES) +FEATURE_SURFACE_FILES = frozenset( + { + ".agents/backend-matrix.md", + ".agents/feature-matrix.md", + ".agents/model-matrix.md", + ".agents/quantization-matrix.md", + } +) +FEATURE_SURFACE_PREFIXES = ("src/vllm/model_executor/models/",) + +# These are exact user-facing configuration/build/install entrypoints. Do not +# broaden this to all cmake files: toolchain internals are checkpoints but do +# not necessarily alter installation instructions. +USER_USAGE_FILES = frozenset( + { + ".env.example", + "CMakeLists.txt", + "cmake/install.cmake", + "examples/CMakeLists.txt", + "examples/cli/main.cpp", + "examples/server/main.cpp", + "include/vllm.h", + } +) +USER_USAGE_PREFIXES = ( + "include/vllm/", + "src/vllm/entrypoints/", + "examples/cli/", + "examples/server/", +) +LIVE_STATE_FILES = frozenset({".agents/state.md", ".agents/coordination.md"}) + +# README permission and obligation come only from underlying landing sources. +# Co-edited public projections can never justify README churn. +LANDING_SOURCE_FILES = frozenset( + { + ".agents/mission.md", + "CMakeLists.txt", + "benchmarks/demo/footprint_gb10.json", + "benchmarks/demo/qwen36_27b_c1_c32.json", + "examples/cli/main.cpp", + "examples/server/main.cpp", + } +) -def _preview(triggers: list[str]) -> str: - preview = ", ".join(triggers[:5]) - if len(triggers) > 5: - preview += f", ... (+{len(triggers) - 5})" +def classify_changed_paths(paths: list[str]) -> set[str]: + """Classify changed paths by semantic obligation, not mutable directory.""" + + classes: set[str] = set() + path_set = set(paths) + for path in sorted(path_set): + if path in PUBLIC_SURFACES: + continue + if path in GOVERNANCE_FILES: + classes.add("governance") + continue + if path in LIVE_STATE_FILES: + classes.update({"feature_checkpoint", "live_state"}) + if path in FEATURE_SURFACE_FILES or path.startswith(FEATURE_SURFACE_PREFIXES): + classes.update({"feature_checkpoint", "feature_surface"}) + if path in USER_USAGE_FILES or path.startswith(USER_USAGE_PREFIXES): + classes.update({"feature_checkpoint", "user_usage"}) + if path in LANDING_SOURCE_FILES: + classes.add("landing_page") + if path in FEATURE_CHECKPOINT_FILES or path.startswith( + FEATURE_CHECKPOINT_PREFIXES + ): + classes.add("feature_checkpoint") + + if "README.md" in path_set: + classes.add("readme_changed") + return classes + + +def required_public_surfaces(change_classes: set[str]) -> set[str]: + """Project semantic classes to surfaces through fully parsed policy rules.""" + + return { + binding.surface + for binding in public_rule_bindings().values() + if binding.change_class in change_classes + } + + +def _preview(paths: set[str]) -> str: + ordered = sorted(paths) + preview = ", ".join(ordered[:5]) + if len(ordered) > 5: + preview += f", ... (+{len(ordered) - 5})" return preview def checkpoint_errors(paths: set[str]) -> list[str]: - """Return missing-public-document errors for one atomic change.""" + """Return missing or unjustified public-projection errors for one change.""" + + classes = classify_changed_paths(sorted(paths)) + required = required_public_surfaces(classes) + bindings = public_rule_bindings() errors: list[str] = [] + for rule_id in PUBLIC_RULE_IDS: + binding = bindings[rule_id] + if binding.surface in required and binding.surface not in paths: + errors.append( + f"{binding.change_class} change ({_preview(paths)}) requires " + f"{binding.surface} in the same change under {rule_id}" + ) - triggers = sorted(path for path in paths if is_checkpoint_path(path)) - if triggers: - errors += [ - "feature/iteration checkpoint changed " - f"({_preview(triggers)}) but did not update {path} in the same change" - for path in PUBLIC_CHECKPOINTS - if path not in paths - ] - - feature_triggers = sorted(path for path in paths if is_feature_path(path)) - if feature_triggers and FEATURE_CHECKPOINT not in paths: + if "readme_changed" in classes and "landing_page" not in classes: errors.append( - "feature/model/backend/quantization surface changed " - f"({_preview(feature_triggers)}) but did not update " - f"{FEATURE_CHECKPOINT} in the same change; update the row this " - "moves (support mark, gate, or the Not-supported-yet table) so the " - "public feature matrix cannot drift from the area matrices" + "README.md changed without a landing-page trigger; change it only " + "with an underlying quickstart, install, positioning, or headline " + "benchmark source" ) - return errors +# Compatibility for existing callers. +def is_checkpoint_path(path: str) -> bool: + return "feature_checkpoint" in classify_changed_paths([path]) + + +def is_feature_path(path: str) -> bool: + return "feature_surface" in classify_changed_paths([path]) + + def git(*args: str) -> str: return subprocess.check_output( ["git", *args], cwd=ROOT, text=True, stderr=subprocess.DEVNULL @@ -136,20 +551,27 @@ def parse_args() -> argparse.Namespace: source.add_argument( "--staged", action="store_true", help="check the current staged change" ) + source.add_argument("--range", dest="revision_range", help="check BASE..HEAD") parser.add_argument("--base", help="check every commit after this revision") parser.add_argument("--head", help="range endpoint (requires --base)") args = parser.parse_args() if (args.base is None) != (args.head is None): parser.error("--base and --head must be supplied together") - if args.base is not None and (args.commit is not None or args.staged): + if args.base is not None and ( + args.commit is not None or args.staged or args.revision_range is not None + ): parser.error("a revision range cannot be combined with --commit/--staged") + if args.revision_range is not None: + parts = args.revision_range.split("..") + if len(parts) != 2 or not all(parts): + parser.error("--range must be BASE..HEAD") + args.base, args.head = parts return args def main() -> int: args = parse_args() failures: list[str] = [] - if args.staged: paths = set(git("diff", "--cached", "--name-only").splitlines()) failures.extend(f"staged change: {error}" for error in checkpoint_errors(paths)) @@ -165,23 +587,11 @@ def main() -> int: f"commit {short}: {error}" for error in checkpoint_errors(commit_paths(commit)) ) - if failures: for failure in failures: print(f"ERROR: {failure}", file=sys.stderr) - print( - "Update docs/STATUS.md and docs/BENCHMARKS.md with the current " - "stage/result, including explicit pending or void outcomes, and " - "docs/FEATURES.md when the feature/model/backend/quantization " - "surface moves.", - file=sys.stderr, - ) return 1 - - print( - "OK: feature checkpoints update docs/STATUS.md and docs/BENCHMARKS.md, " - "and feature-surface changes update docs/FEATURES.md." - ) + print("OK: public documentation matches the change's semantic policy triggers.") return 0 diff --git a/scripts/check-protocol-consistency.py b/scripts/check-protocol-consistency.py index 495727549..7dbd61ff7 100644 --- a/scripts/check-protocol-consistency.py +++ b/scripts/check-protocol-consistency.py @@ -1,26 +1,9 @@ #!/usr/bin/env python3 -"""Keep the protocol prose and the checkers that enforce it in agreement. +"""Keep structured policy consumers and their protocol artifacts consistent. -The failure this exists to prevent is real and already happened: the -same-change public-document obligation was migrated from README.md to -docs/STATUS.md in scripts/check-doc-checkpoint.py, AGENTS.md was updated, and -`.agents/workflow.md` -- the operating manual an agent is told to follow every -session -- was not. For a while the manual instructed agents to update README.md -at every checkpoint, which is exactly the drift the migration removed. Prose and -checker disagreed, and the prose is what agents actually read. - -So the obligated surfaces are declared ONCE, as a machine-readable contract -block that both documents carry verbatim, and this gate asserts the block equals -the constants in scripts/check-doc-checkpoint.py. Changing the checker without -changing the prose (or the reverse) is a red build, not a silent divergence. - -The contract block looks like this, and is a normal Markdown table to a reader: - - - | Public surface | Owed by | - |---|---| - | `docs/STATUS.md` | every feature/iteration checkpoint | - +Public-document obligations are controlled policy rows. The document checker +fully parses their stable ID, exact scope, semantic trigger, positive action, +and enforcement list; explanatory details remain in procedures. The same gate now also asserts that `.agents/workflow.md` carries the ROLE INTERVIEW, between `` and its `:end`. That is the @@ -56,13 +39,10 @@ ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) -# Both documents must carry the contract, because both are read as normative: -# AGENTS.md is the canonical index, workflow.md is the session operating manual. -CONTRACT_DOCUMENTS = ("AGENTS.md", ".agents/workflow.md") - -BEGIN = "" -END = "" +from scripts.policy_contract import PolicyRule, load_policy # The session manual must carry the role interview, because agent-preflight.sh # now FAILS a session that has not declared a role. A gate whose precondition is @@ -128,21 +108,6 @@ ), } -# A path in a table cell, e.g. `docs/STATUS.md`. -CELL_PATH = re.compile(r"`([^`]+\.md)`") - -# README.md is a landing page, not a checkpoint surface. Naming it inside the -# contract is the specific regression this gate was built after, so it earns a -# targeted message instead of a bare set-difference. -FORBIDDEN_IN_CONTRACT = { - "README.md": ( - "README.md is a user-facing landing page, not a per-checkpoint status " - "surface; it changes only when a user-visible headline shifts. The " - "per-capability obligation belongs to docs/STATUS.md" - ), -} - - def _load(name: str, relative: str): path = ROOT / relative spec = importlib.util.spec_from_file_location(name, path) @@ -153,52 +118,43 @@ def _load(name: str, relative: str): return module -def obligated_surfaces() -> tuple[str, ...]: - """Return the surfaces check-doc-checkpoint.py actually enforces.""" - checkpoint = _load("doc_checkpoint", "scripts/check-doc-checkpoint.py") - return tuple(checkpoint.PUBLIC_CHECKPOINTS) + (checkpoint.FEATURE_CHECKPOINT,) - +def public_document_rule_errors( + rules: dict[str, PolicyRule] | None = None, +) -> list[str]: + """Validate every public rule through the document checker's closed parser.""" -def contract_paths(text: str) -> list[str] | None: - """Return the paths declared in the contract block, or None if absent.""" - start = text.find(BEGIN) - end = text.find(END) - if start == -1 or end == -1 or end < start: - return None - block = text[start + len(BEGIN) : end] - paths: list[str] = [] - for line in block.splitlines(): - stripped = line.strip() - if not stripped.startswith("|"): + checkpoint = _load( + "doc_checkpoint_for_consistency", "scripts/check-doc-checkpoint.py" + ) + policy = load_policy(ROOT) if rules is None else rules + errors: list[str] = checkpoint.public_namespace_errors(policy) + seen_surfaces: dict[str, str] = {} + + for rule_id in checkpoint.PUBLIC_RULE_IDS: + rule = policy.get(rule_id) + if rule is None: + errors.append(f"public-document checker requires missing policy rule {rule_id}") continue - found = CELL_PATH.findall(stripped) - if found: - paths.append(found[0]) - return paths - - -def document_errors(name: str, text: str, expected: tuple[str, ...]) -> list[str]: - """Return contract problems for one normative document.""" - paths = contract_paths(text) - if paths is None: - return [ - f"{name} is missing the doc-obligation contract block " - f"({BEGIN} ... {END}); it must declare the surfaces that " - "scripts/check-doc-checkpoint.py enforces so prose cannot drift " - "from the gate" - ] + try: + binding = checkpoint.parse_public_rule(rule) + except ValueError as exc: + errors.append(str(exc)) + continue + previous = seen_surfaces.setdefault(binding.surface, rule_id) + if previous != rule_id: + errors.append( + f"public surface {binding.surface!r} is duplicate in " + f"{previous} and {rule_id}" + ) - errors: list[str] = [] - for path in paths: - if path in FORBIDDEN_IN_CONTRACT: - errors.append(f"{name} contract names {path}: {FORBIDDEN_IN_CONTRACT[path]}") - - if tuple(paths) != expected: - errors.append( - f"{name} contract declares {paths!r} but " - f"scripts/check-doc-checkpoint.py enforces {list(expected)!r}; " - "update the prose and the checker in the same change" - ) + if rules is None and not errors: + try: + bindings = checkpoint.public_rule_bindings(ROOT) + except ValueError as exc: + errors.extend(str(exc).splitlines()) + else: + if set(bindings) != set(checkpoint.PUBLIC_RULE_IDS): + errors.append("document-checker bindings omit a required public rule") return errors @@ -270,9 +226,8 @@ def prompt_errors(required: dict[str, tuple[str, ...]] | None = None) -> list[st def main() -> int: - expected = obligated_surfaces() failures: list[str] = [] - blocks: dict[str, list[str] | None] = {} + failures.extend(public_document_rule_errors()) interview = ROOT / INTERVIEW_DOCUMENT if not interview.exists(): @@ -293,33 +248,12 @@ def main() -> int: failures.extend(prompt_errors()) - for name in CONTRACT_DOCUMENTS: - path = ROOT / name - if not path.exists(): - failures.append(f"{name} does not exist") - continue - text = path.read_text(encoding="utf-8") - blocks[name] = contract_paths(text) - failures.extend(document_errors(name, text, expected)) - - present = {name: paths for name, paths in blocks.items() if paths is not None} - if len(present) == len(CONTRACT_DOCUMENTS): - distinct = {tuple(paths) for paths in present.values()} - if len(distinct) > 1: - failures.append( - "the doc-obligation contract differs between " - f"{' and '.join(CONTRACT_DOCUMENTS)}; both must carry the same " - "block verbatim" - ) - if failures: for failure in failures: print(f"ERROR: {failure}", file=sys.stderr) print( - "The obligated public surfaces are defined by PUBLIC_CHECKPOINTS and " - "FEATURE_CHECKPOINT in scripts/check-doc-checkpoint.py. Mirror them " - "in the contract block of every document listed in " - "CONTRACT_DOCUMENTS. The role interview is the block between " + "Public-document policy rows must fully parse through " + "scripts/check-doc-checkpoint.py. The role interview is the block between " f"{INTERVIEW_MARKER} and its :end in {INTERVIEW_DOCUMENT}; it must " "name every answer agent-role.py accepts. The operator's loop is " f"the block between {LOOP_MARKER} and its :end in {LOOP_DOCUMENT}; " @@ -333,9 +267,8 @@ def main() -> int: return 1 print( - "OK: the doc-obligation contract in " - f"{' and '.join(CONTRACT_DOCUMENTS)} matches " - f"scripts/check-doc-checkpoint.py, {INTERVIEW_DOCUMENT} carries the " + "OK: public-document policy matches scripts/check-doc-checkpoint.py, " + f"{INTERVIEW_DOCUMENT} carries the " f"role interview and the orchestration loop, and " f"{len(PROMPT_REQUIRED)} sub-agent prompts carry their binding " "instructions." diff --git a/tests/scripts/test_check_protocol_consistency.py b/tests/scripts/test_check_protocol_consistency.py index 6e75821f4..b8586fd05 100644 --- a/tests/scripts/test_check_protocol_consistency.py +++ b/tests/scripts/test_check_protocol_consistency.py @@ -10,6 +10,8 @@ from __future__ import annotations import contextlib +import csv +import dataclasses import importlib.util import io import re @@ -36,7 +38,18 @@ def _load(name: str, relative: str): consistency = _load("protocol_consistency", "scripts/check-protocol-consistency.py") -EXPECTED = ("docs/STATUS.md", "docs/BENCHMARKS.md", "docs/FEATURES.md") +EXPECTED_PUBLIC_RULES = { + "POL-DOC-STATUS": ("docs/STATUS.md", "feature_checkpoint", "Update"), + "POL-DOC-BENCHMARKS": ( + "docs/BENCHMARKS.md", + "feature_checkpoint", + "Update", + ), + "POL-DOC-FEATURES": ("docs/FEATURES.md", "feature_surface", "Update"), + "POL-DOC-USAGE": ("docs/USAGE.md", "user_usage", "Update"), + "POL-DOC-README": ("README.md", "landing_page", "Update"), + "POL-NOW-COUPLING": (".agents/NOW.md", "live_state", "Refresh"), +} def _tracked_paths(prefix: str) -> set[str] | None: @@ -93,7 +106,20 @@ def _repo_copy(workflow_text: str, *, prompts: bool = True): ROOT / "scripts/check-doc-checkpoint.py", root / "scripts/check-doc-checkpoint.py", ) - shutil.copy(ROOT / "AGENTS.md", root / "AGENTS.md") + shutil.copy(ROOT / ".agents/policy.csv", root / ".agents/policy.csv") + shutil.copy(ROOT / ".agents/waivers.csv", root / ".agents/waivers.csv") + # The policy parser validates every named checker and procedure. Create + # the exact declared paths so this fixture isolates workflow behavior. + with (ROOT / ".agents/policy.csv").open( + newline="", encoding="utf-8" + ) as stream: + for row in csv.DictReader(stream): + named = [*row["enforcement"].split(";"), row["procedure"]] + for relative in named: + target = root / relative.strip() + target.parent.mkdir(parents=True, exist_ok=True) + if not target.exists(): + target.write_text("# fixture\n", encoding="utf-8") if prompts: shutil.copytree(ROOT / ".agents/prompts", root / ".agents/prompts") (root / ".agents/workflow.md").write_text(workflow_text, encoding="utf-8") @@ -106,79 +132,262 @@ def _repo_copy(workflow_text: str, *, prompts: bool = True): consistency.ROOT = saved -def document(*paths: str) -> str: - rows = "\n".join(f"| `{path}` | every checkpoint |" for path in paths) - return "\n".join( - [ - "# Some normative document", - "", - consistency.BEGIN, - "| Public surface | Owed by |", - "|---|---|", - rows, - consistency.END, - "", - "Trailing prose.", - ] - ) +class PublicDocumentPolicyTests(unittest.TestCase): + def _mutated_errors(self, rule_id: str, **changes: str) -> list[str]: + rules = consistency.load_policy(ROOT) + rules[rule_id] = dataclasses.replace(rules[rule_id], **changes) + return consistency.public_document_rule_errors(rules) + + def _extra_rule_errors(self, rule_id: str, **changes: str) -> list[str]: + rules = consistency.load_policy(ROOT) + source = rules["POL-PR-REQUIRED"] + fields = { + "rule_id": rule_id, + "scope": "internal review", + "trigger": "policy review", + "requirement": "Review the policy change.", + } + fields.update(changes) + rules[rule_id] = dataclasses.replace(source, **fields) + return consistency.public_document_rule_errors(rules) + def test_repository_policy_matches_the_checker_semantically(self) -> None: + self.assertEqual(consistency.public_document_rule_errors(), []) -class ContractParsing(unittest.TestCase): - def test_extracts_paths_in_order(self) -> None: - self.assertEqual( - consistency.contract_paths(document(*EXPECTED)), list(EXPECTED) + def test_main_enforces_the_public_document_mapping(self) -> None: + self.assertEqual(consistency.main(), 0) + + def test_missing_public_rule_is_rejected(self) -> None: + rules = consistency.load_policy(ROOT) + rules.pop("POL-DOC-USAGE") + errors = consistency.public_document_rule_errors(rules) + self.assertTrue(any("POL-DOC-USAGE" in error for error in errors), errors) + + def test_unknown_trigger_identifier_is_rejected(self) -> None: + errors = self._mutated_errors("POL-DOC-STATUS", trigger="mutable_tree") + self.assertTrue(any("mutable_tree" in error for error in errors), errors) + + def test_public_rule_must_name_the_document_checker(self) -> None: + errors = self._mutated_errors( + "POL-DOC-USAGE", enforcement="scripts/check-policy.py" ) + self.assertTrue(any("POL-DOC-USAGE" in e and "enforcement" in e for e in errors), errors) - def test_absent_block_is_none(self) -> None: - self.assertIsNone(consistency.contract_paths("# No contract here")) + def test_partial_enforcement_content_is_rejected(self) -> None: + errors = self._mutated_errors( + "POL-DOC-USAGE", + enforcement="scripts/check-doc-checkpoint.py; scripts/check-policy.py; later", + ) + self.assertTrue(any("POL-DOC-USAGE" in e and "enforcement" in e for e in errors), errors) - def test_end_before_begin_is_none(self) -> None: - text = f"{consistency.END}\n| `docs/STATUS.md` |\n{consistency.BEGIN}" - self.assertIsNone(consistency.contract_paths(text)) + def test_duplicate_public_surface_is_rejected(self) -> None: + rules = consistency.load_policy(ROOT) + rules["POL-DOC-USAGE"] = dataclasses.replace( + rules["POL-DOC-USAGE"], scope=rules["POL-DOC-STATUS"].scope + ) + errors = consistency.public_document_rule_errors(rules) + self.assertTrue( + any("POL-DOC-USAGE" in e and "scope" in e for e in errors), errors + ) + def test_each_public_rule_accepts_only_exact_positive_requirement(self) -> None: + for rule_id, (scope, _trigger, verb) in EXPECTED_PUBLIC_RULES.items(): + original = consistency.load_policy(ROOT)[rule_id] + mutations = { + "wrong action": f"Observe {scope}.", + "wrong target": f"{verb} another-page.md.", + "except suffix": original.requirement.rstrip(".") + + " except when state is appended.", + "unless suffix": original.requirement.rstrip(".") + + " unless the change is small.", + "other than suffix": original.requirement.rstrip(".") + + " other than for releases.", + "without suffix": original.requirement.rstrip(".") + + " without benchmark changes.", + "generic trailing text": original.requirement + " Extra words", + "missing period": f"{verb} {scope}", + } + for label, requirement in mutations.items(): + with self.subTest(rule_id=rule_id, mutation=label): + errors = self._mutated_errors(rule_id, requirement=requirement) + self.assertTrue( + any(rule_id in error and "requirement" in error for error in errors), + errors, + ) -class Mutations(unittest.TestCase): - def test_baseline_passes(self) -> None: - self.assertEqual( - consistency.document_errors("doc", document(*EXPECTED), EXPECTED), [] + def test_public_rule_rejects_unparsed_scope_content(self) -> None: + errors = self._mutated_errors( + "POL-DOC-STATUS", scope="docs/STATUS.md except docs/legacy.md" ) + self.assertTrue(any("POL-DOC-STATUS" in e and "scope" in e for e in errors), errors) - def test_missing_block_is_rejected(self) -> None: - errors = consistency.document_errors("doc", "# nothing", EXPECTED) - self.assertTrue(any("missing the doc-obligation contract" in e for e in errors)) - - def test_readme_in_contract_is_rejected_by_name(self) -> None: - """The exact historical regression: README named as a checkpoint.""" - text = document("README.md", "docs/BENCHMARKS.md", "docs/FEATURES.md") - errors = consistency.document_errors("doc", text, EXPECTED) - self.assertTrue(any("README.md" in e and "landing page" in e for e in errors)) - - def test_dropped_surface_is_rejected(self) -> None: - text = document("docs/STATUS.md", "docs/BENCHMARKS.md") - errors = consistency.document_errors("doc", text, EXPECTED) - self.assertTrue(any("enforces" in e for e in errors)) - - def test_reordered_surfaces_are_rejected(self) -> None: - text = document("docs/BENCHMARKS.md", "docs/STATUS.md", "docs/FEATURES.md") - self.assertNotEqual( - consistency.document_errors("doc", text, EXPECTED), [] + def test_public_rule_rejects_unparsed_trigger_content(self) -> None: + errors = self._mutated_errors( + "POL-DOC-STATUS", trigger="feature_checkpoint except docs-only" ) + self.assertTrue(any("POL-DOC-STATUS" in e and "trigger" in e for e in errors), errors) - def test_extra_surface_is_rejected(self) -> None: - text = document(*EXPECTED, "docs/USAGE.md") - self.assertNotEqual(consistency.document_errors("doc", text, EXPECTED), []) + def test_extra_rule_cannot_reuse_a_public_trigger(self) -> None: + errors = self._extra_rule_errors( + "POL-EXTRA-TRIGGER", trigger="feature_checkpoint" + ) + self.assertTrue( + any("POL-EXTRA-TRIGGER" in e and "trigger" in e for e in errors), errors + ) + def test_extra_rule_cannot_reuse_a_public_scope(self) -> None: + errors = self._extra_rule_errors( + "POL-EXTRA-SCOPE", scope="docs/STATUS.md" + ) + self.assertTrue( + any("POL-EXTRA-SCOPE" in e and "scope" in e for e in errors), errors + ) -class LiveTree(unittest.TestCase): - def test_expected_surfaces_come_from_the_checker(self) -> None: - self.assertEqual(consistency.obligated_surfaces(), EXPECTED) + def test_extra_rule_cannot_target_a_public_surface_in_its_requirement(self) -> None: + errors = self._extra_rule_errors( + "POL-EXTRA-REQUIREMENT", requirement="Update docs/STATUS.md." + ) + self.assertTrue( + any( + "POL-EXTRA-REQUIREMENT" in e and "requirement" in e + for e in errors + ), + errors, + ) - def test_repository_contract_is_consistent(self) -> None: - self.assertEqual(consistency.main(), 0) + def test_malformed_positive_requirements_cannot_bypass_public_target_ownership( + self, + ) -> None: + mutations = { + "double terminal period": "Update docs/STATUS.md..", + "triple terminal period": "Update docs/STATUS.md...", + "segment ending period": "Update docs./STATUS.md.", + "empty segment": "Update docs//STATUS.md.", + "dot segment": "Update docs/./STATUS.md.", + "dot-dot segment": "Update docs/../STATUS.md.", + "parent traversal": "Update ../docs/STATUS.md.", + "absolute path": "Update /docs/STATUS.md.", + "backslash separator": r"Update docs\STATUS.md.", + "space inside token": "Update docs/STATUS file.md.", + "tab inside token": "Update docs/STATUS\tfile.md.", + "leading space": " Update docs/STATUS.md.", + "trailing space": "Update docs/STATUS.md. ", + "double space": "Update docs/STATUS.md.", + "tab delimiter": "Update\tdocs/STATUS.md.", + "nbsp delimiter": "Update\u00a0docs/STATUS.md.", + "inserted token": "Update only docs/STATUS.md.", + "carriage return delimiter": "Update\rdocs/STATUS.md.", + "line feed delimiter": "Update\ndocs/STATUS.md.", + "second sentence": ( + "Update docs/STATUS.md. Refresh docs/BENCHMARKS.md." + ), + "second delimiter": "Update docs/STATUS.md.;docs/BENCHMARKS.md.", + } + for label, requirement in mutations.items(): + with self.subTest(mutation=label): + errors = self._extra_rule_errors( + "POL-EXTRA-MALFORMED", requirement=requirement + ) + self.assertTrue( + any( + "POL-EXTRA-MALFORMED" in error + and "requirement" in error + for error in errors + ), + errors, + ) - def test_every_contract_document_exists(self) -> None: - for name in consistency.CONTRACT_DOCUMENTS: - self.assertTrue((ROOT / name).exists(), name) + def test_reserved_target_scan_is_independent_of_positive_grammar(self) -> None: + requirements = ( + "Review docs/STATUS.md before release.", + " Review docs/STATUS.md.", + "Update docs/STATUS.md.", + "Update\tdocs/STATUS.md.", + "Update\u00a0docs/STATUS.md.", + "Update only docs/STATUS.md.", + "Review (docs/STATUS.md).", + "Update\r\ndocs/STATUS.md.", + ) + for requirement in requirements: + with self.subTest(requirement=repr(requirement)): + errors = self._extra_rule_errors( + "POL-EXTRA-LEXICAL", requirement=requirement + ) + self.assertTrue( + any( + "POL-EXTRA-LEXICAL" in error + and "requirement target" in error + for error in errors + ), + errors, + ) + + def test_reserved_target_scan_avoids_path_prefix_collisions(self) -> None: + # These controls independently pin both lexical boundaries. Removing + # the left-boundary check makes the nested/prefixed paths collide; + # removing the right-boundary check makes the suffixed paths collide. + requirements = ( + "Review sub/docs/STATUS.md before release.", + "Review prefix/docs/STATUS.md before release.", + "Review docs/STATUS.md.extra before release.", + "Review .agents/NOW.md.extra before release.", + ) + for requirement in requirements: + with self.subTest(requirement=requirement): + errors = self._extra_rule_errors( + "POL-EXTRA-PREFIX", requirement=requirement + ) + self.assertFalse( + any("requirement target" in error for error in errors), errors + ) + + self._assert_reserved_target_scan_accepts_non_path_delimiters() + + def _assert_reserved_target_scan_accepts_non_path_delimiters(self) -> None: + # Exact references remain reserved when ordinary prose punctuation is + # adjacent. Inverting either boundary predicate makes at least one of + # these positive controls disappear from the ownership scan. + requirements = ( + 'Review "docs/STATUS.md" before release.', + "Review `docs/STATUS.md` before release.", + "Review (docs/STATUS.md) before release.", + "Review docs/STATUS.md: before release.", + "Review docs/STATUS.md, before release.", + "Review docs/STATUS.md; before release.", + "Review docs/STATUS.md! before release.", + "Review docs/STATUS.md? before release.", + ) + for requirement in requirements: + with self.subTest(requirement=requirement): + errors = self._extra_rule_errors( + "POL-EXTRA-DELIMITED", requirement=requirement + ) + self.assertTrue( + any( + "POL-EXTRA-DELIMITED" in error + and "requirement target" in error + for error in errors + ), + errors, + ) + + def test_extra_rule_cannot_duplicate_a_public_semantic_binding(self) -> None: + errors = self._extra_rule_errors( + "POL-EXTRA-BINDING", + scope="docs/STATUS.md", + trigger="feature_checkpoint", + requirement="Update docs/STATUS.md.", + ) + self.assertTrue( + any("POL-EXTRA-BINDING" in e and "duplicate" in e for e in errors), + errors, + ) + + def test_extra_rule_cannot_claim_the_reserved_public_rule_namespace(self) -> None: + errors = self._extra_rule_errors("POL-DOC-EXTRA") + self.assertTrue( + any("POL-DOC-EXTRA" in e and "reserved" in e for e in errors), errors + ) class InterviewBlockTests(unittest.TestCase): diff --git a/tests/scripts/test_doc_checkpoint.py b/tests/scripts/test_doc_checkpoint.py index c16dea4be..cbbcb5ade 100755 --- a/tests/scripts/test_doc_checkpoint.py +++ b/tests/scripts/test_doc_checkpoint.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Unit checks for scripts/check-doc-checkpoint.py.""" +"""Behavior and mutation checks for purpose-specific public documentation.""" from __future__ import annotations @@ -18,97 +18,291 @@ SPEC.loader.exec_module(doc_checkpoint) -class DocumentationCheckpointTests(unittest.TestCase): - def test_production_change_requires_both_public_documents(self) -> None: - errors = doc_checkpoint.checkpoint_errors({"src/vllm/example.cpp"}) - self.assertEqual(len(errors), 2) - self.assertIn("docs/STATUS.md", errors[0]) - self.assertIn("docs/BENCHMARKS.md", errors[1]) +class RequirementParserTests(unittest.TestCase): + def test_positive_action_and_repository_path_are_parsed(self) -> None: + for requirement, expected in ( + ("Update docs/STATUS.md.", ("Update", "docs/STATUS.md")), + ("Refresh .agents/NOW.md.", ("Refresh", ".agents/NOW.md")), + ( + "Update docs/segment-/segment_.", + ("Update", "docs/segment-/segment_"), + ), + ( + "Update docs/name_with-dash.v2.md.", + ("Update", "docs/name_with-dash.v2.md"), + ), + ): + with self.subTest(requirement=requirement): + self.assertEqual( + doc_checkpoint.parse_requirement(requirement), expected + ) + + def test_malformed_positive_requirement_is_rejected_before_target_use( + self, + ) -> None: + invalid = ( + "Update docs/STATUS.md..", + "Update docs/STATUS.md...", + "Update docs./STATUS.md.", + "Update docs/STATUS./index.md.", + "Update docs//STATUS.md.", + "Update docs/./STATUS.md.", + "Update docs/../STATUS.md.", + "Update ../docs/STATUS.md.", + "Update /docs/STATUS.md.", + r"Update docs\STATUS.md.", + "Update docs/STATUS file.md.", + "Update docs/STATUS\tfile.md.", + "Update docs/STATUS.md. Refresh docs/BENCHMARKS.md.", + "Update docs/STATUS.md.;docs/BENCHMARKS.md.", + "Update docs/STATUS?.md.", + "Update docs/STATUS%.md.", + "Update docs/STATUS:md.", + "Update docs/STATUS\x00md.", + "Update docs/STATUS\x1fmd.", + "Update docs/STAT\u00a0US.md.", + "Update docs/STAT\u00e9US.md.", + "Observe docs/STATUS.md.", + "Update docs/STATUS.md", + " Update docs/STATUS.md.", + "Update docs/STATUS.md.", + "Update\tdocs/STATUS.md.", + "Update\u00a0docs/STATUS.md.", + "Update only docs/STATUS.md.", + "Update\r\ndocs/STATUS.md.", + "Update docs/STATUS.md.\n", + "Update docs/STATUS.md.\r", + "Update docs/STATUS.md. ", + ) + for requirement in invalid: + with self.subTest(requirement=requirement): + with self.assertRaises(ValueError): + doc_checkpoint.parse_requirement(requirement) + + def test_repository_path_alphabet_is_closed_and_exact(self) -> None: + expected = frozenset( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" + "0123456789._-" + ) + self.assertEqual(doc_checkpoint.REPOSITORY_PATH_CHARACTERS, expected) + + def test_path_alphabet_mutations_accepting_forbidden_classes_go_red(self) -> None: + forbidden = ( + "%", + ":", + " ", + "\t", + "\r", + "\n", + "\x00", + "\x1f", + "\u00a0", + "\u00e9", + ) + for character in forbidden: + with self.subTest(character=repr(character)): + self.assertNotIn(character, doc_checkpoint.REPOSITORY_PATH_CHARACTERS) + + +class SemanticClassificationTests(unittest.TestCase): + def test_runtime_code_is_a_feature_checkpoint(self) -> None: + self.assertIn( + "feature_checkpoint", + doc_checkpoint.classify_changed_paths(["src/vt/cuda/matmul.cu"]), + ) - def test_one_public_document_does_not_satisfy_checkpoint(self) -> None: - errors = doc_checkpoint.checkpoint_errors( - {"tests/vllm/test_example.cpp", "docs/STATUS.md"} + def test_unlisted_runtime_code_remains_a_checkpoint(self) -> None: + self.assertIn( + "feature_checkpoint", + doc_checkpoint.classify_changed_paths(["scripts/benchmark-grid.py"]), ) - self.assertEqual(len(errors), 1) - self.assertIn("docs/BENCHMARKS.md", errors[0]) - def test_lifecycle_and_diagnostic_records_are_checkpoints(self) -> None: + def test_runtime_checkers_are_not_hidden_by_governance_names(self) -> None: for path in ( - ".agents/roadmap_v1.md", - ".agents/state.md", - ".agents/specs/example.md", - "tools/bench/example.py", + "scripts/check-gemv-invocation-consistency.py", + "tests/scripts/test_check_gemv_invocation_consistency.py", ): with self.subTest(path=path): - self.assertTrue(doc_checkpoint.checkpoint_errors({path})) + classes = doc_checkpoint.classify_changed_paths([path]) + self.assertIn("feature_checkpoint", classes) + self.assertNotIn("governance", classes) - def test_both_public_documents_satisfy_checkpoint(self) -> None: - self.assertEqual( - doc_checkpoint.checkpoint_errors( - { - ".agents/engine-matrix.md", - "docs/STATUS.md", - "docs/BENCHMARKS.md", - } - ), - [], - ) + def test_exact_governance_checker_files_are_exempt(self) -> None: + for path in ( + "scripts/check-policy.py", + "scripts/check-doc-checkpoint.py", + "scripts/check-protocol-consistency.py", + "tests/scripts/test_doc_checkpoint.py", + "tests/scripts/test_check_protocol_consistency.py", + ): + with self.subTest(path=path): + self.assertEqual( + doc_checkpoint.classify_changed_paths([path]), {"governance"} + ) - def test_unrelated_documentation_correction_is_not_a_checkpoint(self) -> None: - self.assertEqual( - doc_checkpoint.checkpoint_errors({"docs/design-notes.md"}), [] - ) + def test_governance_only_task_one_files_are_not_a_checkpoint(self) -> None: + paths = [ + ".agents/policy.csv", + ".agents/waivers.csv", + "scripts/policy_contract.py", + "scripts/check-policy.py", + "tests/scripts/test_policy_contract.py", + ] + self.assertEqual(doc_checkpoint.classify_changed_paths(paths), {"governance"}) + self.assertEqual(doc_checkpoint.checkpoint_errors(set(paths)), []) + def test_governance_design_is_not_misclassified_as_feature_work(self) -> None: + path = "docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md" + self.assertEqual(doc_checkpoint.classify_changed_paths([path]), {"governance"}) + self.assertEqual(doc_checkpoint.checkpoint_errors({path}), []) -class FeatureSurfaceCheckpointTests(unittest.TestCase): - """docs/FEATURES.md is the public mirror of the area matrices.""" + def test_feature_support_paths_have_both_semantic_classes(self) -> None: + classes = doc_checkpoint.classify_changed_paths( + ["src/vllm/model_executor/models/qwen3.cpp"] + ) + self.assertEqual(classes, {"feature_checkpoint", "feature_surface"}) - def test_feature_matrix_change_requires_the_public_feature_page(self) -> None: - errors = doc_checkpoint.checkpoint_errors( - {".agents/feature-matrix.md", "docs/STATUS.md", "docs/BENCHMARKS.md"} + def test_user_interface_paths_are_usage_changes_and_checkpoints(self) -> None: + classes = doc_checkpoint.classify_changed_paths( + ["src/vllm/entrypoints/openai/api_server.cpp"] ) - self.assertEqual(len(errors), 1) - self.assertIn("docs/FEATURES.md", errors[0]) + self.assertEqual(classes, {"feature_checkpoint", "user_usage"}) - def test_every_area_matrix_triggers_the_feature_page(self) -> None: + def test_configuration_and_exact_install_sources_require_usage(self) -> None: for path in ( - ".agents/feature-matrix.md", - ".agents/model-matrix.md", - ".agents/backend-matrix.md", - ".agents/quantization-matrix.md", - "src/vllm/model_executor/models/qwen3.cpp", + ".env.example", + "CMakeLists.txt", + "cmake/install.cmake", ): with self.subTest(path=path): - self.assertTrue(doc_checkpoint.is_feature_path(path)) + classes = doc_checkpoint.classify_changed_paths([path]) + self.assertIn("feature_checkpoint", classes) + self.assertIn("user_usage", classes) - def test_updating_the_feature_page_satisfies_the_obligation(self) -> None: - self.assertEqual( - doc_checkpoint.checkpoint_errors( - { - ".agents/model-matrix.md", - "docs/STATUS.md", - "docs/BENCHMARKS.md", - "docs/FEATURES.md", - } - ), - [], + def test_unrelated_cmake_module_is_not_automatically_install_usage(self) -> None: + classes = doc_checkpoint.classify_changed_paths(["cmake/FindNVTX.cmake"]) + self.assertIn("feature_checkpoint", classes) + self.assertNotIn("user_usage", classes) + + def test_state_append_is_live_state_and_checkpoint(self) -> None: + classes = doc_checkpoint.classify_changed_paths([".agents/state.md"]) + self.assertEqual(classes, {"feature_checkpoint", "live_state"}) + + +class RequiredSurfaceTests(unittest.TestCase): + def assertMissing(self, paths: set[str], surface: str) -> None: + errors = doc_checkpoint.checkpoint_errors(paths) + self.assertTrue(any(surface in error for error in errors), errors) + + def test_feature_checkpoint_requires_status_and_benchmarks(self) -> None: + paths = {"src/vt/cuda/matmul.cu"} + self.assertMissing(paths, "docs/STATUS.md") + self.assertMissing(paths, "docs/BENCHMARKS.md") + + def test_each_checkpoint_surface_is_independently_required(self) -> None: + self.assertMissing( + {"src/vt/cuda/matmul.cu", "docs/STATUS.md"}, "docs/BENCHMARKS.md" + ) + self.assertMissing( + {"src/vt/cuda/matmul.cu", "docs/BENCHMARKS.md"}, "docs/STATUS.md" + ) + + def test_feature_surface_requires_features(self) -> None: + self.assertMissing( + { + ".agents/model-matrix.md", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + }, + "docs/FEATURES.md", + ) + + def test_usage_change_requires_usage(self) -> None: + self.assertMissing( + { + "examples/cli/main.cpp", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + }, + "docs/USAGE.md", ) - def test_non_feature_code_does_not_owe_the_feature_page(self) -> None: - # A scheduler or kernel change is a checkpoint, but it moves no public - # feature row, so it must not be forced to touch FEATURES.md. - errors = doc_checkpoint.checkpoint_errors( - {"src/vt/cuda/matmul.cu", "docs/STATUS.md", "docs/BENCHMARKS.md"} + def test_state_append_requires_fresh_now(self) -> None: + self.assertMissing( + { + ".agents/state.md", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + }, + ".agents/NOW.md", ) - self.assertEqual(errors, []) - def test_feature_page_alone_is_not_a_general_checkpoint_surface(self) -> None: - # Editing FEATURES.md must not satisfy the STATUS/BENCHMARKS obligation. - errors = doc_checkpoint.checkpoint_errors( - {".agents/model-matrix.md", "docs/FEATURES.md"} + def test_readme_churn_without_landing_trigger_is_rejected(self) -> None: + errors = doc_checkpoint.checkpoint_errors({"README.md"}) + self.assertTrue(any("README.md" in error and "trigger" in error for error in errors)) + + def test_readme_is_not_justified_by_coedited_public_projections(self) -> None: + paths = { + "README.md", + "docs/USAGE.md", + "docs/FEATURES.md", + "docs/BENCHMARKS.md", + "docs/STATUS.md", + } + errors = doc_checkpoint.checkpoint_errors(paths) + self.assertTrue(any("README.md" in error for error in errors), errors) + + def test_readme_with_explicit_install_source_is_allowed(self) -> None: + paths = { + "README.md", + "CMakeLists.txt", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/USAGE.md", + } + self.assertEqual(doc_checkpoint.checkpoint_errors(paths), []) + + def test_explicit_landing_source_requires_readme_projection(self) -> None: + paths = { + "CMakeLists.txt", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/USAGE.md", + } + self.assertMissing(paths, "README.md") + + def test_all_required_projections_satisfy_a_user_feature_change(self) -> None: + paths = { + "src/vllm/model_executor/models/qwen3.cpp", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/FEATURES.md", + } + self.assertEqual(doc_checkpoint.checkpoint_errors(paths), []) + + +class PolicyMappingTests(unittest.TestCase): + def test_required_surfaces_are_driven_by_public_policy_rules(self) -> None: + self.assertEqual( + doc_checkpoint.required_public_surfaces( + {"feature_checkpoint", "feature_surface", "user_usage", "live_state"} + ), + { + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/FEATURES.md", + "docs/USAGE.md", + ".agents/NOW.md", + }, ) - self.assertTrue(any("docs/STATUS.md" in e for e in errors), errors) - self.assertTrue(any("docs/BENCHMARKS.md" in e for e in errors), errors) + + def test_every_public_rule_is_completely_parsed(self) -> None: + for rule_id, rule in doc_checkpoint.public_document_rules().items(): + with self.subTest(rule_id=rule_id): + binding = doc_checkpoint.parse_public_rule(rule) + self.assertEqual(binding.rule_id, rule_id) + self.assertEqual(binding.surface, rule.scope) + self.assertEqual(binding.change_class, rule.trigger) if __name__ == "__main__": From c74a930ca162c2d39ba744737e8937a956cb4839 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 01:15:08 +0000 Subject: [PATCH 05/12] policy(prompts): close runtime prompt namespace FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/policy.csv | 4 +- .agents/prompts/implementer.md | 69 ++-- .agents/prompts/operator.md | 33 ++ .agents/prompts/reviewer.md | 87 ++--- scripts/check-doc-checkpoint.py | 3 + scripts/check-prompt-contract.py | 286 ++++++++++++++ scripts/check-protocol-consistency.py | 86 +---- tests/scripts/test_check_prompt_contract.py | 348 ++++++++++++++++++ .../test_check_protocol_consistency.py | 243 ++++++------ tests/scripts/test_doc_checkpoint.py | 5 + 10 files changed, 878 insertions(+), 286 deletions(-) create mode 100644 .agents/prompts/operator.md create mode 100644 scripts/check-prompt-contract.py create mode 100644 tests/scripts/test_check_prompt_contract.py diff --git a/.agents/policy.csv b/.agents/policy.csv index e7e3c056c..01bbc5510 100644 --- a/.agents/policy.csv +++ b/.agents/policy.csv @@ -54,5 +54,5 @@ POL-PR-SIZE,pull request,before ready,Keep each explicit path class within its r POL-READY-SPEC,helper queue,advertising a task as ready,Prove the spec is committed and reachable and the declared gate has a failing mutation.,scripts/check-gate-commands.py; scripts/check-policy.py,never,.agents/coordination.md POL-READY-HARDWARE,helper queue,advertising a task as ready,Declare CPU gateability or the exact required hardware.,scripts/check-gate-commands.py; scripts/check-policy.py,never,.agents/coordination.md POL-READY-DEPS,helper queue,advertising a task as ready,Prove parsed dependencies are satisfied and no live claim exists.,scripts/check-policy.py,never,.agents/coordination.md -POL-PROMPT-ENVELOPE,delegated task,dispatching an agent,Supply goal context constraints done-when required-evidence and authority.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md -POL-PROMPT-BOUNDARIES,runtime prompt,changing an agent prompt,Keep prompts tool-neutral versioned bounded and explicit about method output and stop conditions.,scripts/check-protocol-consistency.py; scripts/check-policy.py,never,.agents/workflow.md +POL-PROMPT-ENVELOPE,delegated task,dispatching an agent,Supply goal context constraints done-when required-evidence and authority or return NEEDS_CONTEXT.,scripts/check-prompt-contract.py; scripts/check-policy.py,never,.agents/workflow.md +POL-PROMPT-BOUNDARIES,runtime prompt,changing an agent prompt,Keep prompts versioned bounded tool-neutral and fully parsed with explicit method output and stop rows.,scripts/check-prompt-contract.py; scripts/check-policy.py,never,.agents/workflow.md diff --git a/.agents/prompts/implementer.md b/.agents/prompts/implementer.md index c254419ea..334e64815 100644 --- a/.agents/prompts/implementer.md +++ b/.agents/prompts/implementer.md @@ -1,40 +1,37 @@ -# Implementer prompt - -You implement one task. A different agent will review it by mutating your code. +--- +prompt-contract-version: 1 +role: implementer +policy-rules: POL-PROMPT-ENVELOPE POL-PROMPT-BOUNDARIES +--- +## Task envelope +- Goal: REQUIRED +- Context: REQUIRED +- Constraints: REQUIRED +- Done when: REQUIRED +- Required evidence: REQUIRED +- Authority: REQUIRED +- Missing input: NEEDS_CONTEXT ## Method +- `IMP-TEST-FIRST` | required | Write the failing test first, run it before implementation, and confirm the stated failure. +- `IMP-VERIFY` | required | Run every declared verification and require exit zero or an unchanged proven baseline. +- `IMP-MUTATE` | required | Delete or invert each behavior named by every added test, require its focused suite to fail, then restore it. +- `IMP-SCOPE` | forbidden | Change files or behavior outside Authority. +- `IMP-EVIDENCE` | evidence | Record each command, exit status, negative mutation, observed failure, and restoration check. -1. Write the failing test first. Run it. Confirm it fails **for the stated - reason**: a test that fails for the wrong reason pins nothing. -2. Implement the minimum that makes it pass. -3. **Mutate every test you wrote**: delete the line it names, confirm red, - restore. Report the results. If a briefed test does not pin what it claims, - fix it and say so; four implementers before you did exactly that and were - right every time. Read that four as a **dated floor** (2026-08-06), not a - running total: it can only grow, and growing never weakens the rule. -4. Run the project gate (`scripts/agent-preflight.sh`, redirected to a file, - never piped) and confirm `EXIT=0`. When a gate is ALREADY red before you - touch anything, capture that failing set as a baseline FIRST: you are green - when the failing set after your change is identical to it. Name the carried - reds in your report. A gate you did not break is not yours to allowlist, and - reaching a green banner is never a reason to weaken one. -5. Commit in your worktree with the required trailers, and return the SHA. - -## Honesty rules - -- **Never let a failure and an absence look the same.** Every recorded defect - class in this repo is that bug: a substring `--grep` crediting a row with - another row's commits, `.get()` on a missing key reporting a live claim as - finished, a git failure mapped to `""` and read as "no evidence". -- **Report what you did not do.** An empty concerns section is itself a claim. -- **Escalate rather than guess.** Report `BLOCKED` or `NEEDS_CONTEXT` with - specifics. Bad work is worse than no work, and you will not be penalised for - stopping. -- **Never weaken a checker, a budget or a test to make something pass.** If the - gate is red, repair the record. - -## Deviating from the brief +## Required output +- status: COMPLETE | BLOCKED | NEEDS_CONTEXT | NEEDS_DECISION +- summary: REQUIRED +- changed_files: LIST +- commands_and_exit_status: EVIDENCE +- negative_mutation: EVIDENCE +- deviations_and_waivers: EVIDENCE | NONE +- risks: EVIDENCE | NONE +- omitted_gates: EVIDENCE | NONE +- commit_sha: SHA | NONE -You may deviate when the brief is wrong, and it sometimes is. State the -deviation explicitly in your report with the evidence that justifies it. Silent -scope expansion is a defect; a disclosed, argued correction is not. +## Stop conditions +- `STOP-AUTHORITY` | BLOCKED | An edit outside Authority is required. +- `STOP-VERIFY` | BLOCKED | Declared verification cannot pass or match a proven baseline. +- `STOP-CONTEXT` | NEEDS_CONTEXT | Missing input changes the contract or public behavior. +- `STOP-DECISION` | NEEDS_DECISION | A required choice exceeds Authority. diff --git a/.agents/prompts/operator.md b/.agents/prompts/operator.md new file mode 100644 index 000000000..eab58fb11 --- /dev/null +++ b/.agents/prompts/operator.md @@ -0,0 +1,33 @@ +--- +prompt-contract-version: 1 +role: operator +policy-rules: POL-PROMPT-ENVELOPE POL-PROMPT-BOUNDARIES POL-OPERATOR-BOUNDARY POL-OPERATOR-VERIFY POL-PR-DISPOSITION POL-REVIEW-FRESH POL-REVIEW-NO-REPAIR POL-REMOTE-UNKNOWN +--- +## Task envelope +- Goal: REQUIRED +- Context: REQUIRED +- Constraints: REQUIRED +- Done when: REQUIRED +- Required evidence: REQUIRED +- Authority: REQUIRED +- Missing input: NEEDS_CONTEXT + +## Method +- `OP-DELEGATE` | required | Delegate implementation and repairs to fresh implementers. +- `OP-VERIFY` | required | Run claimed verification on the returned commit without trusting the implementer report. +- `OP-REVIEW` | required | Dispatch a fresh reviewer for independent static review and targeted scratch mutation. +- `OP-DISPOSITION` | required | Merge a verified PR in-session or close an obsolete PR with its recorded reason. +- `OP-REPAIR` | forbidden | Repair an implementer or reviewer finding in the coordinating context. +- `OP-EVIDENCE` | evidence | Record verification, review, PR disposition, blocker, and remote-state evidence. + +## Required output +- status: MERGED | CLOSED | BLOCKED | REMOTE_UNVERIFIED +- verification: EVIDENCE +- review: EVIDENCE +- disposition: EVIDENCE +- remaining_concern: EVIDENCE | NONE + +## Stop conditions +- `STOP-AUTHORITY` | BLOCKED | A required action exceeds Authority. +- `STOP-REMOTE` | REMOTE_UNVERIFIED | Required remote state cannot be queried. +- `STOP-BLOCKER` | BLOCKED | Non-terminal work cannot name its blocker on the PR. diff --git a/.agents/prompts/reviewer.md b/.agents/prompts/reviewer.md index 92818db7c..16ae6f74d 100644 --- a/.agents/prompts/reviewer.md +++ b/.agents/prompts/reviewer.md @@ -1,54 +1,33 @@ -# Reviewer prompt - -You review one change. You did not write it and you will not fix it. - -## The binding instruction: mutate, don't read - -For each test in the change, **delete or invert the line it names and re-run -the suite. A test that stays green is a finding**, regardless of how it reads. - -This is not a style preference. In the two branches audited to 2026-08, eleven -tests passed with the thing they named deleted, including a gate's own -default (an unrelated line satisfied the assertion), a probe with five -hardcoded fields, and `assertIn("merged", reason)` where the string was -`"unmerged"`. **None was visible by reading the diff.** A reviewer who reads -and comments on style adds nothing this project has not already paid for. - -## Do not trust the report - -Treat the implementer's report as unverified claims about the code. Verify each -against the change. A stated rationale ("kept it simple deliberately", "left -it per YAGNI") is the implementer grading its own work and **never** downgrades -a finding's severity. In that same audit, three implementer reports asserted -something false in good faith; each was caught by reproducing the claim rather -than accepting it. Read every count on this page as a dated floor, not a -running total: it can only grow, and growing never weakens the rule. - -## A plan-mandated finding is still a finding - -Roughly half of all Important findings on the preceding branches were defects in -the **plan text**, not the implementation. A reviewer that treats the plan as -authority cannot find them. Report them, labelled `plan-mandated`, and let the -human decide which governs. - -## Severity - -- **Critical**: corrupts the record, weakens a gate, or leaves a false claim in - a document agents read. -- **Important**: the change cannot be trusted until fixed: wrong or fragile - behavior, a missed requirement, a test that asserts nothing. -- **Minor**: polish. - -Cite `file:line` for every finding and for any check you would otherwise answer -with a bare "yes". Acknowledge what was done well before listing issues. - -## What you may not do - -- Never fix what you found. Findings go back to a fresh implementer. -- Never mutate the reviewed worktree, its index, HEAD or branch state. Work in a - scratch copy. -- Never re-run the full suite merely to reproduce the report's green result; - that confirms nothing the report already claims. This is NOT a budget on - mutation: every mutation you make re-runs the suite, and a review that made - none has not started. Reading may prompt an extra focused check, but reading - is never what decides whether to check. +--- +prompt-contract-version: 1 +role: reviewer +policy-rules: POL-PROMPT-ENVELOPE POL-PROMPT-BOUNDARIES POL-REVIEW-FRESH POL-REVIEW-NO-REPAIR +--- +## Task envelope +- Goal: REQUIRED +- Context: REQUIRED +- Constraints: REQUIRED +- Done when: REQUIRED +- Required evidence: REQUIRED +- Authority: REQUIRED +- Missing input: NEEDS_CONTEXT + +## Method +- `REV-STATIC` | required | Perform independent static review of requirements, plan, diff, and relevant surrounding code. +- `REV-MUTATION` | required | In a scratch copy, delete or invert each behavior changed tests claim to pin and rerun its focused test. +- `REV-FULL-GATE` | required | Run the full declared gate exactly once on the unchanged reviewed commit. +- `REV-NO-REPAIR` | forbidden | Repair a finding. +- `REV-WORKTREE` | forbidden | Mutate the reviewed worktree, index, HEAD, or branch. +- `REV-FINDINGS` | evidence | List findings first in severity-descending order with severity, path:line, evidence, violated rule or requirement, and required remediation. + +## Required output +- findings: NONE | LIST[severity,path:line,evidence,violated_rule_or_requirement,required_remediation]; ORDER=severity_descending +- verdict: PASS | FAIL +- static_review: EVIDENCE +- mutation_evidence: EVIDENCE +- full_gate: EVIDENCE +- remaining_concern: EVIDENCE | NONE + +## Stop conditions +- `STOP-SCRATCH` | FAIL | A safe scratch mutation environment cannot be created. +- `STOP-EVIDENCE` | FAIL | Required source, diff, test, or gate evidence is unavailable. diff --git a/scripts/check-doc-checkpoint.py b/scripts/check-doc-checkpoint.py index 18eb2a1ea..c88155c61 100755 --- a/scripts/check-doc-checkpoint.py +++ b/scripts/check-doc-checkpoint.py @@ -344,6 +344,7 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: GOVERNANCE_FILES = frozenset( { ".agents/prompts/implementer.md", + ".agents/prompts/operator.md", ".agents/prompts/reviewer.md", ".agents/governance-tasks.csv", ".agents/policy.csv", @@ -353,6 +354,7 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: "scripts/check-gate-commands.py", "scripts/policy_contract.py", "scripts/check-policy.py", + "scripts/check-prompt-contract.py", "scripts/check-pr-size.py", "scripts/check-protocol-consistency.py", "scripts/check-role-discipline.py", @@ -360,6 +362,7 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: "tests/scripts/test_agent_role.py", "tests/scripts/test_claim_view.py", "tests/scripts/test_policy_contract.py", + "tests/scripts/test_check_prompt_contract.py", "tests/scripts/test_doc_checkpoint.py", "tests/scripts/test_check_protocol_consistency.py", "docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md", diff --git a/scripts/check-prompt-contract.py b/scripts/check-prompt-contract.py new file mode 100644 index 000000000..5452744f3 --- /dev/null +++ b/scripts/check-prompt-contract.py @@ -0,0 +1,286 @@ +#!/usr/bin/env python3 +"""Validate the closed grammar of versioned runtime prompt contracts.""" + +from __future__ import annotations + +import sys +from collections import Counter +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +from scripts.policy_contract import load_policy # noqa: E402 + + +PROMPT_BUDGET = 4096 +ROLES = ("implementer", "reviewer", "operator") +RUNTIME_PROMPT_FILES = frozenset(f"{role}.md" for role in ROLES) +SECTION_NAMES = ("Task envelope", "Method", "Required output", "Stop conditions") +METADATA_KEYS = ("prompt-contract-version", "role", "policy-rules") +ENVELOPE_ROWS = ( + "- Goal: REQUIRED", + "- Context: REQUIRED", + "- Constraints: REQUIRED", + "- Done when: REQUIRED", + "- Required evidence: REQUIRED", + "- Authority: REQUIRED", + "- Missing input: NEEDS_CONTEXT", +) +ROLE_RULES = { + "implementer": ( + "POL-PROMPT-ENVELOPE", + "POL-PROMPT-BOUNDARIES", + ), + "reviewer": ( + "POL-PROMPT-ENVELOPE", + "POL-PROMPT-BOUNDARIES", + "POL-REVIEW-FRESH", + "POL-REVIEW-NO-REPAIR", + ), + "operator": ( + "POL-PROMPT-ENVELOPE", + "POL-PROMPT-BOUNDARIES", + "POL-OPERATOR-BOUNDARY", + "POL-OPERATOR-VERIFY", + "POL-PR-DISPOSITION", + "POL-REVIEW-FRESH", + "POL-REVIEW-NO-REPAIR", + "POL-REMOTE-UNKNOWN", + ), +} +METHOD_ROWS = { + "implementer": ( + "- `IMP-TEST-FIRST` | required | Write the failing test first, run it before implementation, and confirm the stated failure.", + "- `IMP-VERIFY` | required | Run every declared verification and require exit zero or an unchanged proven baseline.", + "- `IMP-MUTATE` | required | Delete or invert each behavior named by every added test, require its focused suite to fail, then restore it.", + "- `IMP-SCOPE` | forbidden | Change files or behavior outside Authority.", + "- `IMP-EVIDENCE` | evidence | Record each command, exit status, negative mutation, observed failure, and restoration check.", + ), + "reviewer": ( + "- `REV-STATIC` | required | Perform independent static review of requirements, plan, diff, and relevant surrounding code.", + "- `REV-MUTATION` | required | In a scratch copy, delete or invert each behavior changed tests claim to pin and rerun its focused test.", + "- `REV-FULL-GATE` | required | Run the full declared gate exactly once on the unchanged reviewed commit.", + "- `REV-NO-REPAIR` | forbidden | Repair a finding.", + "- `REV-WORKTREE` | forbidden | Mutate the reviewed worktree, index, HEAD, or branch.", + "- `REV-FINDINGS` | evidence | List findings first in severity-descending order with severity, path:line, evidence, violated rule or requirement, and required remediation.", + ), + "operator": ( + "- `OP-DELEGATE` | required | Delegate implementation and repairs to fresh implementers.", + "- `OP-VERIFY` | required | Run claimed verification on the returned commit without trusting the implementer report.", + "- `OP-REVIEW` | required | Dispatch a fresh reviewer for independent static review and targeted scratch mutation.", + "- `OP-DISPOSITION` | required | Merge a verified PR in-session or close an obsolete PR with its recorded reason.", + "- `OP-REPAIR` | forbidden | Repair an implementer or reviewer finding in the coordinating context.", + "- `OP-EVIDENCE` | evidence | Record verification, review, PR disposition, blocker, and remote-state evidence.", + ), +} +OUTPUT_ROWS = { + "implementer": ( + "- status: COMPLETE | BLOCKED | NEEDS_CONTEXT | NEEDS_DECISION", + "- summary: REQUIRED", + "- changed_files: LIST", + "- commands_and_exit_status: EVIDENCE", + "- negative_mutation: EVIDENCE", + "- deviations_and_waivers: EVIDENCE | NONE", + "- risks: EVIDENCE | NONE", + "- omitted_gates: EVIDENCE | NONE", + "- commit_sha: SHA | NONE", + ), + "reviewer": ( + "- findings: NONE | LIST[severity,path:line,evidence,violated_rule_or_requirement,required_remediation]; ORDER=severity_descending", + "- verdict: PASS | FAIL", + "- static_review: EVIDENCE", + "- mutation_evidence: EVIDENCE", + "- full_gate: EVIDENCE", + "- remaining_concern: EVIDENCE | NONE", + ), + "operator": ( + "- status: MERGED | CLOSED | BLOCKED | REMOTE_UNVERIFIED", + "- verification: EVIDENCE", + "- review: EVIDENCE", + "- disposition: EVIDENCE", + "- remaining_concern: EVIDENCE | NONE", + ), +} +STOP_ROWS = { + "implementer": ( + "- `STOP-AUTHORITY` | BLOCKED | An edit outside Authority is required.", + "- `STOP-VERIFY` | BLOCKED | Declared verification cannot pass or match a proven baseline.", + "- `STOP-CONTEXT` | NEEDS_CONTEXT | Missing input changes the contract or public behavior.", + "- `STOP-DECISION` | NEEDS_DECISION | A required choice exceeds Authority.", + ), + "reviewer": ( + "- `STOP-SCRATCH` | FAIL | A safe scratch mutation environment cannot be created.", + "- `STOP-EVIDENCE` | FAIL | Required source, diff, test, or gate evidence is unavailable.", + ), + "operator": ( + "- `STOP-AUTHORITY` | BLOCKED | A required action exceeds Authority.", + "- `STOP-REMOTE` | REMOTE_UNVERIFIED | Required remote state cannot be queried.", + "- `STOP-BLOCKER` | BLOCKED | Non-terminal work cannot name its blocker on the PR.", + ), +} + + +def _parse_metadata(lines: list[str], label: str) -> tuple[dict[str, str], int, list[str]]: + errors: list[str] = [] + if not lines or lines[0] != "---": + return {}, 0, [f"{label}: missing exact metadata opening delimiter"] + try: + end = lines.index("---", 1) + except ValueError: + return {}, len(lines), [f"{label}: missing exact metadata closing delimiter"] + + pairs: list[tuple[str, str]] = [] + for line in lines[1:end]: + if ": " not in line: + errors.append(f"{label}: unparsed metadata line {line!r}") + continue + key, value = line.split(": ", 1) + pairs.append((key, value)) + counts = Counter(key for key, _ in pairs) + duplicates = sorted(key for key, count in counts.items() if count > 1) + if duplicates: + errors.append(f"{label}: duplicate metadata keys: {', '.join(duplicates)}") + keys = tuple(key for key, _ in pairs) + if keys != METADATA_KEYS: + errors.append(f"{label}: metadata keys must be exactly {METADATA_KEYS!r}") + return dict(pairs), end + 1, errors + + +def _parse_sections( + lines: list[str], start: int, label: str +) -> tuple[list[tuple[str, tuple[str, ...]]], list[str]]: + sections: list[tuple[str, list[str]]] = [] + errors: list[str] = [] + for line_number, line in enumerate(lines[start:], start=start + 1): + if not line: + continue + if line.startswith("## "): + sections.append((line[3:], [])) + continue + if not sections: + errors.append(f"{label}:{line_number}: unparsed normative line {line!r}") + continue + sections[-1][1].append(line) + return [(name, tuple(rows)) for name, rows in sections], errors + + +def _row_errors( + label: str, section: str, actual: tuple[str, ...], expected: tuple[str, ...] +) -> list[str]: + errors: list[str] = [] + counts = Counter(actual) + duplicates = sorted(row for row, count in counts.items() if count > 1) + if duplicates: + errors.append(f"{label}: {section} has duplicate rows: {duplicates!r}") + for row in expected: + if row not in actual: + errors.append(f"{label}: {section} is missing row {row!r}") + for row in actual: + if row not in expected: + errors.append(f"{label}: {section} has unparsed row {row!r}") + if not errors and actual != expected: + errors.append(f"{label}: {section} rows are out of order") + return errors + + +def validate_prompt(path: Path, role: str, known_rule_ids: set[str]) -> list[str]: + """Return every closed-grammar defect in one runtime prompt.""" + + label = path.as_posix() + if role not in ROLES: + return [f"{label}: unknown runtime role {role!r}"] + if not path.is_file(): + return [f"{label}: missing runtime prompt"] + if path.stat().st_size > PROMPT_BUDGET: + return [f"{label}: exceeds the {PROMPT_BUDGET}-byte prompt budget"] + try: + text = path.read_text(encoding="utf-8") + except UnicodeDecodeError as exc: + return [f"{label}: not valid UTF-8: {exc}"] + + lines = text.splitlines() + metadata, body_start, errors = _parse_metadata(lines, label) + expected_metadata = { + "prompt-contract-version": "1", + "role": role, + "policy-rules": " ".join(ROLE_RULES[role]), + } + for key, expected in expected_metadata.items(): + if metadata.get(key) != expected: + errors.append(f"{label}: metadata {key!r} must be exactly {expected!r}") + missing_registry_rules = set(ROLE_RULES[role]) - known_rule_ids + if missing_registry_rules: + errors.append( + f"{label}: required policy rules are absent from policy.csv: " + + ", ".join(sorted(missing_registry_rules)) + ) + + sections, section_errors = _parse_sections(lines, body_start, label) + errors.extend(section_errors) + names = tuple(name for name, _ in sections) + if names != SECTION_NAMES: + errors.append(f"{label}: H2 sections must be exactly {SECTION_NAMES!r}") + counts = Counter(names) + duplicates = sorted(name for name, count in counts.items() if count > 1) + if duplicates: + errors.append(f"{label}: duplicate H2 sections: {', '.join(duplicates)}") + + actual_by_name = {name: rows for name, rows in sections if counts[name] == 1} + expected_by_name = { + "Task envelope": ENVELOPE_ROWS, + "Method": METHOD_ROWS[role], + "Required output": OUTPUT_ROWS[role], + "Stop conditions": STOP_ROWS[role], + } + for name, expected_rows in expected_by_name.items(): + actual_rows = actual_by_name.get(name) + if actual_rows is not None: + errors.extend(_row_errors(label, name, actual_rows, expected_rows)) + return errors + + +def repository_errors(root: Path, known_rule_ids: set[str]) -> list[str]: + """Validate every runtime prompt owned by this contract checker.""" + + errors: list[str] = [] + prompt_root = root / ".agents/prompts" + if prompt_root.is_dir(): + for artifact in sorted(prompt_root.iterdir(), key=lambda path: path.name): + if artifact.name not in RUNTIME_PROMPT_FILES: + errors.append( + f"{artifact.as_posix()}: unexpected runtime prompt artifact; " + f"expected exactly {', '.join(sorted(RUNTIME_PROMPT_FILES))}" + ) + else: + errors.append(f"{prompt_root.as_posix()}: missing runtime prompt namespace") + for role in ROLES: + errors.extend( + validate_prompt( + prompt_root / f"{role}.md", role, known_rule_ids + ) + ) + return errors + + +def main() -> int: + try: + known_rule_ids = set(load_policy(ROOT)) + except ValueError as exc: + print(f"prompt contract BLOCKED: policy registry is invalid\n{exc}") + return 1 + + errors = repository_errors(ROOT, known_rule_ids) + if errors: + print("prompt contract FAILED:") + for error in errors: + print(f" - {error}") + return 1 + print(f"OK: {len(ROLES)} closed runtime prompt contracts") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-protocol-consistency.py b/scripts/check-protocol-consistency.py index 7dbd61ff7..e43b24407 100644 --- a/scripts/check-protocol-consistency.py +++ b/scripts/check-protocol-consistency.py @@ -21,13 +21,9 @@ implementer's word, or that findings are never fixed in the controller's own session. -The same gate finally asserts that the sub-agent prompts under `.agents/prompts` -exist and still carry their binding instructions. Every Important finding across -two branches of this project came from an INDEPENDENT reviewer sub-agent, none -from an implementer's self-review, and the reviewers found them by MUTATING code -rather than reading diffs: eleven tests passed with the thing they named -deleted, and not one was visible by reading. That instruction is the deliverable, -so it is tracked and pinned phrase by phrase, not merely present as a file. +The same gate finally runs the closed runtime-prompt grammar. Phrase presence +cannot establish a method contract: contradictory prose may retain every +required phrase. The semantic validator therefore parses every nonempty line. """ from __future__ import annotations @@ -53,8 +49,8 @@ INTERVIEW_MARKER = "" INTERVIEW_REQUIRED = ("claim operator", "claim helper --row", "claim read-only", "--headless") -# The same manual must carry the operator's LOOP. The prompts in PROMPT_REQUIRED -# below are handed to sub-agents; nothing told the operator how to run one, and +# The same manual must carry the operator's LOOP. The prompts handed to +# sub-agents cannot by themselves tell the operator how to run one, and # the three rules that carry the whole return are exactly the ones an operator # improvises away: dispatch a FRESH reviewer whose instruction is to MUTATE, # run the row's gate YOURSELF rather than believing the author's report, and @@ -76,38 +72,6 @@ "never fix findings yourself", ) -# The reviewer prompt's value is the MUTATION instruction; a reviewer told only -# to "review" reads the diff, and reading found none of the eleven tests that -# passed with their subject deleted. Pin the instruction, not the file. -# -# The reviewer needle is the full "mutate, don't read", not a bare "mutate": -# the prompt also says "never mutate the reviewed worktree" further down, so the -# short form would stay satisfied by an unrelated sentence after the binding -# instruction was deleted. That is the same "an unrelated line satisfied the -# assertion" failure the prompt itself is written to catch. -# -# Two needles pin REPAIRS to earlier drafts of these prompts, because a prompt -# that once contradicted itself can drift back: the reviewer prompt used to -# forbid re-running the full suite (which reads as a budget on the mutations it -# demands two sections earlier), and the implementer prompt used to demand a -# green gate with no answer for reds that were already there before the work -# started, whose only exits were stalling or an allowlist. -PROMPT_REQUIRED = { - ".agents/prompts/reviewer.md": ( - "mutate, don't read", - "delete or invert", - "stays green", - "every mutation you make re-runs the suite", - "plan-mandated", - ), - ".agents/prompts/implementer.md": ( - "failing test first", - "mutate every test", - "capture that failing set as a baseline", - "escalate rather than guess", - ), -} - def _load(name: str, relative: str): path = ROOT / relative spec = importlib.util.spec_from_file_location(name, path) @@ -203,26 +167,13 @@ def loop_errors(text: str) -> list[str]: ] -def prompt_errors(required: dict[str, tuple[str, ...]] | None = None) -> list[str]: - """Each tracked prompt exists and carries its binding instruction.""" - # `required or PROMPT_REQUIRED` would silently promote an explicitly EMPTY - # spec into the full live check, which is this repo's recurring defect - # class: an absence and a value that look the same. Only a missing argument - # means "use the default". - errors: list[str] = [] - spec = PROMPT_REQUIRED if required is None else required - for relative, needles in spec.items(): - path = ROOT / relative - if not path.is_file(): - errors.append(f"{relative} is missing; the prompt is the protocol") - continue - text = path.read_text(encoding="utf-8").lower() - errors.extend( - f"{relative} omits {needle!r}" - for needle in needles - if needle.lower() not in text - ) - return errors +def prompt_contract_errors() -> list[str]: + """Validate every runtime prompt through the closed semantic grammar.""" + + checker = _load( + "prompt_contract_for_consistency", "scripts/check-prompt-contract.py" + ) + return checker.repository_errors(ROOT, set(load_policy(ROOT))) def main() -> int: @@ -246,7 +197,7 @@ def main() -> int: else: failures.extend(loop_errors(loop_doc.read_text(encoding="utf-8"))) - failures.extend(prompt_errors()) + failures.extend(prompt_contract_errors()) if failures: for failure in failures: @@ -258,10 +209,8 @@ def main() -> int: "name every answer agent-role.py accepts. The operator's loop is " f"the block between {LOOP_MARKER} and its :end in {LOOP_DOCUMENT}; " f"it must carry {', '.join(repr(n) for n in LOOP_REQUIRED)} inside " - "the block. The sub-agent prompts in " - f"{', '.join(PROMPT_REQUIRED)} must carry their binding " - "instructions verbatim; a prompt that lives only in an operator's " - "head is not a protocol.", + "the block. Every runtime prompt must satisfy the closed grammar " + "in scripts/check-prompt-contract.py; unknown prose is a failure.", file=sys.stderr, ) return 1 @@ -269,9 +218,8 @@ def main() -> int: print( "OK: public-document policy matches scripts/check-doc-checkpoint.py, " f"{INTERVIEW_DOCUMENT} carries the " - f"role interview and the orchestration loop, and " - f"{len(PROMPT_REQUIRED)} sub-agent prompts carry their binding " - "instructions." + f"role interview and the orchestration loop, and all runtime prompts " + "satisfy the closed semantic contract." ) return 0 diff --git a/tests/scripts/test_check_prompt_contract.py b/tests/scripts/test_check_prompt_contract.py new file mode 100644 index 000000000..950468dd6 --- /dev/null +++ b/tests/scripts/test_check_prompt_contract.py @@ -0,0 +1,348 @@ +#!/usr/bin/env python3 +"""Mutation tests for the closed runtime-prompt contract grammar.""" + +from __future__ import annotations + +import importlib.util +import io +import shutil +import sys +import tempfile +import unittest +from contextlib import redirect_stdout +from pathlib import Path +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[2] + + +def _load_checker(): + path = ROOT / "scripts/check-prompt-contract.py" + spec = importlib.util.spec_from_file_location("check_prompt_contract", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +prompt_contract = _load_checker() +KNOWN_RULE_IDS = { + "POL-PROMPT-ENVELOPE", + "POL-PROMPT-BOUNDARIES", + "POL-REVIEW-FRESH", + "POL-REVIEW-NO-REPAIR", + "POL-OPERATOR-BOUNDARY", + "POL-OPERATOR-VERIFY", + "POL-PR-DISPOSITION", + "POL-REMOTE-UNKNOWN", +} + + +class PromptContractTests(unittest.TestCase): + def shipped(self, role: str) -> str: + return (ROOT / ".agents/prompts" / f"{role}.md").read_text(encoding="utf-8") + + def validate_text(self, text: str, role: str) -> list[str]: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / f"{role}.md" + path.write_text(text, encoding="utf-8") + return prompt_contract.validate_prompt(path, role, KNOWN_RULE_IDS) + + def assert_rejected(self, role: str, old: str, new: str) -> None: + text = self.shipped(role) + self.assertIn(old, text) + damaged = text.replace(old, new, 1) + self.assertNotEqual(damaged, text) + self.assertTrue(self.validate_text(damaged, role)) + + def test_all_runtime_prompts_pass_and_fit_the_budget(self) -> None: + for role in ("implementer", "reviewer", "operator"): + with self.subTest(role=role): + path = ROOT / ".agents/prompts" / f"{role}.md" + self.assertLessEqual(path.stat().st_size, 4096) + self.assertEqual( + prompt_contract.validate_prompt(path, role, KNOWN_RULE_IDS), [] + ) + + def test_metadata_is_closed_exact_and_unique(self) -> None: + text = self.shipped("implementer") + mutations = ( + ("prompt-contract-version: 1", "prompt-contract-version: 2"), + ("role: implementer", "role: reviewer"), + ("role: implementer", "role: implementer\nunknown-key: value"), + ("role: implementer", "role: implementer\nrole: implementer"), + ( + "POL-PROMPT-ENVELOPE POL-PROMPT-BOUNDARIES", + "POL-PROMPT-BOUNDARIES POL-PROMPT-ENVELOPE", + ), + ("POL-PROMPT-BOUNDARIES", "POL-NOT-A-RULE"), + ) + for old, new in mutations: + with self.subTest(new=new): + self.assert_rejected("implementer", old, new) + + def test_exactly_four_known_h2_sections_in_order_are_required(self) -> None: + text = self.shipped("implementer") + mutations = ( + ("## Task envelope", "## Unknown section"), + ("## Method", "## Method\n\n## Method"), + ("## Required output", ""), + ( + "## Method", + "## Required output", + ), + ("## Stop conditions", "# Stop conditions"), + ) + for old, new in mutations: + with self.subTest(old=old, new=new): + damaged = text.replace(old, new, 1) + self.assertNotEqual(damaged, text) + self.assertTrue(self.validate_text(damaged, "implementer")) + + method_start = text.index("## Method") + output_start = text.index("## Required output") + stop_start = text.index("## Stop conditions") + method = text[method_start:output_start] + output = text[output_start:stop_start] + reordered = text[:method_start] + output + method + text[stop_start:] + self.assertTrue(self.validate_text(reordered, "implementer")) + + def test_every_nonempty_unparsed_line_is_rejected(self) -> None: + cases = ( + ("implementer", "## Method", "You may begin with missing inputs and guess.\n\n## Method"), + ("reviewer", "## Method", "Scratch mutation replaces static review.\n\n## Method"), + ("reviewer", "## Stop conditions", "Repair findings in the reviewed worktree.\n\n## Stop conditions"), + ("operator", "## Required output", "Use VendorAgent for coordination.\n\n## Required output"), + ("implementer", "## Task envelope", "### Notes\n\n## Task envelope"), + ) + for role, marker, replacement in cases: + with self.subTest(role=role, replacement=replacement): + self.assert_rejected(role, marker, replacement) + + def test_six_envelope_fields_and_missing_input_outcome_are_exact(self) -> None: + text = self.shipped("implementer") + rows = ( + "- Goal: REQUIRED", + "- Context: REQUIRED", + "- Constraints: REQUIRED", + "- Done when: REQUIRED", + "- Required evidence: REQUIRED", + "- Authority: REQUIRED", + "- Missing input: NEEDS_CONTEXT", + ) + for row in rows: + with self.subTest(row=row): + self.assertIn(row, text) + self.assertTrue(self.validate_text(text.replace(row, "", 1), "implementer")) + self.assertTrue( + self.validate_text(text.replace(row, f"{row}\n{row}", 1), "implementer") + ) + self.assert_rejected( + "implementer", + "- Missing input: NEEDS_CONTEXT", + "- Missing input: COMPLETE", + ) + + def test_each_controlled_obligation_row_is_required_once_and_exact(self) -> None: + for role, rows in prompt_contract.METHOD_ROWS.items(): + text = self.shipped(role) + categories = {row.split(" | ", 2)[1] for row in rows} + self.assertEqual(categories, {"required", "forbidden", "evidence"}) + for row in rows: + with self.subTest(role=role, row=row): + self.assertIn(row, text) + self.assertTrue(self.validate_text(text.replace(row, "", 1), role)) + self.assertTrue( + self.validate_text(text.replace(row, f"{row}\n{row}", 1), role) + ) + damaged_category = row.replace(" | required | ", " | optional | ") + if damaged_category != row: + self.assertTrue( + self.validate_text(text.replace(row, damaged_category, 1), role) + ) + + def test_output_rows_and_enums_are_exact(self) -> None: + for role, rows in prompt_contract.OUTPUT_ROWS.items(): + text = self.shipped(role) + for row in rows: + with self.subTest(role=role, row=row): + self.assertIn(row, text) + self.assertTrue(self.validate_text(text.replace(row, "", 1), role)) + self.assertTrue( + self.validate_text(text.replace(row, f"{row}\n{row}", 1), role) + ) + self.assert_rejected( + "implementer", + "COMPLETE | BLOCKED | NEEDS_CONTEXT | NEEDS_DECISION", + "COMPLETE | BLOCKED", + ) + self.assert_rejected( + "operator", + "MERGED | CLOSED | BLOCKED | REMOTE_UNVERIFIED", + "MERGED | CLOSED | BLOCKED", + ) + + def test_stop_rows_are_required_once_and_exact(self) -> None: + for role, rows in prompt_contract.STOP_ROWS.items(): + text = self.shipped(role) + for row in rows: + with self.subTest(role=role, row=row): + self.assertIn(row, text) + self.assertTrue(self.validate_text(text.replace(row, "", 1), role)) + self.assertTrue( + self.validate_text(text.replace(row, f"{row}\n{row}", 1), role) + ) + self.assert_rejected( + "reviewer", + "- `STOP-SCRATCH` | FAIL |", + "- `STOP-SCRATCH` | PASS |", + ) + + def test_implementer_accepts_all_declared_status_and_evidence_outputs(self) -> None: + text = self.shipped("implementer") + for value in ("COMPLETE", "BLOCKED", "NEEDS_CONTEXT", "NEEDS_DECISION"): + self.assertIn(value, text) + for field in ( + "summary", + "changed_files", + "commands_and_exit_status", + "negative_mutation", + "deviations_and_waivers", + "risks", + "omitted_gates", + "commit_sha", + ): + self.assertIn(f"- {field}:", text) + + def test_reviewer_contract_has_three_lenses_and_no_repair_escape(self) -> None: + text = self.shipped("reviewer") + for clause_id in ( + "REV-STATIC", + "REV-MUTATION", + "REV-FULL-GATE", + "REV-NO-REPAIR", + "REV-WORKTREE", + ): + self.assertIn(f"`{clause_id}`", text) + self.assertIn("scratch", text.lower()) + self.assertIn("static review", text.lower()) + self.assertIn("full declared gate", text.lower()) + + def test_reviewer_findings_are_first_structured_and_severity_ordered(self) -> None: + text = self.shipped("reviewer") + finding = prompt_contract.OUTPUT_ROWS["reviewer"][0] + self.assertIn("severity_descending", finding) + for field in ( + "severity", + "path:line", + "evidence", + "violated_rule_or_requirement", + "required_remediation", + ): + self.assertIn(field, finding) + self.assert_rejected( + "reviewer", "required_remediation", "suggested_change" + ) + self.assert_rejected( + "reviewer", "severity_descending", "severity_ascending" + ) + + def test_missing_prompt_and_unknown_role_are_distinct_failures(self) -> None: + missing = Path(tempfile.gettempdir()) / "prompt-contract-does-not-exist.md" + self.assertTrue( + any( + "missing" in error + for error in prompt_contract.validate_prompt( + missing, "implementer", KNOWN_RULE_IDS + ) + ) + ) + self.assertTrue( + any( + "unknown runtime role" in error + for error in prompt_contract.validate_prompt( + missing, "inventor", KNOWN_RULE_IDS + ) + ) + ) + + def test_main_calls_semantic_validation_and_returns_nonzero_for_damage(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + prompt_root = root / ".agents/prompts" + prompt_root.mkdir(parents=True) + for role in ("implementer", "reviewer", "operator"): + shutil.copy(ROOT / ".agents/prompts" / f"{role}.md", prompt_root) + reviewer = prompt_root / "reviewer.md" + reviewer.write_text( + reviewer.read_text(encoding="utf-8").replace( + "## Method", + "Scratch mutation replaces static review.\n\n## Method", + 1, + ), + encoding="utf-8", + ) + with ( + mock.patch.object(prompt_contract, "ROOT", root), + mock.patch.object( + prompt_contract, + "load_policy", + return_value=dict.fromkeys(KNOWN_RULE_IDS), + ), + redirect_stdout(io.StringIO()) as output, + ): + self.assertEqual(prompt_contract.main(), 1) + self.assertIn("unparsed", output.getvalue()) + + def test_repository_errors_owns_runtime_prompt_discovery(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + prompt_root = root / ".agents/prompts" + prompt_root.mkdir(parents=True) + for role in ("implementer", "reviewer", "operator"): + shutil.copy(ROOT / ".agents/prompts" / f"{role}.md", prompt_root) + self.assertEqual( + prompt_contract.repository_errors(root, KNOWN_RULE_IDS), [] + ) + (prompt_root / "operator.md").unlink() + errors = prompt_contract.repository_errors(root, KNOWN_RULE_IDS) + self.assertTrue(any("operator.md" in error and "missing" in error for error in errors)) + + def test_runtime_prompt_namespace_rejects_every_noncanonical_artifact(self) -> None: + additions = { + "critic.md": "# malformed runtime prompt\n", + "reviewer-copy.md": self.shipped("reviewer"), + "Reviewer.md": self.shipped("reviewer"), + } + for name, content in additions.items(): + with self.subTest(name=name), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + prompt_root = root / ".agents/prompts" + prompt_root.mkdir(parents=True) + for role in ("implementer", "reviewer", "operator"): + shutil.copy(ROOT / ".agents/prompts" / f"{role}.md", prompt_root) + (prompt_root / name).write_text(content, encoding="utf-8") + errors = prompt_contract.repository_errors(root, KNOWN_RULE_IDS) + self.assertTrue( + any(name in error and "unexpected" in error for error in errors), + errors, + ) + + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + prompt_root = root / ".agents/prompts" + prompt_root.mkdir(parents=True) + for role in ("implementer", "reviewer", "operator"): + shutil.copy(ROOT / ".agents/prompts" / f"{role}.md", prompt_root) + (prompt_root / "adapters").mkdir() + errors = prompt_contract.repository_errors(root, KNOWN_RULE_IDS) + self.assertTrue( + any("adapters" in error and "unexpected" in error for error in errors), + errors, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_check_protocol_consistency.py b/tests/scripts/test_check_protocol_consistency.py index b8586fd05..117193c04 100644 --- a/tests/scripts/test_check_protocol_consistency.py +++ b/tests/scripts/test_check_protocol_consistency.py @@ -21,6 +21,7 @@ import tempfile import unittest from pathlib import Path +from unittest import mock ROOT = Path(__file__).resolve().parents[2] @@ -91,7 +92,13 @@ def _prompt_tree(files: dict[str, str]): @contextlib.contextmanager -def _repo_copy(workflow_text: str, *, prompts: bool = True): +def _repo_copy( + workflow_text: str, + *, + prompts: bool = True, + prompt_damage: tuple[str, str, str] | None = None, + extra_prompt: tuple[str, str] | None = None, +): """Run consistency.main() against a copy of the repo's own documents. Only `.agents/workflow.md` is substituted, so a red from this helper is @@ -106,6 +113,10 @@ def _repo_copy(workflow_text: str, *, prompts: bool = True): ROOT / "scripts/check-doc-checkpoint.py", root / "scripts/check-doc-checkpoint.py", ) + shutil.copy( + ROOT / "scripts/check-prompt-contract.py", + root / "scripts/check-prompt-contract.py", + ) shutil.copy(ROOT / ".agents/policy.csv", root / ".agents/policy.csv") shutil.copy(ROOT / ".agents/waivers.csv", root / ".agents/waivers.csv") # The policy parser validates every named checker and procedure. Create @@ -122,6 +133,19 @@ def _repo_copy(workflow_text: str, *, prompts: bool = True): target.write_text("# fixture\n", encoding="utf-8") if prompts: shutil.copytree(ROOT / ".agents/prompts", root / ".agents/prompts") + if prompt_damage is not None: + role, old, new = prompt_damage + path = root / ".agents/prompts" / f"{role}.md" + text = path.read_text(encoding="utf-8") + damaged = text.replace(old, new, 1) + if damaged == text: + raise AssertionError(f"prompt mutation did not match: {old!r}") + path.write_text(damaged, encoding="utf-8") + if extra_prompt is not None: + name, content = extra_prompt + (root / ".agents/prompts" / name).write_text( + content, encoding="utf-8" + ) (root / ".agents/workflow.md").write_text(workflow_text, encoding="utf-8") saved, consistency.ROOT = consistency.ROOT, root out, err = io.StringIO(), io.StringIO() @@ -441,8 +465,16 @@ class InterviewWiring(unittest.TestCase): ) @contextlib.contextmanager - def _tree(self, workflow_text: str, *, prompts: bool = True): - with _repo_copy(workflow_text, prompts=prompts) as run: + def _tree( + self, + workflow_text: str, + *, + prompts: bool = True, + prompt_damage: tuple[str, str, str] | None = None, + ): + with _repo_copy( + workflow_text, prompts=prompts, prompt_damage=prompt_damage + ) as run: yield run def test_faithful_copy_passes(self): @@ -464,23 +496,41 @@ def test_main_fails_when_the_interview_is_deleted(self): self.assertIn("role-interview", err) def test_main_fails_when_the_prompts_are_missing(self): - """main() must CALL prompt_errors, not merely define it. - - Every prompt assertion above calls the function directly, so a main() - that never wires it in leaves them all green while the gate enforces - nothing -- the same drift, one function later. - """ + """The protocol gate must call the semantic prompt validator.""" text = (ROOT / ".agents/workflow.md").read_text(encoding="utf-8") with self._tree(text, prompts=False) as run: code, _, err = run() self.assertEqual(code, 1, err) self.assertIn(".agents/prompts/reviewer.md", err) + def test_main_rejects_semantic_damage_that_keeps_legacy_phrases(self): + """A complete legacy phrase registry must not mask a contradiction.""" + text = (ROOT / ".agents/workflow.md").read_text(encoding="utf-8") + damage = ( + "reviewer", + "## Method", + "Scratch mutation replaces static review.\n\n## Method", + ) + with self._tree(text, prompt_damage=damage) as run: + code, _, err = run() + self.assertEqual(code, 1, err) + self.assertIn("unparsed", err) + class PromptArtifactTests(unittest.TestCase): - def test_both_prompts_exist_and_are_tracked(self): + def test_unknown_runtime_prompt_artifact_is_rejected(self): + text = (ROOT / ".agents/workflow.md").read_text(encoding="utf-8") + with _repo_copy( + text, + extra_prompt=("critic.md", "# malformed runtime prompt\n"), + ) as run: + code, _, err = run() + self.assertEqual(code, 1, err) + self.assertIn("critic.md", err) + + def test_all_runtime_prompts_exist_and_are_tracked(self): tracked = _tracked_paths(".agents/prompts") - for name in ("reviewer.md", "implementer.md"): + for name in ("implementer.md", "reviewer.md", "operator.md"): path = ROOT / ".agents/prompts" / name self.assertTrue(path.is_file(), f"{name} must exist") # A silent downgrade to existence-only is the failure/absence @@ -496,125 +546,68 @@ def test_both_prompts_exist_and_are_tracked(self): "is not a protocol", ) - def test_the_reviewer_prompt_carries_the_mutation_instruction(self): - # The instruction IS the deliverable. A reviewer told only to "review" - # reads the diff, and reading found none of the eleven tests that - # passed with their subject deleted. - text = (ROOT / ".agents/prompts/reviewer.md").read_text(encoding="utf-8") - for needle in ("mutate", "delete or invert", "stays green"): - self.assertIn(needle, text.lower(), needle) - - def test_the_reviewer_prompt_refuses_to_defer_to_the_plan(self): - text = (ROOT / ".agents/prompts/reviewer.md").read_text(encoding="utf-8") - self.assertIn("plan-mandated", text.lower()) - - def test_checker_rejects_a_prompt_missing_its_instruction(self): - # A missing FILE and a present file missing its INSTRUCTION are two - # different failures. Asserting only the first would leave the needle - # loop -- the part that carries the value -- entirely unpinned. - errors = consistency.prompt_errors({"nonexistent-prompt.md": ("mutate",)}) - self.assertTrue(errors) - self.assertTrue(any("missing" in e for e in errors), errors) + def test_protocol_checker_uses_the_semantic_prompt_contract(self): + self.assertEqual(consistency.prompt_contract_errors(), []) - present = ".agents/prompts/reviewer.md" - self.assertEqual(consistency.prompt_errors({present: ("mutate",)}), []) - omitted = consistency.prompt_errors( - {present: ("no reviewer prompt would ever contain this phrase",)} - ) - self.assertTrue(any("omits" in e for e in omitted), omitted) - - def test_an_explicitly_empty_spec_checks_nothing(self): - # An empty spec must mean "nothing required", not silently fall back to - # the live PROMPT_REQUIRED: an absence and a value that look the same is - # the defect class the implementer prompt names. - with _prompt_tree({}): - self.assertEqual(consistency.prompt_errors({}), []) - self.assertTrue(consistency.prompt_errors()) - - def test_the_checker_enforces_the_phrases_these_tests_demand(self): - # Every assertion above reads the prompt FILES, so emptying, narrowing - # or widening a PROMPT_REQUIRED tuple would leave them all green while - # the gate quietly stopped enforcing what this suite believes it does. - # - # The comparison is EQUALITY, deliberately, not "demanded is a substring - # of enforced". That substring idiom is borrowed from - # test_every_declarable_role_is_named_in_the_interview, where it is safe - # because the demanded side is DERIVED from role.DECLARABLE. Here both - # sides are hand-written literals, and a substring test cannot see the - # one narrowing that matters: reverting the reviewer needle from - # "mutate, don't read" to a bare "mutate" satisfies it while re-opening - # the incidental-match hole check-protocol-consistency.py spends five - # lines arguing is dangerous. Equality means changing what the gate - # enforces is a deliberate two-file act. - demanded = { - ".agents/prompts/reviewer.md": ( - "mutate, don't read", - "delete or invert", - "stays green", - "every mutation you make re-runs the suite", - "plan-mandated", - ), - ".agents/prompts/implementer.md": ( - "failing test first", - "mutate every test", - "capture that failing set as a baseline", - "escalate rather than guess", - ), - } - self.assertEqual( - set(demanded), - set(consistency.PROMPT_REQUIRED), - "PROMPT_REQUIRED covers a different set of prompts than this suite", + +class SemanticPromptBridgeBoundaryTests(unittest.TestCase): + def _source(self) -> str: + return (ROOT / "scripts/check-protocol-consistency.py").read_text( + encoding="utf-8" ) - for relative, needles in demanded.items(): - with self.subTest(prompt=relative): - self.assertEqual( - set(consistency.PROMPT_REQUIRED[relative]), - set(needles), - f"PROMPT_REQUIRED[{relative!r}] no longer enforces exactly " - "the phrases this suite demands; narrowing one is how the " - "gate stops catching what it was built for", - ) - def test_a_bare_mutate_needle_would_not_pin_the_binding_instruction(self): - # The executable justification for the full "mutate, don't read" needle. - # Deleting the ENTIRE binding-instruction section still leaves the word - # "mutate" in the file ("Never mutate the reviewed worktree" under What - # you may not do), so a bare needle stays green through the exact - # deletion it exists to catch. If this test ever goes red because the - # incidental match is gone, the needle may safely be simplified. - relative = ".agents/prompts/reviewer.md" - text = (ROOT / relative).read_text(encoding="utf-8") - without_section = re.sub( - r"## The binding instruction.*?(?=\n## )", "", text, flags=re.S + def assert_source_boundary(self, source: str) -> None: + # These are direct assertions over the complete production source. They + # deliberately do not consult a production token set: emptying or + # renaming an in-checker registry cannot weaken this boundary. + self.assertNotIn(".agents/prompts/", source) + self.assertNotIn("PROMPT_REQUIRED", source) + self.assertNotIn("PHRASE_PINS", source) + + def test_protocol_checker_source_is_only_a_prompt_contract_bridge(self): + self.assert_source_boundary(self._source()) + + def test_source_boundary_rejects_direct_and_indirect_registries_anywhere(self): + mutations = ( + "\nPROMPT_REQUIRED = {}\n", + '\nruntime_prompt = ".agents/prompts/reviewer.md"\nlegacy_rows = {runtime_prompt: ()}\n', + '\nlegacy_rows = ((".agents/prompts/reviewer.md", ("mutate",)),)\n', + '\nlegacy_rows = [[".agents/prompts/reviewer.md", ["mutate"]]]\n', + '\ndef retired_validator():\n return {".agents/prompts/reviewer.md": ("mutate",)}\n', + "\nPHRASE_PINS = ()\n", ) - self.assertNotEqual(without_section, text, "the strip pattern matched nothing") - with _prompt_tree({relative: without_section}): - self.assertEqual( - consistency.prompt_errors({relative: ("mutate",)}), - [], - "a bare 'mutate' no longer matches incidentally", - ) - self.assertTrue( - consistency.prompt_errors({relative: ("mutate, don't read",)}), - "the shipped needle failed to catch the section deletion", - ) + for mutation in mutations: + with self.subTest(mutation=mutation): + with self.assertRaises(AssertionError): + self.assert_source_boundary(self._source() + mutation) - def test_each_required_phrase_is_pinned_individually(self): - # PROMPT_REQUIRED is a hand-written tuple, so a prompt that survives - # losing one of its phrases means that phrase was never enforced. Strip - # each one in turn from a copy of the real file and demand a red. - for relative, needles in consistency.PROMPT_REQUIRED.items(): - text = (ROOT / relative).read_text(encoding="utf-8") - for needle in needles: - with self.subTest(prompt=relative, needle=needle): - damaged = re.sub(re.escape(needle), "", text, flags=re.I) - self.assertNotEqual( - damaged, text, f"{needle!r} does not appear in {relative}" - ) - with _prompt_tree({relative: damaged}): - errors = consistency.prompt_errors({relative: needles}) - self.assertTrue(any("omits" in e for e in errors), errors) + def test_unrelated_module_constants_remain_allowed(self): + extended = self._source() + """ + +PROMPT_BUDGET = 4096 +REQUIRED_PUBLIC_SURFACES = {"docs/STATUS.md": "feature checkpoint"} +""" + self.assert_source_boundary(extended) + + def test_bridge_imports_and_calls_the_authoritative_repository_validator(self): + checker = mock.Mock() + checker.repository_errors.return_value = ["semantic sentinel"] + with ( + mock.patch.object(consistency, "_load", return_value=checker) as load, + mock.patch.object( + consistency, + "load_policy", + return_value={"POL-PROMPT-BOUNDARIES": object()}, + ), + ): + errors = consistency.prompt_contract_errors() + load.assert_called_once_with( + "prompt_contract_for_consistency", "scripts/check-prompt-contract.py" + ) + checker.repository_errors.assert_called_once_with( + consistency.ROOT, {"POL-PROMPT-BOUNDARIES"} + ) + self.assertEqual(errors, ["semantic sentinel"]) class OrchestrationLoopTests(unittest.TestCase): diff --git a/tests/scripts/test_doc_checkpoint.py b/tests/scripts/test_doc_checkpoint.py index cbbcb5ade..13eecf5ac 100755 --- a/tests/scripts/test_doc_checkpoint.py +++ b/tests/scripts/test_doc_checkpoint.py @@ -131,8 +131,11 @@ def test_exact_governance_checker_files_are_exempt(self) -> None: for path in ( "scripts/check-policy.py", "scripts/check-doc-checkpoint.py", + "scripts/check-prompt-contract.py", "scripts/check-protocol-consistency.py", + ".agents/prompts/operator.md", "tests/scripts/test_doc_checkpoint.py", + "tests/scripts/test_check_prompt_contract.py", "tests/scripts/test_check_protocol_consistency.py", ): with self.subTest(path=path): @@ -146,7 +149,9 @@ def test_governance_only_task_one_files_are_not_a_checkpoint(self) -> None: ".agents/waivers.csv", "scripts/policy_contract.py", "scripts/check-policy.py", + "scripts/check-prompt-contract.py", "tests/scripts/test_policy_contract.py", + "tests/scripts/test_check_prompt_contract.py", ] self.assertEqual(doc_checkpoint.classify_changed_paths(paths), {"governance"}) self.assertEqual(doc_checkpoint.checkpoint_errors(set(paths)), []) From 7794e05d19dbbf553218d0c53ad35c8a2d1f3ab6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 10:34:42 +0000 Subject: [PATCH 06/12] policy(claims): validate live helper readiness Replace mutable claim snapshots with explicit local/live validation and require base-committed, executable gate and mutation proof before a READY task can be advertised. Preserve the retired timestamp procedure in the completed record. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .../completed/pre-cutover-claim-protocol.md | 26 + .agents/coordination.md | 37 +- scripts/check-doc-checkpoint.py | 27 + scripts/claim-view.py | 470 +++++++++----- scripts/ready-for-helper.py | 590 +++++++++++++++--- tests/scripts/test_claim_view.py | 281 ++++++--- tests/scripts/test_doc_checkpoint.py | 50 ++ tests/scripts/test_ready_for_helper.py | 315 ++++++++++ 8 files changed, 1443 insertions(+), 353 deletions(-) create mode 100644 .agents/completed/pre-cutover-claim-protocol.md create mode 100644 tests/scripts/test_ready_for_helper.py diff --git a/.agents/completed/pre-cutover-claim-protocol.md b/.agents/completed/pre-cutover-claim-protocol.md new file mode 100644 index 000000000..0d216a4b5 --- /dev/null +++ b/.agents/completed/pre-cutover-claim-protocol.md @@ -0,0 +1,26 @@ +# Pre-cutover claim snapshot protocol + +This is the evidence-preserving archive of the claim-view procedure retired by +the live-claim cutover. Feature lifecycle rows and claim narratives remain in +`.agents/coordination.md`; only this obsolete procedure and its empty generated +snapshot moved. + +## Retired procedure + +The former `scripts/claim-view.py --apply` queried open pull requests and wrote +a generated table into `.agents/coordination.md`. Offline `--check` accepted the +committed table for up to 14 days. An open `row/` PR rendered as a +reservation; merging or closing it removed the row only after another apply. + +The last committed snapshot was generated on 2026-08-04 and contained no rows: + +| Row | PR | State | Agent | Updated | +|---|---|---|---|---| +| _none_ | | | | | + +## Why it was retired + +The timestamp and TTL made remote state look locally authoritative between +refreshes. The replacement separates network-independent `--check-local` from +remote-authoritative `--check-live`; a failed query is `REMOTE_UNVERIFIED` and +cannot be interpreted as an unclaimed task. diff --git a/.agents/coordination.md b/.agents/coordination.md index abae8871a..030912e0c 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1601,18 +1601,31 @@ items a-runner/b stay with the async/GDN `runner.cpp` owners. | `CLAIM-DSV4-GGUF-LOADER` | `QUANT-GGUF-IQ2_XXS` (INVENTORIED→ACTIVE), `QUANT-GGUF-Q2_K` (INVENTORIED→ACTIVE); cross-refs `MODEL-TEXT-deepseek-v4-deepseek-v4-for-causal-lm` (stays `SPIKE`, owned by `CLAIM-DEEPSEEK-V4-IMPL`) | Claude Code (opus-4-8) | isolated worktree `.claude/worktrees/gguf-iquant-dsv4` (CPU-only `build-cpu` `-DVLLM_CPP_CUDA=OFF`; NO GPU, NO 90 GB download — the dequant unit gate uses known packed bytes; the GGUF header was HTTP-range-read, no download) | branch `feat/gguf-iquant-dsv4`, base `main` `4d1be010` (confirmed via `git rev-parse HEAD`) | GGUF IQ2_XXS + Q2_K dequant, the DeepSeek-V4-Flash single-Spark GGUF quant-path brick (W1). Owns ONLY the GGUF/quant PATH (NOT the forward — the forward TUs stay owned by `CLAIM-DEEPSEEK-V4-*`): the two `dequantize_row_*` decoders + grid/sign tables in `src/vt/cpu/cpu_quant_dequant.cpp`; the `kQ2_K`/`kIQ2_XXS` vt block dtype registration in `src/vt/dtype.{h,cpp}` + `src/vt/ops.cpp`; the id-16 reader trait in `gguf_reader.cpp` + ids-10/16 dispatch in `gguf_dequant.cpp`; `tests/vllm/test_gguf_dequant.cpp` + `tests/vt/test_ops_quant_traits.cpp`; the two `QUANT-GGUF-*` rows; NEW `.agents/specs/gguf-iquant-dsv4.md`; the V4 GGUF-loadable note on the model-matrix V4 row (row stays SPIKE); the record surfaces. **NON-COLLISION:** additive within the existing GGUF dequant switch + vt block table — does NOT touch any DeepSeek-V4 forward TU (`deepseek_v4.{cpp,h}`/`_dsa`/`_weights`), README, or Metal; the k-quant/NVFP4 decoders are byte-unchanged. | `ACTIVE` | 2026-07-29 — **W1 LANDED + CPU-GATED (foreground, NOT pushed).** IQ2_XXS (id 16, codebook `iq2xxs_grid`+signs+4-bit scale) + Q2_K (id 10, nibble sub-scale/min) ported 1:1 from llama.cpp `ggml-quants.c` `237ad9b96`; both DEQUANT-ONLY (no vec_dot ⇒ route to expand-bf16). `test_gguf_dequant` **15/15·480** (hand-derived literals: IQ2_XXS grid[1] byte0=0x2b→5.375, ksigns[1] flips j=0,7→±3.0, db 0.125/0.375; Q2_K 5.75/-0.25/2.5/0.25) + `test_ops_quant_traits` **9/9·5643** (dequant-only contract). All 7 changed TUs clean under full `-Werror`; the `voxtral.cpp` GCC-13 `-Werror=array-bounds` FP PROVEN pre-existing (fails at base with this diff's `dtype.h` reverted), neutralized only to link the test binaries. **W2 (V4-GGUF loader) DERIVED not landed:** HTTP-range-read the real `UD-IQ2_XXS` header — `general.architecture=deepseek4`, `general.file_type=19` (=IQ2_XXS), `split.tensors.count=1328`, full `deepseek4.*` config-KV schema; the tensor NAME manifest is beyond the CDN range cap + uncached ⇒ the V4 registry GGUF reject STAYS. Residuals: V4 forward (W3-W8, multi-Spark) + the V4-GGUF name map (W2, manifest-blocked) + a vec_dot perf leaf. | - - - -GENERATED from open pull requests by `scripts/claim-view.py --apply`. -Do not hand-edit: an open `row/` PR IS the reservation, and it -is released by merging or closing it. - -| Row | PR | State | Agent | Updated | -|---|---|---|---|---| -| _none_ | | | | | - - +## Live claim authority + +Claims are not snapshotted in this record. `scripts/claim-view.py --check-live` +validates current pull requests; `--check-local` proves that no stale snapshot +was recommitted. Remote failure is `REMOTE_UNVERIFIED`, never an empty claim +set. The retired timestamp/TTL procedure is preserved in +[completed/pre-cutover-claim-protocol.md](completed/pre-cutover-claim-protocol.md). + +A `READY` row is advertised to helpers only after its base-committed spec +contains exactly one closed execution contract: + +```text + +``` + +Both values are argv arrays, never shell strings. `ready-for-helper.py` reads +the spec and referenced executables from the configured base commit, expands +that exact commit into a disposable checkout, and runs both commands with no +shell, a sanitized environment, bounded diagnostics, and a timeout. The gate +must exit zero and the mutation command must exit nonzero. A mutable worktree +file, prose assertion, missing/nonregular program, or unsafe argv cannot prove +readiness. CPU or exact gate hardware, satisfied dependencies, `READY` +lifecycle, and the absence of a repository-matching live claim remain required. **16-bit CPU GEMM: wide x86 ISA tiers + tiled sgemm (`KERNEL-GEMM-CPU-ELEM-X86WIDE` + `KERNEL-GEMM-CPU-TILED`, 2026-08-06, `CLAIM-KERNEL-CPU-ELEM-WIDE-1`).** Claude diff --git a/scripts/check-doc-checkpoint.py b/scripts/check-doc-checkpoint.py index c88155c61..6ed262558 100755 --- a/scripts/check-doc-checkpoint.py +++ b/scripts/check-doc-checkpoint.py @@ -359,13 +359,34 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: "scripts/check-protocol-consistency.py", "scripts/check-role-discipline.py", "scripts/claim-view.py", + "scripts/ready-for-helper.py", "tests/scripts/test_agent_role.py", "tests/scripts/test_claim_view.py", + "tests/scripts/test_ready_for_helper.py", "tests/scripts/test_policy_contract.py", "tests/scripts/test_check_prompt_contract.py", "tests/scripts/test_doc_checkpoint.py", "tests/scripts/test_check_protocol_consistency.py", "docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md", + ".agents/completed/pre-cutover-claim-protocol.md", + } +) + +# coordination.md normally moves live feature state and therefore owes NOW plus +# the checkpoint projections. This one closed migration removes only its +# obsolete generated-claim snapshot. Scope the exception to the complete, +# exact cutover transaction so a later coordination edit -- alone or alongside +# feature work -- cannot inherit a blanket governance bypass. +CLAIM_CUTOVER_FILES = frozenset( + { + "scripts/claim-view.py", + "scripts/ready-for-helper.py", + "tests/scripts/test_claim_view.py", + "tests/scripts/test_ready_for_helper.py", + ".agents/coordination.md", + ".agents/completed/pre-cutover-claim-protocol.md", + "scripts/check-doc-checkpoint.py", + "tests/scripts/test_doc_checkpoint.py", } ) FEATURE_CHECKPOINT_FILES = frozenset( @@ -446,9 +467,15 @@ def classify_changed_paths(paths: list[str]) -> set[str]: classes: set[str] = set() path_set = set(paths) + exact_claim_cutover = ( + path_set == CLAIM_CUTOVER_FILES + ) for path in sorted(path_set): if path in PUBLIC_SURFACES: continue + if exact_claim_cutover and path == ".agents/coordination.md": + classes.add("governance") + continue if path in GOVERNANCE_FILES: classes.add("governance") continue diff --git a/scripts/claim-view.py b/scripts/claim-view.py index 06483f3ab..8cd8601af 100755 --- a/scripts/claim-view.py +++ b/scripts/claim-view.py @@ -1,188 +1,368 @@ #!/usr/bin/env python3 -"""Derive the claim view from open PRs instead of maintaining it by hand. (W2) +"""Validate helper claims from live pull-request state. -`coordination.md`'s hand-written claim table rotted for a structural reason: -claiming was free and never expired, while releasing cost anchor work. On -2026-08-04 it carried 106 rows asserting "implementation in flight" with nobody -flying them. - -A PR is already a self-evidently live-or-dead claim: open = reserved, merged or -closed = released, no upkeep. So the claim view is GENERATED from PR state into a -delimited block, and the block is a REPORT that is never hand-edited. - -Two modes, deliberately split so CI never needs the network: - - scripts/claim-view.py --apply # query GitHub, rewrite the block - scripts/claim-view.py --check # offline: the block is well-formed, - # its row IDs exist, and it is not stale - -The legacy hand-maintained table above the block stays until the protocol cuts -over, because `check-agent-record.py` still requires every SPIKE/ACTIVE row's -owner to appear there. Removing it before those rows have PRs would strand them. +The remote is the authority. Local preflight checks only that no obsolete +snapshot has been committed; ready/integration checks consume live PR JSON. +An unavailable remote is neither an empty claim set nor success. """ from __future__ import annotations import argparse +import importlib.util import json import re import subprocess import sys -import time from pathlib import Path +from urllib.parse import urlsplit ROOT = Path(__file__).resolve().parents[1] COORD = ROOT / ".agents/coordination.md" +ROW_BRANCH = re.compile(r"^row/([A-Za-z0-9][A-Za-z0-9_.-]*)$") +REMOTE_UNVERIFIED_EXIT = 4 +OPEN_STATE = "OPEN" +LIVE_FIELDS = ( + "number,state,headRefName,isDraft,title,author,headRepository" +) +FIXTURE_KEYS = frozenset({"expected", "prs"}) +EXPECTED_KEYS = frozenset({"repository", "base", "task_id", "head", "number"}) +REPOSITORY = re.compile(r"^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$") + + +class RemoteUnverified(RuntimeError): + """The authoritative PR state could not be obtained or decoded.""" + + +def _load_record(root: Path): + path = root / "scripts/check-agent-record.py" + spec = importlib.util.spec_from_file_location("claim_view_agent_record", path) + if spec is None or spec.loader is None: + raise ValueError(f"cannot load canonical record parser from {path}") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module -BEGIN = "" -END = "" -GENERATED = re.compile(r"") -# A claim with no PR behind it expires: this is the TTL that stops the rot. -STALE_AFTER_DAYS = 14 +def _location(row: object, root: Path) -> str: + path = Path(row.path) + try: + rendered = path.relative_to(root).as_posix() + except ValueError: + rendered = path.as_posix() + return f"{rendered}:{row.line_no}" -ROW_BRANCH = re.compile(r"^row/([A-Za-z0-9_.-]+)$") +def canonical_task_rows(root: Path = ROOT) -> list: + """Parse the task record once, rejecting duplicate identity before indexing.""" -def git(*args: str) -> str: - return subprocess.check_output(["git", *args], cwd=ROOT, text=True).strip() + record = _load_record(root) + errors: list[str] = [] + rows = [] + locations: dict[str, list[str]] = {} + for path in record.MATRIX_PATHS: + for row in record.parse_claim_rows(path, errors): + rows.append(row) + locations.setdefault(row.item_id, []).append(_location(row, root)) + errors.extend( + f"duplicate task ID {item_id} at " + ", ".join(found) + for item_id, found in sorted(locations.items()) + if len(found) > 1 + ) + if errors: + raise ValueError("canonical task record is invalid: " + "; ".join(errors)) + return rows -def known_row_ids() -> set[str]: - """Every stable row ID the matrices define.""" - import importlib.util +def known_task_ids(root: Path = ROOT) -> set[str]: + """Return task IDs from the existing canonical matrix record. - spec = importlib.util.spec_from_file_location( - "agent_record", ROOT / "scripts/check-agent-record.py" - ) - module = importlib.util.module_from_spec(spec) - sys.modules["agent_record"] = module - spec.loader.exec_module(module) - rows: set[str] = set() - for path in module.MATRIX_PATHS: - rows |= {r.item_id for r in module.parse_claim_rows(path, [])} - return rows + The task-aware helper transaction (and its governance-task registry) was + deliberately deferred from this PR. This adapter therefore has one source: + the matrices already parsed by check-agent-record.py. Live validation and + readiness both call this function, so they cannot disagree about identity. + """ + return {row.item_id for row in canonical_task_rows(root)} -def fetch_open_prs() -> list[dict]: - out = subprocess.check_output( - ["gh", "pr", "list", "--state", "open", "--limit", "200", - "--json", "number,headRefName,isDraft,title,author,updatedAt"], - cwd=ROOT, text=True, - ) - return json.loads(out) +def fetch_prs() -> list[dict]: + """Fetch the open PR set; raise a typed unknown-state result on failure.""" -def render(prs: list[dict], stamp: str) -> str: - rows = [] - for pr in sorted(prs, key=lambda p: p["number"]): - match = ROW_BRANCH.match(pr.get("headRefName", "")) - if not match: - continue - state = "draft" if pr.get("isDraft") else "ready" - author = (pr.get("author") or {}).get("login", "?") - rows.append( - f"| `{match.group(1)}` | #{pr['number']} | {state} | {author} | " - f"{pr.get('updatedAt', '')[:10]} |" + try: + result = subprocess.run( + [ + "gh", "pr", "list", "--state", "open", "--limit", "500", + "--json", LIVE_FIELDS, + ], + cwd=ROOT, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, ) - body = "\n".join(rows) if rows else "| _none_ | | | | |" - return "\n".join([ - BEGIN, - f"", - "", - "GENERATED from open pull requests by `scripts/claim-view.py --apply`.", - "Do not hand-edit: an open `row/` PR IS the reservation, and it", - "is released by merging or closing it.", - "", - "| Row | PR | State | Agent | Updated |", - "|---|---|---|---|---|", - body, - "", - END, - ]) - - -def block_bounds(text: str) -> tuple[int, int] | None: - start, end = text.find(BEGIN), text.find(END) - if start == -1 or end == -1 or end < start: - return None - return start, end + len(END) - - -def check_errors(text: str, rows: set[str]) -> list[str]: - """Offline validation of the generated block.""" - bounds = block_bounds(text) - if bounds is None: - return [ - f".agents/coordination.md is missing the claim view block " - f"({BEGIN} ... {END}); regenerate with scripts/claim-view.py --apply" - ] - block = text[bounds[0]:bounds[1]] + except OSError as exc: + raise RemoteUnverified(str(exc)) from exc + if result.returncode != 0: + detail = result.stderr.strip() or f"gh exited {result.returncode}" + raise RemoteUnverified(detail) + try: + value = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise RemoteUnverified(f"invalid PR JSON: {exc}") from exc + if not isinstance(value, list) or any(not isinstance(item, dict) for item in value): + raise RemoteUnverified("PR response must be a JSON array of objects") + return value + + +def _closed_object(pairs: list[tuple[str, object]]) -> dict: + result: dict = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key {key!r}") + result[key] = value + return result + + +def _validate_expected(value: object) -> dict: + if not isinstance(value, dict): + raise RemoteUnverified("fixture expected must be an object") + unknown = sorted(set(value) - EXPECTED_KEYS) + if unknown: + raise RemoteUnverified("fixture expected has unknown keys: " + ", ".join(unknown)) + repository = value.get("repository") + if not isinstance(repository, str) or REPOSITORY.fullmatch(repository) is None: + raise RemoteUnverified("fixture expected.repository is required and must be owner/name") + base = value.get("base") + if base is not None and (not isinstance(base, str) or not base.strip()): + raise RemoteUnverified("fixture expected.base must be a nonempty string") + task_id = value.get("task_id") + if task_id is not None and ( + not isinstance(task_id, str) + or re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", task_id) is None + ): + raise RemoteUnverified("fixture expected.task_id must be a canonical task ID") + head = value.get("head") + if head is not None and ( + not isinstance(head, str) + or ROW_BRANCH.fullmatch(head) is None + or task_id is None + ): + raise RemoteUnverified("fixture expected.head requires task_id and canonical row head") + number = value.get("number") + if number is not None and _pr_number(number) is None: + raise RemoteUnverified("fixture expected.number must be a positive integer") + return value + + +def load_pr_fixture(path: Path) -> tuple[list[dict], dict]: + try: + value = json.loads( + path.read_text(encoding="utf-8"), object_pairs_hook=_closed_object + ) + except (OSError, json.JSONDecodeError, ValueError) as exc: + raise RemoteUnverified(f"cannot read PR fixture {path}: {exc}") from exc + if not isinstance(value, dict) or set(value) != FIXTURE_KEYS: + raise RemoteUnverified("PR fixture must be a closed {expected, prs} object") + prs = value["prs"] + if not isinstance(prs, list) or any(not isinstance(item, dict) for item in prs): + raise RemoteUnverified("fixture prs must be an array of objects") + return prs, _validate_expected(value["expected"]) + + +def repository_identity(root: Path = ROOT) -> str: + """Derive exact owner/name authority from this checkout's origin URL.""" + + result = subprocess.run( + ["git", "remote", "get-url", "origin"], + cwd=root, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + if result.returncode != 0: + raise RemoteUnverified("cannot derive expected repository from git origin") + remote = result.stdout.strip() + if "://" in remote: + path = urlsplit(remote).path + else: + match = re.fullmatch(r"[^@:/]+@[^:/]+:(.+)", remote) + path = match.group(1) if match else remote + candidate = path.strip("/") + if candidate.endswith(".git"): + candidate = candidate[:-4] + parts = candidate.split("/") + identity = "/".join(parts[-2:]) if len(parts) >= 2 else "" + if REPOSITORY.fullmatch(identity) is None: + raise RemoteUnverified(f"cannot derive owner/name from origin {remote!r}") + return identity + + +def _pr_number(value: object) -> int | None: + return value if isinstance(value, int) and not isinstance(value, bool) and value > 0 else None + + +def validate_live_claims( + prs: list[dict], + known_tasks: set[str], + expected: dict | None = None, +) -> list[str]: + """Validate a complete live claim set with deterministic diagnostics.""" + errors: list[str] = [] + numbers: dict[int, int] = {} + task_claims: dict[str, list[int]] = {} + valid_claims: list[tuple[int, str, str, dict]] = [] + expected_repository = expected.get("repository") if expected is not None else None + if prs and ( + not isinstance(expected_repository, str) + or REPOSITORY.fullmatch(expected_repository) is None + ): + errors.append("nonempty PR input lacks expected repository identity") + + for index, item in enumerate(prs): + label = f"PR entry {index + 1}" + number = _pr_number(item.get("number")) + if number is None: + errors.append(f"{label} has malformed PR number") + else: + numbers[number] = numbers.get(number, 0) + 1 + label = f"PR #{number}" - stamp = GENERATED.search(block) - if not stamp: - errors.append("the claim view has no stamp") - else: - try: - age = time.time() - time.mktime(time.strptime(stamp.group(1), "%Y-%m-%d")) - if age > STALE_AFTER_DAYS * 86400: - errors.append( - f"the claim view was generated {int(age // 86400)} days ago, over " - f"the {STALE_AFTER_DAYS}-day TTL; a claim with no live PR behind " - "it must expire rather than rot. Re-run --apply" - ) - except ValueError: - errors.append(f"unparseable claim-view stamp {stamp.group(1)!r}") - - for line in block.splitlines(): - if not line.startswith("| `"): - continue - row = line.split("`")[1] - if row not in rows: - errors.append(f"the claim view references unknown row {row}") - return errors + state = item.get("state") + if state not in {"OPEN", "CLOSED", "MERGED"}: + errors.append(f"{label} has malformed state {state!r}") + repository = item.get("headRepository") + repository_name = ( + repository.get("nameWithOwner") if isinstance(repository, dict) else None + ) + if not isinstance(repository_name, str) or REPOSITORY.fullmatch(repository_name) is None: + errors.append(f"{label} has malformed head repository identity") + elif expected_repository is not None and repository_name != expected_repository: + errors.append( + f"{label} repository {repository_name!r} does not match expected " + f"{expected_repository!r}" + ) + + head = item.get("headRefName") + if not isinstance(head, str): + errors.append(f"{label} has malformed head identity") + continue + match = ROW_BRANCH.fullmatch(head) + if match is None: + # Unrelated PRs are not claims. A row-like malformed branch is an + # attempted claim and must not disappear from validation. + if head.startswith("row/"): + errors.append(f"{label} has malformed row head {head!r}") + continue -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - mode = parser.add_mutually_exclusive_group() - mode.add_argument("--apply", action="store_true", help="query GitHub and rewrite") - mode.add_argument("--check", action="store_true", help="offline validation (default)") - args = parser.parse_args() + task = match.group(1) + if state != OPEN_STATE: + errors.append(f"{label} for task {task} is {state}, not a live claim") + if task not in known_tasks: + errors.append(f"{label} claims unknown task {task}") + if number is not None: + task_claims.setdefault(task, []).append(number) + valid_claims.append((number, task, head, item)) + + for number, count in sorted(numbers.items()): + if count > 1: + errors.append(f"PR number #{number} is ambiguous ({count} entries)") + for task, claims in sorted(task_claims.items()): + distinct = sorted(set(claims)) + if len(distinct) > 1: + rendered = ", ".join(f"PR #{number}" for number in distinct) + errors.append(f"task {task} has duplicate open claims: {rendered}") + + if expected is not None and "task_id" in expected: + task = expected.get("task_id") + if not isinstance(task, str) or task not in known_tasks: + errors.append(f"expected claim has unknown or malformed task_id {task!r}") + else: + matches = [claim for claim in valid_claims if claim[1] == task] + if not matches: + if len(valid_claims) == 1: + number, _, head, _ = valid_claims[0] + expected_head = expected.get("head", f"row/{task}") + errors.append( + f"PR #{number} has wrong head {head!r}; expected {expected_head!r}" + ) + errors.append(f"expected live claim for task {task} is missing") + else: + expected_head = expected.get("head", f"row/{task}") + for number, _, head, item in matches: + if head != expected_head: + errors.append( + f"PR #{number} has wrong head {head!r}; expected {expected_head!r}" + ) + expected_number = expected.get("number") + if expected_number is not None and not any( + number == expected_number for number, *_ in matches + ): + got = ", ".join(f"#{number}" for number, *_ in matches) + errors.append( + f"expected PR #{expected_number} for task {task}; found {got}" + ) + return sorted(set(errors)) + + +def claimed_tasks(prs: list[dict]) -> set[str]: + """Return only canonical OPEN row claims after callers validate the set.""" + + result = set() + for item in prs: + match = ROW_BRANCH.fullmatch(item.get("headRefName", "")) + if match is not None and item.get("state") == OPEN_STATE: + result.add(match.group(1)) + return result + + +def local_errors(text: str) -> list[str]: + if "" in text or "" in text: + return [ + "committed claim-view snapshots are forbidden; live PR state is authoritative" + ] + return [] - text = COORD.read_text(encoding="utf-8") - if args.apply: +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument("--check-local", action="store_true") + mode.add_argument("--check-live", action="store_true") + mode.add_argument("--check", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--pr-json", type=Path, help="offline live-state fixture") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + if args.check_local or args.check: + if args.pr_json is not None: + _parser().error("--pr-json requires --check-live") + failures = local_errors(COORD.read_text(encoding="utf-8")) + else: try: - prs = fetch_open_prs() - except (subprocess.CalledProcessError, FileNotFoundError) as exc: - print(f"ERROR: could not query GitHub ({exc}); --apply needs `gh`", - file=sys.stderr) - return 1 - stamp = time.strftime("%Y-%m-%d") - rendered = render(prs, stamp) - bounds = block_bounds(text) - if bounds: - text = text[:bounds[0]] + rendered + text[bounds[1]:] - else: - anchor = "## Handoff queue" - if anchor not in text: - print("ERROR: no place to insert the claim view", file=sys.stderr) - return 1 - text = text.replace(anchor, rendered + "\n\n" + anchor, 1) - COORD.write_text(text, encoding="utf-8") - claimed = sum(1 for line in rendered.splitlines() if line.startswith("| `")) - print(f"claim view regenerated: {claimed} row(s) reserved by open PRs") - return 0 - - failures = check_errors(text, known_row_ids()) + if args.pr_json: + prs, expected = load_pr_fixture(args.pr_json) + else: + prs = fetch_prs() + expected = {"repository": repository_identity(ROOT)} + failures = validate_live_claims(prs, known_task_ids(ROOT), expected) + except (RemoteUnverified, ValueError) as exc: + print(f"REMOTE_UNVERIFIED: {exc}", file=sys.stderr) + return REMOTE_UNVERIFIED_EXIT + if failures: for failure in failures: print(f"ERROR: {failure}", file=sys.stderr) return 1 - print("OK: the claim view is present, well-formed and inside its TTL.") + mode = "local record" if (args.check_local or args.check) else "live PR claims" + print(f"OK: {mode} validated.") return 0 diff --git a/scripts/ready-for-helper.py b/scripts/ready-for-helper.py index 62171ea28..535793167 100755 --- a/scripts/ready-for-helper.py +++ b/scripts/ready-for-helper.py @@ -1,150 +1,548 @@ #!/usr/bin/env python3 -"""Compute which rows a helper may pick. (W3) - -A helper picking an arbitrary roadmap row hits what the 2026-08-04 audit found: -the row's state is a lie, its spec is missing, its anchors do not exist. It then -produces a PR the operator must reject. So helpers only ever see a queue, and a -row enters that queue only when all five conditions hold: - -1. a committed `.agents/specs/.md` covering the spike contract; -2. its gates are defined and runnable (a real test/evidence anchor); -3. it is CPU-gateable, or its hardware need is named on the row; -4. it does not depend on an unmerged PR; -5. no open PR already claims it (read from the generated claim view). - - scripts/ready-for-helper.py # print the queue - scripts/ready-for-helper.py --check # assert the computation is sound - -`--check` is what CI runs: it never asserts the queue is NON-empty (an empty -queue is a true and useful answer today, since 98 rows fail condition 1 or 2), -only that every row it WOULD offer really satisfies the conditions. -""" +"""Offer only helper tasks whose readiness is mechanically proven.""" from __future__ import annotations import argparse +import io import importlib.util +import json +import os import re +import shutil +import stat +import subprocess import sys +import tarfile +import tempfile +from pathlib import PurePosixPath from pathlib import Path ROOT = Path(__file__).resolve().parents[1] -PICKABLE_STATES = {"READY", "INVENTORIED"} -# Words that mean "this needs hardware"; the row must say so explicitly. -HARDWARE_HINTS = ("dgx", "gb10", "sm_121", "cuda", "gpu", "metal", "m4", "thor") +PICKABLE_STATES = frozenset({"READY"}) +SATISFIED_STATES = frozenset({"DONE", "NOT-APPLICABLE"}) +PLACEHOLDERS = frozenset({"", "-", "none", "n/a", "not applicable"}) +LINK = re.compile(r"\[[^]]*\]\(([^)]+)\)") +CONTRACT = re.compile( + r"", re.DOTALL +) +CONTRACT_KEYS = frozenset({"gate", "mutation"}) +ALLOWED_PROGRAMS = frozenset( + {"python3", "python", "pytest", "ctest", "cmake", "make", "bash", "sh"} +) +SHELL_META = re.compile(r"[;&|`$<>\\\r\n*?{}\[\]]") +ENV_ASSIGNMENT = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=") +RUN_TIMEOUT_SECONDS = 30 +DIAGNOSTIC_LIMIT = 2000 +CPU_GATE = re.compile(r"(?im)^\s*(?:[-*]\s*)?(?:gate\s+)?hardware\s*:\s*CPU\s*\.??\s*$") +EXACT_HARDWARE = re.compile( + r"(?im)^\s*(?:[-*]\s*)?(?:gate\s+)?hardware\s*:\s*" + r"(?=[^\n]*(?:\bGB[0-9]+\b|\bsm_[0-9]+\b|\bsm[0-9]{2,3}\b))" + r"[^\n]+\s*\.??\s*$" +) def _load(name: str, relative: str): - spec = importlib.util.spec_from_file_location(name, ROOT / relative) + path = ROOT / relative + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load {path}") module = importlib.util.module_from_spec(spec) sys.modules[name] = module spec.loader.exec_module(module) return module -rec = _load("agent_record", "scripts/check-agent-record.py") +rec = _load("ready_agent_record", "scripts/check-agent-record.py") claim_view = _load("claim_view", "scripts/claim-view.py") -def reserved_rows() -> set[str]: - """Rows already claimed by an open PR, per the generated view.""" - text = (ROOT / ".agents/coordination.md").read_text(encoding="utf-8") - bounds = claim_view.block_bounds(text) - if bounds is None: - return set() - return { - line.split("`")[1] - for line in text[bounds[0]:bounds[1]].splitlines() - if line.startswith("| `") - } +def _field(row, name: str) -> str: + direct = row.field(name) + if direct: + return direct + wanted = rec.normalize_header(name) + for index, header in enumerate(row.header): + if header == wanted or (name == "dependencies" and "dependenc" in header): + return row.cells[index] if index < len(row.cells) else "" + return "" + +def _spec_paths(row, root: Path) -> list[str]: + """Return lexical repository paths without consulting mutable worktree files.""" -def spec_ok(row) -> bool: - for candidate in rec.local_spec_paths(row): - path = candidate if candidate.is_absolute() else ROOT / candidate - if not path.exists(): + spec_root = PurePosixPath(".agents/specs") + paths: list[str] = [] + for target in LINK.findall(_field(row, "spec")): + raw = target.strip().strip("<>").split("#", 1)[0] + if not raw or raw.startswith(("http://", "https://")): continue try: - if rec.has_substantive_spec_content( - path.read_text(encoding="utf-8").splitlines() - ): - return True - except OSError: + matrix_relative = row.path.relative_to(root).parent.as_posix() + except ValueError: continue - return False - - -def hardware_declared(row) -> bool: - """CPU-gateable, or the hardware need is named on the row.""" - blob = " ".join(row.cells).lower() - needs_hw = any(hint in blob for hint in HARDWARE_HINTS) - return (not needs_hw) or ("dgx" in blob or "gb10" in blob or "hardware" in blob) - - -def evaluate(row, reserved: set[str]) -> list[str]: - """Return the conditions this row FAILS (empty means pickable).""" - missing = [] - if not spec_ok(row): - missing.append("no substantive spec") - if not rec.is_test_anchor(row.field("tests"), row.path): - missing.append("no runnable gate/evidence anchor") - if not hardware_declared(row): - missing.append("hardware need not named") - if row.item_id in reserved: + candidate = PurePosixPath(matrix_relative) / PurePosixPath(raw) + if candidate.is_absolute() or ".." in candidate.parts: + continue + normalized = PurePosixPath(*[part for part in candidate.parts if part != "."]) + try: + normalized.relative_to(spec_root) + except ValueError: + continue + if normalized.suffix == ".md": + paths.append(normalized.as_posix()) + return paths + + +def _git(root: Path, *args: str, text: bool = True) -> subprocess.CompletedProcess: + return subprocess.run( + ["git", *args], + cwd=root, + text=text, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + + +def _resolve_base(root: Path, base: str) -> tuple[str | None, str | None]: + if not isinstance(base, str) or not base.strip(): + return None, "base commit is not configured" + result = _git(root, "rev-parse", "--verify", "--end-of-options", f"{base}^{{commit}}") + commit = result.stdout.strip() if result.returncode == 0 else "" + if not re.fullmatch(r"[0-9a-f]{40,64}", commit): + return None, f"base commit {base!r} is not configured and reachable" + return commit, None + + +def _base_mode(root: Path, commit: str, relative: str) -> str | None: + result = _git(root, "ls-tree", "-z", commit, "--", relative, text=False) + if result.returncode != 0 or not result.stdout: + return None + entries = result.stdout.split(b"\0") + exact = [] + for entry in entries: + if not entry or b"\t" not in entry: + continue + metadata, path = entry.split(b"\t", 1) + if path.decode("utf-8", "surrogateescape") == relative: + exact.append(metadata.split(b" ", 1)[0].decode("ascii", "replace")) + return exact[0] if len(exact) == 1 else None + + +def _base_blob(root: Path, commit: str, relative: str) -> str | None: + mode = _base_mode(root, commit, relative) + if mode not in {"100644", "100755"}: + return None + result = _git(root, "show", f"{commit}:{relative}", text=False) + if result.returncode != 0: + return None + try: + return result.stdout.decode("utf-8") + except UnicodeDecodeError: + return None + + +def _reachable_spec(row, root: Path, commit: str) -> tuple[str, str] | None: + for relative in _spec_paths(row, root): + text = _base_blob(root, commit, relative) + if text is not None: + return relative, text + return None + + +def _pairs_no_duplicates(pairs: list[tuple[str, object]]) -> dict: + result = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate key {key!r}") + result[key] = value + return result + + +def _parse_contract(text: str) -> tuple[dict[str, list[str]] | None, list[str]]: + matches = list(CONTRACT.finditer(text)) + if len(matches) != 1: + return None, ["spec needs exactly one structured helper-readiness:v1 block"] + try: + value = json.loads( + matches[0].group("body"), object_pairs_hook=_pairs_no_duplicates + ) + except (json.JSONDecodeError, ValueError) as exc: + return None, [f"structured readiness JSON is invalid: {exc}"] + if not isinstance(value, dict) or set(value) != CONTRACT_KEYS: + return None, ["structured readiness object must contain exactly gate and mutation"] + errors = [] + for name in ("gate", "mutation"): + argv = value.get(name) + if ( + not isinstance(argv, list) + or not argv + or len(argv) > 64 + or any(not isinstance(arg, str) or not arg or len(arg) > 1024 for arg in argv) + ): + errors.append( + f"structured {name} must be a nonempty bounded argv string array" + ) + return (value if not errors else None), errors + + +def _validate_argv( + name: str, argv: list[str], root: Path, commit: str +) -> list[str]: + errors: list[str] = [] + for arg in argv: + path = PurePosixPath(arg) + if ( + path.is_absolute() + or ".." in path.parts + or re.search(r"(?:^|[=/])\.\.(?:/|$)", arg) + or re.search(r"(?:^|=)/", arg) + or SHELL_META.search(arg) + or ENV_ASSIGNMENT.match(arg) + or arg == "-c" + or arg.startswith("-c") + or arg == "--eval" + or arg.startswith("--eval=") + or "\x00" in arg + ): + errors.append(f"unsafe {name} argv element {arg!r}") + if errors: + return errors + + program = argv[0] + if "/" not in program: + if program not in ALLOWED_PROGRAMS: + return [f"{name} executable {program!r} is not in the allowlist"] + executable = shutil.which(program, path=os.defpath) + if executable is None or not Path(executable).resolve().is_file(): + return [f"{name} executable {program!r} is missing or nonregular"] + if program in {"python3", "python", "bash", "sh"} and ( + len(argv) < 2 or argv[1].startswith("-") + ): + return [f"unsafe {name} argv: {program} requires a relative base script"] + if program in {"python3", "python", "bash", "sh"} and _base_mode( + root, commit, PurePosixPath(argv[1]).as_posix() + ) not in {"100644", "100755"}: + return [f"{name} argv path {argv[1]!r} is not a regular base file"] + else: + mode = _base_mode(root, commit, PurePosixPath(program).as_posix()) + if mode != "100755": + return [f"{name} executable must be a regular executable base file"] + + for arg in argv[1:]: + if arg.startswith("-"): + continue + mode = _base_mode(root, commit, PurePosixPath(arg).as_posix()) + if "/" not in arg and mode is None: + continue + if mode not in {"100644", "100755"}: + errors.append(f"{name} argv path {arg!r} is not a regular base file") + return errors + + +def _bounded(value: str) -> str: + value = value.replace("\r\n", "\n").replace("\r", "\n") + return value[:DIAGNOSTIC_LIMIT] + + +def _safe_tmp_root() -> Path | None: + candidate = Path(os.environ.get("TMPDIR", "/tmp")) + try: + info = candidate.lstat() + except OSError: + return None + if ( + not candidate.is_absolute() + or candidate.is_symlink() + or not stat.S_ISDIR(info.st_mode) + or candidate.resolve() != candidate + or info.st_uid not in {0, os.getuid()} + or (info.st_mode & stat.S_IWOTH and not info.st_mode & stat.S_ISVTX) + ): + return None + return candidate + + +def _execute_contract( + root: Path, commit: str, contract: dict[str, list[str]] +) -> list[str]: + errors = [] + for name in ("gate", "mutation"): + errors.extend(_validate_argv(name, contract[name], root, commit)) + if errors: + return errors + + safe_tmp = _safe_tmp_root() + if safe_tmp is None: + return ["configured safe temporary directory is unavailable"] + try: + archive = _git(root, "archive", "--format=tar", commit, text=False) + if archive.returncode != 0: + return ["could not materialize exact base archive"] + with tempfile.TemporaryDirectory(prefix="vllm-helper-ready-", dir=safe_tmp) as directory: + checkout = Path(directory) / "checkout" + checkout.mkdir(mode=0o700) + with tarfile.open(fileobj=io.BytesIO(archive.stdout), mode="r:") as stream: + stream.extractall(checkout, filter="data") + run_tmp = checkout / ".helper-tmp" + run_tmp.mkdir(mode=0o700) + environment = { + "HOME": str(checkout), + "LANG": "C", + "LC_ALL": "C", + "PATH": os.defpath, + "PYTHONHASHSEED": "0", + "TMPDIR": str(run_tmp), + } + results = {} + for name in ("gate", "mutation"): + try: + results[name] = subprocess.run( + contract[name], + cwd=checkout, + env=environment, + text=True, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + shell=False, + timeout=RUN_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + errors.append(f"{name} could not execute: {_bounded(str(exc))}") + gate = results.get("gate") + mutation = results.get("mutation") + if gate is not None and gate.returncode != 0: + detail = _bounded(gate.stderr or gate.stdout).strip() + errors.append( + f"gate exited {gate.returncode}" + (f": {detail}" if detail else "") + ) + if mutation is not None and mutation.returncode == 0: + errors.append("mutation unexpectedly exited 0") + except (OSError, tarfile.TarError) as exc: + errors.append(f"could not materialize exact base archive: {_bounded(str(exc))}") + return errors + + +def _hardware_proof(text: str) -> bool: + return bool(CPU_GATE.search(text) or EXACT_HARDWARE.search(text)) + + +def _parse_dependencies(value: str, known_tasks: set[str]) -> tuple[list[str], bool]: + normalized = value.strip().strip("`").strip().casefold() + if normalized in PLACEHOLDERS: + return [], True + quoted = re.findall(r"`([A-Za-z0-9][A-Za-z0-9_.-]*)`", value) + if not quoted or any(item not in known_tasks for item in quoted): + return [], False + residue = re.sub(r"`[A-Za-z0-9][A-Za-z0-9_.-]*`", "", value) + residue = re.sub(r"[\s,;+&()]+", "", residue) + if residue: + return [], False + return quoted, True + + +def evaluate( + row, + live_claims: set[str], + known_tasks: set[str], + task_states: dict[str, str], + *, + root: Path = ROOT, + base: str = "origin/main", +) -> list[str]: + """Return every failed readiness proof for one candidate row.""" + + missing: list[str] = [] + if row.item_id not in known_tasks: + missing.append("unknown task ID") + if row.state not in PICKABLE_STATES: + missing.append(f"lifecycle {row.state} is not pickable") + return missing + + commit, base_error = _resolve_base(root, base) + if base_error is not None or commit is None: + missing.append(base_error or "base commit is unavailable") + text = "" + else: + spec = _reachable_spec(row, root, commit) + if spec is None: + text = "" + else: + _, text = spec + + if text: + contract, contract_errors = _parse_contract(text) + missing.extend(contract_errors) + if contract is not None and commit is not None: + missing.extend(_execute_contract(root, commit, contract)) + elif base_error is None: + missing.append("no base-reachable committed spec") + if not _hardware_proof(text): + missing.append("CPU or exact gate hardware is not declared") + + dependencies, parsed = _parse_dependencies( + _field(row, "dependencies"), known_tasks + ) + if not parsed: + missing.append("dependencies are not fully parsed") + else: + for dependency in dependencies: + state = task_states.get(dependency) + if state not in SATISFIED_STATES: + missing.append( + f"dependency {dependency} is {state or 'UNKNOWN'}, not satisfied" + ) + + if row.item_id in live_claims: missing.append("already claimed by an open PR") return missing -def queue() -> tuple[list, dict[str, int]]: - rows = [] - for path in rec.MATRIX_PATHS: - rows.extend(rec.parse_claim_rows(path, [])) - reserved = reserved_rows() +def _rows(root: Path = ROOT) -> list: + return claim_view.canonical_task_rows(root) + + +def queue( + prs: list[dict] | None = None, + *, + root: Path = ROOT, + base: str = "origin/main", + expected: dict | None = None, +) -> tuple[list, dict[str, int]]: + if prs is None: + # Local callers (including onboarding and preflight before Task22's + # final wiring) get no advertised work, never an assumed-empty remote. + rows = _rows(root) + return [], { + "live claims not verified": sum( + row.state in PICKABLE_STATES for row in rows + ) + } + rows = _rows(root) + known = {row.item_id for row in rows} + if expected is None: + expected = {"repository": claim_view.repository_identity(root), "base": base} + claim_errors = claim_view.validate_live_claims(prs, known, expected) + if claim_errors: + raise ValueError("; ".join(claim_errors)) + states = {row.item_id: row.state for row in rows} + live = claim_view.claimed_tasks(prs) pickable, reasons = [], {} for row in rows: - if row.state not in PICKABLE_STATES: - continue - missing = evaluate(row, reserved) - if missing: - for reason in missing: + failures = evaluate( + row, + live, + known, + states, + root=root, + base=base, + ) + if failures: + for reason in failures: reasons[reason] = reasons.get(reason, 0) + 1 else: pickable.append(row) return pickable, reasons -def main() -> int: +def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--check", action="store_true") - args = parser.parse_args() + mode = parser.add_mutually_exclusive_group() + mode.add_argument("--check", action="store_true") + mode.add_argument("--check-local", action="store_true") + mode.add_argument("--check-live", action="store_true") + parser.add_argument("--pr-json", type=Path, help="offline live-state fixture") + parser.add_argument("--base", default="origin/main") + return parser - pickable, reasons = queue() - reserved = reserved_rows() - if args.check: - # Never assert the queue is non-empty: an empty queue is a true answer. - # Assert instead that everything offered really satisfies every rule. - bad = [r.item_id for r in pickable if evaluate(r, reserved)] - if bad: - for item in bad: - print(f"ERROR: {item} is offered to helpers but fails a condition", - file=sys.stderr) +def main(argv: list[str] | None = None) -> int: + parser = _parser() + args = parser.parse_args(argv) + # Compatibility until the final gate-wiring task updates agent-preflight: + # bare --check is deliberately LOCAL_ONLY. It validates no remote absence + # and advertises no queue. Supplying --pr-json, or printing the queue + # without --check, is the remote-aware path. + local_check = args.check or args.check_local + if local_check and args.pr_json is not None: + parser.error("local checks reject --pr-json") + if local_check and args.base != "origin/main": + parser.error("local checks reject --base") + if args.pr_json is not None and not args.check_live: + parser.error("--pr-json requires --check-live") + if local_check: + local = claim_view.local_errors( + (ROOT / ".agents/coordination.md").read_text(encoding="utf-8") + ) + try: + rows = _rows(ROOT) + known = {row.item_id for row in rows} + except ValueError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + if local: + for failure in local: + print(f"ERROR: {failure}", file=sys.stderr) + return 1 + unknown = sorted(row.item_id for row in rows if row.item_id not in known) + if unknown: + print("ERROR: local rows contain unknown task IDs: " + ", ".join(unknown), file=sys.stderr) return 1 print( - f"OK: {len(pickable)} row(s) are READY-FOR-HELPER; every offered row " - "satisfies spec, gate, hardware and reservation conditions." + "OK: LOCAL_ONLY readiness structure validated; live claims were not " + "verified and no helper queue was advertised." + ) + return 0 + if not args.check_live and args.pr_json is None: + pickable, reasons = queue(None, base=args.base) + print(f"READY-FOR-HELPER queue: {len(pickable)} task(s) [LOCAL_ONLY]\n") + print("Live claims were not verified; no helper task is advertised.") + if reasons: + for reason, count in sorted(reasons.items()): + print(f" {count:4} {reason}") + return 0 + try: + if args.pr_json is not None: + prs, expected = claim_view.load_pr_fixture(args.pr_json) + else: + prs = claim_view.fetch_prs() + expected = { + "repository": claim_view.repository_identity(ROOT), + "base": args.base, + } + known = claim_view.known_task_ids(ROOT) + failures = claim_view.validate_live_claims(prs, known, expected) + if failures: + for failure in failures: + print(f"ERROR: {failure}", file=sys.stderr) + return 1 + base = expected.get("base", args.base) + pickable, reasons = queue(prs, base=base, expected=expected) + except claim_view.RemoteUnverified as exc: + print(f"REMOTE_UNVERIFIED: {exc}", file=sys.stderr) + return claim_view.REMOTE_UNVERIFIED_EXIT + except ValueError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + + if args.check: + print( + f"OK: {len(pickable)} task(s) are READY-FOR-HELPER; every offered " + "task has reachable spec, failing-mutation gate, hardware, dependencies, " + "lifecycle and live-claim proof." ) return 0 - print(f"READY-FOR-HELPER queue: {len(pickable)} row(s)\n") - for row in pickable[:40]: - print(f" {row.item_id[:52]:52} {row.state:12} {row.path.name}") - if len(pickable) > 40: - print(f" ... (+{len(pickable) - 40} more)") + print(f"READY-FOR-HELPER queue: {len(pickable)} task(s)\n") + for row in pickable: + print(f" {row.item_id[:64]:64} {row.state:8} {row.path.name}") if reasons: print("\nWhy the rest are not pickable:") - for reason, count in sorted(reasons.items(), key=lambda kv: -kv[1]): + for reason, count in sorted(reasons.items(), key=lambda item: (-item[1], item[0])): print(f" {count:4} {reason}") return 0 diff --git a/tests/scripts/test_claim_view.py b/tests/scripts/test_claim_view.py index 27da326d0..7f6cc0655 100644 --- a/tests/scripts/test_claim_view.py +++ b/tests/scripts/test_claim_view.py @@ -1,18 +1,17 @@ #!/usr/bin/env python3 -"""Unit and mutation checks for W2-W4. - -scripts/claim-view.py - the generated, PR-derived claim view -scripts/ready-for-helper.py - the pickable queue -scripts/check-pr-size.py - the reviewability cap -""" +"""Behavior checks for live, PR-derived helper claims.""" from __future__ import annotations import importlib.util +import io +import json import sys -import time +import tempfile import unittest +from contextlib import redirect_stderr from pathlib import Path +from unittest import mock ROOT = Path(__file__).resolve().parents[2] @@ -27,108 +26,190 @@ def _load(name: str, relative: str): view = _load("claim_view", "scripts/claim-view.py") -ready = _load("ready_for_helper", "scripts/ready-for-helper.py") -prsize = _load("pr_size", "scripts/check-pr-size.py") -KNOWN = {"ENG-FOO", "KV-BAR"} +KNOWN = {"ENG-FOO", "KV-BAR", "internal-policy-optimization-1"} +EXPECTED = {"repository": "owner/repo", "base": "HEAD"} + + +def pr(number: int, task: str, *, state: str = "OPEN", head: str | None = None) -> dict: + return { + "number": number, + "state": state, + "headRefName": head if head is not None else f"row/{task}", + "isDraft": True, + "title": task, + "author": {"login": "helper"}, + "headRepository": {"nameWithOwner": "owner/repo"}, + } + + +class KnownTasks(unittest.TestCase): + def test_shared_adapter_reads_canonical_matrix_ids(self) -> None: + ids = view.known_task_ids(ROOT) + self.assertIn("ENG-ASYNC-SCHED", ids) + self.assertNotIn("NOT-A-REAL-TASK", ids) + + def test_duplicate_ids_within_or_across_matrices_are_rejected_with_locations(self) -> None: + class DuplicateRow: + def __init__(self, path: Path, line_no: int) -> None: + self.item_id = "ENG-DUP" + self.path = path + self.line_no = line_no + + first = ROOT / ".agents/engine-matrix.md" + second = ROOT / ".agents/feature-matrix.md" + fake = mock.Mock() + fake.MATRIX_PATHS = (first, second) + fake.parse_claim_rows.side_effect = ( + [DuplicateRow(first, 10), DuplicateRow(first, 11)], + [DuplicateRow(second, 20)], + ) + with mock.patch.object(view, "_load_record", return_value=fake): + with self.assertRaisesRegex( + ValueError, + r"duplicate task ID ENG-DUP.*engine-matrix\.md:10.*engine-matrix\.md:11.*feature-matrix\.md:20", + ): + view.known_task_ids(ROOT) + + +class LiveClaimValidation(unittest.TestCase): + def test_one_well_formed_open_claim_passes(self) -> None: + self.assertEqual( + view.validate_live_claims([pr(7, "ENG-FOO")], KNOWN, EXPECTED), [] + ) + def test_nonempty_claim_input_requires_expected_repository_authority(self) -> None: + errors = view.validate_live_claims([pr(7, "ENG-FOO")], KNOWN) + self.assertEqual(errors, ["nonempty PR input lacks expected repository identity"]) + + def test_every_claim_must_match_the_expected_repository(self) -> None: + candidate = pr(7, "ENG-FOO") + candidate["headRepository"] = {"nameWithOwner": "attacker/repo"} + errors = view.validate_live_claims([candidate], KNOWN, EXPECTED) + self.assertEqual( + errors, + ["PR #7 repository 'attacker/repo' does not match expected 'owner/repo'"], + ) + unrelated = pr(8, "ENG-FOO", head="feature/not-a-claim") + unrelated["headRepository"] = {"nameWithOwner": "attacker/repo"} + self.assertEqual( + view.validate_live_claims([unrelated], KNOWN, EXPECTED), + ["PR #8 repository 'attacker/repo' does not match expected 'owner/repo'"], + ) + + def test_duplicate_task_claims_are_rejected_deterministically(self) -> None: + errors = view.validate_live_claims( + [pr(9, "ENG-FOO"), pr(7, "ENG-FOO")], KNOWN, EXPECTED + ) + self.assertEqual( + errors, + ["task ENG-FOO has duplicate open claims: PR #7, PR #9"], + ) -def block(stamp: str, rows: str = "| `ENG-FOO` | #7 | draft | someone | 2026-08-04 |") -> str: - return "\n".join([ - view.BEGIN, - f"", - "", - "| Row | PR | State | Agent | Updated |", - "|---|---|---|---|---|", - rows, - "", - view.END, - ]) + def test_unknown_task_is_rejected(self) -> None: + errors = view.validate_live_claims([pr(7, "UNKNOWN")], KNOWN, EXPECTED) + self.assertEqual(errors, ["PR #7 claims unknown task UNKNOWN"]) + def test_noncanonical_and_closed_row_heads_are_rejected(self) -> None: + cases = ( + (pr(7, "ENG-FOO", head="row/KV-BAR"), {"task_id": "ENG-FOO"}, "wrong head"), + (pr(7, "ENG-FOO", state="CLOSED"), None, "is CLOSED"), + (pr(7, "ENG-FOO", state="MERGED"), None, "is MERGED"), + ) + for candidate, expected, message in cases: + with self.subTest(message=message): + authority = {**EXPECTED, **(expected or {})} + errors = view.validate_live_claims([candidate], KNOWN, authority) + self.assertTrue(any(message in error for error in errors), errors) + + def test_expected_claim_must_exist_and_match_number_and_head(self) -> None: + expected = {"task_id": "ENG-FOO", "head": "row/ENG-FOO", "number": 8} + errors = view.validate_live_claims( + [pr(7, "ENG-FOO")], KNOWN, {**EXPECTED, **expected} + ) + self.assertTrue(any("expected PR #8" in error for error in errors), errors) -class ClaimView(unittest.TestCase): - def test_valid_block_passes(self) -> None: - today = time.strftime("%Y-%m-%d") - self.assertEqual(view.check_errors(block(today), KNOWN), []) + missing = view.validate_live_claims( + [], KNOWN, {**EXPECTED, "task_id": "ENG-FOO"} + ) + self.assertEqual(missing, ["expected live claim for task ENG-FOO is missing"]) + + def test_malformed_or_ambiguous_pr_identity_is_rejected(self) -> None: + malformed = pr(7, "ENG-FOO") + malformed["state"] = "open" + duplicate_number = [pr(7, "ENG-FOO"), pr(7, "KV-BAR")] + errors = view.validate_live_claims( + [malformed, *duplicate_number], KNOWN, EXPECTED + ) + self.assertTrue(any("malformed state" in error for error in errors), errors) + self.assertTrue(any("PR number #7 is ambiguous" in error for error in errors), errors) + + def test_row_claim_requires_unambiguous_repository_identity(self) -> None: + candidate = pr(7, "ENG-FOO") + candidate["headRepository"] = None + errors = view.validate_live_claims([candidate], KNOWN, EXPECTED) + self.assertEqual( + errors, + ["PR #7 has malformed head repository identity"], + ) - def test_missing_block_is_rejected(self) -> None: - errors = view.check_errors("# coordination\n\nno block here", KNOWN) - self.assertTrue(any("missing the claim view" in e for e in errors)) - def test_missing_stamp_is_rejected(self) -> None: - text = block(time.strftime("%Y-%m-%d")).replace( - f"", "" +class ClaimViewCli(unittest.TestCase): + def test_local_check_rejects_a_committed_snapshot(self) -> None: + text = "# Coordination\n\n\nold\n\n" + self.assertEqual( + view.local_errors(text), + ["committed claim-view snapshots are forbidden; live PR state is authoritative"], ) - self.assertTrue(any("no \n\n" + f"Gate hardware: {hardware}.\n", + encoding="utf-8", + ) + + def commit(self, message: str = "fixture") -> None: + subprocess.run(["git", "add", "-A"], cwd=self.root, check=True) + subprocess.run(["git", "commit", "-qm", message], cwd=self.root, check=True) + + def close(self) -> None: + self.temp.cleanup() + + +class ReadinessProof(unittest.TestCase): + def setUp(self) -> None: + self.repo = RepositoryFixture() + self.row = Row(self.repo.root) + self.states = {"ENG-FOO": "READY", "KV-BAR": "DONE"} + + def tearDown(self) -> None: + self.repo.close() + + def evaluate(self, row: Row | None = None, **kwargs) -> list[str]: + return ready.evaluate( + row or self.row, + live_claims=kwargs.pop("live_claims", set()), + known_tasks=kwargs.pop("known_tasks", set(self.states)), + task_states=kwargs.pop("task_states", self.states), + root=self.repo.root, + base="HEAD", + **kwargs, + ) + + def test_complete_proof_is_pickable(self) -> None: + self.assertEqual(self.evaluate(), []) + + def test_each_lifecycle_state_is_enforced(self) -> None: + for state in ("INVENTORIED", "SPIKE", "ACTIVE", "GATING", "BLOCKED", "DONE"): + with self.subTest(state=state): + errors = self.evaluate(Row(self.repo.root, state=state)) + self.assertTrue(any("lifecycle" in error for error in errors), errors) + + def test_unknown_task_uses_the_same_shared_adapter_boundary(self) -> None: + errors = self.evaluate(known_tasks={"KV-BAR"}) + self.assertIn("unknown task ID", errors) + + def test_spec_must_be_committed_and_reachable_from_base(self) -> None: + path = self.repo.root / ".agents/specs/foo.md" + path.write_text("# rewritten worktree file without a contract\n", encoding="utf-8") + self.assertEqual(self.evaluate(), []) + subprocess.run(["git", "rm", "--cached", "-q", ".agents/specs/foo.md"], cwd=self.repo.root, check=True) + subprocess.run(["git", "commit", "-qm", "remove spec from base"], cwd=self.repo.root, check=True) + errors = self.evaluate() + self.assertTrue(any("base-reachable committed spec" in error for error in errors), errors) + + def test_spec_must_be_a_regular_blob_and_base_must_resolve_to_a_commit(self) -> None: + spec = self.repo.root / ".agents/specs/foo.md" + spec.unlink() + os.symlink("../../tests/gate.py", spec) + self.repo.commit("symlink spec") + self.assertIn("no base-reachable committed spec", self.evaluate()) + errors = ready.evaluate( + self.row, + live_claims=set(), + known_tasks=set(self.states), + task_states=self.states, + root=self.repo.root, + base="refs/heads/does-not-exist", + ) + self.assertTrue(any("base commit" in error for error in errors), errors) + + def test_gate_needs_executable_command_and_failing_mutation(self) -> None: + cases = ( + ({"gate": "python3 tests/gate.py", "mutation": ["python3", "tests/mutation.py"]}, "structured"), + ({"gate": ["python3", "-c", "pass"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "-cpass"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "--eval=pass"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["/usr/bin/python3", "tests/gate.py"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "../gate.py"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "--config=../gate.py"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "--config=/etc/passwd"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["MALICE=1", "tests/gate.py"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["python3", "tests/gate.py;echo"], "mutation": ["python3", "tests/mutation.py"]}, "unsafe gate"), + ({"gate": ["missing-program", "tests/gate.py"], "mutation": ["python3", "tests/mutation.py"]}, "gate executable"), + ) + spec = self.repo.root / ".agents/specs/foo.md" + for contract, expected in cases: + with self.subTest(contract=contract): + spec.write_text( + "# Gates\n\n\n\nGate hardware: CPU.\n", + encoding="utf-8", + ) + self.repo.commit("bad contract") + errors = self.evaluate() + self.assertTrue(any(expected in error for error in errors), errors) + + def test_gate_and_mutation_are_executed_from_exact_base_not_worktree(self) -> None: + gate = self.repo.root / "tests/gate.py" + mutation = self.repo.root / "tests/mutation.py" + gate.write_text("raise SystemExit(7)\n", encoding="utf-8") + mutation.write_text("raise SystemExit(0)\n", encoding="utf-8") + self.assertEqual(self.evaluate(), []) + + self.repo.commit("make exact base fail readiness") + errors = self.evaluate() + self.assertTrue(any("gate exited 7" in error for error in errors), errors) + self.assertTrue(any("mutation unexpectedly exited 0" in error for error in errors), errors) + + def test_gate_runs_with_sanitized_environment_and_timeout(self) -> None: + gate = self.repo.root / "tests/gate.py" + gate.write_text( + "import os, time\n" + "assert 'HOST_SECRET_FOR_TEST' not in os.environ\n" + "assert set(os.environ) <= {'HOME','LANG','LC_ALL','PATH','PYTHONHASHSEED','TMPDIR'}\n" + "time.sleep(2)\n", + encoding="utf-8", + ) + self.repo.commit("slow sanitized gate") + with mock.patch.dict(os.environ, {"HOST_SECRET_FOR_TEST": "must-not-leak"}), mock.patch.object( + ready, "RUN_TIMEOUT_SECONDS", 0.05 + ): + errors = self.evaluate() + self.assertTrue(any("gate could not execute" in error and "timed out" in error for error in errors), errors) + + def test_missing_or_nonregular_base_argv_paths_are_rejected(self) -> None: + for argument in ("tests/missing.py", "tests"): + with self.subTest(argument=argument): + self.repo.write_spec(gate=["python3", argument]) + self.repo.commit("bad argv path") + self.assertTrue( + any("regular base file" in error for error in self.evaluate()) + ) + + def test_hardware_must_say_cpu_or_name_exact_hardware(self) -> None: + self.repo.write_spec(hardware="GPU") + self.repo.commit("generic hardware") + self.assertTrue(any("exact gate hardware" in error for error in self.evaluate())) + self.repo.write_spec(hardware="NVIDIA GB10 sm_121") + self.repo.commit("exact hardware") + self.assertEqual(self.evaluate(), []) + + def test_dependencies_must_parse_and_be_done(self) -> None: + self.assertEqual(self.evaluate(Row(self.repo.root, dependencies="`KV-BAR`")), []) + pending_states = {**self.states, "KV-BAR": "ACTIVE"} + errors = self.evaluate(Row(self.repo.root, dependencies="`KV-BAR`"), task_states=pending_states) + self.assertIn("dependency KV-BAR is ACTIVE, not satisfied", errors) + errors = self.evaluate(Row(self.repo.root, dependencies="after somebody reviews it")) + self.assertIn("dependencies are not fully parsed", errors) + + def test_live_claim_excludes_task(self) -> None: + self.assertIn("already claimed by an open PR", self.evaluate(live_claims={"ENG-FOO"})) + + +class QueueAndCli(unittest.TestCase): + def test_queue_consumes_claim_views_shared_known_task_adapter(self) -> None: + row = Row(ROOT) + with mock.patch.object( + ready.claim_view, "canonical_task_rows", return_value=[row] + ) as rows: + pickable, reasons = ready.queue([], root=ROOT, base="HEAD") + rows.assert_called_once_with(ROOT) + self.assertNotIn("unknown task ID", reasons) + + def test_print_queue_rejects_remote_unavailability(self) -> None: + with mock.patch.object(ready.claim_view, "fetch_prs", side_effect=ready.claim_view.RemoteUnverified("offline")): + self.assertEqual(ready.main(["--check-live"]), ready.claim_view.REMOTE_UNVERIFIED_EXIT) + + def test_bare_check_is_explicitly_local_only_and_never_advertises_queue(self) -> None: + output = io.StringIO() + with mock.patch.object( + ready.claim_view, "fetch_prs", side_effect=AssertionError("network") + ), mock.patch.object(ready, "queue", side_effect=AssertionError("queue")), redirect_stdout(output): + self.assertEqual(ready.main(["--check"]), 0) + self.assertIn("LOCAL_ONLY", output.getvalue()) + self.assertIn("no helper queue was advertised", output.getvalue()) + + def test_fixture_claims_are_consumed_without_network(self) -> None: + with tempfile.TemporaryDirectory() as directory: + fixture = Path(directory) / "prs.json" + fixture.write_text( + json.dumps( + {"expected": {"repository": "owner/repo", "base": "HEAD"}, "prs": []} + ), + encoding="utf-8", + ) + with mock.patch.object(ready, "queue", return_value=([], {})) as queue: + self.assertEqual( + ready.main(["--check-live", "--pr-json", str(fixture)]), 0 + ) + queue.assert_called_once() + + def test_deprecated_check_rejects_fixture_and_live_options(self) -> None: + for args in ( + ["--check", "--pr-json", "claims.json"], + ["--check", "--check-live"], + ["--check", "--base", "HEAD"], + ): + with self.subTest(args=args), self.assertRaises(SystemExit): + ready.main(args) + + def test_explicit_local_check_is_available_and_never_fetches(self) -> None: + with mock.patch.object( + ready.claim_view, "fetch_prs", side_effect=AssertionError("network") + ): + self.assertEqual(ready.main(["--check-local"]), 0) + + def test_duplicate_row_ids_are_rejected_before_state_indexing(self) -> None: + with mock.patch.object( + ready.claim_view, + "canonical_task_rows", + side_effect=ValueError("duplicate task ID ENG-FOO at a:1 and b:2"), + ): + with self.assertRaisesRegex(ValueError, "duplicate task ID ENG-FOO"): + ready._rows(ROOT) + + +if __name__ == "__main__": + unittest.main() From bfcf0c3bcc03d45881b1ea38aab02e334cd93826 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 11:19:27 +0000 Subject: [PATCH 07/12] policy: enforce trailers waivers and PR budgets Require POL-COMMIT-TRAILERS, POL-AI-ATTRIBUTION, POL-WAIVER-EXACT, POL-PR-REQUIRED, POL-PATH-CLASSIFICATION, POL-PR-SIZE and checker-change semantic evidence at their executable boundaries. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- scripts/check-commit-trailers.py | 329 +++++++++++ scripts/check-doc-checkpoint.py | 4 + scripts/check-pr-size.py | 603 ++++++++++++++++++-- scripts/check-role-discipline.py | 99 +++- tests/scripts/test_check_commit_trailers.py | 275 +++++++++ tests/scripts/test_check_pr_size.py | 285 +++++++++ tests/scripts/test_doc_checkpoint.py | 15 + tests/scripts/test_policy_waivers.py | 81 +++ 8 files changed, 1628 insertions(+), 63 deletions(-) create mode 100755 scripts/check-commit-trailers.py create mode 100755 tests/scripts/test_check_commit_trailers.py create mode 100755 tests/scripts/test_check_pr_size.py create mode 100755 tests/scripts/test_policy_waivers.py diff --git a/scripts/check-commit-trailers.py b/scripts/check-commit-trailers.py new file mode 100755 index 000000000..66b53a73a --- /dev/null +++ b/scripts/check-commit-trailers.py @@ -0,0 +1,329 @@ +#!/usr/bin/env python3 +"""Validate commit messages with Git's own trailer parser.""" + +from __future__ import annotations + +import argparse +import re +import subprocess +import sys +from pathlib import Path +from typing import Iterable + +try: + from scripts.policy_contract import PolicyRule, Waiver, load_policy, load_waivers +except ModuleNotFoundError: # direct execution from outside the repository root + from policy_contract import PolicyRule, Waiver, load_policy, load_waivers + + +ROOT = Path(__file__).resolve().parents[1] +RAW_PROTOCOL_MARKER = "FOLLOWING_AGENTS_PROTOCOL" +PROTOCOL_RULE = "POL-COMMIT-TRAILERS" +ATTRIBUTION_RULE = "POL-AI-ATTRIBUTION" +ASSISTED_BY = re.compile( + r"[A-Za-z0-9][A-Za-z0-9_.-]*:[A-Za-z0-9][A-Za-z0-9_.+-]*" + r"(?: \[[A-Za-z0-9][A-Za-z0-9_. +:/-]*\])+\Z" +) +# Closed, reviewable vocabulary for obvious agent/vendor/model/tool authorship. +# Boundaries prevent human names such as "Alice" from matching the token "ai". +AI_AUTHORSHIP_TOKENS = ( + "ai", + "agent", + "anthropic", + "bot", + "chatgpt", + "claude", + "claudecode", + "codex", + "copilot", + "gemini", + "gpt", + "llm", + "openai", +) +AI_IDENTITY = re.compile( + r"(? str: + result = subprocess.run( + ["git", "-C", str(repo), *args], + input=input_text, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + if result.returncode != 0: + detail = result.stderr.strip() or "Git command failed" + raise ValueError(detail) + return result.stdout.strip() + + +def parsed_trailers(message: str) -> str: + """Return exactly what ``git interpret-trailers --parse`` returns.""" + + result = subprocess.run( + ["git", "interpret-trailers", "--parse"], + input=message, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True, + ) + return result.stdout + + +def _paragraphs(message: str) -> list[str]: + normalized = message.replace("\r\n", "\n").replace("\r", "\n").strip("\n") + return re.split(r"\n[ \t]*\n+", normalized) if normalized else [] + + +def _trailer_map(message: str) -> dict[str, list[tuple[str, str]]]: + trailers: dict[str, list[tuple[str, str]]] = {} + for line in parsed_trailers(message).splitlines(): + if ": " not in line: + continue + key, value = line.split(": ", 1) + trailers.setdefault(key.casefold(), []).append((key, value)) + return trailers + + +def _strict_errors(message: str) -> list[str]: + errors: list[str] = [] + paragraphs = _paragraphs(message) + marker_indexes = [ + index for index, paragraph in enumerate(paragraphs) + if paragraph == RAW_PROTOCOL_MARKER + ] + if len(marker_indexes) != 1 or marker_indexes[0] == len(paragraphs) - 1: + errors.append( + f"[{PROTOCOL_RULE}] {RAW_PROTOCOL_MARKER} must appear exactly once " + "as a separate paragraph before the trailer paragraph" + ) + + trailers = _trailer_map(message) + protocol = trailers.get("following-agents-protocol", []) + if len(protocol) != 1: + errors.append( + f"[{PROTOCOL_RULE}] Following-Agents-Protocol must appear exactly once" + ) + elif protocol[0][1] != "true": + errors.append( + f"[{PROTOCOL_RULE}] Following-Agents-Protocol must be exactly true" + ) + + declarations = trailers.get("ai-assisted", []) + assisted = trailers.get("assisted-by", []) + if len(declarations) != 1: + errors.append(f"[{ATTRIBUTION_RULE}] AI-Assisted must appear exactly once") + else: + declaration = declarations[0][1] + if declaration not in {"true", "false"}: + errors.append(f"[{ATTRIBUTION_RULE}] AI-Assisted must be true or false") + elif declaration == "true" and not assisted: + errors.append( + f"[{ATTRIBUTION_RULE}] AI-Assisted true requires Assisted-by" + ) + elif declaration == "false" and assisted: + errors.append( + f"[{ATTRIBUTION_RULE}] AI-Assisted false must omit Assisted-by" + ) + + for _, value in assisted: + if ASSISTED_BY.fullmatch(value) is None: + errors.append(f"[{ATTRIBUTION_RULE}] malformed Assisted-by value {value!r}") + + assisted_identities: set[str] = set() + for _, value in assisted: + if ASSISTED_BY.fullmatch(value) is None: + continue + identity, remainder = value.split(":", 1) + model = remainder.split(" ", 1)[0] + assisted_identities.update((identity.casefold(), model.casefold())) + assisted_identities.update( + tool.casefold() for tool in re.findall(r"\[([^]]+)\]", value) + ) + for key in ("signed-off-by", "co-authored-by"): + for original, value in trailers.get(key, []): + folded_value = value.casefold() + if AI_IDENTITY.search(value) or any( + identity in folded_value for identity in assisted_identities + ): + errors.append( + f"[{ATTRIBUTION_RULE}] AI authorship trailer {original} is forbidden" + ) + return errors + + +def validate_waiver_targets(repo: Path, waivers: Iterable[Waiver]) -> None: + """Reject path waivers whose supposedly exact target is a directory.""" + + for waiver in waivers: + if not waiver.scope.startswith("path:"): + continue + target = waiver.scope.removeprefix("path:") + candidate = repo / target + if candidate.is_dir(): + raise ValueError( + f"waiver {waiver.waiver_id} path target {target!r} is a directory" + ) + + +def validate_commit_message(message: str, *, strict: bool) -> list[str]: + """Return trailer-contract errors for one complete commit message.""" + + if not strict: + marker_count = sum( + paragraph == RAW_PROTOCOL_MARKER for paragraph in _paragraphs(message) + ) + if marker_count == 1: + return [] + return _strict_errors(message) + + +def exact_waiver( + waivers: Iterable[Waiver], rule_id: str, scope: str +) -> Waiver | None: + """Select one exact, already-validated waiver or fail on ambiguity.""" + + matches = [ + waiver + for waiver in waivers + if waiver.rule_id == rule_id and waiver.scope == scope + ] + if len(matches) > 1: + raise ValueError(f"duplicate applicable waivers for {rule_id} {scope}") + return matches[0] if matches else None + + +def _resolve_commit(repo: Path, revision: str) -> str: + if not revision or "\x00" in revision or "\n" in revision: + raise ValueError(f"invalid revision {revision!r}") + if not revision.startswith("refs/"): + candidates = ( + f"refs/heads/{revision}", + f"refs/tags/{revision}", + f"refs/remotes/{revision}", + ) + matches = 0 + for candidate in candidates: + result = subprocess.run( + ["git", "-C", str(repo), "show-ref", "--verify", "--quiet", candidate], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if result.returncode == 0: + matches += 1 + elif result.returncode not in {1}: + raise ValueError(f"could not resolve revision {revision!r}") + if matches > 1: + raise ValueError(f"ambiguous revision {revision!r}") + resolved = _git( + repo, "rev-parse", "--verify", "--end-of-options", f"{revision}^{{commit}}" + ).splitlines() + if len(resolved) != 1 or re.fullmatch(r"[0-9a-f]{40}", resolved[0]) is None: + raise ValueError(f"revision {revision!r} did not resolve to one commit") + return resolved[0] + + +def _is_ancestor(repo: Path, older: str, newer: str) -> bool: + result = subprocess.run( + ["git", "-C", str(repo), "merge-base", "--is-ancestor", older, newer], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if result.returncode not in {0, 1}: + raise ValueError("could not establish commit ancestry") + return result.returncode == 0 + + +def _error_rule(error: str) -> str: + match = re.match(r"\[([^]]+)\]", error) + return match.group(1) if match else PROTOCOL_RULE + + +def validate_range( + repo: Path, + base: str, + head: str, + *, + cutover: str | None, + rules: dict[str, PolicyRule] | None, + waivers: Iterable[Waiver], +) -> list[str]: + """Validate an exact first-parent-independent ``BASE..HEAD`` commit set.""" + + base_oid = _resolve_commit(repo, base) + head_oid = _resolve_commit(repo, head) + if not _is_ancestor(repo, base_oid, head_oid): + raise ValueError("range base must be an ancestor of range head") + cutover_oid = _resolve_commit(repo, cutover) if cutover is not None else None + if cutover_oid is not None and not _is_ancestor(repo, cutover_oid, head_oid): + raise ValueError("cutover must be reachable from range head") + + commits_text = _git(repo, "rev-list", "--reverse", f"{base_oid}..{head_oid}") + failures: list[str] = [] + for commit in (line for line in commits_text.splitlines() if line): + if cutover_oid is None: + strict = True + elif _is_ancestor(repo, cutover_oid, commit): + strict = True + elif _is_ancestor(repo, commit, cutover_oid): + strict = False + else: + raise ValueError(f"commit {commit} is incomparable with cutover") + + message = _git(repo, "show", "-s", "--format=%B", commit) + "\n" + for error in validate_commit_message(message, strict=strict): + rule_id = _error_rule(error) + if rules is not None and rule_id not in rules: + raise ValueError(f"policy registry is missing {rule_id}") + if exact_waiver(waivers, rule_id, f"commit:{commit}") is not None: + continue + failures.append(f"{commit[:12]}: {error}") + return failures + + +def _range(value: str) -> tuple[str, str]: + if value.count("..") != 1 or "..." in value: + raise argparse.ArgumentTypeError("range must be exactly BASE..HEAD") + base, head = value.split("..", 1) + if not base or not head: + raise argparse.ArgumentTypeError("range must be exactly BASE..HEAD") + return base, head + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--range", dest="revision_range", type=_range, required=True) + parser.add_argument("--cutover") + args = parser.parse_args() + try: + rules = load_policy(ROOT) + waivers = load_waivers(ROOT, rules) + validate_waiver_targets(ROOT, waivers) + failures = validate_range( + ROOT, + *args.revision_range, + cutover=args.cutover, + rules=rules, + waivers=waivers, + ) + except (OSError, subprocess.SubprocessError, ValueError) as exc: + print(f"commit trailer check FAILED: {exc}", file=sys.stderr) + return 1 + if failures: + print("commit trailer check FAILED:", file=sys.stderr) + for failure in failures: + print(f" - {failure}", file=sys.stderr) + return 1 + print("OK: commit trailer contract") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/check-doc-checkpoint.py b/scripts/check-doc-checkpoint.py index 6ed262558..4100fecfc 100755 --- a/scripts/check-doc-checkpoint.py +++ b/scripts/check-doc-checkpoint.py @@ -351,6 +351,7 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: ".agents/waivers.csv", "scripts/agent-role.py", "scripts/check-doc-checkpoint.py", + "scripts/check-commit-trailers.py", "scripts/check-gate-commands.py", "scripts/policy_contract.py", "scripts/check-policy.py", @@ -366,6 +367,9 @@ def public_rule_bindings(root: Path = ROOT) -> dict[str, PublicRuleBinding]: "tests/scripts/test_policy_contract.py", "tests/scripts/test_check_prompt_contract.py", "tests/scripts/test_doc_checkpoint.py", + "tests/scripts/test_check_commit_trailers.py", + "tests/scripts/test_policy_waivers.py", + "tests/scripts/test_check_pr_size.py", "tests/scripts/test_check_protocol_consistency.py", "docs/superpowers/specs/2026-08-07-internal-policy-optimization-design.md", ".agents/completed/pre-cutover-claim-protocol.md", diff --git a/scripts/check-pr-size.py b/scripts/check-pr-size.py index fd176f273..166c70675 100755 --- a/scripts/check-pr-size.py +++ b/scripts/check-pr-size.py @@ -1,73 +1,586 @@ #!/usr/bin/env python3 -"""Keep helper PRs cheap to review. (W4) - -Parallel helpers plus a serial operator means review is the bottleneck, and the -mitigation is that each PR is small enough to review quickly: one row, bounded -diff. The cap applies to `row/*` PRs, which are the protocol's PRs. Other -branches are reported, not failed, so pre-existing work is not retroactively -punished for a rule it was not written under. - -Record and protocol paths are exempt: an honest state entry or a matrix update -can be long, and shrinking evidence to satisfy a line cap would be the wrong -incentive entirely. -""" +"""Enforce finite review budgets for explicit repository path classes.""" from __future__ import annotations import argparse +import importlib.util +import os +import re +import shutil import subprocess import sys -from pathlib import Path +import tempfile +from dataclasses import dataclass +from pathlib import Path, PurePosixPath + +try: + from scripts.policy_contract import PolicyRule, load_policy +except ModuleNotFoundError: + from policy_contract import PolicyRule, load_policy + ROOT = Path(__file__).resolve().parents[1] -MAX_FEATURE_LINES = 900 -EXEMPT_PREFIXES = (".agents/", "docs/", "tests/scripts/", "scripts/", ".github/") +PATH_CLASSES = frozenset( + { + "product", + "governance_checker", + "governance_test", + "governance_support", + "policy", + "procedure", + "append_only_record", + "project_record", + "public_document", + "design", + "ci", + "configuration", + "asset", + "evidence", + "vendored_dependency", + } +) +PATH_CLASS_BUDGETS = { + "product": 900, + "governance_checker": 6000, + "governance_test": 3000, + "governance_support": 1800, + "policy": 1200, + "procedure": 3000, + "append_only_record": 5000, + "project_record": 4000, + "public_document": 2500, + "design": 1500, + "ci": 800, + "configuration": 800, + "asset": 3000, + "evidence": 8000, + "vendored_dependency": 8000, +} + +POLICY_FILES = frozenset( + { + ".agents/policy.csv", + ".agents/waivers.csv", + ".agents/governance-tasks.csv", + ".agents/policy-cutover", + } +) +APPEND_ONLY_FILES = frozenset( + { + ".agents/state.md", + ".agents/benchmark-record.md", + ".agents/parity-ledger.md", + } +) +PROJECT_RECORD_FILES = frozenset( + { + ".agents/NOW.md", + ".agents/coordination.md", + ".agents/roadmap_v1.md", + ".agents/porting-inventory.md", + ".agents/engine-matrix.md", + ".agents/feature-matrix.md", + ".agents/model-matrix.md", + ".agents/quantization-matrix.md", + ".agents/kernel-matrix.md", + ".agents/backend-matrix.md", + ".agents/sglang-matrix.md", + } +) +PROCEDURE_FILES = frozenset( + { + "AGENTS.md", + ".agents/workflow.md", + ".agents/verification.md", + ".agents/porting.md", + ".agents/directives.md", + ".agents/ai-coding-assistants.md", + ".agents/benchmark-protocol.md", + ".agents/discipline.md", + ".agents/gates.md", + ".agents/prompts/implementer.md", + ".agents/prompts/operator.md", + ".agents/prompts/reviewer.md", + ".agents/backends.md", + ".agents/developer-preferences.example.md", + ".agents/environment.md", + ".agents/mission.md", + ".agents/parity-lever-protocol.md", + ".agents/test-porting.md", + ".agents/upstream-sync.md", + ".agents/vllm-v1-v2.md", + } +) +GOVERNANCE_SUPPORT_FILES = frozenset( + { + "scripts/policy_contract.py", + "scripts/agent-role.py", + "scripts/claim-view.py", + "scripts/ready-for-helper.py", + "scripts/agent-preflight.sh", + } +) +PUBLIC_DOCUMENT_FILES = frozenset( + { + "README.md", + "CONTRIBUTING.md", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/FEATURES.md", + "docs/USAGE.md", + } +) +CHECKER = re.compile(r"scripts/check-[a-z0-9]+(?:-[a-z0-9]+)*\.(?:py|sh)\Z") +CHECKER_TEST = re.compile(r"tests/scripts/test_[a-z0-9]+(?:_[a-z0-9]+)*\.py\Z") +CI = re.compile(r"\.github/(?:workflows/[A-Za-z0-9_.-]+\.ya?ml|dependabot\.yml|pull_request_template\.md)\Z") +DESIGN = re.compile(r"docs/superpowers/specs/[0-9]{4}-[0-9]{2}-[0-9]{2}-[a-z0-9-]+\.md\Z") +DOC = re.compile(r"docs/[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*\.(?:md|png|svg|json)\Z") +SPEC = re.compile(r"\.agents/specs/[A-Za-z0-9_.-]+\.md\Z") +SPEC_EVIDENCE = re.compile(r"\.agents/specs/[A-Za-z0-9_.-]+\.(?:patch|json|log)\Z") +COMPLETED = re.compile(r"\.agents/completed/[A-Za-z0-9_.-]+\.md\Z") +SYNC_RECORD = re.compile(r"\.agents/sync/[A-Za-z0-9_.-]+\.md\Z") +HOOK = re.compile(r"\.githooks/(?:README\.md|[A-Za-z0-9_.-]+)\Z") +BENCH_EVIDENCE = re.compile(r"(?:benchmarks/(?:demo|media)|docs/bench-evidence)/[A-Za-z0-9_.-]+\.(?:json|png|gif|mp4|log)\Z") +ASSET = re.compile(r"assets/[A-Za-z0-9_.-]+\.(?:png|svg)\Z") + +CHECKER_EVIDENCE_OVERRIDES = { + "scripts/check-policy.py": "tests/scripts/test_policy_contract.py", + "scripts/check-role-discipline.py": "tests/scripts/test_check_pr_size.py", + "scripts/check-doc-checkpoint.py": "tests/scripts/test_doc_checkpoint.py", + "scripts/check-protocol-consistency.py": "tests/scripts/test_check_protocol_consistency.py", +} + +# New entrypoints cannot appear in their own historical policy enforcement +# cells before they exist. This closed creation map binds their affected rules +# until the policy cutover can name the entrypoint directly. +CREATED_CHECKER_RULES = { + "scripts/check-commit-trailers.py": ( + "POL-COMMIT-TRAILERS", + "POL-AI-ATTRIBUTION", + "POL-WAIVER-EXACT", + ), +} +CREATION_MUTATIONS = { + "scripts/check-commit-trailers.py": ( + b"#!/usr/bin/env python3\n" + b'"""Deliberately disabled creation-contract mutation."""\n' + b"def parsed_trailers(message): return ''\n" + b"def validate_commit_message(message, *, strict): return []\n" + b"def validate_range(*args, **kwargs): return []\n" + b"def exact_waiver(*args, **kwargs): return None\n" + b"def validate_waiver_targets(*args, **kwargs): return None\n" + ), +} +EVIDENCE_TIMEOUT_SECONDS = 120 +TEST_COUNT = re.compile(r"Ran ([0-9]+) tests? in ") + + +@dataclass(frozen=True) +class ChangedPath: + path: str + added: int | None + removed: int | None + + @property + def lines(self) -> int | None: + if self.added is None or self.removed is None: + return None + return self.added + self.removed + + +@dataclass(frozen=True) +class EvidenceResult: + checker: str + test_module: str + rule_ids: tuple[str, ...] + head_tests: int + head_passed: bool + base_tests: int + base_failed: bool + detail: str = "" + + +def _canonical_path(path: str) -> bool: + candidate = PurePosixPath(path) + return ( + bool(path) + and not candidate.is_absolute() + and "\\" not in path + and "//" not in path + and candidate.as_posix() == path + and all(part not in {"", ".", ".."} for part in candidate.parts) + ) + + +def classify_path(path: str) -> str: + """Return one closed path class or reject an unknown/noncanonical path.""" + + if not _canonical_path(path): + raise ValueError(f"noncanonical repository path {path!r}") + if path in POLICY_FILES: + return "policy" + if path in APPEND_ONLY_FILES: + return "append_only_record" + if path in PROJECT_RECORD_FILES: + return "project_record" + if path == ".agents/upstream-inventory.json": + return "project_record" + if path in PROCEDURE_FILES or SPEC.fullmatch(path) or COMPLETED.fullmatch(path): + return "procedure" + if SPEC_EVIDENCE.fullmatch(path) or SYNC_RECORD.fullmatch(path) or BENCH_EVIDENCE.fullmatch(path): + return "evidence" + if path in GOVERNANCE_SUPPORT_FILES: + return "governance_support" + if DESIGN.fullmatch(path): + return "design" + if path in PUBLIC_DOCUMENT_FILES or DOC.fullmatch(path): + return "public_document" + if CHECKER.fullmatch(path): + return "governance_checker" + if CHECKER_TEST.fullmatch(path): + return "governance_test" + if CI.fullmatch(path): + return "ci" + if HOOK.fullmatch(path): + return "ci" + if ASSET.fullmatch(path): + return "asset" + if path.startswith("third_party/"): + return "vendored_dependency" + if path in { + "CMakeLists.txt", ".env.example", ".gitignore", ".dockerignore", + ".clang-format", ".gitattributes", "flake.lock", "flake.nix", + "LICENSE", "NOTICE", + } or re.fullmatch(r"docker/Dockerfile\.[A-Za-z0-9_.-]+", path): + return "configuration" + if path.startswith(("src/", "include/", "examples/", "tools/", "cmake/", "tests/", "scripts/", "benchmarks/", "triton_kernels/")): + return "product" + raise ValueError(f"unclassified repository path {path!r}") + + +def recognized_evidence(checker_path: str) -> str: + override = CHECKER_EVIDENCE_OVERRIDES.get(checker_path) + if override is not None: + return override + name = Path(checker_path).stem.removeprefix("check-").replace("-", "_") + return f"tests/scripts/test_check_{name}.py" + +def requires_reviewed_pr(path: str) -> bool: + """Return whether a canonical governed path requires reviewed PR arrival.""" -def git(*args: str) -> str: - return subprocess.check_output(["git", *args], cwd=ROOT, text=True).strip() + return classify_path(path) in PATH_CLASSES -def counted_lines(base: str, head: str) -> tuple[int, list[str]]: - out = git("diff", "--numstat", f"{base}...{head}") - total, files = 0, [] - for line in out.splitlines(): - parts = line.split("\t") - if len(parts) != 3: +def parse_numstat(output: str) -> list[ChangedPath]: + changes: list[ChangedPath] = [] + seen: set[str] = set() + for number, line in enumerate(output.splitlines(), start=1): + fields = line.split("\t") + if len(fields) != 3: + raise ValueError(f"numstat line {number} must have exactly three fields") + added_raw, removed_raw, path = fields + if path in seen: + raise ValueError(f"duplicate numstat path {path!r}") + seen.add(path) + classify_path(path) + if added_raw == removed_raw == "-": + changes.append(ChangedPath(path, None, None)) continue - added, removed, path = parts - if path.startswith(EXEMPT_PREFIXES): + if not added_raw.isascii() or not added_raw.isdecimal() or not removed_raw.isascii() or not removed_raw.isdecimal(): + raise ValueError(f"numstat line {number} has invalid line counts") + changes.append(ChangedPath(path, int(added_raw), int(removed_raw))) + return changes + + +def change_errors( + changes: list[ChangedPath], + *, + evidence_results: dict[str, EvidenceResult] | None = None, +) -> list[str]: + errors: list[str] = [] + totals = {path_class: 0 for path_class in PATH_CLASSES} + changed_paths = {change.path: change for change in changes} + for change in changes: + try: + path_class = classify_path(change.path) + except ValueError as exc: + errors.append(str(exc)) continue - if added == "-" or removed == "-": + if change.lines is None: + errors.append(f"binary change {change.path!r} has no reviewable line budget") + continue + totals[path_class] += change.lines + if path_class == "governance_checker": + evidence = recognized_evidence(change.path) + evidence_change = changed_paths.get(evidence) + if evidence_change is None or evidence_change.lines is None or evidence_change.lines <= 0: + errors.append( + f"checker change {change.path!r} requires semantic mutation evidence in {evidence}" + ) + elif evidence_results is not None: + proof = evidence_results.get(change.path) + if proof is None: + errors.append( + f"checker change {change.path!r} has no executable mutation result" + ) + elif proof.checker != change.path: + errors.append(f"checker evidence identity mismatch for {change.path!r}") + elif proof.test_module != evidence.removesuffix(".py").replace("/", "."): + errors.append(f"checker evidence test mismatch for {change.path!r}") + elif not proof.rule_ids: + errors.append(f"checker change {change.path!r} has no affected POL rule IDs") + elif proof.head_tests <= 0: + errors.append(f"checker change {change.path!r} executed no HEAD tests") + elif not proof.head_passed: + errors.append(f"HEAD checker/test pair failed for {change.path!r}: {proof.detail}") + elif proof.base_tests <= 0: + errors.append(f"checker change {change.path!r} executed no BASE mutation tests") + elif not proof.base_failed: + errors.append( + f"BASE checker stayed green for {change.path!r}; changed test is not semantic evidence" + ) + for path_class in sorted(PATH_CLASSES): + budget = PATH_CLASS_BUDGETS[path_class] + if totals[path_class] > budget: + errors.append( + f"{path_class} changes total {totals[path_class]} lines, over the {budget}-line budget" + ) + return errors + + +def git(*args: str, repo: Path = ROOT) -> str: + return subprocess.check_output( + ["git", *args], cwd=repo, text=True, stderr=subprocess.STDOUT + ) + + +def resolve_commit(repo: Path, revision: str) -> str: + if not revision or "\x00" in revision or "\n" in revision: + raise ValueError(f"invalid revision {revision!r}") + try: + oid = git( + "rev-parse", + "--verify", + "--end-of-options", + f"{revision}^{{commit}}", + repo=repo, + ).strip() + except subprocess.CalledProcessError as exc: + raise ValueError(f"could not resolve revision {revision!r}") from exc + if re.fullmatch(r"[0-9a-f]{40}", oid) is None: + raise ValueError(f"revision {revision!r} did not resolve to one commit") + return oid + + +def require_ancestor(repo: Path, base_oid: str, head_oid: str) -> None: + result = subprocess.run( + ["git", "-C", str(repo), "merge-base", "--is-ancestor", base_oid, head_oid], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + timeout=EVIDENCE_TIMEOUT_SECONDS, + shell=False, + ) + if result.returncode == 1: + raise ValueError("base must be an ancestor of head") + if result.returncode != 0: + raise ValueError("could not establish base/head ancestry") + + +def changed_paths(base: str, head: str, *, repo: Path = ROOT) -> list[ChangedPath]: + base_oid = resolve_commit(repo, base) + head_oid = resolve_commit(repo, head) + require_ancestor(repo, base_oid, head_oid) + return parse_numstat( + git("diff", "--no-renames", "--numstat", base_oid, head_oid, repo=repo) + ) + + +def load_role_discipline(): + spec = importlib.util.spec_from_file_location( + "check_role_discipline", ROOT / "scripts/check-role-discipline.py" + ) + if spec is None or spec.loader is None: + raise ValueError("could not load role-discipline checker") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def affected_policy_rules( + checker_path: str, rules: dict[str, PolicyRule], root: Path +) -> tuple[str, ...]: + direct = tuple( + sorted( + rule_id + for rule_id, rule in rules.items() + if checker_path in {part.strip() for part in rule.enforcement.split(";")} + ) + ) + declared = CREATED_CHECKER_RULES.get(checker_path, ()) + rule_ids = tuple(sorted(set(direct) | set(declared))) + if not rule_ids: + raise ValueError(f"{checker_path} has no affected POL rule mapping") + for rule_id in rule_ids: + rule = rules.get(rule_id) + if rule is None: + raise ValueError(f"{checker_path} maps unknown policy rule {rule_id}") + if not (root / rule.procedure).is_file(): + raise ValueError( + f"{checker_path} rule {rule_id} has no existing procedure {rule.procedure}" + ) + return rule_ids + + +def _sanitized_env(home: Path) -> dict[str, str]: + return { + "PATH": os.defpath, + "HOME": str(home), + "LANG": "C.UTF-8", + "LC_ALL": "C.UTF-8", + "PYTHONDONTWRITEBYTECODE": "1", + "GIT_CONFIG_NOSYSTEM": "1", + } + + +def _run_test_module(worktree: Path, module: str) -> tuple[int, bool, str]: + try: + result = subprocess.run( + [sys.executable, "-m", "unittest", "-v", module], + cwd=worktree, + env=_sanitized_env(worktree), + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=EVIDENCE_TIMEOUT_SECONDS, + shell=False, + ) + except subprocess.TimeoutExpired as exc: + raise ValueError(f"semantic evidence timed out for {module}") from exc + counts = [int(value) for value in TEST_COUNT.findall(result.stdout)] + if len(counts) != 1 or counts[0] <= 0: + raise ValueError(f"semantic evidence did not execute tests for {module}") + return counts[0], result.returncode == 0, result.stdout[-2000:] + + +def _base_checker(repo: Path, base_oid: str, checker_path: str) -> bytes: + result = subprocess.run( + ["git", "-C", str(repo), "show", f"{base_oid}:{checker_path}"], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=EVIDENCE_TIMEOUT_SECONDS, + shell=False, + ) + if result.returncode == 0: + return result.stdout + if checker_path not in CREATION_MUTATIONS: + raise ValueError( + f"{checker_path} is absent at BASE and has no closed creation mutation" + ) + return CREATION_MUTATIONS[checker_path] + + +def executable_evidence( + repo: Path, + base: str, + head: str, + changes: list[ChangedPath], + rules: dict[str, PolicyRule], +) -> dict[str, EvidenceResult]: + """Prove each checker change red-before/green-after in an isolated worktree.""" + + base_oid = resolve_commit(repo, base) + head_oid = resolve_commit(repo, head) + require_ancestor(repo, base_oid, head_oid) + changed = {item.path for item in changes} + checkers = sorted( + item.path + for item in changes + if classify_path(item.path) == "governance_checker" + ) + results: dict[str, EvidenceResult] = {} + for checker_path in checkers: + evidence_path = recognized_evidence(checker_path) + if evidence_path not in changed: continue - total += int(added) + int(removed) - files.append(path) - return total, files + module = evidence_path.removesuffix(".py").replace("/", ".") + rule_ids = affected_policy_rules(checker_path, rules, repo) + container = Path( + tempfile.mkdtemp(prefix="vllm-policy-evidence-", dir="/dev/shm") + ) + worktree = container / "worktree" + try: + subprocess.run( + [ + "git", + "-C", + str(repo), + "worktree", + "add", + "--quiet", + "--detach", + str(worktree), + head_oid, + ], + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=EVIDENCE_TIMEOUT_SECONDS, + shell=False, + env=_sanitized_env(container), + ) + head_count, head_passed, head_detail = _run_test_module(worktree, module) + target = worktree / checker_path + target.write_bytes(_base_checker(repo, base_oid, checker_path)) + target.chmod(0o755) + base_count, base_passed, base_detail = _run_test_module(worktree, module) + results[checker_path] = EvidenceResult( + checker=checker_path, + test_module=module, + rule_ids=rule_ids, + head_tests=head_count, + head_passed=head_passed, + base_tests=base_count, + base_failed=not base_passed, + detail=head_detail if not head_passed else base_detail, + ) + finally: + subprocess.run( + ["git", "-C", str(repo), "worktree", "remove", "--force", str(worktree)], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + timeout=EVIDENCE_TIMEOUT_SECONDS, + shell=False, + env=_sanitized_env(container), + ) + shutil.rmtree(container, ignore_errors=True) + return results def main() -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--base", required=True) parser.add_argument("--head", required=True) - parser.add_argument("--branch", default="", help="head branch name") + parser.add_argument("--branch", default="", help="accepted for CI compatibility") args = parser.parse_args() - - total, files = counted_lines(args.base, args.head) - is_row_pr = args.branch.startswith("row/") - if total <= MAX_FEATURE_LINES: - print(f"OK: {total} non-exempt changed lines across {len(files)} file(s).") - return 0 - - message = ( - f"{total} non-exempt changed lines across {len(files)} file(s), over the " - f"{MAX_FEATURE_LINES} cap. Split into one row per PR so review stays " - "cheap; record and protocol paths are already exempt." - ) - if is_row_pr: - print(f"ERROR: {message}", file=sys.stderr) + del args.branch + try: + base_oid = resolve_commit(ROOT, args.base) + head_oid = resolve_commit(ROOT, args.head) + changes = changed_paths(base_oid, head_oid) + rules = load_policy(ROOT) + evidence = executable_evidence(ROOT, base_oid, head_oid, changes, rules) + errors = change_errors(changes, evidence_results=evidence) + except (OSError, ValueError, subprocess.CalledProcessError) as exc: + print(f"ERROR: PR size check could not classify the change: {exc}", file=sys.stderr) + return 1 + if errors: + for error in errors: + print(f"ERROR: {error}", file=sys.stderr) return 1 - print(f"REPORT: {message} (not a row/* PR, not enforced)") + print("OK: every explicit path class is within its review budget.") return 0 diff --git a/scripts/check-role-discipline.py b/scripts/check-role-discipline.py index 6ba0d3651..6850184be 100755 --- a/scripts/check-role-discipline.py +++ b/scripts/check-role-discipline.py @@ -8,11 +8,9 @@ helper session, or the developer all satisfy it the same way, and a direct push of feature code does not. -Integration work stays direct on purpose. `scripts/`, `.agents/`, `docs/` and -`.github/` are NOT feature paths, because the operator must be able to fix a -gate, resolve a conflict or repair the record without a round trip -- an operator -who cannot touch anything cannot review, which is the rubber-stamp failure the -protocol is designed to avoid. +Non-product paths are classified by closed lexical forms rather than exempting +whole mutable trees. This checker still owns the feature-arrival rule; the PR +and size gates independently review the other classes. ACTIVATION. ENFORCING since the cutover commit 44e8225cf (user-directed 2026-08-05). Every commit from that one ONWARD must land feature code through a @@ -33,6 +31,7 @@ import subprocess import sys from pathlib import Path +from pathlib import PurePosixPath ROOT = Path(__file__).resolve().parents[1] @@ -50,14 +49,17 @@ "cmake/", ) FEATURE_FILES = {"CMakeLists.txt"} -# Protocol/record tooling the operator legitimately maintains in place. -INTEGRATION_PREFIXES = ( - "tests/scripts/", - "scripts/", - ".agents/", - "docs/", - ".github/", -) +INTEGRATION_FILES = { + ".agents/state.md", + "docs/STATUS.md", + "docs/BENCHMARKS.md", + "docs/FEATURES.md", + "docs/USAGE.md", + "README.md", +} +CHECKER_PATH = re.compile(r"scripts/check-[A-Za-z0-9_.-]+\.(?:py|sh)\Z") +CHECKER_TEST_PATH = re.compile(r"tests/scripts/test_[A-Za-z0-9_.-]+\.py\Z") +CI_PATH = re.compile(r"\.github/workflows/[A-Za-z0-9_.-]+\.ya?ml\Z") ROW_BRANCH = re.compile(r"row/[A-Za-z0-9_.-]+") PR_REFERENCE = re.compile(r"\(#\d+\)|#\d+") @@ -70,7 +72,22 @@ def git(*args: str) -> str: def is_feature_path(path: str) -> bool: - if path.startswith(INTEGRATION_PREFIXES): + candidate = PurePosixPath(path) + if ( + not path + or candidate.is_absolute() + or "\\" in path + or "//" in path + or candidate.as_posix() != path + or any(part in {"", ".", ".."} for part in candidate.parts) + ): + return True + if ( + path in INTEGRATION_FILES + or CHECKER_PATH.fullmatch(path) + or CHECKER_TEST_PATH.fullmatch(path) + or CI_PATH.fullmatch(path) + ): return False return path in FEATURE_FILES or path.startswith(FEATURE_PREFIXES) @@ -120,8 +137,29 @@ def commit_violations( return [ f"{commit}: feature code ({preview}) reached main without a reviewed " "row/* PR. Feature work goes through a helper session or a sub-agent on " - "a `row/` branch; the operator merges it. Integration paths " - "(scripts/, .agents/, docs/, .github/) are exempt by design" + "a `row/` branch; the operator merges it" + ] + + +def policy_commit_violations( + commit: str, + parents: list[str], + subject: str, + body: str, + paths: list[str], + merged_messages: tuple[str, ...] = (), +) -> list[str]: + """Enforce POL-PR-REQUIRED for every tracked repository change.""" + + governed = sorted(path for path in paths if path) + if not governed or arrives_via_row_pr(parents, subject, body, merged_messages): + return [] + preview = ", ".join(governed[:4]) + if len(governed) > 4: + preview += f", ... (+{len(governed) - 4})" + return [ + f"{commit}: repository change ({preview}) reached main without a reviewed " + "row/* PR" ] @@ -142,7 +180,9 @@ def inspect(commit: str) -> list[str]: # The messages of the branches this commit MERGES IN (parents[1:]), for the # synthetic-PR-merge case in arrives_via_row_pr. merged = tuple(git("log", "-1", "--format=%s%n%b", parent) for parent in parents[1:]) - return commit_violations(short, parents, subject, body, commit_paths(commit), merged) + return policy_commit_violations( + short, parents, subject, body, commit_paths(commit), merged + ) def enforced(commit: str) -> bool: @@ -156,6 +196,25 @@ def enforced(commit: str) -> bool: return False +def has_reached_main(commit: str) -> bool: + """Whether the commit is already contained by the local main ref.""" + + try: + git("merge-base", "--is-ancestor", commit, "refs/heads/main") + return True + except subprocess.CalledProcessError: + pass + try: + branch = git("symbolic-ref", "--quiet", "--short", "HEAD") + except subprocess.CalledProcessError: + # CI checks out main and synthetic PR merges detached. A synthetic merge + # already passes arrives_via_row_pr; a violating detached commit must + # fail closed as landed/integration history. + return True + # An unmerged row head is pending PR disposition, not main history yet. + return not branch.startswith("row/") + + def commits_in_range(base: str, head: str) -> list[str]: try: git("cat-file", "-e", f"{base}^{{commit}}") @@ -183,7 +242,11 @@ def main() -> int: failures, reported = [], [] for commit in commits: for problem in inspect(commit): - (failures if enforced(commit) else reported).append(problem) + # A row head has not reached main yet, so it is reportable pending + # integration rather than a false claim that unmerged work already + # violated the arrival rule. Main history and recognized synthetic + # PR merges remain strict. + (failures if enforced(commit) and has_reached_main(commit) else reported).append(problem) for problem in reported: print(f"REPORT: {problem}", file=sys.stderr) diff --git a/tests/scripts/test_check_commit_trailers.py b/tests/scripts/test_check_commit_trailers.py new file mode 100755 index 000000000..e559f3a6d --- /dev/null +++ b/tests/scripts/test_check_commit_trailers.py @@ -0,0 +1,275 @@ +#!/usr/bin/env python3 +"""Behavior tests for strict Git trailer enforcement.""" + +from __future__ import annotations + +import importlib.util +import subprocess +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-commit-trailers.py" + + +def load_checker(): + spec = importlib.util.spec_from_file_location("check_commit_trailers", CHECKER) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +STRICT_MESSAGE = """policy: enforce trailers + +The body may explain the change. + +FOLLOWING_AGENTS_PROTOCOL + +Following-Agents-Protocol: true +AI-Assisted: true +Assisted-by: Codex:GPT-5 [Codex] +""" + + +class CommitMessageContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.checker = load_checker() + + def assertInvalid(self, message: str, needle: str) -> None: + errors = self.checker.validate_commit_message(message, strict=True) + self.assertTrue(any(needle in error for error in errors), errors) + + def test_valid_ai_assisted_and_human_only_messages(self) -> None: + self.assertEqual( + self.checker.validate_commit_message(STRICT_MESSAGE, strict=True), [] + ) + human = STRICT_MESSAGE.replace( + "AI-Assisted: true\nAssisted-by: Codex:GPT-5 [Codex]\n", + "AI-Assisted: false\n", + ) + self.assertEqual( + self.checker.validate_commit_message(human, strict=True), [] + ) + + def test_substring_or_embedded_legacy_tag_is_not_a_raw_paragraph(self) -> None: + self.assertInvalid( + STRICT_MESSAGE.replace( + "FOLLOWING_AGENTS_PROTOCOL\n\n", + "We are FOLLOWING_AGENTS_PROTOCOL today.\n\n", + ), + "separate paragraph", + ) + self.assertInvalid( + STRICT_MESSAGE.replace( + "FOLLOWING_AGENTS_PROTOCOL\n\n", + "FOLLOWING_AGENTS_PROTOCOL extra\n\n", + ), + "separate paragraph", + ) + + def test_raw_tag_must_be_separate_from_the_trailer_paragraph(self) -> None: + message = STRICT_MESSAGE.replace( + "FOLLOWING_AGENTS_PROTOCOL\n\nFollowing-Agents", + "FOLLOWING_AGENTS_PROTOCOL\nFollowing-Agents", + ) + self.assertInvalid(message, "separate paragraph") + + def test_protocol_and_ai_declarations_are_unique_and_exact(self) -> None: + self.assertInvalid( + STRICT_MESSAGE.replace( + "Following-Agents-Protocol: true", + "Following-Agents-Protocol: false", + ), + "must be exactly true", + ) + self.assertInvalid( + STRICT_MESSAGE.replace( + "Following-Agents-Protocol: true", + "Following-Agents-Protocol: true\nFollowing-Agents-Protocol: true", + ), + "exactly once", + ) + self.assertInvalid( + STRICT_MESSAGE.replace("AI-Assisted: true", "AI-Assisted: maybe"), + "true or false", + ) + self.assertInvalid( + STRICT_MESSAGE.replace( + "AI-Assisted: true", "AI-Assisted: true\nAI-Assisted: false" + ), + "exactly once", + ) + + def test_assistance_attribution_is_required_and_has_closed_syntax(self) -> None: + self.assertInvalid( + STRICT_MESSAGE.replace("Assisted-by: Codex:GPT-5 [Codex]\n", ""), + "Assisted-by", + ) + for malformed in ( + "Codex GPT-5 [Codex]", + "Codex:GPT-5", + "Codex:GPT-5 []", + "Codex: GPT-5 [Codex]", + "Codex:GPT-5 [Codex] trailing", + ): + with self.subTest(malformed=malformed): + self.assertInvalid( + STRICT_MESSAGE.replace("Codex:GPT-5 [Codex]", malformed), + "malformed Assisted-by", + ) + + def test_human_only_declaration_rejects_assistance_attribution(self) -> None: + message = STRICT_MESSAGE.replace("AI-Assisted: true", "AI-Assisted: false") + self.assertInvalid(message, "must omit Assisted-by") + + def test_ai_authorship_trailers_are_forbidden(self) -> None: + for trailer in ( + "Signed-off-by: Codex:GPT-5 [Codex]", + "Co-Authored-By: Claude Opus ", + "Signed-off-by: OpenAI Bot ", + ): + with self.subTest(trailer=trailer): + self.assertInvalid( + STRICT_MESSAGE.rstrip() + "\n" + trailer + "\n", + "AI authorship", + ) + + def test_human_authorship_trailers_remain_allowed(self) -> None: + for trailer in ( + "Signed-off-by: Alice Example ", + "Co-Authored-By: Bob Human ", + ): + with self.subTest(trailer=trailer): + message = STRICT_MESSAGE.rstrip() + "\n" + trailer + "\n" + self.assertEqual( + self.checker.validate_commit_message(message, strict=True), [] + ) + + def test_every_declared_agent_model_and_tool_identity_is_not_an_author(self) -> None: + message = STRICT_MESSAGE.replace( + "Codex:GPT-5 [Codex]", + "Unfamiliar-Agent:Model-Seven [NeutralTool] [SecondTool]", + ) + for identity in ("Model-Seven", "NeutralTool", "SecondTool"): + with self.subTest(identity=identity): + authored = ( + message.rstrip() + + f"\nCo-Authored-By: {identity} <{identity.casefold()}@example.com>\n" + ) + self.assertInvalid(authored, "AI authorship") + + def test_assisted_agent_identity_is_forbidden_as_an_authorship_trailer(self) -> None: + message = STRICT_MESSAGE.replace( + "Codex:GPT-5 [Codex]", "Unfamiliar-Agent:Model-7 [NeutralTool]" + ).rstrip() + message += "\nSigned-off-by: Unfamiliar-Agent \n" + self.assertInvalid(message, "AI authorship") + + def test_legacy_is_only_accepted_in_non_strict_mode(self) -> None: + legacy = "legacy change\n\nFOLLOWING_AGENTS_PROTOCOL\n" + self.assertEqual( + self.checker.validate_commit_message(legacy, strict=False), [] + ) + self.assertTrue( + self.checker.validate_commit_message(legacy, strict=True) + ) + + def test_parser_matches_git_interpret_trailers(self) -> None: + parsed = subprocess.check_output( + ["git", "interpret-trailers", "--parse"], + input=STRICT_MESSAGE, + text=True, + ) + self.assertEqual(self.checker.parsed_trailers(STRICT_MESSAGE), parsed) + + +class RangeContract(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.checker = load_checker() + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.repo = Path(self.tmp.name) + subprocess.run(["git", "init", "-q", str(self.repo)], check=True) + subprocess.run( + ["git", "-C", str(self.repo), "config", "user.email", "test@example.com"], + check=True, + ) + subprocess.run( + ["git", "-C", str(self.repo), "config", "user.name", "Test"], + check=True, + ) + + def tearDown(self) -> None: + self.tmp.cleanup() + + def commit(self, message: str) -> str: + marker = self.repo / "history" + marker.write_text(marker.read_text() + "x" if marker.exists() else "x") + subprocess.run(["git", "-C", str(self.repo), "add", "history"], check=True) + subprocess.run( + ["git", "-C", str(self.repo), "commit", "-q", "-F", "-"], + input=message, + text=True, + check=True, + ) + return subprocess.check_output( + ["git", "-C", str(self.repo), "rev-parse", "HEAD"], text=True + ).strip() + + def test_cutover_commit_itself_is_strict_and_parent_is_legacy(self) -> None: + base = self.commit("base\n\nFOLLOWING_AGENTS_PROTOCOL\n") + before = self.commit("before\n\nFOLLOWING_AGENTS_PROTOCOL\n") + cutover = self.commit(STRICT_MESSAGE) + after = self.commit(STRICT_MESSAGE.replace("policy:", "policy after:")) + errors = self.checker.validate_range( + self.repo, base, after, cutover=cutover, rules=None, waivers=[] + ) + self.assertEqual(errors, []) + self.assertNotEqual(before, cutover) + + def test_post_cutover_legacy_commit_fails(self) -> None: + base = self.commit("base\n\nFOLLOWING_AGENTS_PROTOCOL\n") + cutover = self.commit(STRICT_MESSAGE) + self.commit("after\n\nFOLLOWING_AGENTS_PROTOCOL\n") + errors = self.checker.validate_range( + self.repo, base, "HEAD", cutover=cutover, rules=None, waivers=[] + ) + self.assertTrue(any("Following-Agents-Protocol" in error for error in errors)) + + def test_missing_unreachable_and_non_ancestor_revisions_fail_closed(self) -> None: + base = self.commit("base\n\nFOLLOWING_AGENTS_PROTOCOL\n") + head = self.commit(STRICT_MESSAGE) + with self.assertRaises(ValueError): + self.checker.validate_range( + self.repo, "missing", head, cutover=head, rules=None, waivers=[] + ) + subprocess.run(["git", "-C", str(self.repo), "checkout", "-q", "--detach", base], check=True) + side = self.commit(STRICT_MESSAGE.replace("policy:", "side:")) + with self.assertRaises(ValueError): + self.checker.validate_range( + self.repo, side, head, cutover=head, rules=None, waivers=[] + ) + + def test_ambiguous_revision_name_fails_closed(self) -> None: + base = self.commit("base\n\nFOLLOWING_AGENTS_PROTOCOL\n") + head = self.commit(STRICT_MESSAGE) + subprocess.run( + ["git", "-C", str(self.repo), "branch", "collision", base], check=True + ) + subprocess.run( + ["git", "-C", str(self.repo), "tag", "collision", head], check=True + ) + with self.assertRaises(ValueError): + self.checker.validate_range( + self.repo, base, "collision", cutover=head, rules=None, waivers=[] + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_check_pr_size.py b/tests/scripts/test_check_pr_size.py new file mode 100755 index 000000000..a910a1fbf --- /dev/null +++ b/tests/scripts/test_check_pr_size.py @@ -0,0 +1,285 @@ +#!/usr/bin/env python3 +"""Mutation tests for explicit PR path classes and budgets.""" + +from __future__ import annotations + +import importlib.util +import tempfile +import subprocess +import sys +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +SPEC = importlib.util.spec_from_file_location("check_pr_size", ROOT / "scripts/check-pr-size.py") +assert SPEC is not None and SPEC.loader is not None +checker = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = checker +SPEC.loader.exec_module(checker) + + +class PathClassification(unittest.TestCase): + def test_each_mutable_surface_has_an_explicit_class(self) -> None: + expected = { + "src/vt/x.cpp": "product", + "scripts/check-policy.py": "governance_checker", + "tests/scripts/test_policy_contract.py": "governance_test", + ".agents/policy.csv": "policy", + ".agents/state.md": "append_only_record", + "docs/STATUS.md": "public_document", + ".github/workflows/ci.yml": "ci", + } + for path, path_class in expected.items(): + with self.subTest(path=path): + self.assertEqual(checker.classify_path(path), path_class) + + def test_unknown_and_noncanonical_paths_fail_closed(self) -> None: + for path in ( + "unknown.bin", + "/scripts/check-policy.py", + "scripts/../src/x.cpp", + "scripts//check-policy.py", + r"scripts\check-policy.py", + "./scripts/check-policy.py", + ): + with self.subTest(path=path): + with self.assertRaises(ValueError): + checker.classify_path(path) + + def test_every_tracked_and_current_change_path_is_classified(self) -> None: + paths = set( + subprocess.check_output(["git", "ls-files"], cwd=ROOT, text=True).splitlines() + ) + paths.update( + subprocess.check_output( + ["git", "diff", "--name-only", "origin/main"], cwd=ROOT, text=True + ).splitlines() + ) + failures = [] + for path in sorted(paths): + try: + checker.classify_path(path) + except ValueError: + failures.append(path) + self.assertEqual(failures, []) + + def test_similar_names_do_not_enter_governance_classes(self) -> None: + self.assertNotEqual( + checker.classify_path("scripts/checkpoint-runner.py"), + "governance_checker", + ) + self.assertNotEqual( + checker.recognized_evidence("scripts/check-policy.py"), + "tests/scripts/test_policy_contract_extra.py", + ) + + +class BudgetEnforcement(unittest.TestCase): + def change(self, path: str, lines: int) -> checker.ChangedPath: + return checker.ChangedPath(path, lines, 0) + + def test_every_class_has_a_finite_positive_budget(self) -> None: + self.assertEqual(set(checker.PATH_CLASS_BUDGETS), set(checker.PATH_CLASSES)) + self.assertTrue(all(0 < value < 10000 for value in checker.PATH_CLASS_BUDGETS.values())) + + def test_budget_boundary_passes_and_one_over_fails_on_every_branch(self) -> None: + path = "scripts/check-pr-size.py" + limit = checker.PATH_CLASS_BUDGETS["governance_checker"] + evidence = self.change("tests/scripts/test_check_pr_size.py", 1) + self.assertEqual( + checker.change_errors([self.change(path, limit), evidence]), [] + ) + errors = checker.change_errors([self.change(path, limit + 1), evidence]) + self.assertTrue(any("governance_checker" in error for error in errors), errors) + + def test_oversized_policy_and_governance_test_changes_fail(self) -> None: + for path, path_class in ( + (".agents/policy.csv", "policy"), + ("tests/scripts/test_policy_contract.py", "governance_test"), + ): + with self.subTest(path=path): + errors = checker.change_errors( + [self.change(path, checker.PATH_CLASS_BUDGETS[path_class] + 1)] + ) + self.assertTrue(errors) + + def test_binary_changes_fail_closed_instead_of_becoming_free(self) -> None: + errors = checker.change_errors([checker.ChangedPath("docs/image.png", None, None)]) + self.assertTrue(any("binary" in error for error in errors), errors) + + def test_checker_change_requires_its_recognized_mutation_test(self) -> None: + changed = [ + self.change("scripts/check-pr-size.py", 5), + self.change("tests/scripts/test_unrelated.py", 50), + ] + errors = checker.change_errors(changed, evidence_results={}) + self.assertTrue(any("mutation evidence" in error for error in errors), errors) + changed.append(self.change("tests/scripts/test_check_pr_size.py", 5)) + proof = checker.EvidenceResult( + checker="scripts/check-pr-size.py", + test_module="tests.scripts.test_check_pr_size", + rule_ids=("POL-PATH-CLASSIFICATION", "POL-PR-SIZE"), + head_tests=1, + head_passed=True, + base_tests=1, + base_failed=True, + ) + self.assertEqual( + checker.change_errors( + changed, evidence_results={"scripts/check-pr-size.py": proof} + ), + [], + ) + + def test_unchanged_or_mode_only_test_is_not_semantic_mutation_evidence(self) -> None: + errors = checker.change_errors( + [ + self.change("scripts/check-pr-size.py", 1), + self.change("tests/scripts/test_check_pr_size.py", 0), + ] + ) + self.assertTrue(any("mutation evidence" in error for error in errors), errors) + + def test_fake_or_comment_only_test_edit_cannot_prove_checker_semantics(self) -> None: + changes = [ + self.change("scripts/check-pr-size.py", 1), + self.change("tests/scripts/test_check_pr_size.py", 1), + ] + fake = checker.EvidenceResult( + checker="scripts/check-pr-size.py", + test_module="tests.scripts.test_check_pr_size", + rule_ids=("POL-PATH-CLASSIFICATION",), + head_tests=1, + head_passed=True, + base_tests=1, + base_failed=False, + ) + errors = checker.change_errors( + changes, evidence_results={"scripts/check-pr-size.py": fake} + ) + self.assertTrue(any("BASE checker stayed green" in error for error in errors), errors) + + def test_missing_policy_binding_or_unexecuted_test_fails_closed(self) -> None: + changes = [ + self.change("scripts/check-pr-size.py", 1), + self.change("tests/scripts/test_check_pr_size.py", 1), + ] + for rules, count in (((), 1), (("POL-PR-SIZE",), 0)): + with self.subTest(rules=rules, count=count): + proof = checker.EvidenceResult( + checker="scripts/check-pr-size.py", + test_module="tests.scripts.test_check_pr_size", + rule_ids=rules, + head_tests=count, + head_passed=True, + base_tests=1, + base_failed=True, + ) + self.assertTrue( + checker.change_errors( + changes, + evidence_results={"scripts/check-pr-size.py": proof}, + ) + ) + + def test_arbitrary_test_filename_cannot_claim_mutation_evidence(self) -> None: + errors = checker.change_errors( + [ + self.change("scripts/check-policy.py", 1), + self.change("tests/scripts/test_check_policy.py", 1), + ] + ) + self.assertTrue(any("test_policy_contract.py" in error for error in errors), errors) + + def test_numstat_parser_rejects_malformed_negative_and_duplicate_paths(self) -> None: + for text in ( + "1\t2\n", + "-1\t2\tsrc/x.cpp\n", + "1\t2\tsrc/x.cpp\n3\t4\tsrc/x.cpp\n", + ): + with self.subTest(text=text): + with self.assertRaises(ValueError): + checker.parse_numstat(text) + + def test_real_git_rename_is_delete_plus_add_with_no_rename_syntax(self) -> None: + with tempfile.TemporaryDirectory(dir="/dev/shm") as directory: + repo = Path(directory) + subprocess.run(["git", "init", "-q", str(repo)], check=True) + subprocess.run(["git", "-C", str(repo), "config", "user.name", "Test"], check=True) + subprocess.run(["git", "-C", str(repo), "config", "user.email", "test@example.com"], check=True) + (repo / "src").mkdir() + (repo / "src" / "old.cpp").write_text("line\n", encoding="utf-8") + subprocess.run(["git", "-C", str(repo), "add", "."], check=True) + subprocess.run(["git", "-C", str(repo), "commit", "-qm", "base"], check=True) + base = subprocess.check_output(["git", "-C", str(repo), "rev-parse", "HEAD"], text=True).strip() + subprocess.run(["git", "-C", str(repo), "mv", "src/old.cpp", "src/new.cpp"], check=True) + subprocess.run(["git", "-C", str(repo), "commit", "-qm", "rename"], check=True) + head = subprocess.check_output(["git", "-C", str(repo), "rev-parse", "HEAD"], text=True).strip() + changes = checker.changed_paths(base, head, repo=repo) + self.assertEqual( + [(change.path, change.added, change.removed) for change in changes], + [("src/new.cpp", 1, 0), ("src/old.cpp", 0, 1)], + ) + self.assertFalse(any("{" in change.path or "=>" in change.path for change in changes)) + + def test_missing_and_nonancestor_objects_fail_closed(self) -> None: + with tempfile.TemporaryDirectory(dir="/dev/shm") as directory: + repo = Path(directory) + subprocess.run(["git", "init", "-q", str(repo)], check=True) + subprocess.run(["git", "-C", str(repo), "config", "user.name", "Test"], check=True) + subprocess.run(["git", "-C", str(repo), "config", "user.email", "test@example.com"], check=True) + (repo / "src").mkdir() + (repo / "src" / "base.cpp").write_text("base\n", encoding="utf-8") + subprocess.run(["git", "-C", str(repo), "add", "."], check=True) + subprocess.run(["git", "-C", str(repo), "commit", "-qm", "base"], check=True) + base = subprocess.check_output(["git", "-C", str(repo), "rev-parse", "HEAD"], text=True).strip() + subprocess.run(["git", "-C", str(repo), "checkout", "-q", "--orphan", "side"], check=True) + subprocess.run(["git", "-C", str(repo), "rm", "-qrf", "."], check=True) + (repo / "src").mkdir() + (repo / "src" / "side.cpp").write_text("side\n", encoding="utf-8") + subprocess.run(["git", "-C", str(repo), "add", "."], check=True) + subprocess.run(["git", "-C", str(repo), "commit", "-qm", "side"], check=True) + side = subprocess.check_output(["git", "-C", str(repo), "rev-parse", "HEAD"], text=True).strip() + with self.assertRaises(ValueError): + checker.changed_paths("missing", side, repo=repo) + with self.assertRaisesRegex(ValueError, "ancestor"): + checker.changed_paths(base, side, repo=repo) + + def test_production_pr_classifier_covers_every_governed_path_class(self) -> None: + governed = ( + "src/x.cpp", + ".agents/policy.csv", + "scripts/check-policy.py", + "tests/scripts/test_policy_contract.py", + ".agents/workflow.md", + ".agents/state.md", + ".agents/NOW.md", + "docs/STATUS.md", + ".github/workflows/ci.yml", + "scripts/agent-role.py", + ".env.example", + ) + for path in governed: + with self.subTest(path=path): + self.assertTrue(checker.requires_reviewed_pr(path)) + + def test_production_role_check_requires_pr_for_records_docs_ci_and_support(self) -> None: + role = checker.load_role_discipline() + for path in ( + ".agents/state.md", + "docs/STATUS.md", + ".github/workflows/ci.yml", + "scripts/agent-role.py", + ".env.example", + ): + with self.subTest(path=path): + self.assertTrue( + role.policy_commit_violations( + "abc123", ["parent"], "direct", "", [path] + ) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_doc_checkpoint.py b/tests/scripts/test_doc_checkpoint.py index 9cbd31d5d..77fde35fd 100755 --- a/tests/scripts/test_doc_checkpoint.py +++ b/tests/scripts/test_doc_checkpoint.py @@ -129,6 +129,7 @@ def test_runtime_checkers_are_not_hidden_by_governance_names(self) -> None: def test_exact_governance_checker_files_are_exempt(self) -> None: for path in ( + "scripts/check-commit-trailers.py", "scripts/check-policy.py", "scripts/check-doc-checkpoint.py", "scripts/check-prompt-contract.py", @@ -137,12 +138,26 @@ def test_exact_governance_checker_files_are_exempt(self) -> None: "tests/scripts/test_doc_checkpoint.py", "tests/scripts/test_check_prompt_contract.py", "tests/scripts/test_check_protocol_consistency.py", + "tests/scripts/test_check_commit_trailers.py", + "tests/scripts/test_policy_waivers.py", + "tests/scripts/test_check_pr_size.py", ): with self.subTest(path=path): self.assertEqual( doc_checkpoint.classify_changed_paths([path]), {"governance"} ) + def test_similar_governance_names_remain_feature_checkpoints(self) -> None: + for path in ( + "scripts/check-commit-trailers-extra.py", + "tests/scripts/test_policy_waivers_extra.py", + "tests/scripts/test_check_pr_size.md", + ): + with self.subTest(path=path): + classes = doc_checkpoint.classify_changed_paths([path]) + self.assertIn("feature_checkpoint", classes) + self.assertNotIn("governance", classes) + def test_governance_only_task_one_files_are_not_a_checkpoint(self) -> None: paths = [ ".agents/policy.csv", diff --git a/tests/scripts/test_policy_waivers.py b/tests/scripts/test_policy_waivers.py new file mode 100755 index 000000000..837b81d4b --- /dev/null +++ b/tests/scripts/test_policy_waivers.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +"""Exact waiver selection tests at the enforcement boundary.""" + +from __future__ import annotations + +import datetime as dt +import importlib.util +import tempfile +import unittest +from pathlib import Path + +from scripts.policy_contract import Waiver + + +ROOT = Path(__file__).resolve().parents[2] +SPEC = importlib.util.spec_from_file_location( + "check_commit_trailers", ROOT / "scripts/check-commit-trailers.py" +) +assert SPEC is not None and SPEC.loader is not None +checker = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(checker) + + +class ExactWaiverSelection(unittest.TestCase): + def setUp(self) -> None: + self.waiver = Waiver( + waiver_id="WAIVER-TRAILER-001", + rule_id="POL-COMMIT-TRAILERS", + scope="commit:" + "a" * 40, + owner="maintainer", + reason="bounded migration", + evidence="PR-128", + expires=dt.date(2026, 8, 9), + ) + + def test_rule_and_complete_scope_must_match_exactly(self) -> None: + self.assertEqual( + checker.exact_waiver( + [self.waiver], "POL-COMMIT-TRAILERS", "commit:" + "a" * 40 + ), + self.waiver, + ) + for rule, scope in ( + ("POL-COMMIT-TRAILER", "commit:" + "a" * 40), + ("POL-COMMIT-TRAILERS-EXTRA", "commit:" + "a" * 40), + ("POL-COMMIT-TRAILERS", "commit:" + "a" * 39), + ("POL-COMMIT-TRAILERS", "commit:" + "a" * 40 + "b"), + ("POL-COMMIT-TRAILERS", "path:commit:" + "a" * 40), + ): + with self.subTest(rule=rule, scope=scope): + self.assertIsNone(checker.exact_waiver([self.waiver], rule, scope)) + + def test_duplicate_applicable_waivers_fail_closed(self) -> None: + duplicate = Waiver( + **{**self.waiver.__dict__, "waiver_id": "WAIVER-TRAILER-002"} + ) + with self.assertRaises(ValueError): + checker.exact_waiver( + [self.waiver, duplicate], + "POL-COMMIT-TRAILERS", + "commit:" + "a" * 40, + ) + + def test_path_waiver_may_name_a_file_but_not_an_existing_directory(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "src").mkdir() + (root / "src" / "one.cpp").write_text("x", encoding="utf-8") + directory_waiver = Waiver( + **{**self.waiver.__dict__, "scope": "path:src"} + ) + file_waiver = Waiver( + **{**self.waiver.__dict__, "scope": "path:src/one.cpp"} + ) + with self.assertRaisesRegex(ValueError, "directory"): + checker.validate_waiver_targets(root, [directory_waiver]) + checker.validate_waiver_targets(root, [file_waiver]) + + +if __name__ == "__main__": + unittest.main() From 00927ed611f4c5b720ceb158f6174be1e5470b03 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 8 Aug 2026 11:45:02 +0000 Subject: [PATCH 08/12] Consolidate agent policy procedures Make policy.csv the sole rule registry, replace the oversized bootstrap with checked compact routing, preserve the spec/TDD/review/verification lifecycle, archive legacy policy prose, and repair active and historical links. Affected rules: POL-AUTH-REGISTRY, POL-AUTH-PRECEDENCE, POL-BOOT-ROLE, POL-BOOT-NOW, POL-BOOT-TASK, POL-SPIKE-FIRST, POL-REVIEW-FRESH, POL-REVIEW-NO-REPAIR, POL-OPERATOR-VERIFY, POL-PR-DISPOSITION, POL-DOC-STATUS, POL-DOC-BENCHMARKS, POL-DOC-FEATURES, POL-DOC-USAGE, POL-DOC-README, POL-NOW-COUPLING, POL-EVIDENCE-PRESERVE, POL-CHECKER-CHANGE. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex] --- .agents/NOW.md | 6 +- .agents/backend-matrix.md | 4 +- .agents/backends.md | 4 +- .agents/benchmark-record.md | 2 +- .../ai-coding-assistants-legacy.md} | 2 +- .../benchmark-protocol-legacy.md} | 16 +- .../mvp-gates-legacy.md} | 4 +- .../operator-helper-protocol-legacy.md} | 0 .../policy-directives-legacy.md} | 25 +- .../porting-discipline-legacy.md} | 2 +- .agents/completed/roadmap_mvp_v0.md | 2 +- .../roadmap_v1_inventory_spikes_2026-07-10.md | 2 +- .../test-porting-legacy.md} | 4 +- .agents/parity-ledger.md | 16 +- .agents/policy.csv | 58 +-- .agents/porting.md | 44 ++ .agents/roadmap_v1.md | 6 +- .../specs/backend-fanout-metal-vulkan-xpu.md | 6 +- .agents/specs/cuda-sglang-low-concurrency.md | 2 +- .agents/specs/gemma4-multimodal.md | 2 +- .agents/specs/glm-dsa-latest-deepseek.md | 6 +- .agents/specs/kv-persistence-lmcache.md | 4 +- .agents/specs/lmcache-cpp-client-connector.md | 4 +- .agents/specs/mla-deepseek-campaign.md | 2 +- .agents/specs/model-factory-registry.md | 6 +- .agents/specs/multimodal-track.md | 2 +- .agents/specs/prefix-prompt-caching-parity.md | 4 +- .agents/specs/quantization-coverage.md | 4 +- .agents/specs/session-onboarding.md | 4 +- .agents/specs/sglang-parity-oracle.md | 8 +- .agents/specs/sweep-gemma.md | 6 +- .agents/specs/sweep-olmo2.md | 6 +- .agents/specs/sweep-recent-dense-batch.md | 4 +- .agents/specs/vulkan-full-support.md | 6 +- .agents/state.md | 24 +- .agents/verification.md | 52 ++ .agents/workflow.md | 414 +++++----------- AGENTS.md | 457 ++++-------------- docs/ROCM.md | 4 +- docs/STATUS.md | 7 +- scripts/check-doc-checkpoint.py | 53 ++ scripts/policy_contract.py | 253 +++++++++- tests/scripts/test_doc_checkpoint.py | 18 + tests/scripts/test_policy_contract.py | 132 +++++ 44 files changed, 900 insertions(+), 787 deletions(-) rename .agents/{ai-coding-assistants.md => completed/ai-coding-assistants-legacy.md} (96%) rename .agents/{benchmark-protocol.md => completed/benchmark-protocol-legacy.md} (94%) rename .agents/{gates.md => completed/mvp-gates-legacy.md} (97%) rename .agents/{specs/operator-helper-protocol.md => completed/operator-helper-protocol-legacy.md} (100%) rename .agents/{directives.md => completed/policy-directives-legacy.md} (97%) rename .agents/{discipline.md => completed/porting-discipline-legacy.md} (99%) rename .agents/{test-porting.md => completed/test-porting-legacy.md} (95%) create mode 100644 .agents/porting.md create mode 100644 .agents/verification.md diff --git a/.agents/NOW.md b/.agents/NOW.md index ed5f240f6..0aa3f8bef 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -35,8 +35,8 @@ bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Unchanged: token-exact (or the ratified distributional gate) against the pinned vLLM oracle, AND ≥ vLLM on every throughput axis / ≤ on latency and memory, on -both gate models, reproduced 2–3x on an idle box. See [gates.md](gates.md) and -[benchmark-protocol.md](benchmark-protocol.md). Parity pin: vLLM `555967922` +both gate models, reproduced 2–3x on an idle box. See the +[verification procedure](verification.md). Parity pin: vLLM `555967922` (0.26.0.dev0). Method rules hardened (AGENTS.md): the STRUCTURAL lens (same kernel, different @@ -63,7 +63,7 @@ throughput ⇒ audit the context; per-shape MEASUREMENT arbitrates). on the 10 audited rows (audit §➁a); its fix owes an 8-row adjudication. **Operator/helper protocol** -([spec](specs/operator-helper-protocol.md)): roles DECLARED then MATERIALIZED +([spec](workflow.md)): roles DECLARED then MATERIALIZED into a lock or worktree+PR; operator merges PRs first and does features only via sub-agents; helpers use worktrees on `row/` and open a DRAFT PR at the START, which IS the claim. **W0-W5 LANDED**; role discipline ENFORCING, diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 56806aca0..47f71cc06 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -11,7 +11,7 @@ Upstream references are pinned to vLLM `555967922` (0.26.0.dev0 + transformers Build availability is not runtime support. A target remains `INVENTORIED` until it has a native build, ported tests, hardware execution, and a trace showing the intended dispatch. Performance claims additionally require same-workload runs -under [benchmark-protocol.md](benchmark-protocol.md). +under [verification.md](verification.md). **How an architecture is added (the additive contract, `BACKEND-CUDA-ARCH-ADDITIVITY`).** Since [arch-additivity](specs/cuda-arch-additivity.md) landed, arch support is @@ -241,7 +241,7 @@ memory. Floating competitor versions do not count. | ID | Item | Upstream/reference | Our code | Tests/evidence | Spike/spec | State | Owner | |---|---|---|---|---|---|---|---| -| `BACKEND-GATE-CUDA-VLLM` | CUDA correctness and every-axis parity vs vLLM v0.25.0 | current pin `e24d1b24`; oracle `702f481`; async default `vllm/config/vllm.py:952-1043`; depth-2 engine path `vllm/v1/engine/core.py:519-607`; async output `vllm/v1/worker/gpu/async_utils.py:12-70`; [benchmark protocol](benchmark-protocol.md) | Existing schema-v5 validator/summary remain. The production [packed component driver](../scripts/dgx-gdn-packed-component.sh#L1) and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py#L1) enforce exact corpus/build/oracle provenance, exact and bounded raw-metric validation, duration consistency, exact commands, pairwise stability, one-lock correctness, thermal/memory health, symlink-safe sealing and marker-last provenance | [Component/summary tests](../tests/tools/test_gdn_packed_component.py#L1) pass and the [2026-07-15 new-binding ledger row](parity-ledger.md) records the sealed evidence. **NEW BINDING `9ecd9d0`: 114/124** (full production default set = async + Triton GDN cubin + bit-identical fast RMSNorm + bit-identical fast gated-RMSNorm; supersedes `246a23c`'s 49/124, `3f256ab`'s 55/124, all retained immutable in the ledger). Mem **4/4**, c1 **20/20**, c2 **20/20**, c16 19/20, c4 & c32 18/20, c8 15/20. The bit-identical (0-ulp) fast decode-kernel stack (`348d12d`+`9ecd9d0`) closed **+62 axes** — confirming the decode deficit was norm/quant/act kernel glue. Ratified as "effective parity" via two-grid totality (114+111→115/124; 110 pass-in-both + 5 coin-flip), NOT a clean 124/124. Remaining gate: 10 axes (35B every-axis closure c1/c2 low-conc residual). 35B performance blocked until 27B reaches a clean 124/124. (2026-07-31 audit `CLAIM-ROADMAP-V1-AUDIT`: this row previously lagged the roadmap by two bindings; now synced) | [CUDA inventory](specs/cuda-architecture-inventory.md); [online gate](specs/cuda-online-serving-gate.md); [packed decode](specs/gdn-packed-decode.md); [W3-I](specs/nvfp4-fused-silu-producer.md) | `PARTIAL` | - | +| `BACKEND-GATE-CUDA-VLLM` | CUDA correctness and every-axis parity vs vLLM v0.25.0 | current pin `e24d1b24`; oracle `702f481`; async default `vllm/config/vllm.py:952-1043`; depth-2 engine path `vllm/v1/engine/core.py:519-607`; async output `vllm/v1/worker/gpu/async_utils.py:12-70`; [benchmark protocol](verification.md) | Existing schema-v5 validator/summary remain. The production [packed component driver](../scripts/dgx-gdn-packed-component.sh#L1) and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py#L1) enforce exact corpus/build/oracle provenance, exact and bounded raw-metric validation, duration consistency, exact commands, pairwise stability, one-lock correctness, thermal/memory health, symlink-safe sealing and marker-last provenance | [Component/summary tests](../tests/tools/test_gdn_packed_component.py#L1) pass and the [2026-07-15 new-binding ledger row](parity-ledger.md) records the sealed evidence. **NEW BINDING `9ecd9d0`: 114/124** (full production default set = async + Triton GDN cubin + bit-identical fast RMSNorm + bit-identical fast gated-RMSNorm; supersedes `246a23c`'s 49/124, `3f256ab`'s 55/124, all retained immutable in the ledger). Mem **4/4**, c1 **20/20**, c2 **20/20**, c16 19/20, c4 & c32 18/20, c8 15/20. The bit-identical (0-ulp) fast decode-kernel stack (`348d12d`+`9ecd9d0`) closed **+62 axes** — confirming the decode deficit was norm/quant/act kernel glue. Ratified as "effective parity" via two-grid totality (114+111→115/124; 110 pass-in-both + 5 coin-flip), NOT a clean 124/124. Remaining gate: 10 axes (35B every-axis closure c1/c2 low-conc residual). 35B performance blocked until 27B reaches a clean 124/124. (2026-07-31 audit `CLAIM-ROADMAP-V1-AUDIT`: this row previously lagged the roadmap by two bindings; now synced) | [CUDA inventory](specs/cuda-architecture-inventory.md); [online gate](specs/cuda-online-serving-gate.md); [packed decode](specs/gdn-packed-decode.md); [W3-I](specs/nvfp4-fused-silu-producer.md) | `PARTIAL` | - | | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | SGLang corpus, harness, image and exact-checkpoint/token-ID preflights; P1 CPU harness is implemented/gated while P2 image/model/GPU classification remains | [SGLang v0.5.13](https://github.com/sgl-project/sglang/releases/tag/v0.5.13), commit `28b095c`; digest-pinned CUDA 13 image | [corpus](../tools/bench/make_serve_low_corpus.py#L143); [client/preflights](../tools/bench/run_serve_low.py#L124); [summary](../tools/bench/summarize_serve_low.py#L238); [memory](../tools/bench/sample_process_memory.py#L198); [driver](../scripts/dgx-sglang-low-concurrency.sh#L1) | [16 CPU contract cases](../tests/tools/test_serve_low_client.py#L103); [CTest registration](../tests/CMakeLists.txt#L12); dry-run manifest green; no image/model/GPU evidence or performance result | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `GATING` | - | | `BACKEND-GATE-CUDA-SGLANG` | Binding CUDA low-concurrency serving vs SGLang | same pinned SGLang/checkpoints as `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | [bench main:1](../examples/bench/main.cpp#L1), [server main:1](../examples/server/main.cpp#L1) | no binding run; HTTP TTFT/ITL cannot be measured honestly yet | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `BLOCKED` on `SERVE-ASYNC-LLM` and successful exact-equivalence preflight | - | | `BACKEND-GATE-CUDA-SGLANG-PREFIX` | Binding deterministic shared-prefix cache-on serving vs the faster equivalent vLLM/SGLang floor, separate from cache-neutral serving | SGLang v0.5.15 `f63458b`; digest `d0a667e`; [DGX recipe correction/results](https://github.com/Weschera/qwen-sglang-dgx-spark/tree/03253ef98c01de59a21c85b9a5cc6a27a871c383); vLLM v0.25.0 explicit Qwen hybrid `mamba_cache_mode=align` | current prefix hashes/managers [kv_cache_utils.cpp:259](../src/vllm/v1/core/kv_cache_utils.cpp#L259), [kv_cache_manager.cpp:124](../src/vllm/v1/core/kv_cache_manager.cpp#L124); no local GDN/Mamba aligned-state retention or binding long-prefix harness yet | Source/config audit rejects the original cache-asymmetric 10--40x claim and treats the reported residual 25--45% SGLang lead as unproven: cache-on vLLM is 0.23.1, KV dtypes/memory differ, MTP is enabled, only 35B is measured, and full axes/hits/memory/traces/repetitions are absent. PX1/PX2 implementation, exact 27B equivalence/hit proof and all performance evidence remain pending | [SGLang shared-prefix extension](specs/cuda-sglang-low-concurrency.md#shared-prefix-extension-2026-07-12) | `READY` for PX1 harness/counter work; PX2 begins with the `KV-MAMBA-ALIGN` leaf spike, and binding execution also needs exact v0.5.15 equivalence plus `SERVE-ASYNC-LLM` | - | diff --git a/.agents/backends.md b/.agents/backends.md index 953242b92..2bff50157 100644 --- a/.agents/backends.md +++ b/.agents/backends.md @@ -107,12 +107,12 @@ upstream platform PR ports mechanically. ## Non-negotiables -- The NVIDIA gate (see [gates.md](gates.md)) is not delayed by portability +- The NVIDIA gate (see [verification.md](verification.md)) is not delayed by portability work: M0.2 bakes the interface requirements above (cheap now, unpayable later); backend implementations are post-MVP. - A new backend lands like any port: parity harness (same golden dumps — they are backend-independent), behavioral suites unchanged, benchmark - honesty per [benchmark-protocol.md](benchmark-protocol.md). `DONE` additionally + honesty per [verification.md](verification.md). `DONE` additionally requires match-or-beat against the applicable native floor: llama.cpp for CPU/GGUF and Vulkan, oMLX/MLX-LM for Apple MLX, production vLLM for its native platforms, plus the SGLang low-concurrency CUDA sweep. The canonical status diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 100bd1810..7d83c10b9 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -10628,7 +10628,7 @@ scripts/dgx-online-serving.sh --execute --model 27 \ performance or broader roadmap execution. The complete contract is in the -[benchmark protocol](benchmark-protocol.md) and +[benchmark protocol](verification.md) and [online serving gate spec](specs/cuda-online-serving-gate.md). **Breadth sweep note (2026-07-21):** the active phase is model-architecture breadth (recent-first), each held to token-exact + vLLM-speed on every axis. Ranked queue + CUDA-arch additivity audit: [.agents/specs/breadth-sweep-plan.md](specs/breadth-sweep-plan.md). CUDA archs beyond same-family sm_120 are HW-blocked (only GB10 testable). diff --git a/.agents/ai-coding-assistants.md b/.agents/completed/ai-coding-assistants-legacy.md similarity index 96% rename from .agents/ai-coding-assistants.md rename to .agents/completed/ai-coding-assistants-legacy.md index 7dea8a44a..d00e17d85 100644 --- a/.agents/ai-coding-assistants.md +++ b/.agents/completed/ai-coding-assistants-legacy.md @@ -38,7 +38,7 @@ an unreviewed change. If you cannot explain and defend a line, do not submit it. Every commit MUST carry the trailer `FOLLOWING_AGENTS_PROTOCOL`. This is a deliberate speed-bump: it asserts the contributor has **read -[AGENTS.md](../AGENTS.md)** and follows this project's protocol (mirror vLLM, +[AGENTS.md](../../AGENTS.md)** and follows this project's protocol (mirror vLLM, ground every check in vLLM source, compare vs the vLLM oracle on the identical workload, keep the parity ledger updated, etc.). CI (`commit-protocol-tag` in `.github/workflows/ci.yml`) rejects any commit lacking it, so a contributor who diff --git a/.agents/benchmark-protocol.md b/.agents/completed/benchmark-protocol-legacy.md similarity index 94% rename from .agents/benchmark-protocol.md rename to .agents/completed/benchmark-protocol-legacy.md index f62ad3d19..4d76e9540 100644 --- a/.agents/benchmark-protocol.md +++ b/.agents/completed/benchmark-protocol-legacy.md @@ -27,7 +27,7 @@ it becomes binding once the spike proves the workload/features are equivalent. output lengths, sampling, cache/prefix state, concurrency and serving features. If conversion is necessary, record it and do not turn the converted result into a binding floor until correctness/quality equivalence is established. Full -matrix: [specs/competitive-benchmarks.md](specs/competitive-benchmarks.md). +matrix: [specs/competitive-benchmarks.md](../specs/competitive-benchmarks.md). **Cache policy is part of the workload, never an incidental default.** CUDA serving has at least two independent gates: a cache-neutral/cache-off workload @@ -87,8 +87,8 @@ measured separately. Missing hardware leaves the row in `GATING` with an exact handoff recipe; it never converts an unmeasured milestone into `DONE`. Every feature/iteration checkpoint also updates the public -[`README.md`](../README.md) stage and -[`docs/BENCHMARKS.md`](../docs/BENCHMARKS.md) disposition **in the same +[`README.md`](../../README.md) stage and +[`docs/BENCHMARKS.md`](../../docs/BENCHMARKS.md) disposition **in the same change**. This includes an attempted benchmark that fails closed: record it as `FAILED` or `VOID`, name why no ratio is binding, and retain the exact next reproduction command. A CPU-only/non-performance feature records @@ -110,10 +110,10 @@ checkpoint. serving, include the low-concurrency sweep when latency/scheduling could change. For new backends, the area spike fixes representative models that fit the hardware without pretending a small-model gate proves 27B/35B scale. -5. **Record every axis + every ratio** in [parity-ledger.md](parity-ledger.md), +5. **Record every axis + every ratio** in [parity-ledger.md](../parity-ledger.md), then refresh the concise accepted/pending/failed/void summary in - [`docs/BENCHMARKS.md`](../docs/BENCHMARKS.md) and the current stage in - [`README.md`](../README.md). A ledger row that leaves ANY axis below an + [`docs/BENCHMARKS.md`](../../docs/BENCHMARKS.md) and the current stage in + [`README.md`](../../README.md). A ledger row that leaves ANY axis below an applicable floor is an open gap. ## Reproduction is a GATE @@ -162,7 +162,7 @@ rumor; a reproducible run is evidence. server files may instead be evicted with `POSIX_FADV_DONTNEED` only when a retained report hashes that inventory and `mincore(2)` proves **zero resident pages** afterward; an unverified/best-effort advisory call does not qualify. - See [specs/competitive-benchmarks.md](specs/competitive-benchmarks.md) + See [specs/competitive-benchmarks.md](../specs/competitive-benchmarks.md) § "Folded: spark-bench". If a result cannot be reproduced on demand under these rules, it does not count @@ -181,5 +181,5 @@ toward the gate. every axis, both models**. Post-MVP backend rows additionally remain open while below their applicable native floor. -See also: [gates.md](gates.md) (the MVP gates), [parity-lever-protocol.md](parity-lever-protocol.md) +See also: [gates.md](mvp-gates-legacy.md) (the MVP gates), [parity-lever-protocol.md](../parity-lever-protocol.md) (how to find the levers to close a below-vLLM axis). diff --git a/.agents/gates.md b/.agents/completed/mvp-gates-legacy.md similarity index 97% rename from .agents/gates.md rename to .agents/completed/mvp-gates-legacy.md index bd0e91621..da5bd4ffc 100644 --- a/.agents/gates.md +++ b/.agents/completed/mvp-gates-legacy.md @@ -11,7 +11,7 @@ 1. **Throughput parity vs vLLM and every equivalent faster floor** on the release-gate environment (`${GATE_HOST}`; currently the DGX Spark GB10 - profile in [environment.md](environment.md)): serve + profile in [environment.md](../environment.md)): serve **Qwen3.6-35B-A3B (NVFP4)** and **Qwen3.6-27B (NVFP4)** with prefill AND decode throughput matching vLLM at large concurrency (request-rate sweeps, measured with our `bench serve` equivalent, same box, same models). @@ -50,7 +50,7 @@ vLLM is the parity oracle for BOTH correctness and performance. Every change that could affect either MUST be compared against vLLM, apples-to-apples, and -BOTH numbers recorded in the [parity ledger](parity-ledger.md): +BOTH numbers recorded in the [parity ledger](../parity-ledger.md): - **Correctness:** op dumps + model logits/greedy vs the pinned pip-vLLM oracle (`${VLLM_ORACLE}`, forward-math-identical to the pin). A new diff --git a/.agents/specs/operator-helper-protocol.md b/.agents/completed/operator-helper-protocol-legacy.md similarity index 100% rename from .agents/specs/operator-helper-protocol.md rename to .agents/completed/operator-helper-protocol-legacy.md diff --git a/.agents/directives.md b/.agents/completed/policy-directives-legacy.md similarity index 97% rename from .agents/directives.md rename to .agents/completed/policy-directives-legacy.md index 111a286fe..cc33b9ce5 100644 --- a/.agents/directives.md +++ b/.agents/completed/policy-directives-legacy.md @@ -28,7 +28,7 @@ library / C ABI, as the CLI, and as the OpenAI server, with a runnable example each). Every argument, flag, and endpoint documented MUST be grounded in actual source, never invented. -**Keep [`docs/STATUS.md`](../docs/STATUS.md) CURRENT at EVERY feature/iteration +**Keep [`docs/STATUS.md`](../../docs/STATUS.md) CURRENT at EVERY feature/iteration checkpoint. THAT is the per-capability status surface, NOT `README.md`.** In the SAME change that shifts a feature's lifecycle state, update its ONE binding current-state line in the matching `docs/STATUS.md` section or table (typically @@ -60,7 +60,7 @@ user-facing sections exist, that `docs/STATUS.md` exists and carries the capability ledger, and that the README stays inside its length, paragraph, and table-cell budgets; do not weaken the checker to bypass the obligation. -**Keep [`docs/BENCHMARKS.md`](../docs/BENCHMARKS.md) CURRENT at the SAME +**Keep [`docs/BENCHMARKS.md`](../../docs/BENCHMARKS.md) CURRENT at the SAME checkpoint.** Every feature/iteration records its benchmark disposition there in the same change: accepted numbers with exact workload/reference/evidence, or an explicit `PENDING`, `NOT APPLICABLE`, `FAILED`, or `VOID` reason and the @@ -77,7 +77,7 @@ not a style preference: that is exactly how the page reached 11,405 lines and 171 claim-titled sections and stopped being readable by users (converted 2026-08-04). Per-attempt narrative, refuted hypotheses, profiler tables and superseded numbers go to the append-only -[`.agents/benchmark-record.md`](benchmark-record.md) **in the same +[`.agents/benchmark-record.md`](../benchmark-record.md) **in the same change** (move it, never delete evidence). If sections have accumulated anyway, `scripts/roll-benchmark-record.py --apply` moves every non-canonical section into the record verbatim. `scripts/check-public-doc-tables.py` (CI-gated, with @@ -87,7 +87,7 @@ the house no-em-dash rule, and that the page still points at the record; do not weaken the checker to bypass the obligation. When a DENOMINATOR turns out to be wrong, correct every ratio built on it, never keep the flattering one. -**Keep [`docs/FEATURES.md`](../docs/FEATURES.md) CURRENT — same-change +**Keep [`docs/FEATURES.md`](../../docs/FEATURES.md) CURRENT — same-change obligation.** It is the PUBLIC feature surface: what we support, next to vLLM, SGLang and llama.cpp. It is a KEYED TABLE under the same rules as `docs/BENCHMARKS.md` above (one row per feature, updated in place, no appended @@ -130,7 +130,7 @@ target — a new `VLLM_CPP_CUDA_ARCHITECTURES` value or a new configure-time pipeline (`-DVLLM_CPP_TRITON=ON -DVLLM_CPP_TRITON_REGEN=ON -DVLLM_CPP_TRITON_VENDORED_ARCH=sm_XX`; cross-compiled, no target board needed; 57 artifacts + MANIFEST matching the `sm_121a` fileset — see -[.agents/specs/triton-aot-per-arch.md](specs/triton-aot-per-arch.md) and +[.agents/specs/triton-aot-per-arch.md](../specs/triton-aot-per-arch.md) and `cmake/TritonAOT.cmake`), with `scripts/check-triton-aot-drift.sh` rc=0; or (b) if regen is not yet possible on that arch, record the GDN-decode gap HONESTLY in `backend-matrix.md`/`kernel-matrix.md` (that arch runs the hand @@ -252,7 +252,7 @@ norm+quant, MoE, sm_121 "blackwell_sm12x" kernels), **cutlass**, **cuBLASLt** (nvjet), **DeepGEMM**, and **torch/Inductor** (the fused Triton it codegens). Read the actual pinned vLLM code (`${VLLM_SOURCE}` @ pin `555967922`, vLLM 0.26.0.dev0 — advanced 2026-07-26 from the prior `e24d1b24`/0.25.0 pin; see -[.agents/specs/pin-advance.md](specs/pin-advance.md)) AND, as +[.agents/specs/pin-advance.md](../specs/pin-advance.md)) AND, as needed, the installed dependency source (`${DEPENDENCY_SOURCE}`, for example `flashinfer/cute_dsl/*.py` and `flashinfer/gemm/`), cite `file:line` on every side, and mirror what you find. **NEVER declare a lever "build-specific", @@ -386,7 +386,7 @@ child kernels. Verify the export has graph-node kernel rows whenever graph launc occur; a whole-graph-only report is attribution-incomplete and cannot select a lever.** This applies to every subagent and every parity check. Full method: -[.agents/parity-lever-protocol.md](parity-lever-protocol.md) § Verify the +[.agents/parity-lever-protocol.md](../parity-lever-protocol.md) § Verify the whole chain. ## STANDING DIRECTIVE — port the TESTS with the code (upstream tests = the spec) @@ -399,7 +399,7 @@ cases, with the upstream test file cited in the header. Specs test that can't pass yet is checked in SKIPPED with a tracked reason, never dropped. This ground-rules our work against what vLLM actually guarantees and turns the suite into the regression net. Full protocol: -[.agents/test-porting.md](test-porting.md). +[.agents/test-porting.md](test-porting-legacy.md). ## STANDING DIRECTIVE — always compare vs vLLM (the oracle), same workload @@ -408,7 +408,7 @@ apples-to-apples against vLLM and both numbers + the ratio recorded in the ledger: **correctness** vs the pinned pip-vLLM oracle (`${VLLM_ORACLE}`), **performance** vs `vllm bench throughput` on the *identical* workload. Never re-base the bench config without re-running vLLM on it. This is non-negotiable -and applies to subagents. Full rule: [.agents/gates.md](gates.md) +and applies to subagents. Full rule: [.agents/gates.md](mvp-gates-legacy.md) § PROTOCOL DIRECTIVE. **Acceptance rule — match or beat vLLM on EVERY axis, never below.** Benchmark @@ -421,7 +421,7 @@ any axis = an open gap, not a done change; "near parity" is NOT met. seed, build, vLLM oracle cmd), re-run ≥2–3× to confirm within run-noise, use a same-binary A/B, and run only on an idle box (contended runs are void). A number that doesn't reproduce does not count. Full protocol: -[.agents/benchmark-protocol.md](benchmark-protocol.md). +[.agents/benchmark-protocol.md](benchmark-protocol-legacy.md). **Additional competitor floor — equivalent SGLang binds wherever it is faster.** vLLM remains mandatory on every workload, but an exact SGLang arm is @@ -460,7 +460,7 @@ unchanged. Each participating agent reads BOTH the pinned vLLM `file:line` on both sides, and reports what vLLM does DIFFERENTLY that makes it faster. Then verify each diff adversarially (real? on the gate hot path?), rank by gain÷effort, drive the top lever, re-measure vs vLLM, repeat. Full protocol: -[.agents/parity-lever-protocol.md](parity-lever-protocol.md). Caveat: a +[.agents/parity-lever-protocol.md](../parity-lever-protocol.md). Caveat: a real per-op comparison needs a CLEAN slice of OURS (not inferred proportions). A vLLM edge that is an Inductor/DeepGEMM/flashinfer FUSION is NOT a ceiling: only *eager* op-by-op dispatch can't fuse, and our engine is not limited to eager — we @@ -504,5 +504,4 @@ So: same-kernel-different-throughput is a real RED FLAG worth the structural sca declaring a "ceiling" prematurely — but confirm the anomaly PER-SHAPE (never on an aggregate derived metric), and verify the reference's rationale actually applies to YOUR measured situation before you build the fix. Full method: -[.agents/parity-lever-protocol.md](parity-lever-protocol.md) § The STRUCTURAL lens. - +[.agents/parity-lever-protocol.md](../parity-lever-protocol.md) § The STRUCTURAL lens. diff --git a/.agents/discipline.md b/.agents/completed/porting-discipline-legacy.md similarity index 99% rename from .agents/discipline.md rename to .agents/completed/porting-discipline-legacy.md index 1f8e132f1..397d390c6 100644 --- a/.agents/discipline.md +++ b/.agents/completed/porting-discipline-legacy.md @@ -91,4 +91,4 @@ vs logits/greedy-decode, server vs OpenAI conformance. Upstream vLLM (Python) is a test-time oracle only — never a runtime dependency. - **We port Model Runner V2** (`vllm/v1/worker/gpu/`), not the legacy runner — - see [vllm-v1-v2.md](vllm-v1-v2.md). + see [vllm-v1-v2.md](../vllm-v1-v2.md). diff --git a/.agents/completed/roadmap_mvp_v0.md b/.agents/completed/roadmap_mvp_v0.md index 903f15917..95ef65ed2 100644 --- a/.agents/completed/roadmap_mvp_v0.md +++ b/.agents/completed/roadmap_mvp_v0.md @@ -9,7 +9,7 @@ Milestones decompose into **workable units** (one unit ≈ one focused PR an agent can pick up, implement, test, and ledger in a session or few). Order within a milestone is roughly dependency order. Every unit's Definition of Done (DoD) includes: mirrored-structure sources with upstream-commit headers, -tests per [discipline.md](../discipline.md), inventory markers flipped, ledger +tests per [discipline.md](porting-discipline-legacy.md), inventory markers flipped, ledger row appended, state log updated. Status: ☐ open · 🚧 in progress · ✅ done. Keep this current. diff --git a/.agents/completed/roadmap_v1_inventory_spikes_2026-07-10.md b/.agents/completed/roadmap_v1_inventory_spikes_2026-07-10.md index 4b9203a72..a6824e83c 100644 --- a/.agents/completed/roadmap_v1_inventory_spikes_2026-07-10.md +++ b/.agents/completed/roadmap_v1_inventory_spikes_2026-07-10.md @@ -12,7 +12,7 @@ finished coverage work and its handoff. | `INV-QUANT` | 76 rows across GGUF/llama.cpp, vLLM-native NVIDIA formats and MLX-native formats; recognition, materialization, quant compute, e2e and performance tracked separately | [quantization matrix](../quantization-matrix.md), [coverage spike](../specs/quantization-coverage.md) | spike GGUF compute-in-quant storage and dispatch | | `INV-KERNEL` | 30 practical families grounded through vLLM and its execution dependencies | [kernel matrix](../kernel-matrix.md), [family spike](../specs/kernel-family-inventory.md) | spike raw-pointer/shape/stride/stream adapter ABI | | `INV-BACKEND` | 13 CUDA targets, 18 component target rules, 8 platform/ABI rows and 9 native competitor gates | [backend matrix](../backend-matrix.md), [CUDA/backend spike](../specs/cuda-architecture-inventory.md) | spike the common architecture spine, then parallel target ports | -| `INV-COMPETITORS` | vLLM remains universal CUDA oracle; SGLang low-concurrency, llama.cpp CPU/Vulkan/Metal, and oMLX/MLX-LM Apple floors added | [benchmark spike](../specs/competitive-benchmarks.md), [benchmark protocol](../benchmark-protocol.md) | run the isolated DGX SGLang series after the active PR #3 GPU claim | +| `INV-COMPETITORS` | vLLM remains universal CUDA oracle; SGLang low-concurrency, llama.cpp CPU/Vulkan/Metal, and oMLX/MLX-LM Apple floors added | [benchmark spike](../specs/competitive-benchmarks.md), [benchmark protocol](benchmark-protocol-legacy.md) | run the isolated DGX SGLang series after the active PR #3 GPU claim | | `INV-FEATURE-ANCHORS` | legacy cross-cutting feature surface migrated to stable engine/serving rows; 26 code-bearing claims narrowed to their evidenced slices | [engine matrix](../engine-matrix.md), [anchor-backfill spike](../specs/feature-anchor-backfill.md) | agents write the missing leaf spikes before implementation or `DONE` claims | ## Corrections made by the coverage pass diff --git a/.agents/test-porting.md b/.agents/completed/test-porting-legacy.md similarity index 95% rename from .agents/test-porting.md rename to .agents/completed/test-porting-legacy.md index d52c5fc05..b75ecddee 100644 --- a/.agents/test-porting.md +++ b/.agents/completed/test-porting-legacy.md @@ -28,9 +28,9 @@ So test porting is part of the mirror obligation, not an optional extra. 3. **Traceability is 1:1 and named**: keep our test case names derived from upstream ones (`test_schedule_spec_decode` → `TEST_CASE("schedule_spec_decode …")`) and record the upstream→ours test-file mapping in - [porting-inventory.md](porting-inventory.md) alongside the code mapping, so + [porting-inventory.md](../porting-inventory.md) alongside the code mapping, so the upstream sync cycle can diff TEST deltas mechanically. -4. **Upstream sync ports test deltas too** ([upstream-sync.md](upstream-sync.md)): +4. **Upstream sync ports test deltas too** ([upstream-sync.md](../upstream-sync.md)): when a synced vLLM PR touches `tests/`, the port includes the test delta. A PR that only changes tests still gets synced — it's a spec change. 5. **Specs must inventory their tests**: every `.agents/specs/.md` diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 8fceb8e7f..2e1559b66 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -300,9 +300,9 @@ Columns: | 2026-07-10 (Triton-AOT follow-up: bf16 `chunk_o` source path + per-stream GDN scratch pools; GPU validation pending) | Adds the missing bf16-output AOT `chunk_o` specs (`gdn_chunko_bf16_h48/h32`) and source-wires `TryTritonChunkO` for f32 output plus bf16 output when `VLLM_CPP_TRITON_CHUNKO_BF16` is compiled. The bf16 path is deliberately artifact-gated so today's vendored Triton build keeps working until regenerated artifacts are added. Adds per-stream grow-only pools for GDN chunk scratch/metadata (`gcum/u/w/v_new/hstate/tok0/len/boh/cidx`) and the Triton WU A/Ai intermediates, with `VT_GDN_TRITON_CHUNK_POOL=0` and `VT_GDN_TRITON_WU_POOL=0` as same-binary A/B escapes. Integrates with main's vendored AOT workflow (`VLLM_CPP_TRITON_REGEN=ON` refreshes `src/vt/cuda/triton_aot_vendored//`; normal builds consume vendored artifacts without Python). README status rows updated to reflect the post-MVP throughput gate; the GGUF full-engine acceptance golden is routed out of generic op parity and owned by `test_qwen36_gguf_engine`. | FLA/vLLM `chunk_o.py` stores the recurrence output in bf16 when the model dtype is bf16; vLLM gets allocation reuse through PyTorch's caching allocator rather than raw per-layer `cudaMallocAsync` churn. Vendored AOT artifacts are a vllm.cpp build-system deviation, preserving the same generated CUDA driver ABI. | **Verified locally:** `git diff --check`; `cmake -S . -B build-cpu -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_SERVER=OFF`; `cmake --build build-cpu -j$(nproc)`; `ctest --test-dir build-cpu --output-on-failure` = 90/90. **GPU validation pending:** this environment did not provide a CUDA-capable build/run path, so no throughput claim and no default flip for `VT_GDN_OUT_BF16`. Next CUDA pass must regenerate/check in the bf16 `chunk_o` AOT artifacts, run `VLLM_CPP_TRITON=ON` `test_ops_gdn`, and do same-binary A/B for pools + bf16 output vs fresh vLLM denominators. | | 2026-07-10 (B4 CPU DECISION MEASUREMENT — llama.cpp CPU vs our CPU path, same GGUF/same box: llama.cpp decode **54–75×** faster, prefill **≈1,480×** faster, peak RSS **2.7×** smaller; the roadmap "vendor IF proven faster" criterion is MET, `QUANT-GGUF-COMPUTE` threadpool→compute-in-quant confirmed as the structural route) | ROAD-V1-C4 / `BENCH-CPU-LLAMA` evidence: same-file CPU A/B on `Qwen3.5-2B-UD-Q8_K_XL.gguf` (`qwen35` dense, 1.94B, 2.68 GiB) on the idle x86 box (AMD Ryzen 9 9950X3D VM, 20 cores / 1 thread-per-core, 84 GB). Arms: (a) llama.cpp local fork `237ad9b96` (b9892 — the `QUANT-GGUF-COMPUTE` upstream pin), CPU build, 20 threads: `llama-bench -m -p 512,128 -n 128,32` (r=5) plus isolated `-p 0 -n 32 -r 2` under `/usr/bin/time -v`; (b) ours at `7c91a42` (branch `bench/quant-gguf-compute-b4-cpu-floor` = `83010c7` + measurement-enabling dense-`qwen35` GGUF loader path, gguf ctest 4/4 green): `vllm-bench --model --num-prompts 1 --input-len 128 --output-len 32 --concurrency 1` plus in16/out8 smoke ×3, greedy seed 0. Our CPU path is the known floor: single-threaded scalar ops (`src/vt/cpu/cpu_ops.cpp`, zero threading — 99–100% of one core observed for the whole run) over load-time bf16 expansion (`gguf_dequant.cpp` `DequantGgufRowToBf16`); llama.cpp runs 20-thread vectorized compute-in-quant. | llama.cpp `ggml/src/ggml*.c` tensor traits + `tools/llama-bench` at `237ad9b96` (own measurement per benchmark-protocol repro rules; llama.cpp is the CPU-GGUF perf oracle per specs/quantization-coverage.md §Gates and specs/competitive-benchmarks.md `BENCH-CPU-LLAMA`) | **llama.cpp: pp512 372.65±39.94 t/s, pp128 339.63±17.16, tg128 9.28±0.32, tg32 9.19±2.15 (combined run) and tg32 12.75±0.32 isolated; peak RSS 2.80 GiB. Ours: prefill 128 tok TTFT 559,724 ms = 0.229 t/s; decode TPOT 5,848 ms = 0.171 t/s; smoke ×3 TTFT 76,038/75,083/74,666 ms, TPOT 5,845/5,734/5,704 ms (spread <2% — reproduces); peak RSS 7.43 GiB. Ratios at the matched pp128/tg32 point: decode 54–75×, prefill ≈1,480×, memory 2.65× — all three axes decisively behind. DECISION: B4 "vendor IF proven faster" criterion MET — port/vendor llama.cpp's CPU compute path per the pinned upstream: threadpool first (~N_core× on every op, ~1 wk), then compute-in-quant ggml tensor-traits GEMM (research estimate ~3.3× decode / ~10× prefill on top); keep dequant-to-bf16 as the `VT_CPU_REF` parity oracle (compute-in-quant is not bit-identical). No lifecycle state shifts: `QUANT-GGUF-COMPUTE` and `BACKEND-GATE-CPU-LLAMACPP` stay `INVENTORIED` pending claim-sized leaf specs — this row is their measured perf-floor evidence. Caveat: single-request bench (concurrency 1) on one x86 box; the GB10 Arm-core rerun and a server-endpoint series belong to the gate row once the compute path lands.** | | 2026-07-10 (`ENG-CORE-BUSY-LOOP` W1: EngineCoreProc busy loop + input/output queue split + in-proc client — async-serving block leaf 1) | New `EngineCoreProc : EngineCore` (`include/vllm/v1/engine/core_proc.h` + `src/vllm/v1/engine/core_proc.cpp`): mutex+condvar `BlockingQueue` input/output queues with queue.Queue blocking-get/put_nowait semantics, `run_busy_loop` (`_process_input_queue` block-for-work + drain, `_process_engine_step` step_fn→output_queue), `EngineShutdownState` with abort-mode (finish-all-as-ABORTED + abort outputs) and drain-mode shutdown, ADD-rejected-during-shutdown, WAKEUP + ENGINE_CORE_DEAD sentinels, `EngineCoreRequestType` values kept; step_fn selection mirrored with `max_concurrent_batches > 1` rejected until `step_with_batch_queue` (W3 `ENG-ASYNC-SCHED`). New `InprocClient` (`core_client.{h,cpp}`): owns the proc + the dedicated engine `std::thread` under the run_engine_core fatal-error guard; blocking `get_output` raising `EngineDeadError` on the dead sentinel; `add_request_async`/`abort_requests_async` (empty/dead-guarded). Sync `LLMEngine` path untouched (additive files; only `EngineCore` members private→protected for the upstream subclass shape). Deferred per spec: UTILITY/START_DP_WAVE, DP waves, aborts_queue, idle-state callbacks, post_step, multi-client index. | `vllm/v1/engine/core.py:915-916,1259-1480` + `vllm/v1/engine/core_client.py:779-893`, exceptions.py:9 @ e24d1b24 (recorded deviation D2, async-serving spec: in-proc std::thread + direct queue sharing instead of forked process + ZMQ; client API shapes kept so a multiproc client is a drop-in) | **G2: clean full CPU build zero warnings; ctest 93/93 incl. new `test_engine_core_proc` (9 cases / 82 asserts — normal cycle, abort cycle, abort-after-finish ported from upstream `tests/v1/engine/test_engine_core_client.py:266,523`; WAKEUP idle shutdown, abort-mode + drain-mode shutdown, shutdown ADD-reject, EngineDeadError, batch-queue reject); 25/25 stability reruns of the threaded suite. G1 (both greedy engine gates re-run) + G4 (offline throughput A/B vs pre-change binary) DEFERRED to the gating handoff: GPU held by the `SERVE-GATE-ONLINE` campaign (flock holder + queued waiter verified) — W1 is additive host-side code not wired into the sync path, but the gates run before the row may claim DONE.** | -| 2026-07-10 (`ENG-PRIORITY-SCHED` W4: priority request queue + policy + priority preemption + `priority` plumbing — async-serving block leaf 4) | New `PriorityRequestQueue` (`src/vllm/v1/core/sched/request_queue.cpp:101` + header): a binary heap over `Request*` ordered by `RequestPriorityLess` (`Request.__lt__`: (priority, arrival_time, request_id, identity)) built with the std::*_heap algorithms; `create_request_queue(kPriority)` no longer throws (`request_queue.cpp:186`). Scheduler priority preemption (`scheduler.cpp:178`): when the policy is priority, the OOM victim is `max(running, key=(priority, arrival_time))` with the scheduled-this-step undo (restore token_budget, drop block reservation, `req_index -= 1`), mirroring `scheduler.py:546-572`; FCFS tail-pop unchanged and stays the default. `Request.priority` field + `RequestPriorityLess` (`request.h`/`request.cpp:92`), `EngineCoreRequest.priority`, and the OpenAI `priority` request field are plumbed end to end (protocol → serving → `LLMEngine::add_request`/`generate` → `InputProcessor::process_inputs` → `EngineCoreRequest` → `Request`); `SchedulerPolicyFromString` (`config/scheduler.cpp:21`) rejects unknown policy strings (upstream `SchedulingPolicy(value)` ValueError), and `EngineParams.policy`/`MakeSchedulerConfig` expose the policy config surface. Default behavior is byte-identical FCFS. | `vllm/v1/core/sched/request_queue.py:131-208` (PriorityRequestQueue, create_request_queue); `vllm/v1/core/sched/scheduler.py:546-572` (priority preemption victim); `vllm/v1/request.py:309-320` (`__lt__`), `:83` (priority); `vllm/config/scheduler.py:109-117,175-178` (policy field + SchedulingPolicy validation) @ e24d1b24 | **G2: clean full CPU build zero warnings; ctest 93/93. Ported tests (test-porting.md): 12 priority-scheduler cases in `tests/vllm/v1/test_scheduler.cpp:674` — basic/arrival/mixed ordering, preempts-lowest-priority-running, no-preempt-when-space, victim-selection, equal-priority tiebreak, waiting-queue-order, fcfs-fallback, limited-slots, ascending-priority (heap_property), and preemption-then-resumption-out-of-KV (V2 no-connector variant) — from `tests/v1/core/test_scheduler.py:2382-2856,2978`; 14 priority-queue cases + the seeded random ordering/heap-property property test in `tests/vllm/v1/test_request_queue.cpp:238,429` — from `test_priority_scheduler_random.py`. DEVIATION recorded: our M1.3 KVCacheManager forces caching ON (enable_caching=false deferred), so the ported block-math cases give each request a DISTINCT prompt to keep caching-ON behaviorally equal to upstream's caching-OFF (they test priority order/victim selection, not the prefix cache); the EC/KV-connector `test_scheduler.py:3769` variant is not ported (no connectors). G1 (both greedy engine gates re-run priority-vs-fcfs token-exactness) DEFERRED to the gating handoff: GPU held by the `SERVE-GATE-ONLINE` campaign — the priority policy is not the default so the greedy gates run FCFS unchanged, but the priority-vs-fcfs token-exact A/B runs before the row may claim DONE.** | +| 2026-07-10 (`ENG-PRIORITY-SCHED` W4: priority request queue + policy + priority preemption + `priority` plumbing — async-serving block leaf 4) | New `PriorityRequestQueue` (`src/vllm/v1/core/sched/request_queue.cpp:101` + header): a binary heap over `Request*` ordered by `RequestPriorityLess` (`Request.__lt__`: (priority, arrival_time, request_id, identity)) built with the std::*_heap algorithms; `create_request_queue(kPriority)` no longer throws (`request_queue.cpp:186`). Scheduler priority preemption (`scheduler.cpp:178`): when the policy is priority, the OOM victim is `max(running, key=(priority, arrival_time))` with the scheduled-this-step undo (restore token_budget, drop block reservation, `req_index -= 1`), mirroring `scheduler.py:546-572`; FCFS tail-pop unchanged and stays the default. `Request.priority` field + `RequestPriorityLess` (`request.h`/`request.cpp:92`), `EngineCoreRequest.priority`, and the OpenAI `priority` request field are plumbed end to end (protocol → serving → `LLMEngine::add_request`/`generate` → `InputProcessor::process_inputs` → `EngineCoreRequest` → `Request`); `SchedulerPolicyFromString` (`config/scheduler.cpp:21`) rejects unknown policy strings (upstream `SchedulingPolicy(value)` ValueError), and `EngineParams.policy`/`MakeSchedulerConfig` expose the policy config surface. Default behavior is byte-identical FCFS. | `vllm/v1/core/sched/request_queue.py:131-208` (PriorityRequestQueue, create_request_queue); `vllm/v1/core/sched/scheduler.py:546-572` (priority preemption victim); `vllm/v1/request.py:309-320` (`__lt__`), `:83` (priority); `vllm/config/scheduler.py:109-117,175-178` (policy field + SchedulingPolicy validation) @ e24d1b24 | **G2: clean full CPU build zero warnings; ctest 93/93. Ported tests (porting.md): 12 priority-scheduler cases in `tests/vllm/v1/test_scheduler.cpp:674` — basic/arrival/mixed ordering, preempts-lowest-priority-running, no-preempt-when-space, victim-selection, equal-priority tiebreak, waiting-queue-order, fcfs-fallback, limited-slots, ascending-priority (heap_property), and preemption-then-resumption-out-of-KV (V2 no-connector variant) — from `tests/v1/core/test_scheduler.py:2382-2856,2978`; 14 priority-queue cases + the seeded random ordering/heap-property property test in `tests/vllm/v1/test_request_queue.cpp:238,429` — from `test_priority_scheduler_random.py`. DEVIATION recorded: our M1.3 KVCacheManager forces caching ON (enable_caching=false deferred), so the ported block-math cases give each request a DISTINCT prompt to keep caching-ON behaviorally equal to upstream's caching-OFF (they test priority order/victim selection, not the prefix cache); the EC/KV-connector `test_scheduler.py:3769` variant is not ported (no connectors). G1 (both greedy engine gates re-run priority-vs-fcfs token-exactness) DEFERRED to the gating handoff: GPU held by the `SERVE-GATE-ONLINE` campaign — the priority policy is not the default so the greedy gates run FCFS unchanged, but the priority-vs-fcfs token-exact A/B runs before the row may claim DONE.** | | 2026-07-10 (`MODEL-FACTORY-registry` spike accepted after C2 handoff recovery) | Claim-sized contract for replacing the live `num_experts==0` model dispatch with an ordered architecture-to-type-erased-factory registry, registering both existing Qwen3.5 paths through it, mirroring vLLM's reject-unknown branches and capability metadata, and preserving the full two-model token/performance gate. The recovered draft was corrected against the pin: 32 previously-supported entries (not 36), subset-registry formatting for the default supported-list message, a central ordered table rather than cross-TU static registration, and the include/src header split. | Pinned vLLM `model_executor/models/registry.py:701-796,998-1082,1166-1190,1244-1296,1396-1404` plus `tests/models/test_registry.py` and `test_initialization.py`; no runtime behavior changes in this docs-only spike. | Nine-section spike contract and exact local/upstream anchors reviewed; record checker and 13-test mutation suite pass. Implementation remains `READY`, not supported/DONE. | -| 2026-07-10 (user-directed per-feature performance checkpoints) | Clarify the already-mandatory performance protocol: every feature or milestone that can affect speed/latency/memory is benchmarked as its own checkpoint before another speed-sensitive change is stacked. Each checkpoint requires an attributable same-binary pre/post A/B, a fresh same-box applicable floor, correctness first, all axes, 2–3 uncontended reproductions, and exact ledger commands; unavailable hardware leaves the row `GATING`. | Process-only clarification; vLLM remains the CUDA floor, same-file llama.cpp the CPU+GGUF floor, and backend-native floors remain as tabulated in `benchmark-protocol.md`. | `workflow.md` Definition of Done and `benchmark-protocol.md` acceptance rules now state the checkpoint/no-stacking invariant explicitly; no runtime behavior or support state changed. | +| 2026-07-10 (user-directed per-feature performance checkpoints) | Clarify the already-mandatory performance protocol: every feature or milestone that can affect speed/latency/memory is benchmarked as its own checkpoint before another speed-sensitive change is stacked. Each checkpoint requires an attributable same-binary pre/post A/B, a fresh same-box applicable floor, correctness first, all axes, 2–3 uncontended reproductions, and exact ledger commands; unavailable hardware leaves the row `GATING`. | Process-only clarification; vLLM remains the CUDA floor, same-file llama.cpp the CPU+GGUF floor, and backend-native floors remain as tabulated in `verification.md`. | `workflow.md` Definition of Done and `verification.md` acceptance rules now state the checkpoint/no-stacking invariant explicitly; no runtime behavior or support state changed. | | 2026-07-10 (`ENG-EXPERT-STREAM` spike grounding repair) | Recovered the stopped workflow writer and rewrote the expert-streaming implementation contract against the live 35B Marlin/loader path. Capacity/NVMe math remains, but implementation now requires a bank-only loader (no 16.88 GiB host expert copies), versioned pre-repacked bank with layer-wide scale factors, fixed contiguous C-slot Marlin arrays, device-router-ID readback, logical→slot rewrite before `moe_align`, and exact chunk sweeps for C