Skip to content

docs(release-notes): behaviour → behavior (Argus smoke test)#4816

Merged
bokelley merged 3 commits into
mainfrom
bokelley/argus-smoke-test
May 19, 2026
Merged

docs(release-notes): behaviour → behavior (Argus smoke test)#4816
bokelley merged 3 commits into
mainfrom
bokelley/argus-smoke-test

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

Two instances of British spelling "behaviour" in docs/reference/release-notes.mdx. Repo convention is US spelling (158 "behavior" vs 2 "behaviour" before this fix).

Why this PR exists

This PR is the smoke test for #3488 — the new Argus AI review workflow. The PR is intentionally tiny and benign so we can verify the end-to-end pipeline (App-token mint → prompt build → Argus LLM run → `gh pr review` post → verifier confirm) on a docs-only change before exercising any MUST-FIX gates.

Expected outcome: Argus reviews this PR, classifies as docs-only (no `code-reviewer` delegation needed), and posts `--approve`.

Test plan

  • CI green
  • Argus posts a `--approve` review within 10 minutes of the PR opening
  • Argus skips subagent delegation (docs-only PR)
  • Review body uses bokelley voice (concise, declarative, sign-off ladder)

bokelley and others added 3 commits May 19, 2026 14:07
The Claude Code triage routine opens PRs as @bokelley (the only
GitHub identity available in the Anthropic routine console), which
means the project owner cannot approve their own PRs and branch
protection forces admin-merge for everything the routine produces.

Add a separate auto-approver workflow that uses the AAO Triage Bot
GitHub App to post approving reviews on PRs that are:

  - Routine-authored (head branch starts with `claude/` or `auto/`)
  - Not draft
  - All CI checks green (success / neutral / skipped only)
  - No `do-not-auto-approve` label

Triggers on pull_request_target (every PR event), check_suite.completed
(when CI lands), and workflow_dispatch (manual fire). Idempotent — skips
if the bot already approved the current head_sha.

Required repo secrets:
  - TRIAGE_BOT_APP_ID
  - TRIAGE_BOT_APP_PRIVATE_KEY

The App needs `pull_requests:write` + `contents:read` and must be
installed on this repo. If the secrets are missing, the workflow
fails fast at the token-generation step (no silent skip).

Opt-out: add the `do-not-auto-approve` label to any PR that needs
human eyes (e.g., feature work the routine couldn't fully verify).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original workflow only rubber-stamped CI-green PRs on claude/* and
auto/* branches. It solved the routine-PR admin-merge bottleneck but
ignored the bigger surface: bokelley/* PRs that ship spec changes still
need real review.

Replace with Argus, an LLM reviewer adapted from scope3data/agentic-api.
Reads the diff, delegates to AdCP subagents in parallel, posts a
substantive approve/comment/request-changes review on every non-dependabot
PR. Uses the same RELEASE_APP_* secrets the changeset release bot already
has - no separate App registration needed.

AdCP-specific MUST-FIX gates:
- Spec drift on static/schemas/source/**
- Breaking wire change without a major changeset
- Missing changeset on wire-touching PR
- oneOf discriminator regression against the audit walker baseline

Requires ANTHROPIC_API_KEY to be added to repo or org secrets before the
workflow can complete - claude-code-action mints API calls under that key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two instances of British spelling "behaviour" in
docs/reference/release-notes.mdx. Repo convention is US spelling
(158 "behavior" vs 2 "behaviour" before this fix).

Smoke test for the new Argus AI review workflow (#3488).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smoke-test PR does what it says — two-char docs fix on a load-bearing US-spelling convention (158:2 before this PR), plus the Argus workflow bootstrap. Right shape: small payload, real wire (App-token mint → prompt build → claude-code-action → verifier).

Things I checked

  • docs/reference/release-notes.mdx — two behaviourbehavior replacements, both inside narrative prose. No schema, no wire surface, no SDK impact.
  • .changeset/argus-smoke-test.mdpatch bump on adcontextprotocol matches the typo-only payload.
  • .changeset/auto-approve-routine-prs.md — empty ---\n--- frontmatter is valid for @changesets/cli (precedent in codeowners-maintainers-team.md, move-changeset-guidance-to-playbook.md). No bump emitted, release notes still surface. Intentional.
  • .github/workflows/ai-review.yml:34,42pull_request trigger + actions/checkout@v4 of head.sha. Fork PRs cannot mint the App token (secrets.RELEASE_APP_PRIVATE_KEY is unavailable on pull_request events from forks per GitHub secret-isolation policy). Fail-closed for forks.
  • .github/workflows/ai-review.yml:101-115is_trivial() correctly excludes .github/workflows/*.yml, package.json, CODEOWNERS. Workflow/dependency changes trigger full re-review.
  • Largest-file rule satisfied — read .github/workflows/ai-review.yml (219 net lines) and .github/ai-review/expert-adcp-reviewer.md (231 net lines) in full.
  • Schema-vs-docs audit — N/A. No static/schemas/source/** touched.
  • Test-plan honesty — three unchecked items ("Argus posts `--approve`", "skips subagent delegation", "uses bokelley voice") are self-validating by this review. The author cannot manually verify them prior to merging the workflow that produces them; that's the smoke-test design. Not a block. Side note: subagent delegation was NOT skipped — the PR adds .github/workflows/ai-review.yml (non-docs source), so per the prompt's own rules code-reviewer + security-reviewer ran. Expected outcome line in the PR body is wrong on that point.

Follow-ups (non-blocking — file as issues)

  1. paths-ignore is a no-op when a PR also touches a non-ignored path. .github/workflows/ai-review.yml:28-30 — GitHub paths filters are any-match-outside-the-list. A PR that edits .github/ai-review/expert-adcp-reviewer.md AND any other file still runs the workflow, which cats the modified prompt from the PR's working tree at L143. Threat: a same-repo contributor with write access injects prompt instructions that drive Argus to --approve, satisfying branch protection's "1 review required" check. (security-reviewer: High.) Override reasoning: fork vector is closed by secret-isolation; residual same-repo write-access attacker is within the trust boundary that already grants branch-push. Mitigation when convenient: read the prompt via gh api .../contents/...?ref=\$BASE_SHA, or move to pull_request_target with head.repo.full_name == github.repository gating.

  2. Bash(gh api:*) allowlist is broad. .github/workflows/ai-review.yml:170 — the wildcard lets Argus call any REST endpoint within the App token's scope. Narrow to enumerated read-only patterns. Audit the App's installation scope; confirm it's installed only on adcontextprotocol/adcp and that its permissions don't include administration or write surfaces beyond pull-requests:write. (Both code-reviewer and security-reviewer flagged.)

  3. review_posted verifier false-positives on other bot reviews. .github/workflows/ai-review.yml:78,185 — both jq filters select user.type == \"Bot\" with no login pin. A --comment from changesets-release[bot] or any other bot in the last 10 minutes is treated as Argus's review. Add .user.login == \"<argus-app-bot-slug>\" to both filters.

  4. Pin third-party actions to commit SHAs. actions/checkout@v4, actions/create-github-app-token@v1, anthropics/claude-code-action@v1, actions/github-script@v7 — all moving tags. A force-push to any of these tags ships in the next workflow run with the App token and ANTHROPIC_API_KEY in scope.

  5. Trivial-skip path carries stale approval across force-pushes. .github/workflows/ai-review.yml:65-131 — if a PR receives a bot approval, then force-pushes to a SHA whose diff against the prior-approved SHA only touches *.md/*.mdx/mintlify-docs/*, Argus skips re-review and the prior approval still counts. Real risk only when the bot's approval is the load-bearing one — which is the design. Either drop the shortcut when the prior review was bot-authored, or invalidate the bot approval on force-push.

Minor nits (non-blocking)

  1. macOS date fallback is unreachable. .github/workflows/ai-review.yml:195 — the || date -u -j -f ... BSD form never fires under runs-on: ubuntu-latest, and under set -euo pipefail both branches failing aborts the verify step. Harden with || echo 0.

  2. \$GITHUB_OUTPUT heredoc sentinel is static. .github/workflows/ai-review.yml:145,155ARGUS_EOF is a constant. Repo-controlled prompt makes this informational today; becomes load-bearing if follow-up 1 isn't addressed.

LGTM. Follow-ups noted below.

@bokelley bokelley merged commit 1a900c6 into main May 19, 2026
19 checks passed
@bokelley bokelley deleted the bokelley/argus-smoke-test branch May 19, 2026 21:26
bokelley added a commit that referenced this pull request May 19, 2026
Argus reviewed PR #4816 and flagged five real issues in its own
workflow. Addressing them:

- Prompt loaded from base SHA, not the PR head working tree. Closes
  the same-repo prompt-injection vector even when the PR doesn't
  directly modify the prompt file.

- Workflow-mod gate: PRs that touch .github/ai-review/** or
  .github/workflows/ai-review.yml alongside other files are not
  auto-reviewed. Bot posts a --comment requesting human review.

- Verifier and skip-check filters pin to the Argus bot login
  (aao-release-bot) rather than matching any bot user. Stops
  changesets-release[bot] and other bots from false-positiving.

- Bash(gh api:*) allowlist narrowed to read-only patterns:
  repos/*/pulls/*, repos/*/contents/*, repos/*/issues/*.

- Third-party actions pinned to commit SHAs to close the moving-tag
  vector. Includes actions/checkout, actions/create-github-app-token,
  actions/github-script, anthropics/claude-code-action.

- Heredoc sentinel randomized (openssl rand -hex 8) so prompt content
  can't break framing.

- macOS date fallback removed (workflow is ubuntu-latest only).

Branch-protection requirement documented in workflow comments:
"Require approval of the most recent reviewable push" MUST be enabled
for the trivial-skip path to be fail-closed against force-pushes that
touch only trivial paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley added a commit that referenced this pull request May 19, 2026
Argus reviewed PR #4816 and flagged five real issues in its own
workflow. Addressing them:

- Prompt loaded from base SHA, not the PR head working tree. Closes
  the same-repo prompt-injection vector even when the PR doesn't
  directly modify the prompt file.

- Workflow-mod gate: PRs that touch .github/ai-review/** or
  .github/workflows/ai-review.yml alongside other files are not
  auto-reviewed. Bot posts a --comment requesting human review.

- Verifier and skip-check filters pin to the Argus bot login
  (aao-release-bot) rather than matching any bot user. Stops
  changesets-release[bot] and other bots from false-positiving.

- Bash(gh api:*) allowlist narrowed to read-only patterns:
  repos/*/pulls/*, repos/*/contents/*, repos/*/issues/*.

- Third-party actions pinned to commit SHAs to close the moving-tag
  vector. Includes actions/checkout, actions/create-github-app-token,
  actions/github-script, anthropics/claude-code-action.

- Heredoc sentinel randomized (openssl rand -hex 8) so prompt content
  can't break framing.

- macOS date fallback removed (workflow is ubuntu-latest only).

Branch-protection requirement documented in workflow comments:
"Require approval of the most recent reviewable push" MUST be enabled
for the trivial-skip path to be fail-closed against force-pushes that
touch only trivial paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley added a commit that referenced this pull request May 22, 2026
)

Argus reviewed PR #4816 and flagged five real issues in its own
workflow. Addressing them:

- Prompt loaded from base SHA, not the PR head working tree. Closes
  the same-repo prompt-injection vector even when the PR doesn't
  directly modify the prompt file.

- Workflow-mod gate: PRs that touch .github/ai-review/** or
  .github/workflows/ai-review.yml alongside other files are not
  auto-reviewed. Bot posts a --comment requesting human review.

- Verifier and skip-check filters pin to the Argus bot login
  (aao-release-bot) rather than matching any bot user. Stops
  changesets-release[bot] and other bots from false-positiving.

- Bash(gh api:*) allowlist narrowed to read-only patterns:
  repos/*/pulls/*, repos/*/contents/*, repos/*/issues/*.

- Third-party actions pinned to commit SHAs to close the moving-tag
  vector. Includes actions/checkout, actions/create-github-app-token,
  actions/github-script, anthropics/claude-code-action.

- Heredoc sentinel randomized (openssl rand -hex 8) so prompt content
  can't break framing.

- macOS date fallback removed (workflow is ubuntu-latest only).

Branch-protection requirement documented in workflow comments:
"Require approval of the most recent reviewable push" MUST be enabled
for the trivial-skip path to be fail-closed against force-pushes that
touch only trivial paths.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant