[Incident] Isolate interactive review-agent mentions from scheduled sweep concurrency

[seonghobae]

Buyer-visible incident

Exact-head review requests can remain unacknowledged even when a maintainer posts a trusted @opencode-agent or @cwl-noema-review comment. This blocks otherwise merge-ready security and product pull requests behind an invisible routing failure.

Observed examples include repeated exact-head requests on .github#813 and mightyETL#121 without the router's durable receipt marker or eyes acknowledgement. The Noema-only .github#813 request is not subject to the OpenCode repository allowlist, so an allowlist rejection cannot explain that request's absence.

Root cause

The protected-main workflow .github/workflows/agent-mention-router.yml currently assigns both event classes to one workflow-level concurrency group:

concurrency:
  group: review-agent-mention-router-${{ github.repository }}
  cancel-in-progress: false

That group contains both:

• interactive issue_comment runs, whose local job has a five-minute timeout; and
• organization-wide scheduled sweeps every five minutes, whose sweep job has a fifteen-minute timeout.

GitHub Actions permits only one running and, by default, one pending run in one concurrency group. When another run enters the group, it cancels and replaces the existing pending run even when cancel-in-progress is false. Therefore a five-minute schedule event can cancel a pending interactive comment run while an earlier sweep is still running. Repeated schedule arrivals can silently discard trusted review requests before the router reads their comments, dispatches the agent, writes the durable ledger, or acknowledges the request.

This is a deterministic configuration failure, not a model-provider or review-quality failure.

Considered approaches

A. Job-level isolated concurrency — recommended

Remove the shared workflow-level group and assign separate job groups:

route-local-agent-mention:
  concurrency:
    group: review-agent-mention-router-local-${{ github.repository }}
    queue: max

sweep-organization-agent-mentions:
  concurrency:
    group: review-agent-mention-router-sweep-${{ github.repository }}
    cancel-in-progress: false

This preserves ordered interactive requests without cancellation, while the scheduled group retains the default single pending run so stale sweeps coalesce without interrupting the running sweep. Local and sweep jobs may overlap; the existing exact-name artifact ledger remains the cross-path idempotency authority.

B. Split local routing and scheduled sweep into two workflow files

This provides strong operational isolation but duplicates trigger, permission, checkout, and quality-contract surface. It is unnecessary if job-level concurrency is sufficient.

C. Give every comment a unique concurrency group

This prevents cancellation but permits unbounded parallel comment routing and increases duplicate-dispatch and Actions-consumption pressure. Existing ledger idempotency would still work, but serialization is preferable.

Bounded repair contract

1. Add a RED workflow-contract test proving the inherited shared group can cancel a pending interactive request.
2. Remove workflow-level concurrency from the router.
3. Add the local job group with queue: max.
4. Add a separate scheduled-sweep group with cancel-in-progress: false and no queue: max.
5. Preserve all triggers, conditions, permissions, credentials, reviewer identities, dispatch payloads, exact-head claims, artifact-ledger names, allowlists, and failure semantics.
6. Add authoritative doctoring with incident evidence, concurrency semantics, rollout, monitoring, rollback, and APA 7 references.
7. Update CHANGELOG.md and the permanent quality workflow path contract when necessary.
8. Verify the complete central suite, 100% production statement/branch coverage, 100% production docstrings, Python compilation, YAML contract, and git diff --check.
9. After protected merge, post one fresh exact-head Noema request on .github#813 and one on mightyETL#121; require a durable router receipt before relying on downstream review evidence.

Security and operations boundary

• No model credential, reviewer credential, token name, repository allowlist, write scope, or branch-protection rule changes.
• No COPILOT_GITHUB_TOKEN and no new model provider.
• The local and sweep jobs remain central-only and use their existing least-privileged job permissions.
• The durable exact-name Actions artifact remains the idempotency ledger; a receipt is acknowledgement of dispatch, not approval.
• Queue depth and routing latency should be observable through bounded run-level evidence without recording comment bodies, payloads, principals, tokens, or model output.
• Rollback must not restore a shared event-class concurrency group unless a replacement proves that interactive requests cannot be canceled by scheduled work.

Standards and primary documentation — APA 7th

GitHub. (2026). Concurrency. GitHub Docs. https://docs.github.com/en/actions/concepts/workflows-and-actions/concurrency

GitHub. (2026). Control the concurrency of workflows and jobs. GitHub Docs. https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency

GitHub. (2026). Events that trigger workflows. GitHub Docs. https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows

GitHub. (2026). Troubleshooting workflows. GitHub Docs. https://docs.github.com/en/actions/how-tos/troubleshoot-workflows

GitHub documents that a concurrency group has one running and, by default, one pending run; a new queued run replaces the existing pending run. It also documents queue: max, job-level concurrency, issue_comment default-branch behavior, and the possibility that scheduled work is delayed or dropped under load.

[seonghobae]

Fresh RCA/feasibility update from current live state:

Two independent protected-main defects now sit on the same end-to-end mention path and must not be conflated.

1. Queue-loss root cause — current-main replacement PR fix(automation): replay isolated review-agent queues on current main #835 remains open/Ready/mergeable at exact head 4881c7860a1ed503917e68cc096dbfb1cdb75b36. Its isolated local/sweep concurrency repair has successful exact-head Agent Mention Router Quality CI, Strix, CodeQL, Python Security, Security Scan, Semgrep, Secret Scan, OSV, Scorecard, and SBOM evidence. It fixes scheduled-sweep replacement of pending interactive work.
2. Dispatch-rejection root cause — PR fix(actions): bound agent mention dispatch envelope #840 remains open/Ready/mergeable at exact head ba93f537302086ae54a919ea45c3ca96f2458415. Runtime evidence behind fix(actions): bound agent mention dispatch envelope #840 proved the OpenCode mention route emitted 14 top-level client_payload properties and GitHub rejected it with HTTP 422 before downstream review. fix(actions): bound agent mention dispatch envelope #840 replaces both violating hops with the bounded cwl.agent-invocation/v2 envelope; its current exact-head Agent Mention Router Quality CI, Strix, CodeQL, Python Security, Security Scan, Semgrep, Secret Scan, OSV, Scorecard, and SBOM checks are successful and inline threads are resolved/outdated. Its predecessor-head review does not transfer; current-head automated review and counted independent human approval remain separate gates.

Feasibility decision: a protected-main operational mention test before #840 integrates cannot prove #835 fixed end-to-end review delivery, because the request can still deterministically fail at the independent 422 payload boundary. Conversely, #840 alone does not prove interactive comments cannot be replaced by scheduled sweep concurrency. Do not patch #835 to work around #840, broaden credentials, or weaken either gate.

Operational closure for this incident therefore requires both accepted repairs on protected main, with any shared-file/base movement reconciled and exact-head evidence regenerated normally. Only then post fresh trusted Noema/OpenCode review-only comments and require durable receipt/acknowledgement plus downstream dispatch while a scheduled sweep is active. #814 remains open until that protected-main runtime evidence exists; static PR checks and predecessor runs are not incident closure.