Add scheduled holistic reviews for all PRs and path-specific review instructions

[jeffhandley]

Summary

Reworks the repository's customized code-review automation into two complementary parts:

1. A deterministic orchestrator plus a per-PR agentic workflow that replaces the previous pull_request-triggered code-review.md
2. Path-specific review instructions broken out of the code-review skill so GitHub's built-in Copilot code review agent applies the same dotnet/runtime-specific rules.

The dispatched agentic workflow submits a single, cumulative "Holistic Review" per commit range rather than a stateless diff-only pass, and it is designed to run alongside — not replace — the built-in Copilot reviewer.

Behavior: draft, ready, and closed pull requests

• Ready (non-draft), open PRs are polled on a 10-minute schedule; the orchestrator dispatches one worker run per PR whenever it is new or its head commit changed since the last durable review.
• Draft PRs are excluded from the scheduled poll. They can still be reviewed on demand: holistic-review-orchestrator's workflow_dispatch accepts a comma-separated pr_numbers input ("including drafts and retry-limited review targets") that bypasses the draft filter for exactly the requested PRs, and an unchanged head with an existing durable review is not reviewed again.
• Scheduled runs skip forks while manual runs remain available. The dispatch job runs for workflow_dispatch or when the repository is not a fork, preventing scheduled fork runs without blocking explicit manual dispatch.
• Manual and scheduled dispatch share the exact same state, retry, and dedup logic — there is no separate manual code path to keep in sync.

Coexistence with the built-in Copilot code review agent

The Holistic Review worker and GitHub's built-in Copilot code review agent are fully independent; neither waits on, hides, or edits the other's output, and both remain visible on the PR alongside human reviews. The path-specific instruction files under .github/instructions/ are consumed natively by both: the Holistic Review skill explicitly loads them, and Copilot's built-in reviewer already applies any .instructions.md file whose applyTo glob matches the changed paths. Every Holistic Review ends with an explicit disclosure distinguishing it from the built-in review and linking back to the holistic-review.md workflow source.

Reviews are strictly additive: the worker's only two safe outputs are up to 10 inline create_pull_request_review_comment calls and exactly one submit_pull_request_review, whose event is restricted to COMMENT. It never submits APPROVE or REQUEST_CHANGES, and it never modifies, hides, or supersedes an earlier review or comment from this workflow, the built-in reviewer, or a human.

Architecture: deterministic orchestrator + gh-aw worker

• holistic-review-orchestrator.yml is a conventional, deterministic GitHub Actions workflow — not an agentic workflow. Its top-level permissions: {} grants only actions: write / pull-requests: write inside its single job, and it does nothing but call the GitHub REST API to decide what to dispatch and to read/write one managed state comment per PR. It never checks out PR content and never runs a model, keeping it secure even though it processes untrusted PR metadata.
• holistic-review.md is the gh-aw-compiled agentic worker (compiled to holistic-review.lock.yml with gh-aw v0.82.6, pinned via .github/aw/actions-lock.json to github/gh-aw-actions/setup@v0.82.6). The orchestrator dispatches it once per PR via the Actions API with the PR number, base ref, head SHA, the previously reviewed head and base SHA (when re-reviewing), a bounded review-history JSON blob, and a complete synthetic aw_context (run_id, repo, workflow_id, item_type, item_number) so gh-aw's safe-output layer has full pull-request context even though the triggering event is workflow_dispatch, not pull_request.
• The orchestrator and worker communicate only through workflow_dispatch inputs at dispatch time and the worker's own submitted review afterward, which the orchestrator discovers by matching the run's display_title (Holistic Review #<PR> (<head SHA>)) and by recognizing gh-aw's automatic <!-- gh-aw-agentic-workflow: Holistic Review, ... workflow_id: holistic-review --> review-footer marker.

See github/gh-aw's documentation on compiled agentic workflows, safe outputs, and workflow_dispatch-based dispatch, and GitHub's own workflow_dispatch and triggering-a-workflow documentation, for background on the primitives this design builds on.

Why review state lives in a pull request comment

Each PR carries exactly one managed, orchestrator-owned issue comment holding a versioned (version: 5) JSON state object: the last dispatched and last reviewed (commit, base ref, base SHA) pairs, the ID of the worker run that produced the last recorded review, a bounded initial-plus-latest review-history array, and a bounded review_attempt_count. Comments were chosen over gh-aw's repo-memory/cache mechanisms because gh-aw v0.82.6 supports deterministic reads of repo-memory/comment-memory but not deterministic write-back of durable state from a dispatch-triggered workflow. A pull-requests: write-scoped comment is visible, auditable state that survives reruns, needs no contents: write, and is trivially inspectable by anyone reading the PR. Crucially, a submitted, marker-tagged worker review is the authoritative record of what has actually been reviewed — not the state comment; the comment only suppresses duplicate in-flight dispatches, and legacy comment formats (including a pre-existing machine-only JSON format and an older HTML-marker format) are recognized once and migrated to the current schema in place.

Retries and durable-review authority

A worker run that completes without submitting a review — a transient provider error, a threat-detection replacement, or a safe-output formatting failure — is never treated as "reviewed." The orchestrator only advances last_reviewed_* when it finds a matching, marker-tagged submitted review for the exact dispatched commit and base; otherwise it clears that dispatch record and redispatches the same target. Retries for the same (head, base) target are bounded at MAX_REVIEW_ATTEMPTS = 5; once exhausted, the PR is reported as retry-limited in the workflow step summary rather than dispatched indefinitely, and it can still be retried explicitly through the manual pr_numbers input. Because attempts are keyed by (head commit, base branch), a rebase, force-push, or base retarget resets the attempt counter for the new target instead of inheriting an unrelated failure count.

Review scope: holistic initial review vs. patch-differential incremental review

A trusted, deterministic pre-agent step — not the model — computes the review scope before the agent starts: it resolves the current merge base, computes a patch ID for the PR's cumulative diff, and (for re-reviews) computes the previous merge base, a previous patch ID, a git range-diff, and a raw patch diff, writing all of it to metadata.json plus range-diff.txt/patch-diff.txt for the agent to read.

• Initial review: analyzes the complete current_merge_base..head range — the PR's actual base-to-head diff, not its head compared against the current state of the base branch.
• Re-review (incremental): uses two distinct scopes. It re-reads the complete current base-to-head range only to refresh the cumulative Motivation/Approach/Summary assessment, explicitly comparing against the initial and most recent recorded reviews (retrieved by ID, not rediscovered from the general review list) and stating whether each is unchanged or changed and why. Detailed/actionable findings are restricted to the prepared range-diff/patch-diff between the previous and current cumulative patches — not a raw tree diff between the two head commits — so a rebase that pulls in unrelated upstream changes cannot manufacture new findings. If the previous and current heads are identical but the merge base changed (a pure base retarget), the same prepared patch comparison is authoritative: only code whose inclusion or semantics changed because of the retarget is reviewed, and unchanged portions of the PR patch are not rediscovered. If the patch truly has not changed, the worker still submits a COMMENT review recording that fact and the refreshed Assessment History, so every successful run leaves a durable record.

Model selection

The workflow does not hard-code a model. engine.model resolves from the workflow-specific HOLISTIC_REVIEW_MODEL Actions variable, allowing the model to be selected at environment, repository, or organization scope without editing or recompiling the workflow.

Security model

• Least privilege throughout. The orchestrator's job-scoped permissions are actions: write / pull-requests: write only; the worker's top-level permissions are contents: read, issues: read, pull-requests: read, and its actual review/comment capability comes from gh-aw's safe-output layer rather than a broadly scoped token held by the agent.
• GitHub reads go through the job-scoped, read-only token. tools.github.github-token is pinned to ${{ secrets.GITHUB_TOKEN }}, so the agent's gh/GitHub-proxy calls cannot be backed by a broader GH_AW_GITHUB_* secret, and the declared read-only permissions are actually authoritative.
• PR content — including its own configuration files — is treated as untrusted input. The worker checks out the dispatched head commit, then removes and restores every agent-configuration path gh-aw v0.82.6 recognizes (the complete .github tree, every supported engine configuration directory, and all recognized root instruction files) from main before loading any guidance, because a plain checkout alone would leave PR-added files in those paths behind. PR versions of those paths — along with PR descriptions, comments, source comments, test data, and other PR-controlled text — are treated strictly as untrusted review content, never as instructions; changed files under a trusted overlay path are still reviewed from an explicit commit read or the PR diff, never from the restored worktree copy.
• A narrow, read-only shell allowlist with no build/test/execute capability. The worker cannot run builds or tests, restore or install dependencies, execute PR-provided scripts or binaries, or make direct outbound HTTP requests; there is no web-fetch tool. The allowlist further excludes general-purpose process-launch surfaces (awk, find, xargs, rg, sed, sort, gh) so an allowlisted command cannot spawn another process or modify the workspace; the prompt separately forbids Git/GitHub CLI aliases, hooks, pagers, external helpers, credential helpers, and other child-process-spawning options, and treats the remaining compiler-injected git/gh/sort surface honestly as defense in depth rather than a strict sandbox.
• Egress is narrowed and routed through a CLI-side proxy. network.allowed is limited to defaults (no dotnet feed access, since builds/restores are disallowed), and tools.cli-proxy: true alongside tools.github.mode: gh-proxy ensures no native MCP endpoints remain mounted in the CLI — only the CLI-mounted safeoutputs server — avoiding a firewall interaction where native endpoints bypassed the intended proxy path and were denied.
• Both safe outputs are bound to the exact dispatched PR, not a wildcard or an agent-supplied target: target: ${{ github.event.inputs.pr_number }} on both create-pull-request-review-comment and submit-pull-request-review. A review cannot be redirected to any PR other than the one the orchestrator dispatched.
• Agent checkouts don't carry credentials. Because gh-aw strips Git credentials before the agent runs, pre-agent-steps perform a token-scoped fetch of exactly the commits the review needs (current head, prior head, and prior base) while a token is still available, so the agent operates only on local Git objects afterward, including across force-pushes and retargets.

Shared review instructions

The review rules that used to live entirely in .github/skills/code-review/SKILL.md are split into path-scoped .github/instructions/*.instructions.md files, each with an applyTo glob, so they're consumed identically by the Holistic Review skill and by GitHub's built-in Copilot code review agent:

• review-all-src.instructions.md (src/**) — reviewer mindset, the Holistic PR Assessment (motivation, evidence, approach, cost-benefit, scope, risk, codebase fit), PR hygiene, code reuse, established conventions (including preserving pre-existing alphabetical ordering in modified .csproj item groups and similar lists, flagging only ordering regressions the PR introduces, not pre-existing unsorted entries), and documentation/comment rules.
• review-csharp.instructions.md (**/*.cs) — C#-specific error handling, thread safety, security, correctness, performance/allocation, API design, and style rules.
• review-native.instructions.md (C/C++/asm globs) — JIT-specific correctness, C++ style, VM/interpreter conventions, native size/offset overflow guarding, platform defines, and P/Invoke marshalling.
• review-all-tests.instructions.md (test globs) — testing conventions and regression-test expectations.
• review-core-runtime.instructions.md (src/coreclr/**,src/native/corehost/**) — CoreCLR/compiler/host-specific correctness, collectibility, allocation, and PR-prerequisite guidance that should not load for ordinary managed-library changes.

The broad C#, native, and test files explicitly state that PR-level gates are preparation guidance during authoring or local experimentation, not reasons to block exploratory work unless review is requested. SKILL.md retains the review process and points at these files for substantive guidance. Its area-agent discovery runs only when sub-agent tooling and matching agents actually exist; an instruction file does not imply an agent. The dispatched worker has no such tooling and explicitly skips agent discovery and multi-model fan-out.

Testing: multi-iteration time-travel replay simulation

Because this changes how and when reviews are triggered — not just what they say — it was validated by replaying real PR event timelines (opening commit, subsequent commits, and human review feedback) against the candidate workflow tree in a disposable simulation repository, resetting all simulation PRs and state between rounds and starting the next round only from a clean, freshly deployed tree.

---

Note

This PR description was drafted with the assistance of GitHub Copilot.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR restructures the repo’s customized Copilot-based review automation into a scheduled “orchestrator + per-PR worker” model and moves detailed review guidance into path-scoped .github/instructions/*.instructions.md files so both the custom workflow and GitHub’s built-in Copilot code review can share the same rule source.

Changes:

• Converts code-review into a workflow_dispatch per-PR worker and adds a scheduled orchestrator that deterministically selects PRs needing (re)review and dispatches the worker.
• Splits language-/area-specific review guidance out of the code-review skill into path-scoped instruction files under .github/instructions/.
• Updates the Agentic Workflows action pinning to include actions/cache/*@v4 to support the new orchestrator’s state caching.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
.github/workflows/code-review.md	Reworks the workflow into a per-PR workflow_dispatch worker with updated safe-outputs / concurrency and PAT-pool handling.
.github/workflows/code-review.lock.yml	Regenerated lockfile reflecting the worker model and updated tool/container configuration.
.github/workflows/code-review-orchestrator.md	New scheduled orchestrator that computes a deterministic dispatch list and fans out worker dispatches.
.github/workflows/code-review-orchestrator.lock.yml	New lockfile for the orchestrator workflow.
.github/skills/code-review/SKILL.md	Slims the skill to process + points at path-scoped instruction files; updates output header text.
.github/instructions/review-all-src.instructions.md	New “general” source review criteria for src/**.
.github/instructions/review-csharp.instructions.md	New C#-specific review criteria (**/*.cs).
.github/instructions/review-native.instructions.md	New native/interop review criteria (C/C++/asm globs).
.github/instructions/review-all-tests.instructions.md	New test review criteria (tests globs).
.github/aw/actions-lock.json	Adds pinned SHAs for actions/cache/restore@v4 and actions/cache/save@v4.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

[tannergooding]

A few structural notes on the instructions/skill/worker split, focused on where content is applied vs. where it should live. Individual points inline.

Note

These review comments were drafted with the assistance of GitHub Copilot.

[tannergooding]

Not for this PR -- flagging as potential future work since it's pre-existing in main and out of scope here.

.github/skills/code-review/SKILL.md bakes in transient, environment-specific guidance that will age poorly in a durable, source-controlled skill:

• The Multi-Model Review section hard-codes Do not use gpt-5.4 -- known reliability issues causing sub-agent timeouts in >90% of affected runs. A specific model version + failure-rate belongs in config/ops state, not a checked-in review skill; it's stale the moment that model is fixed or retired.
• More broadly, Step 2: Discover Area-Specific Agents and Multi-Model Review both assume an interactive environment that can fan out sub-agents. As this PR now routes the dispatched single-engine worker through the same skill (see my inline note on holistic-review.md), it's worth a follow-up to either data-drive the model blocklist or gate those sections on sub-agent support so both consumers read cleanly.

Note

This comment was drafted with the assistance of GitHub Copilot.

[tannergooding]

Generally LGTM and the remaining nits are mostly pre-existing, so I'm not going to block on it. We can handle it in a follow up after this big refactoring

[jeffhandley]

Thanks @T-Gro, @tannergooding, and @PranavSenthilnathan — I’ve incorporated the in-scope feedback and replied inline on each thread.

The final design now:

• uses a conventional deterministic orchestrator, with scheduled runs skipped on forks while manual dispatch remains available;
• stores durable, retry-bounded state in one managed PR comment while treating submitted reviews as authoritative;
• submits additive, non-blocking COMMENT reviews with line-anchored findings and never hides earlier feedback;
• computes initial and incremental scope deterministically from cumulative PR patches, correctly handling rebases and base retargets;
• restores trusted agent configuration from main, binds safe outputs to the dispatched PR, and narrows GitHub, network, and tool access;
• splits review guidance into general, C#, native, test, and core-runtime scopes, with authoring caveats in the broad instruction files;
• gates area-agent discovery on actual tooling and available agents, while the dispatched worker explicitly skips fan-out.

I validated the workflow with pinned gh-aw v0.82.6, deterministic recompilation, YAML and Bash syntax checks, and the multi-iteration simulation. The simulation also confirmed that pull-requests: write is sufficient for the managed PR state comment; issues: write is not required.

The model-specific reliability guidance called out in the follow-up comment remains intentionally deferred as separate work.

Note

This summary was drafted with GitHub Copilot on behalf of the PR author.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated no new comments.