Skip to content

Auto-detect PR reviewers and publish resolution feedback - #27

Merged
ether-moon merged 4 commits into
mainfrom
ether_karrot/auto-reviewers
Jul 21, 2026
Merged

Auto-detect PR reviewers and publish resolution feedback#27
ether-moon merged 4 commits into
mainfrom
ether_karrot/auto-reviewers

Conversation

@ether-moon

@ether-moon ether-moon commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • Auto-detect CodeRabbit, Claude, and Codex reviewers without reviewer-specific flags.
  • Track current-HEAD completion signals while avoiding new-review triggers after resolver pushes.
  • Publish idempotent per-thread resolution feedback after the publication gate.

Changes

  • Derive reviewer providers from saved PR evidence instead of caller-supplied adapters.
  • Resolve only fixed and accepted_as_is threads, leaving unresolved feedback open.
  • Emit the exact @coderabbitai resolve command only when all queued CodeRabbit items are resolved.
  • Reject free-form bot mentions, review triggers, and edit-delegation commands in queued feedback.
  • Update shipping documentation, resolver contracts, mocks, and state-machine evaluations.

Test Plan

  • /bin/bash plugins/skill-set/tests/run.sh
  • git diff --check

Summary by CodeRabbit

  • New Features
    • Automatically detects CodeRabbit, Claude, and Codex reviewer activity and tracks completion signals without reviewer-specific configuration.
    • Adds queued, structured per-thread resolution feedback that’s published after the publication gate succeeds.
    • Publishes eligible thread resolutions, including the official @coderabbitai resolve command for CodeRabbit when applicable.
  • Bug Fixes
    • Restores unattended initial commit/push behavior and removes redundant confirmation prompts.
    • Prevents duplicate feedback, unintended reruns, and any new automated reviews or edit-delegation during publication.
  • Documentation
    • Updated skill and command documentation to reflect the revised authorization, publication gate, and queued feedback contract.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@ether-moon, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 31 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 225cc786-fe65-49e6-8285-da3e1d89a6f2

📥 Commits

Reviewing files that changed from the base of the PR and between 45bc084 and 0b1a720.

📒 Files selected for processing (5)
  • CHANGELOG.md
  • plugins/skill-set/bin/skill-set-pr
  • plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh
  • plugins/skill-set/tests/fixtures/mock-gh-pr
  • plugins/skill-set/tests/test-pr-runner.sh
📝 Walkthrough

Walkthrough

shipping-pr now auto-detects Claude, CodeRabbit, and Codex reviewers, tracks current-HEAD evidence, and publishes validated per-thread feedback after the publication gate with provider-specific resolution behavior.

Changes

Shipping PR reviewer and publication flow

Layer / File(s) Summary
Automatic reviewer detection and snapshot state
plugins/skill-set/bin/skill-set-pr, plugins/skill-set/skills/shipping-pr/reference/polling.md, plugins/skill-set/tests/...
Reviewer activity is automatically detected across supported providers, persisted in state, and evaluated from current-HEAD evidence for blocked, polling, timeout, and completion decisions.
Queued feedback and publication gate contract
plugins/skill-set/agents/..., plugins/skill-set/skills/shipping-pr/..., plugins/skill-set/commands/pr/ship.md, CHANGELOG.md
Resolver outputs now use separate summary and per-thread feedback files with exact thread coverage, constrained outcomes, expected-HEAD validation, authorization rules, and post-gate publication ordering.
Validated thread feedback publication
plugins/skill-set/bin/skill-set-pr
publish validates thread feedback, derives provider behavior from saved snapshot evidence, records publication metadata, and publishes idempotent replies and eligible resolutions after the gate.
Fixtures and publication regression coverage
plugins/skill-set/tests/..., plugins/skill-set/evals/shipping-pr/...
Tests and mock GitHub fixtures cover multi-provider detection, review evidence, thread replies, thread resolution, idempotency, invalid inputs, authorization, and happy-path publication.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant shipping-pr
  participant skill-set-pr
  participant GitHub
  shipping-pr->>skill-set-pr: initialize and snapshot PR
  skill-set-pr->>GitHub: collect reviewer evidence
  GitHub-->>skill-set-pr: statuses, reviews, comments, reactions
  shipping-pr->>skill-set-pr: publish results and thread feedback
  skill-set-pr->>GitHub: validate gate, reply to threads, resolve eligible threads
  skill-set-pr->>GitHub: publish summary and derived CodeRabbit resolve command
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: automatic reviewer detection plus publishing resolution feedback.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ether_karrot/auto-reviewers

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh (1)

35-58: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicated GraphQL reply/resolve/node mock logic across two fixture scripts. Both mocks now implement the same id=/body= argument parsing, the same three-way GraphQL branch (reply/resolve/node), and the same fixture-file persistence side effects.

  • plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh#L35-L58: extract this block (and the paired reactions/check-runs/status stubs at L75-L89) into a small shared helper script sourced by both mocks.
  • plugins/skill-set/tests/fixtures/mock-gh-pr#L161-L196: source the same shared helper instead of re-implementing the identical parsing/branching logic (and its L223-L260 stub counterparts).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh`
around lines 35 - 58, The GraphQL reply/resolve/node handling and paired
reactions, check-runs, and status stubs are duplicated between
plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh
lines 35-58 and 75-89 and plugins/skill-set/tests/fixtures/mock-gh-pr lines
161-196 and 223-260. Extract the shared argument parsing, GraphQL branches,
persistence side effects, and stub logic into one helper script, then source
that helper from both mock-gh scripts and remove their duplicate
implementations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh`:
- Around line 35-58: The GraphQL reply/resolve/node handling and paired
reactions, check-runs, and status stubs are duplicated between
plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh
lines 35-58 and 75-89 and plugins/skill-set/tests/fixtures/mock-gh-pr lines
161-196 and 223-260. Extract the shared argument parsing, GraphQL branches,
persistence side effects, and stub logic into one helper script, then source
that helper from both mock-gh scripts and remove their duplicate
implementations.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 31d661c6-435b-48f2-af34-8fd1e4929088

📥 Commits

Reviewing files that changed from the base of the PR and between 4abd281 and 24972f6.

📒 Files selected for processing (15)
  • CHANGELOG.md
  • plugins/skill-set/agents/pr-review-feedback.md
  • plugins/skill-set/agents/resolving-pr-blockers.md
  • plugins/skill-set/bin/skill-set-pr
  • plugins/skill-set/commands/pr/ship.md
  • plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/mock-gh
  • plugins/skill-set/evals/shipping-pr/state-machine-happy-path/fixtures/scaffold.sh
  • plugins/skill-set/evals/shipping-pr/state-machine-happy-path/prompt.md
  • plugins/skill-set/skills/shipping-pr/SKILL.md
  • plugins/skill-set/skills/shipping-pr/reference/blocker-resolution.md
  • plugins/skill-set/skills/shipping-pr/reference/polling.md
  • plugins/skill-set/skills/shipping-pr/reference/troubleshooting.md
  • plugins/skill-set/tests/fixtures/mock-gh-pr
  • plugins/skill-set/tests/test-pr-runner.sh
  • plugins/skill-set/tests/test-shipping-pr-happy-path-eval.sh

@ether-moon
ether-moon merged commit 19d66fb into main Jul 21, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant