Rewrite pr-review-toolkit review workflow - #54
Conversation
|
Warning Review limit reached
More reviews will be available in 32 minutes and 18 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughThe ChangesPR Review Toolkit Rewrite
Sequence Diagram(s)sequenceDiagram
actor User
participant Skill as review-pr skill
participant Workflow as Workflow (review-pr.js)
participant MCP as GitHub MCP read tools
participant Specialists as Specialist agents
participant SynthAgent as Synthesis agent
participant GitHubWrite as GitHub MCP write tools
User->>Skill: /pr-review-toolkit:review-pr pr-url
Skill->>Workflow: invoke(owner, repo, pullNumber)
rect rgba(70, 130, 180, 0.5)
note over Workflow,MCP: Collect phase (read-only)
Workflow->>MCP: get PR metadata
Workflow->>MCP: get paginated changed-file pages
Workflow->>MCP: get review threads
end
rect rgba(100, 160, 100, 0.5)
note over Workflow,Specialists: Analyze phase (parallel)
Workflow->>Specialists: run selected lens agents with shared PrContext
Specialists-->>Workflow: findings + positiveObservations
end
rect rgba(160, 100, 160, 0.5)
note over Workflow,SynthAgent: Synthesize phase
Workflow->>SynthAgent: aggregate findings → REVIEW_BOARD_SCHEMA
SynthAgent-->>Workflow: raw review board
Workflow->>Workflow: finalizeBoard()
end
Workflow-->>Skill: structured review board
Skill->>User: Present Review Board (grouped by outcome)
User->>Skill: select findings / challenge / draft
Skill->>User: Draft + Preview (no write tools)
User->>Skill: approve exact preview
Skill->>GitHubWrite: create pending review, add line comments, submit
GitHubWrite-->>Skill: review posted
Skill->>User: confirmation
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Capture durable review principles, implementation plan, and context for reworking the PR review toolkit around a Workflow-backed comprehensive review model. Assisted-by: Codex:gpt-5
Add Workflow runtime findings from empirical testing against a 509-file PR. Key discoveries: args arrive as JSON strings requiring parsing, and structured-output response size (not workflow variable capacity) is the bottleneck for large file collection — requiring workflow-level pagination for the rewrite's collection phase. Assisted-by: Claude:claude-opus-4-6
Assisted-by: Codex:gpt-5
Assisted-by: Codex:gpt-5
Assisted-by: Codex:gpt-5
Assisted-by: Codex:gpt-5
Assisted-by: Codex:gpt-5
Assisted-by: Codex:gpt-5
Validate required config args before workflow execution, log when reviewer agents produce no findings, widen dedup key to reduce collision risk, clarify intentional schema extension, and remove misleading ID instruction from synthesizer prompt. Assisted-by: Claude:claude-opus-4-6
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pr-review-toolkit/README.md`:
- Line 76: The README link text uses the lowercase product name for the GitHub
plugin, which should be branded consistently. Update the sentence in the
documentation to use GitHub with the proper capitalization, keeping the existing
link target and surrounding wording unchanged.
In `@pr-review-toolkit/skills/review-pr/review-pr.js`:
- Around line 1403-1416: The selected reviewer loop in review-pr.js only warns
when result is missing, but an empty findings array is also a diagnostic “empty
findings” case that currently disappears. Update the selected.forEach handling
to detect when result.findings is an empty array and emit a log message for that
case as well, alongside the existing no-result warning, using the same selected
name and REVIEWERS[name] context so the empty-reviewer path is visible in
diagnostics.
- Around line 729-736: The candidateLocation function is coercing invalid or
non-numeric finding.line values to 0, which creates bad GitHub locations and
affects overlap handling. Update candidateLocation so it only sets location.line
when rawLocation.line or finding.line can be parsed into a valid number, and
leave it unset when parsing fails. Keep the behavior localized to
candidateLocation and preserve the existing path resolution logic.
- Around line 1028-1040: The actionPlanForBoard() summary is incorrectly
flattening discussionOnly findings into the suggestions bucket, which can hide
critical and important items. Update the bucket construction so discussionOnly
entries are grouped by each item’s own severity alongside the other actionable
findings, while keeping suggestions limited to true suggestion-level items.
Preserve the existing severity ordering and use actionPlanForBoard() and its
critical/important/suggestions fields as the main place to adjust the
classification logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 35cba42b-07d3-4bc8-b4f7-a1c4d0fb5cac
📒 Files selected for processing (8)
README.mdpr-review-toolkit/.claude-plugin/plugin.jsonpr-review-toolkit/README.mdpr-review-toolkit/docs/PR_REVIEW_REQUIREMENTS.mdpr-review-toolkit/docs/WORKFLOW_REWRITE_CONTEXT.mdpr-review-toolkit/docs/WORKFLOW_REWRITE_PLAN.mdpr-review-toolkit/skills/review-pr/SKILL.mdpr-review-toolkit/skills/review-pr/review-pr.js
There was a problem hiding this comment.
Pull request overview
This PR rewrites the pr-review-toolkit plugin's review-pr workflow from a parallel-agents/scored-findings pipeline into a shared-context "review board" model. The workflow now owns PR collection (metadata, paginated changed files, review threads), runs specialist reviewer lenses against one shared PrContext, and synthesizes findings into outcome-grouped buckets (recommendedToPost, possiblePlusOnes, partialOverlaps, discussionOnly, alreadyCovered, discarded). The skill conversation then presents the board, supports interactive selection/drafting, and posts only after explicit approval. The change removes the previous per-finding verifier and resolved-thread verifier phases, drops the git/Bash-based local-diff path, and adds extensive design documentation.
Changes:
- Replaced the finding/verify/contextualize pipeline in
review-pr.jswith collect → analyze → synthesize phases, new schemas, a reviewer registry, dedup/merge/routing helpers, args validation, and empty-finding logging. - Rewrote
SKILL.mdaround launching the analysis workflow, presenting the board, free-text interaction, drafting, preview, and approved posting; trimmedallowed-tools(removed git Bash, addedAskUserQuestion). - Added three planning/requirements/context docs and refreshed both READMEs and
plugin.json(1.4.0 → 1.5.0).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
pr-review-toolkit/skills/review-pr/review-pr.js |
Core rewrite: new phases, schemas, reviewer registry, manifest/thread context, board synthesis & normalization helpers. |
pr-review-toolkit/skills/review-pr/SKILL.md |
New skill flow: launch workflow, present board, interact, draft, preview, post; updated tool allowlist and return shape. |
pr-review-toolkit/README.md |
Documents review flow, board outcomes, interaction/posting, MCP permissions, and validation guidance. |
pr-review-toolkit/docs/WORKFLOW_REWRITE_PLAN.md |
New implementation plan, data models, migration phases, and capability-spike results. |
pr-review-toolkit/docs/WORKFLOW_REWRITE_CONTEXT.md |
New background/context for the rewrite. |
pr-review-toolkit/docs/PR_REVIEW_REQUIREMENTS.md |
New enduring requirements/principles document. |
pr-review-toolkit/.claude-plugin/plugin.json |
Version bump to 1.5.0 and updated description. |
README.md |
Updated marketplace table description for the plugin. |
Key points raised in line comments: a paginated collector page size of 100 that conflicts with the spike's observed ~17-entry structured-output cap (possible silent manifest truncation on large PRs), a thread-overlap heuristic where the same-line proximity bonus can force unrelated findings into the overlap buckets, and a now-stale "setup section" reference in the embedded code-reviewer prompt.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Keep invalid finding line values out of board locations, preserve discussion-only findings in severity action-plan buckets, and log empty reviewer findings as diagnostics. Assisted-By: Codex:gpt-5
0e87df8 to
5b9228b
Compare
| let duplicateCount = 0 | ||
| let partialOverlapCount = 0 | ||
| let newCount = 0 | ||
| const pr = normalizePr(prResult) |
Summary
review-prskill workflow script with enriched candidate findings, strengthened review board synthesis, organized reviewer registry, and improved interaction patternsTest plan
/reviewagainst a test PR and verify the full workflow executes end-to-endclaude plugin validate ./pr-review-toolkitSummary by CodeRabbit
New Features
Documentation