Add /refine-plan skill for implementation plan review and refinement

[williamthorsen]

Problem

Implementation plans passed to /orchestrate-dev are often under-specified, leaving decision gaps that the coder fills with its own judgment. This causes implementation divergence from user expectations, particularly in two areas:

1. UX behavior: When the plan says "add a settings panel" without specifying interactions, layouts, or states, the coder improvises — and its choices often don't match what was intended.
2. Technical choices for new functionality: When there's no existing codebase pattern to follow, the coder makes autonomous decisions about patterns, data structures, and API shapes that diverge from expectations.

The cost model is clear: undoing work done wrong is far more expensive than spending tokens upfront to get the plan right. The code review cycle catches code defects, but it cannot catch the case where correct code implements the wrong thing because the plan was under-specified. The plan matched the requirements; it just didn't say enough.

Current workflow

1. Use Claude's planning mode (with interactive Q&A) to generate a ticket and a plan
2. Save the plan with /save-plan under a project and ticket ID
3. Clear context and orchestrate: "Orchestrate the implementation of path/to/ticket.md. This plan is proposed: path/to/plan.md."

Between steps 2 and 3, manual review of the plan typically produces actionable findings — missing UX details, unspecified technical decisions, incorrect assumptions about the codebase. This manual review step should be automated.

Proposed solution

A /refine-plan skill that performs a single review-and-revise round on a saved implementation plan, checking for both completeness (decision gaps the coder would fill) and correctness (factual accuracy against the codebase), before the plan is passed to /orchestrate-dev.

Input

• A saved plan file (the {timestamp}_{slug}_plan.md artifact from /save-plan)
• A ticket or requirements source (file path or URL — GitHub issue URL, Jira URL, etc.)

Review criteria (single reviewer, comprehensive rubric)

Completeness

Focus	What the reviewer checks
UX specification	For UI-facing steps: are interactions, layouts, states (empty, loading, error), and user flows specified? Or will the coder have to decide?
Technical decisions	For new functionality: are patterns, data structures, API shapes, and library choices specified? Or will the coder have to choose?
Behavioral specification	For logic-heavy steps: are edge cases, defaults, and failure modes specified? Or will the coder have to improvise?

Correctness

Focus	What the reviewer checks
Factual accuracy	Do referenced files, utilities, and APIs exist and work as the plan assumes?
Structural soundness	Are dependencies ordered correctly? Are steps appropriately scoped?
Requirements traceability	Does every requirement have a step? Does every step trace to a requirement?

Flow

plan + ticket
  → reviewer (completeness + correctness findings)
  → split:
       findings the planner can resolve autonomously
         (correctness fixes, filling gaps from codebase context)
       gaps requiring user input
         (UX preferences, technical choices where multiple valid options exist)
  → present gaps to user as specific questions
  → user answers
  → planner revises plan (incorporating all findings + user answers)
  → save refined plan

Artifact layout

{ticket-dir}/
├── {timestamp}_{slug}_plan.md              ← original plan (preserved)
├── {timestamp}_{slug}_plan-review.md       ← review findings
├── {timestamp}_{slug}_plan-v2.md           ← refined plan

All are ticket-level artifacts using the standard {timestamp}_{slug}_{artifact-type} naming convention. No role-based naming (these aren't run artifacts).

Updated workflow

After implementation, the workflow becomes:

1. Use Claude's planning mode to generate a ticket and a plan
2. Save the plan with /save-plan
3. /refine-plan path/to/plan.md <ticket-url-or-path> ← new step
4. Clear context and orchestrate: "Orchestrate the implementation of <ticket-url-or-path> using this implementation plan: path/to/plan-v2.md."

Note the confidence upgrade in step 4: "using this implementation plan" (follow it) instead of "this plan is proposed" (validate and potentially deviate). The review step justifies this higher confidence.

Scope

• This is a standalone skill, not a new phase in the orchestrate pipeline
• It uses Task tool agent dispatches, not the orchestrate engine
• It performs one review-and-revise round (no convergence loop)
• It is invoked manually between saving the plan and orchestrating implementation

Acceptance criteria

Must have

• A /refine-plan skill exists at packages/agents/content/skills/refine-plan/SKILL.md
• The skill accepts a plan file path and a ticket source (file path or URL) as input
• A reviewer agent is dispatched to analyze the plan for completeness and correctness
• Completeness review covers UX specification gaps, unspecified technical decisions, and unspecified behavioral details
• Correctness review covers factual accuracy (file/API existence), structural soundness (dependency ordering), and requirements traceability
• Findings are split into those the planner can resolve autonomously and those requiring user input
• Decision gaps requiring user input are presented to the user as specific questions
• A planner/reviser agent is dispatched to produce the refined plan incorporating findings and user answers
• The original plan is preserved (not overwritten)
• The review findings are saved as {timestamp}_{slug}_plan-review.md
• The refined plan is saved as {timestamp}_{slug}_plan-v2.md
• All artifacts are saved at the ticket level (same directory as the original plan)

Should have

• The reviewer verifies factual claims by reading files referenced in the plan
• The reviewer flags steps where the coder will need to make UX decisions that aren't specified
• The reviewer flags steps where the coder will need to choose between multiple valid technical approaches
• The refined plan includes a brief summary of what changed from the original (for user awareness)

Nice to have

• The skill works with orchestration plans (orchestration-plan.md) in addition to prose plans, for cases where /plan-orchestrable-steps is used instead of planning mode

[williamthorsen]

Discussion insights (from planning session, 2026-03-03)

These notes capture architectural insights from the session that explored plan iteration patterns. They are not directly actionable for this ticket but are relevant to future skill and pipeline development.

---

Plan review vs. code review: where the analogy holds and where it breaks

Code review works because code has precise, verifiable semantics — a reviewer can trace logic, check edge cases, verify types. Plans are higher-level abstractions with fewer "surfaces" for defects. The research on agentic workflow patterns (see references below) consistently shows that plan iteration has a steeper diminishing-returns curve than code iteration: one critique round captures the vast majority of value for plans, whereas code benefits from 2-3 rounds.

The analogy holds strongest for completeness checking (are there gaps?) and requirements traceability (does the plan cover the spec?). It is weakest for technical approach validation (is this the right way to solve the problem?), where the reviewer has the same information and similar blind spots as the planner.

The real problem: specification completeness, not plan correctness

The discussion revealed that implementation divergence is primarily caused by under-specification, not by incorrect plans. Plans that say "add a settings panel" without specifying UX details produce correct code that does the wrong thing. The code review cycle cannot catch this because the code matches the plan — the plan just didn't say enough.

This means the most valuable review criterion is: "Where in this plan will the coder have to make decisions that aren't specified?" — a decision-gap analysis rather than a correctness check.

Three established patterns for plan iteration in agentic systems

1. Self-reflection (single agent critiques its own output) — cheapest, but limited by the planner's own blind spots. Suitable as an enhancement to the planner's prompt, not as a separate skill.

2. Separate critic (two-agent loop) — the approach chosen for this ticket. Different evaluation mode (generative vs. evaluative) provides genuine value even with the same underlying model.

3. Multi-agent debate (multiple critics in parallel) — highest coverage but expensive. The signal-to-noise ratio is lower for plans than code. Considered overkill for a single-round skill.

The orchestrate pipeline as a generic iteration engine

The orchestrate engine could serve as a generic pipeline for iterative back-and-forth among subagents — not just code+review but also plan+review, design+critique, etc. The infrastructure (MCP events, artifact registration, run directories, visualization) is artifact-type-agnostic. However, the review-cycle module (review-cycle.md) is deeply code-specific (diffs, code simplifier, file-type-based reviewer activation). A plan review cycle would need its own module rather than parameterizing the existing one.

If future work moves plan review into the orchestrate pipeline (e.g., /orchestrate-planning), the MCP event schema's phase_started enum would need to be extended to accept new phase values. This is a small, well-scoped change that supports the longer-term vision of the engine as a generic multi-agent iteration platform.

Design review (out of scope, future consideration)

A separate concern was identified: design review — producing an excellent vision/strategy document (no implementation details) through automated critique. This is a distinct use case from implementation plan refinement:

• Design review evaluates the quality of strategic thinking (alternatives considered, assumptions challenged, problem-solution fit)
• Plan review evaluates specification completeness and correctness (will the coder have enough detail to implement correctly?)

Design review benefits more from interactive collaboration and is best left as a human-driven process for now. If automated later, it would be a separate skill (e.g., /orchestrate-design) using the orchestrate engine with a design-specific pipeline.

References

• The 2026 guide to agentic workflow architectures
• Agentic design patterns: the 2026 guide
• AFLOW: automating agentic workflow generation
• Agents at work: the 2026 playbook
• 4 agentic AI design patterns

[williamthorsen]

Implementation plan

Plan: Add /refine-plan skill for implementation plan review and refinement

Issue: #157
Branch: 157

Context

Implementation plans passed to /orchestrate-dev are often under-specified, leaving decision gaps that the coder fills with its own judgment. This causes divergence from user expectations, particularly in UX behavior and technical choices for new functionality. The cost model is clear: undoing work done wrong is far more expensive than spending tokens upfront to get the plan right.

This change adds a /refine-plan skill that performs a single review-and-revise round on a saved implementation plan, checking for completeness (decision gaps the coder would fill) and correctness (factual accuracy against the codebase), before the plan is passed to /orchestrate-dev.

Deliverables

File	Action	Purpose
packages/agents/content/skills/refine-plan/SKILL.md	Create	Main skill — orchestrates the review-and-revise flow
packages/agents/content/subagents/plan-reviewer.md	Create	Reviews plans for completeness and correctness
packages/agents/content/subagents/plan-reviser.md	Create	Incorporates findings and user answers into a revised plan
packages/agents/content/skills/save-artifact/SKILL.md	Modify	Add plan-review and plan-v2 artifact types
packages/agents/content/skills/_data/artifact-conventions.md	Modify	Add plan-review and plan-v2 to ticket-level list
packages/agents/content/subagents/_data/claude.yml	Modify	Add model: sonnet for both new subagents
packages/agents/content/subagents/_data/rovodev.yml	Modify	Add empty entries (inherit _defaults) for both new subagents

Implementation steps

Step 1: Register new artifact types

Files: packages/agents/content/skills/_data/artifact-conventions.md, packages/agents/content/skills/save-artifact/SKILL.md

Add two new ticket-level artifact types in alphabetical order after plan:

• plan-review — Plan review findings (completeness and correctness analysis)
• plan-v2 — Refined implementation plan after review and revision

In artifact-conventions.md (line ~534), insert between plan and pull-request.
In save-artifact/SKILL.md (line ~44), insert between plan and pull-request.

Step 2: Create plan-reviewer subagent

File: packages/agents/content/subagents/plan-reviewer.md

Frontmatter:

---
name: plan-reviewer
description: Review implementation plans for completeness and correctness. Outputs structured findings categorized as auto-resolvable or requiring user input.
tools: [Read, Grep, Glob, Bash, Write]
maxTurns: 30
skills:
  - development-workflows
  - software-engineering
---

Key design elements:

Role: Plan reviewer. Analyzes an implementation plan against its requirements and the actual codebase. Does NOT revise the plan — only identifies issues.

Finding scheme — C/X (not F/W/T/R/S/L):

• C{n} — Completeness findings (decision gaps)
• X{n} — Correctness findings (factual errors, structural issues)
• Each finding tagged auto or user for resolution routing

Rationale for a separate scheme: F/W/T/R/S/L is severity-oriented for code review where merge-blocking is the key decision. Plan review needs a resolution-oriented scheme where the key decision is "who resolves this — the reviser autonomously, or the user?"

Completeness criteria (C findings):

Focus	Check	Resolution
UX specification	Interactions, layouts, states, user flows specified?	user — UX decisions are preference-based
Technical decisions	Patterns, data structures, API shapes specified?	auto if codebase pattern applies; user if ambiguous
Behavioral specification	Edge cases, defaults, failure modes specified?	auto if conventions dictate; user if domain-specific

Correctness criteria (X findings):

Focus	Check	Resolution
Factual accuracy	Referenced files, utilities, APIs exist?	auto
Structural soundness	Dependencies ordered? Steps scoped correctly?	auto
Requirements traceability	Every requirement has a step? Every step traces to a requirement?	user if intentionally omitted; auto if overlooked

Output format:

# Plan review

## Summary
{1-3 sentence assessment}
**Findings:** {total} ({auto count} auto-resolvable, {user count} require user input)

## Auto-resolvable findings
### X1: {title}
- **Focus:** Factual accuracy
- **Description:** {what is wrong}
- **Evidence:** {file path or codebase pattern}

## Decision gaps
### C1: {title}
- **Focus:** UX specification
- **Description:** {what is missing}
- **Evidence:** {context}
- **Question:** {specific question for the user}

## Requirements coverage
| Requirement | Plan step | Status |
|---|---|---|
| {requirement} | Step {N} | Covered / Gap / Partial |

Return protocol:

Phase: plan-review
Status: completed|failed
Artifact: {path}
AutoResolvable: {integer}
UserQuestions: {integer}

Step 3: Create plan-reviser subagent

File: packages/agents/content/subagents/plan-reviser.md

Frontmatter:

---
name: plan-reviser
description: Revise implementation plans based on review findings and user answers. Produces a refined plan preserving the original structure while addressing all findings.
tools: [Read, Grep, Glob, Bash, Write]
maxTurns: 30
skills:
  - development-workflows
  - software-engineering
---

Key design elements:

Role: Plan reviser. Produces a refined plan incorporating review findings (auto-resolvable corrections) and user answers (decision gap resolutions). Preserves original structure and intent.

Process:

1. Read original plan, review findings, and user answers
2. For auto-resolvable findings: explore codebase to gather resolution information
3. For user-required findings: incorporate user answers
4. Produce revised plan matching original format
5. Append "Changes from original" summary table

Output: Complete plan document (not a diff) in the same format as the original, plus:

## Changes from original
| Finding | Resolution |
|---|---|
| C1: {title} | {how resolved} |
| X2: {title} | {how corrected} |

Format handling: Detects and preserves plan format:

• Prose plans → same markdown structure
• Orchestration plans → .md + .json companion (if original had one)

Return protocol:

Phase: plan-revision
Status: completed|failed
Artifact: {path}

Step 4: Create /refine-plan skill

File: packages/agents/content/skills/refine-plan/SKILL.md

Frontmatter:

---
name: refine-plan
description: Review and refine an implementation plan for completeness and correctness
user-invocable: true
---

Arguments:

• Plan file path (required): path to a saved plan artifact
• Ticket source (required): file path or URL (GitHub issue, Jira, etc.)

Flow (6 steps):

1. Validate inputs and resolve context

   • Verify plan file exists (Read). Error and stop if not.
   • Resolve ticket: GitHub URL → gh issue view; file path → Read; other URL → WebFetch.
   • Use get-branch-context for ticket_id and project_slug.
   • Resolve artifacts.base_dir (preferences cascade → default ~/.ai).
   • mkdir -p artifact directory.

2. Detect plan format

   • Read plan file. Detect prose vs orchestration by content structure.
   • Check for .json companion if orchestration format.

3. Dispatch plan-reviewer (Task tool, subagent_type: plan-reviewer, max_turns: 30)

   • Pass: plan path, format, ticket content, output path {dir}/{timestamp}_{slug}_plan-review.md
   • Extract slug from original plan filename (reuse for artifact grouping)
   • Parse return: Status, AutoResolvable, UserQuestions

4. Present user questions (if any)

   • If 0 user questions → skip to step 5 with empty answers
   • If 0 total findings → skip reviser entirely, report plan needs no refinement
   • Otherwise: read review artifact, extract decision gaps, present as numbered questions, wait for user response

5. Dispatch plan-reviser (Task tool, subagent_type: plan-reviser, max_turns: 30)

   • Pass: original plan path, review path, user answers, ticket content, output path {dir}/{new-timestamp}_{slug}_plan-v2.md
   • Parse return: Status, Artifact

6. Report completion

   Plan refined:
     Review:  {path to plan-review.md}
     Revised: {path to plan-v2.md}
   

Edge cases:

• Plan file not found → error and stop
• Ticket URL unreachable → error with suggestion
• Reviewer finds 0 findings → skip reviser, report plan is ready
• Reviewer finds only auto-resolvable → skip user interaction, proceed to reviser
• Reviewer or reviser fails → report failure and stop (no retry)

Step 5: Add platform overlays

Files: packages/agents/content/subagents/_data/claude.yml, packages/agents/content/subagents/_data/rovodev.yml

In claude.yml, append:

plan-reviewer:
  model: sonnet

plan-reviser:
  model: sonnet

In rovodev.yml, append:

plan-reviewer: {} # inherits _defaults

plan-reviser: {} # inherits _defaults

Key design decisions

1. Two subagents, not one — Follows the separation principle: reviewers find issues, producers fix them. Prevents the reviewer from softening findings to ease revision.

2. C/X finding scheme, not F/W/T/R/S/L — Plan review needs resolution routing (auto vs. user), not severity-based merge-blocking. Different domain, different scheme.

3. Natural conversation for user questions — Follows the plan-orchestrable-steps pattern: present questions as formatted text, wait for free-form response. More flexible than AskUserQuestion for open-ended decisions.

4. No MCP run tracking — This is a standalone skill producing ticket-level artifacts, not an orchestrated pipeline run. Aligns with issue scope.

5. Slug reuse from original plan — All three artifacts (plan, plan-review, plan-v2) share the same slug for visual grouping in file listings.

Verification

1. Install the agents package: cd packages/agents && pnpm run build
2. Typecheck: pnpm run ws typecheck (should pass — content files only, no TS changes)
3. Lint: pnpm run ws lint (should pass)
4. Manual test: Invoke /refine-plan {path-to-plan} {ticket-url} on an existing saved plan and verify:
   • Review artifact is saved at the expected ticket-level path
   • Decision gaps are presented as numbered questions
   • After answering, revised plan is saved with "Changes from original" table
   • Original plan is preserved unmodified
5. Content validation: Verify all frontmatter fields (name, description, tools, maxTurns, skills) follow existing conventions by comparing with orchestrated-planner.md and orchestrated-reviewer.md