Include rich summary in orchestrated run output

[williamthorsen]

Problem

At the end of an orchestrated run, the orchestrator writes a run-summary.md artifact that contains a phase-status table and file list, but the conversational summary it outputs to the user is significantly richer — it includes a narrative of what was built, insights, and deferred items. This richer summary is ephemeral (lost when the conversation ends) while the artifact persists but is less informative.

The summary artifact should capture interpretive, ephemeral content that can't be regenerated from structured data or git history.

Design considerations

The original draft proposed four new sections. Through refinement, two were dropped:

Original section	Disposition	Rationale
Metrics table	Dropped	Git-derivable stats (files, commits) and structured data (review rounds) already persist elsewhere
Review findings addressed	Transformed	Became "Insights" — notable observations, not a rote finding→resolution ledger
What was built	Kept	Synthesized narrative of end-to-end result — ephemeral without this
Deferred items	Expanded	Absorbs existing "Deviations from reference plan" + adds skipped acceptance criteria

Key principle: the summary should uniquely preserve content that can't be regenerated later. Git-derivable statistics and data already in run-index.json or reviewer artifacts are noise — interpretive synthesis is the value.

The coder's change-summary.md artifact covers only a single iteration. The orchestrator sees all iterations (initial implementation + review fix cycles) and is uniquely positioned to write a synthesized end-to-end narrative.

Proposed solution

Important note: This document will be accessible from the visualizations! So it should contain the kind of information that would be useful to a developer who has scheduled an orchestrated run and wants to know "What happened? What was done? What did we get out of this run?"

Update the Phase 5 (Summary) section of the orchestrate skill (packages/agents/content/skills/orchestrate/SKILL.md) to add three sections to the run-summary.md template:

1. What was built

A synthesized narrative of the end-to-end result across all coder iterations. Describes what each major component or subsystem does and why — not just file paths. Draws from accumulated context across all coder change-summaries and review outcomes. Focuses on the final state, not the iteration history.

If the run failed or needs manual review, describes what was completed and what remains.

2. Insights

Notable observations that emerged during the run. Only items worth preserving — the section is omitted entirely if nothing notable emerged. Examples:

• Architectural patterns discovered or validated
• Design trade-offs surfaced during review
• Conventions or project-specific patterns learned
• Surprising findings from reviewers that revealed something non-obvious
• Technical debt or risks identified but not in scope to address

3. Deferred items

Items intentionally not addressed during the run, with rationale. Replaces and expands the existing conditional "Deviations from reference plan" block. Covers:

• Deviations from reference plan (when external plan was provided)
• Acceptance criteria from the ticket that were intentionally not addressed
• Any other intentional omissions

Omitted when nothing was deferred.

Implementation

The changes are confined to packages/agents/content/skills/orchestrate/SKILL.md, specifically the Phase 5 template. No changes to subagent definitions, the review-cycle module, run-index.json schema, or artifact-conventions.md are needed — all data is already available in the orchestrator's conversation context at summary time.

The orchestrator should also output the same rich summary in the conversation so that it matches the persistent artifact.

Acceptance criteria

• run-summary.md includes a "What was built" section with a synthesized end-to-end narrative derived from accumulated coder context
• run-summary.md includes an "Insights" section with notable observations (omitted when nothing notable emerged)
• run-summary.md includes a "Deferred items" section covering plan deviations and skipped acceptance criteria (omitted when nothing was deferred)
• The same rich summary is output to the user in the conversation (matching the artifact content)

[williamthorsen]

Implementation plan

Context

At the end of an orchestrated run, the orchestrator writes a run-summary.md artifact containing a phase-status table and file list. The conversational summary it outputs is significantly richer — narrative, insights, deferred items — but ephemeral. This change makes the persistent artifact as informative as the conversation by adding three sections to the Phase 5 template.

The original issue proposed four new sections. Through refinement, we dropped two (metrics table, per-finding resolution log) because they duplicate data already available in git history and reviewer artifacts. The remaining three sections capture interpretive, ephemeral content — the kind that can't be regenerated from structured data.

Scope

Single file change: packages/agents/content/skills/orchestrate/SKILL.md, Phase 5 template only.

No changes to subagent definitions, review-cycle module, run-index.json schema, or artifact-conventions.md. All data sources are already available in the orchestrator's conversation context at summary time.

Implementation

Step 1: Update the Phase 5 template

Replace the existing Phase 5 template (lines 378–408 of SKILL.md) with the expanded version below. The existing sections (Task, Run, Phases) are unchanged. Three new sections are inserted, and "Files changed" moves to the end.

New template:

# Orchestration summary

## Task

{task description}

## Run

- **Run ID:** {run-id}
- **Ticket:** {ticket-id}
- **Status:** {completed|failed|needs_manual_review}
- **Duration:** {start} → {end}

## Phases

| Phase           | Status                         | Notes                                                               |
| --------------- | ------------------------------ | ------------------------------------------------------------------- |
| Architecture    | {ran/skipped}                  | {impact level or skip reason}{if external plan: ", plan validated"} |
| Planning        | {ran/skipped}                  | {step count or skip reason}{if external plan: ", N deviations"}     |
| Implementation  | {completed/failed}             |                                                                     |
| Review          | {approved/needs_manual_review} | {aggregated criticality, reviewers with findings, re-review ran}    |
| Code simplifier | {ran/skipped}                  | {actionable findings, fix cycle ran/not needed}                     |
| Holistic review | {ran/skipped}                  | {criticality, late-stage fixes}                                     |

## What was built

{Synthesized narrative of the end-to-end result. Describe each major component or subsystem that was implemented — what it does and why, not just file paths. Draw from the accumulated context across all coder change-summaries and review outcomes. Focus on the final state, not the iteration history.}

{If the run failed or needs manual review, describe what was completed and what remains.}

## Insights

{Notable observations that emerged during the run. Include only items worth preserving — omit this section entirely if nothing notable emerged.}

{Examples of what belongs here:}
{- Architectural patterns discovered or validated}
{- Design trade-offs surfaced during review}
{- Conventions or project-specific patterns learned}
{- Surprising findings from reviewers that revealed something non-obvious}
{- Technical debt or risks identified but not in scope to address}

## Deferred items

{Items intentionally not addressed during this run, with rationale for each.}

{Include:}
{- Deviations from reference plan (when external plan was provided)}
{- Acceptance criteria from the ticket that were intentionally not addressed}
{- Any other intentional omissions}

{Omit this section if nothing was deferred.}

## Files changed

{from git diff --name-only}

Step 2: Add conversational output instruction

After the template block, add an instruction for the orchestrator to output the same summary content in the conversation:

After writing the artifact, present the same summary to the user in the conversation. The conversational output should match the artifact content — do not abbreviate or omit sections.

What changed from the original proposal

Original section	Disposition	Rationale
Metrics table	Dropped	Git-derivable stats (files, commits) and structured data (review rounds) already persist elsewhere
What was built	Kept	Synthesized narrative of end-to-end result — ephemeral without this
Review findings addressed	Transformed	Became "Insights" — notable observations, not a rote finding→resolution ledger
Deferred items	Expanded	Absorbs existing "Deviations from reference plan" + adds skipped acceptance criteria

Verification

1. Read the updated SKILL.md and confirm the Phase 5 template includes all three new sections with clear guidance for the orchestrator
2. Confirm the template placeholders use consistent formatting with the rest of SKILL.md
3. Run pnpm run ws typecheck and pnpm run ws lint in the agents package (content-only change, but verify no issues)
4. Run pnpm run ws test in the agents package

[williamthorsen]

Implementation notes

PR: #88

Template authoring pattern: single vs. fragmented placeholder blocks

The code-simplifier caught a readability issue in the initial implementation: the ## Insights and ## Deferred items sections used multiple separate {...} placeholder blocks (one per paragraph/bullet) instead of wrapping each section's instructions in a single block. Fragmented blocks can confuse the orchestrator about instruction boundaries — a label like {Examples of what belongs here:} on its own line looks like it might be literal output rather than an instruction. Consolidating into single blocks per section is the clearer pattern for template authoring in SKILL.md.

Aspect reviewer activation on content-only changes

Only 2 of 4 aspect reviewers activated for this markdown-only change. The aspect-silent-failure-reviewer and aspect-test-reviewer correctly skipped based on file-type pattern rules (.md is not in the source-code extension list). This is the expected behavior — the activation logic prevents unnecessary review cycles on non-code changes while still running the always-on reviewers (core + aspect-code).

External plan step filtering

The reference plan included a Step 3 to update the GitHub issue with revised acceptance criteria. The planner correctly identified this as out of scope for the coder agent and dropped it from the implementation plan. Issue management is an orchestrator/human responsibility, not a code-change concern. This is a good pattern to follow when plans include non-code actions.