Replace LLM-driven run-index.json assembly with MCP orchestration tools and structured return contracts

[williamthorsen]

Problem

The orchestrate skill's run-index.json is assembled entirely by the orchestrator LLM. At every phase boundary, the orchestrator must:

1. Parse unstructured text from subagent Task returns (looking for Key: value lines)
2. Construct JSON updates to run-index.json with the correct schema
3. Fall back to reading and parsing artifact files when Task return parsing fails
4. Maintain cumulative state (iteration counts, reviewer maps, aggregated criticality) across a long conversation

This works, but it's fragile in predictable ways:

• Parse drift: subagents occasionally format return values differently (extra whitespace, markdown formatting, omitted fields), triggering fallback logic that itself requires LLM interpretation.
• Schema drift: the orchestrator sometimes writes run-index.json fields with slightly wrong shapes — e.g., nesting reviewers one level too deep, using inconsistent timestamp formats, or omitting roleType on artifact entries. These are detectable by a consumer but shouldn't happen at all.
• Cumulative state errors: by Phase 4b the orchestrator is tracking review round counts, per-reviewer criticality histories, fix-cycle outcomes, and selective re-review decisions. The LLM sometimes loses track — miscounting iterations, forgetting to update a reviewer's reReviewCriticality, or recording coderFixCycleRan: false when one did run.
• Non-determinism: two identical runs can produce structurally different run-index.json files (different field ordering, inconsistent null vs. absent, varying artifact entry completeness) because the JSON is free-text composed by the LLM each time.

The root cause is that run-index.json construction is a deterministic bookkeeping task being performed by a non-deterministic actor. The orchestrator should be making decisions (which phase to run, how to interpret findings); it should not be hand-writing JSON.

Proposed solution: Structured return contracts + MCP orchestration tools

Replace the current text-parsing pipeline with two complementary mechanisms:

1. Structured return contracts for subagents

Define a formal return schema per subagent role. Instead of the orchestrator parsing free-text Key: value lines from Task returns, each subagent returns a structured object with validated fields.

Per-role contracts:

Role	Required return fields	Types
architect	phase, status, impact, artifact	string, enum(completed|failed), enum(none|low|medium|high), string
planner	phase, status, steps, artifact	string, enum(completed|failed), integer, string
coder	phase, status, qualityGates, artifact	string, enum(completed|failed), enum(passed|failed|skipped), string
reviewer (all types)	phase, status, criticality, artifact	string, enum(completed|failed), enum(none|low|medium|high), string

This replaces the current "Task return parsing" section of the orchestrate skill. The orchestrator reads typed fields instead of regex-matching text lines. Fallback to artifact-file parsing is eliminated (or becomes a true error path rather than a routine one).

2. MCP tools for run-index.json lifecycle

Expose run-index.json management as a set of MCP tools. The orchestrator calls tools with typed parameters; the tools validate inputs and perform atomic, schema-correct JSON updates.

Proposed tool surface:

orchestration_init_run(
  projectSlug: string,
  ticketId: string,
  projectRoot: string,
  branch: string,
  task: string,
  model: string,
  pipeline: string[],
  config: { externalPlan, mergeBaseSha, diffBase, maxReviewRounds, fixLowFindings }
) → { runId, runDir, artifactDir }

Initializes run-index.json with version, context, config, timestamps. Returns computed paths. Deterministic — no LLM interpretation needed.

orchestration_record_phase_decision(
  runId: string,
  phase: string,           // e.g., "architecture", "parallelReview"
  run: boolean,
  disposition: "executed" | "skipped" | "absent",
  reason?: string
) → { ok }

Writes a validated phaseDecisions entry. Rejects unknown phase names (enforcing the pipeline spec).

orchestration_record_phase_result(
  runId: string,
  phase: string,
  data: object             // phase-specific; validated per phase schema
) → { ok }

Writes a validated phases entry. The data parameter is validated against a per-phase schema:

• architecture: { impact, guidanceProvided }
• planning: { steps, externalPlanDeviations? }
• implementation: { status, qualityGates }
• parallelReview: { aggregatedCriticality, reviewers: { [key]: { status, criticality, reReviewCriticality? } }, coderFixCycleRan, selectiveReReview, iterationCount }
• codeSimplifier: { ran, actionableFindings, coderFixCycleRan }
• holisticReview: { criticality, coderFixCycleRan }

orchestration_register_artifact(
  runId: string,
  filename: string,
  role: string,
  agent: string,
  type: string,
  phase: string,
  iteration?: number,
  note?: string
) → { ok }

Appends a validated artifact entry with auto-generated createdAt and auto-resolved roleType (from the role→roleType taxonomy already defined in artifact-conventions.md).

orchestration_complete_run(
  runId: string,
  status: "completed" | "failed" | "needs_manual_review"
) → { completedAt, finalStatus }

Sets completedAt timestamp and final status. Validates that all pipeline phases have either a phaseDecisions entry or a phases entry.

orchestration_get_run_state(
  runId: string
) → { full run-index.json contents }

Read-only accessor. The orchestrator uses this to check cumulative state (e.g., remaining review rounds) instead of maintaining it in conversation memory.

What changes in the orchestrate skill

The orchestrate skill instructions would change from:

"Parse the Task return for Impact: {value}. If missing, read the artifact file and look for ### Impact level:. Record a parseWarning in run-index.json..."

To:

"Read the impact field from the architect's structured return. Call orchestration_record_phase_decision(phase='architecture', run=true, disposition='executed'), then orchestration_record_phase_result(phase='architecture', data={ impact, guidanceProvided }). Call orchestration_register_artifact(...) for the architecture artifact."

The orchestrator still makes all the same decisions (should this phase run? what does this criticality mean for flow control?). It just stops doing JSON bookkeeping.

What changes for subagents

Each subagent's prompt adds a structured return block. For example, the orchestrated-architect's prompt currently ends with a free-text return. It would instead end with a contract like:

## Return contract
When complete, return these fields:
- Phase: architecture
- Status: completed | failed
- Impact: none | low | medium | high
- Artifact: {filename}

The mechanism for structured returns depends on the Task tool's capabilities. If the Task tool supports structured output schemas, use them. If not, the MCP tools can accept the raw Task return string and parse it with a deterministic parser (not LLM interpretation) — the key difference being that parsing moves from the orchestrator's conversation to a validated tool.

Implementation approach

Phase 1: MCP tool server

Build the orchestration tools as an MCP server (or extend an existing one). The tool implementations are straightforward file I/O with JSON Schema validation:

• Store a runs/ registry mapping runId → runDir for path resolution
• Each write tool reads the current run-index.json, validates the update, applies it, and writes back
• Validation rejects unknown phases, invalid enum values, duplicate artifact entries, etc.
• All timestamps are generated server-side (UTC ISO 8601), eliminating format inconsistency

Phase 2: Structured return contracts

Update subagent prompts to include return contracts. Update the orchestrate skill to read structured fields instead of parsing text. Remove the "Task return parsing" and "Fallback logic" sections.

Phase 3: Migrate orchestrate skill instructions

Rewrite phase-by-phase instructions to use tool calls instead of JSON construction. The skill becomes significantly shorter — phase boundary logic reduces to tool calls plus flow-control decisions.

Scope

• The MCP tools, structured return contracts, and orchestrate skill updates
• Updates to artifact-conventions.md for the new tool-based workflow
• Updates to codeassembly consumer code (status-adapter.ts, canonical.ts, etc.) if the run-index.json schema changes — but the goal is schema consistency, not schema change, so consumer updates should be minimal

Out of scope

• Changes to which phases exist or how flow-control decisions are made (those stay in the skill)
• Changes to the artifact file format (markdown artifacts are unchanged)
• Changes to how subagents do their actual work (review, code, plan)

Success criteria

• run-index.json files from orchestrated runs are schema-valid 100% of the time (vs. current ~90%)
• No parseWarning entries in run-index.json (the concept is eliminated)
• The orchestrate skill's phase-boundary instructions are ≤50% of their current token count
• Cumulative state (review iteration counts, per-reviewer histories) is always correct because it's read from the file via orchestration_get_run_state() rather than tracked in conversation memory

[williamthorsen]

Closing as duplicate of #74. The analysis in this issue informed two additions to the #74 implementation plan: (1) formalized subagent return contracts (Step 4c) and (2) get_run_state for cumulative flow-control decisions (Step 4d). Both are attributed to #83 in the plan's decisions table.