Schema validation for run-index.json with Zod

[williamthorsen]

Description

The Factory status-adapter.ts validates run-index.json (v2) and status.json (v1) using ~177 lines of hand-rolled type-guard functions (isRecord, isOptionalString, isValidPhaseStatus, etc.). While this works and has comprehensive test coverage, it has several drawbacks:

• No reusable schema definition — the shape of run-index.json is documented only implicitly by TypeScript interfaces and validation functions scattered across status-adapter.ts
• No standalone validation tool — validating run output requires running the full Express server; there is no CLI script to validate files on disk
• Maintenance burden — every new field requires updating both a TypeScript interface and a corresponding type-guard function, with no compile-time guarantee they stay in sync
• No structured error reporting — validation throws a generic "Invalid run-index.json at {path}" with no detail on which field failed

This ticket introduces a schema-validation library, defines schemas for the run-index.json data structure, creates a standalone validation script, and refactors the status-adapter to use the schemas.

Proposed library: Zod

Zod is the proposed library:

1. TanStack Router compatibility — TanStack Router (anticipated addition to Factory) uses Zod for route-param validation out of the box
2. TypeScript-first — schemas infer TypeScript types directly via z.infer<>, eliminating duplication between interfaces and validation functions
3. Rich error reporting — ZodError provides structured, per-field error paths, messages, and codes
4. Ecosystem adoption — widely used with React Hook Form, tRPC, and other tools in the TypeScript ecosystem
5. Small footprint — tree-shakeable, no runtime dependencies

Alternatives considered

Library	Pros	Cons
Valibot	Smaller bundle (~1kB), similar API	Less ecosystem integration, no TanStack Router synergy
AJV + JSON Schema	Industry-standard, language-agnostic	Requires separate type generation, verbose, heavier runtime
Yup	Mature, well-known	Weaker TypeScript integration, larger bundle, declining momentum

If the team decides against Zod, Valibot is the closest alternative.

Architecture context

Data flow

Disk: run-index.json (v2) / status.json (v1)
  → status-adapter.ts: parseRunData() reads + validates + normalizes
    → CanonicalRunStatus (shared/types/canonical.ts)
      → project-scanner.ts: discovers runs, extracts status/startedAt
      → API routes → client

Key files

File	Role
packages/factory/src/server/adapters/status-adapter.ts	Validation + normalization (~320 lines, ~177 lines are type guards)
packages/factory/src/server/adapters/__tests__/status-adapter.test.ts	Comprehensive validation tests (~900 lines)
packages/factory/src/shared/types/canonical.ts	Domain model: CanonicalRunStatus, Phases, ArtifactEntry, etc. (18 exported types)
packages/factory/src/server/services/project-scanner.ts	Consumes parseRunData()
packages/factory/src/__test-helpers__/fixtures.ts	createMockRunStatus(), emptyPhases(), createCompletedRunPhases()

V2 run-index.json structure

{
  "version": 2,
  "context": {
    "runId": "string",           // required
    "projectSlug": "string",     // required
    "ticketId": "string?",       // optional
    "projectRoot": "string",     // required
    "branch": "string",          // required
    "task": "string",            // required
    "startedAt": "string",       // required (ISO datetime)
    "completedAt": "string?",    // optional, nullable
    "status": "RunStatus",       // enum: in_progress | completed | failed | needs_manual_review
    "phases": {                  // each value is nullable (null = not yet reached)
      "architecture": { "status": "PhaseStatus", "impactLevel?": "string", "artifact?": "string" },
      "planning": { "status": "PhaseStatus", "stepCount?": "number", "artifacts?": ["string"] },
      "implementation": { "status": "PhaseStatus", "artifact?": "string", "qualityGates?": "..." },
      "parallelReview": { "aggregatedCriticality": "Criticality", "reviewRoundsUsed": "number", "..." },
      "review": { "status": "PhaseStatus", "iterations?": "number", "finalCriticality?": "Criticality" },
      "codeSimplifier": { "ran": "boolean", "actionableFindings": "boolean", "..." },
      "holisticReview": { "status": "PhaseStatus", "criticality?": "Criticality", "..." }
    },
    "phaseDecisions?": { "[phase]": { "run": "boolean", "reason?": "string" } }
  },
  "config": {
    "externalPlan?": "boolean",
    "mergeBaseSha?": "string",
    "diffBase?": "string",
    "maxReviewRounds?": "number",
    "fixLowFindings?": "boolean",
    "mode?": "string",
    "model?": "string"
  },
  "artifacts?": [
    {
      "filename": "string",      // required
      "role": "string",          // required
      "roleType": "string",      // required
      "agent": "string",         // required
      "type": "string",          // required
      "phase": "string",         // required
      "createdAt": "string",     // required
      "iteration?": "number",    // optional
      "note?": "string"          // optional
    }
  ]
}

Enum values

• RunStatus: in_progress, completed, failed, needs_manual_review
• PhaseStatus: completed, skipped, failed, in_progress, approved
• Criticality: none, low, medium, high
• ReviewerStatus: completed, skipped, failed

Validation behaviors to preserve

These behaviors are verified by the existing 900-line test suite:

• null phase entries are valid (phase not yet reached)
• completedAt accepts null, undefined, or string
• phases object may contain unknown phase keys (forward-compatible)
• Bare {} phase entries are accepted
• phaseDecisions entries require run: boolean; reason is optional
• Phase entry fields status, criticality, finalCriticality, aggregatedCriticality accept null or their respective enum values
• Artifact entries require all 7 base fields; iteration and note are optional

Run-index.json file locations

Run-index.json files are produced by the orchestrated-development pipeline and stored at:

~/.ai/projects/{projectSlug}/tickets/{ticketId}/{runId}/run-index.json

Key design decisions

1. Schemas live in server/adapters/schemas/ — V2RunIndex and V1StatusObject describe on-disk formats read only by the server. Client code never touches raw formats.

2. canonical.ts types stay hand-written — CanonicalRunStatus is a flat domain model; V2RunIndex is nested (context + config). The normalizeV2/normalizeV1 functions map between these shapes. Replacing canonical types with z.infer<> would require complex transforms and touch 15+ importing files. Additionally, exactOptionalPropertyTypes is enabled — ArtifactEntry uses iteration?: number (property can be absent), which differs from Zod's .optional() producing T | undefined (must be present).

3. Phase entries use .passthrough() for forward-compatibility — The current validation allows unknown keys in phase objects and only validates known ones. A z.object({...}).passthrough() preserves this.

4. Zod is a runtime dependency — Schemas are used at runtime in the server adapter, not just at build time. However, since schemas are in server/adapters/, they are not included in the client bundle (Vite tree-shakes them).

Acceptance criteria

Must have

• Add zod as a runtime dependency in packages/factory/ with exact version pinning
• Create V2 run-index.json Zod schema defining all structures (RunIndex, Context, Config, Phases, ArtifactEntry, enums)
• Create V1 status.json Zod schema reusing shared enum/phase schemas
• Schema tests covering all enum values, optional/nullable fields, forward-compatible phase entries, and meaningful Zod error paths
• Refactor status-adapter.ts to replace hand-rolled type guards with Zod .safeParse() — preserve public API and error messages
• All existing status-adapter tests pass unchanged
• Create CLI validation script that validates single files or scans directories, reports per-field errors, and exits non-zero on failure
• Register script as pnpm run validate:run-index in factory package.json
• CLI script tests

Should have

• Summary output when scanning directories (total files, passed, failed)

Nice to have

• JSON Schema export from Zod schemas for language-agnostic documentation
• Integration with project-scanner.ts to surface structured validation errors instead of silently skipping invalid runs

[williamthorsen]

Implementation plan (from planning session, 2026-02-27)

These notes capture initial exploration and a proposed approach. The implementer should verify assumptions and adapt as needed.

Implementation plan

Key design decisions

1. Schemas live in server/adapters/schemas/ — V2RunIndex and V1StatusObject are on-disk formats read only by the server. Client code never touches them.

2. canonical.ts types stay hand-written — The canonical types are a different shape from the raw JSON (flat vs nested). Replacing them with z.infer<> would require complex transforms and touch 15+ importing files. Schemas validate the wire format; normalizeV2/normalizeV1 transform to canonical.

3. Phase entries use .passthrough() for forward-compatibility — The current validation allows unknown keys in phase objects and only validates known ones (status, criticality, etc.). A z.object({...}).passthrough() preserves this behavior idiomatically.

4. CLI script at src/scripts/ — Top-level within the package for discoverability, registered as a package.json script.

Step 1: Install Zod

Add zod (exact version) to packages/factory/package.json dependencies and run pnpm install.

Files: packages/factory/package.json, pnpm-lock.yaml

Step 2: Create V2 run-index.json Zod schema

New file: packages/factory/src/server/adapters/schemas/run-index-schema.ts

Define and export:

Schema	Description
runStatusSchema	z.enum(['in_progress', 'completed', 'failed', 'needs_manual_review'])
phaseStatusSchema	z.enum(['completed', 'skipped', 'failed', 'in_progress', 'approved'])
criticalitySchema	z.enum(['none', 'low', 'medium', 'high'])
phaseEntrySchema	z.object({ status, criticality, finalCriticality, aggregatedCriticality }).partial().passthrough() — each field is z.union([z.null(), enumSchema]).optional()
phasesSchema	z.record(z.union([z.null(), phaseEntrySchema])) — unknown phase keys allowed
phaseDecisionSchema	z.object({ run: z.boolean(), reason: z.string().optional() })
phaseDecisionMapSchema	z.record(phaseDecisionSchema)
artifactEntrySchema	7 required string fields + optional iteration: number, note: string
v2ContextSchema	Required: runId, projectSlug, projectRoot, branch, task, startedAt, status. Optional: ticketId, completedAt (string | null), phaseDecisions. Contains phases.
v2ConfigSchema	All 7 fields optional (externalPlan, mergeBaseSha, diffBase, maxReviewRounds, fixLowFindings, mode, model)
v2RunIndexSchema	{ version: z.literal(2), context, config, artifacts?: ArtifactEntry[] }

Step 3: Create V1 status.json Zod schema

New file: packages/factory/src/server/adapters/schemas/status-json-schema.ts

Flat structure reusing runStatusSchema, phasesSchema, phaseDecisionMapSchema from the run-index schema. V1 uses phaseDecision (singular) and has no mode/model/artifacts fields.

Step 4: Create schema tests

New file: packages/factory/src/server/adapters/schemas/__tests__/run-index-schema.test.ts

Cover:

• Valid V2 data parses; all enums accepted/rejected
• Phase entries: null accepted, bare {} accepted, invalid status/criticality rejected, unknown keys pass
• completedAt: string, null, undefined all valid
• version: 2 required
• Artifacts: required fields enforced, optional iteration/note
• Zod error paths are meaningful (e.g., context.status)

New file: packages/factory/src/server/adapters/schemas/__tests__/status-json-schema.test.ts

Analogous coverage for V1 format.

Step 5: Refactor status-adapter.ts to use Zod schemas

Modified file: packages/factory/src/server/adapters/status-adapter.ts

Replace the 177 lines of hand-rolled validation with:

import { v2RunIndexSchema } from './schemas/run-index-schema.js';
import { v1StatusSchema } from './schemas/status-json-schema.js';

// Derive internal types from Zod (module-private, no blast radius)
type V2RunIndex = z.infer<typeof v2RunIndexSchema>;
type V1StatusObject = z.infer<typeof v1StatusSchema>;

Each parse function becomes:

const result = v2RunIndexSchema.safeParse(raw);
if (!result.success) {
  throw new Error(`Invalid run-index.json at ${filePath}`);
}
return normalizeV2(result.data);

Keep: parseRunData, parseRunIndex, parseStatusFile (public API), normalizeV1, normalizeV2 (transformation logic).

Remove: All isValid*, isOptional*, isRecord, hasValid* functions plus the three VALID_* Sets. Net ~170 lines removed.

Error messages preserved: Existing tests check for 'Invalid run-index.json at {path}' — the refactored code throws the same strings.

Step 6: Verify existing tests pass unchanged

File: packages/factory/src/server/adapters/__tests__/status-adapter.test.ts (902 lines)

Run full test suite. All tests should pass with no changes since the adapter's public API and error messages are unchanged. If any edge-case differences emerge between Zod's validation and the hand-rolled guards, adjust the schema (not the tests) to preserve current behavior.

Step 7: Create CLI validation script

New file: packages/factory/src/scripts/validate-run-index.ts

• Accepts a path argument (single file or directory)
• For directories: recursively finds all run-index.json files
• Validates each against v2RunIndexSchema.safeParse()
• Reports per-file: PASS/FAIL with Zod error details (field paths + messages)
• Exits non-zero if any file fails
• Summary line: N passed, M failed out of T files

Step 8: Register CLI script

Modified file: packages/factory/package.json

Add script entry:

"validate:run-index": "tsx src/scripts/validate-run-index.ts"

Usage: cd packages/factory && pnpm run validate:run-index ~/.ai/projects/codeassembly

Step 9: Add CLI script tests

New file: packages/factory/src/scripts/__tests__/validate-run-index.test.ts

Test the core validation/scanning logic (not the process entry point) by importing the library functions with mocked fs.

Files summary

File	Action
packages/factory/package.json	Modify (add zod dep + script)
packages/factory/src/server/adapters/schemas/run-index-schema.ts	Create (~90 lines)
packages/factory/src/server/adapters/schemas/status-json-schema.ts	Create (~25 lines)
packages/factory/src/server/adapters/schemas/__tests__/run-index-schema.test.ts	Create (~200 lines)
packages/factory/src/server/adapters/schemas/__tests__/status-json-schema.test.ts	Create (~100 lines)
packages/factory/src/server/adapters/status-adapter.ts	Modify (−170 lines)
packages/factory/src/scripts/validate-run-index.ts	Create (~60 lines)
packages/factory/src/scripts/__tests__/validate-run-index.test.ts	Create (~60 lines)

Not modified: canonical.ts, api.ts, all 15 client/server files importing canonical types, run-workspace-script.ts.

Verification

1. pnpm run ws test — all existing + new schema tests pass
2. pnpm run ws typecheck — no type errors
3. pnpm run ws lint — clean
4. pnpm run ws check — full quality gate
5. cd packages/factory && pnpm run validate:run-index ~/.ai/projects/codeassembly — validates real run data on disk
6. Manual: confirm a deliberately malformed run-index.json reports per-field errors