Lede guidance can only be evaluated by shipping it

[williamthorsen]

Problem

Four generations of lede doctrine have been designed by reasoning and evaluated by shipping. The first production lede exposes the failure, revisions burn author attention in the middle of shipping real code, and each diagnosis becomes the next rule. Nothing can compare guidance variants against the same input, attribute a good lede to the ingredient that produced it, or evaluate a candidate without contending for the capacity that concurrent live sessions share.

Context

The drafting context that matters is not a cold prompt. The common production shape is one session that runs /review-branch (sometimes a re-review), then /create-pr, then /merge-pr, so the lede is composed inside the conversation that holds the full review. That contamination is a suspected driver of catalog ledes, and some doctrine exists only to counter it; a candidate evaluated cold is optimized for the wrong context.

Candidates are data, not environment: numbered variants installed side by side, selected per run by a read-only chooser, so nothing installed is ever swapped, concurrent sessions cannot interfere, and no sandboxed harness home is needed.

The decision corpus (capture-lede-decision) supplies fixtures: merged changes whose diffs, artifacts, and shipped ledes are recoverable. While the experiment runs, production drafting draws incidentally from the same candidate pool -- no worse than the incumbent's 6-of-7 revision rate -- but production decisions are not captured and form no part of the evaluation.

Proposed solution

A harness of isolated, modular scripts, generic over which guidance file is under test.

• Candidate directory: numbered variants, each carrying a short manifest naming its settings on the axes under test (guidance mass, exemplars none/finals/pairs, stance question vs rules), so survey results attribute to ingredients. Generation 1 spans the space widely -- a minimal drafting ask, a reader-stance variant, the incumbent doctrine as control, and a stance-plus-exemplars variant -- with later generations composed from winning ingredients to shrink the variance. Every candidate carries the four-rule conventions floor (lede voice, tier and register, title discipline, migration sentences), authored once as a partial and inlined at install time.
• Fixture: a worktree at a merged change's pre-merge state plus a frozen pre-draft session, built once by running the real review-then-create-pr sequence and stopping just before the lede is composed; reusable indefinitely.
• Runner: each iteration forks the frozen session headlessly, composes the ## What reading the chooser-named candidate, and appends the draft to a results ledger under an opaque run id. Batch size is a parameter, so the operator controls capacity contention.
• Survey and tally: drafts presented per fixture, blind and shuffled; preferences recorded; unmasking last; tallies by candidate and by axis.
• Production wiring: summarize-change's doctrine-read step resolves through the chooser, deterministically per branch.

Acceptance criteria

Must have

• Numbered candidate files with axis manifests exist, and generation 1 holds at least four materially different variants including the incumbent doctrine.
• Every candidate carries the conventions floor, authored once as a partial and inlined at install time.
• A fixture is built from a merged change once and reused indefinitely: worktree state plus a frozen pre-draft session carrying the review conversation.
• An iteration forks the frozen session, never mutates it, and has no side effects outside the harness's results directory.
• Drafts are recorded under opaque run ids, and the id-to-candidate mapping is not visible until preferences are recorded.
• The survey presents each fixture's drafts blind and shuffled, and the tally reports preferences by candidate and by axis.
• Production drafting resolves its doctrine through the chooser; the chooser is read-only and deterministic per branch.
• The guidance file under test is a harness parameter, not a hardcoded path.
• New/modified behavior in this change is covered by tests.
• Scripts carry usage help, and the harness carries a README covering the candidate-manifest format and the fixture-building flow.

Should have

• A cold mode runs the same candidate and fixture without the frozen session, so the context effect is measurable.
• The runner accepts an authentication override, so a batch can bill to an API key instead of the subscription.