Capture lede decisions as an accumulating corpus

[williamthorsen]

Problem

Every lede rewrite discards the knowledge that produced it. A ## What section gets replaced with a better one, and the pair -- what was wrong, what fixed it -- exists only for the moment of the edit. Nothing accumulates, so doctrine refinement has no evidence base and proceeds from recall.

Context

#1056's blind comparison returned 20-0 for the current doctrine against no doctrine, which changes the question a corpus has to answer: not whether the doctrine helps, but which of its rules are load-bearing and which are accretion. A record of what still went wrong under the doctrine already in force answers that directly.

Both texts are already on disk. The _pull-request.md artifact holds the ## What the agent published, and the _merge.md artifact holds the lede that actually merged, which merge-pr sources from the final pull-request body. Across 501 ticket directories, 225 pairs are comparable and 86 differ. What is missing is not the text but the author's judgment about it.

That judgment cannot be recovered from the text. Of the 86 differing pairs, roughly a third are heavy rewrites that partly reflect content changes from commits landed during review rather than lede refinements. A lede shipped unchanged is likewise ambiguous between "this was good" and "there was no time to fix it" -- the expedience that makes the existing pairs unfit as models.

Proposed solution

Record a lede decision at merge, after the merge completes: the agent's lede, the merged lede, the doctrine version in force, and an optional comment.

The corpus stores positive signals only. A record exists because the author looked and decided -- accepted the agent's lede, or revised it. Declining to decide writes nothing, and the absence of a record carries no meaning; in particular it is not an acceptance. A busy week then yields less data rather than misleading data, which is the failure that makes the existing pairs unusable.

Records land in the shared knowledge substrate alongside the knowledge base and the agent-mistake record, not in a store of their own.

Four alternatives were considered and rejected:

• No backfill of the 86 existing pairs. They were edited for expedience rather than as models.
• Merge is the only capture point. The changelog-writer dispatch is not one: no skill dispatches its rewrite mode, and the caller's pruned-draft population is 10 pairs across the whole history.
• The comment is free text, not a doctrine-rule picker. A fixed rule list at capture time would presuppose which rules matter, which is the question the corpus exists to answer. Categorization belongs to the refinement pass reading the corpus.
• No second candidate and no quality rating. An A/B between two independent samples, and a rating on accepted ledes, are both worth revisiting once the single-decision record is earning its keep.

Acceptance criteria

Must have

• A decision record pairs the agent's lede with the merged lede and carries the work type, tier, and scope of the change, provenance reaching the change it described, and a fingerprint of the doctrine version in force.
• A record is written only when the author affirms a decision. Declining writes nothing, and a merge that never reaches the prompt leaves no trace.
• A lede accepted unchanged is recorded as distinctly as one that was revised.
• The author can explain the decision in free text, and can decline to.
• Records are queryable as a group and by work type, so a refinement pass can read them without walking the store.
• The store states that its contents are outcome-selected, that absence of a record is not an acceptance, and that neither may seed a comparison's fixture draw.
• Records group by doctrine version with nothing recorded at install time.
• New and modified behavior is covered by tests.
• The capture command and the record shape are documented where a later refinement pass can find them.

Should have

• A pull request merged outside the merge flow can still be recorded.

Related: #1056 (supplied the 20-0 result that reframes the question), #1055 (consumes decision records in place of the twenty hand-built golds), #1090 (the general doctrine derived downstream).

[williamthorsen]

Direction refined after the #1056 run. Four additions, one withdrawal.

The batch gold-set approach is dropped: it produced a preference with no reason attached, at the slowest possible rate of accumulation. The replacement is to stop running experiments and instrument the work that already happens.

1. Revision rate is the zero-tax metric, and should come first

Edit distance between the lede the agent emitted and the one that actually shipped in the PR body. It costs the author nothing, since they already revise or don't. It is collected on every PR whether or not anyone engages with the rest of this, it segments by work type, tier, and scope, and it answers #1055 directly: "the doctrine version that landed on 2026-07-19 needed rewriting on N% of PRs; the version before it needed M%."

2. Do not hook the edit; reconcile against the result

Capturing the author's direct edits to a published PR body looked like the hard part. Inverting it removes the problem: the agent records what it emitted along with a run id, and a later batch job diffs that against gh pr view --json body. The refinement pair falls out of reconciliation whenever it runs, with no instrumentation of the editing itself and no hook on the author's tools.

3. Two ledes per PR, author picks

At 5-20 PRs a day this accumulates 150-600 judgments a month, which removes the statistical-power ceiling that made a batch design useless (at n=20 only effects above roughly 75% preference are detectable at all).

Two variants must be kept distinct, because they are different experiments:

• Two samples from the same guidance. No independent variable; this measures the author's preferences rather than the guidance. Its value is the corpus: the systematic differences between chosen and rejected entries are the implicit doctrine, and that is the "because" Build a gold set and measure the lede guidance stack #1056 failed to capture.
• One sample from current guidance, one from a variant. A real continuous A/B, and what the versioning below makes measurable.

The first carries a benefit independent of measurement: best-of-2 sampling improves every PR shipped, today. The cost is a forced choice on every PR, and fatigued picks are noisy picks, so this may want sampling rather than always-on. Set against that, revision rate alone under-measures whenever a mediocre lede is accepted rather than fixed, which is an argument for having both.

4. Version the doctrine

A version marker on lede-voice.md, concision.md, and changelog-writer.md, stamped onto every captured record. This converts the programme from A/B into an interrupted time series: a before/after at every guidance change, automatically and permanently, with no arm construction, no blinding, and no fixture draw, always against the immediately prior version. Confounded by drift in the author's standard, in the model, and in the mix of work; volume is what defeats that, and the PR rate supplies volume.

Withdrawn: the factual-correction carve-out

Earlier framing assumed revisions would need triage to separate factual corrections from stylistic ones. There is no such population. Across hundreds of PRs the author recalls zero factual inaccuracies in a lede, which is what the architecture predicts: changelog-writer re-voices an outcome that summarize-change has already extracted, with the diff and the ticket in context, so a factual error would require inventing something it was never asked to supply. Every emitted-against-shipped diff is clean doctrine evidence and needs no triage layer, no classifier pass, and no extra input at revision time.

The residual worth watching is not factual: revisions that change which outcome is foregrounded, or how a change's reach is characterized. Those look like corrections but are salience judgments, which is doctrine evidence anyway, so they need no separation either.

Generalization

The record shape (emitted artifact, shipped artifact, optional preference, doctrine version, artifact type) applies beyond ledes: commit bodies, ticket bodies, review comments, and plans all have agent-emitted first drafts that a human revises.

[williamthorsen]

One must-have criterion is carried by a ticket in another repository rather than met here.

• The store states that its contents are outcome-selected, that absence of a record is not an acceptance, and that neither may seed a comparison's fixture draw.

The capture-lede-decision skill states all three, so an implementer reading it is covered. The store-side statement -- an assertion note in the knowledge base, plus the new fields declared in its schema -- lands in the vault repository, tracked as williamthorsen/codeassembly-kb#1.