Skip to content

#1122 Capture lede decisions as an accumulating corpus - #1132

Merged
williamthorsen merged 11 commits into
mainfrom
1122
Jul 31, 2026
Merged

#1122 Capture lede decisions as an accumulating corpus#1132
williamthorsen merged 11 commits into
mainfrom
1122

Conversation

@williamthorsen

Copy link
Copy Markdown
Owner

What

After a pull request merges, agents are now instructed to ask the author whether the agent's lede (the paragraph summarizing the change) shipped as written or was rewritten before merge. Each decision adds to a body of evidence for refining the lede guidance: the agent's draft, the text that shipped, and the guidance in force at the time. A pull request that merged elsewhere can be recorded on request.

Why

Every lede rewrite discarded the knowledge that produced it. A ## What section was replaced with a better one, and the pair — what was wrong, what fixed it — existed only for the moment of the edit. Nothing accumulated, so refining the lede guidance had no evidence base and proceeded from recall.

The texts were already on disk, in the pull-request and merge artifacts, but the author's judgment about them was not, and it cannot be recovered from the texts. A third of the differing pairs in history are heavy rewrites that partly reflect commits landed during review rather than lede refinements, and a lede that shipped unchanged is ambiguous between "this was good" and "there was no time to fix it."

Details

🎉 Features

  • A decision is recorded only when the author affirms one. Declining writes nothing, and the absence of a record explicitly does not mean the lede was accepted — a lede that shipped unchanged under time pressure is not an accepted lede. The corpus therefore yields less data on a busy week rather than misleading data.
  • Records are recallable as a group and by work type, and carry a content fingerprint of the guidance in force, so they group by guidance generation with nothing bookkept at install time.
  • The author's explanation is free text. Naming which rule a fix invoked belongs to the refinement pass reading the corpus, not to the moment of capture — a fixed rule list offered at capture time would presuppose which rules matter, which is the question the corpus exists to answer.
  • Decisions are stored as knowledge-base events alongside the existing knowledge base, so recall, validation, and store resolution are the ones already in use.

♻️ Refactoring

  • The refusal message for a capture with no named store is shared across the helpers that emit it, rather than copied per helper.

🧪 Tests

  • Episode resolution, record composition, and the command surface are covered, including each distinct failure code, the artifact shapes taken from real pull-request and merge artifacts, and the case where two ledes differ only by whitespace.

Closes #1122

…ifacts

Adds the resolution step behind lede-decision capture: given a ticket's artifact directory, it produces the lede an agent published, the lede that reached the merge commit, whether the two differ once whitespace is normalized, the change's work type, tier, and scope, and a content fingerprint of the doctrine that governed the agent's text.

Fingerprinting the doctrine by content rather than recording a version is what lets records group by doctrine generation with nothing written at install time; the mapping back to a commit stays recoverable by re-hashing the file's history.

Each lede accepts an override file, so a pull request merged outside the merge flow can still be resolved from text the caller supplies. Every missing input reports a distinct reason rather than throwing, so a caller can report and continue.
…e a store

The message naming the registered knowledge stores, emitted when a capture is refused for want of a named destination, now has a single definition beside the equivalent already shared by the discovery-based helpers. Two verbatim copies stood behind it, and a third was about to join them.
Adds the command behind lede-decision capture. Inspecting reports the resolved episode and writes nothing, so a caller can show both ledes before asking the author to decide; recording writes one event carrying the verdict, both ledes, an optional comment, and the change's identity and doctrine fingerprint.

A decision joins the existing event substrate rather than introducing a record type of its own, so it recalls beside every other captured event. Its tags carry the group, the verdict, and the work type under a namespace, which keeps a work type from colliding with the topical tags an event already uses.

The body records the merged lede whenever the two texts differ, independently of the verdict: the verdict holds what the author says they did, and the body holds what happened.

Every failure the command can foresee comes back as a structured result rather than a throw, so a caller running this after an irreversible merge can report one line and carry on.
Episode resolution now recovers the ticket id from a change summary whose frontmatter carries it unquoted, which is how a wholly numeric id is written. A string-only read dropped it, so a decision resolved without explicit flags reached the record with no ticket. A prefixed key such as `MAC-42` arrives as a string and is unaffected.
…a lede

Adds `capture-lede-decision`, which records whether the lede an agent wrote for a pull request shipped as written or was rewritten before merge. It reads both ledes from the ticket's own artifacts, so neither has to be supplied, and it writes one knowledge-base event carrying the verdict, both texts, an optional free-text comment, and a fingerprint of the lede doctrine in force at the time.

The corpus it builds holds positive signals only: a record exists because someone looked at a lede and decided, and declining to decide writes nothing. The absence of a record therefore means nothing was evaluated, never that a lede was accepted, which keeps a week spent shipping under time pressure from manufacturing approvals nobody gave.

Recall the corpus as a group by its `lede-decision` tag, or one work type at a time. Because it holds only changes someone chose to evaluate, it reads what good looks like and what typically fails, and must not seed a comparison's fixture draw.
Merging a pull request now ends by asking whether the lede that shipped was the one the agent wrote or a rewrite, and records the answer. The question comes after the merge, so declining costs a data point and nothing else, and a failure to record can never read as a failure to merge.
…parser

Deciding which mode an invocation selects, and validating that it names exactly one, now sits in its own function rather than inline among the field reads. Resolving the directory the doctrine ships in likewise reads as one named step.
…here

The capture skill now spells out how to supply the merged lede for a pull request that never went through the merge flow, and so wrote no merge artifact for the command to read.
A pull request merged outside the merge flow records its actual lede. The documented route supplies a command that extracts the `## What` section into the file `--merged-lede-file` reads, so the whole pull-request body no longer lands in the corpus as the lede.

`merge-pr` asks about a lede decision only after a merge that produced a commit. A Bitbucket merge, which the delegate leaves to the user, is skipped rather than reported as a capture failure.
A caller that supplies a home directory has it honored for every lookup the lede-decision capture makes, including the install manifest that names the recorded agents version. An unset `HOME` resolves to the real home rather than to the filesystem root.
The lede fixture accepts only the overrides it applies. A test that asks it for a work type fails to compile rather than silently receiving the fixture's own.
@williamthorsen williamthorsen added feature Added or improved external functionality scope:agents labels Jul 31, 2026
@github-actions

Copy link
Copy Markdown

Dependency audit

Production dependency audit passed.

@williamthorsen williamthorsen self-assigned this Jul 31, 2026
@williamthorsen
williamthorsen marked this pull request as ready for review July 31, 2026 03:00
@williamthorsen
williamthorsen merged commit 32e9044 into main Jul 31, 2026
3 checks passed
@williamthorsen
williamthorsen deleted the 1122 branch July 31, 2026 03:04
williamthorsen added a commit that referenced this pull request Aug 4, 2026
…e-v0.2.2 codeassembly-v0.4.0 factory-v0.2.2 codeassembly-mcp-v0.2.2

codeassembly-v0.4.0
- #1153 feat: Make codeassembly and kb CLI tools publishable (#1164)
- #1091 fix: Anchor a project-deployed link where its target deploys (#1159)
- #1156 refactor: Rename packages to publishable names (#1157)
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- refactor: Refine typescript-preferences
- #1150 fix: State doc-description form and make comment mood opt-in (#1151)
- deps: Upgrade first-party linting deps to latest version
- #1137 feat: Establish personal rulebooks for code layout and TypeScript preferences (#1144)
- #1126 feat: Check a package's own guidance content before it ships (#1138)
- refactor: Fix lint
- tooling: Remove redundant lint rules
- #1133 fix: Reject an anchor link that names no heading (#1135)
- #1122 feat: Capture lede decisions as an accumulating corpus (#1132)
- #1125 feat: Honor invocation tokens in rulebook bodies (#1129)
- #1107 feat: Render rulebook links and path tokens per harness (#1124)
- #1115 feat: Sync guidance at build and install so an upgrade cannot leave it stale (#1123)
- #1114 feat: Adopt a dependency's guidance by naming the package (#1121)
- #1088 feat: Deliver project ambient rulebooks per harness (#1113)
- #1087 refactor: Rename the authoring rulebook and mark its enforced rules (#1092)
- #1095 tooling: Move compilation out of the install lifecycle into a bootstrap step (#1102)
- #1094 refactor: Remove the ambient ripgrep dependency from the test suite (#1097)
- deps: Upgrade all deps to latest version
- tooling: Upgrade all deps to latest version & modernize configs
- refactor: Fix lint
- #1077 feat: Show the proposed edit above post-review menu options (#1078)
- #1068 feat: Deliver ambient rulebooks mechanically, retiring GLOBAL.md (#1075)
- #1069 feat: Add a no-second-person rule to the lede-voice doctrine (#1070)
- #1050 feat: Add a redundancy rule to the lede-voice doctrine (#1058)
- #1035 internal: Add lifecycle workspace with the canonical envelope, vocabulary & lane fold (#1049)
- #1006 internal: Extend lifecycle-event instrumentation to five high-traffic skills (#1034)
- #1028 feat: Rule out absence-of-removed-code tests (#1032)
- #114 feat: Let reviewers emit gated insights into review artifacts (#1031)
- #1027 internal: Retire input.received and redundant skill.progress emits (#1030)
- #1026 fix: Name the side effects an approval ask authorizes (#1029)
- #1022 fix: Normalize action and question label identifiers across asks blocks (#1025)

factory-v0.2.2
- #1156 refactor: Rename packages to publishable names (#1157)
- deps: Upgrade all deps to latest version
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- tooling: Remove redundant lint rules
- deps: Upgrade deps to latest version
- deps: Upgrade all deps to latest version
- deps: Upgrade all deps to latest version
- tooling: Upgrade all deps to latest version & modernize configs
- refactor: Fix lint
- deps: Upgrade all deps to latest version

kb-v0.3.0
- #1153 feat: Make codeassembly and kb CLI tools publishable (#1164)
- #1156 refactor: Rename packages to publishable names (#1157)
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- tooling: Remove redundant lint rules
- #1095 tooling: Move compilation out of the install lifecycle into a bootstrap step (#1102)
- tooling: Upgrade all deps to latest version & modernize configs
- refactor: Fix lint

codeassembly-lifecycle-v0.2.0
- #1153 feat: Make codeassembly and kb CLI tools publishable (#1164)
- #1156 refactor: Rename packages to publishable names (#1157)
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- tooling: Remove redundant lint rules
- #1095 tooling: Move compilation out of the install lifecycle into a bootstrap step (#1102)
- tooling: Upgrade all deps to latest version & modernize configs
- #1038 feat: Add the read-only git adapter for worktree and base-branch ground truth (#1059)
- #1051 feat: Bound fold memory and rescan cost with a retention window (#1057)
- #1035 internal: Add lifecycle workspace with the canonical envelope, vocabulary & lane fold (#1049)

codeassembly-mcp-v0.2.2
- deps: Upgrade all deps to latest version
- #1156 refactor: Rename packages to publishable names (#1157)
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- #1095 tooling: Move compilation out of the install lifecycle into a bootstrap step (#1102)
- tooling: Upgrade all deps to latest version & modernize configs
- refactor: Fix lint

codeassembly-run-core-v0.2.2
- #1156 refactor: Rename packages to publishable names (#1157)
- #1152 tooling: Run every test in the default gate, classified by what it reaches (#1155)
- #1110 tooling: Migrate Vitest to nmr's centralized model (#1154)
- #1095 tooling: Move compilation out of the install lifecycle into a bootstrap step (#1102)
- tooling: Upgrade all deps to latest version & modernize configs
- refactor: Fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Added or improved external functionality scope:agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Capture lede decisions as an accumulating corpus

1 participant