The orchestration harness: a reviewer that mutates, and a gate command that can fail - #76
Merged
Conversation
…, the loop Implementation plan for .agents/specs/orchestration-harness.md. Shaped by one measurement: of 97 gated rows, only 30 name a command that can FAIL. 46 have a Gates section with no command, 20 have no Gates section, 1 has no spec. A gate demanding one would be red on 67 rows on day one and would have to be relaxed to pass. So the checker ships as a classifier (step 2), the debt is recorded (step 3), and only then does it become a SHRINK-ONLY RATCHET (step 4) — the count of rows with a runnable gate command may never fall. Green today, stricter every time someone fixes a row, and never relaxed to pass. Same ordering the live-state audit used, for the same reason. The reviewer prompt is step 1 because it is the highest-value piece and depends on nothing: across two branches every Important finding came from an independent reviewer and none from an implementer's self-review. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…B step 1)
Across two branches every Important finding came from an independent reviewer
and none from an implementer's self-review, and the reviewers found them by
MUTATING code rather than reading diffs. Eleven tests passed with the thing they
named deleted; none was visible by reading. A prompt that lives only in an
operator's head is not a protocol, so the instruction is tracked and gated.
check-protocol-consistency.py now asserts .agents/prompts/{reviewer,implementer}.md
exist and carry their binding phrases, pinned one phrase at a time so a prompt
that quietly loses "delete or invert" or "escalate rather than guess" is a red
build rather than a shorter file.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…AIL (B step 2)
Report only. 72 of 97 gated rows cannot state a runnable command today, so a
gate demanding one would be red on arrival and would have to be relaxed to
pass -- and a relaxed gate is worse than none. The ratchet is step 4, after
step 3 records the debt.
Three rules did not survive their own tests. All three are the recorded "a
failure and an absence look the same" class, and the third is that class
landed on the one output that matters:
- The cannot-fail test passed with the cannot-fail rule DELETED. The no-op
shells were never recognised as commands at all, so `_CANNOT_FAIL` rejected
nothing it was not already rejecting. `is_command` now recognises them
deliberately, so the rejection is the load-bearing branch.
- "A command is not a backticked filename" was pinned only by `docs/`, which
no rule ever matched. On the record `sha256_cbor` was credited via `sh`,
`python@3.14` via `python`, and `tests/foo.cpp` and a bare `tests/` via the
`tests/` prefix. Tool names now need a whole-word boundary and a path must
actually be INVOKED. Two rows lose a gate they never had.
- The 97-row DENOMINATOR was pinned by nothing. Asserting GATED_STATES'
literal value says nothing about audit() using it: deleting the filter left
the suite fully green while the report moved to 726 rows. The test now
asserts audit() yields only gated states, and that the filter excludes
something -- and that the audited-matrix LIST is 7, so a widened
MATRIX_PATHS cannot double-count a file into the denominator.
`flock <lock> -c '<gate>'` (the mandated GPU-gate shape, quoted out of reach of
every other rule) and a bare `./built-binary` are now recognised. Both fire on
real spec content and neither moves a verdict today.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…es it (B step 3)
72 of 97 gated rows cannot state a command that can fail. That is a finding
about MECHANICAL checkability, not about whether the work was verified -- many
of those rows carry more evidence than the rows that score runnable, and the
artifact says so plainly and first. SPEC-DFLASH-GGUF scores gates-no-command
while carrying a mutation proof; BACKEND-VULKAN scores runnable on a pip
install.
Recorded before the ratchet so the baseline is a decision, not a pasted number.
Hand-verified 6 rows (3 runnable, 3 gates-no-command) before trusting the
classifier: 6/6 verdicts hold as the stated rule defines them.
The debt is NOT homogeneous and the split is counted: 18 DONE / 37 ACTIVE /
11 READY / 5 BLOCKED / 1 GATING. The 16 READY+BLOCKED rows carry PROSPECTIVE
gates and no evidence to transcribe -- they become runnable when the work is
done. Only the DONE rows support the "better evidenced than runnable" reading.
Three imperfections recorded rather than fixed, because the ratchet pins this
number:
- the vocabulary misses 7 gate shapes, but the MEASURED exposure inverts the
expectation: 0 rows would flip today, while naively adding bare tool names
would falsely credit 21 disjoint rows -- 15 from a bare binary name, 6 from
bare compute-sanitizer -- taking the baseline 25 -> 46 on nothing. The
flock bug again;
- classify_row reads only specs[0], changing 12 verdicts (6 -> runnable, so
the all-specs count is 31; 6 no-gates-section -> gates-no-command, so that
total is 14 not 20);
- 4 of the 25 runnable credits are not gates (two MLX pip installs, git
diff --stat which exits 0 in every state a gate could meet, git diff
--check).
Also recorded, because step 4 pins this count and cannot write the rule without
it: the baseline's principal FALSE-RED mode is the population moving. A runnable
row that is deleted, merged, or transitioned out of GATED_STATES drops the count
on a legitimate record edit. Rule specified: a drop is a regression only if the
row still exists and is still gated; if the population shrank, re-pin in the
same change naming the row and reason. This needs the SET of runnable row IDs,
not an integer -- pinning a bare count makes a regression and a record edit
indistinguishable.
Also found: sglang-matrix.md is listed as audited and contributes ZERO rows --
not because its rows are below READY, but because it carries a classification
in place of a lifecycle state, so the row parser returns nothing AND no error.
An absence that looks like a pass.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Task 3's artifact specified this and it is right: a bare integer cannot tell "this row lost its gate command" from "this row legitimately left the gated population". The population already moved 3 rows mid-branch, which is why 97 was never pinned — so a count would go red on a legitimate record edit, and the natural fix is to lower the number, which is the gate erasing its own finding. ratchet_errors now reports the two cases separately, and a test asserts they are distinguishable. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Shrink-only ratchet, the same shape as STATUS_RATCHET -- but over the SET of row
IDs carrying a command that can FAIL, never a count. A count cannot tell "this
row lost its gate command" from "this row was deleted, merged, or transitioned
out of GATED_STATES"; the population already moved 3 rows mid-branch, which is
why the 97 total was deliberately never pinned. A count would go red on a
legitimate record edit, and the natural "fix" is to lower the number, which is
the gate erasing its own finding.
ratchet_errors reports the two cases as separate, differently worded errors, and
a test constructs BOTH IN ONE RUN and asserts the lost row is named only in the
"Repair the row" message and the departed row only in the "left the gated
population" one. Deleting the `& present` split -- the whole distinction -- turns
that test red; each single-case test alone survives it.
It ships GREEN (25/25 baseline rows runnable, exit 0) because it was wired AFTER
the debt was recorded in .agents/specs/gate-command-audit-2026-08-06.md, so it
never had to be relaxed to pass, and it gets stricter every time someone
transcribes a row's evidence into an invocation.
Wired into preflight and CI, which also stops step 2's suite rotting in no gate
at all. Dispatched WITH --check in both: report mode exits 0 whatever the record
says, so a CHECKERS entry without the flag would install a gate that cannot fail
-- the exact defect this file classifies, wearing this file's own face. The same
reason --check is handled BEFORE --json, which also returns 0 unconditionally.
Three deviations from the brief, all disclosed in the task report:
- the wiring test asserts membership in the parsed CHECKERS/SUITES bash arrays,
not a substring of the file. `check-gate-commands` appears twice in preflight
(the array entry and the --check case branch), and the briefed substring
assertion was MEASURED green under all three preflight mutations;
- two tests added: the --json/--check ordering above, and the both-cases-in-one-
run distinguishability proof;
- the interface is RUNNABLE_BASELINE, as the brief's Interfaces section and the
artifact specify; its step-2 text naming RUNNABLE_RATCHET is stale.
The 25 pinned credits are a FLOOR, not a certificate: four are weak (two MLX pip
installs, git diff --check, and TOOLS-STREAMING-PARSER on git diff --stat, which
exits 0 unconditionally in a repo). Pinned anyway, per risk 3 -- a ratchet that
waits for a clean baseline never starts. The command vocabulary is UNCHANGED:
widening it naively would falsely credit 21 rows, taking the baseline 25 -> 46.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…it (B step 5)
check-protocol-consistency.py exists because an obligation was once migrated in
AGENTS.md and the checker but not in the manual, which went on instructing
agents to do the thing the migration removed. Prose is what agents read, so the
loop lands in workflow.md and the checker asserts it is there -- the same way
the role interview landed.
Subsystem B had shipped its machinery (tracked prompts, the gate-command
classifier, the recorded debt, the ratchet) and nothing an agent reads said how
to RUN a row: that a reviewer must mutate rather than read, that the controller
runs the row's gate itself instead of believing the author's report, and that
findings are never repaired in the coordinating session.
loop_errors() scopes its needles to the block between the markers rather than
searching the whole manual. workflow.md already discusses gates, prompts and
"closing the loop", so a whole-file search stays green on a loop gutted down to
its two markers -- the same incidental-match failure the reviewer prompt is
written to catch. test_the_needles_must_be_INSIDE_the_block is the executable
justification, and it is the only test that goes red on the wider variant.
Whole-branch review fixes, folded in here:
THE RATCHET IS AN EXACT PIN, NOT SHRINK-ONLY. Six documents said shrink-only,
and every one was false. `--check` is shrink-only, but the suite asserts
RUNNABLE_BASELINE EQUALS the audited runnable set, and that equality is what
makes "just lower the number" go red. It also makes GROWTH go red: adding a
legitimate gate command to a row's spec leaves --check at 0 while the suite,
preflight and CI turn red until the set is re-pinned. That is the intended cost
and it is stronger than shrink-only, so the true contract -- any movement, up or
down, re-pins RUNNABLE_BASELINE in the same change, naming the rows and the
reason -- now appears everywhere the old wording did: the classifier's docstring
and baseline comment, workflow.md, ci.yml, the audit artifact (with a
superseding note over the old "25 / a floor on a count" passage, which also
contradicted its own next subsection), and the plan.
audit() SWALLOWED PARSE ERRORS. It built an errors list, passed it to
parse_claim_rows and never read it. Corrupting a matrix therefore surfaced as
"these baseline rows left the gated population ... re-pin RUNNABLE_BASELINE" --
a parse FAILURE wearing the face of a legitimate record edit, recommending the
one action the audit says must never be taken blindly. That is this branch's own
named defect class inside the file that names it, and it was only partly masked:
check-agent-record covers 5 matrices, feature-matrix.md's 4 gated rows were in
no parse gate at all. audit() now raises RecordParseError and every mode fails,
--json included; the message says the matrix did not PARSE and says not to
re-pin off it.
sglang-matrix.md IS DROPPED FROM THE AUDITED SET, resolving an obligation the
artifact left as a directive ("it must not leave it listed and empty") that step
4 did not discharge and its test then cemented. It carries a classification
column, not a lifecycle state, so it parsed 0 rows of 87 with 0 errors -- listed
and empty, an absence reading as a pass. Dropping is the cheaper of the two
options the artifact allows and is defensible on the merits. The test now pins
the JUSTIFICATION (zero rows AND zero errors there) rather than the membership,
so the matrix comes back the moment it gains lifecycle rows. The runnable set is
unchanged at 25, because zero rows left.
The loop block moves BELOW the manual's numbered protocol. It had been inserted
between the role interview and the unheaded list, re-parenting "0. Declare your
role" under an operator-only heading -- so a helper skipping that section skipped
declaring its role -- and putting two ordered lists adjacent with colliding
numbering.
Also: KERNEL-GEMM-CPU-ELEM is marked as the FIFTH weak credit (a bare `ctest -j2`
lifted from prose describing a flake, not a gate); the audit artifact is linked
from orchestration-harness.md, the only document an agent reads that reaches it;
and the two undated counts in workflow.md and implementer.md are marked as dated
floors the way reviewer.md already marks its own.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
No row ID — protocol work. Subsystem B of two, per
.agents/specs/orchestration-harness.md(merged in #73). Subsystem A shipped the session interview; this is the operator's loop.What changed
The loop an operator follows to run a row through sub-agents existed only as folklore. This lands it as a gated protocol, in four pieces:
.agents/prompts/reviewer.mdandimplementer.mdare tracked artifacts, andcheck-protocol-consistency.pypins their binding instructions, not merely that the files exist. The reviewer's instruction is mutate, don't read.scripts/check-gate-commands.pyclassifies whether each gated row's spec names a command that can actually fail.true,echo ok,:and anything piped are not gates —cmd | tailreportstail's status..agents/specs/gate-command-audit-2026-08-06.mdrecords what that found..agents/workflow.md.Evidence
All 7 commits pass
check-doc-checkpoint.py --commit <sha>individually. The final review ran 34 mutations across both suites and every one went red, including every "wire it in report mode" and "just lower the baseline" attack.scripts/agent-preflight.shpasses except the pre-existing failure belowtest_check_gate_commands.py(22),test_check_protocol_consistency.py(38)docs/STATUS.md,docs/BENCHMARKS.mdSpeed claims
Honest gaps
CI will be red, and not because of this branch.
check-fusion-consistencyfails onmainitself:minimax_h3_video_vae_device.cppfrom #26 hand-rolls three gated-MLP epilogue call sites without routing gate-up through the merged-GEMM seam, and is not on the allowlist. That is one of the three MUST-route seams inAGENTS.mdT0. User-ruled to skip for now — nothing was allowlisted, no model was touched, no checker was weakened, and every task verified its failing-gate set was exactly that pair.72 of 97 gated rows cannot state a command that can fail — and that is not a claim that the work is unverified. Many of those rows carry real evidence in prose, in
.agents/parity-ledger.md, and in test anchors; what they lack is a command a machine can re-run. The hand-verified sample made the point sharply:SPEC-DFLASH-GGUF, classifiedgates-no-command, carries a mutation proof (302/302 exit 0;Q4_K_M→ 21/302 red), whileBACKEND-VULKAN, classifiedrunnable, is credited an MLXpip install. The artifact says this above the fold, not in a footnote.The ratchet is an exact pin, not a shrink-only floor. Any movement — up or down — re-pins
RUNNABLE_BASELINEin the same change, naming the rows and the reason. Growth is welcome but never silent. An earlier draft of six documents called it shrink-only and promised growth was free; the final review proved that adding a legitimate gate command reddens the suite, so the prose was corrected rather than the assertion relaxed. The equality check is what makes "just lower the number" go red.Known limits of the classifier, measured and recorded rather than fixed: the command vocabulary misses real shapes, and naive widening would falsely credit 21 distinct rows (baseline 25 → 46) because a bare tool name matches prose that merely mentions it — adding
flocknaively produced a count matching an earlier prediction, which felt like corroboration and was a bug.classify_rowreads only the first spec link, so the honest all-specs count is 31, not 25. Four of the 25 credits are weak (twopip installlines,git diff --check, and one resting ongit diff --stat, which exits 0 unconditionally).sglang-matrix.mdwas dropped from the audited set with the reason recorded — it uses a classification column, not a lifecycle state.One residual, parked not fixed.
audit()now fails loudly on a real parse error, verified. But a row whose ID cell is malformed is dropped byparse_claim_rowswith no error appended, so the gate still prints "left the gated population… re-pin" — a failure wearing absence's face, which is this branch's own named defect class. Not load-bearing; the fix is a per-matrix parsed-row floor.Not run: the C++
ctestsuite. This branch changes zero C++/CMake files.🤖 Generated with Claude Code