factory: reviewer verdict — structured output + review against the issue, not just the diff - #157
Merged
Merged
Conversation
…sue (toon-meta#275) Port of toon-protocol/toon-meta#299 to buzz: - New .sandcastle/review-verdict.ts: <review> verdict schema + extraction (sandbox.run() ignores an output option in sandcastle 0.12.0, so the module extracts/validates itself, with one resume() retry then a hard fail), plus host-side postBlockingVerdict (PR review + needs:human via REST) and resolveIssueFromPrBody. - agent-implement-issue.ts: reviewer phase runs runReviewerWithVerdict; a blocking verdict is never auto-merged (falls back to PR mode) and lands findings + needs:human on the opened PR. - agent-review-pr.ts: materialise the PR head as a local branch, resolve the Spec-axis issue from the PR body's Closes #n, require the verdict, post blocking findings after sandbox close. - review-prompt.md: two-axis (Standards + Spec) review against the target issue's acceptance criteria, WHAT YOU FIX vs WHAT IS BLOCKING, and the machine-parsed REQUIRED VERDICT block. - agent-review.yml: checkout pins ref: main (sandcastle checks the PR head out in its own worktree; one branch cannot be in two worktrees) and the header documents the verdict contract. Forensics steps untouched (block#278). - main.ts: reviewer promptArgs gain ISSUE_NUMBER/ISSUE_TITLE so the shared prompt's placeholders always resolve. zod was already a root devDependency (^3.25.0, v3) — no lockfile change. Part of toon-protocol/toon-meta#270 Part of toon-protocol/toon-meta#275 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gives the factory reviewer a verdict channel and the missing Spec axis. Port of toon-protocol/toon-meta#299 to buzz (toon-meta#275, epic toon-meta#270).
What changes
<review>{"verdict":"clean"|"blocking","blockingFindings":[{file,line,summary,why}]}</review>. The schema is declared assandcastle.Output.object({ tag: "review", schema }); extraction/validation is performed by the new.sandcastle/review-verdict.tsbecause — verified against the@ai-hero/sandcastle@0.12.0dist — the engine's structured-output surface exists only on top-levelrun(), and thesandbox.run()the runners use silently ignores anoutputoption. Extraction mirrors the engine's semantics exactly (last tag wins, fence-aware unwrap, JSON parse, zod schema validation).SandboxRunResult.resume()(the engine's own structured-output retry mechanism, exactly one iteration, with engine-style error feedback); if the verdict is still malformed the runner throws and the Actions job goes red — a missing verdict is never mistaken for a clean one.ISSUE_NUMBER/ISSUE_TITLEviapromptArgs; the standaloneagent:reviewrunner resolves the issue from the PR body'sCloses #n(no reference → Standards-only review). The prompt instructs the reviewer togh issue viewthe target FIRST and review the diff against its acceptance criteria. buzz's repo-specific review steps and gate commands (just fmt-check/clippy/test-unit,just desktop-*/web-*) are preserved.COMMENT) andneeds:humanis applied — pure REST viagh api(porcelaingh pr editis broken org-wide by the projectCards GraphQL deprecation). In auto-merge mode a blocking verdict refuses to merge and falls back to PR mode.agent-review.ymlnow checks outmain(git refuses one branch in two worktrees — sandcastle checks the PR head out in its own worktree under.sandcastle/worktrees/), and the runner materialises the PR head as a local branch so the engine'sworktree add -b … HEADfallback cannot silently review an empty diff.main.ts(the reserved autonomous loop) passesISSUE_NUMBER/ISSUE_TITLEto its reviewer so the shared prompt's placeholders always resolve; it does not yet consume the verdict (auto-merge wiring is toon-meta#270 work).Forensics steps (toon-meta#278) and the ~200k context-budget prompt wording (toon-meta#273) are preserved byte-for-byte.
zodwas already a root devDependency (^3.25.0, v3) — no dependency or lockfile change.Verification
SANDCASTLE_ISSUE_NUMBER/SANDCASTLE_PR_NUMBER, nothing else).changesjob's release/mobile/desktop contract scripts + file-size ratchet tests) run green locally;actionlintgreen on the workflow. Biome findings in.sandcastle/are pre-existing onmain(the directory is not format-gated); no new format errors introduced.agent:reviewrun after merge, since the label→runner executes main's code.Part of toon-protocol/toon-meta#270
Part of toon-protocol/toon-meta#275
🤖 Generated with Claude Code