Unattended dispatch: the board sweep tick retires manual worker dispatch - #23
Conversation
…ual worker dispatch The unattended phase the Symphony comparison reserved (§9): import the orchestrator's functions (liveness), refuse its form (residency). One mechanical five-minute tick — recover / cancel / dispatch / review / land / relay / report — plus an implement-dispatch.sh that mechanizes the dispatch ritual. Grill decisions recorded: sweep-first transport, board-is-truth dispatch policy, cap 5, both merge tiers armed, PR-open-no-merge exit.
…al, mechanized Triggered (<issue>) + --sweep modes. Registry-first dedupe and slot counting (a pre-gate worker's meta exists before its ticket moves — board state alone would double-dispatch), idle owners never block (fresh re-dispatch is doctrine; board-bind strips), strict render aborts on any surviving placeholder, spike lane routed, engine label > env > codex, gateway env identical to the ritual's. Hermetic suite reuses the shared mock-gh so the REAL _board.py eligibility and board-bind run under test.
… orchestrator functions, no residency) Seven idempotent passes over durable state: bounded recovery (resume with a nudge, 3 lifetime attempts, then park needs-human — resume is the recovery verb, not re-dispatch), board-driven cancel on terminal states only (parks are pauses; review/land species exempt), implement + review lane sweeps, one-shot land dispatch on the human Approve signal, the L2 answer relay (comment newer than last session activity, machine prefixes excluded, relayed id recorded before firing), and the read-only reconcile report. mkdir lock with stale-steal; pass isolation — one failing lane never stops the rest.
…oard state sweep-setup.md (launchd user agent first — cron-context TCC hazard named), issue-dispatch.yml + land-on-approve.yml runner-day templates (same security posture as pr-review-dispatch.yml), dispatch ritual names its mechanical executable, TECH-DEBT 1/2/10-L2 closed with evidence, protocol pins. Fix surfaced by pre-arm registry inspection: a stale working meta on an in-review ticket (obra#489's real meta) ate an implement slot forever — _slots_used now counts only workers whose ticket is still in an active lane (ready-for-agent pre-gate window / in-progress build).
…crete steps First tick is run by hand per sweep-setup.md before the timer is trusted.
…outed-gate verdicts, idempotent tick 2, launchd armed
launchd-context shells get EPERM under ~/Documents (folder protection), verified by probe; the sweep itself is innocent. sweep-setup.md now leads with the three remedies (one-time bash grant / terminal-session timer / relocation); ExecPlan records the finding with evidence.
…d relay now work in the production flow Fresh-context review (native reviewer; codex launch declined) confirmed: (P1) RECOVER had no branch for finalize's noop — an already-finalized error/idle meta (failed resume fork, fast-failed spawn) was abandoned forever; the meta's own status is now the fallback verdict. (P1) RELAY gated on status=idle which nothing sets on the --no-wait park path, and the naive normalize-first fix would bump meta.updated past the human's comment — rebuilt on finalize-normalize + the current turn's TRANSCRIPT MTIME as the turn-end ordering signal. (P2) DEFAULT_BRANCH fallback was unreachable under errexit+pipefail. (P3) missing DAEMON_HOME read as a held lock on fresh machines. Root cause of both P1s pinned durably: the finalize stub now refuses verdicts the real daemon-finalize cannot produce, so fixtures can no longer seed impossible states.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a75e7630a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| live) | ||
| tx="$(_transcript "$current")" | ||
| if [ -n "$tx" ]; then | ||
| age="$(( ( $(date +%s) - $(stat -f %m "$tx" 2>/dev/null || stat -c %Y "$tx") ) / 60 ))" |
There was a problem hiding this comment.
On GNU/Linux, stat -f %m "$tx" writes filesystem information before failing, so the || stat -c %Y fallback appends the epoch after that text; the arithmetic expansion then sees tokens like File: and, with set -u, exits the entire sweep. I reproduced this via tests/issue-tracker/test-board-sweep.sh: any live in-progress worker with a transcript hits this path, so Linux cron/runner sweeps abort before dispatch/review/land instead of recovering stalls.
Useful? React with 👍 / 👎.
| rows="$(gh pr list -R "$BOARD_REPO" --state open --label confident-ready \ | ||
| --json number,reviewDecision,labels 2>/dev/null)" || rows="[]" |
There was a problem hiding this comment.
When a repo has more than 30 open confident-ready PRs, this LAND pass only considers the first default page; I checked the gh pr list manual and --limit defaults to 30. Because older approved PRs are never passed to land-dispatch.sh while the first page remains open (for example dry-run landers or long-running/conflicted landers), the unattended sweep can leave approved PRs unlanded indefinitely unless a human notices and dispatches them manually.
Useful? React with 👍 / 👎.
| <key>DOPERPOWERS_HOME</key><string>/Users/new/Documents/GitHub/doperpowers/.claude/worktrees/unattended-sweep</string> | ||
| <key>LOCAL_REPO</key><string>/Users/new/Documents/GitHub/ida-solution</string> | ||
| <key>AUTO_MERGE_ENABLED</key><string>true</string> | ||
| <key>LAND_ENABLED</key><string>true</string> | ||
| <key>IMPLEMENT_MAX_CONCURRENT</key><string>5</string> |
There was a problem hiding this comment.
This committed launchd agent is armed for /Users/new/.../ida-solution with live merge flags, and the new event templates carry the same personal repo/actor values instead of placeholders. In any checkout except that one operator's machine, copying the documented infra either fails outright or silently dispatches against the wrong local clone, which conflicts with this repo's core-plugin rule against project-specific configuration.
Useful? React with 👍 / 👎.
| spawn_out="$(DAEMON_CLAUDE_SETTINGS="${CLODEX_SETTINGS:-$HOME/.claude/clodex-settings.json}" \ | ||
| DAEMON_CLAUDE_EFFORT="${CLODEX_EFFORT:-xhigh}" \ | ||
| "$DAEMON_SCRIPTS/daemon-spawn.sh" --no-wait "$name" "$prompt" "$LOCAL_REPO" "$name" \ | ||
| "${IMPLEMENT_MODEL:-fable}")" \ | ||
| || { echo "#$n: worker spawn failed" >&2; return 1; } |
There was a problem hiding this comment.
When the issue-event trigger and the sweep (or a manual dispatch) race on the same eligible ticket, daemon-spawn.sh --no-wait starts an implement worker before board-bind.sh can establish exclusive ownership. Unlike the review/land paths, this bootstrap has no bind-ready barrier, and board transitions do not verify registry ownership, so the losing worker can still post a gate verdict, change the ticket, or commit in its worktree before the later bind failure retires it, violating the one-worker-per-ticket dedupe this automation relies on.
Useful? React with 👍 / 👎.
| tk = str(m.get("ticket") or "").lstrip("#") | ||
| if not tk or tk not in tickets: | ||
| continue |
There was a problem hiding this comment.
The sweep joins daemon registry entries to the current board by numeric ticket id only. With the documented one-timer-per-consumer-repo setup and the default shared DAEMON_HOME, a worker bound to repo A's #42 will be treated as the owner of repo B's #42 if that issue exists, so RECOVER/CANCEL/RELAY can resume, retire, or answer the wrong daemon and implement dispatch can incorrectly dedupe against it. Persist and compare the target repo (or another repo-unique key) before accepting a meta as bound to this board.
Useful? React with 👍 / 👎.
… stalled-pipeline finding All three live implement workers opened DRAFT closing PRs and the review sweep deliberately skips drafts — tickets stranded in in-review with no reviewer ever attaching. Draft was only ever the spike lane's marker; the implement protocol now says so explicitly (clause + pin). ExecPlan reconciles the parallel-session coordination record: the other session's d4cca59 diagnosed the launchd TCC wall and its loop ran the interim ticks; this session's loop (pid 25917) carries them now, launchd stays loaded to self-arm on the grant, and dual runners proved the N-concurrent-sweeps safety claim live.
…fix SIGPIPE false-FAILs in four suites Live shakedown finding: claude-species implement workers have no self-finalizer, so their metas linger status=working and board-bind protects them as stable owners — three reviewers spawned and were retired on bind-failed. review-dispatch now runs the same normalize-owners-preflight land-dispatch already had; validated live (next tick bound all three reviewers). Separately, the 'flaky' suites were a harness bug: printf big-log | grep -Fq under pipefail SIGPIPEs when grep matches early, reporting FAIL on a PRESENT match — every captured failure carried its expected string plus 'Broken pipe'. All four suites now use herestrings.
… rule
The shakedown fix hard-banned draft closing PRs ("never as a draft").
The validated failure state is narrower: a draft *waiting for the review
loop*, which skips drafts. Per the constraint-minimization golden rule,
the clause now states the outcome (done → open ready for review) and the
consequence (drafts get no reviewer until marked ready), leaving the
means to the worker's judgment. Test pin follows the consequence
sentence instead of the ban phrase.
…epointed The GitHub tree left the TCC-protected ~/Documents (structural fix for the launchd/daemon TCC wall recorded in sweep-setup.md). Runner templates and the sweep plist now point at /Users/new/Developer/GitHub; historical docs keep the paths that were true when written.
…bodies Per the golden rule, the dispatch prompts shed everything the worker doesn't need handed to it: - Skill invocation is two sentences (use the skill; protocol = the dispatcher-pinned copy at the bound path, authoritative for the turn). The redefinition dance, the .agents/skills spoof guard (codex-era), the placeholder lecture, and 'never proceed from this bootstrap alone' are gone. The pinned path stays: it is the version-pinning device, and it's derived from the dispatch script's own location at render time, so it survives relocation and VM moves. - PR_BODY / ISSUE_BODY are no longer inlined (up to 20k chars each): workers read the PR and ticket live via gh — fresher, no truncation, smaller spawn args. RISK_MANIFEST + REPO_FACTS stay inlined on the review side only (BASE-ref snapshots a PR must not be able to edit); the implement worker reads .doperpowers/repo-facts.md from its worktree. - Static engine text folded into the protocols (one route, one engine post-clodex-migration): execution.md → implementing-tickets SKILL.md Execution; engine-codex-review.md + fallback-engine.md → reviewing-prs SKILL.md START ENGINE. references/engine-blocks/ retired on both sides; the review bootstrap binds REVIEW_ENGINE / CODEX_REVIEW_MODEL / CODEX_REVIEW_EFFORT as plain values. - implement-dispatch drops DEFAULT_BRANCH resolution and all body/facts temp-file plumbing; review-dispatch drops the issue-body fetch and the engine-block rendering. Suites: implementing-tickets (protocol-content, implement-dispatch), reviewing-prs (skill-entrypoint 165 asserts, review-dispatch, land-dispatch), issue-tracker (board-sweep, board-scripts), skill-links cross-doc linter (168 refs), shellcheck — all green.
…re, instance-value markers, snapshot-cost note - issue-dispatch.yml: the dedupe comment no longer overclaims — the registry-check-to-meta window allows a rare double-spawn (board-bind strips one owner; both processes run out the turn) - sweep-setup.md: states the private/trusted-board assumption RELAY rests on, plus its newest-comment-only semantics (remedy: comment again); plist intro names the ONE placeholder it actually has - issue-dispatch.yml + land-on-approve.yml: actor allowlist and LOCAL_REPO marked as EDIT-ME instance values for future adopters - implement-dispatch.sh: known-cost note — ~3 board snapshots per dispatched ticket; cache before the board grows ~10x Hermetic-test gap for the normalize-owners preflight filed as #24 instead of blocking. Lock-steal race left as designed (idempotent).
…ce checkout — worktree pin retired with PR #23's merge
1 participant
Problem
Every worker in the board pipeline was dispatched by a human running a script: the dispatch ritual by hand for implement workers,
review-dispatch.shby hand while the Actions runner stays blocked (ida-solution#302),land-dispatch.shby hand after approving,board-answer.shby hand to relay answers, and dead workers resurrected manually. The Symphony comparison (docs/doperpowers/2026-07-11-symphony-comparison.md §9) reserved this as the unattended phase: import the orchestrator's functions (liveness), refuse its form (residency).What this branch does
One mechanical five-minute tick —
skills/issue-tracker/scripts/board-sweep.sh, cron/launchd, no model calls, all state in GitHub + the daemon registry — runs seven idempotent passes: RECOVER (bounded resume of dead/stalled workers, 3 attempts then parkneeds-human; resume is the recovery verb, not re-dispatch), CANCEL (board-driven kill on terminal tickets only; parks are pauses), DISPATCH (implement-dispatch.sh --sweep— the dispatch ritual mechanized, registry-first dedupe, cap 5), REVIEW (review-dispatch.sh --sweep), LAND (one-shot land dispatch on the human Approve signal), RELAY (a fresh human comment on a parked ticket resumes the bound worker — TECH-DEBT #10 L2), REPORT (reconcile into the sweep log). Operator surface:references/sweep-setup.md(launchd user agent first — the cron-context TCC hazard is named) + runner-day event templates for all three lanes. TECH-DEBT items 1, 2 (verified NA on the one-harness path), and 10-L2 closed.Evidence
tests/implementing-tickets/test-implement-dispatch.sh(36 asserts),tests/issue-tracker/test-board-sweep.sh(31 asserts); full battery + shellcheck + cross-doc lint green.[gate] pass — codex/DIRECTwith substantive verdicts — the v7.21.x schema's first live exercise); tick 2 dispatched nothing (idempotence); launchd timer ticks run clean.daemon-finalizecannot produce; the stub now refuses impossible verdicts.Full narrative:
docs/doperpowers/execplans/2026-07-18-unattended-sweep-dispatch.md.Post-merge
scripts/bump-version.sh+ release, then repoint the armed plist'sDOPERPOWERS_HOMEfrom the worktree to the marketplace path.claude agentsview) — hermeticity ticket, untouched here.This PR is deliberately left unmerged — the human merges (it rewires their attention surface).
Authoring disclosure: Claude Fable 5 on Claude Code, doperpowers worktree pipeline; fresh-context review by a native reviewer subagent.