From 415ff740ab17c44795e6646e26e0993130f83dd4 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 06:59:58 +0900 Subject: [PATCH 01/12] =?UTF-8?q?feat(orchestrating-daemons):=20host-stamp?= =?UTF-8?q?ed=20registry=20+=20host-aware=20pid=20liveness=20=E2=80=94=20m?= =?UTF-8?q?anaged-agents=20steals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopted from Anthropic's 'Scaling Managed Agents: Decoupling the brain from the hands' (transfer analysis in docs/doperpowers/2026-07-12-managed-agents-steals.md): - Every registration/resume stamps `host` (DAEMON_HOST, default hostname) into daemon metas; _pid_alive() in _lib.sh treats a foreign-host pid as dead regardless of kill -0 (a registry migrated on a state volume carries pid numbers, not processes). Empty host = legacy meta = local. - All four pid consumers routed through host-aware checks: codex-resume one-turn guard, daemon-retire kill, review/land-dispatch _is_live dedupe, review-dispatch _wt_occupied worktree-removal guard. - Steals doc also records the state-volume convention (steal 2) and the token-wired-remote scope-split recipe (steal 3, narrows TECH-DEBT accepted note on GH_TOKEN in worker env — pointer added). - Tests: foreign-host pid → resume proceeds + re-stamps, retire leaves the unrelated process alone, dispatchers retire+respawn; same-host controls unchanged. All 5 suites + shell lint green. --- .../2026-07-12-managed-agents-steals.md | 153 ++++++++++++++++++ docs/doperpowers/TECH-DEBT.md | 6 +- .../scripts/_codex_lib.sh | 4 +- skills/orchestrating-daemons/scripts/_lib.sh | 18 +++ .../scripts/codex-resume.sh | 7 +- .../scripts/codex-spawn.sh | 2 +- .../scripts/daemon-resume.sh | 4 +- .../scripts/daemon-retire.sh | 2 +- .../scripts/daemon-spawn.sh | 4 +- skills/reviewing-prs/scripts/land-dispatch.sh | 23 ++- .../reviewing-prs/scripts/review-dispatch.sh | 31 ++-- .../test-codex-scripts.sh | 32 ++++ tests/reviewing-prs/test-land-dispatch.sh | 19 +++ tests/reviewing-prs/test-review-dispatch.sh | 53 ++++++ 14 files changed, 330 insertions(+), 28 deletions(-) create mode 100644 docs/doperpowers/2026-07-12-managed-agents-steals.md diff --git a/docs/doperpowers/2026-07-12-managed-agents-steals.md b/docs/doperpowers/2026-07-12-managed-agents-steals.md new file mode 100644 index 0000000000..9ac14b3d3e --- /dev/null +++ b/docs/doperpowers/2026-07-12-managed-agents-steals.md @@ -0,0 +1,153 @@ +# Managed-Agents steals — what we take, what we skip, and why + +**Source.** Anthropic engineering blog, *"Scaling Managed Agents: Decoupling +the brain from the hands"* (https://www.anthropic.com/engineering/managed-agents), +read 2026-07-12 during the production-infrastructure discussion (cloud host +for the board pipeline: implement / review / land workers, feedback intake). +This doc records the transfer analysis and what was actually adopted, the +same way `2026-07-11-symphony-comparison.md` did for OpenAI Symphony. + +**The article in one line.** Virtualize the agent into three interfaces — +**session** (append-only durable log), **harness** (the loop; stateless, +rebootable via `wake(sessionId)`), **sandbox** (execution environment; +provisioned on demand, disposable) — so each can fail or be replaced without +the others. Pets-vs-cattle applied to agents: the pet is not the machine, +it is *state fused to compute*. + +## 0. Mapping onto our architecture + +| Managed Agents | doperpowers equivalent | already decoupled? | +|---|---|---| +| session (durable event log) | claude JSONL transcripts (`~/.claude/projects`), codex rollouts (`~/.codex/sessions`) — plus, for the human-relevant slice, GitHub issue comments (FD-9 answer relay) | **yes** — resume = re-materialize the brain from the log | +| harness (stateless loop, `wake(sessionId)`) | `claude --bg --resume` / `codex exec resume` under the daemon registry | **yes** — every resume is already a fresh process over the log | +| sandbox (disposable hands) | git worktrees under `LOCAL_REPO/.claude/worktrees/` | **yes** — reconstructible from git at any time | +| the coupling they broke | our registry binds ticket → **(pid on one host, session file on one host)** | **no — this was our pet**, fixed by steal 1 | + +The conclusion that shaped the infra discussion: what must be persistent is +the **state volume** (registry + session stores + clones), never the compute. +A worker host — VM, Fly machine, container — is a disposable body around +that volume. Recovery semantics were already right before this doc: the +board is the source of truth, sessions are a resume *optimization*, and the +worst case after total state loss is a fresh re-dispatch with the durable +issue-comment record. The steals below close the gaps that remained. + +## 1. Steal 1 — host-stamped registry, host-aware liveness (IMPLEMENTED) + +**Their form.** The container became cattle the moment the harness stopped +assuming it shared a box with the sandbox; failure detection became "the +tool call errored," not "nurse the box." + +**Our defect.** Registry metas recorded a bare `pid`, and four consumers +tested liveness with `kill -0`: + +- `codex-resume.sh` — one-turn-per-daemon guard +- `daemon-retire.sh` — kills the live turn before retiring +- `review-dispatch.sh` `_is_live` + `_wt_occupied` — dedupe and + worktree-removal guards +- `land-dispatch.sh` `_is_live` — same + +A pid is only meaningful on the machine that recorded it. Move the registry +to a new host on a state volume (host died, host rebuilt, migrated to +cloud) and a **reused pid number** reads as a live worker: resume refuses +("a turn is still running"), retire signals an unrelated process, dispatch +skips a dead reviewer as active, worktree removal is blocked by a ghost. +Every failure is silent and directional — the pipeline stalls open. + +**The fix (shipped with this doc).** Every registration and resume stamps +`host` (`DAEMON_HOST`, default `hostname`) into the meta; `_pid_alive()` +in `_lib.sh` treats a host mismatch as dead regardless of `kill -0` +(empty `host` = legacy meta = local, preserving old behavior). All four +consumers now route through host-aware checks. Claude-engine metas get the +stamp too: their `short`/`current` ids are just as host-local (`claude +agents` is a per-host view). Pinned by tests in +`tests/orchestrating-daemons/test-codex-scripts.sh` (foreign-host pid: +resume proceeds + re-stamps, retire leaves the process alone) and +`tests/reviewing-prs/test-{review,land}-dispatch.sh` (foreign-host pid: +treated dead → retire+respawn; same-host control still skips as active). + +This is the registry's cattle key without the migration project: metas are +now effectively keyed (host, pid, session), so a future multi-host or +container fleet only adds a dispatcher — no semantic change. + +## 2. Steal 2 — the state-volume convention (CONVENTION, this doc is the spec) + +**Their form.** Session outlives harness; harness outlives container; +nothing in the compute needs to survive. + +**Ours.** When provisioning a dedicated worker host, ALL mutable pipeline +state lives under paths that sit on one detachable volume, and the compute +is fully reproducible from a setup script (cloud-init or equivalent): + +| state | path | owner | +|---|---|---| +| daemon registry + codex run scratch | `DAEMON_HOME` (default `~/.claude/orchestrating-daemons`) | ours, env-overridable everywhere (verified: all scripts honor it) | +| claude sessions + jobs | `~/.claude/projects`, `~/.claude/jobs` | harness-owned — placed on the volume by making `$HOME` (or the whole user dir) volume-backed | +| codex sessions + auth | `~/.codex` | harness-owned, same treatment | +| canonical clones + worktrees | `LOCAL_REPO` per consumer repo | ours | + +Host dies → new host + attach volume → every parked session resumes; the +host-aware liveness from steal 1 is exactly what makes the stale pids in +the migrated registry harmless. Nothing else in the pipeline needs backup +discipline: the board (GitHub) already carries ticket state and the +human-answer record. + +Non-goal: we did NOT adopt per-worker ephemeral compute (their full cattle +shape). At single-tenant scale with a few concurrent workers, dispatch +latency is noise against multi-hour turns, and the refactor would reopen +freshly shaken-down rituals for a TTFT win we cannot feel. + +## 3. Steal 3 — token-wired remote (RECIPE, apply at host provisioning) + +**Their form.** Credentials are never reachable from where generated code +runs: the repo token is wired into the git remote during sandbox init +(push/pull work; the agent never handles the token), everything else sits +behind a vault + proxy. + +**Ours, honestly scoped.** Full unreachability is impossible in this +design — a worker must hold a board-write token (`gh` issue transitions, +PR creation) *by contract*. The stealable part is **scope separation**: + +- `LOCAL_REPO` on the worker host is cloned with a fine-grained token + (contents: read/write on that repo ONLY) embedded in the remote URL. + Worktrees share the parent repo's remote config, so worker `git push` + just works — the push credential never enters worker env. +- Worker env `GH_TOKEN` gets a second fine-grained token scoped to + issues + pull-requests (board writes, PR open/comment) with NO contents + write. `_codex_launch` already prefers a pre-set `GH_TOKEN` over the + keychain capture, so injection needs zero code. +- Net effect: a prompt-injected worker can vandalize the board (visible, + reversible, audited) but cannot push code with authority the reviewer + didn't grant, and neither token can spawn broader sessions. + +Not applied on the current Mac host — the keychain token is the user's +full-power credential either way, so the split is theater there. The +TECH-DEBT accepted note ("GH_TOKEN in worker env") now points here as its +narrowing path. + +## 4. Non-steals + +- **Harness out of the container / on-demand sandbox provisioning.** Their + TTFT win (p50 −60%) comes from thousands of multi-tenant sessions paying + container boot before first token. Our workers run minutes-to-hours; + boot cost is invisible. Skip until worker concurrency outgrows one host. +- **MCP vault + proxy for tool credentials.** Right shape at multi-tenant + scale; for us it adds a resident service (the thing our no-orchestrator + architecture deliberately avoids) to protect one token we can scope + instead. +- **`getEvents()` context-interrogation layer.** Our equivalent pressure + is already answered compositionally: FD-7 closing artifacts (PR body as + the durable orientation record) + transcript-on-disk + fork-carries-context + resume. A positional event-slicing API over transcripts is capability we + have no consumer for. +- **Session-as-REPL-object context management.** Same reason — no consumer; + the harnesses own their context engineering. + +## 5. Relationship to the Symphony comparison + +Symphony gave us the *work semantics* debates (FD-1..FD-9); Managed Agents +gives the *substrate* doctrine. They agree on the load-bearing point: the +durable log is the identity of the work, the loop is disposable. Symphony +puts that log in an orchestrator's memory + Linear; Managed Agents puts it +in a session store; we put it in GitHub (board) + session files (resume +cache). The steals above keep that stack true when the pipeline leaves the +Mac it was born on. diff --git a/docs/doperpowers/TECH-DEBT.md b/docs/doperpowers/TECH-DEBT.md index 7fe0d97a64..3f7cf9b553 100644 --- a/docs/doperpowers/TECH-DEBT.md +++ b/docs/doperpowers/TECH-DEBT.md @@ -118,7 +118,11 @@ design (FU-7) removed the main consumer, so exposure is small. - **GH_TOKEN in worker env** — visible to the worker's subprocesses; parity with what claude workers reach via the keychain, recorded in FU-3's - security note. + security note. A narrowing recipe now exists (token-wired remote: push + scope in the clone's remote URL, board scope in env — see + `2026-07-12-managed-agents-steals.md` §steal-3); apply it when + provisioning a dedicated worker host, where scoped tokens are real. On + this Mac the keychain token is full-power anyway, so the note stands. - **Mini ssh probe noise** — codex's `keepRemoteControlAwakeWhilePluggedIn` probes the unreachable `mini` host at spawn; user config, harmless. - **FU-2 known limitation** — a daemon resumed once and never diff --git a/skills/orchestrating-daemons/scripts/_codex_lib.sh b/skills/orchestrating-daemons/scripts/_codex_lib.sh index 92fc0aa37f..e7ccd1d342 100755 --- a/skills/orchestrating-daemons/scripts/_codex_lib.sh +++ b/skills/orchestrating-daemons/scripts/_codex_lib.sh @@ -7,7 +7,9 @@ # shell around the process finalizes the registry when the turn ends. The meta # is the same JSON contract as claude daemons plus: # engine "codex" -# pid codex process pid of the CURRENT turn (liveness: kill -0) +# pid codex process pid of the CURRENT turn (liveness: _pid_alive — +# kill -0 gated on `host` matching this machine; a registry that +# migrated on a state volume carries pids that are dead here) # effort model_reasoning_effort # event_log JSONL event stream of the current turn (--json stdout) # The codex session id (thread.started) keys the registry, so board-bind.sh / diff --git a/skills/orchestrating-daemons/scripts/_lib.sh b/skills/orchestrating-daemons/scripts/_lib.sh index 7653cf9bf1..cf8cd3f599 100755 --- a/skills/orchestrating-daemons/scripts/_lib.sh +++ b/skills/orchestrating-daemons/scripts/_lib.sh @@ -22,6 +22,13 @@ set -euo pipefail DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" mkdir -p "$DAEMON_HOME" +# Host identity — stamped into metas as `host` at every registration. A pid +# (and a `claude agents` short) is only meaningful on the machine that recorded +# it: when the registry lives on a state volume that migrates to a new host, +# a reused pid number would otherwise read as a live worker and wrongly block +# resume / dedupe / retire. Override with DAEMON_HOST for tests. +DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" + # How long the spawn/resume WATCHER polls `claude agents` for a turn to finish — # a wait bound only, NOT a turn budget. The toolkit never kills a turn: every turn # is an independent `--bg` process that keeps running regardless. Default 5 hours; @@ -37,6 +44,17 @@ _err_path() { printf '%s/%s.err' "$DAEMON_HOME" "$1"; } # Strip ANSI color codes (the `claude --bg` banner is colored even when piped). _strip_ansi() { sed -E 's/\x1b\[[0-9;]*m//g'; } +# rc 0 iff pid <1> is a live process OF THIS HOST. <2> is the meta's recorded +# `host` (empty = legacy meta with no host field → assume local, preserving +# pre-host behavior). A host mismatch is DEAD regardless of kill -0: the +# process did not migrate with the state volume, only its number did. +_pid_alive() { + local pid="$1" host="${2:-}" + [ -n "$pid" ] || return 1 + [ -z "$host" ] || [ "$host" = "$DAEMON_HOST" ] || return 1 + kill -0 "$pid" 2>/dev/null +} + # Merge key=value pairs into a daemon's metadata JSON (creates it if absent). # Usage: _meta_set field1 value1 [field2 value2 ...] # diff --git a/skills/orchestrating-daemons/scripts/codex-resume.sh b/skills/orchestrating-daemons/scripts/codex-resume.sh index a7c76c5257..3e10f32b59 100755 --- a/skills/orchestrating-daemons/scripts/codex-resume.sh +++ b/skills/orchestrating-daemons/scripts/codex-resume.sh @@ -19,10 +19,12 @@ msg="${2:?missing message}" [ "$(_meta_get "$uuid" engine)" = "codex" ] \ || { echo "codex-resume: $uuid is not a codex daemon — use daemon-resume.sh" >&2; exit 1; } -# One turn per daemon at a time (same invariant as claude daemons). +# One turn per daemon at a time (same invariant as claude daemons). Host-aware: +# a pid recorded on another machine (registry migrated on a state volume) is +# dead by definition — resuming HERE is exactly the recovery act. if [ "$(_meta_get "$uuid" status)" = "working" ]; then pid="$(_meta_get "$uuid" pid)" - if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then + if _pid_alive "$pid" "$(_meta_get "$uuid" host)"; then echo "codex-resume: a turn is still running (pid $pid) — wait for it or retire" >&2 exit 1 fi @@ -116,6 +118,7 @@ fi turns="$(_meta_get "$uuid" turns)"; turns=$((${turns:-1} + 1)) _meta_set "$uuid" pid "$(cat "$run.pid" 2>/dev/null || printf '')" \ + host "$DAEMON_HOST" \ event_log "$run.events.jsonl" status "working" turns "$turns" updated "$(_now)" [ -f "$run.rc" ] && _meta_set "$uuid" \ status "$(_codex_final_status "$(cat "$run.rc")" "$run.events.jsonl")" updated "$(_now)" diff --git a/skills/orchestrating-daemons/scripts/codex-spawn.sh b/skills/orchestrating-daemons/scripts/codex-spawn.sh index d4daf8bda6..ba4fa16140 100755 --- a/skills/orchestrating-daemons/scripts/codex-spawn.sh +++ b/skills/orchestrating-daemons/scripts/codex-spawn.sh @@ -89,7 +89,7 @@ status="working" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$(printf '%.8s' "$uuid")" name "$name" \ task "$task" cwd "$runcwd" worktree "$worktree" model "$model" effort "$effort" \ - engine "codex" pid "$pid" event_log "$run.events.jsonl" \ + engine "codex" pid "$pid" host "$DAEMON_HOST" event_log "$run.events.jsonl" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" # The wrapper may have finalized between our check and the write — re-apply. [ -f "$run.rc" ] && _meta_set "$uuid" \ diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index dc8efe0e0f..b03853bbab 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -96,7 +96,7 @@ if [ "$poll_rc" -ne 0 ] || [ -z "$newuuid" ]; then # chain to the new turn and mark status=working. Do NOT write the reply file # or bump turns; no final reply has landed. daemon-reply.sh reads the CURRENT # session's transcript, so it will surface the reply once the turn finishes. - _meta_set "$uuid" current "$newuuid" short "$newshort" status "working" updated "$(_now)" + _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" status "working" updated "$(_now)" # The fork is confirmed live — the superseded turn's session can go. if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 @@ -117,7 +117,7 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error # Reply file stays keyed by the ORIGINAL uuid; read the reply from the new turn. _record_reply "$newuuid" "$uuid" "$state" -_meta_set "$uuid" current "$newuuid" short "$newshort" \ +_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" \ status "$status" updated "$(_now)" turns "$((turns + 1))" # Purge the superseded turn: deregister it (jobs entry + supervisor record) and diff --git a/skills/orchestrating-daemons/scripts/daemon-retire.sh b/skills/orchestrating-daemons/scripts/daemon-retire.sh index fd5a7bf81c..fd2637b804 100755 --- a/skills/orchestrating-daemons/scripts/daemon-retire.sh +++ b/skills/orchestrating-daemons/scripts/daemon-retire.sh @@ -24,7 +24,7 @@ if [ "$engine" = "codex" ]; then status="$(_meta_get "$uuid" status)" pid="$(_meta_get "$uuid" pid)" if { [ "$status" = "working" ] || [ "$status" = "blocked" ]; } && \ - [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then + _pid_alive "$pid" "$(_meta_get "$uuid" host)"; then kill "$pid" 2>/dev/null || true # Killing the pid makes _codex_launch's own detached wrapper (the one # that finalizes status+reply once `wait` on the codex pid returns) wake diff --git a/skills/orchestrating-daemons/scripts/daemon-spawn.sh b/skills/orchestrating-daemons/scripts/daemon-spawn.sh index 8c4f149dd0..aa75c1b527 100755 --- a/skills/orchestrating-daemons/scripts/daemon-spawn.sh +++ b/skills/orchestrating-daemons/scripts/daemon-spawn.sh @@ -75,7 +75,7 @@ if [ "$nowait" -eq 1 ]; then esac _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" echo "daemon spawned (no-wait): $name [$short / $uuid] status=$status (reply: daemon-reply.sh $short)" exit 0 @@ -104,7 +104,7 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error _record_reply "$uuid" "$uuid" "$state" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" wtnote=""; [ -n "$worktree" ] && wtnote=" worktree=$runcwd (branch worktree-$wt)" diff --git a/skills/reviewing-prs/scripts/land-dispatch.sh b/skills/reviewing-prs/scripts/land-dispatch.sh index a12af50560..cfc37443c7 100755 --- a/skills/reviewing-prs/scripts/land-dispatch.sh +++ b/skills/reviewing-prs/scripts/land-dispatch.sh @@ -35,6 +35,8 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" +# Host identity for pid checks — same semantics and default as review-dispatch. +DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="${BOARD_SCRIPTS:-$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)}" @@ -61,7 +63,8 @@ case "${LAND_ENABLED:-false}" in *) LAND_MODE="dry-run" ;; esac -# Newest land-pr- registry entry → "uuid|status|current|engine|pid" (empty if none). +# Newest land-pr- registry entry → "uuid|status|current|engine|pid|host" +# (empty if none). _land_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' import glob, json, os @@ -80,15 +83,18 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""))) + print("%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""))) PY } -# rc 0 when the worker's CURRENT turn is live (same semantics as review-dispatch). -_is_live() { # +# rc 0 when the worker's CURRENT turn is live (same semantics as review-dispatch: +# a codex pid counts only on the host that recorded it). +_is_live() { # if [ "$2" = "codex" ]; then - [ -n "$3" ] && kill -0 "$3" 2>/dev/null + [ -n "$3" ] || return 1 + [ -z "${4:-}" ] || [ "$4" = "$DAEMON_HOST" ] || return 1 + kill -0 "$3" 2>/dev/null return fi claude agents --json --all 2>/dev/null | CUR="$1" python3 -c ' @@ -159,10 +165,11 @@ fi meta="$(_land_meta "$pr")" if [ -n "$meta" ]; then uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" - current="${rest%%|*}"; rest="${rest#*|}"; w_engine="${rest%%|*}"; w_pid="${rest#*|}" + current="${rest%%|*}"; rest="${rest#*|}"; w_engine="${rest%%|*}"; rest="${rest#*|}" + w_pid="${rest%%|*}"; w_host="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$w_engine" "$w_pid"; then + if _is_live "$current" "$w_engine" "$w_pid" "$w_host"; then echo "#$pr: skip active land worker"; exit 0 fi _retire "$uuid" ;; diff --git a/skills/reviewing-prs/scripts/review-dispatch.sh b/skills/reviewing-prs/scripts/review-dispatch.sh index 8ed3c56ce4..867ab235b5 100755 --- a/skills/reviewing-prs/scripts/review-dispatch.sh +++ b/skills/reviewing-prs/scripts/review-dispatch.sh @@ -52,6 +52,9 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" +# Host identity for pid checks — a pid recorded on another machine (registry +# migrated on a state volume) is not a live worker here. Same default as _lib.sh. +DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)" @@ -84,7 +87,8 @@ ENGINE_BLOCK_FILE="$SKILL_DIR/references/engine-blocks/engine-codex-review.md" FALLBACK_FILE="$SKILL_DIR/references/engine-blocks/fallback-engine.md" REVIEW_ENGINE="$SCRIPT_DIR/review-engine.sh" -# Newest review-pr- registry entry → "uuid|status|current|engine|pid" (empty if none). +# Newest review-pr- registry entry → "uuid|status|current|engine|pid|host" +# (empty if none). _reviewer_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' import glob, json, os @@ -103,16 +107,19 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""))) + print("%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""))) PY } # rc 0 when the reviewer's CURRENT turn is live: claude → session uuid visible -# in `claude agents`; codex → recorded pid alive. -_is_live() { # +# in `claude agents`; codex → recorded pid alive ON THIS HOST (a foreign-host +# pid is dead by definition — only its number migrated with the registry). +_is_live() { # if [ "$2" = "codex" ]; then - [ -n "$3" ] && kill -0 "$3" 2>/dev/null + [ -n "$3" ] || return 1 + [ -z "${4:-}" ] || [ "$4" = "$DAEMON_HOST" ] || return 1 + kill -0 "$3" 2>/dev/null return fi claude agents --json --all 2>/dev/null | CUR="$1" python3 -c ' @@ -143,7 +150,7 @@ sys.exit(0 if any(a.get("cwd") == os.environ["WT"] for a in d) else 1)' && retur # count ONLY codex metas with a live pid; a stale claude-engine `working` # meta must NOT start blocking removal (the claude path's fail-open # behavior above is unchanged). - DAEMON_HOME="$DAEMON_HOME" WT="$1" python3 - <<'PY' + DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" WT="$1" python3 - <<'PY' import glob, json, os, sys home = os.environ["DAEMON_HOME"]; wt = os.environ["WT"] for p in glob.glob(os.path.join(home, "*.json")): @@ -157,6 +164,9 @@ for p in glob.glob(os.path.join(home, "*.json")): continue if m.get("status") not in ("working", "blocked"): continue + host = str(m.get("host") or "") + if host and host != os.environ["DAEMON_HOST"]: + continue # foreign-host pid — the process did not migrate with the registry pid = str(m.get("pid") or "") if pid.isdigit(): try: @@ -291,15 +301,16 @@ PY # Dedupe verdict for PR <1> in mode <2> (triggered|sweep), cr-label flag <3>. # Prints: "dispatch" | "respawn " | "skip ". _decide() { - local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid + local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid whost if [ "$cr" = "1" ]; then echo "skip confident-ready label (remove it to force re-review)"; return; fi meta="$(_reviewer_meta "$pr")" if [ -z "$meta" ]; then echo "dispatch"; return; fi uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" - current="${rest%%|*}"; rest="${rest#*|}"; engine="${rest%%|*}"; pid="${rest#*|}" + current="${rest%%|*}"; rest="${rest#*|}"; engine="${rest%%|*}"; rest="${rest#*|}" + pid="${rest%%|*}"; whost="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$engine" "$pid"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; + if _is_live "$current" "$engine" "$pid" "$whost"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; retired) echo "dispatch" ;; *) if [ "$mode" = "triggered" ]; then echo "respawn $uuid" diff --git a/tests/orchestrating-daemons/test-codex-scripts.sh b/tests/orchestrating-daemons/test-codex-scripts.sh index 2c5f6c3bf4..280a240e0e 100755 --- a/tests/orchestrating-daemons/test-codex-scripts.sh +++ b/tests/orchestrating-daemons/test-codex-scripts.sh @@ -308,6 +308,38 @@ kill "$stale_pid" 2>/dev/null || true wait "$stale_pid" 2>/dev/null || true assert_equals "$(meta_field "$uuid_stale" status)" "retired" "terminal codex record is still retired" +echo "== host-aware liveness: a foreign-host pid is dead by definition ==" +# A registry that migrated on a state volume carries pids recorded on the old +# machine. A reused (live-HERE) pid number must not read as a live worker: +# resume proceeds (resuming here IS the recovery act) and retire never signals +# an unrelated local process. +assert_equals "$(meta_field "$uuid_e" host)" "$(hostname)" "codex-spawn stamps host" +sleep 30 & foreign_pid=$! +uuid_mig="hostmig1-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","name":"migrated","short":"hostmig1","engine":"codex","pid":"%s","host":"old-host","status":"working","cwd":"%s"}' \ + "$uuid_mig" "$uuid_mig" "$foreign_pid" "$WORK" > "$DAEMON_HOME/$uuid_mig.json" +out="$("$SCRIPTS_DIR/codex-resume.sh" "$uuid_mig" "resume after migration" 2>&1)" +assert_not_contains "$out" "a turn is still running" "resume does not mistake a foreign-host pid for a live turn" +assert_contains "$out" "stub reply: resume after migration" "resume proceeds on the new host" +assert_equals "$(meta_field "$uuid_mig" host)" "$(hostname)" "resume re-stamps host to this machine" +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "resume never signals the unrelated local process behind the foreign pid" +else + fail "resume never signals the unrelated local process behind the foreign pid" +fi +uuid_mig2="hostmig2-0000-4000-8000-000000000000" +printf '{"uuid":"%s","name":"migrated2","short":"hostmig2","engine":"codex","pid":"%s","host":"old-host","status":"working"}' \ + "$uuid_mig2" "$foreign_pid" > "$DAEMON_HOME/$uuid_mig2.json" +"$SCRIPTS_DIR/daemon-retire.sh" "$uuid_mig2" >/dev/null +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "retire leaves a foreign-host working pid alone" +else + fail "retire leaves a foreign-host working pid alone" +fi +assert_equals "$(meta_field "$uuid_mig2" status)" "retired" "foreign-host record is still retired" +kill "$foreign_pid" 2>/dev/null || true +wait "$foreign_pid" 2>/dev/null || true + echo "== _meta_set serializes concurrent RMW — no lost fields (FU-1) ==" # The lost-update race: the detached _codex_launch wrapper and the foreground # spawn each read-modify-write the same meta; without serialization the later diff --git a/tests/reviewing-prs/test-land-dispatch.sh b/tests/reviewing-prs/test-land-dispatch.sh index 32e8c3e3aa..a7594fba2c 100755 --- a/tests/reviewing-prs/test-land-dispatch.sh +++ b/tests/reviewing-prs/test-land-dispatch.sh @@ -250,6 +250,25 @@ reset_state; seed_lander idle assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "finished land worker retired (explicit dispatch = fresh signal)" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "finished land worker re-dispatched" +# host-aware: a working codex land worker whose live-here pid was recorded on +# another host (registry migrated on a state volume) is DEAD → retire+respawn. +reset_state +sleep 300 & LANDPID=$! +PID="$LANDPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "feed0000-0000-4000-8000-000000000000", + "current": "feed0000-0000-4000-8000-000000000000", + "name": "land-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], + "feed0000-0000-4000-8000-000000000000.json"), "w")) +PY +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host live pid → land worker treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "foreign-host live pid → respawned" +kill "$LANDPID" 2>/dev/null; wait "$LANDPID" 2>/dev/null || true + # ---- ticketless PR -------------------------------------------------------------------- echo "ticketless:" reset_state diff --git a/tests/reviewing-prs/test-review-dispatch.sh b/tests/reviewing-prs/test-review-dispatch.sh index 3d6f349876..88d97b932c 100755 --- a/tests/reviewing-prs/test-review-dispatch.sh +++ b/tests/reviewing-prs/test-review-dispatch.sh @@ -610,6 +610,59 @@ out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "stale claude-engine meta in the same cwd does not block removal (fail-open preserved)" reset_state +# (d) host-aware: a codex meta whose pid is live HERE but was recorded on +# another host (registry migrated on a state volume) is NOT an occupant — +# the process did not migrate, only its number did. +sleep 300 & WTPID=$! +WT="$WT" PID="$WTPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8002-0000-4000-8000-000000000000", + "current": "cdec8002-0000-4000-8000-000000000000", + "name": "occupant", "engine": "codex", "cwd": os.environ["WT"], + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8002-0000-4000-8000-000000000000.json"), "w")) +PY +: > "$SPAWN_LOG" +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host codex pid does not block removal — dispatch proceeds" +kill "$WTPID" 2>/dev/null; wait "$WTPID" 2>/dev/null || true +reset_state + +# ---- dedupe is host-aware too --------------------------------------------------- +# A working codex reviewer meta whose live-here pid carries a foreign host must +# read as DEAD in _decide: respawn, not "skip active reviewer". +echo "dedupe (host-aware):" +sleep 300 & DEDUPID=$! +PID="$DEDUPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:cdec8003" "foreign-host live pid → reviewer treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host live pid → respawned" +reset_state +# control: same live pid, HOST MATCHING this machine → still a live occupant +PID="$DEDUPID" H="$(hostname)" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$out" "active reviewer" "same-host live pid still skips as active" +assert_equals "$(cat "$SPAWN_LOG")" "" "same-host live pid spawns nothing" +kill "$DEDUPID" 2>/dev/null; wait "$DEDUPID" 2>/dev/null || true +reset_state + echo if [[ "$FAILURES" -gt 0 ]]; then echo "$FAILURES test(s) FAILED"; exit 1 From dde32ade35b58ae03f41cb1cc4cdbe0ceb990189 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 16:44:00 +0900 Subject: [PATCH 02/12] =?UTF-8?q?feat(infra):=20worker-host=20provisioning?= =?UTF-8?q?=20=E2=80=94=20cloud-init=20body,=20state-volume=20soul,=20seed?= =?UTF-8?q?ing=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runnable form of the steals doc §2 (state/compute separation): - infra/worker-host/cloud-init.yaml — disposable body for a CX43/Ubuntu 24.04 host: blank-only volume format + /data mount before user creation, worker user (fixed uid, home ON the volume, no sudo), Node LTS, gh, codex, claude (native installer as worker), tailscale, swap. No Docker by design (host- process registry model, codex Landlock on a real kernel, cloud-init IS the reproducibility). - infra/worker-host/env.example — seeding contract: two-token scope split (board token in env, push token only in clone remotes), setup-token / auth.json paths, SSL_CERT_FILE pin. - infra/worker-host/README.md — layer split, hcloud creation, seeding checklist, Linux verification gate (dogfood cell), body-rebuild drill. - _codex_launch + review-engine.sh: SSL_CERT_FILE probe now also covers /etc/ssl/certs/ca-certificates.crt — TECH-DEBT #8 RESOLVED. Suites: codex-scripts, review-engine, shell lint — green; cloud-init YAML syntax-checked. --- .../2026-07-12-managed-agents-steals.md | 4 + docs/doperpowers/TECH-DEBT.md | 10 +- infra/worker-host/README.md | 108 ++++++++++++++++++ infra/worker-host/cloud-init.yaml | 84 ++++++++++++++ infra/worker-host/env.example | 31 +++++ .../scripts/_codex_lib.sh | 11 +- skills/reviewing-prs/scripts/review-engine.sh | 9 +- 7 files changed, 246 insertions(+), 11 deletions(-) create mode 100644 infra/worker-host/README.md create mode 100644 infra/worker-host/cloud-init.yaml create mode 100644 infra/worker-host/env.example diff --git a/docs/doperpowers/2026-07-12-managed-agents-steals.md b/docs/doperpowers/2026-07-12-managed-agents-steals.md index 9ac14b3d3e..8dd3ccb449 100644 --- a/docs/doperpowers/2026-07-12-managed-agents-steals.md +++ b/docs/doperpowers/2026-07-12-managed-agents-steals.md @@ -91,6 +91,10 @@ the migrated registry harmless. Nothing else in the pipeline needs backup discipline: the board (GitHub) already carries ticket state and the human-answer record. +Runnable form: `infra/worker-host/` (cloud-init for the body, env.example +for the seeding contract, README with the layer split, verification gate, +and rebuild drill). + Non-goal: we did NOT adopt per-worker ephemeral compute (their full cattle shape). At single-tenant scale with a few concurrent workers, dispatch latency is noise against multi-hour turns, and the refactor would reopen diff --git a/docs/doperpowers/TECH-DEBT.md b/docs/doperpowers/TECH-DEBT.md index 3f7cf9b553..ec6cf4e3b5 100644 --- a/docs/doperpowers/TECH-DEBT.md +++ b/docs/doperpowers/TECH-DEBT.md @@ -109,10 +109,12 @@ post-clause: disable collab tools via spawn-time `-c` config. ### 8. `SSL_CERT_FILE` is macOS-specific -`_codex_launch` exports `/etc/ssl/cert.pem` when present (guarded — silently -skipped elsewhere). A Linux worker host needs its distro bundle path wired -in; the review engine block documents the requirement. The in-thread review -design (FU-7) removed the main consumer, so exposure is small. +**RESOLVED 2026-07-12**: `_codex_launch` and `review-engine.sh` now probe +`/etc/ssl/cert.pem` (macOS) then `/etc/ssl/certs/ca-certificates.crt` +(Debian/Ubuntu) and export the first hit; `infra/worker-host/env.example` +also pins the Linux path explicitly as belt-and-suspenders. Remaining +exposure: other distros' bundle paths — extend the probe list if a non-Debian +host ever appears. ### 9. Accepted notes (recorded, no action intended) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md new file mode 100644 index 0000000000..4c8e96e870 --- /dev/null +++ b/infra/worker-host/README.md @@ -0,0 +1,108 @@ +# Worker host provisioning — body / soul / seeding + +Runnable counterpart of `docs/doperpowers/2026-07-12-managed-agents-steals.md` +§2 (state-volume convention). One persistent Linux VM runs the whole board +pipeline: implement / review / land workers (detached processes under the +daemon registry), the Actions runner that turns board events into dispatches, +and nothing else. No Docker: workers are host processes keyed by +(host, pid, session) in the registry, codex brings its own Landlock sandbox +on a real kernel, and single-tenant reproducibility comes from cloud-init, +not images. + +Everything splits by lifetime: + +| layer | what | lives | +|---|---|---| +| 1 — body | OS, packages, CLIs, runner binary | disposable; `cloud-init.yaml` recreates it | +| 2 — soul | registry, sessions, auth files, clones, worktrees | `/data` volume = the worker user's `$HOME` | +| 3 — seeding | tokens, codex auth.json, runner registration, clones | by hand, ONCE per soul (never per body) | + +## 1. Create the body + +Hetzner (CX43, Ubuntu 24.04, one Volume) — adjust names to taste: + +```bash +hcloud volume create --name doper-data --size 40 --location fsn1 +hcloud server create --name doper-worker-1 --type cx43 --image ubuntu-24.04 \ + --location fsn1 --volume doper-data --user-data-from-file cloud-init.yaml \ + --ssh-key +``` + +First boot formats the volume ONLY if blank, mounts it at `/data`, creates +`worker` (uid 1010, home `/data/worker`, no sudo), installs git/jq/python3/ +build-essential, Node LTS, `gh`, `codex` (global npm), `claude` (native +installer, as worker), Tailscale, and an 8G swapfile. + +After boot: + +```bash +tailscale up # then close public SSH: +ufw allow in on tailscale0; ufw allow ssh; ufw enable # drop 'allow ssh' once tailscale login works +``` + +## 2. Seed the soul (once) + +As `worker` (`sudo -iu worker`): + +1. **Env file** — copy `env.example` → `~/.env`, fill it (see the file for + the two-token scope split), `chmod 600 ~/.env`. +2. **codex auth** — from your logged-in machine: + `scp ~/.codex/auth.json worker@host:~/.codex/auth.json` (portable by + design; device-code flow is the fallback if the workspace enables it). +3. **claude auth** — nothing to do beyond `CLAUDE_CODE_OAUTH_TOKEN` in + `~/.env` (generated locally with `claude setup-token`). +4. **Canonical clones, push scope wired into the remote** (steal 3) — one + per consumer repo, using a contents-only fine-grained PAT that appears + NOWHERE else: + ```bash + git clone https://oauth2:@github.com/OWNER/REPO.git ~/repos/REPO + ``` + Worktrees share the parent's remote config, so worker `git push` works + without the token ever entering worker env. Verify the split: + `gh api repos/OWNER/REPO --jq .permissions` under `GH_TOKEN` must show + push=false. +5. **doperpowers itself** — `git clone` this repo to `~/doperpowers` + (dispatch scripts are invoked by path from here; workers get skills via + vendoring / plugin install exactly as on the Mac). +6. **Actions runner** (event trigger, PRIVATE repos only — standing + constraint): download the runner into `~/runner`, then + `./config.sh --url https://github.com/OWNER/REPO --token --labels doper-worker` + and as root `./svc.sh install worker && ./svc.sh start`. Runner jobs must + only run the dispatch ritual (render → spawn --no-wait → bind) and exit; + workers are detached processes outside the job, so job timeouts never + touch them. + +## 3. Verification gate (before trusting it) + +The rituals were shakedown-tested on macOS; this is the Linux pass. Run one +dogfood cell end-to-end on the VM and check, in order: + +1. `codex exec` smoke turn — auth.json accepted, Landlock sandbox active + (kernel ≥5.13; Ubuntu 24.04 is 6.8). +2. Nested-codex TLS — `SSL_CERT_FILE` resolves (`_codex_launch` and + `review-engine.sh` probe `/etc/ssl/certs/ca-certificates.crt` since + 2026-07-12); a review-engine call from inside a worker completes. +3. Board write under env `GH_TOKEN` — a `board-transition.sh` against a test + issue; push from a worktree uses the remote-wired credential. +4. Full cell: dispatch an implement worker on a toy ticket → PR → review + worker → verdict. The claude-engine branch counts double here + (TECH-DEBT #3: untested by the original shakedown). + +## 4. Rebuild drill (why this layout exists) + +Body loss is a non-event: create a new server with the same +`cloud-init.yaml`, attach the same volume, re-run layer-1's `tailscale up`, +restart the runner service. Every parked session resumes — the registry's +stale pids are neutralized by host-aware liveness (metas carry `host`; a +foreign-host pid reads as dead). Nothing in layer 3 repeats, because nothing +in layer 3 lived on the body. + +## 5. Not configured here, on purpose + +- **Watchdog** — T1 tech-debt (import of Symphony's liveness sweep); when it + exists it becomes a cron/timer on this host. Until then + `review-dispatch.sh --sweep` on a timer is the available partial. +- **Feedback intake** (customer feedback → issue) — serverless territory + (Cloudflare Worker → GitHub API), not this VM. +- **ufw auto-enable in cloud-init** — deliberate: enabling the firewall + before Tailscale is joined can lock you out of a fresh body. diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml new file mode 100644 index 0000000000..6a5f1bcd47 --- /dev/null +++ b/infra/worker-host/cloud-init.yaml @@ -0,0 +1,84 @@ +#cloud-config +# Doperpowers worker host — the DISPOSABLE BODY (layer 1 of 3). +# +# Design: docs/doperpowers/2026-07-12-managed-agents-steals.md (steal 2). +# Everything this file installs can be thrown away and rebuilt; everything +# that must survive lives on the attached state volume mounted at /data +# (the worker user's HOME sits on it, so ~/.claude, ~/.codex, the daemon +# registry, and all clones ride the volume automatically). Secrets are NEVER +# in this file — they are seeded once by hand (layer 3, see README.md). +# +# Target: Hetzner Cloud CX43 (8 vCPU / 16 GB, x86), Ubuntu 24.04 LTS, with +# one Hetzner Volume attached at server creation. Works on any KVM VM with +# a by-id-addressable data disk if you adjust the bootcmd glob. + +hostname: doper-worker-1 +timezone: Etc/UTC + +package_update: true +packages: + - git + - curl + - jq + - python3 + - build-essential # npm native-module fallback builds (no arm64/x64 prebuilt) + - unzip + - ca-certificates # provides /etc/ssl/certs/ca-certificates.crt (SSL_CERT_FILE) + +# ---- state volume (the soul) -------------------------------------------------- +# First boot formats the attached volume ONLY if it is blank (blkid probe), +# so re-provisioning a new body against an existing soul never wipes it. +# cloud-init runs bootcmd + mounts before users-groups, so the worker user's +# home can live on the volume. +bootcmd: + - | + dev="$(readlink -f /dev/disk/by-id/scsi-0HC_Volume_* 2>/dev/null | head -1)" + if [ -b "$dev" ] && ! blkid "$dev" >/dev/null 2>&1; then + mkfs.ext4 -L doperdata "$dev" + fi +mounts: + - [ "LABEL=doperdata", /data, ext4, "defaults,nofail,discard", "0", "2" ] + +swap: + filename: /swap.img + size: 8G + maxsize: 8G + +# ---- users --------------------------------------------------------------------- +# Fixed uid so a REBUILT body maps the volume's existing file ownership. +# No sudo: the worker runs pipeline processes only; admin happens as root. +users: + - default + - name: worker + uid: 1010 + shell: /bin/bash + homedir: /data/worker + lock_passwd: true + +runcmd: + # Node.js LTS (consumer-repo builds/tests: npm ci, playwright, etc.) + - curl -fsSL https://deb.nodesource.com/setup_22.x | bash - + - apt-get install -y nodejs + # gh CLI (board transitions, PR ops) — official apt repo + - mkdir -p -m 755 /etc/apt/keyrings + - curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg + - chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg + - echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list + - apt-get update && apt-get install -y gh + # Tailscale (SSH access path — close public 22 after `tailscale up`, see README) + - curl -fsSL https://tailscale.com/install.sh | sh + # codex CLI (global; worker-agnostic binary, auth is per-user on the volume) + - npm install -g @openai/codex + # claude CLI — native installer, installed AS the worker (lands in ~/.local/bin + # on the volume; harmless to re-run on a rebuilt body) + - sudo -u worker bash -lc 'curl -fsSL https://claude.ai/install.sh | bash' + # worker shell wiring: source the seeded env file (layer 3) on login and in + # non-interactive bash -lc invocations (systemd runner jobs use those) + - | + sudo -u worker bash -c ' + touch /data/worker/.env && chmod 600 /data/worker/.env + grep -q "source ~/.env" /data/worker/.bashrc 2>/dev/null || \ + printf "\n# doperpowers worker env (seeded by hand — see infra/worker-host)\n[ -f ~/.env ] && set -a && source ~/.env && set +a\n" >> /data/worker/.bashrc + ' + +final_message: "doper-worker body ready after $UPTIME s — proceed to layer 3 seeding (infra/worker-host/README.md)" diff --git a/infra/worker-host/env.example b/infra/worker-host/env.example new file mode 100644 index 0000000000..b0a8e9f917 --- /dev/null +++ b/infra/worker-host/env.example @@ -0,0 +1,31 @@ +# Doperpowers worker host — per-user environment (layer 3: seeded by hand). +# Copy to /data/worker/.env, chmod 600. Sourced by ~/.bashrc (wired by +# cloud-init) with `set -a`, so every value is exported. +# +# TWO TOKENS, TWO SCOPES (steal 3 — token-wired remote, see +# docs/doperpowers/2026-07-12-managed-agents-steals.md §3): +# - GH_TOKEN: fine-grained PAT, issues + pull-requests read/write on the +# consumer repos, NO contents write. This is what workers reach. +# - The push credential (contents read/write only) is NOT here — it is +# embedded in each canonical clone's remote URL at clone time. +GH_TOKEN= + +# Claude subscription auth for headless workers: run `claude setup-token` +# on a machine with a browser, paste the printed token here (valid ~1 year). +CLAUDE_CODE_OAUTH_TOKEN= + +# codex auth is FILE-based, not env-based: copy your logged-in +# ~/.codex/auth.json to /data/worker/.codex/auth.json (chmod 600). + +# TLS trust anchors as a file bundle for NESTED codex calls. The scripts +# autodetect this path since 2026-07-12; pinning it here is belt-and-suspenders. +SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt + +# Engine defaults (label > WORKER_ENGINE > codex). +WORKER_ENGINE=codex +CODEX_MODEL=gpt-5.6-sol +CODEX_EFFORT=high + +# DAEMON_HOME / DAEMON_HOST are deliberately NOT set: DAEMON_HOME defaults to +# ~/.claude/orchestrating-daemons (already on the volume via $HOME), and +# DAEMON_HOST must stay the real hostname so host-aware pid liveness works. diff --git a/skills/orchestrating-daemons/scripts/_codex_lib.sh b/skills/orchestrating-daemons/scripts/_codex_lib.sh index e7ccd1d342..aa4eea25ec 100755 --- a/skills/orchestrating-daemons/scripts/_codex_lib.sh +++ b/skills/orchestrating-daemons/scripts/_codex_lib.sh @@ -226,10 +226,13 @@ _codex_launch() { # review engine call) cannot reach the OS keychain/trustd under the # sandbox, so its rustls has no trust anchors and every connection dies # with `invalid peer certificate: UnknownIssuer`. /etc/ssl/cert.pem is the - # sandbox-readable file bundle (verified live: nested exec fails without, - # completes with). The outer codex itself is unsandboxed and unaffected. - if [ -z "${SSL_CERT_FILE:-}" ] && [ -f /etc/ssl/cert.pem ]; then - export SSL_CERT_FILE=/etc/ssl/cert.pem + # sandbox-readable file bundle on macOS (verified live: nested exec fails + # without, completes with); Debian/Ubuntu ships it as ca-certificates.crt. + # The outer codex itself is unsandboxed and unaffected. + if [ -z "${SSL_CERT_FILE:-}" ]; then + for _cert in /etc/ssl/cert.pem /etc/ssl/certs/ca-certificates.crt; do + if [ -f "$_cert" ]; then export SSL_CERT_FILE="$_cert"; break; fi + done fi # A NESTED codex (e.g. review-engine.sh run by a codex worker) resolves # its code-mode command host to /usr/local/bin (absent here) instead of diff --git a/skills/reviewing-prs/scripts/review-engine.sh b/skills/reviewing-prs/scripts/review-engine.sh index f2bf10e285..d1cadd18d5 100755 --- a/skills/reviewing-prs/scripts/review-engine.sh +++ b/skills/reviewing-prs/scripts/review-engine.sh @@ -42,9 +42,12 @@ effort="${CODEX_REVIEW_EFFORT:-xhigh}" source_codex_home="${CODEX_HOME:-$HOME/.codex}" # TLS trust anchors as a FILE bundle — a nested codex cannot reach the OS -# keychain/trustd under the outer seatbelt (shakedown FU-6). -if [ -z "${SSL_CERT_FILE:-}" ] && [ -f /etc/ssl/cert.pem ]; then - export SSL_CERT_FILE=/etc/ssl/cert.pem +# keychain/trustd under the outer seatbelt (shakedown FU-6). macOS ships the +# bundle as cert.pem, Debian/Ubuntu as ca-certificates.crt. +if [ -z "${SSL_CERT_FILE:-}" ]; then + for _cert in /etc/ssl/cert.pem /etc/ssl/certs/ca-certificates.crt; do + if [ -f "$_cert" ]; then export SSL_CERT_FILE="$_cert"; break; fi + done fi # A nested codex resolves its code-mode command host to /usr/local/bin # (absent here) instead of ~/.local/bin — point it explicitly. From 76cbd9e2a39e15910659af2c84735a67d26c54f9 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 17:02:52 +0900 Subject: [PATCH 03/12] =?UTF-8?q?feat(infra):=20triage=20tenant=20layer=20?= =?UTF-8?q?=E2=80=94=20separate=20uid,=20systemd=20timer,=20seeding=20cont?= =?UTF-8?q?ract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First production tenant of the worker host: the triaging-feedback poller moves off the Mac's launchd onto the VM as its burn-in workload. - cloud-init: triage user (uid 1011, /data/triage — separate from worker because its env holds SUPABASE_SERVICE_ROLE_KEY while worker executes PR-derived code); doper-triage systemd oneshot+timer replacing the launchd label (same single-instance serialization the reclaim design needs); ExecCondition on the seeded .env makes pre-seed ticks silent no-ops and rebuilds zero-touch. - env.triage.example: skill-dir .env contract — VM deltas vs Mac (explicit GH_TOKEN issues-scope since no keychain; contents-READ-only PAT in the ida-solution remote; triage never pushes). - README: §5 tenant seeding + one-tick verification + Mac handoff; rebuild drill and stale feedback-intake bullet updated. --- infra/worker-host/README.md | 59 +++++++++++++++++++++++----- infra/worker-host/cloud-init.yaml | 48 +++++++++++++++++++++- infra/worker-host/env.triage.example | 36 +++++++++++++++++ 3 files changed, 132 insertions(+), 11 deletions(-) create mode 100644 infra/worker-host/env.triage.example diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index 4c8e96e870..e7e147a4db 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -4,7 +4,8 @@ Runnable counterpart of `docs/doperpowers/2026-07-12-managed-agents-steals.md` §2 (state-volume convention). One persistent Linux VM runs the whole board pipeline: implement / review / land workers (detached processes under the daemon registry), the Actions runner that turns board events into dispatches, -and nothing else. No Docker: workers are host processes keyed by +and the feedback-triage poller (a separate unix user — see §5), and nothing +else. No Docker: workers are host processes keyed by (host, pid, session) in the registry, codex brings its own Landlock sandbox on a real kernel, and single-tenant reproducibility comes from cloud-init, not images. @@ -29,9 +30,11 @@ hcloud server create --name doper-worker-1 --type cx43 --image ubuntu-24.04 \ ``` First boot formats the volume ONLY if blank, mounts it at `/data`, creates -`worker` (uid 1010, home `/data/worker`, no sudo), installs git/jq/python3/ -build-essential, Node LTS, `gh`, `codex` (global npm), `claude` (native -installer, as worker), Tailscale, and an 8G swapfile. +`worker` (uid 1010, home `/data/worker`, no sudo) and `triage` (uid 1011, +home `/data/triage`, no sudo — see §5 for why they are separate), installs +git/jq/python3/build-essential, Node LTS, `gh`, `codex` (global npm), +`claude` (native installer, as worker), Tailscale, an 8G swapfile, and the +`doper-triage` systemd timer (armed but inert until §5's seeding). After boot: @@ -95,14 +98,52 @@ Body loss is a non-event: create a new server with the same restart the runner service. Every parked session resumes — the registry's stale pids are neutralized by host-aware liveness (metas carry `host`; a foreign-host pid reads as dead). Nothing in layer 3 repeats, because nothing -in layer 3 lived on the body. - -## 5. Not configured here, on purpose +in layer 3 lived on the body. The triage tenant needs no step at all: the +timer is re-armed by cloud-init and its `ExecCondition` finds the seeded +`.env` already on the volume. + +## 5. Triage tenant (`skills/triaging-feedback` poller) + +The first production tenant, and deliberately the burn-in one: real value +from day one (24/7 feedback triage instead of a sleep-prone Mac's launchd), +minimal blast radius (ticket-only — the poller diagnoses and files tickets; +it never writes code or pushes). It has **no soul on this host**: all durable +state lives in Supabase (`feedback.triage_state`/`triage_lease`) and GitHub +(tickets), so unlike the workers it doesn't even need the state volume — it +sits on it only because `$HOME` does. + +**Why a separate unix user.** The dispatcher env holds +`SUPABASE_SERVICE_ROLE_KEY` (RLS bypass — the strongest secret here), while +`worker` runs implement/review workers that execute PR-derived code. User +separation (uid 1011, mode-600 `.env`) is the cheap wall between them. + +Seed as `triage` (`sudo -iu triage`) — prerequisites first: the p86 +migration must be live in Supabase and the descriptive labels created on +ida-solution (`references/setup.md` §0/§4): + +1. **Clones** — `git clone` doperpowers to `~/doperpowers`; clone + ida-solution to `~/repos/ida-solution` with a contents-READ-only + fine-grained PAT in the remote URL (the poller fetches origin every tick, + never pushes). +2. **Deps** — `cd ~/doperpowers/skills/triaging-feedback && npm ci`. +3. **Env** — copy `env.triage.example` → that same skill dir's `.env`, + fill it, `chmod 600`. Keep `TRIAGE_K=1` until ticket quality is trusted + (`references/setup.md` §5). +4. **Verify one tick** — `sudo systemctl start doper-triage.service`, then + `journalctl -u doper-triage.service -f`. The timer (installed by + cloud-init, 10-minute cadence) takes over from there. +5. **Mac handoff** — once a VM tick has filed a correct ticket, unload the + Mac's launchd label (`references/setup.md` §6). Atomic claim + lease + makes the overlap window safe, but two pollers is a config smell, not a + feature. + +## 6. Not configured here, on purpose - **Watchdog** — T1 tech-debt (import of Symphony's liveness sweep); when it exists it becomes a cron/timer on this host. Until then `review-dispatch.sh --sweep` on a timer is the available partial. -- **Feedback intake** (customer feedback → issue) — serverless territory - (Cloudflare Worker → GitHub API), not this VM. +- **Feedback intake** (in-app 피드백 → Supabase `feedback` table) — that is + the product's own write path; this host only polls the table. No + serverless intake layer is needed for triage. - **ufw auto-enable in cloud-init** — deliberate: enabling the firewall before Tailscale is joined can lock you out of a fresh body. diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml index 6a5f1bcd47..ff72d42770 100644 --- a/infra/worker-host/cloud-init.yaml +++ b/infra/worker-host/cloud-init.yaml @@ -45,8 +45,12 @@ swap: maxsize: 8G # ---- users --------------------------------------------------------------------- -# Fixed uid so a REBUILT body maps the volume's existing file ownership. -# No sudo: the worker runs pipeline processes only; admin happens as root. +# Fixed uids so a REBUILT body maps the volume's existing file ownership. +# No sudo: pipeline users run pipeline processes only; admin happens as root. +# `triage` is a SEPARATE user from `worker` on purpose: its dispatcher env +# holds SUPABASE_SERVICE_ROLE_KEY (RLS bypass — the strongest secret on this +# host), while implement/review workers under `worker` execute PR-derived +# code. Same user would let a compromised worker read that key. users: - default - name: worker @@ -54,6 +58,43 @@ users: shell: /bin/bash homedir: /data/worker lock_passwd: true + - name: triage + uid: 1011 + shell: /bin/bash + homedir: /data/triage + lock_passwd: true + +# ---- triage tenant: systemd timer (body-side; arming is guarded) --------------- +# Replaces the Mac's launchd label (skills/triaging-feedback/references/setup.md +# §6). systemd gives the same single-instance serialization the reclaim design +# depends on: a timer never re-triggers a oneshot service that is still running. +# ExecCondition makes pre-seed ticks silent no-ops, so the timer can be enabled +# unconditionally — a rebuilt body against a seeded soul resumes polling with +# zero manual steps. +write_files: + - path: /etc/systemd/system/doper-triage.service + content: | + [Unit] + Description=Doperpowers feedback triage — one poll tick + + [Service] + Type=oneshot + User=triage + Environment=HOME=/data/triage + ExecCondition=/usr/bin/test -f /data/triage/doperpowers/skills/triaging-feedback/.env + ExecStart=/data/triage/doperpowers/skills/triaging-feedback/scripts/feedback-poll.sh + - path: /etc/systemd/system/doper-triage.timer + content: | + [Unit] + Description=Doperpowers feedback triage — 10-minute poll + + [Timer] + OnCalendar=*:00/10 + RandomizedDelaySec=30 + Persistent=false + + [Install] + WantedBy=timers.target runcmd: # Node.js LTS (consumer-repo builds/tests: npm ci, playwright, etc.) @@ -80,5 +121,8 @@ runcmd: grep -q "source ~/.env" /data/worker/.bashrc 2>/dev/null || \ printf "\n# doperpowers worker env (seeded by hand — see infra/worker-host)\n[ -f ~/.env ] && set -a && source ~/.env && set +a\n" >> /data/worker/.bashrc ' + # triage tenant timer — safe to arm before seeding (ExecCondition guards) + - systemctl daemon-reload + - systemctl enable --now doper-triage.timer final_message: "doper-worker body ready after $UPTIME s — proceed to layer 3 seeding (infra/worker-host/README.md)" diff --git a/infra/worker-host/env.triage.example b/infra/worker-host/env.triage.example new file mode 100644 index 0000000000..492cacc712 --- /dev/null +++ b/infra/worker-host/env.triage.example @@ -0,0 +1,36 @@ +# Triage tenant env (layer 3: seeded by hand) — the skill-dir .env that +# feedback-poll.sh sources. Copy to +# /data/triage/doperpowers/skills/triaging-feedback/.env (chmod 600) +# Authoritative reference for every knob (models, K, timeouts, trust roles): +# skills/triaging-feedback/references/setup.md §2. This file records only the +# required set plus the VM-specific deltas vs the Mac deployment. + +# The strongest secret on this host (RLS bypass). This is WHY triage runs as +# its own unix user: implement/review workers under `worker` execute +# PR-derived code and must never be able to read this file. +SUPABASE_URL= +SUPABASE_SERVICE_ROLE_KEY= + +# Codex SDK auth — API-billed, no auth.json needed for this tenant. +OPENAI_API_KEY= + +# VM delta: no macOS keychain here, so the board scripts' `gh` calls need an +# explicit token. Fine-grained PAT: issues + pull-requests read/write on +# ida-solution, NO contents write (triage is ticket-only — it never pushes). +GH_TOKEN= + +# Consumer repo checkout. Clone it with a SECOND fine-grained PAT — contents +# READ-only — embedded in the remote URL (the poller fetches origin every +# tick; it never pushes): +# git clone https://oauth2:@github.com/IDA-solution/ida-solution.git /data/triage/repos/ida-solution +TRIAGE_REPO_PATH=/data/triage/repos/ida-solution +TRIAGE_BASE_BRANCH=feat/m4.5-polish + +# Board scripts, invoked with cwd = TRIAGE_REPO_PATH so BOARD_REPO is +# inferred from that checkout's remote (setup.md §3). +TRIAGE_BOARD_SCRIPTS_DIR=/data/triage/doperpowers/skills/issue-tracker/scripts + +# Start conservative on a new host: observe ticket quality before raising K +# (setup.md §5), and keep the kill switch handy. +TRIAGE_K=1 +TRIAGE_ENABLED=true From 447e9d1b8cb6904b7a7745b6c27469aa4f218f57 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 21:44:58 +0900 Subject: [PATCH 04/12] =?UTF-8?q?fix(infra):=20worker-host=20review=20find?= =?UTF-8?q?ings=20=E2=80=94=20per-body=20hostname,=20runner=20env/label,?= =?UTF-8?q?=20honest=20PAT=20boundary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five findings from PR #10 review, all confirmed against the code: - Static hostname doper-worker-1 broke host-aware liveness on the rebuild path (_lib.sh stamps $(hostname); same name = old metas read local, a recycled pid could alias an unrelated process). runcmd now sets doper-worker-$(machine-id prefix) — unique per body since machine-id is minted at first boot — with preserve_hostname guarding per-boot reverts. - Runner registration label doper-worker never matched the shipped workflow's runs-on [self-hosted, claude-review]; jobs would queue forever. - Runner jobs read ~/runner/.env + .path, not .bashrc (non-login; Ubuntu .bashrc early-returns non-interactive). §2.6 now seeds both; the cloud-init comment claiming bash -lc coverage is corrected. - Rebuild drill said 'restart the runner service' but the svc.sh systemd unit lived on the discarded root fs — drill now reinstalls it. - README claimed the push PAT never enters worker reach; any worker-uid process can read it via git remote get-url. Text now states the honest boundary (workers ARE contents-write principals) and requires branch protection before first run. --- infra/worker-host/README.md | 50 ++++++++++++++++++++++++------- infra/worker-host/cloud-init.yaml | 19 ++++++++++-- infra/worker-host/env.example | 4 ++- 3 files changed, 58 insertions(+), 15 deletions(-) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index e7e147a4db..e037de6332 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -34,7 +34,11 @@ First boot formats the volume ONLY if blank, mounts it at `/data`, creates home `/data/triage`, no sudo — see §5 for why they are separate), installs git/jq/python3/build-essential, Node LTS, `gh`, `codex` (global npm), `claude` (native installer, as worker), Tailscale, an 8G swapfile, and the -`doper-triage` systemd timer (armed but inert until §5's seeding). +`doper-triage` systemd timer (armed but inert until §5's seeding). It also +sets the hostname to `doper-worker-` — unique per body, +which is what makes the registry's host-aware pid liveness hold across +rebuilds (the Hetzner server *name* can stay `doper-worker-1`; only the OS +hostname matters to the registry). After boot: @@ -61,7 +65,14 @@ As `worker` (`sudo -iu worker`): git clone https://oauth2:@github.com/OWNER/REPO.git ~/repos/REPO ``` Worktrees share the parent's remote config, so worker `git push` works - without the token ever entering worker env. Verify the split: + without the token ever entering worker env. Be honest about what that + buys: any process running as `worker` — including PR-derived code — can + still read the token (`git remote get-url origin`), so workers ARE + contents-write principals on the consumer repos. The containment is the + token's shape, not its hiding place: contents-only scope, consumer repos + only, and **branch protection on main/integration branches (require + PRs)** so the token cannot rewrite protected refs. Enable that protection + before the first worker runs. Verify the env-token split: `gh api repos/OWNER/REPO --jq .permissions` under `GH_TOKEN` must show push=false. 5. **doperpowers itself** — `git clone` this repo to `~/doperpowers` @@ -69,11 +80,23 @@ As `worker` (`sudo -iu worker`): vendoring / plugin install exactly as on the Mac). 6. **Actions runner** (event trigger, PRIVATE repos only — standing constraint): download the runner into `~/runner`, then - `./config.sh --url https://github.com/OWNER/REPO --token --labels doper-worker` - and as root `./svc.sh install worker && ./svc.sh start`. Runner jobs must - only run the dispatch ritual (render → spawn --no-wait → bind) and exit; - workers are detached processes outside the job, so job timeouts never - touch them. + `./config.sh --url https://github.com/OWNER/REPO --token --labels claude-review` + — the label must be `claude-review`: the shipped `pr-review-dispatch.yml` + selects `runs-on: [self-hosted, claude-review]`, and a runner registered + under any other label leaves those jobs queued forever. Runner jobs read + env from `~/runner/.env` and PATH from `~/runner/.path`, NOT from + `.bashrc` (non-login shells; Ubuntu's stock `.bashrc` early-returns for + non-interactive). From a worker login shell where + `command -v gh git node python3 claude` all resolve: + ```bash + grep -Ev '^\s*(#|$)' ~/.env > ~/runner/.env # runner .env is bare KEY=VALUE + echo "DOPERPOWERS_HOME=/data/worker/doperpowers" >> ~/runner/.env + echo "$PATH" > ~/runner/.path + ``` + Then as root `cd /data/worker/runner && ./svc.sh install worker && + ./svc.sh start`. Runner jobs must only run the dispatch ritual (render → + spawn --no-wait → bind) and exit; workers are detached processes outside + the job, so job timeouts never touch them. ## 3. Verification gate (before trusting it) @@ -95,10 +118,15 @@ dogfood cell end-to-end on the VM and check, in order: Body loss is a non-event: create a new server with the same `cloud-init.yaml`, attach the same volume, re-run layer-1's `tailscale up`, -restart the runner service. Every parked session resumes — the registry's -stale pids are neutralized by host-aware liveness (metas carry `host`; a -foreign-host pid reads as dead). Nothing in layer 3 repeats, because nothing -in layer 3 lived on the body. The triage tenant needs no step at all: the +and reinstall the runner service — its registration and config survive in +`/data/worker/runner`, but the systemd unit `svc.sh install` wrote lived on +the discarded root filesystem, so there is nothing to merely "restart": +`cd /data/worker/runner && sudo ./svc.sh install worker && sudo ./svc.sh start`. +Every parked session resumes — the registry's stale pids are neutralized by +host-aware liveness (metas carry `host`; the rebuilt body's machine-id +suffix gives it a NEW hostname, so every old meta reads foreign and its pid +as dead). Nothing else in layer 3 repeats, because nothing else in layer 3 +lived on the body. The triage tenant needs no step at all: the timer is re-armed by cloud-init and its `ExecCondition` finds the seeded `.env` already on the volume. diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml index ff72d42770..7270aed072 100644 --- a/infra/worker-host/cloud-init.yaml +++ b/infra/worker-host/cloud-init.yaml @@ -12,7 +12,13 @@ # one Hetzner Volume attached at server creation. Works on any KVM VM with # a by-id-addressable data disk if you adjust the bootcmd glob. -hostname: doper-worker-1 +# hostname is set in runcmd, NOT here — it must be UNIQUE PER BODY. The daemon +# registry stamps metas with $(hostname) (host-aware pid liveness in +# orchestrating-daemons/_lib.sh); a rebuilt body reusing the old name would +# make the previous body's metas read as local, so a recycled pid could alias +# an unrelated process into resume/dedupe/retire. preserve_hostname stops +# cloud-init's per-boot update module from reverting to the metadata name. +preserve_hostname: true timezone: Etc/UTC package_update: true @@ -97,6 +103,11 @@ write_files: WantedBy=timers.target runcmd: + # Per-body hostname (see header note). /etc/machine-id is minted at this + # body's first boot, so every rebuild gets a fresh suffix — old registry + # metas read as foreign-host and their pids as dead, which is what lets + # parked sessions resume safely after a rebuild. + - hostnamectl set-hostname "doper-worker-$(cut -c1-6 /etc/machine-id)" # Node.js LTS (consumer-repo builds/tests: npm ci, playwright, etc.) - curl -fsSL https://deb.nodesource.com/setup_22.x | bash - - apt-get install -y nodejs @@ -113,8 +124,10 @@ runcmd: # claude CLI — native installer, installed AS the worker (lands in ~/.local/bin # on the volume; harmless to re-run on a rebuilt body) - sudo -u worker bash -lc 'curl -fsSL https://claude.ai/install.sh | bash' - # worker shell wiring: source the seeded env file (layer 3) on login and in - # non-interactive bash -lc invocations (systemd runner jobs use those) + # worker shell wiring: source the seeded env file (layer 3) in INTERACTIVE + # shells (admin ssh, sudo -iu worker). Actions runner jobs do NOT read + # .bashrc — non-login shells, and Ubuntu's stock .bashrc early-returns for + # non-interactive ones — they get env from ~/runner/.env (README §2.6). - | sudo -u worker bash -c ' touch /data/worker/.env && chmod 600 /data/worker/.env diff --git a/infra/worker-host/env.example b/infra/worker-host/env.example index b0a8e9f917..11e850409e 100644 --- a/infra/worker-host/env.example +++ b/infra/worker-host/env.example @@ -28,4 +28,6 @@ CODEX_EFFORT=high # DAEMON_HOME / DAEMON_HOST are deliberately NOT set: DAEMON_HOME defaults to # ~/.claude/orchestrating-daemons (already on the volume via $HOME), and -# DAEMON_HOST must stay the real hostname so host-aware pid liveness works. +# DAEMON_HOST must stay the real hostname so host-aware pid liveness works +# (cloud-init makes it per-body unique via a machine-id suffix — pinning it +# here would survive rebuilds on the volume and defeat exactly that). From 16a8b2dcdb1286540ba62739efae3e4b36c9cf78 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 22:10:41 +0900 Subject: [PATCH 05/12] fix(infra): /data mount hard gate + codex-auth seeding path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review wave, both findings confirmed: - nofail (kept: a missing volume must degrade to a reachable system, not systemd emergency mode) let provisioning 'succeed' with /data on the disposable root fs — the users module creates the homedirs there before runcmd. runcmd now opens with a mountpoint assertion that aborts loudly; final_message and README §1 direct to 'cloud-init status --long' before seeding instead of claiming readiness. - README §2.2's scp to worker@host always failed: worker has a locked password, no authorized keys, and no ~/.codex. Documented the real path — through the login account, install(1) with owner/mode set from birth, no temp-file exposure. --- infra/worker-host/README.md | 15 ++++++++++++--- infra/worker-host/cloud-init.yaml | 16 +++++++++++++++- 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index e037de6332..e271ff4dc3 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -43,6 +43,8 @@ hostname matters to the registry). After boot: ```bash +cloud-init status --long # must be done/no errors — the runcmd hard + # gate aborts here if /data didn't mount tailscale up # then close public SSH: ufw allow in on tailscale0; ufw allow ssh; ufw enable # drop 'allow ssh' once tailscale login works ``` @@ -53,9 +55,16 @@ As `worker` (`sudo -iu worker`): 1. **Env file** — copy `env.example` → `~/.env`, fill it (see the file for the two-token scope split), `chmod 600 ~/.env`. -2. **codex auth** — from your logged-in machine: - `scp ~/.codex/auth.json worker@host:~/.codex/auth.json` (portable by - design; device-code flow is the fallback if the workspace enables it). +2. **codex auth** — `worker` is deliberately not SSH-reachable (locked + password, no authorized keys), so copy through your login account (root + on Hetzner) and install with ownership and mode set from birth: + ```bash + ssh root@host 'install -d -m 700 -o worker -g worker /data/worker/.codex' + ssh root@host 'install -m 600 -o worker -g worker /dev/stdin /data/worker/.codex/auth.json' \ + < ~/.codex/auth.json + ``` + (auth.json is portable by design; device-code flow is the fallback if + the workspace enables it.) 3. **claude auth** — nothing to do beyond `CLAUDE_CODE_OAUTH_TOKEN` in `~/.env` (generated locally with `claude setup-token`). 4. **Canonical clones, push scope wired into the remote** (steal 3) — one diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml index 7270aed072..6753e9d957 100644 --- a/infra/worker-host/cloud-init.yaml +++ b/infra/worker-host/cloud-init.yaml @@ -103,6 +103,19 @@ write_files: WantedBy=timers.target runcmd: + # HARD GATE: everything below assumes the state volume is mounted. The + # fstab entry keeps `nofail` on purpose (a missing volume must degrade to + # a reachable system, not systemd emergency mode), which means a missing/ + # unlabeled volume silently leaves /data on the DISPOSABLE root fs — the + # users module has already created the homedirs there by now, and seeding + # into them would vanish on rebuild. Abort loudly instead: no CLIs, no + # timer, and `cloud-init status` reports the error. Recovery is to attach + # the volume and recreate the server (runcmd does not re-run). + - | + mountpoint -q /data || { + echo "FATAL: /data is not a mountpoint — state volume missing or unlabeled; aborting provisioning" >&2 + exit 1 + } # Per-body hostname (see header note). /etc/machine-id is minted at this # body's first boot, so every rebuild gets a fresh suffix — old registry # metas read as foreign-host and their pids as dead, which is what lets @@ -138,4 +151,5 @@ runcmd: - systemctl daemon-reload - systemctl enable --now doper-triage.timer -final_message: "doper-worker body ready after $UPTIME s — proceed to layer 3 seeding (infra/worker-host/README.md)" +# final_message prints even when runcmd aborted — hence "check status", not "ready". +final_message: "doper-worker body boot finished after $UPTIME s — confirm with 'cloud-init status --long' BEFORE layer 3 seeding (infra/worker-host/README.md)" From 38716d5903241c0351dc5a87201ef0ccb495e120 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 22:35:36 +0900 Subject: [PATCH 06/12] fix(infra): tenant home hardening, git identity seeding, fail-fast provisioning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third review wave, all three confirmed: - The uid boundary relied on the image's login.defs default home mode; anything group/other-traversable lets triage read worker's clones (push PAT in each .git/config). runcmd now chmods both homes 0700 (idempotent against an existing soul), and the runner .env recipe adds chmod 600 — it carries the same tokens as ~/.env but was born under default umask. - A fresh soul has no git identity and git refuses to auto-detect email from the machine-id hostname, so a worker's first commit dies with 'Author identity unknown'. Seeding §2.4 now sets user.name/user.email once per soul (~/.gitconfig rides the volume). - runcmd merged into one sh script with no set -e meant a failed NodeSource/Tailscale/claude installer still yielded 'cloud-init status: done'; curl|bash additionally masked download failures. First item is now set -e (governs the whole merged script), curl|bash split into download-then-execute (sh is dash — no pipefail; the claude installer keeps the pipe but under an inner bash with pipefail), and provisioning ends with an explicit binary/Node-major/claude-on-worker-PATH gate. --- infra/worker-host/README.md | 10 +++++++++ infra/worker-host/cloud-init.yaml | 34 +++++++++++++++++++++++++++---- 2 files changed, 40 insertions(+), 4 deletions(-) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index e271ff4dc3..9550f44232 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -84,6 +84,15 @@ As `worker` (`sudo -iu worker`): before the first worker runs. Verify the env-token split: `gh api repos/OWNER/REPO --jq .permissions` under `GH_TOKEN` must show push=false. + Seed the commit identity in the same sitting — a fresh volume has none, + and the implementation protocol's first commit dies with "Author identity + unknown" (git refuses to auto-detect from the machine-id hostname): + ```bash + git config --global user.name "doper-worker" + git config --global user.email "" + ``` + Lives in `~/.gitconfig` on the volume — once per soul, like everything + else in this layer. 5. **doperpowers itself** — `git clone` this repo to `~/doperpowers` (dispatch scripts are invoked by path from here; workers get skills via vendoring / plugin install exactly as on the Mac). @@ -100,6 +109,7 @@ As `worker` (`sudo -iu worker`): ```bash grep -Ev '^\s*(#|$)' ~/.env > ~/runner/.env # runner .env is bare KEY=VALUE echo "DOPERPOWERS_HOME=/data/worker/doperpowers" >> ~/runner/.env + chmod 600 ~/runner/.env # holds the same tokens as ~/.env echo "$PATH" > ~/runner/.path ``` Then as root `cd /data/worker/runner && ./svc.sh install worker && diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml index 6753e9d957..00e84588fe 100644 --- a/infra/worker-host/cloud-init.yaml +++ b/infra/worker-host/cloud-init.yaml @@ -103,6 +103,12 @@ write_files: WantedBy=timers.target runcmd: + # runcmd items are merged into ONE #!/bin/sh script, so this set -e governs + # every line below: any installer/apt/npm failure aborts provisioning and + # surfaces in `cloud-init status` instead of yielding a half-built body + # that reports success. (sh is dash — no pipefail — so the curl|bash + # installers below are split into download-then-execute.) + - set -e # HARD GATE: everything below assumes the state volume is mounted. The # fstab entry keeps `nofail` on purpose (a missing volume must degrade to # a reachable system, not systemd emergency mode), which means a missing/ @@ -116,13 +122,19 @@ runcmd: echo "FATAL: /data is not a mountpoint — state volume missing or unlabeled; aborting provisioning" >&2 exit 1 } + # UID boundary hardening: the users module creates the homes with the + # image's login.defs default, which may be group/other-traversable — that + # would let `triage` read worker's clones (the push PAT lives in each + # .git/config). Explicit 0700, idempotent against an existing soul. + - chmod 700 /data/worker /data/triage # Per-body hostname (see header note). /etc/machine-id is minted at this # body's first boot, so every rebuild gets a fresh suffix — old registry # metas read as foreign-host and their pids as dead, which is what lets # parked sessions resume safely after a rebuild. - hostnamectl set-hostname "doper-worker-$(cut -c1-6 /etc/machine-id)" # Node.js LTS (consumer-repo builds/tests: npm ci, playwright, etc.) - - curl -fsSL https://deb.nodesource.com/setup_22.x | bash - + - curl -fsSL https://deb.nodesource.com/setup_22.x -o /tmp/nodesource-setup.sh + - bash /tmp/nodesource-setup.sh - apt-get install -y nodejs # gh CLI (board transitions, PR ops) — official apt repo - mkdir -p -m 755 /etc/apt/keyrings @@ -131,12 +143,14 @@ runcmd: - echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list - apt-get update && apt-get install -y gh # Tailscale (SSH access path — close public 22 after `tailscale up`, see README) - - curl -fsSL https://tailscale.com/install.sh | sh + - curl -fsSL https://tailscale.com/install.sh -o /tmp/tailscale-install.sh + - sh /tmp/tailscale-install.sh # codex CLI (global; worker-agnostic binary, auth is per-user on the volume) - npm install -g @openai/codex # claude CLI — native installer, installed AS the worker (lands in ~/.local/bin - # on the volume; harmless to re-run on a rebuilt body) - - sudo -u worker bash -lc 'curl -fsSL https://claude.ai/install.sh | bash' + # on the volume; harmless to re-run on a rebuilt body). Inner shell is bash, + # so pipefail is available to keep a failed download from being masked. + - sudo -u worker bash -lc 'set -e -o pipefail; curl -fsSL https://claude.ai/install.sh | bash' # worker shell wiring: source the seeded env file (layer 3) in INTERACTIVE # shells (admin ssh, sudo -iu worker). Actions runner jobs do NOT read # .bashrc — non-login shells, and Ubuntu's stock .bashrc early-returns for @@ -150,6 +164,18 @@ runcmd: # triage tenant timer — safe to arm before seeding (ExecCondition guards) - systemctl daemon-reload - systemctl enable --now doper-triage.timer + # Provisioning gate: set -e catches command failures; this catches + # semantically-wrong installs (missing binary, wrong Node major). + - | + for bin in git jq python3 node npm gh tailscale codex; do + command -v "$bin" >/dev/null || { echo "FATAL: $bin missing after provisioning" >&2; exit 1; } + done + node -e 'process.exit(parseInt(process.versions.node) >= 22 ? 0 : 1)' || { + echo "FATAL: node $(node --version) < 22" >&2; exit 1 + } + sudo -u worker bash -lc 'command -v claude' >/dev/null || { + echo "FATAL: claude missing from worker PATH" >&2; exit 1 + } # final_message prints even when runcmd aborted — hence "check status", not "ready". final_message: "doper-worker body boot finished after $UPTIME s — confirm with 'cloud-init status --long' BEFORE layer 3 seeding (infra/worker-host/README.md)" From 7db6b8dc8bfa3ca8ef82f71c7329fd816bd50dcc Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 22:48:39 +0900 Subject: [PATCH 07/12] fix(orchestrating-daemons): bind liveness to host boot --- .../scripts/_codex_lib.sh | 4 +- skills/orchestrating-daemons/scripts/_lib.sh | 35 ++++++---- .../scripts/codex-resume.sh | 4 +- .../scripts/codex-spawn.sh | 2 +- .../scripts/daemon-resume.sh | 4 +- .../scripts/daemon-retire.sh | 6 +- .../scripts/daemon-spawn.sh | 4 +- skills/reviewing-prs/scripts/land-dispatch.sh | 19 +++--- .../reviewing-prs/scripts/review-dispatch.sh | 64 +++++++++++++------ .../test-codex-scripts.sh | 25 ++++++++ .../test-daemon-scripts.sh | 14 ++++ tests/reviewing-prs/test-land-dispatch.sh | 29 +++++++++ tests/reviewing-prs/test-review-dispatch.sh | 50 ++++++++++++++- 13 files changed, 207 insertions(+), 53 deletions(-) diff --git a/skills/orchestrating-daemons/scripts/_codex_lib.sh b/skills/orchestrating-daemons/scripts/_codex_lib.sh index aa4eea25ec..ed9dc97676 100755 --- a/skills/orchestrating-daemons/scripts/_codex_lib.sh +++ b/skills/orchestrating-daemons/scripts/_codex_lib.sh @@ -8,8 +8,8 @@ # is the same JSON contract as claude daemons plus: # engine "codex" # pid codex process pid of the CURRENT turn (liveness: _pid_alive — -# kill -0 gated on `host` matching this machine; a registry that -# migrated on a state volume carries pids that are dead here) +# kill -0 gated on `host` + `boot_id` matching this boot; migrated +# or prior-boot registry pids are dead here) # effort model_reasoning_effort # event_log JSONL event stream of the current turn (--json stdout) # The codex session id (thread.started) keys the registry, so board-bind.sh / diff --git a/skills/orchestrating-daemons/scripts/_lib.sh b/skills/orchestrating-daemons/scripts/_lib.sh index cf8cd3f599..f68c9d8818 100755 --- a/skills/orchestrating-daemons/scripts/_lib.sh +++ b/skills/orchestrating-daemons/scripts/_lib.sh @@ -22,12 +22,19 @@ set -euo pipefail DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" mkdir -p "$DAEMON_HOME" -# Host identity — stamped into metas as `host` at every registration. A pid -# (and a `claude agents` short) is only meaningful on the machine that recorded -# it: when the registry lives on a state volume that migrates to a new host, -# a reused pid number would otherwise read as a live worker and wrongly block -# resume / dedupe / retire. Override with DAEMON_HOST for tests. +# Host + boot identity — stamped into metas at every registration. A pid (and a +# `claude agents` short) is only meaningful in the host's current boot. Hostname +# catches volume migration; boot id catches a rebuilt/rebooted machine that kept +# its name but received a fresh pid namespace. Override both values for tests. DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" +_boot_id() { + if [ -r /proc/sys/kernel/random/boot_id ]; then + cat /proc/sys/kernel/random/boot_id + elif command -v sysctl >/dev/null 2>&1; then + sysctl -n kern.boottime 2>/dev/null | sed -E 's/.*sec = ([0-9]+).*/\1/' || true + fi +} +DAEMON_BOOT_ID="${DAEMON_BOOT_ID:-$(_boot_id)}" # How long the spawn/resume WATCHER polls `claude agents` for a turn to finish — # a wait bound only, NOT a turn budget. The toolkit never kills a turn: every turn @@ -44,14 +51,20 @@ _err_path() { printf '%s/%s.err' "$DAEMON_HOME" "$1"; } # Strip ANSI color codes (the `claude --bg` banner is colored even when piped). _strip_ansi() { sed -E 's/\x1b\[[0-9;]*m//g'; } -# rc 0 iff pid <1> is a live process OF THIS HOST. <2> is the meta's recorded -# `host` (empty = legacy meta with no host field → assume local, preserving -# pre-host behavior). A host mismatch is DEAD regardless of kill -0: the -# process did not migrate with the state volume, only its number did. +# rc 0 iff a meta's recorded host/boot identity belongs to this boot. Empty +# values preserve legacy local behavior. +_identity_local() { + local host="${1:-}" boot_id="${2:-}" + [ -z "$host" ] || [ "$host" = "$DAEMON_HOST" ] || return 1 + [ -z "$boot_id" ] || [ -z "$DAEMON_BOOT_ID" ] || [ "$boot_id" = "$DAEMON_BOOT_ID" ] || return 1 +} + +# rc 0 iff pid <1> is live in THIS HOST BOOT. An identity mismatch is DEAD +# regardless of kill -0: only the number survived. _pid_alive() { - local pid="$1" host="${2:-}" + local pid="$1" [ -n "$pid" ] || return 1 - [ -z "$host" ] || [ "$host" = "$DAEMON_HOST" ] || return 1 + _identity_local "${2:-}" "${3:-}" || return 1 kill -0 "$pid" 2>/dev/null } diff --git a/skills/orchestrating-daemons/scripts/codex-resume.sh b/skills/orchestrating-daemons/scripts/codex-resume.sh index 3e10f32b59..cd6252b807 100755 --- a/skills/orchestrating-daemons/scripts/codex-resume.sh +++ b/skills/orchestrating-daemons/scripts/codex-resume.sh @@ -24,7 +24,7 @@ msg="${2:?missing message}" # dead by definition — resuming HERE is exactly the recovery act. if [ "$(_meta_get "$uuid" status)" = "working" ]; then pid="$(_meta_get "$uuid" pid)" - if _pid_alive "$pid" "$(_meta_get "$uuid" host)"; then + if _pid_alive "$pid" "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then echo "codex-resume: a turn is still running (pid $pid) — wait for it or retire" >&2 exit 1 fi @@ -118,7 +118,7 @@ fi turns="$(_meta_get "$uuid" turns)"; turns=$((${turns:-1} + 1)) _meta_set "$uuid" pid "$(cat "$run.pid" 2>/dev/null || printf '')" \ - host "$DAEMON_HOST" \ + host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ event_log "$run.events.jsonl" status "working" turns "$turns" updated "$(_now)" [ -f "$run.rc" ] && _meta_set "$uuid" \ status "$(_codex_final_status "$(cat "$run.rc")" "$run.events.jsonl")" updated "$(_now)" diff --git a/skills/orchestrating-daemons/scripts/codex-spawn.sh b/skills/orchestrating-daemons/scripts/codex-spawn.sh index ba4fa16140..7d5b80bcfc 100755 --- a/skills/orchestrating-daemons/scripts/codex-spawn.sh +++ b/skills/orchestrating-daemons/scripts/codex-spawn.sh @@ -89,7 +89,7 @@ status="working" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$(printf '%.8s' "$uuid")" name "$name" \ task "$task" cwd "$runcwd" worktree "$worktree" model "$model" effort "$effort" \ - engine "codex" pid "$pid" host "$DAEMON_HOST" event_log "$run.events.jsonl" \ + engine "codex" pid "$pid" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" event_log "$run.events.jsonl" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" # The wrapper may have finalized between our check and the write — re-apply. [ -f "$run.rc" ] && _meta_set "$uuid" \ diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index b03853bbab..357b59d144 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -96,7 +96,7 @@ if [ "$poll_rc" -ne 0 ] || [ -z "$newuuid" ]; then # chain to the new turn and mark status=working. Do NOT write the reply file # or bump turns; no final reply has landed. daemon-reply.sh reads the CURRENT # session's transcript, so it will surface the reply once the turn finishes. - _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" status "working" updated "$(_now)" + _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" status "working" updated "$(_now)" # The fork is confirmed live — the superseded turn's session can go. if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 @@ -117,7 +117,7 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error # Reply file stays keyed by the ORIGINAL uuid; read the reply from the new turn. _record_reply "$newuuid" "$uuid" "$state" -_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" \ +_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" updated "$(_now)" turns "$((turns + 1))" # Purge the superseded turn: deregister it (jobs entry + supervisor record) and diff --git a/skills/orchestrating-daemons/scripts/daemon-retire.sh b/skills/orchestrating-daemons/scripts/daemon-retire.sh index fd2637b804..ec9683faa6 100755 --- a/skills/orchestrating-daemons/scripts/daemon-retire.sh +++ b/skills/orchestrating-daemons/scripts/daemon-retire.sh @@ -24,7 +24,7 @@ if [ "$engine" = "codex" ]; then status="$(_meta_get "$uuid" status)" pid="$(_meta_get "$uuid" pid)" if { [ "$status" = "working" ] || [ "$status" = "blocked" ]; } && \ - _pid_alive "$pid" "$(_meta_get "$uuid" host)"; then + _pid_alive "$pid" "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then kill "$pid" 2>/dev/null || true # Killing the pid makes _codex_launch's own detached wrapper (the one # that finalizes status+reply once `wait` on the codex pid returns) wake @@ -42,7 +42,9 @@ if [ "$engine" = "codex" ]; then fi resume_hint="codex resume $uuid" else - [ -n "$short" ] && claude stop "$short" >/dev/null 2>&1 || true + if _identity_local "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then + [ -n "$short" ] && claude stop "$short" >/dev/null 2>&1 || true + fi resume_hint="claude --resume $uuid" fi diff --git a/skills/orchestrating-daemons/scripts/daemon-spawn.sh b/skills/orchestrating-daemons/scripts/daemon-spawn.sh index aa75c1b527..cdfa8b8533 100755 --- a/skills/orchestrating-daemons/scripts/daemon-spawn.sh +++ b/skills/orchestrating-daemons/scripts/daemon-spawn.sh @@ -75,7 +75,7 @@ if [ "$nowait" -eq 1 ]; then esac _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" host "$DAEMON_HOST" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" echo "daemon spawned (no-wait): $name [$short / $uuid] status=$status (reply: daemon-reply.sh $short)" exit 0 @@ -104,7 +104,7 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error _record_reply "$uuid" "$uuid" "$state" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" host "$DAEMON_HOST" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" wtnote=""; [ -n "$worktree" ] && wtnote=" worktree=$runcwd (branch worktree-$wt)" diff --git a/skills/reviewing-prs/scripts/land-dispatch.sh b/skills/reviewing-prs/scripts/land-dispatch.sh index cfc37443c7..b65f070727 100755 --- a/skills/reviewing-prs/scripts/land-dispatch.sh +++ b/skills/reviewing-prs/scripts/land-dispatch.sh @@ -35,8 +35,8 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" -# Host identity for pid checks — same semantics and default as review-dispatch. -DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" +# shellcheck source=../../orchestrating-daemons/scripts/_lib.sh +. "$SKILL_DIR/../orchestrating-daemons/scripts/_lib.sh" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="${BOARD_SCRIPTS:-$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)}" @@ -63,7 +63,7 @@ case "${LAND_ENABLED:-false}" in *) LAND_MODE="dry-run" ;; esac -# Newest land-pr- registry entry → "uuid|status|current|engine|pid|host" +# Newest land-pr- registry entry → "uuid|status|current|engine|pid|host|boot" # (empty if none). _land_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' @@ -83,17 +83,18 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""))) + print("%s|%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""), + m.get("boot_id", ""))) PY } # rc 0 when the worker's CURRENT turn is live (same semantics as review-dispatch: # a codex pid counts only on the host that recorded it). -_is_live() { # +_is_live() { # + _identity_local "${4:-}" "${5:-}" || return 1 if [ "$2" = "codex" ]; then [ -n "$3" ] || return 1 - [ -z "${4:-}" ] || [ "$4" = "$DAEMON_HOST" ] || return 1 kill -0 "$3" 2>/dev/null return fi @@ -166,10 +167,10 @@ meta="$(_land_meta "$pr")" if [ -n "$meta" ]; then uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" current="${rest%%|*}"; rest="${rest#*|}"; w_engine="${rest%%|*}"; rest="${rest#*|}" - w_pid="${rest%%|*}"; w_host="${rest#*|}" + w_pid="${rest%%|*}"; rest="${rest#*|}"; w_host="${rest%%|*}"; w_boot="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$w_engine" "$w_pid" "$w_host"; then + if _is_live "$current" "$w_engine" "$w_pid" "$w_host" "$w_boot"; then echo "#$pr: skip active land worker"; exit 0 fi _retire "$uuid" ;; diff --git a/skills/reviewing-prs/scripts/review-dispatch.sh b/skills/reviewing-prs/scripts/review-dispatch.sh index 867ab235b5..fde86f4867 100755 --- a/skills/reviewing-prs/scripts/review-dispatch.sh +++ b/skills/reviewing-prs/scripts/review-dispatch.sh @@ -52,9 +52,8 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" -# Host identity for pid checks — a pid recorded on another machine (registry -# migrated on a state volume) is not a live worker here. Same default as _lib.sh. -DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" +# shellcheck source=../../orchestrating-daemons/scripts/_lib.sh +. "$SKILL_DIR/../orchestrating-daemons/scripts/_lib.sh" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)" @@ -87,7 +86,7 @@ ENGINE_BLOCK_FILE="$SKILL_DIR/references/engine-blocks/engine-codex-review.md" FALLBACK_FILE="$SKILL_DIR/references/engine-blocks/fallback-engine.md" REVIEW_ENGINE="$SCRIPT_DIR/review-engine.sh" -# Newest review-pr- registry entry → "uuid|status|current|engine|pid|host" +# Newest review-pr- registry entry → "uuid|status|current|engine|pid|host|boot" # (empty if none). _reviewer_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' @@ -107,18 +106,19 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""))) + print("%s|%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""), + m.get("boot_id", ""))) PY } # rc 0 when the reviewer's CURRENT turn is live: claude → session uuid visible # in `claude agents`; codex → recorded pid alive ON THIS HOST (a foreign-host # pid is dead by definition — only its number migrated with the registry). -_is_live() { # +_is_live() { # + _identity_local "${4:-}" "${5:-}" || return 1 if [ "$2" = "codex" ]; then [ -n "$3" ] || return 1 - [ -z "${4:-}" ] || [ "$4" = "$DAEMON_HOST" ] || return 1 kill -0 "$3" 2>/dev/null return fi @@ -133,24 +133,45 @@ sys.exit(0 if any(a.get("sessionId") == os.environ["CUR"] for a in d) else 1)' _retire() { "$DAEMON_SCRIPTS/daemon-retire.sh" "$1" >/dev/null 2>&1 || true; } -# rc 0 when some `claude agents` row's cwd equals worktree path <1> — a live -# daemon (of ANY kind, not just a review worker) is sitting in it. This is -# defense-in-depth ON TOP OF the registry dedupe check above, not a -# replacement for it: fail-open on an `agents` flake (empty/unreadable list → -# not occupied → removal proceeds) is correct here, do not invert it. +# rc 0 when some LOCAL `claude agents` row's cwd equals worktree path <1>. A +# visible row with matching foreign registry metadata migrated with the session +# store, not the process, so it does not occupy the worktree. Unmanaged rows +# have no identity evidence and remain conservatively local/occupied. _wt_occupied() { - claude agents --json --all 2>/dev/null | WT="$1" python3 -c ' -import json, os, sys + claude agents --json --all 2>/dev/null | \ + DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" DAEMON_BOOT_ID="$DAEMON_BOOT_ID" WT="$1" python3 -c ' +import glob, json, os, sys try: d = json.load(sys.stdin) except Exception: d = [] -sys.exit(0 if any(a.get("cwd") == os.environ["WT"] for a in d) else 1)' && return 0 +metas = {} +for p in glob.glob(os.path.join(os.environ["DAEMON_HOME"], "*.json")): + if p.endswith(".reply.json"): + continue + try: + m = json.load(open(p)) + except Exception: + continue + if (m.get("engine") or "claude") == "claude" and m.get("current"): + metas[str(m["current"])] = m +def local(m): + host = str(m.get("host") or "") + boot = str(m.get("boot_id") or "") + return (not host or host == os.environ["DAEMON_HOST"]) and \ + (not boot or not os.environ["DAEMON_BOOT_ID"] or boot == os.environ["DAEMON_BOOT_ID"]) +for a in d: + if a.get("cwd") != os.environ["WT"]: + continue + m = metas.get(str(a.get("sessionId") or "")) + if m is None or local(m): + sys.exit(0) +sys.exit(1)' && return 0 # codex workers never appear in `claude agents` — scan the registry, but # count ONLY codex metas with a live pid; a stale claude-engine `working` # meta must NOT start blocking removal (the claude path's fail-open # behavior above is unchanged). - DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" WT="$1" python3 - <<'PY' + DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" DAEMON_BOOT_ID="$DAEMON_BOOT_ID" WT="$1" python3 - <<'PY' import glob, json, os, sys home = os.environ["DAEMON_HOME"]; wt = os.environ["WT"] for p in glob.glob(os.path.join(home, "*.json")): @@ -167,6 +188,9 @@ for p in glob.glob(os.path.join(home, "*.json")): host = str(m.get("host") or "") if host and host != os.environ["DAEMON_HOST"]: continue # foreign-host pid — the process did not migrate with the registry + boot_id = str(m.get("boot_id") or "") + if boot_id and os.environ["DAEMON_BOOT_ID"] and boot_id != os.environ["DAEMON_BOOT_ID"]: + continue # prior-boot pid — the pid namespace did not survive the reboot pid = str(m.get("pid") or "") if pid.isdigit(): try: @@ -301,16 +325,16 @@ PY # Dedupe verdict for PR <1> in mode <2> (triggered|sweep), cr-label flag <3>. # Prints: "dispatch" | "respawn " | "skip ". _decide() { - local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid whost + local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid whost wboot if [ "$cr" = "1" ]; then echo "skip confident-ready label (remove it to force re-review)"; return; fi meta="$(_reviewer_meta "$pr")" if [ -z "$meta" ]; then echo "dispatch"; return; fi uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" current="${rest%%|*}"; rest="${rest#*|}"; engine="${rest%%|*}"; rest="${rest#*|}" - pid="${rest%%|*}"; whost="${rest#*|}" + pid="${rest%%|*}"; rest="${rest#*|}"; whost="${rest%%|*}"; wboot="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$engine" "$pid" "$whost"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; + if _is_live "$current" "$engine" "$pid" "$whost" "$wboot"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; retired) echo "dispatch" ;; *) if [ "$mode" = "triggered" ]; then echo "respawn $uuid" diff --git a/tests/orchestrating-daemons/test-codex-scripts.sh b/tests/orchestrating-daemons/test-codex-scripts.sh index 280a240e0e..fcd6881c19 100755 --- a/tests/orchestrating-daemons/test-codex-scripts.sh +++ b/tests/orchestrating-daemons/test-codex-scripts.sh @@ -40,6 +40,7 @@ export DAEMON_HOME="$TEST_ROOT/registry" export STUB_STATE="$TEST_ROOT/stub" export DAEMON_TIMEOUT=10 export DAEMON_UUID_POLL=5 +export DAEMON_BOOT_ID="boot-current" WORK="$TEST_ROOT/work" mkdir -p "$HOME" "$WORK" "$STUB_STATE" # Fake code-mode host under the hermetic HOME so _codex_launch's only-if-exists @@ -314,6 +315,7 @@ echo "== host-aware liveness: a foreign-host pid is dead by definition ==" # resume proceeds (resuming here IS the recovery act) and retire never signals # an unrelated local process. assert_equals "$(meta_field "$uuid_e" host)" "$(hostname)" "codex-spawn stamps host" +assert_equals "$(meta_field "$uuid_e" boot_id)" "$DAEMON_BOOT_ID" "codex-spawn stamps boot id" sleep 30 & foreign_pid=$! uuid_mig="hostmig1-0000-4000-8000-000000000000" printf '{"uuid":"%s","current":"%s","name":"migrated","short":"hostmig1","engine":"codex","pid":"%s","host":"old-host","status":"working","cwd":"%s"}' \ @@ -337,6 +339,29 @@ else fail "retire leaves a foreign-host working pid alone" fi assert_equals "$(meta_field "$uuid_mig2" status)" "retired" "foreign-host record is still retired" + +# Rebuilding the same named host also creates a fresh pid namespace. A stale +# pid from the prior boot must therefore be treated like a foreign-host pid. +uuid_reboot="reboot01-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","name":"rebooted","short":"reboot01","engine":"codex","pid":"%s","host":"%s","boot_id":"boot-old","status":"working","cwd":"%s"}' \ + "$uuid_reboot" "$uuid_reboot" "$foreign_pid" "$(hostname)" "$WORK" > "$DAEMON_HOME/$uuid_reboot.json" +if out="$("$SCRIPTS_DIR/codex-resume.sh" "$uuid_reboot" "resume after reboot" 2>&1)"; then + pass "resume treats a prior-boot pid as dead" +else + fail "resume treats a prior-boot pid as dead"; echo " in: $out" +fi +assert_contains "$out" "stub reply: resume after reboot" "resume proceeds after a same-host reboot" +assert_equals "$(meta_field "$uuid_reboot" boot_id)" "$DAEMON_BOOT_ID" "resume re-stamps boot id" + +uuid_reboot2="reboot02-0000-4000-8000-000000000000" +printf '{"uuid":"%s","name":"rebooted2","short":"reboot02","engine":"codex","pid":"%s","host":"%s","boot_id":"boot-old","status":"working"}' \ + "$uuid_reboot2" "$foreign_pid" "$(hostname)" > "$DAEMON_HOME/$uuid_reboot2.json" +"$SCRIPTS_DIR/daemon-retire.sh" "$uuid_reboot2" >/dev/null +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "retire leaves a prior-boot pid alone" +else + fail "retire leaves a prior-boot pid alone" +fi kill "$foreign_pid" 2>/dev/null || true wait "$foreign_pid" 2>/dev/null || true diff --git a/tests/orchestrating-daemons/test-daemon-scripts.sh b/tests/orchestrating-daemons/test-daemon-scripts.sh index f71c07cae1..f9eaceb03c 100755 --- a/tests/orchestrating-daemons/test-daemon-scripts.sh +++ b/tests/orchestrating-daemons/test-daemon-scripts.sh @@ -33,6 +33,11 @@ assert_contains() { if printf '%s' "$1" | grep -Fq -- "$2"; then pass "$3"; else fail "$3"; echo " expected to find: $2"; echo " in: $1"; fi } +assert_not_contains() { + if printf '%s' "$1" | grep -Fq -- "$2"; then + fail "$3"; echo " expected NOT to find: $2"; echo " in: $1" + else pass "$3"; fi +} assert_file_exists() { if [[ -f "$1" ]]; then pass "$2"; else fail "$2"; echo " missing: $1"; fi } @@ -45,6 +50,7 @@ export HOME="$TEST_ROOT/home" export DAEMON_HOME="$TEST_ROOT/registry" export STUB_STATE="$TEST_ROOT/stub" export DAEMON_TIMEOUT=10 +export DAEMON_BOOT_ID="boot-current" WORK="$TEST_ROOT/work" mkdir -p "$HOME" "$WORK" "$STUB_STATE/agents" @@ -332,6 +338,14 @@ assert_contains "$(cat "$STUB_STATE/log/calls.log")" "stop $SHORT2" "retire stop "$SCRIPTS_DIR/daemon-retire.sh" "$SHORT" purge >/dev/null assert_file_absent "$DAEMON_HOME/$UUID.json" "retire purge removes the registry record" +FOREIGN_UUID="face0000-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","short":"face0000","name":"foreign","engine":"claude","host":"old-host","boot_id":"boot-old","status":"working"}' \ + "$FOREIGN_UUID" "$FOREIGN_UUID" > "$DAEMON_HOME/$FOREIGN_UUID.json" +: > "$STUB_STATE/log/calls.log" +"$SCRIPTS_DIR/daemon-retire.sh" "$FOREIGN_UUID" >/dev/null +assert_not_contains "$(cat "$STUB_STATE/log/calls.log")" "stop face0000" "retire does not stop a foreign-host Claude session" +assert_contains "$(cat "$DAEMON_HOME/$FOREIGN_UUID.json")" '"status": "retired"' "foreign-host Claude record is still retired" + # ---- 6) worktree isolation (native --worktree threading) --------------------- echo "worktree isolation:" WT_REPO="$TEST_ROOT/wtrepo"; mkdir -p "$WT_REPO" diff --git a/tests/reviewing-prs/test-land-dispatch.sh b/tests/reviewing-prs/test-land-dispatch.sh index a7594fba2c..0f313fbb44 100755 --- a/tests/reviewing-prs/test-land-dispatch.sh +++ b/tests/reviewing-prs/test-land-dispatch.sh @@ -41,6 +41,7 @@ export BIND_LOG="$TEST_ROOT/bind.log"; : > "$BIND_LOG" export EDIT_LOG="$TEST_ROOT/edit.log"; : > "$EDIT_LOG" export PROMPT_DIR="$TEST_ROOT/prompts"; mkdir -p "$PROMPT_DIR" export STUB_COUNT="$TEST_ROOT/count" +export DAEMON_BOOT_ID="boot-current" # real git: bare origin + working clone with main and a PR head branch ORIGIN="$TEST_ROOT/origin.git" @@ -245,6 +246,20 @@ reset_state; seed_lander working # agents.json [] → session gone assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "dead land worker retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "dead land worker respawned" +reset_state +python3 - <<'PY' +import json, os +u = "feed0000-0000-4000-8000-000000000000" +json.dump({"uuid": u, "current": u, "name": "land-pr-5", "engine": "claude", + "host": "old-host", "boot_id": "boot-old", "status": "working", + "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo '[{"id": "feedcafe", "sessionId": "feed0000-0000-4000-8000-000000000000"}]' > "$MOCK_DIR/agents.json" +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host Claude land worker is retired despite a visible migrated session" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "foreign-host Claude land worker is respawned" + reset_state; seed_lander idle "$DISPATCH" 5 > /dev/null assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "finished land worker retired (explicit dispatch = fresh signal)" @@ -267,6 +282,20 @@ PY "$DISPATCH" 5 > /dev/null assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host live pid → land worker treated as dead and retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "foreign-host live pid → respawned" +reset_state +PID="$LANDPID" H="$(hostname)" python3 - <<'PY' +import json, os +json.dump({"uuid": "feed0000-0000-4000-8000-000000000000", + "current": "feed0000-0000-4000-8000-000000000000", + "name": "land-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": "boot-old", + "status": "working", "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], + "feed0000-0000-4000-8000-000000000000.json"), "w")) +PY +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "prior-boot live pid → land worker treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "prior-boot live pid → respawned" kill "$LANDPID" 2>/dev/null; wait "$LANDPID" 2>/dev/null || true # ---- ticketless PR -------------------------------------------------------------------- diff --git a/tests/reviewing-prs/test-review-dispatch.sh b/tests/reviewing-prs/test-review-dispatch.sh index 88d97b932c..e8415175e8 100755 --- a/tests/reviewing-prs/test-review-dispatch.sh +++ b/tests/reviewing-prs/test-review-dispatch.sh @@ -40,6 +40,7 @@ export MOCK_LOG="$TEST_ROOT/gh-calls.log"; : > "$MOCK_LOG" export SPAWN_LOG="$TEST_ROOT/spawn.log"; : > "$SPAWN_LOG" export PROMPT_DIR="$TEST_ROOT/prompts"; mkdir -p "$PROMPT_DIR" export STUB_COUNT="$TEST_ROOT/count" +export DAEMON_BOOT_ID="boot-current" # real git: bare origin + working clone with main and a PR head branch ORIGIN="$TEST_ROOT/origin.git" @@ -242,6 +243,20 @@ out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "dead reviewer retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "dead reviewer respawned" +reset_state +H="old-host" B="boot-old" python3 - <<'PY' +import json, os +u = "feed0000-0000-4000-8000-000000000000" +json.dump({"uuid": u, "current": u, "name": "review-pr-5", "engine": "claude", + "host": os.environ["H"], "boot_id": os.environ["B"], "status": "working", + "updated": "2026-07-08T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo '[{"id": "feedcafe", "sessionId": "feed0000-0000-4000-8000-000000000000"}]' > "$MOCK_DIR/agents.json" +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host Claude reviewer is retired despite a visible migrated session" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host Claude reviewer is respawned" + reset_state; seed_reviewer idle out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "triggered mode retires a finished reviewer" @@ -349,6 +364,23 @@ if [ -d "$WT" ]; then pass "worktree still exists after the refused dispatch"; e fail "worktree still exists after the refused dispatch"; fi reset_state +# A session restored from a foreign state volume can still be visible in the +# local dashboard, but its foreign registry identity must not occupy the worktree. +U="foreign1-0000-4000-8000-000000000000" WT="$WT" python3 - <<'PY' +import json, os +u = os.environ["U"] +json.dump({"uuid": u, "current": u, "name": "other-daemon", "engine": "claude", + "cwd": os.environ["WT"], "host": "old-host", "boot_id": "boot-old", + "status": "working", "updated": "2026-07-08T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo "[{\"id\": \"foreign1\", \"sessionId\": \"foreign1-0000-4000-8000-000000000000\", \"cwd\": \"$WT\"}]" \ + > "$MOCK_DIR/agents.json" +out="$("$DISPATCH" 5 2>&1)" || true +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host Claude session does not occupy the worktree" +assert_not_contains "$out" "live daemon occupies" "foreign-host Claude session does not block removal" +reset_state + # ---- sweep failure isolation ---------------------------------------------------- echo "sweep failure isolation:" # PR 4 (earlier in the sweep order) fails mid-dispatch; PR 5 must still be @@ -647,13 +679,27 @@ out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "retire:cdec8003" "foreign-host live pid → reviewer treated as dead and retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host live pid → respawned" reset_state -# control: same live pid, HOST MATCHING this machine → still a live occupant +# A rebuilt host can keep its hostname while getting a fresh pid namespace. PID="$DEDUPID" H="$(hostname)" python3 - <<'PY' import json, os json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", "current": "cdec8003-0000-4000-8000-000000000000", "name": "review-pr-5", "engine": "codex", - "pid": os.environ["PID"], "host": os.environ["H"], "status": "working", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": "boot-old", + "status": "working", "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:cdec8003" "prior-boot live pid → reviewer treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "prior-boot live pid → respawned" +reset_state +# control: same live pid, HOST MATCHING this machine → still a live occupant +PID="$DEDUPID" H="$(hostname)" B="$DAEMON_BOOT_ID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": os.environ["B"], "status": "working", "updated": "2026-07-10T00:00:00Z"}, open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) PY From b525ed0a726a6bd5a78a1e057428df751a7dfa74 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 22:51:45 +0900 Subject: [PATCH 08/12] fix(orchestrating-daemons,infra): strip RUNNER_TRACKING_ID at detach; triage verify from admin shell MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourth review wave, both findings confirmed: - Detached workers spawned from an Actions dispatch job inherited RUNNER_TRACKING_ID, and the runner's post-job cleanup kills any surviving process whose environ carries that marker — nohup/--bg detach the session, not the env, so the primary automated dispatch path reaped its own workers at job exit (stale 'working' meta left behind). All three detach points (daemon-spawn, daemon-resume fork, _codex_launch wrapper) now spawn under env -u RUNNER_TRACKING_ID — a no-op outside Actions. README §2.6 names this as the mechanism instead of implying nohup suffices. - README §5.4 had the first-tick verification run 'sudo systemctl' inside the sudo -iu triage shell — triage has no sudo, a locked password, and no journal access. Verification now runs from the admin shell; §5 intro marks step 4 as the exception. Suites: test-daemon-scripts, test-codex-scripts, test-review-dispatch all pass; shell lint clean. --- infra/worker-host/README.md | 12 +++++++++--- skills/orchestrating-daemons/scripts/_codex_lib.sh | 6 +++++- .../orchestrating-daemons/scripts/daemon-resume.sh | 5 ++++- skills/orchestrating-daemons/scripts/daemon-spawn.sh | 7 ++++++- 4 files changed, 24 insertions(+), 6 deletions(-) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index 9550f44232..dd922023c2 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -115,7 +115,10 @@ As `worker` (`sudo -iu worker`): Then as root `cd /data/worker/runner && ./svc.sh install worker && ./svc.sh start`. Runner jobs must only run the dispatch ritual (render → spawn --no-wait → bind) and exit; workers are detached processes outside - the job, so job timeouts never touch them. + the job. What actually makes that true is not nohup/--bg but the spawn + paths stripping `RUNNER_TRACKING_ID` (daemon-spawn / daemon-resume / + `_codex_launch`) — the runner's post-job cleanup kills any surviving + process whose environ still carries that job marker, detached or not. ## 3. Verification gate (before trusting it) @@ -164,7 +167,8 @@ sits on it only because `$HOME` does. `worker` runs implement/review workers that execute PR-derived code. User separation (uid 1011, mode-600 `.env`) is the cheap wall between them. -Seed as `triage` (`sudo -iu triage`) — prerequisites first: the p86 +Seed as `triage` (`sudo -iu triage`; step 4 is the exception — it runs from +the admin shell) — prerequisites first: the p86 migration must be live in Supabase and the descriptive labels created on ida-solution (`references/setup.md` §0/§4): @@ -176,7 +180,9 @@ ida-solution (`references/setup.md` §0/§4): 3. **Env** — copy `env.triage.example` → that same skill dir's `.env`, fill it, `chmod 600`. Keep `TRIAGE_K=1` until ticket quality is trusted (`references/setup.md` §5). -4. **Verify one tick** — `sudo systemctl start doper-triage.service`, then +4. **Verify one tick** — from the ADMIN/root shell, not the triage shell + (`triage` has no sudo, a locked password, and cannot read the system + journal): `systemctl start doper-triage.service`, then `journalctl -u doper-triage.service -f`. The timer (installed by cloud-init, 10-minute cadence) takes over from there. 5. **Mac handoff** — once a VM tick has filed a correct ticket, unload the diff --git a/skills/orchestrating-daemons/scripts/_codex_lib.sh b/skills/orchestrating-daemons/scripts/_codex_lib.sh index ed9dc97676..8a278edd8d 100755 --- a/skills/orchestrating-daemons/scripts/_codex_lib.sh +++ b/skills/orchestrating-daemons/scripts/_codex_lib.sh @@ -241,7 +241,11 @@ _codex_launch() { if [ -z "${CODEX_CODE_MODE_HOST_PATH:-}" ] && [ -x "$HOME/.local/bin/codex-code-mode-host" ]; then export CODEX_CODE_MODE_HOST_PATH="$HOME/.local/bin/codex-code-mode-host" fi - nohup bash -c ' + # env -u RUNNER_TRACKING_ID: an Actions runner's post-job cleanup kills any + # process whose environ still carries the job's tracking marker — the + # wrapper and its codex child must outlive a runner dispatch job (see + # daemon-spawn.sh). No-op outside Actions. + nohup env -u RUNNER_TRACKING_ID bash -c ' set -u DIR="$1"; cwd="$2"; taskf="$3"; run="$4"; shift 4 # shellcheck source=/dev/null diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index 357b59d144..091cda0c8f 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -62,8 +62,11 @@ args+=( "$msg" ) # the meta is stranded status=working with `current` still on the old (stopped) # turn. A nonzero exit AND a banner with no parseable short id both land in the # same error path. +# env -u RUNNER_TRACKING_ID: the forked agent must survive a runner dispatch +# job's post-job cleanup, which kills marker-carrying processes (see +# daemon-spawn.sh). newshort="" -if banner="$(cd "$cwd" && claude "${args[@]}" &1 | _strip_ansi)"; then +if banner="$(cd "$cwd" && env -u RUNNER_TRACKING_ID claude "${args[@]}" &1 | _strip_ansi)"; then newshort="$(printf '%s\n' "$banner" | sed -n 's/.*backgrounded · \([0-9a-f][0-9a-f]*\).*/\1/p' | head -1)" fi if [ -z "$newshort" ]; then diff --git a/skills/orchestrating-daemons/scripts/daemon-spawn.sh b/skills/orchestrating-daemons/scripts/daemon-spawn.sh index cdfa8b8533..73a5b97078 100755 --- a/skills/orchestrating-daemons/scripts/daemon-spawn.sh +++ b/skills/orchestrating-daemons/scripts/daemon-spawn.sh @@ -49,7 +49,12 @@ fi [ -n "$model" ] && args+=( --model "$model" ) args+=( "$task" ) -banner="$(cd "$cwd" && claude "${args[@]}" &1 | _strip_ansi)" +# env -u RUNNER_TRACKING_ID: under a GitHub Actions runner the job env carries +# this tracking marker, and the runner's post-job cleanup kills any surviving +# process whose environ still has it — nohup/--bg detach the session, not the +# env. Stripping it at spawn is what actually lets the daemon outlive a +# dispatch job (a no-op everywhere else: env -u of an unset var). +banner="$(cd "$cwd" && env -u RUNNER_TRACKING_ID claude "${args[@]}" &1 | _strip_ansi)" short="$(printf '%s\n' "$banner" | sed -n 's/.*backgrounded · \([0-9a-f][0-9a-f]*\).*/\1/p' | head -1)" [ -n "$short" ] || { echo "spawn failed — could not parse background id from:" >&2; echo "$banner" >&2; exit 1; } From d49e3522e3d57567cc8a99669230af3573673d46 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 23:08:17 +0900 Subject: [PATCH 09/12] =?UTF-8?q?fix(infra):=20codex-auth=20seeding=20?= =?UTF-8?q?=E2=80=94=20GNU=20install=20refuses=20pipe-backed=20/dev/stdin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fifth review wave, confirmed empirically (GNU coreutils install exits 1 with 'skipping file /dev/stdin, as it was replaced while being copied' and creates nothing): the §2.2 recipe could never seed auth.json on the Ubuntu target. Replaced with umask 077 + cat (file born 0600, still no exposure window) followed by chown to worker. --- infra/worker-host/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md index dd922023c2..e54c693b00 100644 --- a/infra/worker-host/README.md +++ b/infra/worker-host/README.md @@ -57,11 +57,13 @@ As `worker` (`sudo -iu worker`): the two-token scope split), `chmod 600 ~/.env`. 2. **codex auth** — `worker` is deliberately not SSH-reachable (locked password, no authorized keys), so copy through your login account (root - on Hetzner) and install with ownership and mode set from birth: + on Hetzner). GNU `install` refuses a pipe-backed `/dev/stdin` ("replaced + while being copied"), so write under `umask 077` — the file is born 0600 + with no exposure window — then hand it to `worker`: ```bash ssh root@host 'install -d -m 700 -o worker -g worker /data/worker/.codex' - ssh root@host 'install -m 600 -o worker -g worker /dev/stdin /data/worker/.codex/auth.json' \ - < ~/.codex/auth.json + ssh root@host 'umask 077 && cat > /data/worker/.codex/auth.json \ + && chown worker:worker /data/worker/.codex/auth.json' < ~/.codex/auth.json ``` (auth.json is portable by design; device-code flow is the fallback if the workspace enables it.) From 5bb3888e2be61810d41e340f5668d002715fe709 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 23:07:36 +0900 Subject: [PATCH 10/12] fix(orchestrating-daemons): register resumed turn early --- .../scripts/daemon-resume.sh | 61 +++++++++---------- .../test-daemon-scripts.sh | 29 +++++++++ 2 files changed, 57 insertions(+), 33 deletions(-) diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index 091cda0c8f..15acfd7774 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -76,6 +76,25 @@ if [ -z "$newshort" ]; then exit 1 fi +# Register local ownership as soon as the new UUID materializes, BEFORE the +# terminal watcher can block for hours. This is the liveness handoff: dispatch +# must see the active fork as local, not keep reading the migrated source host. +newuuid="" +if uuid_out="$(_poll_uuid "$newshort")"; then + newuuid="${uuid_out%% *}" +fi +case "$newuuid" in + ""|*[!0-9a-f-]*) + _meta_set "$uuid" status "error" pending_short "$newshort" updated "$(_now)" + echo "resume: forked agent $newshort produced no usable session uuid; kept previous current (recover via meta pending_short)." >&2 + exit 1 ;; +esac +_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ + status "working" updated "$(_now)" +# The fork is confirmed live — the superseded turn's session can go now rather +# than waiting for terminal state. +if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi + # Wait for the forked turn to finish, PRESERVING the watcher's exit status so we # can tell a terminal turn (rc 0) from a timeout (rc != 0). Parse via parameter # expansion, not word-splitting: the watcher's timeout line can lead with an @@ -83,35 +102,17 @@ fi # state token for the uuid. The cwd is fixed at spawn and never changes. poll_rc=0 poll_out="$(_poll_until_done "$newshort" "$((DAEMON_TIMEOUT / 2))")" || poll_rc=$? -newuuid="${poll_out%% *}"; state="${poll_out#* }"; state="${state%% *}" -case "$newuuid" in - *[!0-9a-f-]*) newuuid="" ;; # defensive: a jumbled poll line is not a session uuid +polled_uuid="${poll_out%% *}"; state="${poll_out#* }"; state="${state%% *}" +case "$polled_uuid" in + *[!0-9a-f-]*) polled_uuid="" ;; # defensive: a jumbled poll line is not a session uuid esac +[ -n "$polled_uuid" ] && newuuid="$polled_uuid" -# A terminal poll can still carry an EMPTY sessionId (`claude agents` row with -# no uuid) — leading-space parsing maps that to newuuid="". Never finalize meta -# from such a row: route it through the same recovery path as a no-uuid timeout. -if [ "$poll_rc" -ne 0 ] || [ -z "$newuuid" ]; then - # Watcher expired before the turn reached a terminal state — or the row it - # returned had no usable uuid. - if [ -n "$newuuid" ]; then - # The fork DID launch and is still running — record that truth: advance the - # chain to the new turn and mark status=working. Do NOT write the reply file - # or bump turns; no final reply has landed. daemon-reply.sh reads the CURRENT - # session's transcript, so it will surface the reply once the turn finishes. - _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" status "working" updated "$(_now)" - # The fork is confirmed live — the superseded turn's session can go. - if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi - echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 - echo " run daemon-reply.sh $uuid once it lands to read the reply." >&2 - else - # NO usable uuid (agent never appeared, or its row had an empty sessionId). - # Keep `short`/`current` on the previous (consistent) session so daemon-reply - # never reads a half-existent turn; stash the parsed short as `pending_short` - # so the new turn stays recoverable by hand. - _meta_set "$uuid" status "error" pending_short "$newshort" updated "$(_now)" - echo "resume: forked agent $newshort produced no usable session uuid; kept previous current (recover via meta pending_short)." >&2 - fi +# The UUID was already confirmed and registered above. A watcher timeout leaves +# that truthful working state in place; no reply or turn-count update has landed. +if [ "$poll_rc" -ne 0 ]; then + echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 + echo " run daemon-reply.sh $uuid once it lands to read the reply." >&2 exit 1 fi @@ -123,12 +124,6 @@ _record_reply "$newuuid" "$uuid" "$state" _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" updated "$(_now)" turns "$((turns + 1))" -# Purge the superseded turn: deregister it (jobs entry + supervisor record) and -# drop its transcript so `claude agents` shows exactly one session per daemon — -# the current turn. The fork carried the full conversation forward, so nothing -# is lost. -if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi - echo "daemon resumed: $name [$newshort / $uuid] status=$(_meta_get "$uuid" status) turns=$(_meta_get "$uuid" turns) current=$newuuid" echo "--- reply ---" _reply_text "$uuid" diff --git a/tests/orchestrating-daemons/test-daemon-scripts.sh b/tests/orchestrating-daemons/test-daemon-scripts.sh index f9eaceb03c..b5a67b92b7 100755 --- a/tests/orchestrating-daemons/test-daemon-scripts.sh +++ b/tests/orchestrating-daemons/test-daemon-scripts.sh @@ -50,6 +50,7 @@ export HOME="$TEST_ROOT/home" export DAEMON_HOME="$TEST_ROOT/registry" export STUB_STATE="$TEST_ROOT/stub" export DAEMON_TIMEOUT=10 +export DAEMON_UUID_POLL=5 export DAEMON_BOOT_ID="boot-current" WORK="$TEST_ROOT/work" mkdir -p "$HOME" "$WORK" "$STUB_STATE/agents" @@ -363,6 +364,34 @@ echo "failure windows:" spawn_short() { printf '%s' "$1" | sed -n 's/.*\[\([0-9a-f]*\) \/ .*/\1/p' | head -1; } spawn_uuid() { printf '%s' "$1" | sed -n 's/.*\[[0-9a-f]* \/ \([0-9a-f-]*\)\].*/\1/p' | head -1; } +# A resumed turn must become locally owned as soon as its UUID appears, before +# the long terminal-state watcher returns. Otherwise migrated metadata remains +# foreign while a real local turn is running and dispatch can duplicate it. +M_OUT="$("$SCRIPTS_DIR/daemon-spawn.sh" "migrated-resume" "seed-m" "$WORK")" +M_SHORT="$(spawn_short "$M_OUT")"; M_UUID="$(spawn_uuid "$M_OUT")" +python3 - "$DAEMON_HOME/$M_UUID.json" <<'PY' +import json, sys +p = sys.argv[1] +m = json.load(open(p)) +m["host"] = "old-host" +m["boot_id"] = "boot-old" +json.dump(m, open(p, "w"), indent=2) +PY +STUB_BG_STATE=running "$SCRIPTS_DIR/daemon-resume.sh" "$M_SHORT" "continue locally" > "$TEST_ROOT/migrated-resume.out" 2>&1 & +M_RESUME_PID=$! +for _ in $(seq 1 20); do + M_CUR="$(sed -n 's/.*"current": "\([^"]*\)".*/\1/p' "$DAEMON_HOME/$M_UUID.json")" + [ -n "$M_CUR" ] && [ "$M_CUR" != "$M_UUID" ] && break + sleep 0.25 +done +M_META="$(cat "$DAEMON_HOME/$M_UUID.json")" +assert_contains "$M_META" '"host": "'"$(hostname)"'"' "running resume is re-stamped to the local host before terminal polling" +assert_contains "$M_META" '"boot_id": "boot-current"' "running resume is re-stamped to the local boot before terminal polling" +[ -n "${M_CUR:-}" ] && [ "$M_CUR" != "$M_UUID" ] \ + && pass "running resume advances current before terminal polling" \ + || fail "running resume advances current before terminal polling" +wait "$M_RESUME_PID" 2>/dev/null || true + # (a) The fork command itself fails (session not resumable). Resume must exit # nonzero, flip status=error, and leave `current`/turns untouched — the daemon # must not be silently advanced past a launch that never happened. From 402df1b229d026bf69f2929360691e70686151d6 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 23:31:17 +0900 Subject: [PATCH 11/12] fix(orchestrating-daemons): gate superseded-turn stop/purge on recorded host identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A migrated meta's short is host-local and reusable — stop/rm through it can hit an unrelated local agent. Capture the identity verdict before the fork re-stamps the meta; foreign metas skip both (leftover old-host session files are cosmetic, a purge through a reused short is destructive). --- .../scripts/daemon-resume.sh | 17 ++++++++++++++--- .../test-daemon-scripts.sh | 9 +++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index 15acfd7774..a7eb613390 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -43,10 +43,18 @@ wtpath=""; [ -n "$(_meta_get "$uuid" worktree)" ] && wtpath="$cwd" # before the fork rework have no `current`, so fall back to the daemon's uuid). cur="$(_meta_get "$uuid" current)"; [ -n "$cur" ] || cur="$uuid" curshort="$(_meta_get "$uuid" short)" +# The recorded short is HOST-LOCAL (8-hex, reusable): stop/rm through it only +# when the meta's identity belongs to this boot — on a migrated meta the same +# short may name an unrelated local agent. Capture the verdict BEFORE the fork +# re-stamps the meta to this host. Foreign → skip stop and purge entirely (a +# leftover transcript/jobs entry from the old host is cosmetic; a purge through +# a reused short is destructive). +cur_is_local=0 +_identity_local "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)" && cur_is_local=1 # Release the current bg turn (idempotent — harmless if already stopped). This # also drops it from the active `claude agents` view before the next turn forks. -[ -n "$curshort" ] && claude stop "$curshort" >/dev/null 2>&1 || true +[ -n "$curshort" ] && [ "$cur_is_local" -eq 1 ] && claude stop "$curshort" >/dev/null 2>&1 || true _meta_set "$uuid" status "working" updated "$(_now)" @@ -92,8 +100,11 @@ esac _meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "working" updated "$(_now)" # The fork is confirmed live — the superseded turn's session can go now rather -# than waiting for terminal state. -if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi +# than waiting for terminal state. Never purge through a foreign short: the +# meta was re-stamped local above, but curshort still names the OLD host's turn. +if [ "$cur" != "$newuuid" ] && [ "$cur_is_local" -eq 1 ]; then + _session_purge "$curshort" "$cur" "$wtpath" +fi # Wait for the forked turn to finish, PRESERVING the watcher's exit status so we # can tell a terminal turn (rc 0) from a timeout (rc != 0). Parse via parameter diff --git a/tests/orchestrating-daemons/test-daemon-scripts.sh b/tests/orchestrating-daemons/test-daemon-scripts.sh index b5a67b92b7..4850889611 100755 --- a/tests/orchestrating-daemons/test-daemon-scripts.sh +++ b/tests/orchestrating-daemons/test-daemon-scripts.sh @@ -377,6 +377,10 @@ m["host"] = "old-host" m["boot_id"] = "boot-old" json.dump(m, open(p, "w"), indent=2) PY +# The migrated short may be REUSED by an unrelated local agent — resume must +# never stop/rm through it. Seed a jobs dir standing in for that local agent. +mkdir -p "$HOME/.claude/jobs/$M_SHORT" +: > "$STUB_STATE/log/calls.log" STUB_BG_STATE=running "$SCRIPTS_DIR/daemon-resume.sh" "$M_SHORT" "continue locally" > "$TEST_ROOT/migrated-resume.out" 2>&1 & M_RESUME_PID=$! for _ in $(seq 1 20); do @@ -391,6 +395,11 @@ assert_contains "$M_META" '"boot_id": "boot-current"' "running resume is re-stam && pass "running resume advances current before terminal polling" \ || fail "running resume advances current before terminal polling" wait "$M_RESUME_PID" 2>/dev/null || true +M_CALLS="$(cat "$STUB_STATE/log/calls.log")" +assert_not_contains "$M_CALLS" "stop $M_SHORT" "migrated resume never stops through the foreign short" +assert_not_contains "$M_CALLS" "rm $M_SHORT" "migrated resume never rms through the foreign short" +[ -d "$HOME/.claude/jobs/$M_SHORT" ] && pass "migrated resume leaves the reused short's jobs dir intact" \ + || fail "migrated resume leaves the reused short's jobs dir intact" # (a) The fork command itself fails (session not resumable). Resume must exit # nonzero, flip status=error, and leave `current`/turns untouched — the daemon From b7ead32de129e6619bc550608656b5b20b8e8c17 Mon Sep 17 00:00:00 2001 From: SSFSKIM Date: Sun, 12 Jul 2026 23:45:12 +0900 Subject: [PATCH 12/12] fix(orchestrating-daemons): parse the LEADING sec field of kern.boottime MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The greedy .*sec-= match landed inside 'usec = ' and recorded microseconds as the boot identity — a value that can repeat across reboots, defeating the host-boot liveness gate on macOS. Anchor on the leading '{ sec = N,'. Regression cross-checks _boot_id against an independent parse. env.triage.example: header marks the file fork-local (names the fork's real consumer repo on purpose; never upstream) — review finding triaged as void for a personal fork, kernel kept as a permanent guard note. --- infra/worker-host/env.triage.example | 4 ++++ skills/orchestrating-daemons/scripts/_lib.sh | 4 +++- tests/orchestrating-daemons/test-daemon-scripts.sh | 14 ++++++++++++++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/infra/worker-host/env.triage.example b/infra/worker-host/env.triage.example index 492cacc712..93d44bdf07 100644 --- a/infra/worker-host/env.triage.example +++ b/infra/worker-host/env.triage.example @@ -1,3 +1,7 @@ +# FORK-LOCAL deployment recipe — names this fork's real consumer repo on +# purpose (the concrete values ARE the record). Upstream's no-project-config +# rule binds upstream PRs, not this fork; NEVER upstream this file. +# # Triage tenant env (layer 3: seeded by hand) — the skill-dir .env that # feedback-poll.sh sources. Copy to # /data/triage/doperpowers/skills/triaging-feedback/.env (chmod 600) diff --git a/skills/orchestrating-daemons/scripts/_lib.sh b/skills/orchestrating-daemons/scripts/_lib.sh index f68c9d8818..8ce39e8701 100755 --- a/skills/orchestrating-daemons/scripts/_lib.sh +++ b/skills/orchestrating-daemons/scripts/_lib.sh @@ -31,7 +31,9 @@ _boot_id() { if [ -r /proc/sys/kernel/random/boot_id ]; then cat /proc/sys/kernel/random/boot_id elif command -v sysctl >/dev/null 2>&1; then - sysctl -n kern.boottime 2>/dev/null | sed -E 's/.*sec = ([0-9]+).*/\1/' || true + # Anchor on the LEADING `{ sec = N,` field: a greedy `.*sec = ` lands on + # the `sec` inside `usec = ` and records the microseconds instead. + sysctl -n kern.boottime 2>/dev/null | sed -E 's/^\{ sec = ([0-9]+),.*/\1/' || true fi } DAEMON_BOOT_ID="${DAEMON_BOOT_ID:-$(_boot_id)}" diff --git a/tests/orchestrating-daemons/test-daemon-scripts.sh b/tests/orchestrating-daemons/test-daemon-scripts.sh index 4850889611..6827253f6e 100755 --- a/tests/orchestrating-daemons/test-daemon-scripts.sh +++ b/tests/orchestrating-daemons/test-daemon-scripts.sh @@ -347,6 +347,20 @@ printf '{"uuid":"%s","current":"%s","short":"face0000","name":"foreign","engine" assert_not_contains "$(cat "$STUB_STATE/log/calls.log")" "stop face0000" "retire does not stop a foreign-host Claude session" assert_contains "$(cat "$DAEMON_HOME/$FOREIGN_UUID.json")" '"status": "retired"' "foreign-host Claude record is still retired" +# _boot_id must record the BOOT identity — on macOS the sec field, where a +# greedy `.*sec = ` match lands inside `usec = ` and records microseconds. +# Cross-check against an independent parse (Linux: the boot_id file verbatim; +# macOS: the first integer in kern.boottime is the sec field). +if [ -r /proc/sys/kernel/random/boot_id ]; then + BOOT_EXPECT="$(cat /proc/sys/kernel/random/boot_id)" +else + BOOT_EXPECT="$(sysctl -n kern.boottime | grep -oE '[0-9]+' | head -1)" +fi +BOOT_GOT="$(bash -c "source '$SCRIPTS_DIR/_lib.sh' >/dev/null 2>&1; _boot_id")" +[ -n "$BOOT_GOT" ] && [ "$BOOT_GOT" = "$BOOT_EXPECT" ] \ + && pass "_boot_id records the boot identity, not a substring field" \ + || fail "_boot_id records the boot identity, not a substring field (got: $BOOT_GOT, want: $BOOT_EXPECT)" + # ---- 6) worktree isolation (native --worktree threading) --------------------- echo "worktree isolation:" WT_REPO="$TEST_ROOT/wtrepo"; mkdir -p "$WT_REPO"