diff --git a/docs/doperpowers/2026-07-12-managed-agents-steals.md b/docs/doperpowers/2026-07-12-managed-agents-steals.md new file mode 100644 index 0000000000..8dd3ccb449 --- /dev/null +++ b/docs/doperpowers/2026-07-12-managed-agents-steals.md @@ -0,0 +1,157 @@ +# Managed-Agents steals — what we take, what we skip, and why + +**Source.** Anthropic engineering blog, *"Scaling Managed Agents: Decoupling +the brain from the hands"* (https://www.anthropic.com/engineering/managed-agents), +read 2026-07-12 during the production-infrastructure discussion (cloud host +for the board pipeline: implement / review / land workers, feedback intake). +This doc records the transfer analysis and what was actually adopted, the +same way `2026-07-11-symphony-comparison.md` did for OpenAI Symphony. + +**The article in one line.** Virtualize the agent into three interfaces — +**session** (append-only durable log), **harness** (the loop; stateless, +rebootable via `wake(sessionId)`), **sandbox** (execution environment; +provisioned on demand, disposable) — so each can fail or be replaced without +the others. Pets-vs-cattle applied to agents: the pet is not the machine, +it is *state fused to compute*. + +## 0. Mapping onto our architecture + +| Managed Agents | doperpowers equivalent | already decoupled? | +|---|---|---| +| session (durable event log) | claude JSONL transcripts (`~/.claude/projects`), codex rollouts (`~/.codex/sessions`) — plus, for the human-relevant slice, GitHub issue comments (FD-9 answer relay) | **yes** — resume = re-materialize the brain from the log | +| harness (stateless loop, `wake(sessionId)`) | `claude --bg --resume` / `codex exec resume` under the daemon registry | **yes** — every resume is already a fresh process over the log | +| sandbox (disposable hands) | git worktrees under `LOCAL_REPO/.claude/worktrees/` | **yes** — reconstructible from git at any time | +| the coupling they broke | our registry binds ticket → **(pid on one host, session file on one host)** | **no — this was our pet**, fixed by steal 1 | + +The conclusion that shaped the infra discussion: what must be persistent is +the **state volume** (registry + session stores + clones), never the compute. +A worker host — VM, Fly machine, container — is a disposable body around +that volume. Recovery semantics were already right before this doc: the +board is the source of truth, sessions are a resume *optimization*, and the +worst case after total state loss is a fresh re-dispatch with the durable +issue-comment record. The steals below close the gaps that remained. + +## 1. Steal 1 — host-stamped registry, host-aware liveness (IMPLEMENTED) + +**Their form.** The container became cattle the moment the harness stopped +assuming it shared a box with the sandbox; failure detection became "the +tool call errored," not "nurse the box." + +**Our defect.** Registry metas recorded a bare `pid`, and four consumers +tested liveness with `kill -0`: + +- `codex-resume.sh` — one-turn-per-daemon guard +- `daemon-retire.sh` — kills the live turn before retiring +- `review-dispatch.sh` `_is_live` + `_wt_occupied` — dedupe and + worktree-removal guards +- `land-dispatch.sh` `_is_live` — same + +A pid is only meaningful on the machine that recorded it. Move the registry +to a new host on a state volume (host died, host rebuilt, migrated to +cloud) and a **reused pid number** reads as a live worker: resume refuses +("a turn is still running"), retire signals an unrelated process, dispatch +skips a dead reviewer as active, worktree removal is blocked by a ghost. +Every failure is silent and directional — the pipeline stalls open. + +**The fix (shipped with this doc).** Every registration and resume stamps +`host` (`DAEMON_HOST`, default `hostname`) into the meta; `_pid_alive()` +in `_lib.sh` treats a host mismatch as dead regardless of `kill -0` +(empty `host` = legacy meta = local, preserving old behavior). All four +consumers now route through host-aware checks. Claude-engine metas get the +stamp too: their `short`/`current` ids are just as host-local (`claude +agents` is a per-host view). Pinned by tests in +`tests/orchestrating-daemons/test-codex-scripts.sh` (foreign-host pid: +resume proceeds + re-stamps, retire leaves the process alone) and +`tests/reviewing-prs/test-{review,land}-dispatch.sh` (foreign-host pid: +treated dead → retire+respawn; same-host control still skips as active). + +This is the registry's cattle key without the migration project: metas are +now effectively keyed (host, pid, session), so a future multi-host or +container fleet only adds a dispatcher — no semantic change. + +## 2. Steal 2 — the state-volume convention (CONVENTION, this doc is the spec) + +**Their form.** Session outlives harness; harness outlives container; +nothing in the compute needs to survive. + +**Ours.** When provisioning a dedicated worker host, ALL mutable pipeline +state lives under paths that sit on one detachable volume, and the compute +is fully reproducible from a setup script (cloud-init or equivalent): + +| state | path | owner | +|---|---|---| +| daemon registry + codex run scratch | `DAEMON_HOME` (default `~/.claude/orchestrating-daemons`) | ours, env-overridable everywhere (verified: all scripts honor it) | +| claude sessions + jobs | `~/.claude/projects`, `~/.claude/jobs` | harness-owned — placed on the volume by making `$HOME` (or the whole user dir) volume-backed | +| codex sessions + auth | `~/.codex` | harness-owned, same treatment | +| canonical clones + worktrees | `LOCAL_REPO` per consumer repo | ours | + +Host dies → new host + attach volume → every parked session resumes; the +host-aware liveness from steal 1 is exactly what makes the stale pids in +the migrated registry harmless. Nothing else in the pipeline needs backup +discipline: the board (GitHub) already carries ticket state and the +human-answer record. + +Runnable form: `infra/worker-host/` (cloud-init for the body, env.example +for the seeding contract, README with the layer split, verification gate, +and rebuild drill). + +Non-goal: we did NOT adopt per-worker ephemeral compute (their full cattle +shape). At single-tenant scale with a few concurrent workers, dispatch +latency is noise against multi-hour turns, and the refactor would reopen +freshly shaken-down rituals for a TTFT win we cannot feel. + +## 3. Steal 3 — token-wired remote (RECIPE, apply at host provisioning) + +**Their form.** Credentials are never reachable from where generated code +runs: the repo token is wired into the git remote during sandbox init +(push/pull work; the agent never handles the token), everything else sits +behind a vault + proxy. + +**Ours, honestly scoped.** Full unreachability is impossible in this +design — a worker must hold a board-write token (`gh` issue transitions, +PR creation) *by contract*. The stealable part is **scope separation**: + +- `LOCAL_REPO` on the worker host is cloned with a fine-grained token + (contents: read/write on that repo ONLY) embedded in the remote URL. + Worktrees share the parent repo's remote config, so worker `git push` + just works — the push credential never enters worker env. +- Worker env `GH_TOKEN` gets a second fine-grained token scoped to + issues + pull-requests (board writes, PR open/comment) with NO contents + write. `_codex_launch` already prefers a pre-set `GH_TOKEN` over the + keychain capture, so injection needs zero code. +- Net effect: a prompt-injected worker can vandalize the board (visible, + reversible, audited) but cannot push code with authority the reviewer + didn't grant, and neither token can spawn broader sessions. + +Not applied on the current Mac host — the keychain token is the user's +full-power credential either way, so the split is theater there. The +TECH-DEBT accepted note ("GH_TOKEN in worker env") now points here as its +narrowing path. + +## 4. Non-steals + +- **Harness out of the container / on-demand sandbox provisioning.** Their + TTFT win (p50 −60%) comes from thousands of multi-tenant sessions paying + container boot before first token. Our workers run minutes-to-hours; + boot cost is invisible. Skip until worker concurrency outgrows one host. +- **MCP vault + proxy for tool credentials.** Right shape at multi-tenant + scale; for us it adds a resident service (the thing our no-orchestrator + architecture deliberately avoids) to protect one token we can scope + instead. +- **`getEvents()` context-interrogation layer.** Our equivalent pressure + is already answered compositionally: FD-7 closing artifacts (PR body as + the durable orientation record) + transcript-on-disk + fork-carries-context + resume. A positional event-slicing API over transcripts is capability we + have no consumer for. +- **Session-as-REPL-object context management.** Same reason — no consumer; + the harnesses own their context engineering. + +## 5. Relationship to the Symphony comparison + +Symphony gave us the *work semantics* debates (FD-1..FD-9); Managed Agents +gives the *substrate* doctrine. They agree on the load-bearing point: the +durable log is the identity of the work, the loop is disposable. Symphony +puts that log in an orchestrator's memory + Linear; Managed Agents puts it +in a session store; we put it in GitHub (board) + session files (resume +cache). The steals above keep that stack true when the pipeline leaves the +Mac it was born on. diff --git a/docs/doperpowers/TECH-DEBT.md b/docs/doperpowers/TECH-DEBT.md index 7fe0d97a64..ec6cf4e3b5 100644 --- a/docs/doperpowers/TECH-DEBT.md +++ b/docs/doperpowers/TECH-DEBT.md @@ -109,16 +109,22 @@ post-clause: disable collab tools via spawn-time `-c` config. ### 8. `SSL_CERT_FILE` is macOS-specific -`_codex_launch` exports `/etc/ssl/cert.pem` when present (guarded — silently -skipped elsewhere). A Linux worker host needs its distro bundle path wired -in; the review engine block documents the requirement. The in-thread review -design (FU-7) removed the main consumer, so exposure is small. +**RESOLVED 2026-07-12**: `_codex_launch` and `review-engine.sh` now probe +`/etc/ssl/cert.pem` (macOS) then `/etc/ssl/certs/ca-certificates.crt` +(Debian/Ubuntu) and export the first hit; `infra/worker-host/env.example` +also pins the Linux path explicitly as belt-and-suspenders. Remaining +exposure: other distros' bundle paths — extend the probe list if a non-Debian +host ever appears. ### 9. Accepted notes (recorded, no action intended) - **GH_TOKEN in worker env** — visible to the worker's subprocesses; parity with what claude workers reach via the keychain, recorded in FU-3's - security note. + security note. A narrowing recipe now exists (token-wired remote: push + scope in the clone's remote URL, board scope in env — see + `2026-07-12-managed-agents-steals.md` §steal-3); apply it when + provisioning a dedicated worker host, where scoped tokens are real. On + this Mac the keychain token is full-power anyway, so the note stands. - **Mini ssh probe noise** — codex's `keepRemoteControlAwakeWhilePluggedIn` probes the unreachable `mini` host at spawn; user config, harmless. - **FU-2 known limitation** — a daemon resumed once and never diff --git a/infra/worker-host/README.md b/infra/worker-host/README.md new file mode 100644 index 0000000000..e54c693b00 --- /dev/null +++ b/infra/worker-host/README.md @@ -0,0 +1,204 @@ +# Worker host provisioning — body / soul / seeding + +Runnable counterpart of `docs/doperpowers/2026-07-12-managed-agents-steals.md` +§2 (state-volume convention). One persistent Linux VM runs the whole board +pipeline: implement / review / land workers (detached processes under the +daemon registry), the Actions runner that turns board events into dispatches, +and the feedback-triage poller (a separate unix user — see §5), and nothing +else. No Docker: workers are host processes keyed by +(host, pid, session) in the registry, codex brings its own Landlock sandbox +on a real kernel, and single-tenant reproducibility comes from cloud-init, +not images. + +Everything splits by lifetime: + +| layer | what | lives | +|---|---|---| +| 1 — body | OS, packages, CLIs, runner binary | disposable; `cloud-init.yaml` recreates it | +| 2 — soul | registry, sessions, auth files, clones, worktrees | `/data` volume = the worker user's `$HOME` | +| 3 — seeding | tokens, codex auth.json, runner registration, clones | by hand, ONCE per soul (never per body) | + +## 1. Create the body + +Hetzner (CX43, Ubuntu 24.04, one Volume) — adjust names to taste: + +```bash +hcloud volume create --name doper-data --size 40 --location fsn1 +hcloud server create --name doper-worker-1 --type cx43 --image ubuntu-24.04 \ + --location fsn1 --volume doper-data --user-data-from-file cloud-init.yaml \ + --ssh-key +``` + +First boot formats the volume ONLY if blank, mounts it at `/data`, creates +`worker` (uid 1010, home `/data/worker`, no sudo) and `triage` (uid 1011, +home `/data/triage`, no sudo — see §5 for why they are separate), installs +git/jq/python3/build-essential, Node LTS, `gh`, `codex` (global npm), +`claude` (native installer, as worker), Tailscale, an 8G swapfile, and the +`doper-triage` systemd timer (armed but inert until §5's seeding). It also +sets the hostname to `doper-worker-` — unique per body, +which is what makes the registry's host-aware pid liveness hold across +rebuilds (the Hetzner server *name* can stay `doper-worker-1`; only the OS +hostname matters to the registry). + +After boot: + +```bash +cloud-init status --long # must be done/no errors — the runcmd hard + # gate aborts here if /data didn't mount +tailscale up # then close public SSH: +ufw allow in on tailscale0; ufw allow ssh; ufw enable # drop 'allow ssh' once tailscale login works +``` + +## 2. Seed the soul (once) + +As `worker` (`sudo -iu worker`): + +1. **Env file** — copy `env.example` → `~/.env`, fill it (see the file for + the two-token scope split), `chmod 600 ~/.env`. +2. **codex auth** — `worker` is deliberately not SSH-reachable (locked + password, no authorized keys), so copy through your login account (root + on Hetzner). GNU `install` refuses a pipe-backed `/dev/stdin` ("replaced + while being copied"), so write under `umask 077` — the file is born 0600 + with no exposure window — then hand it to `worker`: + ```bash + ssh root@host 'install -d -m 700 -o worker -g worker /data/worker/.codex' + ssh root@host 'umask 077 && cat > /data/worker/.codex/auth.json \ + && chown worker:worker /data/worker/.codex/auth.json' < ~/.codex/auth.json + ``` + (auth.json is portable by design; device-code flow is the fallback if + the workspace enables it.) +3. **claude auth** — nothing to do beyond `CLAUDE_CODE_OAUTH_TOKEN` in + `~/.env` (generated locally with `claude setup-token`). +4. **Canonical clones, push scope wired into the remote** (steal 3) — one + per consumer repo, using a contents-only fine-grained PAT that appears + NOWHERE else: + ```bash + git clone https://oauth2:@github.com/OWNER/REPO.git ~/repos/REPO + ``` + Worktrees share the parent's remote config, so worker `git push` works + without the token ever entering worker env. Be honest about what that + buys: any process running as `worker` — including PR-derived code — can + still read the token (`git remote get-url origin`), so workers ARE + contents-write principals on the consumer repos. The containment is the + token's shape, not its hiding place: contents-only scope, consumer repos + only, and **branch protection on main/integration branches (require + PRs)** so the token cannot rewrite protected refs. Enable that protection + before the first worker runs. Verify the env-token split: + `gh api repos/OWNER/REPO --jq .permissions` under `GH_TOKEN` must show + push=false. + Seed the commit identity in the same sitting — a fresh volume has none, + and the implementation protocol's first commit dies with "Author identity + unknown" (git refuses to auto-detect from the machine-id hostname): + ```bash + git config --global user.name "doper-worker" + git config --global user.email "" + ``` + Lives in `~/.gitconfig` on the volume — once per soul, like everything + else in this layer. +5. **doperpowers itself** — `git clone` this repo to `~/doperpowers` + (dispatch scripts are invoked by path from here; workers get skills via + vendoring / plugin install exactly as on the Mac). +6. **Actions runner** (event trigger, PRIVATE repos only — standing + constraint): download the runner into `~/runner`, then + `./config.sh --url https://github.com/OWNER/REPO --token --labels claude-review` + — the label must be `claude-review`: the shipped `pr-review-dispatch.yml` + selects `runs-on: [self-hosted, claude-review]`, and a runner registered + under any other label leaves those jobs queued forever. Runner jobs read + env from `~/runner/.env` and PATH from `~/runner/.path`, NOT from + `.bashrc` (non-login shells; Ubuntu's stock `.bashrc` early-returns for + non-interactive). From a worker login shell where + `command -v gh git node python3 claude` all resolve: + ```bash + grep -Ev '^\s*(#|$)' ~/.env > ~/runner/.env # runner .env is bare KEY=VALUE + echo "DOPERPOWERS_HOME=/data/worker/doperpowers" >> ~/runner/.env + chmod 600 ~/runner/.env # holds the same tokens as ~/.env + echo "$PATH" > ~/runner/.path + ``` + Then as root `cd /data/worker/runner && ./svc.sh install worker && + ./svc.sh start`. Runner jobs must only run the dispatch ritual (render → + spawn --no-wait → bind) and exit; workers are detached processes outside + the job. What actually makes that true is not nohup/--bg but the spawn + paths stripping `RUNNER_TRACKING_ID` (daemon-spawn / daemon-resume / + `_codex_launch`) — the runner's post-job cleanup kills any surviving + process whose environ still carries that job marker, detached or not. + +## 3. Verification gate (before trusting it) + +The rituals were shakedown-tested on macOS; this is the Linux pass. Run one +dogfood cell end-to-end on the VM and check, in order: + +1. `codex exec` smoke turn — auth.json accepted, Landlock sandbox active + (kernel ≥5.13; Ubuntu 24.04 is 6.8). +2. Nested-codex TLS — `SSL_CERT_FILE` resolves (`_codex_launch` and + `review-engine.sh` probe `/etc/ssl/certs/ca-certificates.crt` since + 2026-07-12); a review-engine call from inside a worker completes. +3. Board write under env `GH_TOKEN` — a `board-transition.sh` against a test + issue; push from a worktree uses the remote-wired credential. +4. Full cell: dispatch an implement worker on a toy ticket → PR → review + worker → verdict. The claude-engine branch counts double here + (TECH-DEBT #3: untested by the original shakedown). + +## 4. Rebuild drill (why this layout exists) + +Body loss is a non-event: create a new server with the same +`cloud-init.yaml`, attach the same volume, re-run layer-1's `tailscale up`, +and reinstall the runner service — its registration and config survive in +`/data/worker/runner`, but the systemd unit `svc.sh install` wrote lived on +the discarded root filesystem, so there is nothing to merely "restart": +`cd /data/worker/runner && sudo ./svc.sh install worker && sudo ./svc.sh start`. +Every parked session resumes — the registry's stale pids are neutralized by +host-aware liveness (metas carry `host`; the rebuilt body's machine-id +suffix gives it a NEW hostname, so every old meta reads foreign and its pid +as dead). Nothing else in layer 3 repeats, because nothing else in layer 3 +lived on the body. The triage tenant needs no step at all: the +timer is re-armed by cloud-init and its `ExecCondition` finds the seeded +`.env` already on the volume. + +## 5. Triage tenant (`skills/triaging-feedback` poller) + +The first production tenant, and deliberately the burn-in one: real value +from day one (24/7 feedback triage instead of a sleep-prone Mac's launchd), +minimal blast radius (ticket-only — the poller diagnoses and files tickets; +it never writes code or pushes). It has **no soul on this host**: all durable +state lives in Supabase (`feedback.triage_state`/`triage_lease`) and GitHub +(tickets), so unlike the workers it doesn't even need the state volume — it +sits on it only because `$HOME` does. + +**Why a separate unix user.** The dispatcher env holds +`SUPABASE_SERVICE_ROLE_KEY` (RLS bypass — the strongest secret here), while +`worker` runs implement/review workers that execute PR-derived code. User +separation (uid 1011, mode-600 `.env`) is the cheap wall between them. + +Seed as `triage` (`sudo -iu triage`; step 4 is the exception — it runs from +the admin shell) — prerequisites first: the p86 +migration must be live in Supabase and the descriptive labels created on +ida-solution (`references/setup.md` §0/§4): + +1. **Clones** — `git clone` doperpowers to `~/doperpowers`; clone + ida-solution to `~/repos/ida-solution` with a contents-READ-only + fine-grained PAT in the remote URL (the poller fetches origin every tick, + never pushes). +2. **Deps** — `cd ~/doperpowers/skills/triaging-feedback && npm ci`. +3. **Env** — copy `env.triage.example` → that same skill dir's `.env`, + fill it, `chmod 600`. Keep `TRIAGE_K=1` until ticket quality is trusted + (`references/setup.md` §5). +4. **Verify one tick** — from the ADMIN/root shell, not the triage shell + (`triage` has no sudo, a locked password, and cannot read the system + journal): `systemctl start doper-triage.service`, then + `journalctl -u doper-triage.service -f`. The timer (installed by + cloud-init, 10-minute cadence) takes over from there. +5. **Mac handoff** — once a VM tick has filed a correct ticket, unload the + Mac's launchd label (`references/setup.md` §6). Atomic claim + lease + makes the overlap window safe, but two pollers is a config smell, not a + feature. + +## 6. Not configured here, on purpose + +- **Watchdog** — T1 tech-debt (import of Symphony's liveness sweep); when it + exists it becomes a cron/timer on this host. Until then + `review-dispatch.sh --sweep` on a timer is the available partial. +- **Feedback intake** (in-app 피드백 → Supabase `feedback` table) — that is + the product's own write path; this host only polls the table. No + serverless intake layer is needed for triage. +- **ufw auto-enable in cloud-init** — deliberate: enabling the firewall + before Tailscale is joined can lock you out of a fresh body. diff --git a/infra/worker-host/cloud-init.yaml b/infra/worker-host/cloud-init.yaml new file mode 100644 index 0000000000..00e84588fe --- /dev/null +++ b/infra/worker-host/cloud-init.yaml @@ -0,0 +1,181 @@ +#cloud-config +# Doperpowers worker host — the DISPOSABLE BODY (layer 1 of 3). +# +# Design: docs/doperpowers/2026-07-12-managed-agents-steals.md (steal 2). +# Everything this file installs can be thrown away and rebuilt; everything +# that must survive lives on the attached state volume mounted at /data +# (the worker user's HOME sits on it, so ~/.claude, ~/.codex, the daemon +# registry, and all clones ride the volume automatically). Secrets are NEVER +# in this file — they are seeded once by hand (layer 3, see README.md). +# +# Target: Hetzner Cloud CX43 (8 vCPU / 16 GB, x86), Ubuntu 24.04 LTS, with +# one Hetzner Volume attached at server creation. Works on any KVM VM with +# a by-id-addressable data disk if you adjust the bootcmd glob. + +# hostname is set in runcmd, NOT here — it must be UNIQUE PER BODY. The daemon +# registry stamps metas with $(hostname) (host-aware pid liveness in +# orchestrating-daemons/_lib.sh); a rebuilt body reusing the old name would +# make the previous body's metas read as local, so a recycled pid could alias +# an unrelated process into resume/dedupe/retire. preserve_hostname stops +# cloud-init's per-boot update module from reverting to the metadata name. +preserve_hostname: true +timezone: Etc/UTC + +package_update: true +packages: + - git + - curl + - jq + - python3 + - build-essential # npm native-module fallback builds (no arm64/x64 prebuilt) + - unzip + - ca-certificates # provides /etc/ssl/certs/ca-certificates.crt (SSL_CERT_FILE) + +# ---- state volume (the soul) -------------------------------------------------- +# First boot formats the attached volume ONLY if it is blank (blkid probe), +# so re-provisioning a new body against an existing soul never wipes it. +# cloud-init runs bootcmd + mounts before users-groups, so the worker user's +# home can live on the volume. +bootcmd: + - | + dev="$(readlink -f /dev/disk/by-id/scsi-0HC_Volume_* 2>/dev/null | head -1)" + if [ -b "$dev" ] && ! blkid "$dev" >/dev/null 2>&1; then + mkfs.ext4 -L doperdata "$dev" + fi +mounts: + - [ "LABEL=doperdata", /data, ext4, "defaults,nofail,discard", "0", "2" ] + +swap: + filename: /swap.img + size: 8G + maxsize: 8G + +# ---- users --------------------------------------------------------------------- +# Fixed uids so a REBUILT body maps the volume's existing file ownership. +# No sudo: pipeline users run pipeline processes only; admin happens as root. +# `triage` is a SEPARATE user from `worker` on purpose: its dispatcher env +# holds SUPABASE_SERVICE_ROLE_KEY (RLS bypass — the strongest secret on this +# host), while implement/review workers under `worker` execute PR-derived +# code. Same user would let a compromised worker read that key. +users: + - default + - name: worker + uid: 1010 + shell: /bin/bash + homedir: /data/worker + lock_passwd: true + - name: triage + uid: 1011 + shell: /bin/bash + homedir: /data/triage + lock_passwd: true + +# ---- triage tenant: systemd timer (body-side; arming is guarded) --------------- +# Replaces the Mac's launchd label (skills/triaging-feedback/references/setup.md +# §6). systemd gives the same single-instance serialization the reclaim design +# depends on: a timer never re-triggers a oneshot service that is still running. +# ExecCondition makes pre-seed ticks silent no-ops, so the timer can be enabled +# unconditionally — a rebuilt body against a seeded soul resumes polling with +# zero manual steps. +write_files: + - path: /etc/systemd/system/doper-triage.service + content: | + [Unit] + Description=Doperpowers feedback triage — one poll tick + + [Service] + Type=oneshot + User=triage + Environment=HOME=/data/triage + ExecCondition=/usr/bin/test -f /data/triage/doperpowers/skills/triaging-feedback/.env + ExecStart=/data/triage/doperpowers/skills/triaging-feedback/scripts/feedback-poll.sh + - path: /etc/systemd/system/doper-triage.timer + content: | + [Unit] + Description=Doperpowers feedback triage — 10-minute poll + + [Timer] + OnCalendar=*:00/10 + RandomizedDelaySec=30 + Persistent=false + + [Install] + WantedBy=timers.target + +runcmd: + # runcmd items are merged into ONE #!/bin/sh script, so this set -e governs + # every line below: any installer/apt/npm failure aborts provisioning and + # surfaces in `cloud-init status` instead of yielding a half-built body + # that reports success. (sh is dash — no pipefail — so the curl|bash + # installers below are split into download-then-execute.) + - set -e + # HARD GATE: everything below assumes the state volume is mounted. The + # fstab entry keeps `nofail` on purpose (a missing volume must degrade to + # a reachable system, not systemd emergency mode), which means a missing/ + # unlabeled volume silently leaves /data on the DISPOSABLE root fs — the + # users module has already created the homedirs there by now, and seeding + # into them would vanish on rebuild. Abort loudly instead: no CLIs, no + # timer, and `cloud-init status` reports the error. Recovery is to attach + # the volume and recreate the server (runcmd does not re-run). + - | + mountpoint -q /data || { + echo "FATAL: /data is not a mountpoint — state volume missing or unlabeled; aborting provisioning" >&2 + exit 1 + } + # UID boundary hardening: the users module creates the homes with the + # image's login.defs default, which may be group/other-traversable — that + # would let `triage` read worker's clones (the push PAT lives in each + # .git/config). Explicit 0700, idempotent against an existing soul. + - chmod 700 /data/worker /data/triage + # Per-body hostname (see header note). /etc/machine-id is minted at this + # body's first boot, so every rebuild gets a fresh suffix — old registry + # metas read as foreign-host and their pids as dead, which is what lets + # parked sessions resume safely after a rebuild. + - hostnamectl set-hostname "doper-worker-$(cut -c1-6 /etc/machine-id)" + # Node.js LTS (consumer-repo builds/tests: npm ci, playwright, etc.) + - curl -fsSL https://deb.nodesource.com/setup_22.x -o /tmp/nodesource-setup.sh + - bash /tmp/nodesource-setup.sh + - apt-get install -y nodejs + # gh CLI (board transitions, PR ops) — official apt repo + - mkdir -p -m 755 /etc/apt/keyrings + - curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg + - chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg + - echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list + - apt-get update && apt-get install -y gh + # Tailscale (SSH access path — close public 22 after `tailscale up`, see README) + - curl -fsSL https://tailscale.com/install.sh -o /tmp/tailscale-install.sh + - sh /tmp/tailscale-install.sh + # codex CLI (global; worker-agnostic binary, auth is per-user on the volume) + - npm install -g @openai/codex + # claude CLI — native installer, installed AS the worker (lands in ~/.local/bin + # on the volume; harmless to re-run on a rebuilt body). Inner shell is bash, + # so pipefail is available to keep a failed download from being masked. + - sudo -u worker bash -lc 'set -e -o pipefail; curl -fsSL https://claude.ai/install.sh | bash' + # worker shell wiring: source the seeded env file (layer 3) in INTERACTIVE + # shells (admin ssh, sudo -iu worker). Actions runner jobs do NOT read + # .bashrc — non-login shells, and Ubuntu's stock .bashrc early-returns for + # non-interactive ones — they get env from ~/runner/.env (README §2.6). + - | + sudo -u worker bash -c ' + touch /data/worker/.env && chmod 600 /data/worker/.env + grep -q "source ~/.env" /data/worker/.bashrc 2>/dev/null || \ + printf "\n# doperpowers worker env (seeded by hand — see infra/worker-host)\n[ -f ~/.env ] && set -a && source ~/.env && set +a\n" >> /data/worker/.bashrc + ' + # triage tenant timer — safe to arm before seeding (ExecCondition guards) + - systemctl daemon-reload + - systemctl enable --now doper-triage.timer + # Provisioning gate: set -e catches command failures; this catches + # semantically-wrong installs (missing binary, wrong Node major). + - | + for bin in git jq python3 node npm gh tailscale codex; do + command -v "$bin" >/dev/null || { echo "FATAL: $bin missing after provisioning" >&2; exit 1; } + done + node -e 'process.exit(parseInt(process.versions.node) >= 22 ? 0 : 1)' || { + echo "FATAL: node $(node --version) < 22" >&2; exit 1 + } + sudo -u worker bash -lc 'command -v claude' >/dev/null || { + echo "FATAL: claude missing from worker PATH" >&2; exit 1 + } + +# final_message prints even when runcmd aborted — hence "check status", not "ready". +final_message: "doper-worker body boot finished after $UPTIME s — confirm with 'cloud-init status --long' BEFORE layer 3 seeding (infra/worker-host/README.md)" diff --git a/infra/worker-host/env.example b/infra/worker-host/env.example new file mode 100644 index 0000000000..11e850409e --- /dev/null +++ b/infra/worker-host/env.example @@ -0,0 +1,33 @@ +# Doperpowers worker host — per-user environment (layer 3: seeded by hand). +# Copy to /data/worker/.env, chmod 600. Sourced by ~/.bashrc (wired by +# cloud-init) with `set -a`, so every value is exported. +# +# TWO TOKENS, TWO SCOPES (steal 3 — token-wired remote, see +# docs/doperpowers/2026-07-12-managed-agents-steals.md §3): +# - GH_TOKEN: fine-grained PAT, issues + pull-requests read/write on the +# consumer repos, NO contents write. This is what workers reach. +# - The push credential (contents read/write only) is NOT here — it is +# embedded in each canonical clone's remote URL at clone time. +GH_TOKEN= + +# Claude subscription auth for headless workers: run `claude setup-token` +# on a machine with a browser, paste the printed token here (valid ~1 year). +CLAUDE_CODE_OAUTH_TOKEN= + +# codex auth is FILE-based, not env-based: copy your logged-in +# ~/.codex/auth.json to /data/worker/.codex/auth.json (chmod 600). + +# TLS trust anchors as a file bundle for NESTED codex calls. The scripts +# autodetect this path since 2026-07-12; pinning it here is belt-and-suspenders. +SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt + +# Engine defaults (label > WORKER_ENGINE > codex). +WORKER_ENGINE=codex +CODEX_MODEL=gpt-5.6-sol +CODEX_EFFORT=high + +# DAEMON_HOME / DAEMON_HOST are deliberately NOT set: DAEMON_HOME defaults to +# ~/.claude/orchestrating-daemons (already on the volume via $HOME), and +# DAEMON_HOST must stay the real hostname so host-aware pid liveness works +# (cloud-init makes it per-body unique via a machine-id suffix — pinning it +# here would survive rebuilds on the volume and defeat exactly that). diff --git a/infra/worker-host/env.triage.example b/infra/worker-host/env.triage.example new file mode 100644 index 0000000000..93d44bdf07 --- /dev/null +++ b/infra/worker-host/env.triage.example @@ -0,0 +1,40 @@ +# FORK-LOCAL deployment recipe — names this fork's real consumer repo on +# purpose (the concrete values ARE the record). Upstream's no-project-config +# rule binds upstream PRs, not this fork; NEVER upstream this file. +# +# Triage tenant env (layer 3: seeded by hand) — the skill-dir .env that +# feedback-poll.sh sources. Copy to +# /data/triage/doperpowers/skills/triaging-feedback/.env (chmod 600) +# Authoritative reference for every knob (models, K, timeouts, trust roles): +# skills/triaging-feedback/references/setup.md §2. This file records only the +# required set plus the VM-specific deltas vs the Mac deployment. + +# The strongest secret on this host (RLS bypass). This is WHY triage runs as +# its own unix user: implement/review workers under `worker` execute +# PR-derived code and must never be able to read this file. +SUPABASE_URL= +SUPABASE_SERVICE_ROLE_KEY= + +# Codex SDK auth — API-billed, no auth.json needed for this tenant. +OPENAI_API_KEY= + +# VM delta: no macOS keychain here, so the board scripts' `gh` calls need an +# explicit token. Fine-grained PAT: issues + pull-requests read/write on +# ida-solution, NO contents write (triage is ticket-only — it never pushes). +GH_TOKEN= + +# Consumer repo checkout. Clone it with a SECOND fine-grained PAT — contents +# READ-only — embedded in the remote URL (the poller fetches origin every +# tick; it never pushes): +# git clone https://oauth2:@github.com/IDA-solution/ida-solution.git /data/triage/repos/ida-solution +TRIAGE_REPO_PATH=/data/triage/repos/ida-solution +TRIAGE_BASE_BRANCH=feat/m4.5-polish + +# Board scripts, invoked with cwd = TRIAGE_REPO_PATH so BOARD_REPO is +# inferred from that checkout's remote (setup.md §3). +TRIAGE_BOARD_SCRIPTS_DIR=/data/triage/doperpowers/skills/issue-tracker/scripts + +# Start conservative on a new host: observe ticket quality before raising K +# (setup.md §5), and keep the kill switch handy. +TRIAGE_K=1 +TRIAGE_ENABLED=true diff --git a/skills/orchestrating-daemons/scripts/_codex_lib.sh b/skills/orchestrating-daemons/scripts/_codex_lib.sh index 92fc0aa37f..8a278edd8d 100755 --- a/skills/orchestrating-daemons/scripts/_codex_lib.sh +++ b/skills/orchestrating-daemons/scripts/_codex_lib.sh @@ -7,7 +7,9 @@ # shell around the process finalizes the registry when the turn ends. The meta # is the same JSON contract as claude daemons plus: # engine "codex" -# pid codex process pid of the CURRENT turn (liveness: kill -0) +# pid codex process pid of the CURRENT turn (liveness: _pid_alive — +# kill -0 gated on `host` + `boot_id` matching this boot; migrated +# or prior-boot registry pids are dead here) # effort model_reasoning_effort # event_log JSONL event stream of the current turn (--json stdout) # The codex session id (thread.started) keys the registry, so board-bind.sh / @@ -224,10 +226,13 @@ _codex_launch() { # review engine call) cannot reach the OS keychain/trustd under the # sandbox, so its rustls has no trust anchors and every connection dies # with `invalid peer certificate: UnknownIssuer`. /etc/ssl/cert.pem is the - # sandbox-readable file bundle (verified live: nested exec fails without, - # completes with). The outer codex itself is unsandboxed and unaffected. - if [ -z "${SSL_CERT_FILE:-}" ] && [ -f /etc/ssl/cert.pem ]; then - export SSL_CERT_FILE=/etc/ssl/cert.pem + # sandbox-readable file bundle on macOS (verified live: nested exec fails + # without, completes with); Debian/Ubuntu ships it as ca-certificates.crt. + # The outer codex itself is unsandboxed and unaffected. + if [ -z "${SSL_CERT_FILE:-}" ]; then + for _cert in /etc/ssl/cert.pem /etc/ssl/certs/ca-certificates.crt; do + if [ -f "$_cert" ]; then export SSL_CERT_FILE="$_cert"; break; fi + done fi # A NESTED codex (e.g. review-engine.sh run by a codex worker) resolves # its code-mode command host to /usr/local/bin (absent here) instead of @@ -236,7 +241,11 @@ _codex_launch() { if [ -z "${CODEX_CODE_MODE_HOST_PATH:-}" ] && [ -x "$HOME/.local/bin/codex-code-mode-host" ]; then export CODEX_CODE_MODE_HOST_PATH="$HOME/.local/bin/codex-code-mode-host" fi - nohup bash -c ' + # env -u RUNNER_TRACKING_ID: an Actions runner's post-job cleanup kills any + # process whose environ still carries the job's tracking marker — the + # wrapper and its codex child must outlive a runner dispatch job (see + # daemon-spawn.sh). No-op outside Actions. + nohup env -u RUNNER_TRACKING_ID bash -c ' set -u DIR="$1"; cwd="$2"; taskf="$3"; run="$4"; shift 4 # shellcheck source=/dev/null diff --git a/skills/orchestrating-daemons/scripts/_lib.sh b/skills/orchestrating-daemons/scripts/_lib.sh index 7653cf9bf1..8ce39e8701 100755 --- a/skills/orchestrating-daemons/scripts/_lib.sh +++ b/skills/orchestrating-daemons/scripts/_lib.sh @@ -22,6 +22,22 @@ set -euo pipefail DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" mkdir -p "$DAEMON_HOME" +# Host + boot identity — stamped into metas at every registration. A pid (and a +# `claude agents` short) is only meaningful in the host's current boot. Hostname +# catches volume migration; boot id catches a rebuilt/rebooted machine that kept +# its name but received a fresh pid namespace. Override both values for tests. +DAEMON_HOST="${DAEMON_HOST:-$(hostname)}" +_boot_id() { + if [ -r /proc/sys/kernel/random/boot_id ]; then + cat /proc/sys/kernel/random/boot_id + elif command -v sysctl >/dev/null 2>&1; then + # Anchor on the LEADING `{ sec = N,` field: a greedy `.*sec = ` lands on + # the `sec` inside `usec = ` and records the microseconds instead. + sysctl -n kern.boottime 2>/dev/null | sed -E 's/^\{ sec = ([0-9]+),.*/\1/' || true + fi +} +DAEMON_BOOT_ID="${DAEMON_BOOT_ID:-$(_boot_id)}" + # How long the spawn/resume WATCHER polls `claude agents` for a turn to finish — # a wait bound only, NOT a turn budget. The toolkit never kills a turn: every turn # is an independent `--bg` process that keeps running regardless. Default 5 hours; @@ -37,6 +53,23 @@ _err_path() { printf '%s/%s.err' "$DAEMON_HOME" "$1"; } # Strip ANSI color codes (the `claude --bg` banner is colored even when piped). _strip_ansi() { sed -E 's/\x1b\[[0-9;]*m//g'; } +# rc 0 iff a meta's recorded host/boot identity belongs to this boot. Empty +# values preserve legacy local behavior. +_identity_local() { + local host="${1:-}" boot_id="${2:-}" + [ -z "$host" ] || [ "$host" = "$DAEMON_HOST" ] || return 1 + [ -z "$boot_id" ] || [ -z "$DAEMON_BOOT_ID" ] || [ "$boot_id" = "$DAEMON_BOOT_ID" ] || return 1 +} + +# rc 0 iff pid <1> is live in THIS HOST BOOT. An identity mismatch is DEAD +# regardless of kill -0: only the number survived. +_pid_alive() { + local pid="$1" + [ -n "$pid" ] || return 1 + _identity_local "${2:-}" "${3:-}" || return 1 + kill -0 "$pid" 2>/dev/null +} + # Merge key=value pairs into a daemon's metadata JSON (creates it if absent). # Usage: _meta_set field1 value1 [field2 value2 ...] # diff --git a/skills/orchestrating-daemons/scripts/codex-resume.sh b/skills/orchestrating-daemons/scripts/codex-resume.sh index a7c76c5257..cd6252b807 100755 --- a/skills/orchestrating-daemons/scripts/codex-resume.sh +++ b/skills/orchestrating-daemons/scripts/codex-resume.sh @@ -19,10 +19,12 @@ msg="${2:?missing message}" [ "$(_meta_get "$uuid" engine)" = "codex" ] \ || { echo "codex-resume: $uuid is not a codex daemon — use daemon-resume.sh" >&2; exit 1; } -# One turn per daemon at a time (same invariant as claude daemons). +# One turn per daemon at a time (same invariant as claude daemons). Host-aware: +# a pid recorded on another machine (registry migrated on a state volume) is +# dead by definition — resuming HERE is exactly the recovery act. if [ "$(_meta_get "$uuid" status)" = "working" ]; then pid="$(_meta_get "$uuid" pid)" - if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then + if _pid_alive "$pid" "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then echo "codex-resume: a turn is still running (pid $pid) — wait for it or retire" >&2 exit 1 fi @@ -116,6 +118,7 @@ fi turns="$(_meta_get "$uuid" turns)"; turns=$((${turns:-1} + 1)) _meta_set "$uuid" pid "$(cat "$run.pid" 2>/dev/null || printf '')" \ + host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ event_log "$run.events.jsonl" status "working" turns "$turns" updated "$(_now)" [ -f "$run.rc" ] && _meta_set "$uuid" \ status "$(_codex_final_status "$(cat "$run.rc")" "$run.events.jsonl")" updated "$(_now)" diff --git a/skills/orchestrating-daemons/scripts/codex-spawn.sh b/skills/orchestrating-daemons/scripts/codex-spawn.sh index d4daf8bda6..7d5b80bcfc 100755 --- a/skills/orchestrating-daemons/scripts/codex-spawn.sh +++ b/skills/orchestrating-daemons/scripts/codex-spawn.sh @@ -89,7 +89,7 @@ status="working" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$(printf '%.8s' "$uuid")" name "$name" \ task "$task" cwd "$runcwd" worktree "$worktree" model "$model" effort "$effort" \ - engine "codex" pid "$pid" event_log "$run.events.jsonl" \ + engine "codex" pid "$pid" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" event_log "$run.events.jsonl" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" # The wrapper may have finalized between our check and the write — re-apply. [ -f "$run.rc" ] && _meta_set "$uuid" \ diff --git a/skills/orchestrating-daemons/scripts/daemon-resume.sh b/skills/orchestrating-daemons/scripts/daemon-resume.sh index dc8efe0e0f..a7eb613390 100755 --- a/skills/orchestrating-daemons/scripts/daemon-resume.sh +++ b/skills/orchestrating-daemons/scripts/daemon-resume.sh @@ -43,10 +43,18 @@ wtpath=""; [ -n "$(_meta_get "$uuid" worktree)" ] && wtpath="$cwd" # before the fork rework have no `current`, so fall back to the daemon's uuid). cur="$(_meta_get "$uuid" current)"; [ -n "$cur" ] || cur="$uuid" curshort="$(_meta_get "$uuid" short)" +# The recorded short is HOST-LOCAL (8-hex, reusable): stop/rm through it only +# when the meta's identity belongs to this boot — on a migrated meta the same +# short may name an unrelated local agent. Capture the verdict BEFORE the fork +# re-stamps the meta to this host. Foreign → skip stop and purge entirely (a +# leftover transcript/jobs entry from the old host is cosmetic; a purge through +# a reused short is destructive). +cur_is_local=0 +_identity_local "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)" && cur_is_local=1 # Release the current bg turn (idempotent — harmless if already stopped). This # also drops it from the active `claude agents` view before the next turn forks. -[ -n "$curshort" ] && claude stop "$curshort" >/dev/null 2>&1 || true +[ -n "$curshort" ] && [ "$cur_is_local" -eq 1 ] && claude stop "$curshort" >/dev/null 2>&1 || true _meta_set "$uuid" status "working" updated "$(_now)" @@ -62,8 +70,11 @@ args+=( "$msg" ) # the meta is stranded status=working with `current` still on the old (stopped) # turn. A nonzero exit AND a banner with no parseable short id both land in the # same error path. +# env -u RUNNER_TRACKING_ID: the forked agent must survive a runner dispatch +# job's post-job cleanup, which kills marker-carrying processes (see +# daemon-spawn.sh). newshort="" -if banner="$(cd "$cwd" && claude "${args[@]}" &1 | _strip_ansi)"; then +if banner="$(cd "$cwd" && env -u RUNNER_TRACKING_ID claude "${args[@]}" &1 | _strip_ansi)"; then newshort="$(printf '%s\n' "$banner" | sed -n 's/.*backgrounded · \([0-9a-f][0-9a-f]*\).*/\1/p' | head -1)" fi if [ -z "$newshort" ]; then @@ -73,6 +84,28 @@ if [ -z "$newshort" ]; then exit 1 fi +# Register local ownership as soon as the new UUID materializes, BEFORE the +# terminal watcher can block for hours. This is the liveness handoff: dispatch +# must see the active fork as local, not keep reading the migrated source host. +newuuid="" +if uuid_out="$(_poll_uuid "$newshort")"; then + newuuid="${uuid_out%% *}" +fi +case "$newuuid" in + ""|*[!0-9a-f-]*) + _meta_set "$uuid" status "error" pending_short "$newshort" updated "$(_now)" + echo "resume: forked agent $newshort produced no usable session uuid; kept previous current (recover via meta pending_short)." >&2 + exit 1 ;; +esac +_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ + status "working" updated "$(_now)" +# The fork is confirmed live — the superseded turn's session can go now rather +# than waiting for terminal state. Never purge through a foreign short: the +# meta was re-stamped local above, but curshort still names the OLD host's turn. +if [ "$cur" != "$newuuid" ] && [ "$cur_is_local" -eq 1 ]; then + _session_purge "$curshort" "$cur" "$wtpath" +fi + # Wait for the forked turn to finish, PRESERVING the watcher's exit status so we # can tell a terminal turn (rc 0) from a timeout (rc != 0). Parse via parameter # expansion, not word-splitting: the watcher's timeout line can lead with an @@ -80,35 +113,17 @@ fi # state token for the uuid. The cwd is fixed at spawn and never changes. poll_rc=0 poll_out="$(_poll_until_done "$newshort" "$((DAEMON_TIMEOUT / 2))")" || poll_rc=$? -newuuid="${poll_out%% *}"; state="${poll_out#* }"; state="${state%% *}" -case "$newuuid" in - *[!0-9a-f-]*) newuuid="" ;; # defensive: a jumbled poll line is not a session uuid +polled_uuid="${poll_out%% *}"; state="${poll_out#* }"; state="${state%% *}" +case "$polled_uuid" in + *[!0-9a-f-]*) polled_uuid="" ;; # defensive: a jumbled poll line is not a session uuid esac +[ -n "$polled_uuid" ] && newuuid="$polled_uuid" -# A terminal poll can still carry an EMPTY sessionId (`claude agents` row with -# no uuid) — leading-space parsing maps that to newuuid="". Never finalize meta -# from such a row: route it through the same recovery path as a no-uuid timeout. -if [ "$poll_rc" -ne 0 ] || [ -z "$newuuid" ]; then - # Watcher expired before the turn reached a terminal state — or the row it - # returned had no usable uuid. - if [ -n "$newuuid" ]; then - # The fork DID launch and is still running — record that truth: advance the - # chain to the new turn and mark status=working. Do NOT write the reply file - # or bump turns; no final reply has landed. daemon-reply.sh reads the CURRENT - # session's transcript, so it will surface the reply once the turn finishes. - _meta_set "$uuid" current "$newuuid" short "$newshort" status "working" updated "$(_now)" - # The fork is confirmed live — the superseded turn's session can go. - if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi - echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 - echo " run daemon-reply.sh $uuid once it lands to read the reply." >&2 - else - # NO usable uuid (agent never appeared, or its row had an empty sessionId). - # Keep `short`/`current` on the previous (consistent) session so daemon-reply - # never reads a half-existent turn; stash the parsed short as `pending_short` - # so the new turn stays recoverable by hand. - _meta_set "$uuid" status "error" pending_short "$newshort" updated "$(_now)" - echo "resume: forked agent $newshort produced no usable session uuid; kept previous current (recover via meta pending_short)." >&2 - fi +# The UUID was already confirmed and registered above. A watcher timeout leaves +# that truthful working state in place; no reply or turn-count update has landed. +if [ "$poll_rc" -ne 0 ]; then + echo "resume: watcher expired after $((DAEMON_TIMEOUT / 2)) polls; forked turn $newshort ($newuuid) is still running (status=working)." >&2 + echo " run daemon-reply.sh $uuid once it lands to read the reply." >&2 exit 1 fi @@ -117,15 +132,9 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error # Reply file stays keyed by the ORIGINAL uuid; read the reply from the new turn. _record_reply "$newuuid" "$uuid" "$state" -_meta_set "$uuid" current "$newuuid" short "$newshort" \ +_meta_set "$uuid" current "$newuuid" short "$newshort" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" updated "$(_now)" turns "$((turns + 1))" -# Purge the superseded turn: deregister it (jobs entry + supervisor record) and -# drop its transcript so `claude agents` shows exactly one session per daemon — -# the current turn. The fork carried the full conversation forward, so nothing -# is lost. -if [ "$cur" != "$newuuid" ]; then _session_purge "$curshort" "$cur" "$wtpath"; fi - echo "daemon resumed: $name [$newshort / $uuid] status=$(_meta_get "$uuid" status) turns=$(_meta_get "$uuid" turns) current=$newuuid" echo "--- reply ---" _reply_text "$uuid" diff --git a/skills/orchestrating-daemons/scripts/daemon-retire.sh b/skills/orchestrating-daemons/scripts/daemon-retire.sh index fd5a7bf81c..ec9683faa6 100755 --- a/skills/orchestrating-daemons/scripts/daemon-retire.sh +++ b/skills/orchestrating-daemons/scripts/daemon-retire.sh @@ -24,7 +24,7 @@ if [ "$engine" = "codex" ]; then status="$(_meta_get "$uuid" status)" pid="$(_meta_get "$uuid" pid)" if { [ "$status" = "working" ] || [ "$status" = "blocked" ]; } && \ - [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then + _pid_alive "$pid" "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then kill "$pid" 2>/dev/null || true # Killing the pid makes _codex_launch's own detached wrapper (the one # that finalizes status+reply once `wait` on the codex pid returns) wake @@ -42,7 +42,9 @@ if [ "$engine" = "codex" ]; then fi resume_hint="codex resume $uuid" else - [ -n "$short" ] && claude stop "$short" >/dev/null 2>&1 || true + if _identity_local "$(_meta_get "$uuid" host)" "$(_meta_get "$uuid" boot_id)"; then + [ -n "$short" ] && claude stop "$short" >/dev/null 2>&1 || true + fi resume_hint="claude --resume $uuid" fi diff --git a/skills/orchestrating-daemons/scripts/daemon-spawn.sh b/skills/orchestrating-daemons/scripts/daemon-spawn.sh index 8c4f149dd0..73a5b97078 100755 --- a/skills/orchestrating-daemons/scripts/daemon-spawn.sh +++ b/skills/orchestrating-daemons/scripts/daemon-spawn.sh @@ -49,7 +49,12 @@ fi [ -n "$model" ] && args+=( --model "$model" ) args+=( "$task" ) -banner="$(cd "$cwd" && claude "${args[@]}" &1 | _strip_ansi)" +# env -u RUNNER_TRACKING_ID: under a GitHub Actions runner the job env carries +# this tracking marker, and the runner's post-job cleanup kills any surviving +# process whose environ still has it — nohup/--bg detach the session, not the +# env. Stripping it at spawn is what actually lets the daemon outlive a +# dispatch job (a no-op everywhere else: env -u of an unset var). +banner="$(cd "$cwd" && env -u RUNNER_TRACKING_ID claude "${args[@]}" &1 | _strip_ansi)" short="$(printf '%s\n' "$banner" | sed -n 's/.*backgrounded · \([0-9a-f][0-9a-f]*\).*/\1/p' | head -1)" [ -n "$short" ] || { echo "spawn failed — could not parse background id from:" >&2; echo "$banner" >&2; exit 1; } @@ -75,7 +80,7 @@ if [ "$nowait" -eq 1 ]; then esac _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" echo "daemon spawned (no-wait): $name [$short / $uuid] status=$status (reply: daemon-reply.sh $short)" exit 0 @@ -104,7 +109,7 @@ status="idle"; [ "$state" = "blocked" ] && status="blocked"; [ "$state" = "error _record_reply "$uuid" "$uuid" "$state" _meta_set "$uuid" \ uuid "$uuid" current "$uuid" short "$short" name "$name" task "$task" cwd "$runcwd" \ - worktree "$worktree" model "$model" \ + worktree "$worktree" model "$model" host "$DAEMON_HOST" boot_id "$DAEMON_BOOT_ID" \ status "$status" created "$(_now)" updated "$(_now)" turns "1" wtnote=""; [ -n "$worktree" ] && wtnote=" worktree=$runcwd (branch worktree-$wt)" diff --git a/skills/reviewing-prs/scripts/land-dispatch.sh b/skills/reviewing-prs/scripts/land-dispatch.sh index a12af50560..b65f070727 100755 --- a/skills/reviewing-prs/scripts/land-dispatch.sh +++ b/skills/reviewing-prs/scripts/land-dispatch.sh @@ -35,6 +35,8 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" +# shellcheck source=../../orchestrating-daemons/scripts/_lib.sh +. "$SKILL_DIR/../orchestrating-daemons/scripts/_lib.sh" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="${BOARD_SCRIPTS:-$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)}" @@ -61,7 +63,8 @@ case "${LAND_ENABLED:-false}" in *) LAND_MODE="dry-run" ;; esac -# Newest land-pr- registry entry → "uuid|status|current|engine|pid" (empty if none). +# Newest land-pr- registry entry → "uuid|status|current|engine|pid|host|boot" +# (empty if none). _land_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' import glob, json, os @@ -80,15 +83,19 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""))) + print("%s|%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""), + m.get("boot_id", ""))) PY } -# rc 0 when the worker's CURRENT turn is live (same semantics as review-dispatch). -_is_live() { # +# rc 0 when the worker's CURRENT turn is live (same semantics as review-dispatch: +# a codex pid counts only on the host that recorded it). +_is_live() { # + _identity_local "${4:-}" "${5:-}" || return 1 if [ "$2" = "codex" ]; then - [ -n "$3" ] && kill -0 "$3" 2>/dev/null + [ -n "$3" ] || return 1 + kill -0 "$3" 2>/dev/null return fi claude agents --json --all 2>/dev/null | CUR="$1" python3 -c ' @@ -159,10 +166,11 @@ fi meta="$(_land_meta "$pr")" if [ -n "$meta" ]; then uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" - current="${rest%%|*}"; rest="${rest#*|}"; w_engine="${rest%%|*}"; w_pid="${rest#*|}" + current="${rest%%|*}"; rest="${rest#*|}"; w_engine="${rest%%|*}"; rest="${rest#*|}" + w_pid="${rest%%|*}"; rest="${rest#*|}"; w_host="${rest%%|*}"; w_boot="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$w_engine" "$w_pid"; then + if _is_live "$current" "$w_engine" "$w_pid" "$w_host" "$w_boot"; then echo "#$pr: skip active land worker"; exit 0 fi _retire "$uuid" ;; diff --git a/skills/reviewing-prs/scripts/review-dispatch.sh b/skills/reviewing-prs/scripts/review-dispatch.sh index 8ed3c56ce4..fde86f4867 100755 --- a/skills/reviewing-prs/scripts/review-dispatch.sh +++ b/skills/reviewing-prs/scripts/review-dispatch.sh @@ -52,6 +52,8 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SKILL_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" DAEMON_SCRIPTS="${DAEMON_SCRIPTS:-$(cd "$SKILL_DIR/../orchestrating-daemons/scripts" && pwd)}" DAEMON_HOME="${DAEMON_HOME:-$HOME/.claude/orchestrating-daemons}" +# shellcheck source=../../orchestrating-daemons/scripts/_lib.sh +. "$SKILL_DIR/../orchestrating-daemons/scripts/_lib.sh" export DAEMON_HOME LOCAL_REPO="${LOCAL_REPO:-$PWD}" BOARD_SCRIPTS="$(cd "$SKILL_DIR/../issue-tracker/scripts" && pwd)" @@ -84,7 +86,8 @@ ENGINE_BLOCK_FILE="$SKILL_DIR/references/engine-blocks/engine-codex-review.md" FALLBACK_FILE="$SKILL_DIR/references/engine-blocks/fallback-engine.md" REVIEW_ENGINE="$SCRIPT_DIR/review-engine.sh" -# Newest review-pr- registry entry → "uuid|status|current|engine|pid" (empty if none). +# Newest review-pr- registry entry → "uuid|status|current|engine|pid|host|boot" +# (empty if none). _reviewer_meta() { DAEMON_HOME="$DAEMON_HOME" PRN="$1" python3 - <<'PY' import glob, json, os @@ -103,16 +106,20 @@ for p in glob.glob(os.path.join(home, "*.json")): best = (key, m) if best: m = best[1] - print("%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), - m.get("engine") or "claude", m.get("pid", ""))) + print("%s|%s|%s|%s|%s|%s|%s" % (m.get("uuid", ""), m.get("status", ""), m.get("current", ""), + m.get("engine") or "claude", m.get("pid", ""), m.get("host", ""), + m.get("boot_id", ""))) PY } # rc 0 when the reviewer's CURRENT turn is live: claude → session uuid visible -# in `claude agents`; codex → recorded pid alive. -_is_live() { # +# in `claude agents`; codex → recorded pid alive ON THIS HOST (a foreign-host +# pid is dead by definition — only its number migrated with the registry). +_is_live() { # + _identity_local "${4:-}" "${5:-}" || return 1 if [ "$2" = "codex" ]; then - [ -n "$3" ] && kill -0 "$3" 2>/dev/null + [ -n "$3" ] || return 1 + kill -0 "$3" 2>/dev/null return fi claude agents --json --all 2>/dev/null | CUR="$1" python3 -c ' @@ -126,24 +133,45 @@ sys.exit(0 if any(a.get("sessionId") == os.environ["CUR"] for a in d) else 1)' _retire() { "$DAEMON_SCRIPTS/daemon-retire.sh" "$1" >/dev/null 2>&1 || true; } -# rc 0 when some `claude agents` row's cwd equals worktree path <1> — a live -# daemon (of ANY kind, not just a review worker) is sitting in it. This is -# defense-in-depth ON TOP OF the registry dedupe check above, not a -# replacement for it: fail-open on an `agents` flake (empty/unreadable list → -# not occupied → removal proceeds) is correct here, do not invert it. +# rc 0 when some LOCAL `claude agents` row's cwd equals worktree path <1>. A +# visible row with matching foreign registry metadata migrated with the session +# store, not the process, so it does not occupy the worktree. Unmanaged rows +# have no identity evidence and remain conservatively local/occupied. _wt_occupied() { - claude agents --json --all 2>/dev/null | WT="$1" python3 -c ' -import json, os, sys + claude agents --json --all 2>/dev/null | \ + DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" DAEMON_BOOT_ID="$DAEMON_BOOT_ID" WT="$1" python3 -c ' +import glob, json, os, sys try: d = json.load(sys.stdin) except Exception: d = [] -sys.exit(0 if any(a.get("cwd") == os.environ["WT"] for a in d) else 1)' && return 0 +metas = {} +for p in glob.glob(os.path.join(os.environ["DAEMON_HOME"], "*.json")): + if p.endswith(".reply.json"): + continue + try: + m = json.load(open(p)) + except Exception: + continue + if (m.get("engine") or "claude") == "claude" and m.get("current"): + metas[str(m["current"])] = m +def local(m): + host = str(m.get("host") or "") + boot = str(m.get("boot_id") or "") + return (not host or host == os.environ["DAEMON_HOST"]) and \ + (not boot or not os.environ["DAEMON_BOOT_ID"] or boot == os.environ["DAEMON_BOOT_ID"]) +for a in d: + if a.get("cwd") != os.environ["WT"]: + continue + m = metas.get(str(a.get("sessionId") or "")) + if m is None or local(m): + sys.exit(0) +sys.exit(1)' && return 0 # codex workers never appear in `claude agents` — scan the registry, but # count ONLY codex metas with a live pid; a stale claude-engine `working` # meta must NOT start blocking removal (the claude path's fail-open # behavior above is unchanged). - DAEMON_HOME="$DAEMON_HOME" WT="$1" python3 - <<'PY' + DAEMON_HOME="$DAEMON_HOME" DAEMON_HOST="$DAEMON_HOST" DAEMON_BOOT_ID="$DAEMON_BOOT_ID" WT="$1" python3 - <<'PY' import glob, json, os, sys home = os.environ["DAEMON_HOME"]; wt = os.environ["WT"] for p in glob.glob(os.path.join(home, "*.json")): @@ -157,6 +185,12 @@ for p in glob.glob(os.path.join(home, "*.json")): continue if m.get("status") not in ("working", "blocked"): continue + host = str(m.get("host") or "") + if host and host != os.environ["DAEMON_HOST"]: + continue # foreign-host pid — the process did not migrate with the registry + boot_id = str(m.get("boot_id") or "") + if boot_id and os.environ["DAEMON_BOOT_ID"] and boot_id != os.environ["DAEMON_BOOT_ID"]: + continue # prior-boot pid — the pid namespace did not survive the reboot pid = str(m.get("pid") or "") if pid.isdigit(): try: @@ -291,15 +325,16 @@ PY # Dedupe verdict for PR <1> in mode <2> (triggered|sweep), cr-label flag <3>. # Prints: "dispatch" | "respawn " | "skip ". _decide() { - local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid + local pr="$1" mode="$2" cr="$3" meta uuid status current rest engine pid whost wboot if [ "$cr" = "1" ]; then echo "skip confident-ready label (remove it to force re-review)"; return; fi meta="$(_reviewer_meta "$pr")" if [ -z "$meta" ]; then echo "dispatch"; return; fi uuid="${meta%%|*}"; rest="${meta#*|}"; status="${rest%%|*}"; rest="${rest#*|}" - current="${rest%%|*}"; rest="${rest#*|}"; engine="${rest%%|*}"; pid="${rest#*|}" + current="${rest%%|*}"; rest="${rest#*|}"; engine="${rest%%|*}"; rest="${rest#*|}" + pid="${rest%%|*}"; rest="${rest#*|}"; whost="${rest%%|*}"; wboot="${rest#*|}" case "$status" in working|blocked) - if _is_live "$current" "$engine" "$pid"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; + if _is_live "$current" "$engine" "$pid" "$whost" "$wboot"; then echo "skip active reviewer"; else echo "respawn $uuid"; fi ;; retired) echo "dispatch" ;; *) if [ "$mode" = "triggered" ]; then echo "respawn $uuid" diff --git a/skills/reviewing-prs/scripts/review-engine.sh b/skills/reviewing-prs/scripts/review-engine.sh index f2bf10e285..d1cadd18d5 100755 --- a/skills/reviewing-prs/scripts/review-engine.sh +++ b/skills/reviewing-prs/scripts/review-engine.sh @@ -42,9 +42,12 @@ effort="${CODEX_REVIEW_EFFORT:-xhigh}" source_codex_home="${CODEX_HOME:-$HOME/.codex}" # TLS trust anchors as a FILE bundle — a nested codex cannot reach the OS -# keychain/trustd under the outer seatbelt (shakedown FU-6). -if [ -z "${SSL_CERT_FILE:-}" ] && [ -f /etc/ssl/cert.pem ]; then - export SSL_CERT_FILE=/etc/ssl/cert.pem +# keychain/trustd under the outer seatbelt (shakedown FU-6). macOS ships the +# bundle as cert.pem, Debian/Ubuntu as ca-certificates.crt. +if [ -z "${SSL_CERT_FILE:-}" ]; then + for _cert in /etc/ssl/cert.pem /etc/ssl/certs/ca-certificates.crt; do + if [ -f "$_cert" ]; then export SSL_CERT_FILE="$_cert"; break; fi + done fi # A nested codex resolves its code-mode command host to /usr/local/bin # (absent here) instead of ~/.local/bin — point it explicitly. diff --git a/tests/orchestrating-daemons/test-codex-scripts.sh b/tests/orchestrating-daemons/test-codex-scripts.sh index 2c5f6c3bf4..fcd6881c19 100755 --- a/tests/orchestrating-daemons/test-codex-scripts.sh +++ b/tests/orchestrating-daemons/test-codex-scripts.sh @@ -40,6 +40,7 @@ export DAEMON_HOME="$TEST_ROOT/registry" export STUB_STATE="$TEST_ROOT/stub" export DAEMON_TIMEOUT=10 export DAEMON_UUID_POLL=5 +export DAEMON_BOOT_ID="boot-current" WORK="$TEST_ROOT/work" mkdir -p "$HOME" "$WORK" "$STUB_STATE" # Fake code-mode host under the hermetic HOME so _codex_launch's only-if-exists @@ -308,6 +309,62 @@ kill "$stale_pid" 2>/dev/null || true wait "$stale_pid" 2>/dev/null || true assert_equals "$(meta_field "$uuid_stale" status)" "retired" "terminal codex record is still retired" +echo "== host-aware liveness: a foreign-host pid is dead by definition ==" +# A registry that migrated on a state volume carries pids recorded on the old +# machine. A reused (live-HERE) pid number must not read as a live worker: +# resume proceeds (resuming here IS the recovery act) and retire never signals +# an unrelated local process. +assert_equals "$(meta_field "$uuid_e" host)" "$(hostname)" "codex-spawn stamps host" +assert_equals "$(meta_field "$uuid_e" boot_id)" "$DAEMON_BOOT_ID" "codex-spawn stamps boot id" +sleep 30 & foreign_pid=$! +uuid_mig="hostmig1-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","name":"migrated","short":"hostmig1","engine":"codex","pid":"%s","host":"old-host","status":"working","cwd":"%s"}' \ + "$uuid_mig" "$uuid_mig" "$foreign_pid" "$WORK" > "$DAEMON_HOME/$uuid_mig.json" +out="$("$SCRIPTS_DIR/codex-resume.sh" "$uuid_mig" "resume after migration" 2>&1)" +assert_not_contains "$out" "a turn is still running" "resume does not mistake a foreign-host pid for a live turn" +assert_contains "$out" "stub reply: resume after migration" "resume proceeds on the new host" +assert_equals "$(meta_field "$uuid_mig" host)" "$(hostname)" "resume re-stamps host to this machine" +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "resume never signals the unrelated local process behind the foreign pid" +else + fail "resume never signals the unrelated local process behind the foreign pid" +fi +uuid_mig2="hostmig2-0000-4000-8000-000000000000" +printf '{"uuid":"%s","name":"migrated2","short":"hostmig2","engine":"codex","pid":"%s","host":"old-host","status":"working"}' \ + "$uuid_mig2" "$foreign_pid" > "$DAEMON_HOME/$uuid_mig2.json" +"$SCRIPTS_DIR/daemon-retire.sh" "$uuid_mig2" >/dev/null +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "retire leaves a foreign-host working pid alone" +else + fail "retire leaves a foreign-host working pid alone" +fi +assert_equals "$(meta_field "$uuid_mig2" status)" "retired" "foreign-host record is still retired" + +# Rebuilding the same named host also creates a fresh pid namespace. A stale +# pid from the prior boot must therefore be treated like a foreign-host pid. +uuid_reboot="reboot01-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","name":"rebooted","short":"reboot01","engine":"codex","pid":"%s","host":"%s","boot_id":"boot-old","status":"working","cwd":"%s"}' \ + "$uuid_reboot" "$uuid_reboot" "$foreign_pid" "$(hostname)" "$WORK" > "$DAEMON_HOME/$uuid_reboot.json" +if out="$("$SCRIPTS_DIR/codex-resume.sh" "$uuid_reboot" "resume after reboot" 2>&1)"; then + pass "resume treats a prior-boot pid as dead" +else + fail "resume treats a prior-boot pid as dead"; echo " in: $out" +fi +assert_contains "$out" "stub reply: resume after reboot" "resume proceeds after a same-host reboot" +assert_equals "$(meta_field "$uuid_reboot" boot_id)" "$DAEMON_BOOT_ID" "resume re-stamps boot id" + +uuid_reboot2="reboot02-0000-4000-8000-000000000000" +printf '{"uuid":"%s","name":"rebooted2","short":"reboot02","engine":"codex","pid":"%s","host":"%s","boot_id":"boot-old","status":"working"}' \ + "$uuid_reboot2" "$foreign_pid" "$(hostname)" > "$DAEMON_HOME/$uuid_reboot2.json" +"$SCRIPTS_DIR/daemon-retire.sh" "$uuid_reboot2" >/dev/null +if kill -0 "$foreign_pid" 2>/dev/null; then + pass "retire leaves a prior-boot pid alone" +else + fail "retire leaves a prior-boot pid alone" +fi +kill "$foreign_pid" 2>/dev/null || true +wait "$foreign_pid" 2>/dev/null || true + echo "== _meta_set serializes concurrent RMW — no lost fields (FU-1) ==" # The lost-update race: the detached _codex_launch wrapper and the foreground # spawn each read-modify-write the same meta; without serialization the later diff --git a/tests/orchestrating-daemons/test-daemon-scripts.sh b/tests/orchestrating-daemons/test-daemon-scripts.sh index f71c07cae1..6827253f6e 100755 --- a/tests/orchestrating-daemons/test-daemon-scripts.sh +++ b/tests/orchestrating-daemons/test-daemon-scripts.sh @@ -33,6 +33,11 @@ assert_contains() { if printf '%s' "$1" | grep -Fq -- "$2"; then pass "$3"; else fail "$3"; echo " expected to find: $2"; echo " in: $1"; fi } +assert_not_contains() { + if printf '%s' "$1" | grep -Fq -- "$2"; then + fail "$3"; echo " expected NOT to find: $2"; echo " in: $1" + else pass "$3"; fi +} assert_file_exists() { if [[ -f "$1" ]]; then pass "$2"; else fail "$2"; echo " missing: $1"; fi } @@ -45,6 +50,8 @@ export HOME="$TEST_ROOT/home" export DAEMON_HOME="$TEST_ROOT/registry" export STUB_STATE="$TEST_ROOT/stub" export DAEMON_TIMEOUT=10 +export DAEMON_UUID_POLL=5 +export DAEMON_BOOT_ID="boot-current" WORK="$TEST_ROOT/work" mkdir -p "$HOME" "$WORK" "$STUB_STATE/agents" @@ -332,6 +339,28 @@ assert_contains "$(cat "$STUB_STATE/log/calls.log")" "stop $SHORT2" "retire stop "$SCRIPTS_DIR/daemon-retire.sh" "$SHORT" purge >/dev/null assert_file_absent "$DAEMON_HOME/$UUID.json" "retire purge removes the registry record" +FOREIGN_UUID="face0000-0000-4000-8000-000000000000" +printf '{"uuid":"%s","current":"%s","short":"face0000","name":"foreign","engine":"claude","host":"old-host","boot_id":"boot-old","status":"working"}' \ + "$FOREIGN_UUID" "$FOREIGN_UUID" > "$DAEMON_HOME/$FOREIGN_UUID.json" +: > "$STUB_STATE/log/calls.log" +"$SCRIPTS_DIR/daemon-retire.sh" "$FOREIGN_UUID" >/dev/null +assert_not_contains "$(cat "$STUB_STATE/log/calls.log")" "stop face0000" "retire does not stop a foreign-host Claude session" +assert_contains "$(cat "$DAEMON_HOME/$FOREIGN_UUID.json")" '"status": "retired"' "foreign-host Claude record is still retired" + +# _boot_id must record the BOOT identity — on macOS the sec field, where a +# greedy `.*sec = ` match lands inside `usec = ` and records microseconds. +# Cross-check against an independent parse (Linux: the boot_id file verbatim; +# macOS: the first integer in kern.boottime is the sec field). +if [ -r /proc/sys/kernel/random/boot_id ]; then + BOOT_EXPECT="$(cat /proc/sys/kernel/random/boot_id)" +else + BOOT_EXPECT="$(sysctl -n kern.boottime | grep -oE '[0-9]+' | head -1)" +fi +BOOT_GOT="$(bash -c "source '$SCRIPTS_DIR/_lib.sh' >/dev/null 2>&1; _boot_id")" +[ -n "$BOOT_GOT" ] && [ "$BOOT_GOT" = "$BOOT_EXPECT" ] \ + && pass "_boot_id records the boot identity, not a substring field" \ + || fail "_boot_id records the boot identity, not a substring field (got: $BOOT_GOT, want: $BOOT_EXPECT)" + # ---- 6) worktree isolation (native --worktree threading) --------------------- echo "worktree isolation:" WT_REPO="$TEST_ROOT/wtrepo"; mkdir -p "$WT_REPO" @@ -349,6 +378,43 @@ echo "failure windows:" spawn_short() { printf '%s' "$1" | sed -n 's/.*\[\([0-9a-f]*\) \/ .*/\1/p' | head -1; } spawn_uuid() { printf '%s' "$1" | sed -n 's/.*\[[0-9a-f]* \/ \([0-9a-f-]*\)\].*/\1/p' | head -1; } +# A resumed turn must become locally owned as soon as its UUID appears, before +# the long terminal-state watcher returns. Otherwise migrated metadata remains +# foreign while a real local turn is running and dispatch can duplicate it. +M_OUT="$("$SCRIPTS_DIR/daemon-spawn.sh" "migrated-resume" "seed-m" "$WORK")" +M_SHORT="$(spawn_short "$M_OUT")"; M_UUID="$(spawn_uuid "$M_OUT")" +python3 - "$DAEMON_HOME/$M_UUID.json" <<'PY' +import json, sys +p = sys.argv[1] +m = json.load(open(p)) +m["host"] = "old-host" +m["boot_id"] = "boot-old" +json.dump(m, open(p, "w"), indent=2) +PY +# The migrated short may be REUSED by an unrelated local agent — resume must +# never stop/rm through it. Seed a jobs dir standing in for that local agent. +mkdir -p "$HOME/.claude/jobs/$M_SHORT" +: > "$STUB_STATE/log/calls.log" +STUB_BG_STATE=running "$SCRIPTS_DIR/daemon-resume.sh" "$M_SHORT" "continue locally" > "$TEST_ROOT/migrated-resume.out" 2>&1 & +M_RESUME_PID=$! +for _ in $(seq 1 20); do + M_CUR="$(sed -n 's/.*"current": "\([^"]*\)".*/\1/p' "$DAEMON_HOME/$M_UUID.json")" + [ -n "$M_CUR" ] && [ "$M_CUR" != "$M_UUID" ] && break + sleep 0.25 +done +M_META="$(cat "$DAEMON_HOME/$M_UUID.json")" +assert_contains "$M_META" '"host": "'"$(hostname)"'"' "running resume is re-stamped to the local host before terminal polling" +assert_contains "$M_META" '"boot_id": "boot-current"' "running resume is re-stamped to the local boot before terminal polling" +[ -n "${M_CUR:-}" ] && [ "$M_CUR" != "$M_UUID" ] \ + && pass "running resume advances current before terminal polling" \ + || fail "running resume advances current before terminal polling" +wait "$M_RESUME_PID" 2>/dev/null || true +M_CALLS="$(cat "$STUB_STATE/log/calls.log")" +assert_not_contains "$M_CALLS" "stop $M_SHORT" "migrated resume never stops through the foreign short" +assert_not_contains "$M_CALLS" "rm $M_SHORT" "migrated resume never rms through the foreign short" +[ -d "$HOME/.claude/jobs/$M_SHORT" ] && pass "migrated resume leaves the reused short's jobs dir intact" \ + || fail "migrated resume leaves the reused short's jobs dir intact" + # (a) The fork command itself fails (session not resumable). Resume must exit # nonzero, flip status=error, and leave `current`/turns untouched — the daemon # must not be silently advanced past a launch that never happened. diff --git a/tests/reviewing-prs/test-land-dispatch.sh b/tests/reviewing-prs/test-land-dispatch.sh index 32e8c3e3aa..0f313fbb44 100755 --- a/tests/reviewing-prs/test-land-dispatch.sh +++ b/tests/reviewing-prs/test-land-dispatch.sh @@ -41,6 +41,7 @@ export BIND_LOG="$TEST_ROOT/bind.log"; : > "$BIND_LOG" export EDIT_LOG="$TEST_ROOT/edit.log"; : > "$EDIT_LOG" export PROMPT_DIR="$TEST_ROOT/prompts"; mkdir -p "$PROMPT_DIR" export STUB_COUNT="$TEST_ROOT/count" +export DAEMON_BOOT_ID="boot-current" # real git: bare origin + working clone with main and a PR head branch ORIGIN="$TEST_ROOT/origin.git" @@ -245,11 +246,58 @@ reset_state; seed_lander working # agents.json [] → session gone assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "dead land worker retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "dead land worker respawned" +reset_state +python3 - <<'PY' +import json, os +u = "feed0000-0000-4000-8000-000000000000" +json.dump({"uuid": u, "current": u, "name": "land-pr-5", "engine": "claude", + "host": "old-host", "boot_id": "boot-old", "status": "working", + "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo '[{"id": "feedcafe", "sessionId": "feed0000-0000-4000-8000-000000000000"}]' > "$MOCK_DIR/agents.json" +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host Claude land worker is retired despite a visible migrated session" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "foreign-host Claude land worker is respawned" + reset_state; seed_lander idle "$DISPATCH" 5 > /dev/null assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "finished land worker retired (explicit dispatch = fresh signal)" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "finished land worker re-dispatched" +# host-aware: a working codex land worker whose live-here pid was recorded on +# another host (registry migrated on a state volume) is DEAD → retire+respawn. +reset_state +sleep 300 & LANDPID=$! +PID="$LANDPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "feed0000-0000-4000-8000-000000000000", + "current": "feed0000-0000-4000-8000-000000000000", + "name": "land-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], + "feed0000-0000-4000-8000-000000000000.json"), "w")) +PY +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host live pid → land worker treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "foreign-host live pid → respawned" +reset_state +PID="$LANDPID" H="$(hostname)" python3 - <<'PY' +import json, os +json.dump({"uuid": "feed0000-0000-4000-8000-000000000000", + "current": "feed0000-0000-4000-8000-000000000000", + "name": "land-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": "boot-old", + "status": "working", "updated": "2026-07-12T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], + "feed0000-0000-4000-8000-000000000000.json"), "w")) +PY +"$DISPATCH" 5 > /dev/null +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "prior-boot live pid → land worker treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait land-pr-5" "prior-boot live pid → respawned" +kill "$LANDPID" 2>/dev/null; wait "$LANDPID" 2>/dev/null || true + # ---- ticketless PR -------------------------------------------------------------------- echo "ticketless:" reset_state diff --git a/tests/reviewing-prs/test-review-dispatch.sh b/tests/reviewing-prs/test-review-dispatch.sh index 3d6f349876..e8415175e8 100755 --- a/tests/reviewing-prs/test-review-dispatch.sh +++ b/tests/reviewing-prs/test-review-dispatch.sh @@ -40,6 +40,7 @@ export MOCK_LOG="$TEST_ROOT/gh-calls.log"; : > "$MOCK_LOG" export SPAWN_LOG="$TEST_ROOT/spawn.log"; : > "$SPAWN_LOG" export PROMPT_DIR="$TEST_ROOT/prompts"; mkdir -p "$PROMPT_DIR" export STUB_COUNT="$TEST_ROOT/count" +export DAEMON_BOOT_ID="boot-current" # real git: bare origin + working clone with main and a PR head branch ORIGIN="$TEST_ROOT/origin.git" @@ -242,6 +243,20 @@ out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "dead reviewer retired" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "dead reviewer respawned" +reset_state +H="old-host" B="boot-old" python3 - <<'PY' +import json, os +u = "feed0000-0000-4000-8000-000000000000" +json.dump({"uuid": u, "current": u, "name": "review-pr-5", "engine": "claude", + "host": os.environ["H"], "boot_id": os.environ["B"], "status": "working", + "updated": "2026-07-08T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo '[{"id": "feedcafe", "sessionId": "feed0000-0000-4000-8000-000000000000"}]' > "$MOCK_DIR/agents.json" +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "foreign-host Claude reviewer is retired despite a visible migrated session" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host Claude reviewer is respawned" + reset_state; seed_reviewer idle out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "retire:feed0000" "triggered mode retires a finished reviewer" @@ -349,6 +364,23 @@ if [ -d "$WT" ]; then pass "worktree still exists after the refused dispatch"; e fail "worktree still exists after the refused dispatch"; fi reset_state +# A session restored from a foreign state volume can still be visible in the +# local dashboard, but its foreign registry identity must not occupy the worktree. +U="foreign1-0000-4000-8000-000000000000" WT="$WT" python3 - <<'PY' +import json, os +u = os.environ["U"] +json.dump({"uuid": u, "current": u, "name": "other-daemon", "engine": "claude", + "cwd": os.environ["WT"], "host": "old-host", "boot_id": "boot-old", + "status": "working", "updated": "2026-07-08T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], u + ".json"), "w")) +PY +echo "[{\"id\": \"foreign1\", \"sessionId\": \"foreign1-0000-4000-8000-000000000000\", \"cwd\": \"$WT\"}]" \ + > "$MOCK_DIR/agents.json" +out="$("$DISPATCH" 5 2>&1)" || true +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host Claude session does not occupy the worktree" +assert_not_contains "$out" "live daemon occupies" "foreign-host Claude session does not block removal" +reset_state + # ---- sweep failure isolation ---------------------------------------------------- echo "sweep failure isolation:" # PR 4 (earlier in the sweep order) fails mid-dispatch; PR 5 must still be @@ -610,6 +642,73 @@ out="$("$DISPATCH" 5)" assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "stale claude-engine meta in the same cwd does not block removal (fail-open preserved)" reset_state +# (d) host-aware: a codex meta whose pid is live HERE but was recorded on +# another host (registry migrated on a state volume) is NOT an occupant — +# the process did not migrate, only its number did. +sleep 300 & WTPID=$! +WT="$WT" PID="$WTPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8002-0000-4000-8000-000000000000", + "current": "cdec8002-0000-4000-8000-000000000000", + "name": "occupant", "engine": "codex", "cwd": os.environ["WT"], + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8002-0000-4000-8000-000000000000.json"), "w")) +PY +: > "$SPAWN_LOG" +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host codex pid does not block removal — dispatch proceeds" +kill "$WTPID" 2>/dev/null; wait "$WTPID" 2>/dev/null || true +reset_state + +# ---- dedupe is host-aware too --------------------------------------------------- +# A working codex reviewer meta whose live-here pid carries a foreign host must +# read as DEAD in _decide: respawn, not "skip active reviewer". +echo "dedupe (host-aware):" +sleep 300 & DEDUPID=$! +PID="$DEDUPID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": "old-host", "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:cdec8003" "foreign-host live pid → reviewer treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "foreign-host live pid → respawned" +reset_state +# A rebuilt host can keep its hostname while getting a fresh pid namespace. +PID="$DEDUPID" H="$(hostname)" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": "boot-old", + "status": "working", "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$(cat "$SPAWN_LOG")" "retire:cdec8003" "prior-boot live pid → reviewer treated as dead and retired" +assert_contains "$(cat "$SPAWN_LOG")" "spawn:--no-wait review-pr-5" "prior-boot live pid → respawned" +reset_state +# control: same live pid, HOST MATCHING this machine → still a live occupant +PID="$DEDUPID" H="$(hostname)" B="$DAEMON_BOOT_ID" python3 - <<'PY' +import json, os +json.dump({"uuid": "cdec8003-0000-4000-8000-000000000000", + "current": "cdec8003-0000-4000-8000-000000000000", + "name": "review-pr-5", "engine": "codex", + "pid": os.environ["PID"], "host": os.environ["H"], "boot_id": os.environ["B"], "status": "working", + "updated": "2026-07-10T00:00:00Z"}, + open(os.path.join(os.environ["DAEMON_HOME"], "cdec8003-0000-4000-8000-000000000000.json"), "w")) +PY +out="$("$DISPATCH" 5)" +assert_contains "$out" "active reviewer" "same-host live pid still skips as active" +assert_equals "$(cat "$SPAWN_LOG")" "" "same-host live pid spawns nothing" +kill "$DEDUPID" 2>/dev/null; wait "$DEDUPID" 2>/dev/null || true +reset_state + echo if [[ "$FAILURES" -gt 0 ]]; then echo "$FAILURES test(s) FAILED"; exit 1