feat(orchestrating-daemons): host-stamped registry + host-aware pid liveness (managed-agents steals)

[SSFSKIM]

What

Adopts the transferable parts of Anthropic's Scaling Managed Agents: Decoupling the brain from the hands into the board pipeline. Full transfer analysis: docs/doperpowers/2026-07-12-managed-agents-steals.md.

Steal 1 (code). Registry metas now carry a host stamp at every registration/resume; _pid_alive() treats a pid recorded on another host as dead regardless of kill -0. Fixes the silent stall class where a state-volume migration (host died/rebuilt/moved to cloud) leaves reused pid numbers reading as live workers: resume refused, retire signaling unrelated processes, dispatch skipping dead reviewers, worktree removal blocked by ghosts. All four pid consumers (codex-resume guard, daemon-retire kill, review/land-dispatch _is_live, _wt_occupied) are host-aware; empty host = legacy meta = local (behavior unchanged).

Steal 2 (convention). State-volume spec for a future dedicated worker host: registry, session stores, clones on one detachable volume; compute disposable.

Steal 3 (recipe). Token-wired-remote scope split (push scope in clone remote URL, board scope in env) recorded as the narrowing path for the GH_TOKEN accepted note — TECH-DEBT pointer added; applies at host provisioning, not on this Mac.

Testing

• tests/orchestrating-daemons/test-codex-scripts.sh — new: foreign-host pid → resume proceeds + re-stamps host, never signals the unrelated local process; retire leaves it alone. PASS
• tests/reviewing-prs/test-review-dispatch.sh — new: foreign-host live pid → retire+respawn; _wt_occupied not occupied; same-host control still skips active. PASS
• tests/reviewing-prs/test-land-dispatch.sh — new: foreign-host live pid → retire+respawn. PASS
• test-daemon-scripts.sh, test-board-scripts.sh, scripts/lint-shell.sh — PASS

Fork-internal change (personal fork; no upstream PR intended).

[SSFSKIM]

Second commit dde32ad adds the runnable provisioning layer: infra/worker-host/ (cloud-init body / state-volume soul / hand-seeded secrets) + SSL_CERT_FILE Linux bundle path in _codex_launch and review-engine.sh (TECH-DEBT #8 resolved). Target: Hetzner CX43, Ubuntu 24.04, no Docker.

[SSFSKIM]

Third commit 76cbd9e: triage tenant layer — the triaging-feedback poller becomes the worker host's first (burn-in) tenant.

• cloud-init.yaml: triage user (uid 1011, /data/triage) separated from worker — its env carries SUPABASE_SERVICE_ROLE_KEY while worker executes PR-derived code; doper-triage systemd oneshot+timer (10 min) replaces the Mac launchd label with the same single-instance serialization, ExecCondition on the seeded .env makes pre-seed ticks no-ops and rebuilds zero-touch.
• env.triage.example: skill-dir .env contract with the VM deltas (explicit issues-scope GH_TOKEN, contents-READ-only PAT in the ida-solution remote — ticket-only, never pushes).
• README.md §5: seeding (p86 migration + labels as prerequisites), one-tick verification, Mac handoff.

Cloud-init YAML validated; shell lint green (no shell changes).

[SSFSKIM]

Integration re-verified against current main (351bf0d): git merge-tree clean, and the three suites touching overlapped files (test-review-dispatch, test-land-dispatch, test-codex-scripts) all pass on the actual merged tree — main's land-mode split (7e11019) and this branch's host-aware liveness compose without conflict. Marked ready for review.

[SSFSKIM]

Review verdict (5 findings) addressed in 447e9d1 — all five confirmed against the code before fixing:

• Rebuild-unique host identity (P1): hostname is now set in runcmd to doper-worker-<machine-id prefix> (machine-id is minted per body), with preserve_hostname: true so cloud-init's per-boot update can't revert it. _lib.sh's DAEMON_HOST=$(hostname) contract is untouched; the README §4 claim about foreign-host neutralization is now actually true on the rebuild path.
• Push token reach (P1): README §2.4 no longer implies the remote-URL embedding hides the PAT from workers — it states workers ARE contents-write principals, and requires branch protection on main/integration branches before the first worker runs. (A credential broker was considered and rejected as over-engineering for a single-tenant host; the honest-boundary + protected-refs model is the fix.)
• Runner env (P1): §2.6 now seeds ~/runner/.env (comment-stripped copy of ~/.env + DOPERPOWERS_HOME) and ~/runner/.path; the cloud-init comment claiming bash -lc coverage for runner jobs was false (Ubuntu's .bashrc early-returns non-interactive) and is corrected — .bashrc wiring is for interactive admin shells only.
• Runner label (P1): registration now uses --labels claude-review, matching the shipped pr-review-dispatch.yml's runs-on.
• Runner service after rebuild (P2): rebuild drill now reinstalls the unit (svc.sh install worker && svc.sh start) instead of 'restarting' a unit that died with the root fs.

cloud-init YAML validity re-verified after the edits.

[SSFSKIM]

Second review wave (2 findings) addressed in 16a8b2d — both confirmed:

• /data mount gate (P1): nofail stays (dropping it trades a silent-wrong-disk failure for systemd emergency mode and an unreachable box), but runcmd now opens with mountpoint -q /data || exit 1 — cloud-init's module order means the homedirs already exist on the root fs at that point, so the gate's job is to abort before CLIs/timer and make cloud-init status report the failure. final_message no longer claims "ready"; it and README §1 both direct to cloud-init status --long before any seeding. Recovery = attach the volume and recreate the server (runcmd is per-instance).
• codex auth seeding (P2): scp … worker@host could never work (locked password, no keys, no ~/.codex). §2.2 now copies through the login account and uses install(1) with -o worker -m 600 reading /dev/stdin — ownership and mode correct from birth, no temp file on the remote.

YAML validity re-verified.

[SSFSKIM]

Third review wave (3 findings) addressed in 38716d5 — all confirmed:

• Tenant home modes (P1): chmod 700 /data/worker /data/triage added to runcmd right after the mount gate (homes exist by then — the users module runs earlier), rather than trusting the image's login.defs default. Idempotent on rebuilds against an existing soul. The §2.6 runner-env recipe now ends with chmod 600 ~/runner/.env.
• Git identity (P1): §2.4 seeds user.name/user.email once per soul — without it git refuses to auto-detect an email from the machine-id hostname and the implementation protocol's first commit fails. triage needs none (ticket-only, never commits).
• Fail-fast provisioning (P2): runcmd items merge into ONE #!/bin/sh script, so a leading set -e item governs everything below it. dash has no pipefail, so the NodeSource/Tailscale curl|bash pipelines are split into download-then-execute; the claude installer keeps its pipe but inside an inner bash with set -e -o pipefail. Provisioning now ends with an explicit gate: required binaries present, Node major ≥22, claude resolvable on the worker's PATH.

Verified: YAML parses; the merged runcmd script passes sh -n; item order is set -e → mount gate → chmod 700 → rest.

[SSFSKIM]

Fourth review wave (2 findings) addressed in b525ed0 — both confirmed:

• RUNNER_TRACKING_ID (P1): real and the nastier one — the Actions runner's post-job cleanup kills any surviving process whose environ still carries the job's tracking marker, so nohup/--bg detachment was never sufficient; a no-wait dispatch job would reap its own worker at job exit and strand a working registry entry. Fixed at all three detach points — daemon-spawn.sh, daemon-resume.sh (the fork also runs under runner dispatch on the resume path), and _codex_launch's nohup wrapper — each now spawns under env -u RUNNER_TRACKING_ID (no-op outside Actions, where the var is unset). README §2.6 now states the stripping as the actual survival mechanism rather than implying detachment suffices.
• Triage first-tick verification (P2): sudo systemctl inside the sudo -iu triage shell can never work (no sudo membership, locked password, no journal read access). §5.4 now runs from the admin shell and the §5 intro marks step 4 as the exception to the as-triage seeding.

Rebased onto 7db6b8d (boot-id-bound liveness) before pushing; test-daemon-scripts, test-codex-scripts, test-review-dispatch all pass on the combined tree, shellcheck lint clean.

[SSFSKIM]

Fifth review wave (1 finding) addressed in d49e352 — confirmed empirically before fixing: GNU coreutils install exits 1 on a pipe-backed /dev/stdin (skipping file '/dev/stdin', as it was replaced while being copied) and creates nothing, so the §2.2 recipe introduced in the second wave could never seed auth.json on the Ubuntu target (it only worked on BSD install, where I wrote it).

Replaced with umask 077 && cat > … && chown worker:worker … over the same root SSH path — the file is born 0600 under the umask, so the no-exposure-window property that motivated install(1) is preserved without depending on GNU-specific pipe handling. Recipe shape verified locally (pipe → umask 077 shell → mode -rw------- with intact content).

[SSFSKIM]

Review trail\n\nEngine: native Codex review via review-engine.sh, model gpt-5.6-sol, effort xhigh.\n\nClosing artifact cross-check:\n- The PR body has no Validation Evidence section. Per protocol, absence is recorded but is not a finding.\n- The five named test suites pass. The board suite initially failed only because sandboxed process inspection denied ps; it passed outside the sandbox.\n- The bare scripts/lint-shell.sh command exits 0 with No shell files found on a clean checkout. Explicitly linting all 13 changed shell files passes.\n- GitHub reports no CI checks.\n\nRound 1:\n- P2, same-host reboot PID reuse: FIX NOW. Added boot_id stamping and host plus boot liveness gates. Regression tests cover resume, retire, review dispatch, and land dispatch.\n- P2, Claude host checks missing from review and land liveness: FIX NOW. Host plus boot identity now gates Claude dedupe, worktree occupancy, and retirement. Unmanaged visible Claude sessions still conservatively block removal.\n- Both fixes were committed and pushed in 7db6b8d. Focused suites and shell lint passed.\n\nRound 2:\n- P1, resumed Claude turns remain foreign until terminal polling finishes: FIX NOW. daemon-resume now registers the new UUID and local host plus boot identity immediately after the UUID appears, before the long terminal poll.\n- The regression failed before the change and passes after it. All four focused suites and shell lint pass.\n- Fix commit 01e31c1 is local only. The first push was rejected because the head advanced to b525ed0; the fix was rebased cleanly. The single retry was rejected again because the head advanced to d49e352. Per protocol, no further rebase and no force-push were attempted.\n- Round 3 was not run because this P1 fix is not on the PR head.\n\nFinding bins:\n- Round 1 P2 reboot identity: FIX NOW, pushed.\n- Round 1 P2 Claude liveness identity: FIX NOW, pushed.\n- Round 2 P1 early resume registration: FIX NOW, verified and committed locally, but blocked from push by repeated head movement.\n\nVerdict: NEEDS ATTENTION. The current PR head still has the unresolved P1. A human must reconcile local commit 01e31c1 onto current head d49e352 and then run the third and final engine round.\n\nTier judgment: not self-merge eligible. The PR targets the default branch main, has no CI checks, exceeds the 150-line and 5-file self-merge bounds, and touches worker provisioning and token/auth handling. No confident-ready label was applied; help wanted marks the human handoff. No board writes were made because this PR has no linked ticket.

[SSFSKIM]

Review trail — reconciliation + final rounds (human-directed session)

Completing the earlier NEEDS ATTENTION verdict:

• Reconciled the stranded P1: local commit 01e31c1 (register resumed turn early) cherry-picked onto d49e352 as 5bb3888 — clean (disjoint files), focused suites + shellcheck green.
• Round 3 (engine: review-engine.sh, gpt-5.6-sol, xhigh, full delta): 1 finding. P2 — daemon-resume stop/purge ran through the meta-recorded short ID without checking its host; a migrated meta's short is host-local and reusable, so claude stop/claude rm could hit an unrelated local agent. Fixed in 402df1b: identity verdict captured BEFORE the fork re-stamps the meta; foreign → skip both. Regression: 3 asserts, fail-before/pass-after verified.
• Round 4: 2 findings. P2 — macOS kern.boottime parse: greedy .*sec = landed inside usec = and recorded microseconds as the boot identity (reproduced live: got 507683, want 1783766484). Fixed in b7ead32 (anchor on leading { sec = N,), regression cross-checks _boot_id against an independent parse, fail-before verified. P1 — project-specific config in core: triaged VOID for this personal fork (the CLAUDE.md rule binds upstream PRs; the concrete values are the deployment record) — kernel kept as a permanent fork-local/never-upstream header guard in env.triage.example.
• Round 5 (fork-context line in criteria): CLEAN — no findings; engine confirms host/boot-aware liveness consistent across registration/resume/retire/dispatch with regression coverage.

All four focused suites + shell lint green on head b7ead32. Landing by human direction (squash).