fix(system-agent): trinity-system adopts a rebuilt base image at the cold boundary (#1816)

[AndriiPasternak31]

Fixes #1816

system_agent_service.ensure_deployed() returned action: none the instant the trinity-system
container reported running — no drift predicate was evaluated at all. Combined with
restart_policy: unless-stopped and a canonical upgrade path (build-base-image.sh → start.sh) that
never touches agent containers, that made the platform orchestrator the most-stale agent in every
fleet, indefinitely and silently.

This closes the system-agent half that #1809 explicitly left out of scope: #1809 fixed image
adoption for the per-agent start path (start_agent_internal, cold-start-gated ninth predicate) and
named the trinity-system bootstrap as not-covered. Last of the three v0.8.5 upgrade holes (#1814
platform images, #1809 agent images, this).

The part that wasn't obvious

recreate_container_with_updated_config resolves the image from the old container's own
Config.Image reference — the tag trinity-agent-base:latest, not the pinned id. So every
config-drift recreate is also an image adoption, and AC2 ("a running trinity-system is never
image-recreated mid-operation") cannot be satisfied by gating the image predicate alone.

Two of the eight config predicates were permanently false for a freshly created system agent:

Predicate	Why
check_agent_auth_token_env_matches (#1159)	_create_system_agent's env dict never wrote TRINITY_AGENT_AUTH_TOKEN — the only writers were crud.py and the two lifecycle recreates
check_full_capabilities_match	the container runs cap_add=FULL_CAPABILITIES but carried no trinity.full-capabilities label, and a missing label reads as false against a fleet default of true

So the first POST /api/agents/trinity-system/start after any fresh provision replaced a running
orchestrator and swapped its image mid-operation. Convergent — a recreate writes both values, so the
second start is clean — which is why it survived this long.

What ships

1. Convergence (_create_system_agent) — writes derive_agent_token(SYSTEM_AGENT_NAME) and pins
trinity.full-capabilities: 'true', so both predicates go quiet.

2. Cold-boundary adoption (ensure_deployed):

Boundary	Behaviour
running	read-only — reports base_image_state (current/stale/unknown), WARNs naming the remedy, raises an operator alarm on stale only. No recreate call exists in this branch (source-pinned between named anchors).
stopped	delegates to start_agent_internal instead of a bare container_start — the cold boundary where the #1809 image gate fires. This is where adoption happens.
absent	unchanged

3. AC2 as a structural gate — start_agent_internal skips the whole needs_recreation block
for a running trinity-system and returns recreate_deferred: "system_agent_running". Deliberately
independent of predicate count: a tenth predicate added next quarter cannot reopen the hole.

4. Two rebuild fences (review + CSO pass, ADOPT-006/007):

• recreate_missing_container — the bug(recovery): soft-delete recovery dead-ends — recover tells you to start, but start 404s when the container is gone #1559 soft-delete recovery rebuild — now refuses
  trinity-system with a 409. start_agent_internal falls through to it when the container lookup
  returns None, and bug: trinity-system never adopts a rebuilt base image — ensure_deployed short-circuits on 'already running' #1816 newly reaches that from the boot path and /restart; ensure_deployed
  runs in every uvicorn worker with no leader lock, so a concurrent recreate can null the lookup
  mid-flight. That path reconstructs a regular agent: it deactivates the system-scoped MCP key
  and mints an agent-scoped one (plaintext unrecoverable ⇒ the orchestrator irreversibly loses its
  permission bypass), drops trinity.is-system, the /template bind and unless-stopped, and arms
  the scope-403 TRINITY_BACKEND_URL. Failing closed is self-healing — ensure_deployed's create
  branch rebuilds it correctly on the next boot. (The race itself is refactor: per-agent start lock — complete the concurrent-start race hardening from #1809 #1817.)
• POST /api/system-agent/restart and /reinitialize are now human-only
  (reject_agent_principal, trinity-ops-agent#232 precedent). assert_admin rejects connector
  principals but not agent ones, and get_current_user hands an agent-scoped key its owner's role —
  so on a default admin-owned install any non-ephemeral agent's injected TRINITY_MCP_API_KEY passed
  it. Tolerable while /restart was a stop+start; not once this PR makes it replace the container.
  No-op for JWT / user-scoped / system-scoped callers.

Also fixed along the way: unless-stopped silently vanished on every agent recreate — the old
container's HostConfig was extracted and never read. trinity-system is created with it, so one
recreate downgraded the orchestrator to "stays down after a crash or host reboot".

Deliberate decisions worth reviewing

• check_full_capabilities_match is system-aware. Pinning the label alone would, on any install
  with agent_full_capabilities=false, produce a mismatch that can never converge — a recreate on
  every start, forever. The predicate (checker) and the recreate's new full_capabilities override
  (writer) both route through one helpers.is_system_agent_name(). That helper is deliberately a
  name test rather than db.is_system_agent: it must be unfailable (a DB error that flips the
  answer would either recreate the orchestrator or leak full capabilities) and it must not widen the
  exemption to any is_system-flagged row.
• check_base_image_matches split into a 3-state core + an unchanged boolean wrapper. The alarm
  cannot be built on a boolean whose True means both "the image is current" and "the check could not
  run" — alarming on that recreates the bug: a rebuilt agent base image is never picked up — no image-drift predicate, so the v0.8.5 "stop/start-with-recreate" upgrade step is a no-op #1809 symptom one layer up. unknown never alarms.
  bug: a rebuilt agent base image is never picked up — no image-drift predicate, so the v0.8.5 "stop/start-with-recreate" upgrade step is a no-op #1809's consumer is byte-identical (state != "drift").
• No TRINITY_BACKEND_URL for the system agent. It gates the agent-side heartbeat loop, and
  authorize_heartbeat accepts only scope == "agent" keys — the system agent's is scope == "system".
  This PR makes the recreate a routine path for it, so arming it would newly create a permanent 5s
  403 loop (~17k backend log lines/day). Whether the orchestrator should be visible to fleet health
  is a real question, and a separate one.
• /reinitialize deliberately does not adopt — already an explicit stop, zero incremental AC
  coverage, and its four-site handle rebinding was the highest-risk edit considered. Pinned by a test.

Consequences operators must know

1. Adoption destroys the container writable layer. Everything installed outside /home/developer —
   apt packages, system pip, /usr/local, /etc — is gone. trinity-system is the one agent that runs
   FULL_CAPABILITIES explicitly for package installation, so this is the most consequential side
   effect of the feature. (/var/lib/trinity/oauth-token is wiped too; harmless — trinity-system uses
   ANTHROPIC_API_KEY.)
2. Adoption re-reads ANTHROPIC_API_KEY from get_anthropic_api_key() and rewrites trinity.cpu /
   trinity.memory from db.get_resource_limits — creation reads them from template.yaml, so the two
   can legitimately disagree.
3. On the canonical upgrade path, adoption is operator-triggered (alarm → Restart), not automatic.
   The system agent is running after every canonical upgrade, and that is the branch that must not act.
4. An admin who changes a running system agent's config and clicks Start now gets no recreate. A
   deliberate divergence from regular-agent semantics; the response names the remedy
   (recreate_deferred).

Verification

Manual, on a live stack — /verify-local --keep Stage 5 (agent-exercise), the blocking check

Mocks cannot model tag-vs-id pinning, which is this bug class — #1809 shipped unit-only and left
exactly this hole. So the adoption was driven against a real trinity-system container on the
Stage-5 stack, rebuilding the real base image mid-flight: 27/27 assertions, 0 failures.

Step	Result
creation converges (on a REAL container)	TRINITY_AGENT_AUTH_TOKEN baked · trinity.full-capabilities=true · restart_policy=unless-stopped
rebuild moves :latest	d9784ee75bfc → d64c89b684b7; the running container stays pinned to the old id
AC2 — running + drift, restart the backend	container id unchanged; still on the old image; WARNING + alarm logged
AC1 — the alarm	base-image-stale-* operator-queue row raised; content carries no image ids/digests
/api/system-agent/status	base_image_state: "stale", enum-only
D1.2 — POST /agents/trinity-system/start while running	id unchanged; response carries recreate_deferred
AC1 — stop it, restart the backend	container REPLACED 12722a8dce22 → 46b86ff9e7e3, adopted d64c89b684b7
what survived the replacement	restart policy · cap_add (9) · cap_drop ALL · ssh port 2227 · TRINITY_MCP_API_KEY · /template + /home/developer mounts · the workspace volume · the capabilities label · the auth token
no 403 heartbeat loop	no TRINITY_BACKEND_URL on the replacement
convergence	a second boot does not recreate again; /status → current

Backend log from the AC2 step, verbatim:

Base image drift for trinity-system: sha256:d9784ee75bfc… -> sha256:d64c89b684b7… (reference 'trinity-agent-base:latest')
System agent is running a stale base image. It will NOT be recreated while running (#1816 AC2).
  Adopt it with POST /api/system-agent/restart, or stop it and let the next backend boot pick it up.
System agent: none - System agent already running, but on a STALE base image — restart it … to adopt

Automated

• /verify-local --keep: PASS — preflight · agent-precheck · build+import-smoke ·
  agent-build+import-smoke · boot+health · Stage 5 agent-exercise · integration, all green.
• Unit: 5154 passed, 1 failed (full tests/unit, alphabetical) — the 1 is the pre-existing
  failure below. 69 tests are new.
• Re-run CI-style (cd tests && pytest unit/ -m "not slow" -p randomly --randomly-seed=…) on two
  local seeds: 12345 → 5154 passed / 1 failed (the pre-existing one); 777 → 5145 passed /
  1 failed / 9 errors. The 9 are all test_subscription_auto_switch_pingpong.py and are not mine
  — proven by re-running the same seed with only the test-order fix below reverted (identical item
  set ⇒ identical shuffle, so this is a controlled comparison rather than a cross-tree seed match,
  which is not comparable): byte-identical result, 9 errors + 1 failure either way.
• tests/lint_sys_modules.py: 203 violations in 60 files, baseline allows 240 — no new violations.
  Pyflakes on every touched file: no new findings.
• Pre-existing failure, proven not mine (identical on pristine origin/dev):
  test_1069_voip_call_path_param — the verify-venv fastapi get_flat_dependant drift.

One real defect the validation pass caught, and fixed

test_1816_system_agent_adoption.py passed alone and in the full alphabetical run, but 21 of its 67
tests failed when collected alongside tests/unit/test_start_agent_skip_inject.py. Five sibling unit
files replace services.agent_service / .helpers with Mock objects in sys.modules at
collection time (the grandfathered #762 class), and this file resolves the real modules lazily,
inside fixtures — i.e. after the contamination. The leak is silent rather than loud:
is_system_agent_name becomes "no agent is the system agent", so the AC2 gate never fires and the
recreate it exists to suppress happens, while every assertion still looks meaningful. Fixed by
importing the real modules at collection time (this file sorts first) and pinning them per-test via
monkeypatch.setitem — self-restoring, so the siblings' own harnesses are untouched. Reproduction now
green: 21 failed → 0; the original 5-file set 21 failed → 108 passed.

Worth noting for the reviewer: this would not reliably have been caught downstream. It is invisible
to an alphabetical full-suite run, and at seed 777 the adoption suite happened to land in a lucky order
and passed even unfixed — so the 3-seed CI matrix catches it only probabilistically.

Known, documented, not fixed here

ensure_deployed runs once per worker lifespan and the staleness-alarm cooldown is per-process, so a
stale boot files one queue item per worker (--workers 2 ⇒ 2 — visible in the run above).
Deliberate: a cross-worker cursor puts Redis or a DB read on the boot path for an advisory alarm, and
the un-guessable timestamped id that stops an agent pre-creating and silencing it (#1632) is inherently
undedupable by on_conflict_do_nothing. Said plainly in both the code and the flow doc.

Not in scope

Deferred	Why
#1817 per-agent start lock	Separate issue, separate PR. The boot double-provision race is pre-existing and not widened here — ADOPT-006 fences its worst consequence.
System-scoped MCP key hygiene	get_agent_container returns None on any Docker error, so deactivate-before-run would turn a transient socket hiccup into an outage of the privileged orchestrator.
System-agent heartbeat	authorize_heartbeat rejects scope='system'; a design question, not this bug.
Rewiring _create_system_agent onto crud.py	Follow-up refactor.

Docs

requirements/infrastructure.md §8.5b (ADOPT-001…007, the new home for #1809's semantics too),
architecture.md (system_agent_service entry, the #1560 lifecycle-clearing paragraph, Invariant #8
"Role ≠ human"), feature-flows/internal-system-agent.md (new "Base-image adoption" section),
feature-flows/agent-lifecycle.md, feature-flows.md index row, plus learnings.md.

---

Reviewers: @webmixgamer (wrote #1809's predicate — please sign off on the 3-state extraction and the
system-aware check_full_capabilities_match), @dolho, @vybe.

[github-actions]

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

[vybe]

Validated via /validate-pr. Fixes #1816 with a closing keyword. The AC2 boundary is honored (running branch is read-only; adoption only at the cold boundary / explicit restart), the staleness alarm follows the #1632 reserved-prefix + #1644 no-load-bearing-queue-item lessons, is_system_agent_name is correctly a deterministic name test that never widens capabilities, and the restart endpoint gets reject_agent_principal per the ops-agent#232 pattern. Docs complete (architecture + 4 flows + requirements + learnings), named #1816 regression suites present, full pytest matrix + prod-image-smoke green.