fix(reliability): clear per-agent Redis runtime state across the agent lifecycle (#1560)

[webmixgamer]

Summary

• agent:circuit:{name} (transport breaker) is keyed by agent name, not container identity, and carries no TTL. Nothing in the agent lifecycle cleared it, so any container replaced under the same name inherited its predecessor's dormant verdict and fast-failed every execution with "Agent circuit breaker open — agent is unhealthy" — without the backend ever contacting the agent.
• Adds services/agent_runtime_state.py as the single enumeration point for every name-keyed per-agent Redis keyspace — the Redis-side twin of the existing AGENT_REFS registry in db/agent_cleanup.py — and wires it into six lifecycle points.
• A parity test fails CI when a new agent:* keyspace ships unregistered, so the next key added cannot be silently forgotten (AC fix: add missing logging_config.py to backend Dockerfile #4).

Root cause, and why the acceptance criteria were incomplete

The issue names delete / rename / create as the fix sites. Investigation showed none of them is the reachable path:

• create is unreachable: is_agent_name_reserved() (db/agents.py:127) sees soft-deleted rows and crud.py 409s, so a name is locked for the whole retention window.
• the name only unlocks at purge: purge_agent_ownership() has exactly one caller — the retention sweep at cleanup_service.py — and it cleared no Redis state.
• the reachable path is start_agent_internal: its needs_recreation branch replaces the container on any config drift (subscription switch, resource change, auth-token rotation, guardrails edit). A single fleet-wide subscription rotation recreates every agent at once and resurrects every stale verdict.

Three sites were therefore added beyond the written ACs: start/recreate, purge, and the trinity-system bootstrap (system_agent_service.ensure_system_agent() recreates a fixed, permanently-recycled name via its own containers_run).

Design: two entry points, different blast radii

Function	Clears	Called from
clear_agent_breakers (sync)	heartbeat markers, transport circuit + probe-lock, dispatch breaker + probe-lock	start/recreate, create, system-agent bootstrap
clear_agent_runtime_state (async)	the above + execution slots	delete, rename (old and new name), purge

Slots are cleared only where the container is provably gone or stopped. force_clear_slots wholesale-DELs agent:slots:{name}; on a running agent that would drop capacity accounting for an in-flight fire-and-forget execution (#1083).

The start-path clear runs before the recreate, not after: recreate_container_with_updated_config brings the replacement up via containers_run(detach=True), so clearing afterwards leaves a window in which a concurrent dispatch reads the predecessor's verdict against an already-running container.

It is guarded on needs_recreation or not was_already_running, so a no-op start of an already-running agent cannot be used to reset a breaker protecting a genuinely wedged agent.

Evidence from a live instance

Before the fix, a dev instance carried 10 orphaned agent:circuit:* keys from long-deleted agents — every one state=dormant, TTL=-1, and every one still inside its deny window:

agent      state     failures  ttl   next_probe_at
agent-a    dormant   175       -1    +2273s   (would DENY)
agent-b    dormant   174       -1    +3307s   (would DENY)
...        (10 total, all dormant, none expiring)

They never decay into a harmless state: the #1561 poller keeps re-arming next_probe_at (last_failure_ts was 964s old on an agent deleted a week earlier). Deleting the keys by hand is futile while #1561 is unfixed — they were rebuilt to failures=9, state=open within 20 minutes.

Reproduced and then verified end-to-end against a real agent:

plant dormant verdict -> POST /task  ->  "circuit breaker open — agent is unhealthy"  (0s, agent never contacted)
                                          agent-server /health -> 200   (it was healthy the whole time)
recreate via config drift             ->  Circuit reset to CLOSED / Dispatch breaker reset to CLOSED
                      -> POST /task  ->  HTTP 200

Relationship to #1561 (PR #1565)

Complementary, both needed, no file overlap. #1561 removes the source (the sync-health poller hammering soft-deleted agents and poisoning their breakers); this PR removes the inheritance. Note that until #1565 lands, the delete-time clear is undone within 60s by that poller — correctness here rests on the purge / create / start clears, not the delete one.

Tests

• tests/unit/test_1560_agent_redis_key_parity.py — bidirectional registry parity; fails CI on an unregistered agent:* keyspace. Mutation-tested (adding a fake keyspace fails it).
• tests/unit/test_1560_breaker_cleared_on_lifecycle.py — per-subsystem fail-open behaviour + the six call sites, incl. that lifecycle.py never calls the slot-clearing variant. Mutation-tested (removing the start-path clear fails it).
• tests/integration/test_1560_breaker_lifecycle.py — against real Redis (real Lua; fakeredis has no EVALSHA): a dormant circuit denies, and after the sweep allow_request() allows — the exact gate task_execution_service reads. Plus an opt-in leg (TRINITY_LIFECYCLE_TEST_AGENT) that drives the full AC Feature/process engine #6 path: plant verdict → task fast-fails → real recreate → task returns 200.

150 passed, 1 skipped across the new tests plus the heartbeat / circuit-breaker / dispatch-wiring / soft-delete / cleanup-sweep / correlated-pause suites.

Deliberately not in scope

• The misleading error string (task_execution_service.py:1016) is unchanged. AC Setup improvements #5 is satisfied by its "either the state is cleared" branch; the wording is the explicitly-named secondary defect of bug: Disabling agent autonomy forces circuit breaker DORMANT and blocks all inbound user responses #1557, which is where it will change, so the two branches don't collide on the same line.
• The stale circuit_breaker_dormant operator-queue entry naming a deleted agent (mentioned in the issue's Technical Notes) is untouched — it reaches into a different subsystem. Worth a follow-up.

Known interaction (authorization-consistency note, not a security issue)

force_circuit_dormant(reason="autonomy_disabled") (agent_service/autonomy.py, #631) is an owner-only side effect, and the explicit POST /circuit-breaker/reset is admin-only. But POST /agents/{name}/start is AuthorizedAgentByName (owner, shared user, or admin), so after this change a shared non-owner can clear the breakers by stopping and starting the agent, and a restart clears the autonomy_disabled dormant marker while the autonomy flag stays off.

This was reviewed adversarially and is not a security finding. The clear touches only agent:*:{that-name} Redis keys; agent_ownership.autonomy_enabled and the schedule enabled flags live in SQLite and are untouched, so no autonomous execution is re-enabled — only background health-probing resumes. A shared user already holds chat/dispatch rights to that exact agent, so no new access, data, tool, or tenant is reachable. The trinity-system path is closed: clear_agent_breakers(SYSTEM_AGENT_NAME) is reachable only via admin reinitialize or a start that can_user_access_agent('trinity-system') denies with a uniform 404.

Worth an operator's eye nonetheless: the breaker-clearing side effect on start (shared) is broader than the dedicated reset endpoint (admin). #1557 removes the autonomy↔breaker conflation entirely, at which point the marker interaction disappears.

Two residual reliability notes, both self-healing and out of scope: a microsecond TOCTOU between purge_agent_ownership(name) and the sweep could drop one slot-ZSET entry for a name recreated inside that window (reclaimed by the slot TTL reaper); and the stale circuit_breaker_dormant operator-queue entry is untouched (see above).

Fixes #1560

[vybe]

/validate-pr ✅ — Fixes #1560. Single-enumeration agent_runtime_state.py registry + parity test + recreate-inheritance integration test; clears both breakers/heartbeat/slots across 6 lifecycle points, clears before containers_run(detach=True), guards slot-clear against in-flight #1083 capacity. Docs (architecture/feature-flows/learnings) updated. No secrets/packaging gaps.