fix(monitoring): health-check the whole fleet, not just running agents (#675)

[dolho]

Root cause

#669 patched the user-visible 500 (NULL status coercion). This is the actual cause of the stale 24h rollups: _run_check_cycle filtered the fleet to running agents before the health check —

running_agents = [a.name for a in agents if a.status == "running"]
if not running_agents:
    return

A stopped / exited / crashed agent was dropped from every cycle, so its agent_health_checks row never refreshed. Production: 8 of 9 agents last refreshed 2026-02-23..04-28; only the one running agent stayed current. The filter had the logic backwards — a down agent is exactly what monitoring should surface.

Fix

Check every agent. list_all_agents_fast() already passes all=True (stopped containers included, normalised to status="stopped"), and perform_health_check already produces a correct "docker exited / unreachable → unhealthy" record for a stopped agent.

The #631 dormant short-circuit still bounds cost for hard-down agents (synthetic detail, no DB writes) — explicitly not modified: re-enabling writes for dormant agents would reopen the SQLite contention #631 fixed. Net behaviour: a long-down agent settles to "unhealthy, circuit dormant, last-checked ≤ dormant-window ago" instead of literal months-stale data — correct and actionable.

Issue asks

Ask	Outcome
Confirm per-agent isolation, add if missing	Already present — perform_fleet_health_check uses asyncio.gather(..., return_exceptions=True) + per-result handling. Hypothesis #1 (one failure short-circuits fleet) did not hold; the bug was the filter. Documented in docstring.
Structured per-pass log	Added — one INFO line/cycle: fleet_health_check pass complete: agents=N (running=R stopped=S) healthy=.. degraded=.. unhealthy=.. critical=.. unknown=.. duration_ms=..
Audit registration path	No separate registry — cycle re-derives the fleet from list_all_agents_fast() each pass. "Not registered" reduces to this filter.
Pull prod logs around stale timestamps	Not done — no production access from this environment.

Tests

tests/unit/test_fleet_check_covers_all_agents.py (4):

• running + stopped both checked (root-cause regression guard)
• all-stopped fleet still checked (the exact months-stale prod shape — pre-fix returned early)
• zero-agent no-op
• pass-summary log emitted with correct running/stopped/status counts

Existing monitoring suite — test_monitoring_dormant_skip / test_monitoring_router_signatures / test_fleet_status_resilience (15) — still green; #631 dormant-skip behaviour unchanged.

Test plan

• 4 new tests pass
• 15 existing monitoring tests pass (no regression, bug: circuit breaker flood causes SQLite DB locks and UI unresponsiveness #631 intact)
• lint clean
• CI: full 6-seed matrix + regression diff

Related to #675

🤖 Generated with Claude Code

[dolho]

Live-verified on the running instance (real _run_check_cycle() against live Docker + DB, fleet of 5: 1 running, 4 stopped):

Structured log fired with real data:
fleet_health_check pass complete: agents=5 (running=1 stopped=4) healthy=0 degraded=0 unhealthy=3 critical=2 unknown=0 duration_ms=484

Stopped agent	Circuit	Outcome
guardrail-test	closed	✅ 4 fresh agent_health_checks rows (had none before)
my-new-agent	closed	✅ 4 fresh rows
another_test	dormant	no rows — #631 synthetic short-circuit (unmodified, by design)
pr-reviewer	dormant	no rows — same

Pre-fix, all 4 stopped agents were filtered out every cycle (only the 1 running agent checked) — the exact #675 root cause / production symptom. Post-fix, the non-dormant stopped agents now refresh correctly with "unhealthy/stopped" records.

The two dormant agents behaving as #631 designed (synthetic detail, no DB write — live aggregate status is still correctly unhealthy/dormant with last_check_at=now, only the historical rows lag) is the documented, intentional boundary — now confirmed empirically rather than just by analysis. Modifying it is explicitly out of #675 scope (reopens the SQLite contention #631 fixed).

[vybe]

Clean fix. Root cause clearly identified, minimal change, #631 dormant-skip explicitly preserved, 4 targeted regression tests. One note: consider updating the issue link from 'Related to' to 'Closes #675' if no further follow-up is planned on #675.