fix: fleet-restart lock — ownership-checked lease refresh + acquire inside try/finally (#1919)

[webmixgamer]

Summary

• The ops:fleet_restart per-iteration lease refresh was a bare EXPIRE gated only on a local flag: after a TTL lapse it would extend a concurrent caller's lease while both fleet loops ran — the exact dual-run the lock exists to prevent, silently. The refresh is now an ownership-checked pre-action gate (GET-compare via the new shared redis_breaker_util.lock_token_matches, same predicate as the release so the two can't drift).
• Foreign token ⇒ stop the run with honest partial accounting: summary/audit gain processed + stopped_early="lease_lost_foreign"; completed restarts stand; the release compare-and-delete still runs (foreign-safe by construction — skipping it on a false "absent" would self-lockout the fleet for the TTL). Absent token (incl. EXPIRE→0 in the GET→EXPIRE sliver) ⇒ SETNX re-acquire and continue — nothing is racing us, so a destructive half-done run completes instead of aborting; lease_reacquired is audited. Refresh Redis errors stay fail-open with one throttled warning per run.
• agents = list_all_agents_fast() moved inside the try/finally (pre-initialized for the finally-audit read), so nothing can ever sit between lock acquire and release; an abnormal exit audits as stopped_early="error" + exception class name only (fix: fleet restart adopts rebuilt base images through the canonical lifecycle path (#1860) #1912 exposure rule), distinguishing an aborted run from an empty fleet.
• TTL 900→2100, sized above the slowest single agent: the restart path runs skill injection, itself bounded at skill_service._INJECT_LOCK_TTL_SECONDS (1800s) — at 900s a mid-agent lapse was arithmetically guaranteed on the slow path, and detection alone would have shipped against a guaranteed trip. Constants are comment-linked so retuning either side re-opens the math visibly.

Scope notes

• The gate shrinks the dual-run window to at most one in-flight agent restart — it cannot eliminate it (per-agent start locks are refactor: per-agent start lock — complete the concurrent-start race hardening from #1809 #1817).
• stopped_early/processed are consumed by API/audit readers; the MobileAdmin PWA toast reads only a generic message today — client-side follow-up noted in mobile-admin-pwa.md.
• The same bug class lives in 6 sibling hand-rolled lock sites (worst: system_seed_service's unconditional tokenless release) — consolidation filed as bug: hand-rolled Redis single-flight locks — 7 divergent copies; system_seed_service releases with an unconditional tokenless delete #1920; this PR deliberately stays scoped to restart_fleet + the first shared helper.

Changes

• src/backend/routers/ops.py — ownership gate, re-acquire, TTL resize, acquire-inside-try, partial-run honesty fields (response summary + audit details)
• src/backend/redis_breaker_util.py — lock_token_matches (dual bytes/str, documented as belt-and-braces under decode_responses=True)
• tests/unit/test_1860_fleet_restart_adoption.py — +11 tests (27 total): 2-agent happy-path lease baseline, foreign stop + release-leaves-foreign, absent re-acquire win/lose, refresh-error scoped fail-open with release still deleting, EXPIRE→0 / EXPIRE-raise, first-iteration loss, bytes token, no-token-values-in-logs, listing-failure releases lock + error audit
• Docs: architecture.md Redis-catalog bullet, agent-lifecycle.md (single-flight section, tests line, changelog row, line refs re-verified), mobile-admin-pwa.md, feature-flows.md index row, 2 learnings.md entries (lease-TTL-vs-slowest-iteration; false-detection-vs-self-safe-cleanup)
• Security: /cso --diff PASS, zero findings — report pair committed under docs/security-reports/

Test Plan

• pytest tests/unit/test_1860_fleet_restart_adoption.py — 27/27
• Live suite tests/test_ops.py against the running stack — 41/41
• Adjacent test_dispatch_breaker.py + test_69_ephemeral_agents.py — 66/66
• Live validation on a real 4-agent stack: baseline (TTL 2098 observed mid-run, lock released), foreign-takeover mid-run (partial stop, foreign token untouched by release, warning + audit row), absent-lease mid-run (re-acquired, run completed, lease_reacquired:true audited), concurrent call → 409, fleet healthy after

Fixes #1919

🤖 Generated with Claude Code

[obasilakis]

Validated via /validate-pr: docs, security, packaging, test adequacy and lock-logic correctness all pass. All 21 CI checks green. Approving.