From 51eebebc614d122747ec5683e40e5008956cc24e Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 23 Jun 2026 11:05:39 -0700 Subject: [PATCH 01/23] docs(watchdog): add acquisition watchdog design spec and implementation plan Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/2026-06-23-acquisition-watchdog.md | 1779 +++++++++++++++++ .../2026-06-23-acquisition-watchdog-design.md | 230 +++ 2 files changed, 2009 insertions(+) create mode 100644 software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md create mode 100644 software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md diff --git a/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md b/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md new file mode 100644 index 000000000..f779ae259 --- /dev/null +++ b/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md @@ -0,0 +1,1779 @@ +# Acquisition Watchdog Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Detect when an acquisition ends prematurely — process crash/hang/kill, fatal error, or user abort — and send a single Slack alert, covering GUI- and MCP-server-driven runs on Ubuntu and Windows. + +**Architecture:** The acquisition engine (`control/core/`) drops on-disk breadcrumbs — a `run.json` written atomically at start, bumped with a throttled heartbeat during the run, and finalized with a reason at end. An independent, lightweight `acquisition_watchdog` process polls `run.json`, detects a dead/stale run or a non-clean end, and posts one Slack alert. A shared dependency-free `squid/slack.py` sender is reused by both the watchdog and the existing in-process `SlackNotifier`, whose end-of-run message is gated to clean successes so failures alert exactly once. + +**Tech Stack:** Python 3.8+, stdlib only (`json`, `urllib`, `configparser`, `socket`, `uuid`, `tempfile`, `os`), `platformdirs` (already a dep), `pyyaml` (already a dep), `pytest`. No new third-party dependencies. + +--- + +## Spec + +`docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md` + +## Reason taxonomy (v1) + +The worker computes one `reason` at the end of `run()`; it drives both the breadcrumb and the in-process finish message: + +| `reason` | When | Watchdog alerts? | Notifier finish msg? | +|---|---|---|---| +| `completed` | loop finished all timepoints, `_acquisition_error_count == 0`, not aborted | no | yes | +| `completed_with_errors` | loop finished but `_acquisition_error_count > 0` | yes | no | +| `error` | uncaught exception, or auto-abort from `TimeoutError` / failed-job abort | yes | no | +| `user_abort` | abort flag set externally (GUI/server) **or** GUI closed mid-run (shutdown aborts + joins) | yes | no | +| *(no end record)* | process crashed/killed/hung before writing end | yes (crash/hang) | n/a | + +(`app_closed` from the design is folded into `user_abort` for v1: the shutdown hook requests the normal abort and joins the worker so it writes a proper `user_abort` end record instead of looking like a crash. A distinct `app_closed` label is future work.) + +## File structure + +| Path | Responsibility | +|---|---| +| `squid/slack.py` | **New.** Dependency-free `post_message(bot_token, channel_id, text, blocks)` via `urllib`. Reused by notifier + watchdog. | +| `squid/acquisition_state.py` | **New.** `run.json` schema, `default_state_dir()`, atomic write, `read_run()`, `RunStateWriter` (+ `NullRunStateWriter`). Engine writes; watchdog reads. Leaf module — must not import `control`. | +| `acquisition_watchdog/__init__.py` | **New.** Package marker. | +| `acquisition_watchdog/config.py` | **New.** Resolve active `.ini`; load `[SlackNotifications]` with stdlib `configparser`. | +| `acquisition_watchdog/alerts.py` | **New.** Format the Slack alert text + blocks for each alert kind. | +| `acquisition_watchdog/monitor.py` | **New.** `pid_alive`, `Monitor.classify`, `Monitor.check_once`, dedup persistence, `run_forever`. | +| `acquisition_watchdog/__main__.py` | **New.** CLI entry: `python -m acquisition_watchdog`. | +| `acquisition_watchdog/systemd/squid-acquisition-watchdog.service` | **New.** Linux user-service unit. | +| `acquisition_watchdog/windows/squid-acquisition-watchdog.xml`, `install.ps1` | **New.** Windows Task Scheduler recipe. | +| `acquisition_watchdog/README.md` | **New.** Install/run docs for both OSes. | +| `control/slack_notifier.py` | **Modify.** Delegate `_post_message` to `squid.slack`; add `reason` field to `AcquisitionStats`; gate `notify_acquisition_finished` to `reason == "completed"`. | +| `control/core/multi_point_controller.py` | **Modify.** Write the start breadcrumb in `run_acquisition()`; pass the writer to the worker. | +| `control/core/multi_point_worker.py` | **Modify.** Heartbeat in the loop + image callback; compute `reason` and write end in `finally`; track `_abort_cause`. | +| `main_hcs.py` | **Modify.** On shutdown-while-acquiring, request abort + join so the worker writes `user_abort`. | +| `tests/...` | **New/modify.** Unit + integration tests per task; autouse fixture redirecting the state dir to tmp. | + +--- + +## Task 1: `squid/slack.py` — shared Slack sender + +**Files:** +- Create: `squid/slack.py` +- Test: `tests/squid/test_slack.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/squid/test_slack.py +import json +from unittest.mock import patch, MagicMock + +import squid.slack as slack + + +def test_post_message_returns_false_without_credentials(): + assert slack.post_message(None, "C123", "hi") == (False, None) + assert slack.post_message("xoxb-1", None, "hi") == (False, None) + + +def test_post_message_builds_authorized_request_and_parses_ok(): + captured = {} + + class FakeResp: + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def read(self): + return json.dumps({"ok": True, "ts": "111.222"}).encode() + + def fake_urlopen(request, timeout=15): + captured["url"] = request.full_url + captured["headers"] = request.headers + captured["body"] = json.loads(request.data.decode()) + return FakeResp() + + with patch("urllib.request.urlopen", side_effect=fake_urlopen): + ok, ts = slack.post_message("xoxb-token", "C123", "hello", blocks=[{"type": "section"}]) + + assert ok is True and ts == "111.222" + assert captured["url"].endswith("/chat.postMessage") + assert captured["headers"]["Authorization"] == "Bearer xoxb-token" + assert captured["body"]["channel"] == "C123" + assert captured["body"]["text"] == "hello" + assert captured["body"]["blocks"] == [{"type": "section"}] + + +def test_post_message_handles_api_error(): + class FakeResp: + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def read(self): + return json.dumps({"ok": False, "error": "channel_not_found"}).encode() + + with patch("urllib.request.urlopen", return_value=FakeResp()): + ok, ts = slack.post_message("xoxb", "C1", "x") + assert ok is False and ts is None +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/squid/test_slack.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'squid.slack'`. + +- [ ] **Step 3: Write the implementation** + +```python +# squid/slack.py +"""Dependency-free Slack chat.postMessage sender. + +Shared by the in-process SlackNotifier (control/slack_notifier.py) and the +standalone acquisition watchdog. Stdlib only — safe to import without the +control/Qt/hardware stack. +""" +import json +import urllib.error +import urllib.request +from typing import Optional, Tuple + +import squid.logging + +_log = squid.logging.get_logger(__name__) + +SLACK_API_BASE = "https://slack.com/api" + + +def post_message( + bot_token: Optional[str], + channel_id: Optional[str], + text: str, + blocks: Optional[list] = None, + timeout: float = 15.0, +) -> Tuple[bool, Optional[str]]: + """Post a message to Slack. Returns (ok, message_ts).""" + if not bot_token or not channel_id: + _log.debug("No Slack bot token or channel configured") + return False, None + + payload = {"channel": channel_id, "text": text} + if blocks: + payload["blocks"] = blocks + data = json.dumps(payload).encode("utf-8") + request = urllib.request.Request( + f"{SLACK_API_BASE}/chat.postMessage", + data=data, + headers={ + "Content-Type": "application/json; charset=utf-8", + "Authorization": f"Bearer {bot_token}", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + result = json.loads(response.read().decode("utf-8")) + if result.get("ok"): + return True, result.get("ts") + _log.warning(f"Slack API error: {result.get('error')}") + return False, None + except urllib.error.URLError as e: + _log.warning(f"Failed to send Slack message: {e}") + return False, None + except Exception as e: + _log.warning(f"Unexpected error sending Slack message: {e}") + return False, None +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/squid/test_slack.py -v` +Expected: PASS (3 tests). Create `tests/squid/__init__.py` if the package import fails. + +- [ ] **Step 5: Commit** + +```bash +git add software/squid/slack.py software/tests/squid/test_slack.py +git commit -m "feat(slack): add dependency-free squid.slack.post_message sender" +``` + +--- + +## Task 2: Delegate `SlackNotifier._post_message` to `squid.slack` + +**Files:** +- Modify: `control/slack_notifier.py` (`_post_message`, lines 160–212; imports lines 10–27) +- Test: `tests/control/test_slack_notifier_send.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/control/test_slack_notifier_send.py +from unittest.mock import patch +from control.slack_notifier import SlackNotifier + + +def test_post_message_delegates_to_squid_slack(): + n = SlackNotifier(bot_token="xoxb-abc", channel_id="C999") + with patch("squid.slack.post_message", return_value=(True, "1.0")) as m: + ok, ts = n._post_message("hello", blocks=[{"type": "section"}]) + assert ok is True and ts == "1.0" + m.assert_called_once_with("xoxb-abc", "C999", "hello", [{"type": "section"}]) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/control/test_slack_notifier_send.py -v` +Expected: FAIL — `_post_message` calls `urllib` directly, so `squid.slack.post_message` is never called (`AssertionError: Expected 'post_message' to have been called once`). + +- [ ] **Step 3: Edit `control/slack_notifier.py`** + +Add the import near the existing `import squid.logging` (line 27): + +```python +import squid.logging +import squid.slack +``` + +Replace the entire `_post_message` method (lines 160–212) with: + +```python + def _post_message(self, text: str, blocks: Optional[list] = None) -> Tuple[bool, Optional[str]]: + return squid.slack.post_message(self.bot_token, self.channel_id, text, blocks) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/control/test_slack_notifier_send.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add software/control/slack_notifier.py software/tests/control/test_slack_notifier_send.py +git commit -m "refactor(slack): route SlackNotifier sends through squid.slack" +``` + +--- + +## Task 3: `squid/acquisition_state.py` — breadcrumb schema + writer + +**Files:** +- Create: `squid/acquisition_state.py` +- Test: `tests/squid/test_acquisition_state.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/squid/test_acquisition_state.py +import json + +import squid.acquisition_state as ast + + +def _expected(): + return {"timepoints": 3, "regions": 1, "fovs": 4, "channels": 2, "z": 1} + + +def test_start_writes_running_record(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="exp1", pid=4321, config_path="/cfg.ini", + output_path=str(tmp_path / "exp1"), expected=_expected(), + machine="micro-1", state_dir=tmp_path, + ) + rec = ast.read_run(tmp_path) + assert rec["status"] == "running" + assert rec["experiment_id"] == "exp1" + assert rec["pid"] == 4321 + assert rec["machine"] == "micro-1" + assert rec["expected"] == _expected() + assert rec["run_id"] == w.run_id + assert rec["reason"] is None + + +def test_beat_is_throttled_but_updates_progress_on_flush(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="e", pid=1, config_path=None, output_path="o", + expected=_expected(), state_dir=tmp_path, + ) + first = ast.read_run(tmp_path)["heartbeat_at"] + # Immediate beat is throttled (< HEARTBEAT_INTERVAL_S since start) -> file unchanged. + w.beat({"timepoint": 1}) + assert ast.read_run(tmp_path)["heartbeat_at"] == first + # Forced beat flushes and records progress. + w.beat({"timepoint": 2}, force=True) + rec = ast.read_run(tmp_path) + assert rec["heartbeat_at"] >= first + assert rec["progress"] == {"timepoint": 2} + + +def test_end_records_reason_and_stats(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="e", pid=1, config_path=None, output_path="o", + expected=_expected(), state_dir=tmp_path, + ) + w.end("user_abort", {"total_images": 7, "errors_encountered": 0}) + rec = ast.read_run(tmp_path) + assert rec["status"] == "ended" + assert rec["reason"] == "user_abort" + assert rec["ended_at"] is not None + assert rec["stats"]["total_images"] == 7 + + +def test_read_run_missing_returns_none(tmp_path): + assert ast.read_run(tmp_path) is None + + +def test_null_writer_is_noop(tmp_path): + w = ast.NullRunStateWriter() + w.beat({"timepoint": 1}) + w.end("completed", {}) + assert ast.read_run(tmp_path) is None + assert w.run_id is None +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/squid/test_acquisition_state.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'squid.acquisition_state'`. + +- [ ] **Step 3: Write the implementation** + +```python +# squid/acquisition_state.py +"""On-disk acquisition run-state breadcrumbs, shared by the acquisition engine +(writer) and the standalone acquisition watchdog (reader). + +Stdlib-only leaf module: must NOT import anything from `control`. +""" +import json +import os +import socket +import tempfile +import time +import uuid +from pathlib import Path +from typing import Optional + +import platformdirs + +import squid.logging + +_log = squid.logging.get_logger(__name__) + +SCHEMA_VERSION = 1 +HEARTBEAT_INTERVAL_S = 5.0 +RUN_FILE_NAME = "run.json" + + +def default_state_dir() -> Path: + """Per-user watchdog state dir, shared by writer and reader. + + Overridable via SQUID_WATCHDOG_STATE_DIR (honored by both processes). + """ + override = os.environ.get("SQUID_WATCHDOG_STATE_DIR") + if override: + return Path(override) + return Path(platformdirs.user_state_path("squid", "cephla")) / "watchdog" + + +def run_file_path(state_dir: Optional[Path] = None) -> Path: + return Path(state_dir) / RUN_FILE_NAME if state_dir else default_state_dir() / RUN_FILE_NAME + + +def _atomic_write_json(path: Path, data: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".run-", suffix=".tmp") + try: + with os.fdopen(fd, "w") as f: + json.dump(data, f) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) # atomic on POSIX and Windows + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise + + +def read_run(state_dir: Optional[Path] = None) -> Optional[dict]: + try: + with open(run_file_path(state_dir)) as f: + return json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return None + + +class RunStateWriter: + """Writes/updates the single run.json for the current acquisition.""" + + def __init__(self, record: dict, state_dir: Optional[Path] = None): + self._record = record + self._state_dir = state_dir + self._last_beat = 0.0 + + @classmethod + def start( + cls, + *, + experiment_id: str, + pid: int, + config_path: Optional[str], + output_path: str, + expected: dict, + machine: Optional[str] = None, + state_dir: Optional[Path] = None, + ) -> "RunStateWriter": + now = time.time() + record = { + "schema_version": SCHEMA_VERSION, + "run_id": uuid.uuid4().hex, + "experiment_id": experiment_id, + "machine": machine or socket.gethostname(), + "pid": pid, + "config_path": config_path, + "output_path": output_path, + "started_at": now, + "heartbeat_at": now, + "progress": {}, + "expected": expected, + "status": "running", + "reason": None, + "ended_at": None, + "stats": None, + } + writer = cls(record, state_dir=state_dir) + writer._flush() + writer._last_beat = now + return writer + + @property + def run_id(self) -> Optional[str]: + return self._record.get("run_id") + + def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: + if progress: + self._record["progress"] = progress + now = time.time() + if not force and (now - self._last_beat) < HEARTBEAT_INTERVAL_S: + return + self._last_beat = now + self._record["heartbeat_at"] = now + self._flush() + + def end(self, reason: str, stats: Optional[dict] = None) -> None: + self._record["status"] = "ended" + self._record["reason"] = reason + self._record["ended_at"] = time.time() + if stats is not None: + self._record["stats"] = stats + self._flush() + + def _flush(self) -> None: + try: + _atomic_write_json(run_file_path(self._state_dir), dict(self._record)) + except OSError as e: + _log.warning(f"Failed to write acquisition run state: {e}") + + +class NullRunStateWriter: + """No-op writer used when breadcrumbs are not wired (tests, side paths).""" + + run_id = None + + def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: + pass + + def end(self, reason: str, stats: Optional[dict] = None) -> None: + pass +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/squid/test_acquisition_state.py -v` +Expected: PASS (5 tests). + +- [ ] **Step 5: Commit** + +```bash +git add software/squid/acquisition_state.py software/tests/squid/test_acquisition_state.py +git commit -m "feat(watchdog): add squid.acquisition_state breadcrumb schema + writer" +``` + +--- + +## Task 4: `acquisition_watchdog/config.py` — config resolution + +**Files:** +- Create: `acquisition_watchdog/__init__.py` (empty) +- Create: `acquisition_watchdog/config.py` +- Test: `tests/acquisition_watchdog/test_config.py` (+ `tests/acquisition_watchdog/__init__.py`) + +- [ ] **Step 1: Write the failing test** + +```python +# tests/acquisition_watchdog/test_config.py +from acquisition_watchdog import config as wdconfig + + +def _write_ini(path, body): + path.write_text(body) + return path + + +def test_resolve_prefers_cli_then_env_then_run_record(tmp_path, monkeypatch): + monkeypatch.delenv("SQUID_CONFIG", raising=False) + assert wdconfig.resolve_config_path("/cli.ini", {"config_path": "/run.ini"}) == __import__("pathlib").Path("/cli.ini") + monkeypatch.setenv("SQUID_CONFIG", "/env.ini") + assert str(wdconfig.resolve_config_path(None, {"config_path": "/run.ini"})) == "/env.ini" + monkeypatch.delenv("SQUID_CONFIG", raising=False) + assert str(wdconfig.resolve_config_path(None, {"config_path": "/run.ini"})) == "/run.ini" + + +def test_load_slack_config_reads_section(tmp_path): + ini = _write_ini( + tmp_path / "c.ini", + "[SLACKNOTIFICATIONS]\nenabled = True\nbot_token = xoxb-xyz\nchannel_id = C42\nwatchdog_enabled = True\n", + ) + cfg = wdconfig.load_slack_config(ini) + assert cfg.enabled is True + assert cfg.bot_token == "xoxb-xyz" + assert cfg.channel_id == "C42" + assert cfg.watchdog_enabled is True + + +def test_load_slack_config_defaults_when_missing(tmp_path): + ini = _write_ini(tmp_path / "c.ini", "[GENERAL]\nfoo = 1\n") + cfg = wdconfig.load_slack_config(ini) + assert cfg.bot_token is None and cfg.channel_id is None + assert cfg.watchdog_enabled is True # defaults to on when section absent + + assert wdconfig.load_slack_config(None).bot_token is None +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_config.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog'`. + +- [ ] **Step 3: Write the implementation** + +```python +# acquisition_watchdog/__init__.py +``` + +```python +# acquisition_watchdog/config.py +"""Resolve the active Squid .ini and read its [SlackNotifications] section, +without importing the heavy control._def module. +""" +import configparser +import os +from pathlib import Path +from typing import NamedTuple, Optional + + +class SlackConfig(NamedTuple): + enabled: bool + bot_token: Optional[str] + channel_id: Optional[str] + watchdog_enabled: bool + + +def resolve_config_path(cli_config: Optional[str], run_record: Optional[dict]) -> Optional[Path]: + """Priority: --config > $SQUID_CONFIG > run.json config_path > cache pointer.""" + if cli_config: + return Path(cli_config) + env = os.environ.get("SQUID_CONFIG") + if env: + return Path(env) + if run_record and run_record.get("config_path"): + return Path(run_record["config_path"]) + cache = Path("cache/config_file_path.txt") + if cache.exists(): + first = cache.read_text().splitlines() + if first: + return Path(first[0].strip()) + return None + + +def load_slack_config(config_path: Optional[Path]) -> SlackConfig: + if not config_path or not Path(config_path).exists(): + return SlackConfig(False, None, None, True) + cp = configparser.ConfigParser() + try: + cp.read(config_path) + except configparser.Error: + return SlackConfig(False, None, None, True) + if not cp.has_section("SLACKNOTIFICATIONS"): + return SlackConfig(False, None, None, True) + sec = cp["SLACKNOTIFICATIONS"] + + def getbool(key: str, default: bool) -> bool: + try: + return sec.getboolean(key, default) + except ValueError: + return default + + token = sec.get("bot_token", fallback=None) or None + channel = sec.get("channel_id", fallback=None) or None + return SlackConfig( + enabled=getbool("enabled", False), + bot_token=token, + channel_id=channel, + watchdog_enabled=getbool("watchdog_enabled", True), + ) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_config.py -v` +Expected: PASS (3 tests). Add empty `tests/acquisition_watchdog/__init__.py` if needed. + +- [ ] **Step 5: Commit** + +```bash +git add software/acquisition_watchdog/__init__.py software/acquisition_watchdog/config.py software/tests/acquisition_watchdog/ +git commit -m "feat(watchdog): add config resolution + [SlackNotifications] loader" +``` + +--- + +## Task 5: `acquisition_watchdog/alerts.py` — alert formatting + +**Files:** +- Create: `acquisition_watchdog/alerts.py` +- Test: `tests/acquisition_watchdog/test_alerts.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/acquisition_watchdog/test_alerts.py +from acquisition_watchdog import alerts + + +def _run(): + return { + "experiment_id": "plateA_2026", + "machine": "micro-1", + "output_path": "/data/plateA_2026", + "progress": {"timepoint": 3, "expected_timepoints": 10, "images": 360}, + "expected": {"timepoints": 10}, + "started_at": 1_700_000_000.0, + "heartbeat_at": 1_700_000_100.0, + } + + +def test_format_alert_includes_key_facts(): + text, blocks = alerts.format_alert("crash", _run()) + assert "plateA_2026" in text + assert "micro-1" in text + blob = str(blocks) + assert "plateA_2026" in blob + assert "3" in blob and "10" in blob # progress vs expected + assert isinstance(blocks, list) and blocks + + +def test_format_alert_each_kind_has_title(): + for kind in ("crash", "hang", "error", "completed_with_errors", "user_abort"): + text, _ = alerts.format_alert(kind, _run()) + assert text # non-empty title line for every kind +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_alerts.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.alerts'`. + +- [ ] **Step 3: Write the implementation** + +```python +# acquisition_watchdog/alerts.py +"""Format watchdog Slack alerts (text + Block Kit blocks).""" +from datetime import datetime, timezone +from typing import Optional, Tuple + +_KIND_TITLE = { + "crash": ":red_circle: Acquisition process died", + "hang": ":large_orange_circle: Acquisition hung (no heartbeat)", + "error": ":red_circle: Acquisition ended with a fatal error", + "completed_with_errors": ":large_orange_circle: Acquisition finished with errors", + "user_abort": ":large_yellow_circle: Acquisition aborted", +} + + +def _fmt_ts(epoch: Optional[float]) -> str: + if not epoch: + return "unknown" + return datetime.fromtimestamp(epoch, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC") + + +def _progress_line(run: dict) -> str: + prog = run.get("progress") or {} + expected = run.get("expected") or {} + tp = prog.get("timepoint", "?") + exp_tp = prog.get("expected_timepoints", expected.get("timepoints", "?")) + images = prog.get("images", "?") + return f"timepoint {tp}/{exp_tp}, {images} images" + + +def format_alert(kind: str, run: dict) -> Tuple[str, list]: + title = _KIND_TITLE.get(kind, f"Acquisition alert: {kind}") + experiment = run.get("experiment_id", "unknown") + machine = run.get("machine", "unknown") + text = f"{title}: {experiment} on {machine}" + + last_seen = run.get("ended_at") or run.get("heartbeat_at") + detail = ( + f"*Experiment:* {experiment}\n" + f"*Machine:* {machine}\n" + f"*Progress:* {_progress_line(run)}\n" + f"*Started:* {_fmt_ts(run.get('started_at'))}\n" + f"*Last seen:* {_fmt_ts(last_seen)}\n" + f"*Output:* {run.get('output_path', 'unknown')}" + ) + blocks = [ + {"type": "header", "text": {"type": "plain_text", "text": title.replace(":red_circle:", "") + .replace(":large_orange_circle:", "").replace(":large_yellow_circle:", "").strip(), + "emoji": True}}, + {"type": "section", "text": {"type": "mrkdwn", "text": detail}}, + ] + return text, blocks +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_alerts.py -v` +Expected: PASS (2 tests). + +- [ ] **Step 5: Commit** + +```bash +git add software/acquisition_watchdog/alerts.py software/tests/acquisition_watchdog/test_alerts.py +git commit -m "feat(watchdog): add Slack alert formatting" +``` + +--- + +## Task 6: `acquisition_watchdog/monitor.py` — poll, classify, dedup + +**Files:** +- Create: `acquisition_watchdog/monitor.py` +- Test: `tests/acquisition_watchdog/test_monitor.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/acquisition_watchdog/test_monitor.py +import time + +import squid.acquisition_state as ast +from acquisition_watchdog.monitor import Monitor + + +def _running(tmp_path, pid, heartbeat_age=0.0, run_id="r1"): + rec = { + "schema_version": 1, "run_id": run_id, "experiment_id": "e", "machine": "m", + "pid": pid, "config_path": None, "output_path": "o", + "started_at": time.time() - 100, "heartbeat_at": time.time() - heartbeat_age, + "progress": {}, "expected": {}, "status": "running", "reason": None, + "ended_at": None, "stats": None, + } + ast._atomic_write_json(ast.run_file_path(tmp_path), rec) + return rec + + +def _mon(tmp_path): + return Monitor(state_dir=tmp_path, heartbeat_timeout=120.0) + + +def test_running_with_live_pid_and_fresh_heartbeat_is_silent(tmp_path): + _running(tmp_path, pid=__import__("os").getpid(), heartbeat_age=1.0) + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) is None + + +def test_dead_pid_is_crash(tmp_path): + _running(tmp_path, pid=2_000_000_000, heartbeat_age=1.0) # impossible pid + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "crash" + + +def test_stale_heartbeat_with_live_pid_is_hang(tmp_path): + _running(tmp_path, pid=__import__("os").getpid(), heartbeat_age=999.0) + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "hang" + + +def test_ended_reasons(tmp_path): + mon = _mon(tmp_path) + for reason, expect in [ + ("completed", None), ("completed_with_errors", "completed_with_errors"), + ("error", "error"), ("user_abort", "user_abort"), + ]: + run = {"run_id": f"x-{reason}", "status": "ended", "reason": reason} + assert mon.classify(run, time.time()) == expect + + +def test_dedup_persists_across_restart(tmp_path, monkeypatch): + _running(tmp_path, pid=2_000_000_000, run_id="dup1") + sent = [] + monkeypatch.setattr("squid.slack.post_message", lambda *a, **k: (sent.append(a) or (True, "1"))) + monkeypatch.setattr( + "acquisition_watchdog.config.load_slack_config", + lambda p: __import__("acquisition_watchdog.config", fromlist=["SlackConfig"]).SlackConfig(True, "xoxb", "C1", True), + ) + Monitor(state_dir=tmp_path).check_once(time.time()) + assert len(sent) == 1 + # Fresh Monitor (simulated restart) must not re-alert the same run_id. + Monitor(state_dir=tmp_path).check_once(time.time()) + assert len(sent) == 1 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_monitor.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.monitor'`. + +- [ ] **Step 3: Write the implementation** + +```python +# acquisition_watchdog/monitor.py +"""Poll the acquisition run-state and alert on premature ends.""" +import json +import os +import time +from pathlib import Path +from typing import Optional, Set + +import squid.acquisition_state as acquisition_state +import squid.logging +import squid.slack +from acquisition_watchdog import alerts, config + +_log = squid.logging.get_logger("acquisition_watchdog") + +ALERT_REASONS = {"completed_with_errors", "error", "user_abort"} + + +def pid_alive(pid: Optional[int]) -> bool: + if not pid: + return False + try: + import psutil + + return psutil.pid_exists(pid) + except ImportError: + pass + if os.name == "posix": + try: + os.kill(pid, 0) + return True + except ProcessLookupError: + return False + except PermissionError: + return True + except OSError: + return False + # Windows without psutil: cannot check reliably; rely on the heartbeat instead. + return True + + +class Monitor: + def __init__( + self, + state_dir: Optional[Path] = None, + cli_config: Optional[str] = None, + poll_interval: float = 5.0, + heartbeat_timeout: float = 120.0, + ): + self._state_dir = Path(state_dir) if state_dir else None + self._cli_config = cli_config + self._poll = poll_interval + self._timeout = heartbeat_timeout + base = self._state_dir or acquisition_state.default_state_dir() + self._alerted_path = base / "alerted.json" + self._alerted = self._load_alerted() + + def _load_alerted(self) -> Set[str]: + try: + with open(self._alerted_path) as f: + return set(json.load(f)) + except (FileNotFoundError, json.JSONDecodeError): + return set() + + def _save_alerted(self) -> None: + try: + self._alerted_path.parent.mkdir(parents=True, exist_ok=True) + with open(self._alerted_path, "w") as f: + json.dump(sorted(self._alerted), f) + except OSError as e: + _log.warning(f"Could not persist alerted set: {e}") + + def classify(self, run: Optional[dict], now: float) -> Optional[str]: + """Return an alert kind ('crash'|'hang'|) or None.""" + if not run or run.get("run_id") in self._alerted: + return None + status = run.get("status") + if status == "running": + if not pid_alive(run.get("pid")): + return "crash" + if (now - (run.get("heartbeat_at") or 0)) > self._timeout: + return "hang" + return None + if status == "ended" and run.get("reason") in ALERT_REASONS: + return run["reason"] + return None + + def check_once(self, now: float) -> None: + run = acquisition_state.read_run(self._state_dir) + kind = self.classify(run, now) + if kind is None: + return + + cfg_path = config.resolve_config_path(self._cli_config, run) + slack_cfg = config.load_slack_config(cfg_path) + if not (slack_cfg.bot_token and slack_cfg.channel_id and slack_cfg.watchdog_enabled): + _log.warning( + f"Premature end ({kind}) for run_id={run.get('run_id')} but Slack is not " + f"configured/enabled; not alerting." + ) + self._mark_alerted(run["run_id"]) + return + + text, blocks = alerts.format_alert(kind, run) + ok, _ = squid.slack.post_message(slack_cfg.bot_token, slack_cfg.channel_id, text, blocks) + if ok: + _log.info(f"Sent watchdog alert ({kind}) for run_id={run.get('run_id')}") + self._mark_alerted(run["run_id"]) + else: + # Leave unmarked so a transient Slack failure retries on the next poll. + _log.warning(f"Failed to send watchdog alert ({kind}) for run_id={run.get('run_id')}; will retry") + + def _mark_alerted(self, run_id: str) -> None: + self._alerted.add(run_id) + self._save_alerted() + + def run_forever(self) -> None: + base = self._state_dir or acquisition_state.default_state_dir() + _log.info(f"Acquisition watchdog started. state_dir={base} heartbeat_timeout={self._timeout}s") + while True: + try: + self.check_once(time.time()) + except Exception as e: + _log.exception(f"Watchdog poll error: {e}") + time.sleep(self._poll) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_monitor.py -v` +Expected: PASS (5 tests). + +- [ ] **Step 5: Commit** + +```bash +git add software/acquisition_watchdog/monitor.py software/tests/acquisition_watchdog/test_monitor.py +git commit -m "feat(watchdog): add poll/classify/dedup monitor" +``` + +--- + +## Task 7: `acquisition_watchdog/__main__.py` — CLI entry + +**Files:** +- Create: `acquisition_watchdog/__main__.py` +- Test: `tests/acquisition_watchdog/test_cli.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/acquisition_watchdog/test_cli.py +from unittest.mock import patch + +from acquisition_watchdog.__main__ import main + + +def test_once_runs_single_check(tmp_path): + with patch("acquisition_watchdog.monitor.Monitor.check_once") as check, \ + patch("acquisition_watchdog.monitor.Monitor.run_forever") as forever: + main(["--once", "--state-dir", str(tmp_path)]) + check.assert_called_once() + forever.assert_not_called() + + +def test_default_runs_forever(tmp_path): + with patch("acquisition_watchdog.monitor.Monitor.run_forever") as forever: + main(["--state-dir", str(tmp_path)]) + forever.assert_called_once() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_cli.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.__main__'`. + +- [ ] **Step 3: Write the implementation** + +```python +# acquisition_watchdog/__main__.py +"""CLI entry point: python -m acquisition_watchdog""" +import argparse +import time +from pathlib import Path +from typing import Optional, Sequence + +import squid.logging +from acquisition_watchdog.monitor import Monitor + + +def main(argv: Optional[Sequence[str]] = None) -> None: + parser = argparse.ArgumentParser( + prog="acquisition_watchdog", + description="Alert on prematurely-ended Squid acquisitions (crash/hang/abort/error).", + ) + parser.add_argument("--config", help="Path to the active configuration .ini ([SlackNotifications]).") + parser.add_argument("--state-dir", help="Override the watchdog state directory.") + parser.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between checks (default 5).") + parser.add_argument( + "--heartbeat-timeout", type=float, default=120.0, + help="Seconds of heartbeat silence (with a live PID) before declaring a hang (default 120).", + ) + parser.add_argument("--once", action="store_true", help="Run a single check and exit.") + args = parser.parse_args(argv) + + log = squid.logging.get_logger("acquisition_watchdog") + monitor = Monitor( + state_dir=Path(args.state_dir) if args.state_dir else None, + cli_config=args.config, + poll_interval=args.poll_interval, + heartbeat_timeout=args.heartbeat_timeout, + ) + if args.once: + monitor.check_once(time.time()) + else: + try: + monitor.run_forever() + except KeyboardInterrupt: + log.info("Acquisition watchdog stopped.") + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_cli.py -v` +Expected: PASS (2 tests). + +- [ ] **Step 5: Commit** + +```bash +git add software/acquisition_watchdog/__main__.py software/tests/acquisition_watchdog/test_cli.py +git commit -m "feat(watchdog): add CLI entry point" +``` + +--- + +## Task 8: Engine — write the start breadcrumb in `run_acquisition()` + +**Files:** +- Modify: `control/core/multi_point_controller.py` (`run_acquisition`, around lines 838–888) +- Modify: `control/core/multi_point_worker.py` (`__init__`, lines 66–110) +- Modify: `tests/control/conftest.py` (add autouse fixture redirecting state dir to tmp) + +- [ ] **Step 1: Add the autouse fixture so tests never touch the real state dir** + +Append to `tests/control/conftest.py`: + +```python +import pytest + + +@pytest.fixture(autouse=True) +def _watchdog_state_to_tmp(tmp_path, monkeypatch): + # Keep acquisition breadcrumbs out of the real user state dir during tests. + monkeypatch.setenv("SQUID_WATCHDOG_STATE_DIR", str(tmp_path / "watchdog")) +``` + +- [ ] **Step 2: Add `run_state_writer` param to `MultiPointWorker.__init__`** + +In `control/core/multi_point_worker.py`, add to the `__init__` signature (after `prewarmed_bp_values`, line 83): + +```python + prewarmed_bp_values: Optional["BackpressureValues"] = None, + run_state_writer=None, + ): +``` + +Add the import near the top of the file (with the other `control`/`squid` imports): + +```python +import squid.acquisition_state +``` + +Store it among the other attribute assignments (near line 110, after `self.request_abort_fn = request_abort_fn`): + +```python + self._run_state = run_state_writer or squid.acquisition_state.NullRunStateWriter() + self._abort_cause = None # set to "error" by auto-abort paths (timeout / failed jobs) +``` + +- [ ] **Step 3: Write the start breadcrumb and pass the writer (controller)** + +In `control/core/multi_point_controller.py`, add near the top with the other imports: + +```python +import squid.acquisition_state +``` + +In `run_acquisition()`, immediately AFTER the `_save_acquisition_yaml(...)` call block (ends ~line 865) and BEFORE `prewarmed_runner, prewarmed_bp_values = self.get_prewarmed_job_runner()` (line 869), insert: + +```python + # Acquisition watchdog: drop the "running" breadcrumb (covers GUI + MCP-server runs). + self._run_state_writer = squid.acquisition_state.NullRunStateWriter() + try: + expected = { + "timepoints": self.Nt, + "regions": len(scan_position_information.scan_region_coords_mm), + "fovs": sum(len(c) for c in scan_position_information.scan_region_fov_coords_mm.values()), + "channels": len(self.selected_configurations), + "z": self.NZ, + } + config_path = (getattr(control._def, "CACHED_CONFIG_FILE_PATH", None) or "").strip() or None + self._run_state_writer = squid.acquisition_state.RunStateWriter.start( + experiment_id=self.experiment_ID, + pid=os.getpid(), + config_path=config_path, + output_path=experiment_path, + expected=expected, + ) + except Exception as e: + self._log.warning(f"Failed to write acquisition watchdog start state: {e}") +``` + +Then add `run_state_writer=self._run_state_writer,` to the `MultiPointWorker(...)` constructor call (within the kwargs block at lines 873–888): + +```python + prewarmed_bp_values=prewarmed_bp_values, + run_state_writer=self._run_state_writer, + ) +``` + +(`os` and `control._def` are already imported in this module; confirm with `grep -n "^import os" software/control/core/multi_point_controller.py` and add `import os` if absent.) + +- [ ] **Step 4: Smoke-test the wiring** + +```python +# tests/control/test_watchdog_breadcrumbs.py +import os + +import squid.acquisition_state as ast +import control.microscope +import tests.control.gui_test_stubs as gts + + +def test_run_acquisition_writes_running_breadcrumb(qtbot): + scope = control.microscope.Microscope.build_from_global_config(True) + mpc = gts.get_test_qt_multi_point_controller(microscope=scope) + mpc.run_acquisition() + rec = ast.read_run(os.environ["SQUID_WATCHDOG_STATE_DIR"]) + assert rec is not None + assert rec["status"] == "running" + assert rec["pid"] == os.getpid() + assert rec["expected"]["timepoints"] >= 1 + mpc.request_abort_aquisition() + scope.close() +``` + +Run: `cd software && python3 -m pytest tests/control/test_watchdog_breadcrumbs.py -v` +Expected: PASS (start breadcrumb present). The `ended` transition is verified in Task 12. + +- [ ] **Step 5: Commit** + +```bash +git add software/control/core/multi_point_controller.py software/control/core/multi_point_worker.py software/tests/control/conftest.py software/tests/control/test_watchdog_breadcrumbs.py +git commit -m "feat(watchdog): write acquisition start breadcrumb from the engine" +``` + +--- + +## Task 9: Engine — heartbeat, reason, and end breadcrumb in the worker + +**Files:** +- Modify: `control/core/multi_point_worker.py` (`run`, lines 449–539; `_image_callback` ~line 1200; failed-job path lines 1023–1027) +- Modify: `control/slack_notifier.py` (`AcquisitionStats`, lines 46–53) + +- [ ] **Step 1: Add `reason` field to `AcquisitionStats`** + +In `control/slack_notifier.py`, extend the dataclass (lines 46–53): + +```python +@dataclass +class AcquisitionStats: + """Statistics for a completed acquisition.""" + + total_images: int + total_timepoints: int + total_duration_seconds: float + errors_encountered: int + experiment_id: str + reason: str = "completed" +``` + +- [ ] **Step 2: Add a heartbeat helper + loop beats (worker)** + +In `control/core/multi_point_worker.py`, add a helper method to `MultiPointWorker`: + +```python + def _run_state_beat(self) -> None: + self._run_state.beat( + { + "timepoint": self.time_point, + "expected_timepoints": self.Nt, + "fov": self._timepoint_fov_count, + "images": self.image_count, + } + ) +``` + +Insert `self._run_state_beat()` at three points in `run()`: + +(a) Right after the top-of-loop abort check (after line 453 `break`), as the first statement of the loop body when not aborting: + +```python + while self.time_point < self.Nt: + # check if abort acquisition has been requested + if self.abort_requested_fn(): + self._log.debug("In run, abort_acquisition_requested=True") + break + self._run_state_beat() +``` + +(b) Inside the timed-acquisition wait loop (lines 494–498), so dt gaps keep the heartbeat fresh: + +```python + while time.time() < self.timestamp_acquisition_started + self.time_point * self.dt: + if self.abort_requested_fn(): + self._log.debug("In run wait loop, abort_acquisition_requested=True") + break + self._run_state_beat() + self._sleep(sleep_time) +``` + +(c) In `_image_callback`, immediately after `self.image_count` is incremented (~line 1200), so long single-timepoint scans keep beating with real imaging progress: + +```python + self.image_count += 1 + self._run_state_beat() +``` + +- [ ] **Step 3: Tag error-driven aborts** + +In the `except TimeoutError` handler (lines 507–510), set the cause before requesting abort: + +```python + except TimeoutError as te: + self._log.error(f"Operation timed out during acquisition, aborting acquisition!") + self._log.error(te) + self._abort_cause = "error" + self.request_abort_fn() +``` + +In the failed-job abort path (lines 1023–1027): + +```python + if not result.none_failed and self._abort_on_failed_job: + self._log.error("Some jobs failed, aborting acquisition because abort_on_failed_job=True") + self._abort_cause = "error" + self.request_abort_fn() + return +``` + +- [ ] **Step 4: Compute `reason` and write the end breadcrumb in `finally`** + +Set a fatal-error flag in the generic handler (lines 511–513): + +```python + except Exception as e: + self._log.exception(e) + self._run_state_fatal = True + raise +``` + +Initialize the flag at the very top of `run()` (next to `this_image_callback_id = None`, line 425): + +```python + def run(self): + this_image_callback_id = None + self._run_state_fatal = False +``` + +In the `finally` block, replace the existing Slack-finish block — from `if self._slack_notifier is not None:` (~line 526) through the final `self.callbacks.signal_acquisition_finished()` (line 539) — so it computes `reason`, writes the end breadcrumb, passes `reason` to `AcquisitionStats`, and still calls `signal_acquisition_finished()` exactly once. The replacement: + +```python + # Determine why the acquisition ended (drives the watchdog + the in-process finish msg). + if self._run_state_fatal: + reason = "error" + elif self.abort_requested_fn(): + reason = "error" if self._abort_cause == "error" else "user_abort" + elif self._acquisition_error_count > 0: + reason = "completed_with_errors" + else: + reason = "completed" + + total_duration = time.time() - self.timestamp_acquisition_started + self._run_state.end( + reason, + { + "total_images": self.image_count, + "total_timepoints": self.time_point, + "total_duration_seconds": total_duration, + "errors_encountered": self._acquisition_error_count, + }, + ) + + # Send Slack acquisition finished notification via callback (ensures ordering with timepoint notifications) + if self._slack_notifier is not None: + try: + stats = AcquisitionStats( + total_images=self.image_count, + total_timepoints=self.time_point, + total_duration_seconds=total_duration, + errors_encountered=self._acquisition_error_count, + experiment_id=self.experiment_ID or "unknown", + reason=reason, + ) + self.callbacks.signal_slack_acquisition_finished(stats) + except Exception as e: + self._log.warning(f"Failed to send Slack acquisition finished notification: {e}") + + self.callbacks.signal_acquisition_finished() +``` + +- [ ] **Step 5: Unit-test the reason logic in isolation** + +```python +# tests/control/test_worker_reason.py +import time +from unittest.mock import MagicMock + +import squid.acquisition_state as ast +from control.core.multi_point_worker import MultiPointWorker + + +def _make_worker(tmp_path, monkeypatch): + # Build a bare worker without running __init__ (we only exercise the finally logic helpers). + w = MultiPointWorker.__new__(MultiPointWorker) + w.time_point = 2 + w.Nt = 5 + w.image_count = 40 + w._acquisition_error_count = 0 + w._abort_cause = None + w._run_state_fatal = False + w.experiment_ID = "e" + w.timestamp_acquisition_started = time.time() - 1 + w._run_state = ast.RunStateWriter.start( + experiment_id="e", pid=1, config_path=None, output_path="o", + expected={}, state_dir=tmp_path, + ) + w.abort_requested_fn = lambda: False + return w + + +def _reason(w): + # Mirror the finally classification. + if w._run_state_fatal: + return "error" + if w.abort_requested_fn(): + return "error" if w._abort_cause == "error" else "user_abort" + if w._acquisition_error_count > 0: + return "completed_with_errors" + return "completed" + + +def test_reason_completed(tmp_path, monkeypatch): + assert _reason(_make_worker(tmp_path, monkeypatch)) == "completed" + + +def test_reason_user_abort(tmp_path, monkeypatch): + w = _make_worker(tmp_path, monkeypatch) + w.abort_requested_fn = lambda: True + assert _reason(w) == "user_abort" + + +def test_reason_error_on_timeout_abort(tmp_path, monkeypatch): + w = _make_worker(tmp_path, monkeypatch) + w.abort_requested_fn = lambda: True + w._abort_cause = "error" + assert _reason(w) == "error" + + +def test_reason_completed_with_errors(tmp_path, monkeypatch): + w = _make_worker(tmp_path, monkeypatch) + w._acquisition_error_count = 3 + assert _reason(w) == "completed_with_errors" +``` + +Run: `cd software && python3 -m pytest tests/control/test_worker_reason.py -v` +Expected: PASS (4 tests). (This test pins the classification table; the in-context version is exercised end-to-end in Task 12.) + +- [ ] **Step 6: Commit** + +```bash +git add software/control/core/multi_point_worker.py software/control/slack_notifier.py software/tests/control/test_worker_reason.py +git commit -m "feat(watchdog): heartbeat + end-reason breadcrumb in acquisition worker" +``` + +--- + +## Task 10: Notifier trim — gate the finish message on a clean end + +**Files:** +- Modify: `control/slack_notifier.py` (`notify_acquisition_finished`, lines 610–646) +- Test: `tests/control/test_notifier_trim.py` + +- [ ] **Step 1: Write the failing test** + +```python +# tests/control/test_notifier_trim.py +from unittest.mock import patch + +import control._def +from control.slack_notifier import SlackNotifier, AcquisitionStats + + +def _stats(reason): + return AcquisitionStats( + total_images=10, total_timepoints=2, total_duration_seconds=5.0, + errors_encountered=0, experiment_id="e", reason=reason, + ) + + +def test_finish_message_sent_only_on_clean_completion(monkeypatch): + monkeypatch.setattr(control._def.SlackNotifications, "NOTIFY_ON_ACQUISITION_FINISHED", True) + n = SlackNotifier(bot_token="x", channel_id="C") + with patch.object(n, "_queue_message") as q: + n.notify_acquisition_finished(_stats("completed")) + assert q.call_count == 1 + + with patch.object(n, "_queue_message") as q: + n.notify_acquisition_finished(_stats("error")) + n.notify_acquisition_finished(_stats("user_abort")) + n.notify_acquisition_finished(_stats("completed_with_errors")) + assert q.call_count == 0 # watchdog owns these alerts +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd software && python3 -m pytest tests/control/test_notifier_trim.py -v` +Expected: FAIL — finish message is queued for all reasons (`assert 3 == 0`). + +- [ ] **Step 3: Edit `notify_acquisition_finished`** + +In `control/slack_notifier.py`, add a guard right after the existing `NOTIFY_ON_ACQUISITION_FINISHED` check at the top of `notify_acquisition_finished` (line ~611): + +```python + def notify_acquisition_finished(self, stats: AcquisitionStats): + if not control._def.SlackNotifications.NOTIFY_ON_ACQUISITION_FINISHED: + return + if stats.reason != "completed": + # Premature/degraded ends are reported once by the acquisition watchdog. + return +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd software && python3 -m pytest tests/control/test_notifier_trim.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add software/control/slack_notifier.py software/tests/control/test_notifier_trim.py +git commit -m "feat(watchdog): notifier reports only clean finishes; watchdog owns premature alerts" +``` + +--- + +## Task 11: Shutdown hook — abort + join on close + +**Files:** +- Modify: `main_hcs.py` (shutdown sequence, lines 437–439) + +- [ ] **Step 1: Edit the shutdown sequence** + +In `main_hcs.py`, replace the shutdown tail (lines 437–439): + +```python + exit_code = app.exec_() + logging.shutdown() # Flush log handlers before os._exit() bypasses Python cleanup + os._exit(exit_code) +``` + +with: + +```python + exit_code = app.exec_() + + # If the app is quitting mid-acquisition, request the normal abort and let the worker + # write its end breadcrumb so the watchdog reports "aborted" rather than a crash. + try: + mpc = getattr(win, "multipointController", None) + if mpc is not None and mpc.acquisition_in_progress(): + log.info("Acquisition in progress at shutdown; requesting abort before exit.") + mpc.request_abort_aquisition() + if getattr(mpc, "thread", None) is not None: + mpc.thread.join(timeout=15.0) + except Exception as e: + log.warning(f"Error during shutdown abort handling: {e}") + + logging.shutdown() # Flush log handlers before os._exit() bypasses Python cleanup + os._exit(exit_code) +``` + +- [ ] **Step 2: Verify it imports and the app still launches** + +Run: `cd software && python3 -c "import ast; ast.parse(open('main_hcs.py').read()); print('parse ok')"` +Expected: `parse ok`. + +Run (manual smoke, simulation): `cd software && timeout 25 python3 main_hcs.py --simulation` — confirm the GUI starts and closes cleanly with no traceback from the new block. (No automated test: `main_hcs.py` is excluded from CI and drives the full GUI.) + +- [ ] **Step 3: Commit** + +```bash +git add software/main_hcs.py +git commit -m "feat(watchdog): write an aborted breadcrumb when quitting mid-acquisition" +``` + +--- + +## Task 12: Integration test — full breadcrumb lifecycle + +**Files:** +- Create: `tests/control/test_watchdog_integration.py` + +- [ ] **Step 1: Write the test** + +```python +# tests/control/test_watchdog_integration.py +import os +import time + +import squid.acquisition_state as ast +import control.microscope +import tests.control.gui_test_stubs as gts + + +def _wait_for(predicate, timeout=30.0, interval=0.2): + deadline = time.time() + timeout + while time.time() < deadline: + if predicate(): + return True + time.sleep(interval) + return False + + +def test_simulated_acquisition_writes_ended_breadcrumb(qtbot): + state_dir = os.environ["SQUID_WATCHDOG_STATE_DIR"] + scope = control.microscope.Microscope.build_from_global_config(True) + mpc = gts.get_test_qt_multi_point_controller(microscope=scope) + + mpc.run_acquisition() + assert _wait_for(lambda: ast.read_run(state_dir) is not None) + assert ast.read_run(state_dir)["status"] == "running" + + # Let it finish (the default test acquisition is short); fall back to abort. + finished = _wait_for(lambda: (ast.read_run(state_dir) or {}).get("status") == "ended", timeout=20.0) + if not finished: + mpc.request_abort_aquisition() + assert _wait_for(lambda: (ast.read_run(state_dir) or {}).get("status") == "ended", timeout=20.0) + + rec = ast.read_run(state_dir) + assert rec["status"] == "ended" + assert rec["reason"] in {"completed", "completed_with_errors", "user_abort", "error"} + assert rec["ended_at"] is not None + scope.close() +``` + +- [ ] **Step 2: Run the test** + +Run: `cd software && python3 -m pytest tests/control/test_watchdog_integration.py -v` +Expected: PASS — `run.json` transitions `running → ended` with a valid reason and `heartbeat_at`/`ended_at` populated. + +- [ ] **Step 3: Commit** + +```bash +git add software/tests/control/test_watchdog_integration.py +git commit -m "test(watchdog): end-to-end breadcrumb lifecycle in simulation" +``` + +--- + +## Task 13: Service recipes + README + +**Files:** +- Create: `acquisition_watchdog/systemd/squid-acquisition-watchdog.service` +- Create: `acquisition_watchdog/windows/squid-acquisition-watchdog.xml` +- Create: `acquisition_watchdog/windows/install.ps1` +- Create: `acquisition_watchdog/README.md` + +- [ ] **Step 1: Linux systemd user unit** + +```ini +# acquisition_watchdog/systemd/squid-acquisition-watchdog.service +# Install (per user): +# mkdir -p ~/.config/systemd/user +# cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/ +# # edit WorkingDirectory + --config below to match this machine, then: +# systemctl --user daemon-reload +# systemctl --user enable --now squid-acquisition-watchdog +[Unit] +Description=Squid acquisition watchdog (alerts on prematurely-ended acquisitions) +After=default.target + +[Service] +Type=simple +WorkingDirectory=%h/Squid/software +ExecStart=/usr/bin/python3 -m acquisition_watchdog --config %h/Squid/software/configuration.ini +Restart=always +RestartSec=5 + +[Install] +WantedBy=default.target +``` + +- [ ] **Step 2: Windows Task Scheduler task + installer** + +```xml + + + + + Squid acquisition watchdog (alerts on prematurely-ended acquisitions) + + + + true + + + + IgnoreNew + false + + PT1M + 999 + + PT0S + + + + pythonw.exe + -m acquisition_watchdog --config C:\Squid\software\configuration.ini + C:\Squid\software + + + +``` + +```powershell +# acquisition_watchdog/windows/install.ps1 +# Run in PowerShell from software\ : .\acquisition_watchdog\windows\install.ps1 +$ErrorActionPreference = "Stop" +$taskName = "SquidAcquisitionWatchdog" +$xmlPath = Join-Path $PSScriptRoot "squid-acquisition-watchdog.xml" +Write-Host "Registering scheduled task '$taskName' from $xmlPath" +Register-ScheduledTask -TaskName $taskName -Xml (Get-Content $xmlPath -Raw) -Force +Write-Host "Done. Edit the task's --config/WorkingDirectory if your install path differs, then log off/on or 'Start' the task." +``` + +- [ ] **Step 3: README** + +```markdown +# acquisition_watchdog/README.md +# Acquisition Watchdog + +Independent process that alerts (via Slack) when a Squid acquisition ends +prematurely — process crash/hang/kill, fatal error, or user abort. Covers runs +launched from the GUI and from the MCP control server. + +## How it works +The Squid GUI writes a `run.json` breadcrumb (start / throttled heartbeat / end) +into a shared state dir. This watchdog polls it and posts one Slack alert when a +run dies, hangs, or ends with a non-clean reason. Clean completions are silent. + +## Run it + cd software + python3 -m acquisition_watchdog --config ./configuration.ini + +Options: `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. + +Slack credentials are read from the `[SlackNotifications]` section of the active +`.ini` (same `bot_token` / `channel_id` the GUI uses). Set `watchdog_enabled = False` +in that section to disable watchdog alerts on a machine. + +## Install as an always-on service +- **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). +- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). + +## State dir +Defaults to `platformdirs.user_state_path("squid","cephla")/watchdog`. Override with +`SQUID_WATCHDOG_STATE_DIR` (must match the GUI's environment) or `--state-dir`. + +## Remote / power-loss coverage (future) +Point `--state-dir` at a shared/synced mount on another host and run this process +there. Per-machine `run-.json` naming and clock-skew tolerance are needed +first (see the design spec, "Future work"). +``` + +- [ ] **Step 4: Commit** + +```bash +git add software/acquisition_watchdog/systemd software/acquisition_watchdog/windows software/acquisition_watchdog/README.md +git commit -m "docs(watchdog): add systemd + Windows service recipes and README" +``` + +--- + +## Task 14: Finalize — format, full test run, commit the spec + +**Files:** +- All new/modified files +- `docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md` + +- [ ] **Step 1: Format with Black** + +Run: `cd software && black --config pyproject.toml squid/slack.py squid/acquisition_state.py acquisition_watchdog/ tests/squid/ tests/acquisition_watchdog/ tests/control/test_watchdog_breadcrumbs.py tests/control/test_watchdog_integration.py tests/control/test_worker_reason.py tests/control/test_notifier_trim.py tests/control/test_slack_notifier_send.py control/slack_notifier.py control/core/multi_point_worker.py control/core/multi_point_controller.py main_hcs.py` +Expected: files reformatted/unchanged; no errors. + +- [ ] **Step 2: Run the watchdog + new unit tests** + +Run: `cd software && python3 -m pytest tests/squid tests/acquisition_watchdog tests/control/test_worker_reason.py tests/control/test_notifier_trim.py tests/control/test_slack_notifier_send.py -v` +Expected: ALL PASS. + +- [ ] **Step 3: Run the engine/integration tests** + +Run: `cd software && python3 -m pytest tests/control/test_watchdog_breadcrumbs.py tests/control/test_watchdog_integration.py tests/control/test_MultiPointWorker.py -v` +Expected: ALL PASS (no regression in the existing worker test). + +- [ ] **Step 4: Full suite (CI parity)** + +Run: `cd software && python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py` +Expected: no new failures attributable to these changes. + +- [ ] **Step 5: Commit the spec + plan and verify Black on the whole tree** + +Run: `cd software && black --config pyproject.toml --check .` +Expected: "All done!" (no files would be reformatted). + +```bash +git add software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md +git commit -m "docs(watchdog): add design spec and implementation plan" +``` + +--- + +## Self-review notes + +- **Spec coverage:** start/heartbeat/end protocol (Tasks 3, 8, 9); watchdog poll/classify/dedup (Task 6); config sharing (Task 4); cross-platform state dir + PID degrade (Tasks 3, 6); deployment recipes (Task 13); notifier split (Tasks 2, 10); server coverage (engine-level instrumentation in Tasks 8–9 — no server-specific code needed). v1 out-of-scope items (progress-stall, power-loss, server-thread health) are intentionally absent. +- **`app_closed`** from the spec is implemented as `user_abort` via the shutdown abort+join (Task 11); noted in the taxonomy above. Update the spec's taxonomy/ shutdown wording to match (done as part of plan authoring). +- **Type consistency:** `RunStateWriter.start(...)`/`beat`/`end`, `read_run`, `default_state_dir`, `NullRunStateWriter` used identically across Tasks 3/6/8/9/12; `SlackConfig` fields (`enabled`,`bot_token`,`channel_id`,`watchdog_enabled`) consistent across Tasks 4/6; `AcquisitionStats.reason` added in Task 9 and consumed in Task 10; `squid.slack.post_message(token, channel, text, blocks)` signature consistent across Tasks 1/2/6. diff --git a/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md b/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md new file mode 100644 index 000000000..a7ce2cb1f --- /dev/null +++ b/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md @@ -0,0 +1,230 @@ +# Acquisition Watchdog + +Detect when an acquisition ends prematurely — process **crash / hang / kill**, **fatal +error**, or **user abort** — and send a single Slack alert. Works for acquisitions +launched from the GUI *and* from the MCP control server, on Ubuntu and Windows. + +## Motivation + +A crashing process cannot report its own death. The existing in-process +`SlackNotifier` (`control/slack_notifier.py`) runs on a daemon thread *inside* the GUI +process, so it can report live errors and a clean finish, but it can never report a +segfault in a camera SDK, an OOM-kill, `os._exit()`, a power-loss of the process, or a +frozen UI — the thing that would send the alert is the thing that died. + +The fix is an **independent process** that watches on-disk breadcrumbs the app leaves +behind. Because both the GUI and the MCP control server run acquisitions through the +same engine (`MultiPointController.run_acquisition()` → `MultiPointWorker.run()`, both +in `control/core/`), instrumenting the **engine** — not the GUI widgets — covers both +launch paths with no extra code. Server-driven runs are unattended, which is exactly +when an alert matters most. + +## Architecture + +Three parts, with a clean dependency DAG (`acquisition_watchdog` → `squid`; `control` → +`squid`; the watchdog never imports `control`): + +``` + GUI process (main_hcs.py, incl. in-process MCP control server) + ┌───────────────────────────────────────────────┐ + │ MultiPointController.run_acquisition() │ writes + │ MultiPointWorker.run(): │ ─────────► /run.json + │ start → write run.json (status=running) │ (atomic os.replace) + │ loop → beat() heartbeat + progress (~5s) │ + │ finally→ write end (status=ended, reason) │ + │ in-process SlackNotifier (unchanged role): │ + │ live errors, progress, finish-with-mosaic │ + └───────────────────────────────────────────────┘ + reads + acquisition_watchdog (independent always-on process) ◄──────── /run.json + poll every ~5s → classify → Slack alert (once per run_id) reads [SlackNotifications] + from the active .ini +``` + +### Part 1 — Breadcrumb protocol (in the acquisition engine) + +A new leaf module `squid/acquisition_state.py` owns the run-state schema and atomic +read/write. The acquisition engine writes; the watchdog reads. It must stay +import-light (stdlib only) and must not import `control`. + +**`run.json`** — a single file in the shared state dir, replaced atomically +(`os.replace`, atomic and torn-read-free on both POSIX and Windows): + +| Field | Type | Notes | +|---|---|---| +| `schema_version` | int | `1` | +| `run_id` | str | `uuid4().hex`; the watchdog's dedup key | +| `experiment_id` | str | from `MultiPointController` | +| `machine` | str | config machine-name if present, else `socket.gethostname()` | +| `pid` | int | `os.getpid()` of the GUI process | +| `config_path` | str | absolute path of the active `.ini` (so the watchdog finds Slack settings with no args) | +| `output_path` | str | experiment output dir | +| `started_at` | float | epoch seconds, UTC | +| `heartbeat_at` | float | epoch seconds; bumped ~every `HEARTBEAT_INTERVAL_S` | +| `progress` | obj | `{timepoint, expected_timepoints, fov, region_fovs, images}` | +| `expected` | obj | `{timepoints, regions, fovs, channels, z}` — `fovs` is total planned across all regions | +| `status` | str | `running` \| `ended` | +| `reason` | str\|null | set when `ended` — see taxonomy below | +| `ended_at` | float\|null | epoch seconds | +| `stats` | obj\|null | `{total_images, errors_encountered, total_duration_seconds}` at end | + +**Write points in the engine:** + +1. **Start** — in `MultiPointController.run_acquisition()`, which owns the experiment id + and acquisition parameters: write the full record with `status=running`, a fresh + `run_id`, and `expected` totals. The `run_id` is handed to the worker so its heartbeat + and end writes update the same record. +2. **Heartbeat** — a `HeartbeatWriter.beat(progress)` helper, called at the worker + loop's existing abort-check points (the per-timepoint, per-FOV, and the long-wait + poll loops in `MultiPointWorker.run()`). `beat()` is cheap: it updates an in-memory + timestamp and **flushes to disk at most every `HEARTBEAT_INTERVAL_S` (default 5 s)**. + Because the long-wait loops (timelapse `dt`, fluidics) already poll the abort flag, + the heartbeat stays fresh whenever the worker thread is alive and freezes only on a + true hang or process death. +3. **End** — in the `finally` of `MultiPointWorker.run()` (which already runs for normal + finish, abort, and caught exceptions): write `status=ended`, `reason`, `ended_at`, + `stats` (reuse the existing `AcquisitionStats` / `_acquisition_error_count`). +4. **App close while acquiring** — `main_hcs.py` shuts down via `os._exit()` (skips + destructors). In the shutdown path, if `acquisition_in_progress()`, request the + normal abort and **join the worker** (bounded timeout) before `os._exit()`, so the + worker writes its normal `user_abort` end record and a deliberate quit is not + misreported as a crash. (A distinct `app_closed` reason is future work.) + +**Reason taxonomy** (computed at the end write): + +| `reason` | When | Watchdog alerts? | +|---|---|---| +| `completed` | loop finished all timepoints, `errors_encountered == 0` | no (silent) | +| `completed_with_errors` | loop finished but `errors_encountered > 0` | yes | +| `error` | uncaught exception, or auto-abort from `TimeoutError` / failed-job abort | yes | +| `user_abort` | abort flag set externally (human / server) **or** GUI closed mid-run (shutdown aborts + joins) | yes | +| *(no end record)* | process crashed/killed/hung before writing end | yes (crash/hang) | + +To distinguish `error` from `user_abort`, the engine records an **abort cause**: the +auto-abort paths (`TimeoutError`, failed-job abort) tag the cause as error-type; a bare +`request_abort_aquisition()` is `user`. The end write maps cause → reason. (`errors_encountered` +already exists and drives `completed` vs `completed_with_errors`.) + +### Part 2 — The watchdog process (`software/acquisition_watchdog/`) + +Independent, lightweight (stdlib + `pyyaml`), **does not import `control`**. + +- **Poll loop** (every `POLL_INTERVAL_S`, default 5 s): read `run.json`; if absent, idle. +- **Classification:** + - `status=running` and (`pid` not alive **or** `now − heartbeat_at > HEARTBEAT_TIMEOUT_S`) → **crash/hang** → alert. + - `status=ended` and `reason ∈ {completed_with_errors, error, user_abort}` → alert. + - `status=ended` and `reason=completed` → silent. +- **PID check** is a best-effort accelerator catching hard death within one poll: + `psutil.pid_exists(pid)` if `psutil` is importable, else POSIX `os.kill(pid, 0)`, else + skip (heartbeat-only). The **heartbeat is the primary, OS-agnostic signal**; PID just + makes a true crash detectable in ~5 s instead of waiting out the heartbeat timeout. +- **Alert once per `run_id`.** Alerted ids are persisted to `/alerted.json` so + a watchdog restart never re-alerts, and so a crash that happened while the watchdog was + down is alerted exactly once on its next start. +- **Defaults** (overridable via CLI flags / config): `POLL_INTERVAL_S=5`, + `HEARTBEAT_INTERVAL_S=5`, `HEARTBEAT_TIMEOUT_S=120` (comfortably above the longest + legitimate single blocking op — long exposures, stage moves, fluidics — while still + catching a hang within ~2 min). + +**Alert payload:** machine name, experiment id, classification (crash / hang / error / +aborted / completed-with-errors), progress vs expected ("stopped at timepoint 3/10, +360 images"), start + last-heartbeat / end timestamps, output path. + +### Part 3 — Notifier trim (minimal) + +`control/slack_notifier.py` keeps live in-run error warnings, per-timepoint progress, and +the finish-with-mosaic summary. It **stops flagging bad *endings* itself** (the +end-of-run failure messaging moves to the watchdog), so a failed run produces exactly one +alert. The ~20-line Slack `chat.postMessage` send is extracted into a shared, +dependency-free `squid/slack.py` (stdlib `urllib`/`json`, no Qt/`control` imports) used by +both the notifier and the watchdog. Image upload (`files.getUploadURLExternal`) stays in +`SlackNotifier` — the watchdog never needs it. + +## Cross-platform (Ubuntu + Windows) + +- **State dir** via `platformdirs` (already used for logs in `squid/logging.py`): + `default_state_dir()` in `squid/acquisition_state.py` returns + `platformdirs.user_state_path("squid", "cephla") / "watchdog"` — `~/.local/state` (or + `~/.cache`) on Linux, `%LOCALAPPDATA%\cephla\squid\…` on Windows. Writer and reader call + the same helper so they always agree. Overridable via `SQUID_WATCHDOG_STATE_DIR` (both) + and `--state-dir` (watchdog). +- **Atomic writes** use `os.replace` (atomic on both OSes). **PID check** is guarded per + above. No POSIX-only calls on the hot path. + +## Config sharing + +The watchdog reads the **same `[SlackNotifications]`** the GUI uses (token, channel, +enabled). Resolution order: `--config` flag → `SQUID_CONFIG` env → +`run.json.config_path` (written by the GUI, so the watchdog auto-discovers the active +`.ini` with no args) → `cache/config_file_path.txt`. Parsed with stdlib `configparser`; +the watchdog never imports `control._def`. One new opt-out key, +`[SlackNotifications] watchdog_enabled` (default `True` when a token+channel are set), lets +a machine disable watchdog alerts without disabling the in-process notifier. + +## Deployment — always-on user service + +Core process is just `python -m acquisition_watchdog [--config ]`, identical on both +OSes. Shipped recipes: + +- **Linux:** a systemd **`--user`** unit (`Restart=always`, `WantedBy=default.target`), + `systemctl --user enable --now squid-acquisition-watchdog`. Runs as the same user as the + GUI, sharing the `platformdirs` state dir. +- **Windows:** a **Task Scheduler** task triggered "at log on" of the user (sample `.xml` + + a small `install.ps1`). Same user, same state dir. + +Both ship in `acquisition_watchdog/` with a README. The identical code can later run as a +**remote monitor** (the option-3 variant) by pointing `--state-dir` at a shared/synced +mount — see Future work. + +## Proposed file layout + +| Path | Role | +|---|---| +| `squid/acquisition_state.py` | run-state schema, `default_state_dir()`, atomic read/write, `HeartbeatWriter` (engine writes, watchdog reads) | +| `squid/slack.py` | shared dependency-free `chat.postMessage` sender | +| `software/acquisition_watchdog/__main__.py` | CLI entry (`python -m acquisition_watchdog`) | +| `software/acquisition_watchdog/monitor.py` | poll loop + classification + dedup | +| `software/acquisition_watchdog/config.py` | resolve active `.ini`, load `[SlackNotifications]` | +| `software/acquisition_watchdog/alerts.py` | format the Slack alert payload | +| `software/acquisition_watchdog/systemd/`, `windows/`, `README.md` | install recipes + docs | +| `control/core/multi_point_controller.py`, `control/core/multi_point_worker.py` | write breadcrumbs (start / beat / end) + abort-cause tagging | +| `control/slack_notifier.py` | stop end-of-run failure messaging; call `squid/slack.py` | +| `main_hcs.py` | write `app_closed` end record on shutdown-while-acquiring | + +Named `acquisition_watchdog` (not `watchdog`) to avoid colliding with the PyPI `watchdog` +filesystem-events package. + +## Tests + +- `squid/acquisition_state.py`: round-trip (start → beats → end), atomic-replace, schema + versioning; `beat()` throttling (many calls, ≤1 flush per interval). +- `acquisition_watchdog/monitor.py`: classification table — synthetic `run.json` for each + state (`running`+stale heartbeat, `running`+dead PID, each `ended` reason, `completed`) + → expected alert/no-alert; dedup (no double alert per `run_id`, persists across a + monitor restart via `alerted.json`). +- `acquisition_watchdog/config.py`: resolution precedence (`--config` > env > + `run.json.config_path` > cache pointer); missing/disabled Slack → logs, no crash. +- `squid/slack.py`: monkeypatch `urllib`, assert request shape; no network. +- PID check: alive (current pid) vs an impossible/known-dead pid, on the available + platform; graceful degrade when `psutil` absent. +- Engine integration (simulation mode): run a short simulated acquisition and assert + `run.json` transitions `running → ended/completed` and `heartbeat_at` advances. Mirror + the abort path → `reason=user_abort`, and a forced job error → `completed_with_errors`. +- Black (120) over the new package; it is not in the formatter excludes. + +## Out of scope (v1) + +- **Progress-stall detection** (process alive but worker wedged) — needs per-step timing + from `acquisition.yaml`; fragile, deferred. v1 catches death + full hang + abort + error. +- **Machine power-loss coverage** — needs the remote-monitor variant. +- **MCP control-server thread health** — server-thread death does not abort an in-flight + acquisition, so it is not an "acquisition ended" event. +- **Multi-microscope aggregation**, a GUI panel for the watchdog, and secrets management + for the bot token (stays in the `.ini` as today). + +## Future work + +- **Remote monitor:** point `--state-dir` at a shared mount; key the state file per + machine (`run-.json`) to avoid collisions, and add a clock-skew tolerance to + the heartbeat comparison (writer/reader no longer share a clock). +- Progress-stall detection; packaging the per-OS install into one script. From 95edd861eb0ce7e6a2326cf1d19a333702b14902 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 23 Jun 2026 11:09:55 -0700 Subject: [PATCH 02/23] feat(slack): add dependency-free squid.slack.post_message sender Co-Authored-By: Claude Opus 4.8 (1M context) --- software/squid/slack.py | 57 ++++++++++++++++++++++++++++++ software/tests/squid/test_slack.py | 56 +++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) create mode 100644 software/squid/slack.py create mode 100644 software/tests/squid/test_slack.py diff --git a/software/squid/slack.py b/software/squid/slack.py new file mode 100644 index 000000000..6bd3eafb3 --- /dev/null +++ b/software/squid/slack.py @@ -0,0 +1,57 @@ +# squid/slack.py +"""Dependency-free Slack chat.postMessage sender. + +Shared by the in-process SlackNotifier (control/slack_notifier.py) and the +standalone acquisition watchdog. Stdlib only — safe to import without the +control/Qt/hardware stack. +""" +import json +import urllib.error +import urllib.request +from typing import Optional, Tuple + +import squid.logging + +_log = squid.logging.get_logger(__name__) + +SLACK_API_BASE = "https://slack.com/api" + + +def post_message( + bot_token: Optional[str], + channel_id: Optional[str], + text: str, + blocks: Optional[list] = None, + timeout: float = 15.0, +) -> Tuple[bool, Optional[str]]: + """Post a message to Slack. Returns (ok, message_ts).""" + if not bot_token or not channel_id: + _log.debug("No Slack bot token or channel configured") + return False, None + + payload = {"channel": channel_id, "text": text} + if blocks: + payload["blocks"] = blocks + data = json.dumps(payload).encode("utf-8") + request = urllib.request.Request( + f"{SLACK_API_BASE}/chat.postMessage", + data=data, + headers={ + "Content-Type": "application/json; charset=utf-8", + "Authorization": f"Bearer {bot_token}", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + result = json.loads(response.read().decode("utf-8")) + if result.get("ok"): + return True, result.get("ts") + _log.warning(f"Slack API error: {result.get('error')}") + return False, None + except urllib.error.URLError as e: + _log.warning(f"Failed to send Slack message: {e}") + return False, None + except Exception as e: + _log.warning(f"Unexpected error sending Slack message: {e}") + return False, None diff --git a/software/tests/squid/test_slack.py b/software/tests/squid/test_slack.py new file mode 100644 index 000000000..2b5265cf2 --- /dev/null +++ b/software/tests/squid/test_slack.py @@ -0,0 +1,56 @@ +# tests/squid/test_slack.py +import json +from unittest.mock import patch, MagicMock + +import squid.slack as slack + + +def test_post_message_returns_false_without_credentials(): + assert slack.post_message(None, "C123", "hi") == (False, None) + assert slack.post_message("xoxb-1", None, "hi") == (False, None) + + +def test_post_message_builds_authorized_request_and_parses_ok(): + captured = {} + + class FakeResp: + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def read(self): + return json.dumps({"ok": True, "ts": "111.222"}).encode() + + def fake_urlopen(request, timeout=15): + captured["url"] = request.full_url + captured["headers"] = request.headers + captured["body"] = json.loads(request.data.decode()) + return FakeResp() + + with patch("urllib.request.urlopen", side_effect=fake_urlopen): + ok, ts = slack.post_message("xoxb-token", "C123", "hello", blocks=[{"type": "section"}]) + + assert ok is True and ts == "111.222" + assert captured["url"].endswith("/chat.postMessage") + assert captured["headers"]["Authorization"] == "Bearer xoxb-token" + assert captured["body"]["channel"] == "C123" + assert captured["body"]["text"] == "hello" + assert captured["body"]["blocks"] == [{"type": "section"}] + + +def test_post_message_handles_api_error(): + class FakeResp: + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def read(self): + return json.dumps({"ok": False, "error": "channel_not_found"}).encode() + + with patch("urllib.request.urlopen", return_value=FakeResp()): + ok, ts = slack.post_message("xoxb", "C1", "x") + assert ok is False and ts is None From 044e40fd246ab960cea71ec2633bec5b7b4802bd Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 23 Jun 2026 11:13:24 -0700 Subject: [PATCH 03/23] refactor(slack): route SlackNotifier sends through squid.slack Co-Authored-By: Claude Opus 4.8 (1M context) --- software/control/slack_notifier.py | 54 +------------------ .../tests/control/test_slack_notifier_send.py | 11 ++++ 2 files changed, 13 insertions(+), 52 deletions(-) create mode 100644 software/tests/control/test_slack_notifier_send.py diff --git a/software/control/slack_notifier.py b/software/control/slack_notifier.py index 343cca5f1..d5421240f 100644 --- a/software/control/slack_notifier.py +++ b/software/control/slack_notifier.py @@ -23,6 +23,7 @@ import control._def import squid.logging +import squid.slack log = squid.logging.get_logger(__name__) @@ -158,58 +159,7 @@ def _worker_loop(self): log.warning(f"Error in Slack worker loop: {e}") def _post_message(self, text: str, blocks: Optional[list] = None) -> Tuple[bool, Optional[str]]: - """Post a message to Slack using chat.postMessage API. - - Args: - text: Fallback text for the message. - blocks: Optional Block Kit blocks for rich formatting. - - Returns: - Tuple of (success, message_ts) where message_ts is the timestamp - of the posted message (used for threading). - """ - token = self.bot_token - channel = self.channel_id - - if not token or not channel: - log.debug("No Slack bot token or channel configured") - return False, None - - try: - log.info(f"Sending Slack message: {text[:50]}") - payload = { - "channel": channel, - "text": text, - } - if blocks: - payload["blocks"] = blocks - - data = json.dumps(payload).encode("utf-8") - request = urllib.request.Request( - f"{self.SLACK_API_BASE}/chat.postMessage", - data=data, - headers={ - "Content-Type": "application/json; charset=utf-8", - "Authorization": f"Bearer {token}", - }, - method="POST", - ) - - with urllib.request.urlopen(request, timeout=15) as response: - result = json.loads(response.read().decode("utf-8")) - if result.get("ok"): - log.info("Slack message sent successfully") - return True, result.get("ts") - else: - log.warning(f"Slack API error: {result.get('error')}") - return False, None - - except urllib.error.URLError as e: - log.warning(f"Failed to send Slack message: {e}") - return False, None - except Exception as e: - log.warning(f"Unexpected error sending Slack message: {e}") - return False, None + return squid.slack.post_message(self.bot_token, self.channel_id, text, blocks) def _upload_image( self, diff --git a/software/tests/control/test_slack_notifier_send.py b/software/tests/control/test_slack_notifier_send.py new file mode 100644 index 000000000..262d95bb0 --- /dev/null +++ b/software/tests/control/test_slack_notifier_send.py @@ -0,0 +1,11 @@ +# tests/control/test_slack_notifier_send.py +from unittest.mock import patch +from control.slack_notifier import SlackNotifier + + +def test_post_message_delegates_to_squid_slack(): + n = SlackNotifier(bot_token="xoxb-abc", channel_id="C999") + with patch("squid.slack.post_message", return_value=(True, "1.0")) as m: + ok, ts = n._post_message("hello", blocks=[{"type": "section"}]) + assert ok is True and ts == "1.0" + m.assert_called_once_with("xoxb-abc", "C999", "hello", [{"type": "section"}]) From 9741fd9720aa08489cc432abf51fd6b6c20431f8 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 23 Jun 2026 11:16:09 -0700 Subject: [PATCH 04/23] feat(watchdog): add squid.acquisition_state breadcrumb schema + writer Co-Authored-By: Claude Opus 4.8 (1M context) --- software/squid/acquisition_state.py | 148 ++++++++++++++++++ .../tests/squid/test_acquisition_state.py | 77 +++++++++ 2 files changed, 225 insertions(+) create mode 100644 software/squid/acquisition_state.py create mode 100644 software/tests/squid/test_acquisition_state.py diff --git a/software/squid/acquisition_state.py b/software/squid/acquisition_state.py new file mode 100644 index 000000000..5ed9da527 --- /dev/null +++ b/software/squid/acquisition_state.py @@ -0,0 +1,148 @@ +# squid/acquisition_state.py +"""On-disk acquisition run-state breadcrumbs, shared by the acquisition engine +(writer) and the standalone acquisition watchdog (reader). + +Stdlib-only leaf module: must NOT import anything from `control`. +""" +import json +import os +import socket +import tempfile +import time +import uuid +from pathlib import Path +from typing import Optional + +import platformdirs + +import squid.logging + +_log = squid.logging.get_logger(__name__) + +SCHEMA_VERSION = 1 +HEARTBEAT_INTERVAL_S = 5.0 +RUN_FILE_NAME = "run.json" + + +def default_state_dir() -> Path: + """Per-user watchdog state dir, shared by writer and reader. + + Overridable via SQUID_WATCHDOG_STATE_DIR (honored by both processes). + """ + override = os.environ.get("SQUID_WATCHDOG_STATE_DIR") + if override: + return Path(override) + return Path(platformdirs.user_state_path("squid", "cephla")) / "watchdog" + + +def run_file_path(state_dir: Optional[Path] = None) -> Path: + return Path(state_dir) / RUN_FILE_NAME if state_dir else default_state_dir() / RUN_FILE_NAME + + +def _atomic_write_json(path: Path, data: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".run-", suffix=".tmp") + try: + with os.fdopen(fd, "w") as f: + json.dump(data, f) + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) # atomic on POSIX and Windows + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise + + +def read_run(state_dir: Optional[Path] = None) -> Optional[dict]: + try: + with open(run_file_path(state_dir)) as f: + return json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return None + + +class RunStateWriter: + """Writes/updates the single run.json for the current acquisition.""" + + def __init__(self, record: dict, state_dir: Optional[Path] = None): + self._record = record + self._state_dir = state_dir + self._last_beat = 0.0 + + @classmethod + def start( + cls, + *, + experiment_id: str, + pid: int, + config_path: Optional[str], + output_path: str, + expected: dict, + machine: Optional[str] = None, + state_dir: Optional[Path] = None, + ) -> "RunStateWriter": + now = time.time() + record = { + "schema_version": SCHEMA_VERSION, + "run_id": uuid.uuid4().hex, + "experiment_id": experiment_id, + "machine": machine or socket.gethostname(), + "pid": pid, + "config_path": config_path, + "output_path": output_path, + "started_at": now, + "heartbeat_at": now, + "progress": {}, + "expected": expected, + "status": "running", + "reason": None, + "ended_at": None, + "stats": None, + } + writer = cls(record, state_dir=state_dir) + writer._flush() + writer._last_beat = now + return writer + + @property + def run_id(self) -> Optional[str]: + return self._record.get("run_id") + + def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: + if progress: + self._record["progress"] = progress + now = time.time() + if not force and (now - self._last_beat) < HEARTBEAT_INTERVAL_S: + return + self._last_beat = now + self._record["heartbeat_at"] = now + self._flush() + + def end(self, reason: str, stats: Optional[dict] = None) -> None: + self._record["status"] = "ended" + self._record["reason"] = reason + self._record["ended_at"] = time.time() + if stats is not None: + self._record["stats"] = stats + self._flush() + + def _flush(self) -> None: + try: + _atomic_write_json(run_file_path(self._state_dir), dict(self._record)) + except OSError as e: + _log.warning(f"Failed to write acquisition run state: {e}") + + +class NullRunStateWriter: + """No-op writer used when breadcrumbs are not wired (tests, side paths).""" + + run_id = None + + def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: + pass + + def end(self, reason: str, stats: Optional[dict] = None) -> None: + pass diff --git a/software/tests/squid/test_acquisition_state.py b/software/tests/squid/test_acquisition_state.py new file mode 100644 index 000000000..552e4d934 --- /dev/null +++ b/software/tests/squid/test_acquisition_state.py @@ -0,0 +1,77 @@ +# tests/squid/test_acquisition_state.py +import json + +import squid.acquisition_state as ast + + +def _expected(): + return {"timepoints": 3, "regions": 1, "fovs": 4, "channels": 2, "z": 1} + + +def test_start_writes_running_record(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="exp1", + pid=4321, + config_path="/cfg.ini", + output_path=str(tmp_path / "exp1"), + expected=_expected(), + machine="micro-1", + state_dir=tmp_path, + ) + rec = ast.read_run(tmp_path) + assert rec["status"] == "running" + assert rec["experiment_id"] == "exp1" + assert rec["pid"] == 4321 + assert rec["machine"] == "micro-1" + assert rec["expected"] == _expected() + assert rec["run_id"] == w.run_id + assert rec["reason"] is None + + +def test_beat_is_throttled_but_updates_progress_on_flush(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="e", + pid=1, + config_path=None, + output_path="o", + expected=_expected(), + state_dir=tmp_path, + ) + first = ast.read_run(tmp_path)["heartbeat_at"] + # Immediate beat is throttled (< HEARTBEAT_INTERVAL_S since start) -> file unchanged. + w.beat({"timepoint": 1}) + assert ast.read_run(tmp_path)["heartbeat_at"] == first + # Forced beat flushes and records progress. + w.beat({"timepoint": 2}, force=True) + rec = ast.read_run(tmp_path) + assert rec["heartbeat_at"] >= first + assert rec["progress"] == {"timepoint": 2} + + +def test_end_records_reason_and_stats(tmp_path): + w = ast.RunStateWriter.start( + experiment_id="e", + pid=1, + config_path=None, + output_path="o", + expected=_expected(), + state_dir=tmp_path, + ) + w.end("user_abort", {"total_images": 7, "errors_encountered": 0}) + rec = ast.read_run(tmp_path) + assert rec["status"] == "ended" + assert rec["reason"] == "user_abort" + assert rec["ended_at"] is not None + assert rec["stats"]["total_images"] == 7 + + +def test_read_run_missing_returns_none(tmp_path): + assert ast.read_run(tmp_path) is None + + +def test_null_writer_is_noop(tmp_path): + w = ast.NullRunStateWriter() + w.beat({"timepoint": 1}) + w.end("completed", {}) + assert ast.read_run(tmp_path) is None + assert w.run_id is None From 5f2539cbd24ee5a90f7efc02c88f53487445f08d Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 23 Jun 2026 11:37:33 -0700 Subject: [PATCH 05/23] feat(watchdog): add config resolution + [SlackNotifications] loader Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/__init__.py | 0 software/acquisition_watchdog/config.py | 60 +++++++++++++++++++ .../tests/acquisition_watchdog/__init__.py | 0 .../tests/acquisition_watchdog/test_config.py | 39 ++++++++++++ 4 files changed, 99 insertions(+) create mode 100644 software/acquisition_watchdog/__init__.py create mode 100644 software/acquisition_watchdog/config.py create mode 100644 software/tests/acquisition_watchdog/__init__.py create mode 100644 software/tests/acquisition_watchdog/test_config.py diff --git a/software/acquisition_watchdog/__init__.py b/software/acquisition_watchdog/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/software/acquisition_watchdog/config.py b/software/acquisition_watchdog/config.py new file mode 100644 index 000000000..a00fee7e2 --- /dev/null +++ b/software/acquisition_watchdog/config.py @@ -0,0 +1,60 @@ +# acquisition_watchdog/config.py +"""Resolve the active Squid .ini and read its [SlackNotifications] section, +without importing the heavy control._def module. +""" +import configparser +import os +from pathlib import Path +from typing import NamedTuple, Optional + + +class SlackConfig(NamedTuple): + enabled: bool + bot_token: Optional[str] + channel_id: Optional[str] + watchdog_enabled: bool + + +def resolve_config_path(cli_config: Optional[str], run_record: Optional[dict]) -> Optional[Path]: + """Priority: --config > $SQUID_CONFIG > run.json config_path > cache pointer.""" + if cli_config: + return Path(cli_config) + env = os.environ.get("SQUID_CONFIG") + if env: + return Path(env) + if run_record and run_record.get("config_path"): + return Path(run_record["config_path"]) + cache = Path("cache/config_file_path.txt") + if cache.exists(): + first = cache.read_text().splitlines() + if first: + return Path(first[0].strip()) + return None + + +def load_slack_config(config_path: Optional[Path]) -> SlackConfig: + if not config_path or not Path(config_path).exists(): + return SlackConfig(False, None, None, True) + cp = configparser.ConfigParser() + try: + cp.read(config_path) + except configparser.Error: + return SlackConfig(False, None, None, True) + if not cp.has_section("SLACKNOTIFICATIONS"): + return SlackConfig(False, None, None, True) + sec = cp["SLACKNOTIFICATIONS"] + + def getbool(key: str, default: bool) -> bool: + try: + return sec.getboolean(key, default) + except ValueError: + return default + + token = sec.get("bot_token", fallback=None) or None + channel = sec.get("channel_id", fallback=None) or None + return SlackConfig( + enabled=getbool("enabled", False), + bot_token=token, + channel_id=channel, + watchdog_enabled=getbool("watchdog_enabled", True), + ) diff --git a/software/tests/acquisition_watchdog/__init__.py b/software/tests/acquisition_watchdog/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/software/tests/acquisition_watchdog/test_config.py b/software/tests/acquisition_watchdog/test_config.py new file mode 100644 index 000000000..791617cda --- /dev/null +++ b/software/tests/acquisition_watchdog/test_config.py @@ -0,0 +1,39 @@ +# tests/acquisition_watchdog/test_config.py +from pathlib import Path + +from acquisition_watchdog import config as wdconfig + + +def _write_ini(path, body): + path.write_text(body) + return path + + +def test_resolve_prefers_cli_then_env_then_run_record(tmp_path, monkeypatch): + monkeypatch.delenv("SQUID_CONFIG", raising=False) + assert wdconfig.resolve_config_path("/cli.ini", {"config_path": "/run.ini"}) == Path("/cli.ini") + monkeypatch.setenv("SQUID_CONFIG", "/env.ini") + assert wdconfig.resolve_config_path(None, {"config_path": "/run.ini"}) == Path("/env.ini") + monkeypatch.delenv("SQUID_CONFIG", raising=False) + assert wdconfig.resolve_config_path(None, {"config_path": "/run.ini"}) == Path("/run.ini") + + +def test_load_slack_config_reads_section(tmp_path): + ini = _write_ini( + tmp_path / "c.ini", + "[SLACKNOTIFICATIONS]\nenabled = True\nbot_token = xoxb-xyz\nchannel_id = C42\nwatchdog_enabled = True\n", + ) + cfg = wdconfig.load_slack_config(ini) + assert cfg.enabled is True + assert cfg.bot_token == "xoxb-xyz" + assert cfg.channel_id == "C42" + assert cfg.watchdog_enabled is True + + +def test_load_slack_config_defaults_when_missing(tmp_path): + ini = _write_ini(tmp_path / "c.ini", "[GENERAL]\nfoo = 1\n") + cfg = wdconfig.load_slack_config(ini) + assert cfg.bot_token is None and cfg.channel_id is None + assert cfg.watchdog_enabled is True # defaults to on when section absent + + assert wdconfig.load_slack_config(None).bot_token is None From 648f07c22cc5b33ce942082bf49492d6053e15ae Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 08:04:41 -0700 Subject: [PATCH 06/23] feat(watchdog): add Slack alert formatting Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/alerts.py | 49 +++++++++++++++++++ .../tests/acquisition_watchdog/test_alerts.py | 30 ++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 software/acquisition_watchdog/alerts.py create mode 100644 software/tests/acquisition_watchdog/test_alerts.py diff --git a/software/acquisition_watchdog/alerts.py b/software/acquisition_watchdog/alerts.py new file mode 100644 index 000000000..a4cbee647 --- /dev/null +++ b/software/acquisition_watchdog/alerts.py @@ -0,0 +1,49 @@ +# acquisition_watchdog/alerts.py +"""Format watchdog Slack alerts (text + Block Kit blocks).""" +from datetime import datetime, timezone +from typing import Optional, Tuple + +_KIND_TITLE = { + "crash": ":red_circle: Acquisition process died", + "hang": ":large_orange_circle: Acquisition hung (no heartbeat)", + "error": ":red_circle: Acquisition ended with a fatal error", + "completed_with_errors": ":large_orange_circle: Acquisition finished with errors", + "user_abort": ":large_yellow_circle: Acquisition aborted", +} + + +def _fmt_ts(epoch: Optional[float]) -> str: + if not epoch: + return "unknown" + return datetime.fromtimestamp(epoch, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC") + + +def _progress_line(run: dict) -> str: + prog = run.get("progress") or {} + expected = run.get("expected") or {} + tp = prog.get("timepoint", "?") + exp_tp = prog.get("expected_timepoints", expected.get("timepoints", "?")) + images = prog.get("images", "?") + return f"timepoint {tp}/{exp_tp}, {images} images" + + +def format_alert(kind: str, run: dict) -> Tuple[str, list]: + title = _KIND_TITLE.get(kind, f"Acquisition alert: {kind}") + experiment = run.get("experiment_id", "unknown") + machine = run.get("machine", "unknown") + text = f"{title}: {experiment} on {machine}" + + last_seen = run.get("ended_at") or run.get("heartbeat_at") + detail = ( + f"*Experiment:* {experiment}\n" + f"*Machine:* {machine}\n" + f"*Progress:* {_progress_line(run)}\n" + f"*Started:* {_fmt_ts(run.get('started_at'))}\n" + f"*Last seen:* {_fmt_ts(last_seen)}\n" + f"*Output:* {run.get('output_path', 'unknown')}" + ) + blocks = [ + {"type": "section", "text": {"type": "mrkdwn", "text": f"*{title}*"}}, + {"type": "section", "text": {"type": "mrkdwn", "text": detail}}, + ] + return text, blocks diff --git a/software/tests/acquisition_watchdog/test_alerts.py b/software/tests/acquisition_watchdog/test_alerts.py new file mode 100644 index 000000000..6acc33da4 --- /dev/null +++ b/software/tests/acquisition_watchdog/test_alerts.py @@ -0,0 +1,30 @@ +# tests/acquisition_watchdog/test_alerts.py +from acquisition_watchdog import alerts + + +def _run(): + return { + "experiment_id": "plateA_2026", + "machine": "micro-1", + "output_path": "/data/plateA_2026", + "progress": {"timepoint": 3, "expected_timepoints": 10, "images": 360}, + "expected": {"timepoints": 10}, + "started_at": 1_700_000_000.0, + "heartbeat_at": 1_700_000_100.0, + } + + +def test_format_alert_includes_key_facts(): + text, blocks = alerts.format_alert("crash", _run()) + assert "plateA_2026" in text + assert "micro-1" in text + blob = str(blocks) + assert "plateA_2026" in blob + assert "3" in blob and "10" in blob # progress vs expected + assert isinstance(blocks, list) and blocks + + +def test_format_alert_each_kind_has_title(): + for kind in ("crash", "hang", "error", "completed_with_errors", "user_abort"): + text, _ = alerts.format_alert(kind, _run()) + assert text # non-empty title line for every kind From a0dad7aede353e0dadbc57c776dc0825c0459886 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 08:08:45 -0700 Subject: [PATCH 07/23] feat(watchdog): add poll/classify/dedup monitor Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/monitor.py | 125 ++++++++++++++++++ .../acquisition_watchdog/test_monitor.py | 75 +++++++++++ 2 files changed, 200 insertions(+) create mode 100644 software/acquisition_watchdog/monitor.py create mode 100644 software/tests/acquisition_watchdog/test_monitor.py diff --git a/software/acquisition_watchdog/monitor.py b/software/acquisition_watchdog/monitor.py new file mode 100644 index 000000000..b9f7a8c9d --- /dev/null +++ b/software/acquisition_watchdog/monitor.py @@ -0,0 +1,125 @@ +# acquisition_watchdog/monitor.py +"""Poll the acquisition run-state and alert on premature ends.""" +import json +import os +import time +from pathlib import Path +from typing import Optional, Set + +import squid.acquisition_state as acquisition_state +import squid.logging +import squid.slack +from acquisition_watchdog import alerts, config + +_log = squid.logging.get_logger("acquisition_watchdog") + +ALERT_REASONS = {"completed_with_errors", "error", "user_abort"} + + +def pid_alive(pid: Optional[int]) -> bool: + if not pid: + return False + try: + import psutil + + return psutil.pid_exists(pid) + except ImportError: + pass + if os.name == "posix": + try: + os.kill(pid, 0) + return True + except ProcessLookupError: + return False + except PermissionError: + return True + except OSError: + return False + # Windows without psutil: cannot check reliably; rely on the heartbeat instead. + return True + + +class Monitor: + def __init__( + self, + state_dir: Optional[Path] = None, + cli_config: Optional[str] = None, + poll_interval: float = 5.0, + heartbeat_timeout: float = 120.0, + ): + self._state_dir = Path(state_dir) if state_dir else None + self._cli_config = cli_config + self._poll = poll_interval + self._timeout = heartbeat_timeout + base = self._state_dir or acquisition_state.default_state_dir() + self._alerted_path = base / "alerted.json" + self._alerted = self._load_alerted() + + def _load_alerted(self) -> Set[str]: + try: + with open(self._alerted_path) as f: + return set(json.load(f)) + except (FileNotFoundError, json.JSONDecodeError): + return set() + + def _save_alerted(self) -> None: + try: + self._alerted_path.parent.mkdir(parents=True, exist_ok=True) + with open(self._alerted_path, "w") as f: + json.dump(sorted(self._alerted), f) + except OSError as e: + _log.warning(f"Could not persist alerted set: {e}") + + def classify(self, run: Optional[dict], now: float) -> Optional[str]: + """Return an alert kind ('crash'|'hang'|) or None.""" + if not run or run.get("run_id") in self._alerted: + return None + status = run.get("status") + if status == "running": + if not pid_alive(run.get("pid")): + return "crash" + if (now - (run.get("heartbeat_at") or 0)) > self._timeout: + return "hang" + return None + if status == "ended" and run.get("reason") in ALERT_REASONS: + return run["reason"] + return None + + def check_once(self, now: float) -> None: + run = acquisition_state.read_run(self._state_dir) + kind = self.classify(run, now) + if kind is None: + return + + cfg_path = config.resolve_config_path(self._cli_config, run) + slack_cfg = config.load_slack_config(cfg_path) + if not (slack_cfg.bot_token and slack_cfg.channel_id and slack_cfg.watchdog_enabled): + _log.warning( + f"Premature end ({kind}) for run_id={run.get('run_id')} but Slack is not " + f"configured/enabled; not alerting." + ) + self._mark_alerted(run["run_id"]) + return + + text, blocks = alerts.format_alert(kind, run) + ok, _ = squid.slack.post_message(slack_cfg.bot_token, slack_cfg.channel_id, text, blocks) + if ok: + _log.info(f"Sent watchdog alert ({kind}) for run_id={run.get('run_id')}") + self._mark_alerted(run["run_id"]) + else: + # Leave unmarked so a transient Slack failure retries on the next poll. + _log.warning(f"Failed to send watchdog alert ({kind}) for run_id={run.get('run_id')}; will retry") + + def _mark_alerted(self, run_id: str) -> None: + self._alerted.add(run_id) + self._save_alerted() + + def run_forever(self) -> None: + base = self._state_dir or acquisition_state.default_state_dir() + _log.info(f"Acquisition watchdog started. state_dir={base} heartbeat_timeout={self._timeout}s") + while True: + try: + self.check_once(time.time()) + except Exception as e: + _log.exception(f"Watchdog poll error: {e}") + time.sleep(self._poll) diff --git a/software/tests/acquisition_watchdog/test_monitor.py b/software/tests/acquisition_watchdog/test_monitor.py new file mode 100644 index 000000000..a1dd2ae36 --- /dev/null +++ b/software/tests/acquisition_watchdog/test_monitor.py @@ -0,0 +1,75 @@ +# tests/acquisition_watchdog/test_monitor.py +import os +import time + +import squid.acquisition_state as ast +from acquisition_watchdog.config import SlackConfig +from acquisition_watchdog.monitor import Monitor + + +def _running(tmp_path, pid, heartbeat_age=0.0, run_id="r1"): + rec = { + "schema_version": 1, + "run_id": run_id, + "experiment_id": "e", + "machine": "m", + "pid": pid, + "config_path": None, + "output_path": "o", + "started_at": time.time() - 100, + "heartbeat_at": time.time() - heartbeat_age, + "progress": {}, + "expected": {}, + "status": "running", + "reason": None, + "ended_at": None, + "stats": None, + } + ast._atomic_write_json(ast.run_file_path(tmp_path), rec) + return rec + + +def _mon(tmp_path): + return Monitor(state_dir=tmp_path, heartbeat_timeout=120.0) + + +def test_running_with_live_pid_and_fresh_heartbeat_is_silent(tmp_path): + _running(tmp_path, pid=os.getpid(), heartbeat_age=1.0) + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) is None + + +def test_dead_pid_is_crash(tmp_path): + _running(tmp_path, pid=2_000_000_000, heartbeat_age=1.0) # impossible pid + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "crash" + + +def test_stale_heartbeat_with_live_pid_is_hang(tmp_path): + _running(tmp_path, pid=os.getpid(), heartbeat_age=999.0) + assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "hang" + + +def test_ended_reasons(tmp_path): + mon = _mon(tmp_path) + for reason, expect in [ + ("completed", None), + ("completed_with_errors", "completed_with_errors"), + ("error", "error"), + ("user_abort", "user_abort"), + ]: + run = {"run_id": f"x-{reason}", "status": "ended", "reason": reason} + assert mon.classify(run, time.time()) == expect + + +def test_dedup_persists_across_restart(tmp_path, monkeypatch): + _running(tmp_path, pid=2_000_000_000, run_id="dup1") + sent = [] + monkeypatch.setattr("squid.slack.post_message", lambda *a, **k: (sent.append(a) or (True, "1"))) + monkeypatch.setattr( + "acquisition_watchdog.config.load_slack_config", + lambda p: SlackConfig(True, "xoxb", "C1", True), + ) + Monitor(state_dir=tmp_path).check_once(time.time()) + assert len(sent) == 1 + # Fresh Monitor (simulated restart) must not re-alert the same run_id. + Monitor(state_dir=tmp_path).check_once(time.time()) + assert len(sent) == 1 From 0985a56c30071551ccfd587a188ace8cadccc2c1 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 09:45:51 -0700 Subject: [PATCH 08/23] feat(watchdog): add CLI entry point Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/__main__.py | 46 +++++++++++++++++++ .../tests/acquisition_watchdog/test_cli.py | 19 ++++++++ 2 files changed, 65 insertions(+) create mode 100644 software/acquisition_watchdog/__main__.py create mode 100644 software/tests/acquisition_watchdog/test_cli.py diff --git a/software/acquisition_watchdog/__main__.py b/software/acquisition_watchdog/__main__.py new file mode 100644 index 000000000..7259cb648 --- /dev/null +++ b/software/acquisition_watchdog/__main__.py @@ -0,0 +1,46 @@ +# acquisition_watchdog/__main__.py +"""CLI entry point: python -m acquisition_watchdog""" +import argparse +import time +from pathlib import Path +from typing import Optional, Sequence + +import squid.logging +from acquisition_watchdog.monitor import Monitor + + +def main(argv: Optional[Sequence[str]] = None) -> None: + parser = argparse.ArgumentParser( + prog="acquisition_watchdog", + description="Alert on prematurely-ended Squid acquisitions (crash/hang/abort/error).", + ) + parser.add_argument("--config", help="Path to the active configuration .ini ([SlackNotifications]).") + parser.add_argument("--state-dir", help="Override the watchdog state directory.") + parser.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between checks (default 5).") + parser.add_argument( + "--heartbeat-timeout", + type=float, + default=120.0, + help="Seconds of heartbeat silence (with a live PID) before declaring a hang (default 120).", + ) + parser.add_argument("--once", action="store_true", help="Run a single check and exit.") + args = parser.parse_args(argv) + + log = squid.logging.get_logger("acquisition_watchdog") + monitor = Monitor( + state_dir=Path(args.state_dir) if args.state_dir else None, + cli_config=args.config, + poll_interval=args.poll_interval, + heartbeat_timeout=args.heartbeat_timeout, + ) + if args.once: + monitor.check_once(time.time()) + else: + try: + monitor.run_forever() + except KeyboardInterrupt: + log.info("Acquisition watchdog stopped.") + + +if __name__ == "__main__": + main() diff --git a/software/tests/acquisition_watchdog/test_cli.py b/software/tests/acquisition_watchdog/test_cli.py new file mode 100644 index 000000000..3c5d7bc51 --- /dev/null +++ b/software/tests/acquisition_watchdog/test_cli.py @@ -0,0 +1,19 @@ +# tests/acquisition_watchdog/test_cli.py +from unittest.mock import patch + +from acquisition_watchdog.__main__ import main + + +def test_once_runs_single_check(tmp_path): + with patch("acquisition_watchdog.monitor.Monitor.check_once") as check, patch( + "acquisition_watchdog.monitor.Monitor.run_forever" + ) as forever: + main(["--once", "--state-dir", str(tmp_path)]) + check.assert_called_once() + forever.assert_not_called() + + +def test_default_runs_forever(tmp_path): + with patch("acquisition_watchdog.monitor.Monitor.run_forever") as forever: + main(["--state-dir", str(tmp_path)]) + forever.assert_called_once() From a5e38bb41ddda5ba10379913cc487d76619b8806 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 10:03:35 -0700 Subject: [PATCH 09/23] feat(watchdog): write acquisition start breadcrumb from the engine Co-Authored-By: Claude Opus 4.8 (1M context) --- .../control/core/multi_point_controller.py | 23 +++++++++++++++++++ software/control/core/multi_point_worker.py | 4 ++++ software/tests/control/conftest.py | 6 +++++ .../control/test_watchdog_breadcrumbs.py | 19 +++++++++++++++ 4 files changed, 52 insertions(+) create mode 100644 software/tests/control/test_watchdog_breadcrumbs.py diff --git a/software/control/core/multi_point_controller.py b/software/control/core/multi_point_controller.py index 680c22b8d..c44787027 100644 --- a/software/control/core/multi_point_controller.py +++ b/software/control/core/multi_point_controller.py @@ -28,6 +28,7 @@ from control.microcontroller import Microcontroller from control.piezo import PiezoStage from squid.abc import CameraFrame, AbstractCamera, AbstractStage +import squid.acquisition_state import squid.logging @@ -872,6 +873,27 @@ def finish_fn(): self.overlap_percent, ) + # Acquisition watchdog: drop the "running" breadcrumb (covers GUI + MCP-server runs). + self._run_state_writer = squid.acquisition_state.NullRunStateWriter() + try: + expected = { + "timepoints": self.Nt, + "regions": len(scan_position_information.scan_region_coords_mm), + "fovs": sum(len(c) for c in scan_position_information.scan_region_fov_coords_mm.values()), + "channels": len(self.selected_configurations), + "z": self.NZ, + } + config_path = (getattr(control._def, "CACHED_CONFIG_FILE_PATH", None) or "").strip() or None + self._run_state_writer = squid.acquisition_state.RunStateWriter.start( + experiment_id=self.experiment_ID, + pid=os.getpid(), + config_path=config_path, + output_path=experiment_path, + expected=expected, + ) + except Exception as e: + self._log.warning(f"Failed to write acquisition watchdog start state: {e}") + # Get pre-warmed job runner and its shared backpressure values # (starts a new one warming for next acquisition) prewarmed_runner, prewarmed_bp_values = self.get_prewarmed_job_runner() @@ -893,6 +915,7 @@ def finish_fn(): slack_notifier=self._slack_notifier, prewarmed_job_runner=prewarmed_runner, prewarmed_bp_values=prewarmed_bp_values, + run_state_writer=self._run_state_writer, ) except Exception: # Clean up pre-warmed runner if worker creation failed. diff --git a/software/control/core/multi_point_worker.py b/software/control/core/multi_point_worker.py index 9c834ed83..2a4380a97 100644 --- a/software/control/core/multi_point_worker.py +++ b/software/control/core/multi_point_worker.py @@ -30,6 +30,7 @@ from control.piezo import PiezoStage from control.models import AcquisitionChannel from squid.abc import AbstractCamera, CameraFrame, CameraFrameFormat +import squid.acquisition_state import squid.logging import control.core.job_processing from control.core.job_processing import ZarrWriteResult @@ -81,6 +82,7 @@ def __init__( slack_notifier=None, prewarmed_job_runner: Optional[JobRunner] = None, prewarmed_bp_values: Optional["BackpressureValues"] = None, + run_state_writer=None, ): self._log = squid.logging.get_logger(__class__.__name__) self._timing = utils.TimingManager("MultiPointWorker Timer Manager") @@ -110,6 +112,8 @@ def __init__( self.callbacks: MultiPointControllerFunctions = callbacks self.abort_requested_fn: Callable[[], bool] = abort_requested_fn self.request_abort_fn: Callable[[], None] = request_abort_fn + self._run_state = run_state_writer or squid.acquisition_state.NullRunStateWriter() + self._abort_cause = None # set to "error" by auto-abort paths (timeout / failed jobs) self.NZ = acquisition_parameters.NZ self.deltaZ = acquisition_parameters.deltaZ diff --git a/software/tests/control/conftest.py b/software/tests/control/conftest.py index 6ca8247f3..485098188 100644 --- a/software/tests/control/conftest.py +++ b/software/tests/control/conftest.py @@ -80,3 +80,9 @@ def firmware_sim_nonstrict(): sim = FirmwareSimSerial(strict=False) yield sim sim.close() + + +@pytest.fixture(autouse=True) +def _watchdog_state_to_tmp(tmp_path, monkeypatch): + # Keep acquisition breadcrumbs out of the real user state dir during tests. + monkeypatch.setenv("SQUID_WATCHDOG_STATE_DIR", str(tmp_path / "watchdog")) diff --git a/software/tests/control/test_watchdog_breadcrumbs.py b/software/tests/control/test_watchdog_breadcrumbs.py new file mode 100644 index 000000000..fc9b27daa --- /dev/null +++ b/software/tests/control/test_watchdog_breadcrumbs.py @@ -0,0 +1,19 @@ +# tests/control/test_watchdog_breadcrumbs.py +import os + +import squid.acquisition_state as ast +import control.microscope +import tests.control.gui_test_stubs as gts + + +def test_run_acquisition_writes_running_breadcrumb(qtbot): + scope = control.microscope.Microscope.build_from_global_config(True) + mpc = gts.get_test_qt_multi_point_controller(microscope=scope) + mpc.run_acquisition() + rec = ast.read_run(os.environ["SQUID_WATCHDOG_STATE_DIR"]) + assert rec is not None + assert rec["status"] == "running" + assert rec["pid"] == os.getpid() + assert rec["expected"]["timepoints"] >= 1 + mpc.request_abort_aquisition() + scope.close() From 7cceeba0ec9e1e843568309fe8453f4cc683f231 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 10:55:46 -0700 Subject: [PATCH 10/23] feat(watchdog): heartbeat + end-reason breadcrumb in acquisition worker Co-Authored-By: Claude Opus 4.8 (1M context) --- .../control/core/multi_point_controller.py | 4 ++ software/control/core/multi_point_worker.py | 41 ++++++++++++++++++- software/control/slack_notifier.py | 1 + software/tests/control/test_worker_reason.py | 41 +++++++++++++++++++ 4 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 software/tests/control/test_worker_reason.py diff --git a/software/control/core/multi_point_controller.py b/software/control/core/multi_point_controller.py index c44787027..56bca471a 100644 --- a/software/control/core/multi_point_controller.py +++ b/software/control/core/multi_point_controller.py @@ -693,6 +693,7 @@ def run_acquisition(self, acquire_current_fov=False): log_memory("ACQUISITION START", include_children=True) thread_started = False + self._run_state_writer = squid.acquisition_state.NullRunStateWriter() try: self._log.info("start multipoint") self._start_position = self.stage.get_pos() @@ -935,6 +936,9 @@ def finish_fn(): self.thread.start() finally: if not thread_started: + # Acquisition never launched a worker — close out the breadcrumb so the + # watchdog doesn't later misread the lingering "running" state as a hang. + self._run_state_writer.end("error", None) self._stop_per_acquisition_log() # Stop memory monitor if acquisition setup failed if self._memory_monitor is not None: diff --git a/software/control/core/multi_point_worker.py b/software/control/core/multi_point_worker.py index 2a4380a97..f9a156be6 100644 --- a/software/control/core/multi_point_worker.py +++ b/software/control/core/multi_point_worker.py @@ -428,8 +428,28 @@ def _is_well_based_acquisition(self) -> bool: ) return True + def _run_state_beat(self) -> None: + self._run_state.beat( + { + "timepoint": self.time_point, + "expected_timepoints": self.Nt, + "fov": self._timepoint_fov_count, + "images": self.image_count, + } + ) + + def _compute_end_reason(self) -> str: + if self._run_state_fatal: + return "error" + if self.abort_requested_fn(): + return "error" if self._abort_cause == "error" else "user_abort" + if self._acquisition_error_count > 0: + return "completed_with_errors" + return "completed" + def run(self): this_image_callback_id = None + self._run_state_fatal = False try: start_time = time.perf_counter_ns() self.camera.start_streaming() @@ -461,6 +481,7 @@ def run(self): if self.abort_requested_fn(): self._log.debug("In run, abort_acquisition_requested=True") break + self._run_state_beat() # Gate on laser engine readiness for the channels this acquisition will fire. # Re-checked every timepoint so dt-induced sleep gaps are handled. @@ -506,6 +527,7 @@ def run(self): if self.abort_requested_fn(): self._log.debug("In run wait loop, abort_acquisition_requested=True") break + self._run_state_beat() self._sleep(sleep_time) elapsed_time = time.perf_counter_ns() - start_time @@ -517,9 +539,11 @@ def run(self): except TimeoutError as te: self._log.error(f"Operation timed out during acquisition, aborting acquisition!") self._log.error(te) + self._abort_cause = "error" self.request_abort_fn() except Exception as e: self._log.exception(e) + self._run_state_fatal = True raise finally: # We do this above, but there are some paths that skip the proper end of the acquisition so make @@ -531,16 +555,29 @@ def run(self): self._finish_jobs() + # Determine why the acquisition ended (drives the watchdog + the in-process finish msg). + reason = self._compute_end_reason() + total_duration = time.time() - self.timestamp_acquisition_started + self._run_state.end( + reason, + { + "total_images": self.image_count, + "total_timepoints": self.time_point, + "total_duration_seconds": total_duration, + "errors_encountered": self._acquisition_error_count, + }, + ) + # Send Slack acquisition finished notification via callback (ensures ordering with timepoint notifications) if self._slack_notifier is not None: try: - total_duration = time.time() - self.timestamp_acquisition_started stats = AcquisitionStats( total_images=self.image_count, total_timepoints=self.time_point, total_duration_seconds=total_duration, errors_encountered=self._acquisition_error_count, experiment_id=self.experiment_ID or "unknown", + reason=reason, ) self.callbacks.signal_slack_acquisition_finished(stats) except Exception as e: @@ -1033,6 +1070,7 @@ def run_coordinate_acquisition(self, current_path): result = self._summarize_runner_outputs() if not result.none_failed and self._abort_on_failed_job: self._log.error("Some jobs failed, aborting acquisition because abort_on_failed_job=True") + self._abort_cause = "error" self.request_abort_fn() return @@ -1337,6 +1375,7 @@ def _image_callback(self, camera_frame: CameraFrame): # Increment image counter for Slack notification stats self._timepoint_image_count += 1 self.image_count += 1 + self._run_state_beat() with self._timing.get_timer("job creation and dispatch"): # Wait for subprocess to be ready before first dispatch diff --git a/software/control/slack_notifier.py b/software/control/slack_notifier.py index d5421240f..8437a2f18 100644 --- a/software/control/slack_notifier.py +++ b/software/control/slack_notifier.py @@ -52,6 +52,7 @@ class AcquisitionStats: total_duration_seconds: float errors_encountered: int experiment_id: str + reason: str = "completed" @dataclass diff --git a/software/tests/control/test_worker_reason.py b/software/tests/control/test_worker_reason.py new file mode 100644 index 000000000..61ed5cd70 --- /dev/null +++ b/software/tests/control/test_worker_reason.py @@ -0,0 +1,41 @@ +# tests/control/test_worker_reason.py +from control.core.multi_point_worker import MultiPointWorker + + +def _make_worker(): + # Bypass __init__; we only exercise _compute_end_reason()'s pure logic. + w = MultiPointWorker.__new__(MultiPointWorker) + w._run_state_fatal = False + w._abort_cause = None + w._acquisition_error_count = 0 + w.abort_requested_fn = lambda: False + return w + + +def test_reason_completed(): + assert _make_worker()._compute_end_reason() == "completed" + + +def test_reason_user_abort(): + w = _make_worker() + w.abort_requested_fn = lambda: True + assert w._compute_end_reason() == "user_abort" + + +def test_reason_error_on_timeout_abort(): + w = _make_worker() + w.abort_requested_fn = lambda: True + w._abort_cause = "error" + assert w._compute_end_reason() == "error" + + +def test_reason_error_on_fatal_exception(): + w = _make_worker() + w._run_state_fatal = True + assert w._compute_end_reason() == "error" + + +def test_reason_completed_with_errors(): + w = _make_worker() + w._acquisition_error_count = 3 + assert w._compute_end_reason() == "completed_with_errors" From ea615c415d062c6609ac769c97f7d1ca0d137bb2 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 21:47:32 -0700 Subject: [PATCH 11/23] feat(watchdog): notifier reports only clean finishes; watchdog owns premature alerts Co-Authored-By: Claude Opus 4.8 (1M context) --- software/control/slack_notifier.py | 3 ++ software/tests/control/test_notifier_trim.py | 30 ++++++++++++++++++++ 2 files changed, 33 insertions(+) create mode 100644 software/tests/control/test_notifier_trim.py diff --git a/software/control/slack_notifier.py b/software/control/slack_notifier.py index 8437a2f18..7dad6d564 100644 --- a/software/control/slack_notifier.py +++ b/software/control/slack_notifier.py @@ -562,6 +562,9 @@ def notify_acquisition_finished(self, stats: AcquisitionStats): """Send an acquisition completion notification to Slack.""" if not control._def.SlackNotifications.NOTIFY_ON_ACQUISITION_FINISHED: return + if stats.reason != "completed": + # Premature/degraded ends are reported once by the acquisition watchdog. + return duration_str = self._format_duration(stats.total_duration_seconds) timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") diff --git a/software/tests/control/test_notifier_trim.py b/software/tests/control/test_notifier_trim.py new file mode 100644 index 000000000..d8a8f50d0 --- /dev/null +++ b/software/tests/control/test_notifier_trim.py @@ -0,0 +1,30 @@ +# tests/control/test_notifier_trim.py +from unittest.mock import patch + +import control._def +from control.slack_notifier import SlackNotifier, AcquisitionStats + + +def _stats(reason): + return AcquisitionStats( + total_images=10, + total_timepoints=2, + total_duration_seconds=5.0, + errors_encountered=0, + experiment_id="e", + reason=reason, + ) + + +def test_finish_message_sent_only_on_clean_completion(monkeypatch): + monkeypatch.setattr(control._def.SlackNotifications, "NOTIFY_ON_ACQUISITION_FINISHED", True) + n = SlackNotifier(bot_token="x", channel_id="C") + with patch.object(n, "_queue_message") as q: + n.notify_acquisition_finished(_stats("completed")) + assert q.call_count == 1 + + with patch.object(n, "_queue_message") as q: + n.notify_acquisition_finished(_stats("error")) + n.notify_acquisition_finished(_stats("user_abort")) + n.notify_acquisition_finished(_stats("completed_with_errors")) + assert q.call_count == 0 # watchdog owns these alerts From 54efda8193d4e35c8b9c2c9558d0485a783f6d24 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 21:50:26 -0700 Subject: [PATCH 12/23] feat(watchdog): write an aborted breadcrumb when quitting mid-acquisition Co-Authored-By: Claude Opus 4.8 (1M context) --- software/main_hcs.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/software/main_hcs.py b/software/main_hcs.py index 47ee7b886..330b018f9 100644 --- a/software/main_hcs.py +++ b/software/main_hcs.py @@ -435,5 +435,18 @@ def launch_claude_code(): # All hardware cleanup (camera, stage, microcontroller) happens in closeEvent, # which completes before os._exit() is called. exit_code = app.exec_() + + # If the app is quitting mid-acquisition, request the normal abort and let the worker + # write its end breadcrumb so the watchdog reports "aborted" rather than a crash. + try: + mpc = getattr(win, "multipointController", None) + if mpc is not None and mpc.acquisition_in_progress(): + log.info("Acquisition in progress at shutdown; requesting abort before exit.") + mpc.request_abort_aquisition() + if getattr(mpc, "thread", None) is not None: + mpc.thread.join(timeout=15.0) + except Exception as e: + log.warning(f"Error during shutdown abort handling: {e}") + logging.shutdown() # Flush log handlers before os._exit() bypasses Python cleanup os._exit(exit_code) From c0f40e44d6b4c0a7cb008d7e4fce0d5ae15c9028 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 21:58:12 -0700 Subject: [PATCH 13/23] test(watchdog): end-to-end breadcrumb lifecycle in simulation Co-Authored-By: Claude Opus 4.8 (1M context) --- .../control/test_watchdog_integration.py | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 software/tests/control/test_watchdog_integration.py diff --git a/software/tests/control/test_watchdog_integration.py b/software/tests/control/test_watchdog_integration.py new file mode 100644 index 000000000..1cf35bd78 --- /dev/null +++ b/software/tests/control/test_watchdog_integration.py @@ -0,0 +1,36 @@ +# tests/control/test_watchdog_integration.py +import os +import time + +import squid.acquisition_state as ast +import control.microscope +import tests.control.gui_test_stubs as gts + + +def _wait_for(predicate, timeout=30.0, interval=0.2): + deadline = time.time() + timeout + while time.time() < deadline: + if predicate(): + return True + time.sleep(interval) + return False + + +def test_simulated_acquisition_writes_ended_breadcrumb(qtbot): + state_dir = os.environ["SQUID_WATCHDOG_STATE_DIR"] + scope = control.microscope.Microscope.build_from_global_config(True) + mpc = gts.get_test_qt_multi_point_controller(microscope=scope) + + mpc.run_acquisition() + assert _wait_for(lambda: ast.read_run(state_dir) is not None) + assert ast.read_run(state_dir)["status"] == "running" + + # Abort and confirm the worker writes the end breadcrumb (deterministic path). + mpc.request_abort_aquisition() + assert _wait_for(lambda: (ast.read_run(state_dir) or {}).get("status") == "ended", timeout=30.0) + + rec = ast.read_run(state_dir) + assert rec["status"] == "ended" + assert rec["reason"] in {"completed", "completed_with_errors", "user_abort", "error"} + assert rec["ended_at"] is not None + scope.close() From 93cc90bf3749ec8c9a24920da737690ff49b3a61 Mon Sep 17 00:00:00 2001 From: You Yan Date: Wed, 24 Jun 2026 22:44:15 -0700 Subject: [PATCH 14/23] docs(watchdog): add systemd + Windows service recipes and README Adds a .gitignore negation so the Windows Task Scheduler XML (a committed deployment artifact) is tracked despite the repo-wide *.xml ignore rule. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 ++ software/acquisition_watchdog/README.md | 33 ++++++++++++++++++ .../squid-acquisition-watchdog.service | 19 ++++++++++ .../acquisition_watchdog/windows/install.ps1 | 7 ++++ .../windows/squid-acquisition-watchdog.xml | Bin 0 -> 1858 bytes 5 files changed, 61 insertions(+) create mode 100644 software/acquisition_watchdog/README.md create mode 100644 software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service create mode 100644 software/acquisition_watchdog/windows/install.ps1 create mode 100644 software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml diff --git a/.gitignore b/.gitignore index 420f18149..7986d5c48 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,8 @@ .DS_Store *.pyc *.xml +# Acquisition watchdog Windows Task Scheduler definition (committed deployment artifact) +!software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml **/.idea/ .migration_complete diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md new file mode 100644 index 000000000..4e2b34243 --- /dev/null +++ b/software/acquisition_watchdog/README.md @@ -0,0 +1,33 @@ +# Acquisition Watchdog + +Independent process that alerts (via Slack) when a Squid acquisition ends +prematurely — process crash/hang/kill, fatal error, or user abort. Covers runs +launched from the GUI and from the MCP control server. + +## How it works +The Squid GUI writes a `run.json` breadcrumb (start / throttled heartbeat / end) +into a shared state dir. This watchdog polls it and posts one Slack alert when a +run dies, hangs, or ends with a non-clean reason. Clean completions are silent. + +## Run it + cd software + python3 -m acquisition_watchdog --config ./configuration.ini + +Options: `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. + +Slack credentials are read from the `[SlackNotifications]` section of the active +`.ini` (same `bot_token` / `channel_id` the GUI uses). Set `watchdog_enabled = False` +in that section to disable watchdog alerts on a machine. + +## Install as an always-on service +- **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). +- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). + +## State dir +Defaults to `platformdirs.user_state_path("squid","cephla")/watchdog`. Override with +`SQUID_WATCHDOG_STATE_DIR` (must match the GUI's environment) or `--state-dir`. + +## Remote / power-loss coverage (future) +Point `--state-dir` at a shared/synced mount on another host and run this process +there. Per-machine `run-.json` naming and clock-skew tolerance are needed +first (see the design spec, "Future work"). diff --git a/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service b/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service new file mode 100644 index 000000000..605e48ca9 --- /dev/null +++ b/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service @@ -0,0 +1,19 @@ +# Install (per user): +# mkdir -p ~/.config/systemd/user +# cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/ +# # edit WorkingDirectory + --config below to match this machine, then: +# systemctl --user daemon-reload +# systemctl --user enable --now squid-acquisition-watchdog +[Unit] +Description=Squid acquisition watchdog (alerts on prematurely-ended acquisitions) +After=default.target + +[Service] +Type=simple +WorkingDirectory=%h/Squid/software +ExecStart=/usr/bin/python3 -m acquisition_watchdog --config %h/Squid/software/configuration.ini +Restart=always +RestartSec=5 + +[Install] +WantedBy=default.target diff --git a/software/acquisition_watchdog/windows/install.ps1 b/software/acquisition_watchdog/windows/install.ps1 new file mode 100644 index 000000000..ee4bb051b --- /dev/null +++ b/software/acquisition_watchdog/windows/install.ps1 @@ -0,0 +1,7 @@ +# Run in PowerShell from software\ : .\acquisition_watchdog\windows\install.ps1 +$ErrorActionPreference = "Stop" +$taskName = "SquidAcquisitionWatchdog" +$xmlPath = Join-Path $PSScriptRoot "squid-acquisition-watchdog.xml" +Write-Host "Registering scheduled task '$taskName' from $xmlPath" +Register-ScheduledTask -TaskName $taskName -Xml (Get-Content $xmlPath -Raw) -Force +Write-Host "Done. Edit the task's --config/WorkingDirectory if your install path differs, then log off/on or 'Start' the task." diff --git a/software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml b/software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml new file mode 100644 index 0000000000000000000000000000000000000000..bacfe282b003319b13bd9f1fe783d55e55df98a9 GIT binary patch literal 1858 zcmbW2UvJY;48{G-r-=3h5Bj>+qyg>EHfLjKEgV))UDsF zXD8Oe>oq=yme@0kSsM`J*VcIZ9xq_d@D$Fam0WW>unt!N(j|yfkJRQlVI>E>;J(1K z?Odk#E}hSa^(EGh?ZdZguXyY7-toO9&+?Adp0kW#7uy#wQuY^EbL=JWQhv&P!JG8R zGk724EoM1h^LyXKzS?hFur|VPfIWOEi&eY`Zi+5dVw&RloRy4-IqL=A!i|WV-{~%i zsrStOkYQzi#P4>D!d{@LW7}8;CGvQzYE@ENQ?Ae#tNA!{CE~s6*YpaR`+(2De!AWf zSyi?d#1xx#TQlPw+l8SBtLVjj)nY0$21`{?$JUm>>KqQ4Tj$P9RaYkrcn;ZF*k=@G z>>g5sgsw=jTzOSKVqZG{r$%Y&J0^DFeUK4%316YBBTnhRPoTZyTMCr_iE~P@sB;Rw zLcRYdMw5m5A?OyeP)|UZfDt-yv!2RS;B@U=5;E0e3Y2!7GXEa63y+c58^5Ha zxmxd#$@Jra$<$(pCcS2$ zPJcvY4y=#rYR>hE8|JX)nx<5LWrzBvo~CW0%(%t>A%5!`z7sp=|3kY{=Gnv%lvl0m bB<|)Ttle>HWcjbxC-7abr&g Date: Wed, 24 Jun 2026 23:03:19 -0700 Subject: [PATCH 15/23] docs(watchdog): replace Windows Task XML with a self-contained install.ps1 Removes the binary UTF-16 Task Scheduler XML and the repo-root .gitignore negation it required; install.ps1 now builds the task inline via New-ScheduledTask* cmdlets. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 - .../acquisition_watchdog/windows/install.ps1 | 40 +++++++++++++++--- .../windows/squid-acquisition-watchdog.xml | Bin 1858 -> 0 bytes 3 files changed, 34 insertions(+), 8 deletions(-) delete mode 100644 software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml diff --git a/.gitignore b/.gitignore index 7986d5c48..420f18149 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,6 @@ .DS_Store *.pyc *.xml -# Acquisition watchdog Windows Task Scheduler definition (committed deployment artifact) -!software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml **/.idea/ .migration_complete diff --git a/software/acquisition_watchdog/windows/install.ps1 b/software/acquisition_watchdog/windows/install.ps1 index ee4bb051b..da549e7b5 100644 --- a/software/acquisition_watchdog/windows/install.ps1 +++ b/software/acquisition_watchdog/windows/install.ps1 @@ -1,7 +1,35 @@ -# Run in PowerShell from software\ : .\acquisition_watchdog\windows\install.ps1 +# Run in PowerShell (as the user who runs the Squid GUI), from software\ : +# .\acquisition_watchdog\windows\install.ps1 +# Registers a logon-triggered scheduled task that runs the acquisition watchdog. +# Edit $workingDir / $configPath below if your install path differs. $ErrorActionPreference = "Stop" -$taskName = "SquidAcquisitionWatchdog" -$xmlPath = Join-Path $PSScriptRoot "squid-acquisition-watchdog.xml" -Write-Host "Registering scheduled task '$taskName' from $xmlPath" -Register-ScheduledTask -TaskName $taskName -Xml (Get-Content $xmlPath -Raw) -Force -Write-Host "Done. Edit the task's --config/WorkingDirectory if your install path differs, then log off/on or 'Start' the task." + +$taskName = "SquidAcquisitionWatchdog" +$workingDir = "C:\Squid\software" +$configPath = "C:\Squid\software\configuration.ini" + +$action = New-ScheduledTaskAction ` + -Execute "pythonw.exe" ` + -Argument "-m acquisition_watchdog --config `"$configPath`"" ` + -WorkingDirectory $workingDir + +$trigger = New-ScheduledTaskTrigger -AtLogOn + +$settings = New-ScheduledTaskSettingsSet ` + -MultipleInstances IgnoreNew ` + -RestartInterval (New-TimeSpan -Minutes 1) ` + -RestartCount 999 ` + -ExecutionTimeLimit (New-TimeSpan -Seconds 0) ` + -AllowStartIfOnBatteries ` + -DontStopIfGoingOnBatteries + +Write-Host "Registering scheduled task '$taskName'..." +Register-ScheduledTask ` + -TaskName $taskName ` + -Action $action ` + -Trigger $trigger ` + -Settings $settings ` + -Description "Squid acquisition watchdog (alerts on prematurely-ended acquisitions)" ` + -Force + +Write-Host "Done. It starts at next logon. Run now with: Start-ScheduledTask -TaskName $taskName" diff --git a/software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml b/software/acquisition_watchdog/windows/squid-acquisition-watchdog.xml deleted file mode 100644 index bacfe282b003319b13bd9f1fe783d55e55df98a9..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1858 zcmbW2UvJY;48{G-r-=3h5Bj>+qyg>EHfLjKEgV))UDsF zXD8Oe>oq=yme@0kSsM`J*VcIZ9xq_d@D$Fam0WW>unt!N(j|yfkJRQlVI>E>;J(1K z?Odk#E}hSa^(EGh?ZdZguXyY7-toO9&+?Adp0kW#7uy#wQuY^EbL=JWQhv&P!JG8R zGk724EoM1h^LyXKzS?hFur|VPfIWOEi&eY`Zi+5dVw&RloRy4-IqL=A!i|WV-{~%i zsrStOkYQzi#P4>D!d{@LW7}8;CGvQzYE@ENQ?Ae#tNA!{CE~s6*YpaR`+(2De!AWf zSyi?d#1xx#TQlPw+l8SBtLVjj)nY0$21`{?$JUm>>KqQ4Tj$P9RaYkrcn;ZF*k=@G z>>g5sgsw=jTzOSKVqZG{r$%Y&J0^DFeUK4%316YBBTnhRPoTZyTMCr_iE~P@sB;Rw zLcRYdMw5m5A?OyeP)|UZfDt-yv!2RS;B@U=5;E0e3Y2!7GXEa63y+c58^5Ha zxmxd#$@Jra$<$(pCcS2$ zPJcvY4y=#rYR>hE8|JX)nx<5LWrzBvo~CW0%(%t>A%5!`z7sp=|3kY{=Gnv%lvl0m bB<|)Ttle>HWcjbxC-7abr&g Date: Thu, 25 Jun 2026 12:33:06 -0700 Subject: [PATCH 16/23] docs(watchdog): note pythonw.exe PATH requirement for Windows install Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md index 4e2b34243..e69263c1d 100644 --- a/software/acquisition_watchdog/README.md +++ b/software/acquisition_watchdog/README.md @@ -21,7 +21,9 @@ in that section to disable watchdog alerts on a machine. ## Install as an always-on service - **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). -- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). +- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). Ensure + `pythonw.exe` is on `PATH`, or edit the `-Execute` value in `install.ps1` to the full + Python path. ## State dir Defaults to `platformdirs.user_state_path("squid","cephla")/watchdog`. Override with From be8ba6ac5eb919a48b72a151ab4a1d1ea00d4d48 Mon Sep 17 00:00:00 2001 From: You Yan Date: Thu, 25 Jun 2026 14:35:34 -0700 Subject: [PATCH 17/23] fix(watchdog): read Slack credentials from cache/slack_settings.yaml (the GUI's real source) The watchdog previously read a non-existent [SlackNotifications] .ini section; real credentials live in cache/slack_settings.yaml (bot_token/channel_id/enabled), written by the GUI Slack dialog. Without this the watchdog never alerts. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/README.md | 13 ++-- software/acquisition_watchdog/__main__.py | 8 ++- software/acquisition_watchdog/config.py | 69 +++++++++---------- software/acquisition_watchdog/monitor.py | 6 +- .../squid-acquisition-watchdog.service | 4 +- .../acquisition_watchdog/windows/install.ps1 | 5 +- .../2026-06-23-acquisition-watchdog-design.md | 27 ++++---- .../tests/acquisition_watchdog/test_config.py | 51 ++++++++------ 8 files changed, 96 insertions(+), 87 deletions(-) diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md index e69263c1d..4d62c285a 100644 --- a/software/acquisition_watchdog/README.md +++ b/software/acquisition_watchdog/README.md @@ -11,13 +11,16 @@ run dies, hangs, or ends with a non-clean reason. Clean completions are silent. ## Run it cd software - python3 -m acquisition_watchdog --config ./configuration.ini + python3 -m acquisition_watchdog -Options: `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. +Options: `--slack-settings`, `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. -Slack credentials are read from the `[SlackNotifications]` section of the active -`.ini` (same `bot_token` / `channel_id` the GUI uses). Set `watchdog_enabled = False` -in that section to disable watchdog alerts on a machine. +Slack credentials are read from `cache/slack_settings.yaml` — the same file the GUI's +Slack settings dialog writes (keys `bot_token`, `channel_id`, `enabled`). Run the +watchdog from the `software/` directory (so the default `cache/slack_settings.yaml` +path resolves), or pass `--slack-settings `. To disable watchdog alerts on a +machine without disabling the GUI's notifications, add `watchdog_enabled: false` to +that YAML. ## Install as an always-on service - **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). diff --git a/software/acquisition_watchdog/__main__.py b/software/acquisition_watchdog/__main__.py index 7259cb648..dcbcb9aaa 100644 --- a/software/acquisition_watchdog/__main__.py +++ b/software/acquisition_watchdog/__main__.py @@ -14,7 +14,11 @@ def main(argv: Optional[Sequence[str]] = None) -> None: prog="acquisition_watchdog", description="Alert on prematurely-ended Squid acquisitions (crash/hang/abort/error).", ) - parser.add_argument("--config", help="Path to the active configuration .ini ([SlackNotifications]).") + parser.add_argument( + "--slack-settings", + help="Path to the Slack settings YAML (defaults to ./cache/slack_settings.yaml, " + "the same file the GUI writes).", + ) parser.add_argument("--state-dir", help="Override the watchdog state directory.") parser.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between checks (default 5).") parser.add_argument( @@ -29,7 +33,7 @@ def main(argv: Optional[Sequence[str]] = None) -> None: log = squid.logging.get_logger("acquisition_watchdog") monitor = Monitor( state_dir=Path(args.state_dir) if args.state_dir else None, - cli_config=args.config, + slack_settings=args.slack_settings, poll_interval=args.poll_interval, heartbeat_timeout=args.heartbeat_timeout, ) diff --git a/software/acquisition_watchdog/config.py b/software/acquisition_watchdog/config.py index a00fee7e2..697ed1801 100644 --- a/software/acquisition_watchdog/config.py +++ b/software/acquisition_watchdog/config.py @@ -1,12 +1,18 @@ # acquisition_watchdog/config.py -"""Resolve the active Squid .ini and read its [SlackNotifications] section, -without importing the heavy control._def module. +"""Load Slack credentials from the same source the Squid GUI uses. + +The GUI stores Slack settings (bot token, channel, enabled) in +`cache/slack_settings.yaml` (written by the Slack settings dialog and loaded at +GUI startup via control.widgets_slack.load_slack_settings_from_cache). This module +reads that same YAML so the watchdog alerts to the same workspace — without +importing the heavy control stack. """ -import configparser import os from pathlib import Path from typing import NamedTuple, Optional +import yaml + class SlackConfig(NamedTuple): enabled: bool @@ -15,46 +21,33 @@ class SlackConfig(NamedTuple): watchdog_enabled: bool -def resolve_config_path(cli_config: Optional[str], run_record: Optional[dict]) -> Optional[Path]: - """Priority: --config > $SQUID_CONFIG > run.json config_path > cache pointer.""" - if cli_config: - return Path(cli_config) - env = os.environ.get("SQUID_CONFIG") +DEFAULT_SLACK_SETTINGS = "cache/slack_settings.yaml" + + +def resolve_slack_settings_path(cli_path: Optional[str]) -> Path: + """Priority: --slack-settings > $SQUID_SLACK_SETTINGS > cache/slack_settings.yaml (cwd-relative).""" + if cli_path: + return Path(cli_path) + env = os.environ.get("SQUID_SLACK_SETTINGS") if env: return Path(env) - if run_record and run_record.get("config_path"): - return Path(run_record["config_path"]) - cache = Path("cache/config_file_path.txt") - if cache.exists(): - first = cache.read_text().splitlines() - if first: - return Path(first[0].strip()) - return None - - -def load_slack_config(config_path: Optional[Path]) -> SlackConfig: - if not config_path or not Path(config_path).exists(): + return Path(DEFAULT_SLACK_SETTINGS) + + +def load_slack_config(path: Optional[Path]) -> SlackConfig: + p = Path(path) if path else Path(DEFAULT_SLACK_SETTINGS) + if not p.exists(): return SlackConfig(False, None, None, True) - cp = configparser.ConfigParser() try: - cp.read(config_path) - except configparser.Error: + with open(p) as f: + data = yaml.safe_load(f) or {} + except Exception: return SlackConfig(False, None, None, True) - if not cp.has_section("SLACKNOTIFICATIONS"): + if not isinstance(data, dict): return SlackConfig(False, None, None, True) - sec = cp["SLACKNOTIFICATIONS"] - - def getbool(key: str, default: bool) -> bool: - try: - return sec.getboolean(key, default) - except ValueError: - return default - - token = sec.get("bot_token", fallback=None) or None - channel = sec.get("channel_id", fallback=None) or None return SlackConfig( - enabled=getbool("enabled", False), - bot_token=token, - channel_id=channel, - watchdog_enabled=getbool("watchdog_enabled", True), + enabled=bool(data.get("enabled", False)), + bot_token=(data.get("bot_token") or None), + channel_id=(data.get("channel_id") or None), + watchdog_enabled=bool(data.get("watchdog_enabled", True)), ) diff --git a/software/acquisition_watchdog/monitor.py b/software/acquisition_watchdog/monitor.py index b9f7a8c9d..e567b2018 100644 --- a/software/acquisition_watchdog/monitor.py +++ b/software/acquisition_watchdog/monitor.py @@ -43,12 +43,12 @@ class Monitor: def __init__( self, state_dir: Optional[Path] = None, - cli_config: Optional[str] = None, + slack_settings: Optional[str] = None, poll_interval: float = 5.0, heartbeat_timeout: float = 120.0, ): self._state_dir = Path(state_dir) if state_dir else None - self._cli_config = cli_config + self._slack_settings = slack_settings self._poll = poll_interval self._timeout = heartbeat_timeout base = self._state_dir or acquisition_state.default_state_dir() @@ -91,7 +91,7 @@ def check_once(self, now: float) -> None: if kind is None: return - cfg_path = config.resolve_config_path(self._cli_config, run) + cfg_path = config.resolve_slack_settings_path(self._slack_settings) slack_cfg = config.load_slack_config(cfg_path) if not (slack_cfg.bot_token and slack_cfg.channel_id and slack_cfg.watchdog_enabled): _log.warning( diff --git a/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service b/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service index 605e48ca9..3749cc194 100644 --- a/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service +++ b/software/acquisition_watchdog/systemd/squid-acquisition-watchdog.service @@ -1,7 +1,7 @@ # Install (per user): # mkdir -p ~/.config/systemd/user # cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/ -# # edit WorkingDirectory + --config below to match this machine, then: +# # edit WorkingDirectory below to match this machine, then: # systemctl --user daemon-reload # systemctl --user enable --now squid-acquisition-watchdog [Unit] @@ -11,7 +11,7 @@ After=default.target [Service] Type=simple WorkingDirectory=%h/Squid/software -ExecStart=/usr/bin/python3 -m acquisition_watchdog --config %h/Squid/software/configuration.ini +ExecStart=/usr/bin/python3 -m acquisition_watchdog Restart=always RestartSec=5 diff --git a/software/acquisition_watchdog/windows/install.ps1 b/software/acquisition_watchdog/windows/install.ps1 index da549e7b5..36fe1ab49 100644 --- a/software/acquisition_watchdog/windows/install.ps1 +++ b/software/acquisition_watchdog/windows/install.ps1 @@ -1,16 +1,15 @@ # Run in PowerShell (as the user who runs the Squid GUI), from software\ : # .\acquisition_watchdog\windows\install.ps1 # Registers a logon-triggered scheduled task that runs the acquisition watchdog. -# Edit $workingDir / $configPath below if your install path differs. +# Edit $workingDir below if your install path differs. $ErrorActionPreference = "Stop" $taskName = "SquidAcquisitionWatchdog" $workingDir = "C:\Squid\software" -$configPath = "C:\Squid\software\configuration.ini" $action = New-ScheduledTaskAction ` -Execute "pythonw.exe" ` - -Argument "-m acquisition_watchdog --config `"$configPath`"" ` + -Argument "-m acquisition_watchdog" ` -WorkingDirectory $workingDir $trigger = New-ScheduledTaskTrigger -AtLogOn diff --git a/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md b/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md index a7ce2cb1f..70e77f61b 100644 --- a/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md +++ b/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md @@ -37,8 +37,8 @@ Three parts, with a clean dependency DAG (`acquisition_watchdog` → `squid`; `c └───────────────────────────────────────────────┘ reads acquisition_watchdog (independent always-on process) ◄──────── /run.json - poll every ~5s → classify → Slack alert (once per run_id) reads [SlackNotifications] - from the active .ini + poll every ~5s → classify → Slack alert (once per run_id) reads bot_token/channel_id + from cache/slack_settings.yaml ``` ### Part 1 — Breadcrumb protocol (in the acquisition engine) @@ -153,18 +153,17 @@ both the notifier and the watchdog. Image upload (`files.getUploadURLExternal`) ## Config sharing -The watchdog reads the **same `[SlackNotifications]`** the GUI uses (token, channel, -enabled). Resolution order: `--config` flag → `SQUID_CONFIG` env → -`run.json.config_path` (written by the GUI, so the watchdog auto-discovers the active -`.ini` with no args) → `cache/config_file_path.txt`. Parsed with stdlib `configparser`; -the watchdog never imports `control._def`. One new opt-out key, -`[SlackNotifications] watchdog_enabled` (default `True` when a token+channel are set), lets -a machine disable watchdog alerts without disabling the in-process notifier. +The watchdog reads the **same `cache/slack_settings.yaml`** the GUI writes and loads +(`bot_token` / `channel_id` / `enabled`), resolved cwd-relative or overridden via the +`--slack-settings` flag or `$SQUID_SLACK_SETTINGS` env. It is parsed with `yaml`; the +watchdog never imports `control._def`. A `watchdog_enabled: false` key (default `true`) +disables watchdog alerts on a machine without disabling the in-process GUI notifier. ## Deployment — always-on user service -Core process is just `python -m acquisition_watchdog [--config ]`, identical on both -OSes. Shipped recipes: +Core process is just `python -m acquisition_watchdog [--slack-settings ]`, identical on +both OSes (run from `software/` so the default `cache/slack_settings.yaml` resolves). Shipped +recipes: - **Linux:** a systemd **`--user`** unit (`Restart=always`, `WantedBy=default.target`), `systemctl --user enable --now squid-acquisition-watchdog`. Runs as the same user as the @@ -184,7 +183,7 @@ mount — see Future work. | `squid/slack.py` | shared dependency-free `chat.postMessage` sender | | `software/acquisition_watchdog/__main__.py` | CLI entry (`python -m acquisition_watchdog`) | | `software/acquisition_watchdog/monitor.py` | poll loop + classification + dedup | -| `software/acquisition_watchdog/config.py` | resolve active `.ini`, load `[SlackNotifications]` | +| `software/acquisition_watchdog/config.py` | resolve & load `cache/slack_settings.yaml` (the GUI's Slack creds) | | `software/acquisition_watchdog/alerts.py` | format the Slack alert payload | | `software/acquisition_watchdog/systemd/`, `windows/`, `README.md` | install recipes + docs | | `control/core/multi_point_controller.py`, `control/core/multi_point_worker.py` | write breadcrumbs (start / beat / end) + abort-cause tagging | @@ -202,8 +201,8 @@ filesystem-events package. state (`running`+stale heartbeat, `running`+dead PID, each `ended` reason, `completed`) → expected alert/no-alert; dedup (no double alert per `run_id`, persists across a monitor restart via `alerted.json`). -- `acquisition_watchdog/config.py`: resolution precedence (`--config` > env > - `run.json.config_path` > cache pointer); missing/disabled Slack → logs, no crash. +- `acquisition_watchdog/config.py`: resolution precedence (`--slack-settings` > env > + default `cache/slack_settings.yaml`); missing/disabled Slack → logs, no crash. - `squid/slack.py`: monkeypatch `urllib`, assert request shape; no network. - PID check: alive (current pid) vs an impossible/known-dead pid, on the available platform; graceful degrade when `psutil` absent. diff --git a/software/tests/acquisition_watchdog/test_config.py b/software/tests/acquisition_watchdog/test_config.py index 791617cda..04f904985 100644 --- a/software/tests/acquisition_watchdog/test_config.py +++ b/software/tests/acquisition_watchdog/test_config.py @@ -1,39 +1,50 @@ # tests/acquisition_watchdog/test_config.py from pathlib import Path +import yaml + from acquisition_watchdog import config as wdconfig -def _write_ini(path, body): - path.write_text(body) +def _write_yaml(path, data): + path.write_text(yaml.safe_dump(data)) return path -def test_resolve_prefers_cli_then_env_then_run_record(tmp_path, monkeypatch): - monkeypatch.delenv("SQUID_CONFIG", raising=False) - assert wdconfig.resolve_config_path("/cli.ini", {"config_path": "/run.ini"}) == Path("/cli.ini") - monkeypatch.setenv("SQUID_CONFIG", "/env.ini") - assert wdconfig.resolve_config_path(None, {"config_path": "/run.ini"}) == Path("/env.ini") - monkeypatch.delenv("SQUID_CONFIG", raising=False) - assert wdconfig.resolve_config_path(None, {"config_path": "/run.ini"}) == Path("/run.ini") +def test_resolve_prefers_cli_then_env_then_default(monkeypatch): + monkeypatch.delenv("SQUID_SLACK_SETTINGS", raising=False) + assert wdconfig.resolve_slack_settings_path("/cli.yaml") == Path("/cli.yaml") + monkeypatch.setenv("SQUID_SLACK_SETTINGS", "/env.yaml") + assert wdconfig.resolve_slack_settings_path(None) == Path("/env.yaml") + monkeypatch.delenv("SQUID_SLACK_SETTINGS", raising=False) + assert wdconfig.resolve_slack_settings_path(None) == Path(wdconfig.DEFAULT_SLACK_SETTINGS) -def test_load_slack_config_reads_section(tmp_path): - ini = _write_ini( - tmp_path / "c.ini", - "[SLACKNOTIFICATIONS]\nenabled = True\nbot_token = xoxb-xyz\nchannel_id = C42\nwatchdog_enabled = True\n", +def test_load_reads_credentials_from_yaml(tmp_path): + p = _write_yaml( + tmp_path / "slack_settings.yaml", + {"enabled": True, "bot_token": "xoxb-xyz", "channel_id": "C42"}, ) - cfg = wdconfig.load_slack_config(ini) + cfg = wdconfig.load_slack_config(p) assert cfg.enabled is True assert cfg.bot_token == "xoxb-xyz" assert cfg.channel_id == "C42" - assert cfg.watchdog_enabled is True + assert cfg.watchdog_enabled is True # default when key absent -def test_load_slack_config_defaults_when_missing(tmp_path): - ini = _write_ini(tmp_path / "c.ini", "[GENERAL]\nfoo = 1\n") - cfg = wdconfig.load_slack_config(ini) - assert cfg.bot_token is None and cfg.channel_id is None - assert cfg.watchdog_enabled is True # defaults to on when section absent +def test_load_watchdog_enabled_false(tmp_path): + p = _write_yaml(tmp_path / "s.yaml", {"bot_token": "x", "channel_id": "C", "watchdog_enabled": False}) + assert wdconfig.load_slack_config(p).watchdog_enabled is False + +def test_load_defaults_when_missing(tmp_path): + cfg = wdconfig.load_slack_config(tmp_path / "nope.yaml") + assert cfg.bot_token is None and cfg.channel_id is None + assert cfg.watchdog_enabled is True assert wdconfig.load_slack_config(None).bot_token is None + + +def test_load_empty_strings_become_none(tmp_path): + p = _write_yaml(tmp_path / "s.yaml", {"bot_token": "", "channel_id": ""}) + cfg = wdconfig.load_slack_config(p) + assert cfg.bot_token is None and cfg.channel_id is None From a816c600acc684210501497b20398438501f198e Mon Sep 17 00:00:00 2001 From: You Yan Date: Thu, 25 Jun 2026 20:40:30 -0700 Subject: [PATCH 18/23] refactor(watchdog): cleanup from /simplify review - Generalize the worker's self-abort into _abort_due_to_error(): every worker self-abort is an error (user aborts arrive via the external flag), so tag the cause in one helper. This also fixes 6 error-abort paths (null capture info, null frame, job dispatch/exec failure, frame-wait timeouts) that previously left _abort_cause unset and were misclassified as user_abort. - Drop the dead SlackConfig.enabled field (the watchdog gates on watchdog_enabled, independent of the GUI's enabled toggle). - Remove the redundant NullRunStateWriter() re-init inside run_acquisition (the pre-try guard already covers the failure path). - Cache Monitor._base instead of recomputing default_state_dir(); drop the redundant per-heartbeat expected_timepoints field (expected.timepoints is authoritative); update progress only after the beat() throttle check. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/config.py | 8 ++--- software/acquisition_watchdog/monitor.py | 7 ++--- .../control/core/multi_point_controller.py | 3 +- software/control/core/multi_point_worker.py | 29 ++++++++++++------- software/squid/acquisition_state.py | 4 +-- .../tests/acquisition_watchdog/test_config.py | 1 - .../acquisition_watchdog/test_monitor.py | 2 +- 7 files changed, 29 insertions(+), 25 deletions(-) diff --git a/software/acquisition_watchdog/config.py b/software/acquisition_watchdog/config.py index 697ed1801..090096a99 100644 --- a/software/acquisition_watchdog/config.py +++ b/software/acquisition_watchdog/config.py @@ -15,7 +15,6 @@ class SlackConfig(NamedTuple): - enabled: bool bot_token: Optional[str] channel_id: Optional[str] watchdog_enabled: bool @@ -37,16 +36,15 @@ def resolve_slack_settings_path(cli_path: Optional[str]) -> Path: def load_slack_config(path: Optional[Path]) -> SlackConfig: p = Path(path) if path else Path(DEFAULT_SLACK_SETTINGS) if not p.exists(): - return SlackConfig(False, None, None, True) + return SlackConfig(None, None, True) try: with open(p) as f: data = yaml.safe_load(f) or {} except Exception: - return SlackConfig(False, None, None, True) + return SlackConfig(None, None, True) if not isinstance(data, dict): - return SlackConfig(False, None, None, True) + return SlackConfig(None, None, True) return SlackConfig( - enabled=bool(data.get("enabled", False)), bot_token=(data.get("bot_token") or None), channel_id=(data.get("channel_id") or None), watchdog_enabled=bool(data.get("watchdog_enabled", True)), diff --git a/software/acquisition_watchdog/monitor.py b/software/acquisition_watchdog/monitor.py index e567b2018..5f49d3f79 100644 --- a/software/acquisition_watchdog/monitor.py +++ b/software/acquisition_watchdog/monitor.py @@ -51,8 +51,8 @@ def __init__( self._slack_settings = slack_settings self._poll = poll_interval self._timeout = heartbeat_timeout - base = self._state_dir or acquisition_state.default_state_dir() - self._alerted_path = base / "alerted.json" + self._base = self._state_dir or acquisition_state.default_state_dir() + self._alerted_path = self._base / "alerted.json" self._alerted = self._load_alerted() def _load_alerted(self) -> Set[str]: @@ -115,8 +115,7 @@ def _mark_alerted(self, run_id: str) -> None: self._save_alerted() def run_forever(self) -> None: - base = self._state_dir or acquisition_state.default_state_dir() - _log.info(f"Acquisition watchdog started. state_dir={base} heartbeat_timeout={self._timeout}s") + _log.info(f"Acquisition watchdog started. state_dir={self._base} heartbeat_timeout={self._timeout}s") while True: try: self.check_once(time.time()) diff --git a/software/control/core/multi_point_controller.py b/software/control/core/multi_point_controller.py index 56bca471a..9f117d989 100644 --- a/software/control/core/multi_point_controller.py +++ b/software/control/core/multi_point_controller.py @@ -875,7 +875,8 @@ def finish_fn(): ) # Acquisition watchdog: drop the "running" breadcrumb (covers GUI + MCP-server runs). - self._run_state_writer = squid.acquisition_state.NullRunStateWriter() + # self._run_state_writer is already a NullRunStateWriter (set before the outer try); it + # stays one if start() below fails, so a breadcrumb write failure never breaks acquisition. try: expected = { "timepoints": self.Nt, diff --git a/software/control/core/multi_point_worker.py b/software/control/core/multi_point_worker.py index f9a156be6..4ce276411 100644 --- a/software/control/core/multi_point_worker.py +++ b/software/control/core/multi_point_worker.py @@ -428,11 +428,20 @@ def _is_well_based_acquisition(self) -> bool: ) return True + def _abort_due_to_error(self) -> None: + """Abort the run due to an internal error (vs a user abort). + + The worker only ever aborts itself on error conditions; user aborts arrive + via the external abort flag. Tagging the cause here lets _compute_end_reason + classify the end as "error" instead of "user_abort". + """ + self._abort_cause = "error" + self.request_abort_fn() + def _run_state_beat(self) -> None: self._run_state.beat( { "timepoint": self.time_point, - "expected_timepoints": self.Nt, "fov": self._timepoint_fov_count, "images": self.image_count, } @@ -539,8 +548,7 @@ def run(self): except TimeoutError as te: self._log.error(f"Operation timed out during acquisition, aborting acquisition!") self._log.error(te) - self._abort_cause = "error" - self.request_abort_fn() + self._abort_due_to_error() except Exception as e: self._log.exception(e) self._run_state_fatal = True @@ -1070,8 +1078,7 @@ def run_coordinate_acquisition(self, current_path): result = self._summarize_runner_outputs() if not result.none_failed and self._abort_on_failed_job: self._log.error("Some jobs failed, aborting acquisition because abort_on_failed_job=True") - self._abort_cause = "error" - self.request_abort_fn() + self._abort_due_to_error() return with self._timing.get_timer("move_to_coordinate"): @@ -1363,13 +1370,13 @@ def _image_callback(self, camera_frame: CameraFrame): self._ready_for_next_trigger.set() if not info: self._log.error("In image callback, no current capture info! Something is wrong. Aborting.") - self.request_abort_fn() + self._abort_due_to_error() return image = camera_frame.frame if not camera_frame or image is None: self._log.warning("image in frame callback is None. Something is really wrong, aborting!") - self.request_abort_fn() + self._abort_due_to_error() return # Increment image counter for Slack notification stats @@ -1399,7 +1406,7 @@ def _image_callback(self, camera_frame: CameraFrame): if job_runner is not None: if not job_runner.dispatch(job): self._log.error("Failed to dispatch multiprocessing job!") - self.request_abort_fn() + self._abort_due_to_error() return else: try: @@ -1408,7 +1415,7 @@ def _image_callback(self, camera_frame: CameraFrame): result = job.run() except Exception: self._log.exception("Failed to execute job, abandoning acquisition!") - self.request_abort_fn() + self._abort_due_to_error() return height, width = image.shape[:2] @@ -1451,7 +1458,7 @@ def acquire_camera_image( with self._timing.get_timer("_ready_for_next_trigger.wait"): if not self._ready_for_next_trigger.wait(self._frame_wait_timeout_s()): self._log.error("Frame callback never set _have_last_triggered_image callback! Aborting acquisition.") - self.request_abort_fn() + self._abort_due_to_error() return # Backpressure check AFTER previous frame dispatched, BEFORE next trigger @@ -1511,7 +1518,7 @@ def acquire_camera_image( non_hw_frame_timeout = 5 * self.camera.get_total_frame_time() / 1e3 + 2 if not self._ready_for_next_trigger.wait(non_hw_frame_timeout): self._log.error("Timed out waiting {non_hw_frame_timeout} [s] for a frame, aborting acquisition.") - self.request_abort_fn() + self._abort_due_to_error() # Let this fall through so we still turn off illumination. Let the caller actually break out # of the acquisition. diff --git a/software/squid/acquisition_state.py b/software/squid/acquisition_state.py index 5ed9da527..fb1f09b64 100644 --- a/software/squid/acquisition_state.py +++ b/software/squid/acquisition_state.py @@ -112,11 +112,11 @@ def run_id(self) -> Optional[str]: return self._record.get("run_id") def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: - if progress: - self._record["progress"] = progress now = time.time() if not force and (now - self._last_beat) < HEARTBEAT_INTERVAL_S: return + if progress: + self._record["progress"] = progress self._last_beat = now self._record["heartbeat_at"] = now self._flush() diff --git a/software/tests/acquisition_watchdog/test_config.py b/software/tests/acquisition_watchdog/test_config.py index 04f904985..060104b7f 100644 --- a/software/tests/acquisition_watchdog/test_config.py +++ b/software/tests/acquisition_watchdog/test_config.py @@ -26,7 +26,6 @@ def test_load_reads_credentials_from_yaml(tmp_path): {"enabled": True, "bot_token": "xoxb-xyz", "channel_id": "C42"}, ) cfg = wdconfig.load_slack_config(p) - assert cfg.enabled is True assert cfg.bot_token == "xoxb-xyz" assert cfg.channel_id == "C42" assert cfg.watchdog_enabled is True # default when key absent diff --git a/software/tests/acquisition_watchdog/test_monitor.py b/software/tests/acquisition_watchdog/test_monitor.py index a1dd2ae36..b59b82711 100644 --- a/software/tests/acquisition_watchdog/test_monitor.py +++ b/software/tests/acquisition_watchdog/test_monitor.py @@ -66,7 +66,7 @@ def test_dedup_persists_across_restart(tmp_path, monkeypatch): monkeypatch.setattr("squid.slack.post_message", lambda *a, **k: (sent.append(a) or (True, "1"))) monkeypatch.setattr( "acquisition_watchdog.config.load_slack_config", - lambda p: SlackConfig(True, "xoxb", "C1", True), + lambda p: SlackConfig("xoxb", "C1", True), ) Monitor(state_dir=tmp_path).check_once(time.time()) assert len(sent) == 1 From 7186e2638e7db41719bf896bf2545713aeacc36f Mon Sep 17 00:00:00 2001 From: You Yan Date: Fri, 26 Jun 2026 16:00:10 -0700 Subject: [PATCH 19/23] fix(watchdog): warn if acquisition thread outlives the shutdown join MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses Copilot review: the shutdown abort joins the worker with a 15s timeout but didn't check the result. If the thread is still alive, os._exit() follows and the watchdog will report a crash — now logged explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/main_hcs.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/software/main_hcs.py b/software/main_hcs.py index 330b018f9..dddc81816 100644 --- a/software/main_hcs.py +++ b/software/main_hcs.py @@ -445,6 +445,11 @@ def launch_claude_code(): mpc.request_abort_aquisition() if getattr(mpc, "thread", None) is not None: mpc.thread.join(timeout=15.0) + if mpc.thread.is_alive(): + log.warning( + "Acquisition thread still alive 15s after abort request; exiting anyway — " + "the watchdog may report this run as a crash." + ) except Exception as e: log.warning(f"Error during shutdown abort handling: {e}") From 1ab8566d88e999a8b2e3c99d06e34e5145423315 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 30 Jun 2026 20:45:16 -0700 Subject: [PATCH 20/23] feat(watchdog): add 'Enable watchdog alerts' toggle to Slack settings dialog Exposes watchdog_enabled in the GUI Slack settings dialog (writes to the same cache/slack_settings.yaml the watchdog reads) plus SlackNotifications.WATCHDOG_ENABLED in _def. Kept independent of the master 'Enable Slack Notifications' switch, so 'no routine pings but still alert me on a crash/hang' is a valid config. Docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/README.md | 4 ++-- software/control/_def.py | 1 + software/control/widgets_slack.py | 12 ++++++++++++ software/docs/slack_notifications.md | 1 + 4 files changed, 16 insertions(+), 2 deletions(-) diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md index 4d62c285a..d35ae8c4a 100644 --- a/software/acquisition_watchdog/README.md +++ b/software/acquisition_watchdog/README.md @@ -19,8 +19,8 @@ Slack credentials are read from `cache/slack_settings.yaml` — the same file th Slack settings dialog writes (keys `bot_token`, `channel_id`, `enabled`). Run the watchdog from the `software/` directory (so the default `cache/slack_settings.yaml` path resolves), or pass `--slack-settings `. To disable watchdog alerts on a -machine without disabling the GUI's notifications, add `watchdog_enabled: false` to -that YAML. +machine without disabling the GUI's notifications, uncheck **"Enable watchdog alerts"** +in the GUI's Slack settings dialog (or set `watchdog_enabled: false` in that YAML directly). ## Install as an always-on service - **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). diff --git a/software/control/_def.py b/software/control/_def.py index c803655ae..a122b8ea1 100644 --- a/software/control/_def.py +++ b/software/control/_def.py @@ -1288,6 +1288,7 @@ class SlackNotifications: NOTIFY_ON_ACQUISITION_START = False NOTIFY_ON_ACQUISITION_FINISHED = True SEND_MOSAIC_SNAPSHOTS = True + WATCHDOG_ENABLED = True # Standalone acquisition watchdog: alert on crash / hang / error / abort try: diff --git a/software/control/widgets_slack.py b/software/control/widgets_slack.py index 65e13869c..dada77cc9 100644 --- a/software/control/widgets_slack.py +++ b/software/control/widgets_slack.py @@ -41,6 +41,7 @@ "send_mosaic_snapshots": ("SEND_MOSAIC_SNAPSHOTS", lambda: True), "notify_on_acquisition_start": ("NOTIFY_ON_ACQUISITION_START", lambda: True), "notify_on_acquisition_finished": ("NOTIFY_ON_ACQUISITION_FINISHED", lambda: True), + "watchdog_enabled": ("WATCHDOG_ENABLED", lambda: True), } @@ -156,6 +157,14 @@ def _setup_ui(self): self.checkbox_notify_finished.setToolTip("Send a Slack message when an acquisition completes") notif_layout.addWidget(self.checkbox_notify_finished) + self.checkbox_watchdog = QCheckBox("Enable watchdog alerts") + self.checkbox_watchdog.setToolTip( + "Let the standalone acquisition watchdog process post a Slack alert when a run ends " + "prematurely (crash / hang / error / abort). Uses the bot token + channel above and is " + "independent of the master toggle; requires the watchdog process to be running." + ) + notif_layout.addWidget(self.checkbox_watchdog) + notif_group.setLayout(notif_layout) layout.addWidget(notif_group) @@ -216,6 +225,7 @@ def _load_settings(self): "send_mosaic_snapshots": self.checkbox_send_mosaic, "notify_on_acquisition_start": self.checkbox_notify_start, "notify_on_acquisition_finished": self.checkbox_notify_finished, + "watchdog_enabled": self.checkbox_watchdog, } lineedit_map = { "bot_token": self.lineedit_bot_token, @@ -254,6 +264,7 @@ def _save_settings(self): control._def.SlackNotifications.SEND_MOSAIC_SNAPSHOTS = self.checkbox_send_mosaic.isChecked() control._def.SlackNotifications.NOTIFY_ON_ACQUISITION_START = self.checkbox_notify_start.isChecked() control._def.SlackNotifications.NOTIFY_ON_ACQUISITION_FINISHED = self.checkbox_notify_finished.isChecked() + control._def.SlackNotifications.WATCHDOG_ENABLED = self.checkbox_watchdog.isChecked() # Update notifier if available if self._slack_notifier: @@ -270,6 +281,7 @@ def _save_settings(self): "send_mosaic_snapshots": self.checkbox_send_mosaic.isChecked(), "notify_on_acquisition_start": self.checkbox_notify_start.isChecked(), "notify_on_acquisition_finished": self.checkbox_notify_finished.isChecked(), + "watchdog_enabled": self.checkbox_watchdog.isChecked(), } try: diff --git a/software/docs/slack_notifications.md b/software/docs/slack_notifications.md index 4157ce9fe..145857377 100644 --- a/software/docs/slack_notifications.md +++ b/software/docs/slack_notifications.md @@ -68,6 +68,7 @@ In the channel, type: | Include mosaic snapshots | Attach screenshot with timepoint notifications | | Notify on acquisition start | Send message when acquisition begins | | Notify on acquisition finished | Send summary when acquisition completes | +| Enable watchdog alerts | Let the standalone acquisition watchdog alert on a crashed / hung / aborted / errored run — independent of the master switch, and requires the watchdog process to be running (see `acquisition_watchdog/README.md`) | ## Notification Examples From 3422fc6bb8b1bc898a7056101444b41794d697b7 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 30 Jun 2026 21:13:57 -0700 Subject: [PATCH 21/23] docs: move design spec + plan out of the PR into the AI-docs archive The acquisition watchdog design spec and implementation plan now live in Cephla-Lab/AI-docs (Squid/done/); this PR carries only code + the package README. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/2026-06-23-acquisition-watchdog.md | 1779 ----------------- .../2026-06-23-acquisition-watchdog-design.md | 229 --- 2 files changed, 2008 deletions(-) delete mode 100644 software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md delete mode 100644 software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md diff --git a/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md b/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md deleted file mode 100644 index f779ae259..000000000 --- a/software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md +++ /dev/null @@ -1,1779 +0,0 @@ -# Acquisition Watchdog Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Detect when an acquisition ends prematurely — process crash/hang/kill, fatal error, or user abort — and send a single Slack alert, covering GUI- and MCP-server-driven runs on Ubuntu and Windows. - -**Architecture:** The acquisition engine (`control/core/`) drops on-disk breadcrumbs — a `run.json` written atomically at start, bumped with a throttled heartbeat during the run, and finalized with a reason at end. An independent, lightweight `acquisition_watchdog` process polls `run.json`, detects a dead/stale run or a non-clean end, and posts one Slack alert. A shared dependency-free `squid/slack.py` sender is reused by both the watchdog and the existing in-process `SlackNotifier`, whose end-of-run message is gated to clean successes so failures alert exactly once. - -**Tech Stack:** Python 3.8+, stdlib only (`json`, `urllib`, `configparser`, `socket`, `uuid`, `tempfile`, `os`), `platformdirs` (already a dep), `pyyaml` (already a dep), `pytest`. No new third-party dependencies. - ---- - -## Spec - -`docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md` - -## Reason taxonomy (v1) - -The worker computes one `reason` at the end of `run()`; it drives both the breadcrumb and the in-process finish message: - -| `reason` | When | Watchdog alerts? | Notifier finish msg? | -|---|---|---|---| -| `completed` | loop finished all timepoints, `_acquisition_error_count == 0`, not aborted | no | yes | -| `completed_with_errors` | loop finished but `_acquisition_error_count > 0` | yes | no | -| `error` | uncaught exception, or auto-abort from `TimeoutError` / failed-job abort | yes | no | -| `user_abort` | abort flag set externally (GUI/server) **or** GUI closed mid-run (shutdown aborts + joins) | yes | no | -| *(no end record)* | process crashed/killed/hung before writing end | yes (crash/hang) | n/a | - -(`app_closed` from the design is folded into `user_abort` for v1: the shutdown hook requests the normal abort and joins the worker so it writes a proper `user_abort` end record instead of looking like a crash. A distinct `app_closed` label is future work.) - -## File structure - -| Path | Responsibility | -|---|---| -| `squid/slack.py` | **New.** Dependency-free `post_message(bot_token, channel_id, text, blocks)` via `urllib`. Reused by notifier + watchdog. | -| `squid/acquisition_state.py` | **New.** `run.json` schema, `default_state_dir()`, atomic write, `read_run()`, `RunStateWriter` (+ `NullRunStateWriter`). Engine writes; watchdog reads. Leaf module — must not import `control`. | -| `acquisition_watchdog/__init__.py` | **New.** Package marker. | -| `acquisition_watchdog/config.py` | **New.** Resolve active `.ini`; load `[SlackNotifications]` with stdlib `configparser`. | -| `acquisition_watchdog/alerts.py` | **New.** Format the Slack alert text + blocks for each alert kind. | -| `acquisition_watchdog/monitor.py` | **New.** `pid_alive`, `Monitor.classify`, `Monitor.check_once`, dedup persistence, `run_forever`. | -| `acquisition_watchdog/__main__.py` | **New.** CLI entry: `python -m acquisition_watchdog`. | -| `acquisition_watchdog/systemd/squid-acquisition-watchdog.service` | **New.** Linux user-service unit. | -| `acquisition_watchdog/windows/squid-acquisition-watchdog.xml`, `install.ps1` | **New.** Windows Task Scheduler recipe. | -| `acquisition_watchdog/README.md` | **New.** Install/run docs for both OSes. | -| `control/slack_notifier.py` | **Modify.** Delegate `_post_message` to `squid.slack`; add `reason` field to `AcquisitionStats`; gate `notify_acquisition_finished` to `reason == "completed"`. | -| `control/core/multi_point_controller.py` | **Modify.** Write the start breadcrumb in `run_acquisition()`; pass the writer to the worker. | -| `control/core/multi_point_worker.py` | **Modify.** Heartbeat in the loop + image callback; compute `reason` and write end in `finally`; track `_abort_cause`. | -| `main_hcs.py` | **Modify.** On shutdown-while-acquiring, request abort + join so the worker writes `user_abort`. | -| `tests/...` | **New/modify.** Unit + integration tests per task; autouse fixture redirecting the state dir to tmp. | - ---- - -## Task 1: `squid/slack.py` — shared Slack sender - -**Files:** -- Create: `squid/slack.py` -- Test: `tests/squid/test_slack.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/squid/test_slack.py -import json -from unittest.mock import patch, MagicMock - -import squid.slack as slack - - -def test_post_message_returns_false_without_credentials(): - assert slack.post_message(None, "C123", "hi") == (False, None) - assert slack.post_message("xoxb-1", None, "hi") == (False, None) - - -def test_post_message_builds_authorized_request_and_parses_ok(): - captured = {} - - class FakeResp: - def __enter__(self): - return self - - def __exit__(self, *a): - return False - - def read(self): - return json.dumps({"ok": True, "ts": "111.222"}).encode() - - def fake_urlopen(request, timeout=15): - captured["url"] = request.full_url - captured["headers"] = request.headers - captured["body"] = json.loads(request.data.decode()) - return FakeResp() - - with patch("urllib.request.urlopen", side_effect=fake_urlopen): - ok, ts = slack.post_message("xoxb-token", "C123", "hello", blocks=[{"type": "section"}]) - - assert ok is True and ts == "111.222" - assert captured["url"].endswith("/chat.postMessage") - assert captured["headers"]["Authorization"] == "Bearer xoxb-token" - assert captured["body"]["channel"] == "C123" - assert captured["body"]["text"] == "hello" - assert captured["body"]["blocks"] == [{"type": "section"}] - - -def test_post_message_handles_api_error(): - class FakeResp: - def __enter__(self): - return self - - def __exit__(self, *a): - return False - - def read(self): - return json.dumps({"ok": False, "error": "channel_not_found"}).encode() - - with patch("urllib.request.urlopen", return_value=FakeResp()): - ok, ts = slack.post_message("xoxb", "C1", "x") - assert ok is False and ts is None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/squid/test_slack.py -v` -Expected: FAIL with `ModuleNotFoundError: No module named 'squid.slack'`. - -- [ ] **Step 3: Write the implementation** - -```python -# squid/slack.py -"""Dependency-free Slack chat.postMessage sender. - -Shared by the in-process SlackNotifier (control/slack_notifier.py) and the -standalone acquisition watchdog. Stdlib only — safe to import without the -control/Qt/hardware stack. -""" -import json -import urllib.error -import urllib.request -from typing import Optional, Tuple - -import squid.logging - -_log = squid.logging.get_logger(__name__) - -SLACK_API_BASE = "https://slack.com/api" - - -def post_message( - bot_token: Optional[str], - channel_id: Optional[str], - text: str, - blocks: Optional[list] = None, - timeout: float = 15.0, -) -> Tuple[bool, Optional[str]]: - """Post a message to Slack. Returns (ok, message_ts).""" - if not bot_token or not channel_id: - _log.debug("No Slack bot token or channel configured") - return False, None - - payload = {"channel": channel_id, "text": text} - if blocks: - payload["blocks"] = blocks - data = json.dumps(payload).encode("utf-8") - request = urllib.request.Request( - f"{SLACK_API_BASE}/chat.postMessage", - data=data, - headers={ - "Content-Type": "application/json; charset=utf-8", - "Authorization": f"Bearer {bot_token}", - }, - method="POST", - ) - try: - with urllib.request.urlopen(request, timeout=timeout) as response: - result = json.loads(response.read().decode("utf-8")) - if result.get("ok"): - return True, result.get("ts") - _log.warning(f"Slack API error: {result.get('error')}") - return False, None - except urllib.error.URLError as e: - _log.warning(f"Failed to send Slack message: {e}") - return False, None - except Exception as e: - _log.warning(f"Unexpected error sending Slack message: {e}") - return False, None -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/squid/test_slack.py -v` -Expected: PASS (3 tests). Create `tests/squid/__init__.py` if the package import fails. - -- [ ] **Step 5: Commit** - -```bash -git add software/squid/slack.py software/tests/squid/test_slack.py -git commit -m "feat(slack): add dependency-free squid.slack.post_message sender" -``` - ---- - -## Task 2: Delegate `SlackNotifier._post_message` to `squid.slack` - -**Files:** -- Modify: `control/slack_notifier.py` (`_post_message`, lines 160–212; imports lines 10–27) -- Test: `tests/control/test_slack_notifier_send.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/control/test_slack_notifier_send.py -from unittest.mock import patch -from control.slack_notifier import SlackNotifier - - -def test_post_message_delegates_to_squid_slack(): - n = SlackNotifier(bot_token="xoxb-abc", channel_id="C999") - with patch("squid.slack.post_message", return_value=(True, "1.0")) as m: - ok, ts = n._post_message("hello", blocks=[{"type": "section"}]) - assert ok is True and ts == "1.0" - m.assert_called_once_with("xoxb-abc", "C999", "hello", [{"type": "section"}]) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/control/test_slack_notifier_send.py -v` -Expected: FAIL — `_post_message` calls `urllib` directly, so `squid.slack.post_message` is never called (`AssertionError: Expected 'post_message' to have been called once`). - -- [ ] **Step 3: Edit `control/slack_notifier.py`** - -Add the import near the existing `import squid.logging` (line 27): - -```python -import squid.logging -import squid.slack -``` - -Replace the entire `_post_message` method (lines 160–212) with: - -```python - def _post_message(self, text: str, blocks: Optional[list] = None) -> Tuple[bool, Optional[str]]: - return squid.slack.post_message(self.bot_token, self.channel_id, text, blocks) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/control/test_slack_notifier_send.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add software/control/slack_notifier.py software/tests/control/test_slack_notifier_send.py -git commit -m "refactor(slack): route SlackNotifier sends through squid.slack" -``` - ---- - -## Task 3: `squid/acquisition_state.py` — breadcrumb schema + writer - -**Files:** -- Create: `squid/acquisition_state.py` -- Test: `tests/squid/test_acquisition_state.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/squid/test_acquisition_state.py -import json - -import squid.acquisition_state as ast - - -def _expected(): - return {"timepoints": 3, "regions": 1, "fovs": 4, "channels": 2, "z": 1} - - -def test_start_writes_running_record(tmp_path): - w = ast.RunStateWriter.start( - experiment_id="exp1", pid=4321, config_path="/cfg.ini", - output_path=str(tmp_path / "exp1"), expected=_expected(), - machine="micro-1", state_dir=tmp_path, - ) - rec = ast.read_run(tmp_path) - assert rec["status"] == "running" - assert rec["experiment_id"] == "exp1" - assert rec["pid"] == 4321 - assert rec["machine"] == "micro-1" - assert rec["expected"] == _expected() - assert rec["run_id"] == w.run_id - assert rec["reason"] is None - - -def test_beat_is_throttled_but_updates_progress_on_flush(tmp_path): - w = ast.RunStateWriter.start( - experiment_id="e", pid=1, config_path=None, output_path="o", - expected=_expected(), state_dir=tmp_path, - ) - first = ast.read_run(tmp_path)["heartbeat_at"] - # Immediate beat is throttled (< HEARTBEAT_INTERVAL_S since start) -> file unchanged. - w.beat({"timepoint": 1}) - assert ast.read_run(tmp_path)["heartbeat_at"] == first - # Forced beat flushes and records progress. - w.beat({"timepoint": 2}, force=True) - rec = ast.read_run(tmp_path) - assert rec["heartbeat_at"] >= first - assert rec["progress"] == {"timepoint": 2} - - -def test_end_records_reason_and_stats(tmp_path): - w = ast.RunStateWriter.start( - experiment_id="e", pid=1, config_path=None, output_path="o", - expected=_expected(), state_dir=tmp_path, - ) - w.end("user_abort", {"total_images": 7, "errors_encountered": 0}) - rec = ast.read_run(tmp_path) - assert rec["status"] == "ended" - assert rec["reason"] == "user_abort" - assert rec["ended_at"] is not None - assert rec["stats"]["total_images"] == 7 - - -def test_read_run_missing_returns_none(tmp_path): - assert ast.read_run(tmp_path) is None - - -def test_null_writer_is_noop(tmp_path): - w = ast.NullRunStateWriter() - w.beat({"timepoint": 1}) - w.end("completed", {}) - assert ast.read_run(tmp_path) is None - assert w.run_id is None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/squid/test_acquisition_state.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'squid.acquisition_state'`. - -- [ ] **Step 3: Write the implementation** - -```python -# squid/acquisition_state.py -"""On-disk acquisition run-state breadcrumbs, shared by the acquisition engine -(writer) and the standalone acquisition watchdog (reader). - -Stdlib-only leaf module: must NOT import anything from `control`. -""" -import json -import os -import socket -import tempfile -import time -import uuid -from pathlib import Path -from typing import Optional - -import platformdirs - -import squid.logging - -_log = squid.logging.get_logger(__name__) - -SCHEMA_VERSION = 1 -HEARTBEAT_INTERVAL_S = 5.0 -RUN_FILE_NAME = "run.json" - - -def default_state_dir() -> Path: - """Per-user watchdog state dir, shared by writer and reader. - - Overridable via SQUID_WATCHDOG_STATE_DIR (honored by both processes). - """ - override = os.environ.get("SQUID_WATCHDOG_STATE_DIR") - if override: - return Path(override) - return Path(platformdirs.user_state_path("squid", "cephla")) / "watchdog" - - -def run_file_path(state_dir: Optional[Path] = None) -> Path: - return Path(state_dir) / RUN_FILE_NAME if state_dir else default_state_dir() / RUN_FILE_NAME - - -def _atomic_write_json(path: Path, data: dict) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".run-", suffix=".tmp") - try: - with os.fdopen(fd, "w") as f: - json.dump(data, f) - f.flush() - os.fsync(f.fileno()) - os.replace(tmp, path) # atomic on POSIX and Windows - except BaseException: - try: - os.unlink(tmp) - except OSError: - pass - raise - - -def read_run(state_dir: Optional[Path] = None) -> Optional[dict]: - try: - with open(run_file_path(state_dir)) as f: - return json.load(f) - except (FileNotFoundError, json.JSONDecodeError): - return None - - -class RunStateWriter: - """Writes/updates the single run.json for the current acquisition.""" - - def __init__(self, record: dict, state_dir: Optional[Path] = None): - self._record = record - self._state_dir = state_dir - self._last_beat = 0.0 - - @classmethod - def start( - cls, - *, - experiment_id: str, - pid: int, - config_path: Optional[str], - output_path: str, - expected: dict, - machine: Optional[str] = None, - state_dir: Optional[Path] = None, - ) -> "RunStateWriter": - now = time.time() - record = { - "schema_version": SCHEMA_VERSION, - "run_id": uuid.uuid4().hex, - "experiment_id": experiment_id, - "machine": machine or socket.gethostname(), - "pid": pid, - "config_path": config_path, - "output_path": output_path, - "started_at": now, - "heartbeat_at": now, - "progress": {}, - "expected": expected, - "status": "running", - "reason": None, - "ended_at": None, - "stats": None, - } - writer = cls(record, state_dir=state_dir) - writer._flush() - writer._last_beat = now - return writer - - @property - def run_id(self) -> Optional[str]: - return self._record.get("run_id") - - def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: - if progress: - self._record["progress"] = progress - now = time.time() - if not force and (now - self._last_beat) < HEARTBEAT_INTERVAL_S: - return - self._last_beat = now - self._record["heartbeat_at"] = now - self._flush() - - def end(self, reason: str, stats: Optional[dict] = None) -> None: - self._record["status"] = "ended" - self._record["reason"] = reason - self._record["ended_at"] = time.time() - if stats is not None: - self._record["stats"] = stats - self._flush() - - def _flush(self) -> None: - try: - _atomic_write_json(run_file_path(self._state_dir), dict(self._record)) - except OSError as e: - _log.warning(f"Failed to write acquisition run state: {e}") - - -class NullRunStateWriter: - """No-op writer used when breadcrumbs are not wired (tests, side paths).""" - - run_id = None - - def beat(self, progress: Optional[dict] = None, force: bool = False) -> None: - pass - - def end(self, reason: str, stats: Optional[dict] = None) -> None: - pass -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/squid/test_acquisition_state.py -v` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add software/squid/acquisition_state.py software/tests/squid/test_acquisition_state.py -git commit -m "feat(watchdog): add squid.acquisition_state breadcrumb schema + writer" -``` - ---- - -## Task 4: `acquisition_watchdog/config.py` — config resolution - -**Files:** -- Create: `acquisition_watchdog/__init__.py` (empty) -- Create: `acquisition_watchdog/config.py` -- Test: `tests/acquisition_watchdog/test_config.py` (+ `tests/acquisition_watchdog/__init__.py`) - -- [ ] **Step 1: Write the failing test** - -```python -# tests/acquisition_watchdog/test_config.py -from acquisition_watchdog import config as wdconfig - - -def _write_ini(path, body): - path.write_text(body) - return path - - -def test_resolve_prefers_cli_then_env_then_run_record(tmp_path, monkeypatch): - monkeypatch.delenv("SQUID_CONFIG", raising=False) - assert wdconfig.resolve_config_path("/cli.ini", {"config_path": "/run.ini"}) == __import__("pathlib").Path("/cli.ini") - monkeypatch.setenv("SQUID_CONFIG", "/env.ini") - assert str(wdconfig.resolve_config_path(None, {"config_path": "/run.ini"})) == "/env.ini" - monkeypatch.delenv("SQUID_CONFIG", raising=False) - assert str(wdconfig.resolve_config_path(None, {"config_path": "/run.ini"})) == "/run.ini" - - -def test_load_slack_config_reads_section(tmp_path): - ini = _write_ini( - tmp_path / "c.ini", - "[SLACKNOTIFICATIONS]\nenabled = True\nbot_token = xoxb-xyz\nchannel_id = C42\nwatchdog_enabled = True\n", - ) - cfg = wdconfig.load_slack_config(ini) - assert cfg.enabled is True - assert cfg.bot_token == "xoxb-xyz" - assert cfg.channel_id == "C42" - assert cfg.watchdog_enabled is True - - -def test_load_slack_config_defaults_when_missing(tmp_path): - ini = _write_ini(tmp_path / "c.ini", "[GENERAL]\nfoo = 1\n") - cfg = wdconfig.load_slack_config(ini) - assert cfg.bot_token is None and cfg.channel_id is None - assert cfg.watchdog_enabled is True # defaults to on when section absent - - assert wdconfig.load_slack_config(None).bot_token is None -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_config.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog'`. - -- [ ] **Step 3: Write the implementation** - -```python -# acquisition_watchdog/__init__.py -``` - -```python -# acquisition_watchdog/config.py -"""Resolve the active Squid .ini and read its [SlackNotifications] section, -without importing the heavy control._def module. -""" -import configparser -import os -from pathlib import Path -from typing import NamedTuple, Optional - - -class SlackConfig(NamedTuple): - enabled: bool - bot_token: Optional[str] - channel_id: Optional[str] - watchdog_enabled: bool - - -def resolve_config_path(cli_config: Optional[str], run_record: Optional[dict]) -> Optional[Path]: - """Priority: --config > $SQUID_CONFIG > run.json config_path > cache pointer.""" - if cli_config: - return Path(cli_config) - env = os.environ.get("SQUID_CONFIG") - if env: - return Path(env) - if run_record and run_record.get("config_path"): - return Path(run_record["config_path"]) - cache = Path("cache/config_file_path.txt") - if cache.exists(): - first = cache.read_text().splitlines() - if first: - return Path(first[0].strip()) - return None - - -def load_slack_config(config_path: Optional[Path]) -> SlackConfig: - if not config_path or not Path(config_path).exists(): - return SlackConfig(False, None, None, True) - cp = configparser.ConfigParser() - try: - cp.read(config_path) - except configparser.Error: - return SlackConfig(False, None, None, True) - if not cp.has_section("SLACKNOTIFICATIONS"): - return SlackConfig(False, None, None, True) - sec = cp["SLACKNOTIFICATIONS"] - - def getbool(key: str, default: bool) -> bool: - try: - return sec.getboolean(key, default) - except ValueError: - return default - - token = sec.get("bot_token", fallback=None) or None - channel = sec.get("channel_id", fallback=None) or None - return SlackConfig( - enabled=getbool("enabled", False), - bot_token=token, - channel_id=channel, - watchdog_enabled=getbool("watchdog_enabled", True), - ) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_config.py -v` -Expected: PASS (3 tests). Add empty `tests/acquisition_watchdog/__init__.py` if needed. - -- [ ] **Step 5: Commit** - -```bash -git add software/acquisition_watchdog/__init__.py software/acquisition_watchdog/config.py software/tests/acquisition_watchdog/ -git commit -m "feat(watchdog): add config resolution + [SlackNotifications] loader" -``` - ---- - -## Task 5: `acquisition_watchdog/alerts.py` — alert formatting - -**Files:** -- Create: `acquisition_watchdog/alerts.py` -- Test: `tests/acquisition_watchdog/test_alerts.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/acquisition_watchdog/test_alerts.py -from acquisition_watchdog import alerts - - -def _run(): - return { - "experiment_id": "plateA_2026", - "machine": "micro-1", - "output_path": "/data/plateA_2026", - "progress": {"timepoint": 3, "expected_timepoints": 10, "images": 360}, - "expected": {"timepoints": 10}, - "started_at": 1_700_000_000.0, - "heartbeat_at": 1_700_000_100.0, - } - - -def test_format_alert_includes_key_facts(): - text, blocks = alerts.format_alert("crash", _run()) - assert "plateA_2026" in text - assert "micro-1" in text - blob = str(blocks) - assert "plateA_2026" in blob - assert "3" in blob and "10" in blob # progress vs expected - assert isinstance(blocks, list) and blocks - - -def test_format_alert_each_kind_has_title(): - for kind in ("crash", "hang", "error", "completed_with_errors", "user_abort"): - text, _ = alerts.format_alert(kind, _run()) - assert text # non-empty title line for every kind -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_alerts.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.alerts'`. - -- [ ] **Step 3: Write the implementation** - -```python -# acquisition_watchdog/alerts.py -"""Format watchdog Slack alerts (text + Block Kit blocks).""" -from datetime import datetime, timezone -from typing import Optional, Tuple - -_KIND_TITLE = { - "crash": ":red_circle: Acquisition process died", - "hang": ":large_orange_circle: Acquisition hung (no heartbeat)", - "error": ":red_circle: Acquisition ended with a fatal error", - "completed_with_errors": ":large_orange_circle: Acquisition finished with errors", - "user_abort": ":large_yellow_circle: Acquisition aborted", -} - - -def _fmt_ts(epoch: Optional[float]) -> str: - if not epoch: - return "unknown" - return datetime.fromtimestamp(epoch, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC") - - -def _progress_line(run: dict) -> str: - prog = run.get("progress") or {} - expected = run.get("expected") or {} - tp = prog.get("timepoint", "?") - exp_tp = prog.get("expected_timepoints", expected.get("timepoints", "?")) - images = prog.get("images", "?") - return f"timepoint {tp}/{exp_tp}, {images} images" - - -def format_alert(kind: str, run: dict) -> Tuple[str, list]: - title = _KIND_TITLE.get(kind, f"Acquisition alert: {kind}") - experiment = run.get("experiment_id", "unknown") - machine = run.get("machine", "unknown") - text = f"{title}: {experiment} on {machine}" - - last_seen = run.get("ended_at") or run.get("heartbeat_at") - detail = ( - f"*Experiment:* {experiment}\n" - f"*Machine:* {machine}\n" - f"*Progress:* {_progress_line(run)}\n" - f"*Started:* {_fmt_ts(run.get('started_at'))}\n" - f"*Last seen:* {_fmt_ts(last_seen)}\n" - f"*Output:* {run.get('output_path', 'unknown')}" - ) - blocks = [ - {"type": "header", "text": {"type": "plain_text", "text": title.replace(":red_circle:", "") - .replace(":large_orange_circle:", "").replace(":large_yellow_circle:", "").strip(), - "emoji": True}}, - {"type": "section", "text": {"type": "mrkdwn", "text": detail}}, - ] - return text, blocks -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_alerts.py -v` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -git add software/acquisition_watchdog/alerts.py software/tests/acquisition_watchdog/test_alerts.py -git commit -m "feat(watchdog): add Slack alert formatting" -``` - ---- - -## Task 6: `acquisition_watchdog/monitor.py` — poll, classify, dedup - -**Files:** -- Create: `acquisition_watchdog/monitor.py` -- Test: `tests/acquisition_watchdog/test_monitor.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/acquisition_watchdog/test_monitor.py -import time - -import squid.acquisition_state as ast -from acquisition_watchdog.monitor import Monitor - - -def _running(tmp_path, pid, heartbeat_age=0.0, run_id="r1"): - rec = { - "schema_version": 1, "run_id": run_id, "experiment_id": "e", "machine": "m", - "pid": pid, "config_path": None, "output_path": "o", - "started_at": time.time() - 100, "heartbeat_at": time.time() - heartbeat_age, - "progress": {}, "expected": {}, "status": "running", "reason": None, - "ended_at": None, "stats": None, - } - ast._atomic_write_json(ast.run_file_path(tmp_path), rec) - return rec - - -def _mon(tmp_path): - return Monitor(state_dir=tmp_path, heartbeat_timeout=120.0) - - -def test_running_with_live_pid_and_fresh_heartbeat_is_silent(tmp_path): - _running(tmp_path, pid=__import__("os").getpid(), heartbeat_age=1.0) - assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) is None - - -def test_dead_pid_is_crash(tmp_path): - _running(tmp_path, pid=2_000_000_000, heartbeat_age=1.0) # impossible pid - assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "crash" - - -def test_stale_heartbeat_with_live_pid_is_hang(tmp_path): - _running(tmp_path, pid=__import__("os").getpid(), heartbeat_age=999.0) - assert _mon(tmp_path).classify(ast.read_run(tmp_path), time.time()) == "hang" - - -def test_ended_reasons(tmp_path): - mon = _mon(tmp_path) - for reason, expect in [ - ("completed", None), ("completed_with_errors", "completed_with_errors"), - ("error", "error"), ("user_abort", "user_abort"), - ]: - run = {"run_id": f"x-{reason}", "status": "ended", "reason": reason} - assert mon.classify(run, time.time()) == expect - - -def test_dedup_persists_across_restart(tmp_path, monkeypatch): - _running(tmp_path, pid=2_000_000_000, run_id="dup1") - sent = [] - monkeypatch.setattr("squid.slack.post_message", lambda *a, **k: (sent.append(a) or (True, "1"))) - monkeypatch.setattr( - "acquisition_watchdog.config.load_slack_config", - lambda p: __import__("acquisition_watchdog.config", fromlist=["SlackConfig"]).SlackConfig(True, "xoxb", "C1", True), - ) - Monitor(state_dir=tmp_path).check_once(time.time()) - assert len(sent) == 1 - # Fresh Monitor (simulated restart) must not re-alert the same run_id. - Monitor(state_dir=tmp_path).check_once(time.time()) - assert len(sent) == 1 -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_monitor.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.monitor'`. - -- [ ] **Step 3: Write the implementation** - -```python -# acquisition_watchdog/monitor.py -"""Poll the acquisition run-state and alert on premature ends.""" -import json -import os -import time -from pathlib import Path -from typing import Optional, Set - -import squid.acquisition_state as acquisition_state -import squid.logging -import squid.slack -from acquisition_watchdog import alerts, config - -_log = squid.logging.get_logger("acquisition_watchdog") - -ALERT_REASONS = {"completed_with_errors", "error", "user_abort"} - - -def pid_alive(pid: Optional[int]) -> bool: - if not pid: - return False - try: - import psutil - - return psutil.pid_exists(pid) - except ImportError: - pass - if os.name == "posix": - try: - os.kill(pid, 0) - return True - except ProcessLookupError: - return False - except PermissionError: - return True - except OSError: - return False - # Windows without psutil: cannot check reliably; rely on the heartbeat instead. - return True - - -class Monitor: - def __init__( - self, - state_dir: Optional[Path] = None, - cli_config: Optional[str] = None, - poll_interval: float = 5.0, - heartbeat_timeout: float = 120.0, - ): - self._state_dir = Path(state_dir) if state_dir else None - self._cli_config = cli_config - self._poll = poll_interval - self._timeout = heartbeat_timeout - base = self._state_dir or acquisition_state.default_state_dir() - self._alerted_path = base / "alerted.json" - self._alerted = self._load_alerted() - - def _load_alerted(self) -> Set[str]: - try: - with open(self._alerted_path) as f: - return set(json.load(f)) - except (FileNotFoundError, json.JSONDecodeError): - return set() - - def _save_alerted(self) -> None: - try: - self._alerted_path.parent.mkdir(parents=True, exist_ok=True) - with open(self._alerted_path, "w") as f: - json.dump(sorted(self._alerted), f) - except OSError as e: - _log.warning(f"Could not persist alerted set: {e}") - - def classify(self, run: Optional[dict], now: float) -> Optional[str]: - """Return an alert kind ('crash'|'hang'|) or None.""" - if not run or run.get("run_id") in self._alerted: - return None - status = run.get("status") - if status == "running": - if not pid_alive(run.get("pid")): - return "crash" - if (now - (run.get("heartbeat_at") or 0)) > self._timeout: - return "hang" - return None - if status == "ended" and run.get("reason") in ALERT_REASONS: - return run["reason"] - return None - - def check_once(self, now: float) -> None: - run = acquisition_state.read_run(self._state_dir) - kind = self.classify(run, now) - if kind is None: - return - - cfg_path = config.resolve_config_path(self._cli_config, run) - slack_cfg = config.load_slack_config(cfg_path) - if not (slack_cfg.bot_token and slack_cfg.channel_id and slack_cfg.watchdog_enabled): - _log.warning( - f"Premature end ({kind}) for run_id={run.get('run_id')} but Slack is not " - f"configured/enabled; not alerting." - ) - self._mark_alerted(run["run_id"]) - return - - text, blocks = alerts.format_alert(kind, run) - ok, _ = squid.slack.post_message(slack_cfg.bot_token, slack_cfg.channel_id, text, blocks) - if ok: - _log.info(f"Sent watchdog alert ({kind}) for run_id={run.get('run_id')}") - self._mark_alerted(run["run_id"]) - else: - # Leave unmarked so a transient Slack failure retries on the next poll. - _log.warning(f"Failed to send watchdog alert ({kind}) for run_id={run.get('run_id')}; will retry") - - def _mark_alerted(self, run_id: str) -> None: - self._alerted.add(run_id) - self._save_alerted() - - def run_forever(self) -> None: - base = self._state_dir or acquisition_state.default_state_dir() - _log.info(f"Acquisition watchdog started. state_dir={base} heartbeat_timeout={self._timeout}s") - while True: - try: - self.check_once(time.time()) - except Exception as e: - _log.exception(f"Watchdog poll error: {e}") - time.sleep(self._poll) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_monitor.py -v` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add software/acquisition_watchdog/monitor.py software/tests/acquisition_watchdog/test_monitor.py -git commit -m "feat(watchdog): add poll/classify/dedup monitor" -``` - ---- - -## Task 7: `acquisition_watchdog/__main__.py` — CLI entry - -**Files:** -- Create: `acquisition_watchdog/__main__.py` -- Test: `tests/acquisition_watchdog/test_cli.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/acquisition_watchdog/test_cli.py -from unittest.mock import patch - -from acquisition_watchdog.__main__ import main - - -def test_once_runs_single_check(tmp_path): - with patch("acquisition_watchdog.monitor.Monitor.check_once") as check, \ - patch("acquisition_watchdog.monitor.Monitor.run_forever") as forever: - main(["--once", "--state-dir", str(tmp_path)]) - check.assert_called_once() - forever.assert_not_called() - - -def test_default_runs_forever(tmp_path): - with patch("acquisition_watchdog.monitor.Monitor.run_forever") as forever: - main(["--state-dir", str(tmp_path)]) - forever.assert_called_once() -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_cli.py -v` -Expected: FAIL — `ModuleNotFoundError: No module named 'acquisition_watchdog.__main__'`. - -- [ ] **Step 3: Write the implementation** - -```python -# acquisition_watchdog/__main__.py -"""CLI entry point: python -m acquisition_watchdog""" -import argparse -import time -from pathlib import Path -from typing import Optional, Sequence - -import squid.logging -from acquisition_watchdog.monitor import Monitor - - -def main(argv: Optional[Sequence[str]] = None) -> None: - parser = argparse.ArgumentParser( - prog="acquisition_watchdog", - description="Alert on prematurely-ended Squid acquisitions (crash/hang/abort/error).", - ) - parser.add_argument("--config", help="Path to the active configuration .ini ([SlackNotifications]).") - parser.add_argument("--state-dir", help="Override the watchdog state directory.") - parser.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between checks (default 5).") - parser.add_argument( - "--heartbeat-timeout", type=float, default=120.0, - help="Seconds of heartbeat silence (with a live PID) before declaring a hang (default 120).", - ) - parser.add_argument("--once", action="store_true", help="Run a single check and exit.") - args = parser.parse_args(argv) - - log = squid.logging.get_logger("acquisition_watchdog") - monitor = Monitor( - state_dir=Path(args.state_dir) if args.state_dir else None, - cli_config=args.config, - poll_interval=args.poll_interval, - heartbeat_timeout=args.heartbeat_timeout, - ) - if args.once: - monitor.check_once(time.time()) - else: - try: - monitor.run_forever() - except KeyboardInterrupt: - log.info("Acquisition watchdog stopped.") - - -if __name__ == "__main__": - main() -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/acquisition_watchdog/test_cli.py -v` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -git add software/acquisition_watchdog/__main__.py software/tests/acquisition_watchdog/test_cli.py -git commit -m "feat(watchdog): add CLI entry point" -``` - ---- - -## Task 8: Engine — write the start breadcrumb in `run_acquisition()` - -**Files:** -- Modify: `control/core/multi_point_controller.py` (`run_acquisition`, around lines 838–888) -- Modify: `control/core/multi_point_worker.py` (`__init__`, lines 66–110) -- Modify: `tests/control/conftest.py` (add autouse fixture redirecting state dir to tmp) - -- [ ] **Step 1: Add the autouse fixture so tests never touch the real state dir** - -Append to `tests/control/conftest.py`: - -```python -import pytest - - -@pytest.fixture(autouse=True) -def _watchdog_state_to_tmp(tmp_path, monkeypatch): - # Keep acquisition breadcrumbs out of the real user state dir during tests. - monkeypatch.setenv("SQUID_WATCHDOG_STATE_DIR", str(tmp_path / "watchdog")) -``` - -- [ ] **Step 2: Add `run_state_writer` param to `MultiPointWorker.__init__`** - -In `control/core/multi_point_worker.py`, add to the `__init__` signature (after `prewarmed_bp_values`, line 83): - -```python - prewarmed_bp_values: Optional["BackpressureValues"] = None, - run_state_writer=None, - ): -``` - -Add the import near the top of the file (with the other `control`/`squid` imports): - -```python -import squid.acquisition_state -``` - -Store it among the other attribute assignments (near line 110, after `self.request_abort_fn = request_abort_fn`): - -```python - self._run_state = run_state_writer or squid.acquisition_state.NullRunStateWriter() - self._abort_cause = None # set to "error" by auto-abort paths (timeout / failed jobs) -``` - -- [ ] **Step 3: Write the start breadcrumb and pass the writer (controller)** - -In `control/core/multi_point_controller.py`, add near the top with the other imports: - -```python -import squid.acquisition_state -``` - -In `run_acquisition()`, immediately AFTER the `_save_acquisition_yaml(...)` call block (ends ~line 865) and BEFORE `prewarmed_runner, prewarmed_bp_values = self.get_prewarmed_job_runner()` (line 869), insert: - -```python - # Acquisition watchdog: drop the "running" breadcrumb (covers GUI + MCP-server runs). - self._run_state_writer = squid.acquisition_state.NullRunStateWriter() - try: - expected = { - "timepoints": self.Nt, - "regions": len(scan_position_information.scan_region_coords_mm), - "fovs": sum(len(c) for c in scan_position_information.scan_region_fov_coords_mm.values()), - "channels": len(self.selected_configurations), - "z": self.NZ, - } - config_path = (getattr(control._def, "CACHED_CONFIG_FILE_PATH", None) or "").strip() or None - self._run_state_writer = squid.acquisition_state.RunStateWriter.start( - experiment_id=self.experiment_ID, - pid=os.getpid(), - config_path=config_path, - output_path=experiment_path, - expected=expected, - ) - except Exception as e: - self._log.warning(f"Failed to write acquisition watchdog start state: {e}") -``` - -Then add `run_state_writer=self._run_state_writer,` to the `MultiPointWorker(...)` constructor call (within the kwargs block at lines 873–888): - -```python - prewarmed_bp_values=prewarmed_bp_values, - run_state_writer=self._run_state_writer, - ) -``` - -(`os` and `control._def` are already imported in this module; confirm with `grep -n "^import os" software/control/core/multi_point_controller.py` and add `import os` if absent.) - -- [ ] **Step 4: Smoke-test the wiring** - -```python -# tests/control/test_watchdog_breadcrumbs.py -import os - -import squid.acquisition_state as ast -import control.microscope -import tests.control.gui_test_stubs as gts - - -def test_run_acquisition_writes_running_breadcrumb(qtbot): - scope = control.microscope.Microscope.build_from_global_config(True) - mpc = gts.get_test_qt_multi_point_controller(microscope=scope) - mpc.run_acquisition() - rec = ast.read_run(os.environ["SQUID_WATCHDOG_STATE_DIR"]) - assert rec is not None - assert rec["status"] == "running" - assert rec["pid"] == os.getpid() - assert rec["expected"]["timepoints"] >= 1 - mpc.request_abort_aquisition() - scope.close() -``` - -Run: `cd software && python3 -m pytest tests/control/test_watchdog_breadcrumbs.py -v` -Expected: PASS (start breadcrumb present). The `ended` transition is verified in Task 12. - -- [ ] **Step 5: Commit** - -```bash -git add software/control/core/multi_point_controller.py software/control/core/multi_point_worker.py software/tests/control/conftest.py software/tests/control/test_watchdog_breadcrumbs.py -git commit -m "feat(watchdog): write acquisition start breadcrumb from the engine" -``` - ---- - -## Task 9: Engine — heartbeat, reason, and end breadcrumb in the worker - -**Files:** -- Modify: `control/core/multi_point_worker.py` (`run`, lines 449–539; `_image_callback` ~line 1200; failed-job path lines 1023–1027) -- Modify: `control/slack_notifier.py` (`AcquisitionStats`, lines 46–53) - -- [ ] **Step 1: Add `reason` field to `AcquisitionStats`** - -In `control/slack_notifier.py`, extend the dataclass (lines 46–53): - -```python -@dataclass -class AcquisitionStats: - """Statistics for a completed acquisition.""" - - total_images: int - total_timepoints: int - total_duration_seconds: float - errors_encountered: int - experiment_id: str - reason: str = "completed" -``` - -- [ ] **Step 2: Add a heartbeat helper + loop beats (worker)** - -In `control/core/multi_point_worker.py`, add a helper method to `MultiPointWorker`: - -```python - def _run_state_beat(self) -> None: - self._run_state.beat( - { - "timepoint": self.time_point, - "expected_timepoints": self.Nt, - "fov": self._timepoint_fov_count, - "images": self.image_count, - } - ) -``` - -Insert `self._run_state_beat()` at three points in `run()`: - -(a) Right after the top-of-loop abort check (after line 453 `break`), as the first statement of the loop body when not aborting: - -```python - while self.time_point < self.Nt: - # check if abort acquisition has been requested - if self.abort_requested_fn(): - self._log.debug("In run, abort_acquisition_requested=True") - break - self._run_state_beat() -``` - -(b) Inside the timed-acquisition wait loop (lines 494–498), so dt gaps keep the heartbeat fresh: - -```python - while time.time() < self.timestamp_acquisition_started + self.time_point * self.dt: - if self.abort_requested_fn(): - self._log.debug("In run wait loop, abort_acquisition_requested=True") - break - self._run_state_beat() - self._sleep(sleep_time) -``` - -(c) In `_image_callback`, immediately after `self.image_count` is incremented (~line 1200), so long single-timepoint scans keep beating with real imaging progress: - -```python - self.image_count += 1 - self._run_state_beat() -``` - -- [ ] **Step 3: Tag error-driven aborts** - -In the `except TimeoutError` handler (lines 507–510), set the cause before requesting abort: - -```python - except TimeoutError as te: - self._log.error(f"Operation timed out during acquisition, aborting acquisition!") - self._log.error(te) - self._abort_cause = "error" - self.request_abort_fn() -``` - -In the failed-job abort path (lines 1023–1027): - -```python - if not result.none_failed and self._abort_on_failed_job: - self._log.error("Some jobs failed, aborting acquisition because abort_on_failed_job=True") - self._abort_cause = "error" - self.request_abort_fn() - return -``` - -- [ ] **Step 4: Compute `reason` and write the end breadcrumb in `finally`** - -Set a fatal-error flag in the generic handler (lines 511–513): - -```python - except Exception as e: - self._log.exception(e) - self._run_state_fatal = True - raise -``` - -Initialize the flag at the very top of `run()` (next to `this_image_callback_id = None`, line 425): - -```python - def run(self): - this_image_callback_id = None - self._run_state_fatal = False -``` - -In the `finally` block, replace the existing Slack-finish block — from `if self._slack_notifier is not None:` (~line 526) through the final `self.callbacks.signal_acquisition_finished()` (line 539) — so it computes `reason`, writes the end breadcrumb, passes `reason` to `AcquisitionStats`, and still calls `signal_acquisition_finished()` exactly once. The replacement: - -```python - # Determine why the acquisition ended (drives the watchdog + the in-process finish msg). - if self._run_state_fatal: - reason = "error" - elif self.abort_requested_fn(): - reason = "error" if self._abort_cause == "error" else "user_abort" - elif self._acquisition_error_count > 0: - reason = "completed_with_errors" - else: - reason = "completed" - - total_duration = time.time() - self.timestamp_acquisition_started - self._run_state.end( - reason, - { - "total_images": self.image_count, - "total_timepoints": self.time_point, - "total_duration_seconds": total_duration, - "errors_encountered": self._acquisition_error_count, - }, - ) - - # Send Slack acquisition finished notification via callback (ensures ordering with timepoint notifications) - if self._slack_notifier is not None: - try: - stats = AcquisitionStats( - total_images=self.image_count, - total_timepoints=self.time_point, - total_duration_seconds=total_duration, - errors_encountered=self._acquisition_error_count, - experiment_id=self.experiment_ID or "unknown", - reason=reason, - ) - self.callbacks.signal_slack_acquisition_finished(stats) - except Exception as e: - self._log.warning(f"Failed to send Slack acquisition finished notification: {e}") - - self.callbacks.signal_acquisition_finished() -``` - -- [ ] **Step 5: Unit-test the reason logic in isolation** - -```python -# tests/control/test_worker_reason.py -import time -from unittest.mock import MagicMock - -import squid.acquisition_state as ast -from control.core.multi_point_worker import MultiPointWorker - - -def _make_worker(tmp_path, monkeypatch): - # Build a bare worker without running __init__ (we only exercise the finally logic helpers). - w = MultiPointWorker.__new__(MultiPointWorker) - w.time_point = 2 - w.Nt = 5 - w.image_count = 40 - w._acquisition_error_count = 0 - w._abort_cause = None - w._run_state_fatal = False - w.experiment_ID = "e" - w.timestamp_acquisition_started = time.time() - 1 - w._run_state = ast.RunStateWriter.start( - experiment_id="e", pid=1, config_path=None, output_path="o", - expected={}, state_dir=tmp_path, - ) - w.abort_requested_fn = lambda: False - return w - - -def _reason(w): - # Mirror the finally classification. - if w._run_state_fatal: - return "error" - if w.abort_requested_fn(): - return "error" if w._abort_cause == "error" else "user_abort" - if w._acquisition_error_count > 0: - return "completed_with_errors" - return "completed" - - -def test_reason_completed(tmp_path, monkeypatch): - assert _reason(_make_worker(tmp_path, monkeypatch)) == "completed" - - -def test_reason_user_abort(tmp_path, monkeypatch): - w = _make_worker(tmp_path, monkeypatch) - w.abort_requested_fn = lambda: True - assert _reason(w) == "user_abort" - - -def test_reason_error_on_timeout_abort(tmp_path, monkeypatch): - w = _make_worker(tmp_path, monkeypatch) - w.abort_requested_fn = lambda: True - w._abort_cause = "error" - assert _reason(w) == "error" - - -def test_reason_completed_with_errors(tmp_path, monkeypatch): - w = _make_worker(tmp_path, monkeypatch) - w._acquisition_error_count = 3 - assert _reason(w) == "completed_with_errors" -``` - -Run: `cd software && python3 -m pytest tests/control/test_worker_reason.py -v` -Expected: PASS (4 tests). (This test pins the classification table; the in-context version is exercised end-to-end in Task 12.) - -- [ ] **Step 6: Commit** - -```bash -git add software/control/core/multi_point_worker.py software/control/slack_notifier.py software/tests/control/test_worker_reason.py -git commit -m "feat(watchdog): heartbeat + end-reason breadcrumb in acquisition worker" -``` - ---- - -## Task 10: Notifier trim — gate the finish message on a clean end - -**Files:** -- Modify: `control/slack_notifier.py` (`notify_acquisition_finished`, lines 610–646) -- Test: `tests/control/test_notifier_trim.py` - -- [ ] **Step 1: Write the failing test** - -```python -# tests/control/test_notifier_trim.py -from unittest.mock import patch - -import control._def -from control.slack_notifier import SlackNotifier, AcquisitionStats - - -def _stats(reason): - return AcquisitionStats( - total_images=10, total_timepoints=2, total_duration_seconds=5.0, - errors_encountered=0, experiment_id="e", reason=reason, - ) - - -def test_finish_message_sent_only_on_clean_completion(monkeypatch): - monkeypatch.setattr(control._def.SlackNotifications, "NOTIFY_ON_ACQUISITION_FINISHED", True) - n = SlackNotifier(bot_token="x", channel_id="C") - with patch.object(n, "_queue_message") as q: - n.notify_acquisition_finished(_stats("completed")) - assert q.call_count == 1 - - with patch.object(n, "_queue_message") as q: - n.notify_acquisition_finished(_stats("error")) - n.notify_acquisition_finished(_stats("user_abort")) - n.notify_acquisition_finished(_stats("completed_with_errors")) - assert q.call_count == 0 # watchdog owns these alerts -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd software && python3 -m pytest tests/control/test_notifier_trim.py -v` -Expected: FAIL — finish message is queued for all reasons (`assert 3 == 0`). - -- [ ] **Step 3: Edit `notify_acquisition_finished`** - -In `control/slack_notifier.py`, add a guard right after the existing `NOTIFY_ON_ACQUISITION_FINISHED` check at the top of `notify_acquisition_finished` (line ~611): - -```python - def notify_acquisition_finished(self, stats: AcquisitionStats): - if not control._def.SlackNotifications.NOTIFY_ON_ACQUISITION_FINISHED: - return - if stats.reason != "completed": - # Premature/degraded ends are reported once by the acquisition watchdog. - return -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd software && python3 -m pytest tests/control/test_notifier_trim.py -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add software/control/slack_notifier.py software/tests/control/test_notifier_trim.py -git commit -m "feat(watchdog): notifier reports only clean finishes; watchdog owns premature alerts" -``` - ---- - -## Task 11: Shutdown hook — abort + join on close - -**Files:** -- Modify: `main_hcs.py` (shutdown sequence, lines 437–439) - -- [ ] **Step 1: Edit the shutdown sequence** - -In `main_hcs.py`, replace the shutdown tail (lines 437–439): - -```python - exit_code = app.exec_() - logging.shutdown() # Flush log handlers before os._exit() bypasses Python cleanup - os._exit(exit_code) -``` - -with: - -```python - exit_code = app.exec_() - - # If the app is quitting mid-acquisition, request the normal abort and let the worker - # write its end breadcrumb so the watchdog reports "aborted" rather than a crash. - try: - mpc = getattr(win, "multipointController", None) - if mpc is not None and mpc.acquisition_in_progress(): - log.info("Acquisition in progress at shutdown; requesting abort before exit.") - mpc.request_abort_aquisition() - if getattr(mpc, "thread", None) is not None: - mpc.thread.join(timeout=15.0) - except Exception as e: - log.warning(f"Error during shutdown abort handling: {e}") - - logging.shutdown() # Flush log handlers before os._exit() bypasses Python cleanup - os._exit(exit_code) -``` - -- [ ] **Step 2: Verify it imports and the app still launches** - -Run: `cd software && python3 -c "import ast; ast.parse(open('main_hcs.py').read()); print('parse ok')"` -Expected: `parse ok`. - -Run (manual smoke, simulation): `cd software && timeout 25 python3 main_hcs.py --simulation` — confirm the GUI starts and closes cleanly with no traceback from the new block. (No automated test: `main_hcs.py` is excluded from CI and drives the full GUI.) - -- [ ] **Step 3: Commit** - -```bash -git add software/main_hcs.py -git commit -m "feat(watchdog): write an aborted breadcrumb when quitting mid-acquisition" -``` - ---- - -## Task 12: Integration test — full breadcrumb lifecycle - -**Files:** -- Create: `tests/control/test_watchdog_integration.py` - -- [ ] **Step 1: Write the test** - -```python -# tests/control/test_watchdog_integration.py -import os -import time - -import squid.acquisition_state as ast -import control.microscope -import tests.control.gui_test_stubs as gts - - -def _wait_for(predicate, timeout=30.0, interval=0.2): - deadline = time.time() + timeout - while time.time() < deadline: - if predicate(): - return True - time.sleep(interval) - return False - - -def test_simulated_acquisition_writes_ended_breadcrumb(qtbot): - state_dir = os.environ["SQUID_WATCHDOG_STATE_DIR"] - scope = control.microscope.Microscope.build_from_global_config(True) - mpc = gts.get_test_qt_multi_point_controller(microscope=scope) - - mpc.run_acquisition() - assert _wait_for(lambda: ast.read_run(state_dir) is not None) - assert ast.read_run(state_dir)["status"] == "running" - - # Let it finish (the default test acquisition is short); fall back to abort. - finished = _wait_for(lambda: (ast.read_run(state_dir) or {}).get("status") == "ended", timeout=20.0) - if not finished: - mpc.request_abort_aquisition() - assert _wait_for(lambda: (ast.read_run(state_dir) or {}).get("status") == "ended", timeout=20.0) - - rec = ast.read_run(state_dir) - assert rec["status"] == "ended" - assert rec["reason"] in {"completed", "completed_with_errors", "user_abort", "error"} - assert rec["ended_at"] is not None - scope.close() -``` - -- [ ] **Step 2: Run the test** - -Run: `cd software && python3 -m pytest tests/control/test_watchdog_integration.py -v` -Expected: PASS — `run.json` transitions `running → ended` with a valid reason and `heartbeat_at`/`ended_at` populated. - -- [ ] **Step 3: Commit** - -```bash -git add software/tests/control/test_watchdog_integration.py -git commit -m "test(watchdog): end-to-end breadcrumb lifecycle in simulation" -``` - ---- - -## Task 13: Service recipes + README - -**Files:** -- Create: `acquisition_watchdog/systemd/squid-acquisition-watchdog.service` -- Create: `acquisition_watchdog/windows/squid-acquisition-watchdog.xml` -- Create: `acquisition_watchdog/windows/install.ps1` -- Create: `acquisition_watchdog/README.md` - -- [ ] **Step 1: Linux systemd user unit** - -```ini -# acquisition_watchdog/systemd/squid-acquisition-watchdog.service -# Install (per user): -# mkdir -p ~/.config/systemd/user -# cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/ -# # edit WorkingDirectory + --config below to match this machine, then: -# systemctl --user daemon-reload -# systemctl --user enable --now squid-acquisition-watchdog -[Unit] -Description=Squid acquisition watchdog (alerts on prematurely-ended acquisitions) -After=default.target - -[Service] -Type=simple -WorkingDirectory=%h/Squid/software -ExecStart=/usr/bin/python3 -m acquisition_watchdog --config %h/Squid/software/configuration.ini -Restart=always -RestartSec=5 - -[Install] -WantedBy=default.target -``` - -- [ ] **Step 2: Windows Task Scheduler task + installer** - -```xml - - - - - Squid acquisition watchdog (alerts on prematurely-ended acquisitions) - - - - true - - - - IgnoreNew - false - - PT1M - 999 - - PT0S - - - - pythonw.exe - -m acquisition_watchdog --config C:\Squid\software\configuration.ini - C:\Squid\software - - - -``` - -```powershell -# acquisition_watchdog/windows/install.ps1 -# Run in PowerShell from software\ : .\acquisition_watchdog\windows\install.ps1 -$ErrorActionPreference = "Stop" -$taskName = "SquidAcquisitionWatchdog" -$xmlPath = Join-Path $PSScriptRoot "squid-acquisition-watchdog.xml" -Write-Host "Registering scheduled task '$taskName' from $xmlPath" -Register-ScheduledTask -TaskName $taskName -Xml (Get-Content $xmlPath -Raw) -Force -Write-Host "Done. Edit the task's --config/WorkingDirectory if your install path differs, then log off/on or 'Start' the task." -``` - -- [ ] **Step 3: README** - -```markdown -# acquisition_watchdog/README.md -# Acquisition Watchdog - -Independent process that alerts (via Slack) when a Squid acquisition ends -prematurely — process crash/hang/kill, fatal error, or user abort. Covers runs -launched from the GUI and from the MCP control server. - -## How it works -The Squid GUI writes a `run.json` breadcrumb (start / throttled heartbeat / end) -into a shared state dir. This watchdog polls it and posts one Slack alert when a -run dies, hangs, or ends with a non-clean reason. Clean completions are silent. - -## Run it - cd software - python3 -m acquisition_watchdog --config ./configuration.ini - -Options: `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. - -Slack credentials are read from the `[SlackNotifications]` section of the active -`.ini` (same `bot_token` / `channel_id` the GUI uses). Set `watchdog_enabled = False` -in that section to disable watchdog alerts on a machine. - -## Install as an always-on service -- **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). -- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). - -## State dir -Defaults to `platformdirs.user_state_path("squid","cephla")/watchdog`. Override with -`SQUID_WATCHDOG_STATE_DIR` (must match the GUI's environment) or `--state-dir`. - -## Remote / power-loss coverage (future) -Point `--state-dir` at a shared/synced mount on another host and run this process -there. Per-machine `run-.json` naming and clock-skew tolerance are needed -first (see the design spec, "Future work"). -``` - -- [ ] **Step 4: Commit** - -```bash -git add software/acquisition_watchdog/systemd software/acquisition_watchdog/windows software/acquisition_watchdog/README.md -git commit -m "docs(watchdog): add systemd + Windows service recipes and README" -``` - ---- - -## Task 14: Finalize — format, full test run, commit the spec - -**Files:** -- All new/modified files -- `docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md` - -- [ ] **Step 1: Format with Black** - -Run: `cd software && black --config pyproject.toml squid/slack.py squid/acquisition_state.py acquisition_watchdog/ tests/squid/ tests/acquisition_watchdog/ tests/control/test_watchdog_breadcrumbs.py tests/control/test_watchdog_integration.py tests/control/test_worker_reason.py tests/control/test_notifier_trim.py tests/control/test_slack_notifier_send.py control/slack_notifier.py control/core/multi_point_worker.py control/core/multi_point_controller.py main_hcs.py` -Expected: files reformatted/unchanged; no errors. - -- [ ] **Step 2: Run the watchdog + new unit tests** - -Run: `cd software && python3 -m pytest tests/squid tests/acquisition_watchdog tests/control/test_worker_reason.py tests/control/test_notifier_trim.py tests/control/test_slack_notifier_send.py -v` -Expected: ALL PASS. - -- [ ] **Step 3: Run the engine/integration tests** - -Run: `cd software && python3 -m pytest tests/control/test_watchdog_breadcrumbs.py tests/control/test_watchdog_integration.py tests/control/test_MultiPointWorker.py -v` -Expected: ALL PASS (no regression in the existing worker test). - -- [ ] **Step 4: Full suite (CI parity)** - -Run: `cd software && python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py` -Expected: no new failures attributable to these changes. - -- [ ] **Step 5: Commit the spec + plan and verify Black on the whole tree** - -Run: `cd software && black --config pyproject.toml --check .` -Expected: "All done!" (no files would be reformatted). - -```bash -git add software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md software/docs/superpowers/plans/2026-06-23-acquisition-watchdog.md -git commit -m "docs(watchdog): add design spec and implementation plan" -``` - ---- - -## Self-review notes - -- **Spec coverage:** start/heartbeat/end protocol (Tasks 3, 8, 9); watchdog poll/classify/dedup (Task 6); config sharing (Task 4); cross-platform state dir + PID degrade (Tasks 3, 6); deployment recipes (Task 13); notifier split (Tasks 2, 10); server coverage (engine-level instrumentation in Tasks 8–9 — no server-specific code needed). v1 out-of-scope items (progress-stall, power-loss, server-thread health) are intentionally absent. -- **`app_closed`** from the spec is implemented as `user_abort` via the shutdown abort+join (Task 11); noted in the taxonomy above. Update the spec's taxonomy/ shutdown wording to match (done as part of plan authoring). -- **Type consistency:** `RunStateWriter.start(...)`/`beat`/`end`, `read_run`, `default_state_dir`, `NullRunStateWriter` used identically across Tasks 3/6/8/9/12; `SlackConfig` fields (`enabled`,`bot_token`,`channel_id`,`watchdog_enabled`) consistent across Tasks 4/6; `AcquisitionStats.reason` added in Task 9 and consumed in Task 10; `squid.slack.post_message(token, channel, text, blocks)` signature consistent across Tasks 1/2/6. diff --git a/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md b/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md deleted file mode 100644 index 70e77f61b..000000000 --- a/software/docs/superpowers/specs/2026-06-23-acquisition-watchdog-design.md +++ /dev/null @@ -1,229 +0,0 @@ -# Acquisition Watchdog - -Detect when an acquisition ends prematurely — process **crash / hang / kill**, **fatal -error**, or **user abort** — and send a single Slack alert. Works for acquisitions -launched from the GUI *and* from the MCP control server, on Ubuntu and Windows. - -## Motivation - -A crashing process cannot report its own death. The existing in-process -`SlackNotifier` (`control/slack_notifier.py`) runs on a daemon thread *inside* the GUI -process, so it can report live errors and a clean finish, but it can never report a -segfault in a camera SDK, an OOM-kill, `os._exit()`, a power-loss of the process, or a -frozen UI — the thing that would send the alert is the thing that died. - -The fix is an **independent process** that watches on-disk breadcrumbs the app leaves -behind. Because both the GUI and the MCP control server run acquisitions through the -same engine (`MultiPointController.run_acquisition()` → `MultiPointWorker.run()`, both -in `control/core/`), instrumenting the **engine** — not the GUI widgets — covers both -launch paths with no extra code. Server-driven runs are unattended, which is exactly -when an alert matters most. - -## Architecture - -Three parts, with a clean dependency DAG (`acquisition_watchdog` → `squid`; `control` → -`squid`; the watchdog never imports `control`): - -``` - GUI process (main_hcs.py, incl. in-process MCP control server) - ┌───────────────────────────────────────────────┐ - │ MultiPointController.run_acquisition() │ writes - │ MultiPointWorker.run(): │ ─────────► /run.json - │ start → write run.json (status=running) │ (atomic os.replace) - │ loop → beat() heartbeat + progress (~5s) │ - │ finally→ write end (status=ended, reason) │ - │ in-process SlackNotifier (unchanged role): │ - │ live errors, progress, finish-with-mosaic │ - └───────────────────────────────────────────────┘ - reads - acquisition_watchdog (independent always-on process) ◄──────── /run.json - poll every ~5s → classify → Slack alert (once per run_id) reads bot_token/channel_id - from cache/slack_settings.yaml -``` - -### Part 1 — Breadcrumb protocol (in the acquisition engine) - -A new leaf module `squid/acquisition_state.py` owns the run-state schema and atomic -read/write. The acquisition engine writes; the watchdog reads. It must stay -import-light (stdlib only) and must not import `control`. - -**`run.json`** — a single file in the shared state dir, replaced atomically -(`os.replace`, atomic and torn-read-free on both POSIX and Windows): - -| Field | Type | Notes | -|---|---|---| -| `schema_version` | int | `1` | -| `run_id` | str | `uuid4().hex`; the watchdog's dedup key | -| `experiment_id` | str | from `MultiPointController` | -| `machine` | str | config machine-name if present, else `socket.gethostname()` | -| `pid` | int | `os.getpid()` of the GUI process | -| `config_path` | str | absolute path of the active `.ini` (so the watchdog finds Slack settings with no args) | -| `output_path` | str | experiment output dir | -| `started_at` | float | epoch seconds, UTC | -| `heartbeat_at` | float | epoch seconds; bumped ~every `HEARTBEAT_INTERVAL_S` | -| `progress` | obj | `{timepoint, expected_timepoints, fov, region_fovs, images}` | -| `expected` | obj | `{timepoints, regions, fovs, channels, z}` — `fovs` is total planned across all regions | -| `status` | str | `running` \| `ended` | -| `reason` | str\|null | set when `ended` — see taxonomy below | -| `ended_at` | float\|null | epoch seconds | -| `stats` | obj\|null | `{total_images, errors_encountered, total_duration_seconds}` at end | - -**Write points in the engine:** - -1. **Start** — in `MultiPointController.run_acquisition()`, which owns the experiment id - and acquisition parameters: write the full record with `status=running`, a fresh - `run_id`, and `expected` totals. The `run_id` is handed to the worker so its heartbeat - and end writes update the same record. -2. **Heartbeat** — a `HeartbeatWriter.beat(progress)` helper, called at the worker - loop's existing abort-check points (the per-timepoint, per-FOV, and the long-wait - poll loops in `MultiPointWorker.run()`). `beat()` is cheap: it updates an in-memory - timestamp and **flushes to disk at most every `HEARTBEAT_INTERVAL_S` (default 5 s)**. - Because the long-wait loops (timelapse `dt`, fluidics) already poll the abort flag, - the heartbeat stays fresh whenever the worker thread is alive and freezes only on a - true hang or process death. -3. **End** — in the `finally` of `MultiPointWorker.run()` (which already runs for normal - finish, abort, and caught exceptions): write `status=ended`, `reason`, `ended_at`, - `stats` (reuse the existing `AcquisitionStats` / `_acquisition_error_count`). -4. **App close while acquiring** — `main_hcs.py` shuts down via `os._exit()` (skips - destructors). In the shutdown path, if `acquisition_in_progress()`, request the - normal abort and **join the worker** (bounded timeout) before `os._exit()`, so the - worker writes its normal `user_abort` end record and a deliberate quit is not - misreported as a crash. (A distinct `app_closed` reason is future work.) - -**Reason taxonomy** (computed at the end write): - -| `reason` | When | Watchdog alerts? | -|---|---|---| -| `completed` | loop finished all timepoints, `errors_encountered == 0` | no (silent) | -| `completed_with_errors` | loop finished but `errors_encountered > 0` | yes | -| `error` | uncaught exception, or auto-abort from `TimeoutError` / failed-job abort | yes | -| `user_abort` | abort flag set externally (human / server) **or** GUI closed mid-run (shutdown aborts + joins) | yes | -| *(no end record)* | process crashed/killed/hung before writing end | yes (crash/hang) | - -To distinguish `error` from `user_abort`, the engine records an **abort cause**: the -auto-abort paths (`TimeoutError`, failed-job abort) tag the cause as error-type; a bare -`request_abort_aquisition()` is `user`. The end write maps cause → reason. (`errors_encountered` -already exists and drives `completed` vs `completed_with_errors`.) - -### Part 2 — The watchdog process (`software/acquisition_watchdog/`) - -Independent, lightweight (stdlib + `pyyaml`), **does not import `control`**. - -- **Poll loop** (every `POLL_INTERVAL_S`, default 5 s): read `run.json`; if absent, idle. -- **Classification:** - - `status=running` and (`pid` not alive **or** `now − heartbeat_at > HEARTBEAT_TIMEOUT_S`) → **crash/hang** → alert. - - `status=ended` and `reason ∈ {completed_with_errors, error, user_abort}` → alert. - - `status=ended` and `reason=completed` → silent. -- **PID check** is a best-effort accelerator catching hard death within one poll: - `psutil.pid_exists(pid)` if `psutil` is importable, else POSIX `os.kill(pid, 0)`, else - skip (heartbeat-only). The **heartbeat is the primary, OS-agnostic signal**; PID just - makes a true crash detectable in ~5 s instead of waiting out the heartbeat timeout. -- **Alert once per `run_id`.** Alerted ids are persisted to `/alerted.json` so - a watchdog restart never re-alerts, and so a crash that happened while the watchdog was - down is alerted exactly once on its next start. -- **Defaults** (overridable via CLI flags / config): `POLL_INTERVAL_S=5`, - `HEARTBEAT_INTERVAL_S=5`, `HEARTBEAT_TIMEOUT_S=120` (comfortably above the longest - legitimate single blocking op — long exposures, stage moves, fluidics — while still - catching a hang within ~2 min). - -**Alert payload:** machine name, experiment id, classification (crash / hang / error / -aborted / completed-with-errors), progress vs expected ("stopped at timepoint 3/10, -360 images"), start + last-heartbeat / end timestamps, output path. - -### Part 3 — Notifier trim (minimal) - -`control/slack_notifier.py` keeps live in-run error warnings, per-timepoint progress, and -the finish-with-mosaic summary. It **stops flagging bad *endings* itself** (the -end-of-run failure messaging moves to the watchdog), so a failed run produces exactly one -alert. The ~20-line Slack `chat.postMessage` send is extracted into a shared, -dependency-free `squid/slack.py` (stdlib `urllib`/`json`, no Qt/`control` imports) used by -both the notifier and the watchdog. Image upload (`files.getUploadURLExternal`) stays in -`SlackNotifier` — the watchdog never needs it. - -## Cross-platform (Ubuntu + Windows) - -- **State dir** via `platformdirs` (already used for logs in `squid/logging.py`): - `default_state_dir()` in `squid/acquisition_state.py` returns - `platformdirs.user_state_path("squid", "cephla") / "watchdog"` — `~/.local/state` (or - `~/.cache`) on Linux, `%LOCALAPPDATA%\cephla\squid\…` on Windows. Writer and reader call - the same helper so they always agree. Overridable via `SQUID_WATCHDOG_STATE_DIR` (both) - and `--state-dir` (watchdog). -- **Atomic writes** use `os.replace` (atomic on both OSes). **PID check** is guarded per - above. No POSIX-only calls on the hot path. - -## Config sharing - -The watchdog reads the **same `cache/slack_settings.yaml`** the GUI writes and loads -(`bot_token` / `channel_id` / `enabled`), resolved cwd-relative or overridden via the -`--slack-settings` flag or `$SQUID_SLACK_SETTINGS` env. It is parsed with `yaml`; the -watchdog never imports `control._def`. A `watchdog_enabled: false` key (default `true`) -disables watchdog alerts on a machine without disabling the in-process GUI notifier. - -## Deployment — always-on user service - -Core process is just `python -m acquisition_watchdog [--slack-settings ]`, identical on -both OSes (run from `software/` so the default `cache/slack_settings.yaml` resolves). Shipped -recipes: - -- **Linux:** a systemd **`--user`** unit (`Restart=always`, `WantedBy=default.target`), - `systemctl --user enable --now squid-acquisition-watchdog`. Runs as the same user as the - GUI, sharing the `platformdirs` state dir. -- **Windows:** a **Task Scheduler** task triggered "at log on" of the user (sample `.xml` - + a small `install.ps1`). Same user, same state dir. - -Both ship in `acquisition_watchdog/` with a README. The identical code can later run as a -**remote monitor** (the option-3 variant) by pointing `--state-dir` at a shared/synced -mount — see Future work. - -## Proposed file layout - -| Path | Role | -|---|---| -| `squid/acquisition_state.py` | run-state schema, `default_state_dir()`, atomic read/write, `HeartbeatWriter` (engine writes, watchdog reads) | -| `squid/slack.py` | shared dependency-free `chat.postMessage` sender | -| `software/acquisition_watchdog/__main__.py` | CLI entry (`python -m acquisition_watchdog`) | -| `software/acquisition_watchdog/monitor.py` | poll loop + classification + dedup | -| `software/acquisition_watchdog/config.py` | resolve & load `cache/slack_settings.yaml` (the GUI's Slack creds) | -| `software/acquisition_watchdog/alerts.py` | format the Slack alert payload | -| `software/acquisition_watchdog/systemd/`, `windows/`, `README.md` | install recipes + docs | -| `control/core/multi_point_controller.py`, `control/core/multi_point_worker.py` | write breadcrumbs (start / beat / end) + abort-cause tagging | -| `control/slack_notifier.py` | stop end-of-run failure messaging; call `squid/slack.py` | -| `main_hcs.py` | write `app_closed` end record on shutdown-while-acquiring | - -Named `acquisition_watchdog` (not `watchdog`) to avoid colliding with the PyPI `watchdog` -filesystem-events package. - -## Tests - -- `squid/acquisition_state.py`: round-trip (start → beats → end), atomic-replace, schema - versioning; `beat()` throttling (many calls, ≤1 flush per interval). -- `acquisition_watchdog/monitor.py`: classification table — synthetic `run.json` for each - state (`running`+stale heartbeat, `running`+dead PID, each `ended` reason, `completed`) - → expected alert/no-alert; dedup (no double alert per `run_id`, persists across a - monitor restart via `alerted.json`). -- `acquisition_watchdog/config.py`: resolution precedence (`--slack-settings` > env > - default `cache/slack_settings.yaml`); missing/disabled Slack → logs, no crash. -- `squid/slack.py`: monkeypatch `urllib`, assert request shape; no network. -- PID check: alive (current pid) vs an impossible/known-dead pid, on the available - platform; graceful degrade when `psutil` absent. -- Engine integration (simulation mode): run a short simulated acquisition and assert - `run.json` transitions `running → ended/completed` and `heartbeat_at` advances. Mirror - the abort path → `reason=user_abort`, and a forced job error → `completed_with_errors`. -- Black (120) over the new package; it is not in the formatter excludes. - -## Out of scope (v1) - -- **Progress-stall detection** (process alive but worker wedged) — needs per-step timing - from `acquisition.yaml`; fragile, deferred. v1 catches death + full hang + abort + error. -- **Machine power-loss coverage** — needs the remote-monitor variant. -- **MCP control-server thread health** — server-thread death does not abort an in-flight - acquisition, so it is not an "acquisition ended" event. -- **Multi-microscope aggregation**, a GUI panel for the watchdog, and secrets management - for the bot token (stays in the `.ini` as today). - -## Future work - -- **Remote monitor:** point `--state-dir` at a shared mount; key the state file per - machine (`run-.json`) to avoid collisions, and add a clock-skew tolerance to - the heartbeat comparison (writer/reader no longer share a clock). -- Progress-stall detection; packaging the per-OS install into one script. From 606f67c70214101ad4c5e28f29b638faed148551 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 30 Jun 2026 21:29:23 -0700 Subject: [PATCH 22/23] docs(watchdog): expand README into a full how-to-use guide Setup (Slack token via GUI), running (manual + always-on service), the GUI 'Enable watchdog alerts' toggle, how to verify, alert taxonomy, behavior notes (GUI-independent, start-order, alert-once), and troubleshooting. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/README.md | 138 +++++++++++++++++++----- 1 file changed, 113 insertions(+), 25 deletions(-) diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md index d35ae8c4a..0d8893a70 100644 --- a/software/acquisition_watchdog/README.md +++ b/software/acquisition_watchdog/README.md @@ -1,38 +1,126 @@ # Acquisition Watchdog -Independent process that alerts (via Slack) when a Squid acquisition ends -prematurely — process crash/hang/kill, fatal error, or user abort. Covers runs -launched from the GUI and from the MCP control server. +An independent process that watches Squid acquisitions and posts a **single Slack alert** +when one ends prematurely — a process **crash / hang / kill**, a **fatal error**, or a +**user abort**. Clean completions stay silent ("no news is good news"). It covers +acquisitions started from the **GUI** and from the **MCP control server**, on **Ubuntu and +Windows**. + +Because it runs as a *separate* process, it can report failures the in-app Slack notifier +can't — a segfault in a camera SDK, an OOM-kill, a frozen UI, or the whole process dying. ## How it works -The Squid GUI writes a `run.json` breadcrumb (start / throttled heartbeat / end) -into a shared state dir. This watchdog polls it and posts one Slack alert when a -run dies, hangs, or ends with a non-clean reason. Clean completions are silent. + +The Squid GUI writes a small `run.json` breadcrumb into a shared state dir: `status=running` +at acquisition start, a throttled heartbeat (+ progress) during the run, and `status=ended` +with a reason at the end. This watchdog polls that file and alerts when a run's process has +died / gone silent, or ended with a non-clean reason. One alert per run (de-duplicated). + +## Prerequisites + +1. **Slack configured in the GUI.** Open *Settings → Slack Notifications*, enter your **Bot + Token** (`xoxb-…`) and **Channel ID** (`C…`), and click *Save*. That writes + `cache/slack_settings.yaml`, which the watchdog reads — there is no separate config. + (Need to create the token? See [`../docs/slack_notifications.md`](../docs/slack_notifications.md).) +2. **"Enable watchdog alerts" checked** (the default) in that same dialog — or the + `watchdog_enabled` key in `cache/slack_settings.yaml`. ## Run it - cd software - python3 -m acquisition_watchdog -Options: `--slack-settings`, `--state-dir`, `--poll-interval` (5s), `--heartbeat-timeout` (120s), `--once`. +From the `software/` directory: + +```bash +cd software +python3 -m acquisition_watchdog +``` + +Leave it running. Start an acquisition; if it crashes / hangs / aborts / errors, you get a +Slack alert. A clean finish produces nothing. + +### Options + +| Flag | Default | Purpose | +|---|---|---| +| `--heartbeat-timeout` | `120` | Seconds of heartbeat silence (with a live process) before declaring a hang. Raise it if you have very long single exposures / fluidics steps. | +| `--poll-interval` | `5` | Seconds between checks. | +| `--once` | — | Run a single check and exit (handy for testing or a cron probe). | +| `--slack-settings ` | `cache/slack_settings.yaml` | Only needed if you don't run from `software/`. | +| `--state-dir ` | platformdirs user-state dir | Must match the GUI's; override here or via `$SQUID_WATCHDOG_STATE_DIR`. | + +## Run it always-on (recommended for a lab microscope) + +A manual run stops when you close the terminal or reboot. To keep it up independently of the +GUI: + +- **Linux (systemd user service):** copy `systemd/squid-acquisition-watchdog.service` to + `~/.config/systemd/user/`, edit `WorkingDirectory` to your `software/` path, then: + ```bash + systemctl --user daemon-reload + systemctl --user enable --now squid-acquisition-watchdog + ``` + It starts at login, restarts on failure, and keeps running across GUI restarts. +- **Windows (Task Scheduler):** run `windows/install.ps1` in PowerShell from `software\`. It + registers a logon-triggered task (via `pythonw.exe`; make sure it's on `PATH`, or edit the + path in the script). -Slack credentials are read from `cache/slack_settings.yaml` — the same file the GUI's -Slack settings dialog writes (keys `bot_token`, `channel_id`, `enabled`). Run the -watchdog from the `software/` directory (so the default `cache/slack_settings.yaml` -path resolves), or pass `--slack-settings `. To disable watchdog alerts on a -machine without disabling the GUI's notifications, uncheck **"Enable watchdog alerts"** -in the GUI's Slack settings dialog (or set `watchdog_enabled: false` in that YAML directly). +## Verify it works -## Install as an always-on service -- **Linux:** see `systemd/squid-acquisition-watchdog.service` (header has steps). -- **Windows:** run `windows/install.ps1` (registers a logon-triggered task). Ensure - `pythonw.exe` is on `PATH`, or edit the `-Execute` value in `install.ps1` to the full - Python path. +```bash +python3 -m acquisition_watchdog --once # one check, then exits — no error means it's healthy +``` + +End-to-end: start a `--simulation` acquisition, `kill -9` the GUI process, and you should get +a crash alert — within one poll (~5 s) when `psutil` is installed (it is, by default), +otherwise within `--heartbeat-timeout` seconds. + +## What triggers an alert + +| Situation | Alert | +|---|---| +| Process died — `running` breadcrumb + PID gone (crash / OOM-kill / power loss) | 🔴 crash | +| Process alive but no heartbeat past the timeout | 🟠 hang | +| Fatal error / auto-abort (timeout, failed save job, camera/frame failure) | 🔴 error | +| Finished, but some save/job errors occurred | 🟠 completed-with-errors | +| Aborted by the user, the MCP server, or by closing the GUI mid-run | 🟡 aborted | +| Finished cleanly | *(silent)* | + +Each alert includes the machine name, experiment, reason, and progress (e.g. "stopped at +timepoint 3/10"). + +## Good to know + +- **Independent of the GUI.** Restarting the software neither starts nor stops the watchdog — + it just picks up the next run. That decoupling is the whole point: a watchdog spawned by the + GUI couldn't survive the GUI crashing. +- **Start order doesn't matter.** Start it before, during, or after the GUI. If it starts + mid-run it monitors from there; if it starts *after* a crash already happened, it reads the + stale `running` breadcrumb, sees the PID is dead, and alerts once. +- **One alert per run.** Alerted run IDs persist in `/alerted.json`, so it never + double-alerts and never re-alerts after a restart. +- **Turn alerts off** on a machine (without disabling the GUI notifier): uncheck *"Enable + watchdog alerts"*, or set `watchdog_enabled: false` in `cache/slack_settings.yaml`. Takes + effect on the next check — no watchdog restart needed. +- **Runs on the same machine as the GUI** (it reads local breadcrumb files). For coverage of a + full machine death / power loss, run it on another host pointed at a shared/synced state dir + (see *Remote / power-loss coverage*). + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No alerts at all | Is the watchdog process actually running? Are `bot_token`/`channel_id` set (GUI → *Test Connection*)? Is *"Enable watchdog alerts"* checked? | +| Log says Slack not configured | Run from `software/` so `cache/slack_settings.yaml` resolves, or pass `--slack-settings `. | +| False "hang" alerts | Raise `--heartbeat-timeout` — a single very long exposure/fluidics step can exceed the default. | +| Crash reported slowly (~2 min) | `psutil` missing → falls back to the heartbeat timeout. Install `psutil` for instant PID-based detection. | ## State dir -Defaults to `platformdirs.user_state_path("squid","cephla")/watchdog`. Override with -`SQUID_WATCHDOG_STATE_DIR` (must match the GUI's environment) or `--state-dir`. + +Defaults to `platformdirs.user_state_path("squid", "cephla")/watchdog`. The GUI (writer) and +the watchdog (reader) must agree on it — run both as the same user, or set +`SQUID_WATCHDOG_STATE_DIR` on both (or `--state-dir` on the watchdog). ## Remote / power-loss coverage (future) -Point `--state-dir` at a shared/synced mount on another host and run this process -there. Per-machine `run-.json` naming and clock-skew tolerance are needed -first (see the design spec, "Future work"). + +Point `--state-dir` at a shared/synced mount on another host and run this process there. +Per-machine `run-.json` naming and a clock-skew tolerance are needed first (see the +design spec). From 67a2b567f807f70a663c4e7339171794d156cdf7 Mon Sep 17 00:00:00 2001 From: You Yan Date: Tue, 30 Jun 2026 21:37:41 -0700 Subject: [PATCH 23/23] docs(watchdog): flesh out the systemd install steps in the README Full copy/enable sequence (mkdir, cp, WorkingDirectory fix via $PWD, daemon-reload, enable --now, status), plus logs (journalctl), enable-linger, and the interpreter note. Co-Authored-By: Claude Opus 4.8 (1M context) --- software/acquisition_watchdog/README.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/software/acquisition_watchdog/README.md b/software/acquisition_watchdog/README.md index 0d8893a70..e8d16457b 100644 --- a/software/acquisition_watchdog/README.md +++ b/software/acquisition_watchdog/README.md @@ -52,13 +52,21 @@ Slack alert. A clean finish produces nothing. A manual run stops when you close the terminal or reboot. To keep it up independently of the GUI: -- **Linux (systemd user service):** copy `systemd/squid-acquisition-watchdog.service` to - `~/.config/systemd/user/`, edit `WorkingDirectory` to your `software/` path, then: +- **Linux (systemd user service)** — run these from the `software/` directory: ```bash + mkdir -p ~/.config/systemd/user + cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/ + # The shipped unit's WorkingDirectory is a placeholder (%h/Squid/software); point it here: + sed -i "s#^WorkingDirectory=.*#WorkingDirectory=$PWD#" ~/.config/systemd/user/squid-acquisition-watchdog.service systemctl --user daemon-reload - systemctl --user enable --now squid-acquisition-watchdog + systemctl --user enable --now squid-acquisition-watchdog # auto-start at login + start now + systemctl --user status squid-acquisition-watchdog # verify it's active ``` - It starts at login, restarts on failure, and keeps running across GUI restarts. + Enable it once and it comes up at every login and restarts on failure (`Restart=always`) — + no need to launch it by hand. Logs: `journalctl --user -u squid-acquisition-watchdog -f`. + To keep it running before/without a graphical login, also run `loginctl enable-linger $USER` + once. (The unit runs `/usr/bin/python3`; if Squid runs on a different interpreter/venv, edit + the `ExecStart=` line to that python.) - **Windows (Task Scheduler):** run `windows/install.ps1` in PowerShell from `software\`. It registers a logon-triggered task (via `pythonw.exe`; make sure it's on `PATH`, or edit the path in the script).