feat: acquisition watchdog — Slack alert on prematurely-ended acquisitions

software/acquisition_watchdog/README.md

@@ -0,0 +1,134 @@
+# Acquisition Watchdog
+
+An independent process that watches Squid acquisitions and posts a **single Slack alert**
+when one ends prematurely — a process **crash / hang / kill**, a **fatal error**, or a
+**user abort**. Clean completions stay silent ("no news is good news"). It covers
+acquisitions started from the **GUI** and from the **MCP control server**, on **Ubuntu and
+Windows**.
+
+Because it runs as a *separate* process, it can report failures the in-app Slack notifier
+can't — a segfault in a camera SDK, an OOM-kill, a frozen UI, or the whole process dying.
+
+## How it works
+
+The Squid GUI writes a small `run.json` breadcrumb into a shared state dir: `status=running`
+at acquisition start, a throttled heartbeat (+ progress) during the run, and `status=ended`
+with a reason at the end. This watchdog polls that file and alerts when a run's process has
+died / gone silent, or ended with a non-clean reason. One alert per run (de-duplicated).
+
+## Prerequisites
+
+1. **Slack configured in the GUI.** Open *Settings → Slack Notifications*, enter your **Bot
+   Token** (`xoxb-…`) and **Channel ID** (`C…`), and click *Save*. That writes
+   `cache/slack_settings.yaml`, which the watchdog reads — there is no separate config.
+   (Need to create the token? See [`../docs/slack_notifications.md`](../docs/slack_notifications.md).)
+2. **"Enable watchdog alerts" checked** (the default) in that same dialog — or the
+   `watchdog_enabled` key in `cache/slack_settings.yaml`.
+
+## Run it
+
+From the `software/` directory:
+
+```bash
+cd software
+python3 -m acquisition_watchdog
+```
+
+Leave it running. Start an acquisition; if it crashes / hangs / aborts / errors, you get a
+Slack alert. A clean finish produces nothing.
+
+### Options
+
+| Flag | Default | Purpose |
+|---|---|---|
+| `--heartbeat-timeout` | `120` | Seconds of heartbeat silence (with a live process) before declaring a hang. Raise it if you have very long single exposures / fluidics steps. |
+| `--poll-interval` | `5` | Seconds between checks. |
+| `--once` | — | Run a single check and exit (handy for testing or a cron probe). |
+| `--slack-settings <path>` | `cache/slack_settings.yaml` | Only needed if you don't run from `software/`. |
+| `--state-dir <path>` | platformdirs user-state dir | Must match the GUI's; override here or via `$SQUID_WATCHDOG_STATE_DIR`. |
+
+## Run it always-on (recommended for a lab microscope)
+
+A manual run stops when you close the terminal or reboot. To keep it up independently of the
+GUI:
+
+- **Linux (systemd user service)** — run these from the `software/` directory:
+  ```bash
+  mkdir -p ~/.config/systemd/user
+  cp acquisition_watchdog/systemd/squid-acquisition-watchdog.service ~/.config/systemd/user/
+  # The shipped unit's WorkingDirectory is a placeholder (%h/Squid/software); point it here:
+  sed -i "s#^WorkingDirectory=.*#WorkingDirectory=$PWD#" ~/.config/systemd/user/squid-acquisition-watchdog.service
+  systemctl --user daemon-reload
+  systemctl --user enable --now squid-acquisition-watchdog   # auto-start at login + start now
+  systemctl --user status squid-acquisition-watchdog         # verify it's active
+  ```
+  Enable it once and it comes up at every login and restarts on failure (`Restart=always`) —
+  no need to launch it by hand. Logs: `journalctl --user -u squid-acquisition-watchdog -f`.
+  To keep it running before/without a graphical login, also run `loginctl enable-linger $USER`
+  once. (The unit runs `/usr/bin/python3`; if Squid runs on a different interpreter/venv, edit
+  the `ExecStart=` line to that python.)
+- **Windows (Task Scheduler):** run `windows/install.ps1` in PowerShell from `software\`. It
+  registers a logon-triggered task (via `pythonw.exe`; make sure it's on `PATH`, or edit the
+  path in the script).
+
+## Verify it works
+
+```bash
+python3 -m acquisition_watchdog --once   # one check, then exits — no error means it's healthy
+```
+
+End-to-end: start a `--simulation` acquisition, `kill -9` the GUI process, and you should get
+a crash alert — within one poll (~5 s) when `psutil` is installed (it is, by default),
+otherwise within `--heartbeat-timeout` seconds.
+
+## What triggers an alert
+
+| Situation | Alert |
+|---|---|
+| Process died — `running` breadcrumb + PID gone (crash / OOM-kill / power loss) | 🔴 crash |
+| Process alive but no heartbeat past the timeout | 🟠 hang |
+| Fatal error / auto-abort (timeout, failed save job, camera/frame failure) | 🔴 error |
+| Finished, but some save/job errors occurred | 🟠 completed-with-errors |
+| Aborted by the user, the MCP server, or by closing the GUI mid-run | 🟡 aborted |
+| Finished cleanly | *(silent)* |
+
+Each alert includes the machine name, experiment, reason, and progress (e.g. "stopped at
+timepoint 3/10").
+
+## Good to know
+
+- **Independent of the GUI.** Restarting the software neither starts nor stops the watchdog —
+  it just picks up the next run. That decoupling is the whole point: a watchdog spawned by the
+  GUI couldn't survive the GUI crashing.
+- **Start order doesn't matter.** Start it before, during, or after the GUI. If it starts
+  mid-run it monitors from there; if it starts *after* a crash already happened, it reads the
+  stale `running` breadcrumb, sees the PID is dead, and alerts once.
+- **One alert per run.** Alerted run IDs persist in `<state_dir>/alerted.json`, so it never
+  double-alerts and never re-alerts after a restart.
+- **Turn alerts off** on a machine (without disabling the GUI notifier): uncheck *"Enable
+  watchdog alerts"*, or set `watchdog_enabled: false` in `cache/slack_settings.yaml`. Takes
+  effect on the next check — no watchdog restart needed.
+- **Runs on the same machine as the GUI** (it reads local breadcrumb files). For coverage of a
+  full machine death / power loss, run it on another host pointed at a shared/synced state dir
+  (see *Remote / power-loss coverage*).
+
+## Troubleshooting
+
+| Symptom | Check |
+|---|---|
+| No alerts at all | Is the watchdog process actually running? Are `bot_token`/`channel_id` set (GUI → *Test Connection*)? Is *"Enable watchdog alerts"* checked? |
+| Log says Slack not configured | Run from `software/` so `cache/slack_settings.yaml` resolves, or pass `--slack-settings <path>`. |
+| False "hang" alerts | Raise `--heartbeat-timeout` — a single very long exposure/fluidics step can exceed the default. |
+| Crash reported slowly (~2 min) | `psutil` missing → falls back to the heartbeat timeout. Install `psutil` for instant PID-based detection. |
+
+## State dir
+
+Defaults to `platformdirs.user_state_path("squid", "cephla")/watchdog`. The GUI (writer) and
+the watchdog (reader) must agree on it — run both as the same user, or set
+`SQUID_WATCHDOG_STATE_DIR` on both (or `--state-dir` on the watchdog).
+
+## Remote / power-loss coverage (future)
+
+Point `--state-dir` at a shared/synced mount on another host and run this process there.
+Per-machine `run-<machine>.json` naming and a clock-skew tolerance are needed first (see the
+design spec).

software/acquisition_watchdog/__main__.py

@@ -0,0 +1,50 @@
+# acquisition_watchdog/__main__.py
+"""CLI entry point: python -m acquisition_watchdog"""
+import argparse
+import time
+from pathlib import Path
+from typing import Optional, Sequence
+
+import squid.logging
+from acquisition_watchdog.monitor import Monitor
+
+
+def main(argv: Optional[Sequence[str]] = None) -> None:
+    parser = argparse.ArgumentParser(
+        prog="acquisition_watchdog",
+        description="Alert on prematurely-ended Squid acquisitions (crash/hang/abort/error).",
+    )
+    parser.add_argument(
+        "--slack-settings",
+        help="Path to the Slack settings YAML (defaults to ./cache/slack_settings.yaml, "
+        "the same file the GUI writes).",
+    )
+    parser.add_argument("--state-dir", help="Override the watchdog state directory.")
+    parser.add_argument("--poll-interval", type=float, default=5.0, help="Seconds between checks (default 5).")
+    parser.add_argument(
+        "--heartbeat-timeout",
+        type=float,
+        default=120.0,
+        help="Seconds of heartbeat silence (with a live PID) before declaring a hang (default 120).",
+    )
+    parser.add_argument("--once", action="store_true", help="Run a single check and exit.")
+    args = parser.parse_args(argv)
+
+    log = squid.logging.get_logger("acquisition_watchdog")
+    monitor = Monitor(
+        state_dir=Path(args.state_dir) if args.state_dir else None,
+        slack_settings=args.slack_settings,
+        poll_interval=args.poll_interval,
+        heartbeat_timeout=args.heartbeat_timeout,
+    )
+    if args.once:
+        monitor.check_once(time.time())
+    else:
+        try:
+            monitor.run_forever()
+        except KeyboardInterrupt:
+            log.info("Acquisition watchdog stopped.")
+
+
+if __name__ == "__main__":
+    main()

software/acquisition_watchdog/alerts.py

@@ -0,0 +1,49 @@
+# acquisition_watchdog/alerts.py
+"""Format watchdog Slack alerts (text + Block Kit blocks)."""
+from datetime import datetime, timezone
+from typing import Optional, Tuple
+
+_KIND_TITLE = {
+    "crash": ":red_circle: Acquisition process died",
+    "hang": ":large_orange_circle: Acquisition hung (no heartbeat)",
+    "error": ":red_circle: Acquisition ended with a fatal error",
+    "completed_with_errors": ":large_orange_circle: Acquisition finished with errors",
+    "user_abort": ":large_yellow_circle: Acquisition aborted",
+}
+
+
+def _fmt_ts(epoch: Optional[float]) -> str:
+    if not epoch:
+        return "unknown"
+    return datetime.fromtimestamp(epoch, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
+
+
+def _progress_line(run: dict) -> str:
+    prog = run.get("progress") or {}
+    expected = run.get("expected") or {}
+    tp = prog.get("timepoint", "?")
+    exp_tp = prog.get("expected_timepoints", expected.get("timepoints", "?"))
+    images = prog.get("images", "?")
+    return f"timepoint {tp}/{exp_tp}, {images} images"
+
+
+def format_alert(kind: str, run: dict) -> Tuple[str, list]:
+    title = _KIND_TITLE.get(kind, f"Acquisition alert: {kind}")
+    experiment = run.get("experiment_id", "unknown")
+    machine = run.get("machine", "unknown")
+    text = f"{title}: {experiment} on {machine}"
+
+    last_seen = run.get("ended_at") or run.get("heartbeat_at")
+    detail = (
+        f"*Experiment:* {experiment}\n"
+        f"*Machine:* {machine}\n"
+        f"*Progress:* {_progress_line(run)}\n"
+        f"*Started:* {_fmt_ts(run.get('started_at'))}\n"
+        f"*Last seen:* {_fmt_ts(last_seen)}\n"
+        f"*Output:* {run.get('output_path', 'unknown')}"
+    )
+    blocks = [
+        {"type": "section", "text": {"type": "mrkdwn", "text": f"*{title}*"}},
+        {"type": "section", "text": {"type": "mrkdwn", "text": detail}},
+    ]
+    return text, blocks

software/acquisition_watchdog/config.py

@@ -0,0 +1,51 @@
+# acquisition_watchdog/config.py
+"""Load Slack credentials from the same source the Squid GUI uses.
+
+The GUI stores Slack settings (bot token, channel, enabled) in
+`cache/slack_settings.yaml` (written by the Slack settings dialog and loaded at
+GUI startup via control.widgets_slack.load_slack_settings_from_cache). This module
+reads that same YAML so the watchdog alerts to the same workspace — without
+importing the heavy control stack.
+"""
+import os
+from pathlib import Path
+from typing import NamedTuple, Optional
+
+import yaml
+
+
+class SlackConfig(NamedTuple):
+    bot_token: Optional[str]
+    channel_id: Optional[str]
+    watchdog_enabled: bool
+
+
+DEFAULT_SLACK_SETTINGS = "cache/slack_settings.yaml"
+
+
+def resolve_slack_settings_path(cli_path: Optional[str]) -> Path:
+    """Priority: --slack-settings > $SQUID_SLACK_SETTINGS > cache/slack_settings.yaml (cwd-relative)."""
+    if cli_path:
+        return Path(cli_path)
+    env = os.environ.get("SQUID_SLACK_SETTINGS")
+    if env:
+        return Path(env)
+    return Path(DEFAULT_SLACK_SETTINGS)
+
+
+def load_slack_config(path: Optional[Path]) -> SlackConfig:
+    p = Path(path) if path else Path(DEFAULT_SLACK_SETTINGS)
+    if not p.exists():
+        return SlackConfig(None, None, True)
+    try:
+        with open(p) as f:
+            data = yaml.safe_load(f) or {}
+    except Exception:
+        return SlackConfig(None, None, True)
+    if not isinstance(data, dict):
+        return SlackConfig(None, None, True)
+    return SlackConfig(
+        bot_token=(data.get("bot_token") or None),
+        channel_id=(data.get("channel_id") or None),
+        watchdog_enabled=bool(data.get("watchdog_enabled", True)),
+    )