feat(#411): canary invariant harness Phase 1 (S-01, E-02, L-03)

.env.example

@@ -260,3 +260,17 @@ OTEL_EXPORTER_OTLP_PROTOCOL=grpc
 
 # Metrics export interval in milliseconds (default: 60 seconds)
 OTEL_METRIC_EXPORT_INTERVAL=60000
+
+# ===========================================
+# CANARY INVARIANT HARNESS (Optional, staging/dev)
+# ===========================================
+
+# Continuous orchestration-invariant watcher (CANARY-001 / Issue #411).
+# Set to 1 on staging/dev to run the 5-min check loop. Production stays 0.
+CANARY_ENABLED=0
+
+# Slack incoming webhook URL for canary green→red transitions.
+# Get from: https://api.slack.com/apps → your app → Incoming Webhooks → Add
+# The URL is the credential — anyone with it can post to that one channel.
+# Unset = canary cycles run silently (violations still persisted to DB).
+CANARY_SLACK_WEBHOOK_URL=

config/canary-fleet.yaml

@@ -0,0 +1,63 @@
+# Canary Invariant Harness — load-generator fleet (CANARY-001 / Issue #411).
+#
+# This manifest defines the synthetic agents that the canary watcher
+# observes. Without traffic, every invariant holds trivially and the
+# harness produces no signal. The fleet is what makes the harness useful.
+#
+# Deploy on staging/dev with:
+#
+#     curl -sS -X POST -H "Authorization: Bearer <admin-token>" \
+#       -H "Content-Type: application/json" \
+#       -d "{\"manifest\": $(jq -Rs . < config/canary-fleet.yaml)}" \
+#       http://localhost:8000/api/systems/deploy
+#
+# The watcher service (src/backend/services/canary_service.py) runs every
+# 5 min independently of this fleet — it'll just observe trivially-green
+# state until the fleet generates load.
+#
+# Permissions are explicitly empty (preset: none) — fleet members observe
+# their own orchestration, not each other.
+
+name: canary-fleet
+description: Canary harness load generators (Issue #411 Phase 1)
+
+agents:
+  # Constant slot churn — fires every minute (cron's minimum). Exercises
+  # S-01 (slot–row bijection) and E-02 (no phantom reversal) by producing
+  # a steady stream of acquire / release cycles.
+  #
+  # `test-echo` is Trinity's minimal stock template (no MCP servers, no
+  # credentials, deterministic short reply) — exactly what slot-churn
+  # invariants need.
+  burst:
+    template: local:test-echo
+    resources:
+      cpu: "1"
+      memory: "512m"
+    schedules:
+      - name: heartbeat
+        cron: "* * * * *"  # every minute
+        message: "ok"
+        timezone: UTC
+        description: Constant load to exercise slot–row bijection (S-01) and reversal (E-02)
+
+  # Slower cadence so multiple slots overlap across cycles. Phase 2
+  # invariants (S-03 / E-01 / E-06) want a longer-lived slot to scrutinize;
+  # Phase 1 only needs the slower cron to broaden snapshot coverage, so
+  # reusing `test-echo` is fine here too.
+  long:
+    template: local:test-echo
+    resources:
+      cpu: "1"
+      memory: "1g"
+    schedules:
+      - name: long-task
+        cron: "*/5 * * * *"  # every 5 min
+        message: "long-task ping"
+        timezone: UTC
+        description: Slower cadence to overlap slots across canary cycles (Phase 2 will swap in a real long task)
+
+# No cross-agent permissions — the canary fleet observes the orchestration
+# layer, not each other.
+permissions:
+  preset: none

docker-compose.yml

@@ -51,6 +51,14 @@ services:
       - SLACK_SIGNING_SECRET=${SLACK_SIGNING_SECRET:-}
       # SSH access host override
       - SSH_HOST=${SSH_HOST:-}
+      # Canary invariant harness (CANARY-001 / Issue #411).
+      # When 1, services/canary_service.py runs the 5-min watcher loop on
+      # staging/dev. Default 0 — production users see no canary activity.
+      - CANARY_ENABLED=${CANARY_ENABLED:-0}
+      # Slack alert sink for canary green→red transitions. URL is the
+      # credential — leaking it lets anyone post to that one channel.
+      # Unset = canary cycles run silently (still persists violations).
+      - CANARY_SLACK_WEBHOOK_URL=${CANARY_SLACK_WEBHOOK_URL:-}
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - ./src/backend:/app

docs/memory/architecture.md

@@ -405,6 +405,7 @@ Services that run continuously in the backend process:
 | **Capacity Maintenance** | `capacity_manager.py` | Calls `CapacityManager.run_maintenance()` every 60s — expires stale queued tasks (>24h) and drains orphans after restart. (BACKLOG-001 / CAPACITY-CONSOLIDATE #428) |
 | **Audit Retention** | `audit_retention_service.py` | Daily APScheduler job at 04:15 UTC that DELETEs `audit_log` rows past the retention window. Configured via `AUDIT_LOG_RETENTION_DAYS` (default 365, floored at 365 — the `audit_log_no_delete` trigger refuses younger rows). Pruning ages out hash-chain history past the cutoff by design. (#552) |
 | **Session Cleanup** | `session_cleanup_service.py` | Periodic JSONL reaper for the Session tab. Default 6h cycle (`poll_interval_seconds`); each cycle diffs every running agent's `~/.claude/projects/-home-developer/<uuid>.jsonl` set against `agent_sessions.cached_claude_session_id` and deletes JSONLs not in the keep set whose mtime is older than `min_age_seconds` (default 1h race guard). Synchronous best-effort `reap_jsonl()` is also called by the session router on user-initiated reset/delete so the disk reclaim is immediate. Uses `execute_command_in_container` (no agent-server endpoint required). (SESSION_TAB Phase 4.2) |
+| **Canary Watcher** | `canary_service.py` | Continuous orchestration-invariant harness (CANARY-001 / Issue #411). Every 5 min: `collect_snapshot()` over Redis × SQLite × agent registries, runs deterministic invariant library (S-01, E-02, L-03 in Phase 1), persists violations to `canary_violations`, classifies green→red transitions and fires one Slack webhook POST per transition (`CANARY_SLACK_WEBHOOK_URL` env var; unset = silent sink). Disabled by default; enable on staging/dev with `CANARY_ENABLED=1`. |
 
 The **agent server** also runs a 15-min `auto_sync` heartbeat loop (gated
 by `GIT_SYNC_AUTO` env var; default-on for non-source-mode GitHub-template
@@ -684,6 +685,55 @@ middleware. Phase 3: MCP tool call audit via transparent wrapper (all 66+
 tools, zero per-tool code). Phase 4: hash chain verification, CSV/JSON
 export, enable/disable toggle. Issue #20 can be closed.
 
+### Canary Invariant Harness (CANARY-001 — Phase 1, NEW: 2026-05-04)
+
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| GET | `/api/canary/violations` | Admin | List violations (filters: invariant_id, severity, tier, start_time, end_time, limit, offset) |
+| GET | `/api/canary/violations/stats` | Admin | Aggregate counts by invariant_id and severity |
+| GET | `/api/canary/violations/{id}` | Admin | Single violation by row id |
+| POST | `/api/canary/run-cycle` | Admin | Run one cycle on demand (delegates to the same `CanaryService.run_cycle()` invoked by the 5-min background loop). Optional body filters which invariants to run. Returns `{snapshot_time, cycle_duration_ms, checks_run, sources_unavailable, violations[], transitions[]}`. Returns 409 with `detail="cycle in progress"` when a background or sibling on-demand cycle is mid-run — empty payload is never silently returned. |
+
+**Storage**: `canary_violations` table in main SQLite DB. JSON-encoded
+`observed_state` column carries invariant-specific payload.
+
+**Phase 1 invariants** (S-01, E-02, L-03):
+- **S-01 — Slot–row bijection**: per agent, set of execution_ids in
+  `agent:slots:{name}` (Redis ZSET, drain sentinels filtered) equals set
+  of execution_ids in `schedule_executions WHERE status='running'`.
+  Severity: critical. Catches PR #378/#403 bug class.
+- **E-02 — No phantom reversal**: an execution row that was in a
+  terminal status in the previous cycle must not appear non-terminal in
+  this snapshot. Phase 1 uses Redis-backed state comparison (key
+  `canary:e02:terminal_seen`) instead of Vector log diff for simplicity.
+  Severity: critical.
+- **L-03 — Delete cascades**: no live row in any cross-cutting table
+  (agent_sharing, agent_schedules, schedule_executions [non-terminal],
+  agent_skills, agent_tags, agent_shared_files, agent_public_links,
+  pending operator_queue, pending access_requests, agent-scoped
+  mcp_api_keys, active chat_sessions) may reference an `agent_name` not
+  in `agent_ownership`; no Redis `agent:slots:{name}` for missing agent.
+  Severity: critical for orphaned `schedule_executions` or Redis slots,
+  major otherwise. Catches Issue #129 bug class.
+
+**Fleet**: `config/canary-fleet.yaml` — synthetic load generators
+(`canary-fleet-burst`, `canary-fleet-long`) deployed via the existing
+systems-deploy API. Without traffic the harness produces trivially-green
+checks; the fleet is what gives the watcher something to watch on
+staging/dev.
+
+**Architecture**: deterministic library (`src/backend/canary/`) shared
+between the 5-min watcher service and the on-demand admin endpoint.
+Library reads state but writes nothing; service writes violations and
+classifies green→red transitions. **Alert sink**: Slack via incoming
+webhook URL configured by `CANARY_SLACK_WEBHOOK_URL` env var (admin-side,
+no Settings UI — the canary is staging/dev-only and the operator already
+has shell access). Unset = silent sink (cycles still run, violations
+still persist). Each transition fires exactly one webhook POST with a
+Block Kit payload (header + body + context with "last red Xm ago"
+badge). Continuing-red invariants don't re-post. No LLM reasoning
+anywhere — the canary's value depends on determinism.
+
 ### Nevermined Payments (NVM-001)
 
 | Method | Path | Auth | Description |
@@ -1380,6 +1430,36 @@ BEGIN SELECT RAISE(ABORT, 'Audit log entries cannot be deleted within retention
 - Cross-cutting platform audit for lifecycle, auth, MCP, credentials events
 - Phase 1 ships infrastructure only; write integration into routers happens in Phase 2
 
+**canary_violations:** (CANARY-001 / Issue #411 — Phase 1, NEW: 2026-05-04)
+```sql
+CREATE TABLE canary_violations (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    invariant_id TEXT NOT NULL,            -- 'S-01', 'E-02', 'L-03', ...
+    tier TEXT NOT NULL,                    -- 'A' | 'B'
+    severity TEXT NOT NULL,                -- 'critical' | 'major' | 'minor'
+    snapshot_time TEXT NOT NULL,           -- ISO 8601 UTC
+    observed_state TEXT NOT NULL,          -- JSON, invariant-specific
+    signal_query TEXT,                     -- the check that fired (debugging aid)
+    created_at TEXT NOT NULL DEFAULT (datetime('now'))
+);
+CREATE INDEX idx_canary_violations_invariant
+    ON canary_violations(invariant_id, snapshot_time DESC);
+CREATE INDEX idx_canary_violations_severity
+    ON canary_violations(severity, snapshot_time DESC);
+CREATE INDEX idx_canary_violations_snapshot
+    ON canary_violations(snapshot_time DESC);
+```
+
+**canary_violations Features:**
+- Append-only in practice (no UPDATE / DELETE in the read API surface).
+- One row per fired check per cycle. `observed_state` carries
+  invariant-specific JSON (slot diffs, ghost agent names, terminal-status
+  reversals).
+- Read via `GET /api/canary/violations`; `GET /api/canary/violations/stats`
+  drives the dashboard tiles.
+- Populated by `services/canary_service.py` on a 5-min loop or on-demand
+  via `POST /api/canary/run-cycle`.
+
 ### Redis
 
 **Credential Storage (DEPRECATED - CRED-002):**

docs/memory/requirements.md

@@ -2005,6 +2005,58 @@ Standalone mobile-friendly admin page for managing agents on the go. Designed as
 
 ---
 
+## 31. Canary Invariant Harness (CANARY-001)
+
+### 31.1 Continuous Orchestration-Invariant Watcher (CANARY-001 — Phase 1)
+- **Implements**: Issue #411 — first three invariants (S-01, E-02, L-03)
+- **Description**: Background watcher service that runs deterministic
+  orchestration-invariant checks against live platform state every 5
+  minutes. Persists violations to a queryable table and classifies
+  green→red transitions for an external alert sink. Catches the bug
+  class behind PRs #378, #403, #129, #226 — race conditions and
+  cross-component state drift that unit tests miss.
+- **Architecture**: deterministic Python library (`src/backend/canary/`)
+  shared between the watcher service (`services/canary_service.py`) and
+  the on-demand admin endpoint (`POST /api/canary/run-cycle`). Library
+  reads state but writes nothing; service writes violations and
+  classifies transitions.
+- **Phase 1 invariants**:
+  - **S-01** Slot–row bijection (Redis ZRANGE vs SQL running rows, drain
+    sentinels filtered)
+  - **E-02** No phantom reversal (terminal executions stay terminal,
+    detected via Redis-backed state comparison)
+  - **L-03** Delete cascades (no orphan rows referencing removed agents
+    in any cross-cutting table; no orphan Redis slot keys)
+- **Storage**: `canary_violations` table; observed_state JSON column.
+- **Activation**: gated by `CANARY_ENABLED=1` env var; disabled by
+  default. Production deployment is staging/dev — the harness watches
+  there, not in user-facing prod.
+- **Fleet**: `config/canary-fleet.yaml` deploys two synthetic agents
+  (`canary-fleet-burst` minute-cron, `canary-fleet-long` 5-min cron) via
+  the existing `/api/systems/deploy` endpoint. Without the fleet, the
+  watcher reports trivially-green cycles with no signal.
+- **Alert sink**: Slack via incoming webhook URL configured by the
+  `CANARY_SLACK_WEBHOOK_URL` env var (admin-side, no Settings UI — the
+  audience is operators with shell access on staging/dev). Each
+  green→red transition fires exactly one webhook POST with a Block Kit
+  payload (severity emoji header, rendered violation summary, context
+  line with snapshot_time + violation count + "last red Xm ago"
+  badge). Unset = silent sink: cycles still run, violations still
+  persist to `canary_violations`, only the outbound POST is skipped.
+  Continuing-red invariants don't re-post. The dashboard-notifications
+  path (writing `agent_notifications` rows via `db.create_notification`)
+  was rejected on the product call.
+- **Determinism**: invariant checks are pure functions
+  `check(snapshot) → list[ViolationReport]`. Same snapshot input always
+  yields the same output. No LLM reasoning anywhere in the canary path.
+- **Phase 2 (deferred)**: S-02, S-03, E-01, E-05, E-06, B-01, B-02,
+  G-01, R-01 (per the catalog at
+  `docs/testing/orchestration-invariant-catalog.md`). Each adds as a new
+  file under `src/backend/canary/invariants/` and a registry entry; the
+  service and API surface stay unchanged.
+
+---
+
 ## Out of Scope
 
 - Multi-tenant deployment (single org only)

docs/security-reports/cso-2026-05-09-411-diff.json

@@ -0,0 +1,42 @@
+{
+  "version": "1.0",
+  "date": "2026-05-09",
+  "mode": "daily",
+  "scope": "diff",
+  "branch": "feature/411-canary-migration",
+  "base": "dev",
+  "phases_run": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
+  "attack_surface": {
+    "public_endpoints": 0,
+    "authenticated_endpoints": 0,
+    "admin_endpoints": 4,
+    "file_upload_points": 0,
+    "websocket_channels": 0,
+    "external_integrations": 1,
+    "background_jobs": 1,
+    "new_env_vars": 2
+  },
+  "findings": [],
+  "supply_chain_summary": {
+    "new_python_deps": 0,
+    "new_node_deps": 0,
+    "new_lockfile_changes": 0
+  },
+  "filter_stats": {
+    "candidates_considered": 5,
+    "dropped_below_gate": 5,
+    "hard_exclusions_applied": ["#1 DoS/resource", "#3 memory leak"]
+  },
+  "totals": {
+    "critical": 0,
+    "high": 0,
+    "medium": 0,
+    "low": 0
+  },
+  "trend": {
+    "prior_report": "cso-2026-05-04-589-diff.md",
+    "carryover": 0,
+    "note": "Prior diff report covered a different change set (#589 network split). No findings carry over."
+  },
+  "summary": "Phase 1 of #411 (canary invariant harness). Admin-only API surface, outbound-only Slack integration with explicit URL-leak mitigation, SQL composition via constants and enum-derived values, env-gated activation. Zero findings at the 8/10 daily-mode gate."
+}

docs/security-reports/cso-2026-05-09-411-diff.md

@@ -0,0 +1,66 @@
+# CSO Diff Audit — feature/411-canary-migration → dev
+
+**Date**: 2026-05-09
+**Mode**: daily (8/10 confidence gate)
+**Scope**: branch diff (19 source files, ~4144 added lines, excluding docs/ and tests/)
+
+## Summary
+
+Zero findings at the daily-mode confidence gate.
+
+| Severity | Count |
+|----------|-------|
+| Critical | 0 |
+| High | 0 |
+| Medium | 0 |
+| Low | 0 |
+
+## Change Set
+
+Phase 1 of #411 — continuous orchestration-invariant harness:
+
+- New module `src/backend/canary/` (snapshot collector + 3 invariants: S-01, E-02, L-03)
+- New service `services/canary_service.py` (5-min watcher, env-gated `CANARY_ENABLED=1`)
+- New service `services/canary_alerts.py` (Slack Block Kit composer, outbound webhook POST)
+- New router `routers/canary.py` (4 admin endpoints)
+- New table `canary_violations` (append-mostly, JSON `observed_state`)
+- New env vars: `CANARY_ENABLED`, `CANARY_SLACK_WEBHOOK_URL`
+
+## Attack Surface (diff scope)
+
+| Surface | Count |
+|---------|-------|
+| Public endpoints | 0 |
+| Authenticated endpoints | 0 |
+| Admin-only endpoints | 4 |
+| File upload points | 0 |
+| WebSocket channels | 0 |
+| External integrations | 1 (Slack outbound) |
+| Background jobs | 1 (env-gated) |
+
+## Why no findings
+
+1. **Admin-only API surface**: every route uses `Depends(require_admin)`. No public, agent, or user-facing exposure.
+2. **Outbound-only Slack integration**: signature verification N/A. URL-leak surface explicitly addressed at `slack_service.py:310-315` — exception class name only, never `str(e)`.
+3. **No SQL injection**: f-string SQL composition draws from `ORPHAN_SCAN_TABLES` constants, the `TaskExecutionStatus` enum, and `pk_col` from `PRAGMA table_info` results. No user input crosses into f-string interpolation.
+4. **Env-gated activation**: `CANARY_ENABLED=0` default; silent-sink fallback when `CANARY_SLACK_WEBHOOK_URL` unset.
+5. **Read-only data flow**: snapshot collector reads Redis and SQLite, writes only to the new dedicated `canary_violations` table.
+6. **No new agent-network exposure**: env vars passed only to backend container; no new ports/volumes/capabilities.
+
+## Candidates Filtered Below the Gate
+
+| Candidate | Drop reason |
+|-----------|-------------|
+| Multi-worker duplicate Slack alerts | Theoretical — Trinity ships single backend worker. Documented in `/review I4` as a forward-looking concern, not currently exploitable. |
+| `canary_violations` unbounded growth | Hard exclusion #3 (resource without proven security impact). |
+| Slack mrkdwn injection via agent_name | Agent names sanitized to `[a-zA-Z0-9_.-]`; admin-only channel; webhooks don't `link_names` by default. |
+| httpx may log webhook URL at DEBUG level | Only at `LOG_LEVEL=DEBUG`; prod runs INFO. Confidence 4/10. |
+| `print()` instead of structured logger in `main.py` startup paths | Logging hygiene, not security. |
+
+## Repudiation Note (non-blocking)
+
+`POST /api/canary/run-cycle` is not currently written to `audit_log`. Admin-gated, so blast radius is bounded, but on-demand cycles aren't traceable to the operator who triggered them. The existing platform audit log doesn't cover every admin endpoint either; flag this for Phase 2 if traceability becomes a stakeholder requirement.
+
+## Trend
+
+Prior report `cso-2026-05-04-589-diff.md` covered #589 (Redis/network split) — different change set, no overlap. No carryover findings.