Research and implement orchestration invariant catalog + continuous canary harness

[vybe]

Summary

Design and build a continuous invariant-based testing harness for Trinity's orchestration layer, running against staging/dev. Goal: prove orchestration invariants hold 24/7 under real traffic and adversarial conditions — catching bugs like #378/#403 (phantom stale-slot failures), #407 (subprocess reaping), and #129 (orphaned executions) before users see them.

Full design: docs/testing/orchestration-invariant-catalog.md

Motivation

Recent orchestration bugs all violated obvious-in-hindsight invariants:

• bug: Execution shows Failed then Success after cleanup (race condition) #378/fix(cleanup): prevent phantom stale-slot failures via JIT re-verify (#378) #403 — "a terminal execution state must be immutable"
• bug: agent-server.py spins at ~83% CPU after claude CLI subprocess becomes defunct #407 — "every subprocess must be reapable on every exit path"
• Active watchdog: remediate stuck executions detected by monitoring #129 — "every running execution maps to either an agent registry entry or a cleanup action within one cycle"
• bug: Cleanup service does not fail execution records when reclaiming stale slots #219/bug: Stale slot cleanup uses fixed 20-min TTL regardless of agent timeout #226 — "slot TTL ≥ execution timeout"

Unit tests don't catch these because they live in cross-component state — Redis × SQLite × Docker × agent process registry. A continuous harness that checks invariants on a live staging instance would have caught all of them.

Proposed approach

See the full catalog in docs/testing/orchestration-invariant-catalog.md. Summary:

Framework layers (bottom-up):

1. Invariant library (~55 invariants across 14 subsystems, each a pure (snapshot) → ViolationReport)
2. Snapshot collector (SQL + Redis + Docker + agent registry, roughly simultaneous)
3. Scenario runner (pytest-style, exercises flows + asserts invariants)
4. Continuous canary (reads-only subset every 5–15 min, alerts on violation)
5. Chaos layer (opt-in, container kill / Redis partition / socket lag)

Tiering: Tier A (must hold always) vs. Tier B (must reconcile within SLA T). Severity: 🔴 critical / 🟡 major / 🟢 minor.

Starting subset (Phase 1 — 10 invariants, ~80% coverage)

ID	Invariant	Reason
S-01	Slot–row bijection	Core orchestration
S-02	No overbooking	Capacity
E-01	Terminal-state closure	No stuck executions
E-02	No phantom reversal	#378/#403
E-06	No completed-but-not-reported	#129
B-01	Queue-status coherence	Backlog integrity
B-02	No queued without slots-full	Drain liveness
L-03	Delete cascades	Dangling references
G-01	No resource leak on restart	Recovery
R-01	No zombie Claude processes	#407

Rollout phases

• Phase 1 — invariant library + snapshot collector + read-only canary for 10-invariant subset. Deploy to staging. Goal: catch one real bug within 30 days.
• Phase 2 — scenario runner exercising all Tier-A invariants through create/delete/trigger flows.
• Phase 3 — Hypothesis stateful testing for scenario generation.
• Phase 4 — chaos injection (Pumba / toxiproxy).

Research questions

Listed in full doc, §Open Questions. Highest-priority:

1. Snapshot source-of-truth — backend API (real code path) vs. direct SQLite read (fidelity). Recommendation: API for Phase 1, direct for Phase 3+.
2. Simultaneity window for snapshots across Redis/SQLite/Docker.
3. Reuse vs. greenfield vs. existing docs/testing/ frameworks (MODULAR_TESTING_STRUCTURE.md, UI_INTEGRATION_TEST.md).
4. Staging topology — is the canary a Trinity agent on the same instance, or a separate process? (Recommend: Trinity agent — dogfoods the platform.)

Acceptance criteria

• docs/testing/orchestration-invariant-catalog.md reviewed and any corrections merged
• Phase 1 design doc (subset of 10 invariants, snapshot format, alert channel) reviewed
• canary_violations DB table + snapshot-collector service implemented
• First 3 invariants (S-01, E-02, L-03) running on staging continuously
• One real violation caught and alerted (or 30 days clean — inform design of Phase 2)

Non-goals (for this issue)

• Replacing unit tests
• Chaos engineering (Phase 4)
• Production deployment of harness

References

• Full catalog: docs/testing/orchestration-invariant-catalog.md
• Architectural invariants: docs/memory/architecture.md §Architectural Invariants
• Recent fixes: PRs bug: Execution shows Failed then Success after cleanup (race condition) #378, fix(cleanup): prevent phantom stale-slot failures via JIT re-verify (#378) #403, bug: agent-server.py spins at ~83% CPU after claude CLI subprocess becomes defunct #407, fix(agent-runtime): reap claude subprocess tree via process groups (#407) #410; Issues Active watchdog: remediate stuck executions detected by monitoring #129, bug: Cleanup service does not fail execution records when reclaiming stale slots #219, bug: Stale slot cleanup uses fixed 20-min TTL regardless of agent timeout #226, bug: Cleanup service misses 'skipped' executions and slow to detect no-session failures #106