feat(scheduler): agent-owned pre-check hook (#454)

docs/memory/architecture.md

@@ -363,6 +363,8 @@ docker exec trinity-vector sh -c "tail -50 /data/logs/agents.json" | jq .
 - `/api/files` - List workspace files (recursive tree structure)
 - `/api/files/download` - Download file content (100MB limit)
 
+**Template-supplied pre-check** (optional, SCHED-COND-001): if the template ships an executable `~/.trinity/pre-check` file, the backend's internal endpoint `POST /api/internal/agents/{name}/pre-check` runs it via `docker exec` before the scheduler fires a cron-triggered chat. The hook is **language-agnostic** — interpreter is selected by the file's shebang line (Python, bash, node, compiled binary, …); Trinity does not invoke `python3` for it. The hook's stdout becomes the chat message; empty stdout + exit 0 records a skipped execution. No HTTP endpoint is exposed on the agent-server for this — the primitive is the same `execute_command_in_container` already used by `services/git_service.py` (persistent-state allowlist), `ssh_service.py`, and the agent terminal.
+
 **Persistent Chat:**
 - All chat messages automatically saved to SQLite (`chat_sessions`, `chat_messages`)
 - Sessions survive container restarts/deletions
@@ -395,7 +397,7 @@ Services that run continuously in the backend process:
 | **Operator Queue Sync** | `operator_queue_service.py` | Polls running agents every 5s, reads `~/.trinity/operator-queue.json`, syncs to DB, writes responses back. (OPS-001) |
 | **Sync Health Service** | `sync_health_service.py` | Polls git-enabled agents every 60s, upserts `agent_sync_state`, emits `sync_failing` operator-queue entries when consecutive_failures ≥ 3. (#389 S1) |
 | **Monitoring Service** | `monitoring_service.py` | Fleet-wide health checks on configurable interval. (MON-001) |
-| **Scheduler Service** | `scheduler_service.py` | APScheduler-based cron job execution. Async fire-and-forget with DB polling for status. |
+| **Scheduler Service** | `scheduler_service.py` | APScheduler-based cron job execution. Async fire-and-forget with DB polling for status. On each cron-triggered fire, optionally invokes the agent's executable `~/.trinity/pre-check` (interpreter chosen by shebang) via the backend's `POST /api/internal/agents/{name}/pre-check` (which `docker exec`s into the agent container). Empty stdout + exit 0 records a skipped execution and does not invoke Claude (SCHED-COND-001, #454). |
 | **Capacity Maintenance** | `capacity_manager.py` | Calls `CapacityManager.run_maintenance()` every 60s — expires stale queued tasks (>24h) and drains orphans after restart. (BACKLOG-001 / CAPACITY-CONSOLIDATE #428) |
 
 The **agent server** also runs a 15-min `auto_sync` heartbeat loop (gated

docs/memory/feature-flows.md

@@ -19,6 +19,7 @@
 | 2026-04-25 | #487 | Telegram file upload Phase 2 — workspace delivery hardening: NFKC sanitizer with collision dedup, spec injection format `[File uploaded by {uploader}]: {name} ({size}) saved to {path}`, all-writes-failed channel error + abort. Same code path benefits Slack inbound. | [telegram-integration.md](feature-flows/telegram-integration.md), [slack-file-sharing.md](feature-flows/slack-file-sharing.md) |
 | 2026-04-23 | #476 | SQLite lexicographic cutoff bug fix — new `iso_cutoff(hours)` helper replaces `datetime('now', ...)` in 15 sites across rate-limit / dashboard / schedules; `max_retries` default flipped `1 → 0`; `cleanup_old_rate_limit_events` wired into `CleanupService` (phase 6, hourly) | [subscription-auto-switch.md](feature-flows/subscription-auto-switch.md), [cleanup-service.md](feature-flows/cleanup-service.md), [scheduler-service.md](feature-flows/scheduler-service.md) |
 | 2026-04-22 | #458 | `.gitignore` init fix — `initialize_git_in_container` now appends missing patterns instead of truncate-and-write; adds `.env`, `.env.*`, `.mcp.json` to the default list and runs for both `/home/developer` and legacy `/home/developer/workspace` (stops credential leak on first GitHub sync) | [github-repo-initialization.md](feature-flows/github-repo-initialization.md) |
+| 2026-04-22 | SCHED-COND-001 (#454) | Conditional schedule pre-check — backend `docker exec`s the template's executable `~/.trinity/pre-check` (language-agnostic, interpreter from shebang) before scheduler fires a cron chat; empty stdout + exit 0 records a skipped execution; fail-open; reuses `ExecutionStatus.SKIPPED` (no schema change, no HTTP edge from scheduler to agent) | [scheduler-pre-check.md](feature-flows/scheduler-pre-check.md) |
 | 2026-04-21 | RELIABILITY-003 (#306) | WebSocket event bus on Redis Streams — replaces in-process broadcast with XADD/XREAD, adds reconnect replay via `?last-event-id=`, 3-failure eviction, MAXLEN trim (tunable) | [websocket-event-bus.md](feature-flows/websocket-event-bus.md) |
 | 2026-04-20 | #420 | Scheduler sync loop fix — `update_schedule_run_times` no longer bumps `updated_at`, stopping the self-triggering re-register of every schedule per tick | [scheduler-service.md](feature-flows/scheduler-service.md) |
 | 2026-04-20 | #418 | Inter-agent timeout honors per-agent `execution_timeout_seconds` — removed 600s hardcoded defaults in MCP `chat_with_agent`/`fan_out` tools and fan-out service; HTTP client ceiling bumped to platform max (7200s) | [fan-out.md](feature-flows/fan-out.md), [mcp-orchestration.md](feature-flows/mcp-orchestration.md), [parallel-headless-execution.md](feature-flows/parallel-headless-execution.md) |

docs/memory/feature-flows/scheduler-pre-check.md

@@ -0,0 +1,140 @@
+# Feature: Conditional Schedule Pre-Check (SCHED-COND-001 / Issue #454)
+
+## Overview
+Optional template-supplied hook that lets a scheduled cron tick be skipped deterministically without waking Claude. Before firing a cron-triggered chat, the scheduler calls a **backend** internal endpoint, which `docker exec`s the executable `~/.trinity/pre-check` file inside the target agent container. Non-empty stdout becomes the chat prompt; empty stdout + exit 0 records a skipped execution. Eliminates per-tick token burn on poll-driven agents (PR reviewers, inbox monitors, alert routers).
+
+The hook is **language-agnostic** — Trinity execs the file directly, so the interpreter is chosen by the file's shebang line. Templates ship Python, bash, node, Go binaries — Trinity does not care.
+
+## User Story
+As the author of a poll-driven agent template, I want a cheap deterministic gate to run before Claude wakes so empty polls (scan→no work) don't burn tokens. As a Trinity operator, I don't want to configure the gate per schedule — the agent template owns it, I just schedule the cadence.
+
+## Entry Points
+- **Template contract**: ship `~/.trinity/pre-check` as a `+x` executable file with a shebang (`#!/usr/bin/env python3`, `#!/bin/bash`, etc.). Prints chat prompt to stdout when work is found; exits 0 with empty stdout to skip; exits non-zero on error (fail-open).
+- **Backend endpoint** (internal, `X-Internal-Secret` auth): `POST /api/internal/agents/{name}/pre-check` — runs the script and returns stdout + exit code. Called only by `trinity-scheduler`.
+- **No operator-facing API change**: schedule CRUD endpoints and the Schedules UI are unchanged. Operators create normal cron schedules; agents own the gate.
+
+## Frontend Layer
+No UI change. Skipped executions appear in the existing schedule executions list alongside `success`/`failed` rows — the frontend already renders the `status` field as a badge.
+
+## Backend Layer
+**Router**: `src/backend/routers/internal.py` — `POST /api/internal/agents/{name}/pre-check`
+
+Uses `services/docker_service.execute_command_in_container` (the same primitive as `services/git_service.py` persistent-state allowlist, `services/ssh_service.py` key provisioning, `services/agent_service/terminal.py`, `routers/system_agent.py`, `adapters/message_router.py` Slack ingest, etc.).
+
+Two exec steps:
+1. `test -f /home/developer/.trinity/pre-check` (5s timeout). If the file doesn't exist, return `{"hook_present": False}` immediately — scheduler treats as "no decision, fire as usual." Note `-f`, not `-x`: a file present but missing the executable bit is treated as "hook present" so the operator gets a 126 in the logs rather than a silent backward-compat fall-through.
+2. `/home/developer/.trinity/pre-check` (60s timeout). Trinity execs the path directly — no `python3` prefix. Interpreter resolution is the file's shebang. Returns the output (capped at 32 KB) and exit code.
+
+Returns:
+```json
+{"hook_present": true, "exit_code": 0, "stdout": "Review PR #1\n", "stderr": ""}
+```
+
+## Scheduler Layer
+**Service**: `src/scheduler/service.py` — `_run_pre_check(agent_name)`
+
+Calls the backend's internal endpoint (not the agent directly — topology stays "scheduler → backend → agent"). Translates the backend response into a scheduler decision:
+
+| Backend response | Scheduler decision |
+|---|---|
+| `hook_present: false` | `None` → fire as usual (backward compat for templates without a hook) |
+| `exit_code != 0` | `None` → fail-open + log stderr (broken hook must not suppress work) |
+| `exit_code == 0`, empty stdout | `{"fire": False, "reason": "pre-check returned empty stdout"}` → record skipped execution |
+| `exit_code == 0`, non-empty stdout | `{"fire": True, "message": stdout.strip()}` → fire with stdout as override |
+| HTTP error / malformed JSON / timeout | `None` → fail-open + log |
+
+Intercept point in `_execute_schedule_with_lock`: called only when `triggered_by == "schedule"` (cron). Manual triggers bypass entirely.
+
+```python
+effective_message = schedule.message
+if triggered_by == "schedule":
+    decision = await self._run_pre_check(schedule.agent_name)
+    if decision is not None:
+        if not decision.get("fire", True):
+            self.db.create_skipped_execution(...)
+            self._publish_event({"type": "schedule_execution_skipped", ...})
+            return
+        override = decision.get("message")
+        if override:
+            effective_message = override
+# ... fires with effective_message
+```
+
+## Data Layer
+**Zero schema change.** Reuses existing:
+- `ExecutionStatus.SKIPPED` (already defined for Issue #46 — APScheduler max-instances drops).
+- `SchedulerDatabase.create_skipped_execution(...)`.
+
+Skip rows carry `status='skipped'`, `error="pre-check: <reason>"`, `duration_ms=0`, `started_at == completed_at`.
+
+## WebSocket Layer
+New event type: `schedule_execution_skipped`.
+
+```json
+{
+  "type": "schedule_execution_skipped",
+  "agent": "pr-reviewer",
+  "schedule_id": "LidXcOwtDsDuFTFGvkUqCw",
+  "execution_id": "I2DWodfpYpuJbs1TTZ42Ig",
+  "schedule_name": "PR review poll",
+  "reason": "pre-check: pre-check returned empty stdout"
+}
+```
+
+## Side Effects
+- Skipped execution row written to `schedule_executions`
+- `schedule.last_run_at` and `next_run_at` updated (so missed-schedule detection still works)
+- WebSocket event broadcast
+- No `/api/internal/execute-task` call, no backend task creation, no Claude invocation, no backlog slot acquisition
+
+## Error Handling
+| Condition | Scheduler behavior |
+|---|---|
+| Agent container doesn't exist | Backend returns 404 → fire as usual (schedule likely stale, let execute-task path handle the 404 surfacing) |
+| `~/.trinity/pre-check` absent | `hook_present: false` → fire as usual (backward compat) |
+| File present but not `+x` (exit 126) | Fail-open — log "hook for X exited 126", fire with `schedule.message`. Operator's signal to `chmod +x` the hook. |
+| Shebang missing or interpreter not found (exit 127) | Fail-open — log shows "command not found"; operator fixes shebang. |
+| Hook exits non-zero for any other reason | Fail-open — log stderr, fire with `schedule.message` |
+| Exec timeout (>60s) | Backend returns non-zero exit → fail-open |
+| Backend unreachable (connection error) | Fail-open — fire as usual, log warning |
+| Backend 5xx / malformed JSON | Fail-open |
+| Exit 0, empty stdout | Record skipped execution, no chat dispatch |
+| Exit 0, non-empty stdout | Fire with stdout as chat message (overrides `schedule.message`) |
+| Manual trigger (`triggered_by != 'schedule'`) | Skip pre-check entirely — explicit operator intent always fires |
+
+## Security
+- Pre-check runs inside the agent's container as `developer`, same sandbox as chat-mode tool calls. No new privilege over what chat-mode tool calls can already do.
+- **Template review expectation**: `.trinity/pre-check` is exec'd directly via the kernel — full process privileges of `developer`, in whatever language the shebang names. It can spawn subprocesses, open sockets, read files. This is intentional (operators already trust the template's `CLAUDE.md`, skills, and tool invocations), but `.trinity/pre-check` should be reviewed with the same scrutiny as any other executable the template ships.
+- Backend endpoint is gated by the existing `X-Internal-Secret` header (C-003). Only `trinity-scheduler` and other internal services can invoke it.
+- Stdout cap at 32 KB on the backend side — oversized output is truncated, still valid as a chat prompt (or truncated to "looks non-empty" which is fine for the fire-with-override path).
+- Fail-open policy means a malicious/broken pre-check cannot suppress scheduled invocations (worst case: wastes tokens — today's baseline).
+
+## Testing
+**Scheduler-side** (`tests/scheduler_tests/test_pre_check.py`, 13 tests):
+- `_run_pre_check` translation — `hook_present: False` → None; non-zero exit → None; empty stdout → skip; non-empty stdout → fire with message; 404 / 5xx / connection error / malformed JSON all → None (fail-open).
+- `_execute_schedule_with_lock` branch — skip records execution with correct reason; fire-true with message uses override; fail-open routes through backend; fire-true without message uses `schedule.message`; manual trigger bypasses pre-check.
+
+Full scheduler suite: 162/162 passing (was 161 before; +1 for the new `malformed JSON` case).
+
+No test file on the agent-server side — there's no agent-server router anymore.
+
+**Live end-to-end** (verified 2026-04-22 in local Trinity on the HTTP-endpoint version before pivoting to docker-exec):
+- Empty scan → skip row in DB, `$0` cost, zero backend chat activity.
+- Open PR → next tick fires with override message, Claude runs `/review`, posts comment.
+- Subsequent tick with existing bot comment → `fire:false` again (stateless dedup via GitHub).
+
+## Related Flows
+- `feature-flows/agent-event-subscriptions.md` — EVT-001, the event-driven analogue (other trigger source, same "non-cron agent invocation" theme).
+- `feature-flows/scheduler-service.md` — base scheduler behavior this extends.
+- `docs/planning/PR_REVIEWER_AGENT.md` — the motivating use case that drove this feature.
+
+## Architectural notes
+- Preserves **Invariant #11 (Docker as source of truth)**: the pre-check primitive is `docker exec`, matching `git_service.py`'s persistent-state allowlist, `ssh_service.py`, the agent terminal, etc.
+- Preserves the **"scheduler → backend → agent"** topology: scheduler never opens a direct HTTP edge to agent-server containers.
+- Reuses **`execute_command_in_container`**, an established async helper in `services/docker_service.py`.
+- No new schema, no new long-lived process, no new primitive — just a new internal endpoint and a scheduler branch.
+
+## Migration / Rollout
+- Zero migration required (no schema change).
+- Existing schedules and agent templates behave identically after deploy (script absent → fall back to today's fire semantics).
+- Templates opt in by shipping `.trinity/pre-check` (executable, `+x`, with a shebang). No Trinity-side flag. File extension is intentionally absent — the file is whatever the shebang says it is.

docs/memory/requirements.md

@@ -373,6 +373,23 @@ Trinity is autonomous agent orchestration and infrastructure — sovereign infra
   - Configurable `POLL_INTERVAL` env var (default 10s)
 - **Root Cause**: TCP connection drops after 15-30 min on long-running scheduled tasks, causing false `failed` status even though agent work completed successfully
 
+### 10.6.1 Conditional Schedule Pre-Check (SCHED-COND-001)
+- **Status**: ✅ Implemented (2026-04-22)
+- **Requirement ID**: SCHED-COND-001
+- **GitHub Issue**: #454
+- **Description**: Optional template-supplied hook that lets a scheduled cron tick be skipped deterministically — the scheduler calls a new internal backend endpoint which `docker exec`s the executable `~/.trinity/pre-check` file inside the target agent container; non-empty stdout becomes the chat prompt, empty stdout + exit 0 records a skipped execution. The hook is language-agnostic (interpreter chosen by shebang). Eliminates Claude token cost on empty polls for poll-driven agents (PR reviewers, inbox monitors, alert routers, RSS watchers).
+- **Key Features**:
+  - Contract: agent templates drop an executable `~/.trinity/pre-check` file with a shebang (`#!/usr/bin/env python3`, `#!/bin/bash`, …). Trinity execs it directly — no `python3` prefix, no language assumption. Stdout is the chat prompt; empty stdout + exit 0 = skip; non-zero exit = fail-open.
+  - Backend endpoint: `POST /api/internal/agents/{name}/pre-check` (X-Internal-Secret gated) runs the script via `execute_command_in_container` — the same primitive used by `git_service.py` (persistent-state allowlist, #384 S3), `ssh_service.py`, `agent_service/terminal.py`, `adapters/message_router.py`, `routers/system_agent.py`, `routers/voice.py`.
+  - Fail-open: script absent, non-zero exit, timeout, backend 5xx / malformed response → scheduler fires as usual. A broken pre-check never silently suppresses scheduled work.
+  - Message override: non-empty stdout replaces `schedule.message` for that one invocation — lets the agent inject real work items (e.g. the PR list) into the chat prompt.
+  - Skip record: empty stdout writes a row to `schedule_executions` with `status='skipped'`, reason, and zero cost — visible in the Trinity UI alongside successful runs.
+  - Manual triggers bypass pre-check entirely (explicit operator intent always fires).
+  - Zero DB schema change (reuses existing `ExecutionStatus.SKIPPED` + `create_skipped_execution`).
+  - **No new HTTP edge**: scheduler calls backend, backend `docker exec`s into agent. Topology stays "scheduler → backend → agent" (Invariant #11).
+- **Test plan**: 13 unit tests covering backend-response translation (hook absent / non-zero exit / empty stdout / fire-with-message / 404 / 5xx / connection error / malformed JSON) + scheduler branch behaviors (skip, override, fail-open, manual-bypass). Full 162-test scheduler suite passes.
+- **Root Cause**: No platform primitive for "deterministic gate before LLM invocation." Previously required per-template daemons backgrounded inside agent containers — invisible to Trinity UI, reimplemented per template, no skip metrics.
+
 ### 10.7 Per-Agent Execution Timeout (TIMEOUT-001)
 - **Status**: ✅ Implemented (2026-03-12)
 - **Requirement ID**: TIMEOUT-001