runner: operator alerts on job failure/recovery - #191
Open
unforced wants to merge 1 commit into
Open
Conversation
A scheduled job's fire failing used to write lastStatus: "error: ..." on its #agent/job note and nothing else happened -- every silent outage to date (e.g. a rate-limited 4am weave with no report) was found by hand. The runner now detects an ok<->error TRANSITION per job and calls an injected `notify` hook: always on a fresh failure (or a job's first-ever fire failing), at most once per 24h on a still-failing job (no spam), and once on recovery. The daemon wires `notify` to the existing Telegram transport's reply() path, gated on two new env vars (PARACHUTE_AGENT_ALERT_CHANNEL / PARACHUTE_AGENT_ALERT_CHAT_ID) -- unconfigured is a silent no-op, and an alert-delivery failure is caught so it never breaks the runner's tick.
unforced
added a commit
that referenced
this pull request
Jul 24, 2026
…erged); restore when #191 lands Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
unforced
added a commit
that referenced
this pull request
Jul 24, 2026
* docs: slim CLAUDE.md to purpose + gotchas (472 -> 157 lines) Context refit per the workspace conventions (docs/process/context.md in ParachuteComputer): CLAUDE.md carries purpose + non-derivable gotchas only; derivable inventories (env-var table, MCP tool tables, discovery detail, testing curl, state-dir listing) become pointers to the code that defines them; the Stage-2 vault protocol and #agent/thread model compress to trigger-level summaries linking their design docs. Kept, compressed to their trigger: the =-binding flag-parser trap (#8), the no-silent-message-loss high-water-mark rule, why-not-the-official-telegram- plugin (anthropics/claude-code#38098), the step-up PIN hard rules (agent#80), and the fail-closed allowInChats semantics. Post-merge-hygiene section deleted (the workspace CLAUDE.md owns it). Docs-only; no version bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB * docs: drop the alert env-var pairing line — it documents PR #191 (unmerged); restore when #191 lands Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB * docs: clean the dangling sentence from the alert-line removal Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the R1 gap in the daemon's risk register: a scheduled
#agent/jobfirefailing wrote
lastStatus: "error: ..."on the note and nothing elsehappened. Every silent failure to date (e.g. a rate-limited 4am weave that
produced no report and went unnoticed for a day) was found by the operator
checking by hand, not surfaced by the system.
src/runner.ts— theRunnernow detects an ok↔error transition perjob fire and calls an optional injected
notify(event)hook: always on afresh failure into error (including a job's very first fire failing),
throttled to at most one alert per 24h on a job that's still failing
(no spam on a stuck job), and always once on recovery (error → ok).
Same "inject the side effect, keep the core pure/testable" shape as the
existing
loadJobs/fire/persistFireseams — no real clock/transport inthe decision logic itself.
src/daemon.ts—buildJobAlertNotify()wires that hook to the existingTelegram transport's
reply()send path (no new transport, no newexternal API surface). Gated on two new env vars:
PARACHUTE_AGENT_ALERT_CHANNEL— name of an already-configured telegramchannel (channels.json) whose bot sends the alert.
PARACHUTE_AGENT_ALERT_CHAT_ID— the chat id (DM or group) to send it to.Both unset → silent no-op (today's behavior, unchanged). Exactly one set →
logs a loud warning once at boot. Both set but the named channel isn't a
live telegram transport → logs + drops per-alert (checked at send time, not
boot, so a channel added later is picked up without a restart). An
alert-delivery failure is caught by the runner's
notifyguard and neverbreaks the tick — the circuit-breaker discipline in
Runner.tick()ispreserved.
CLAUDE.md— documents the two new env vars in the existing table.Test plan
bun run typecheck— cleanbun test ./src/— 1490 pass, 0 fail (1476 baseline + 14 new)runner.test.ts: ok→error alerts once; a job's first-ever fire failingalerts too; ok→ok never alerts; error→ok fires exactly one recovery
alert; a throwing
notifyis swallowed (tick still resolves, bookkeepingstill persists); a still-failing job is throttled to ≤1 alert per window
daemon-job-alerts.test.ts(new): env-var resolution (unset / partial /both set); unreachable/non-telegram alert channel drops without
throwing; a live
TelegramTransportsends viareply()with theconfigured
chat_idand a message containing job id / agent channel /cron / error / note path; a recovery event's message has no error/note
detail; a
reply()failure propagates out ofnotify(so the runner'sguard is what's responsible for swallowing it, not the notify builder)
Not deployed, not merged to main, running daemon untouched.