Skip to content

runner: operator alerts on job failure/recovery - #191

Open
unforced wants to merge 1 commit into
mainfrom
job-failure-alerts
Open

runner: operator alerts on job failure/recovery#191
unforced wants to merge 1 commit into
mainfrom
job-failure-alerts

Conversation

@unforced

Copy link
Copy Markdown
Contributor

Summary

Closes the R1 gap in the daemon's risk register: a scheduled #agent/job fire
failing wrote lastStatus: "error: ..." on the note and nothing else
happened
. Every silent failure to date (e.g. a rate-limited 4am weave that
produced no report and went unnoticed for a day) was found by the operator
checking by hand, not surfaced by the system.

  • src/runner.ts — the Runner now detects an ok↔error transition per
    job fire and calls an optional injected notify(event) hook: always on a
    fresh failure into error (including a job's very first fire failing),
    throttled to at most one alert per 24h on a job that's still failing
    (no spam on a stuck job), and always once on recovery (error → ok).
    Same "inject the side effect, keep the core pure/testable" shape as the
    existing loadJobs/fire/persistFire seams — no real clock/transport in
    the decision logic itself.

  • src/daemon.tsbuildJobAlertNotify() wires that hook to the existing
    Telegram transport's reply()
    send path (no new transport, no new
    external API surface). Gated on two new env vars:

    • PARACHUTE_AGENT_ALERT_CHANNEL — name of an already-configured telegram
      channel (channels.json) whose bot sends the alert.
    • PARACHUTE_AGENT_ALERT_CHAT_ID — the chat id (DM or group) to send it to.

    Both unset → silent no-op (today's behavior, unchanged). Exactly one set →
    logs a loud warning once at boot. Both set but the named channel isn't a
    live telegram transport → logs + drops per-alert (checked at send time, not
    boot, so a channel added later is picked up without a restart). An
    alert-delivery failure is caught by the runner's notify guard and never
    breaks the tick — the circuit-breaker discipline in Runner.tick() is
    preserved.

  • CLAUDE.md — documents the two new env vars in the existing table.

Test plan

  • bun run typecheck — clean
  • bun test ./src/1490 pass, 0 fail (1476 baseline + 14 new)
  • runner.test.ts: ok→error alerts once; a job's first-ever fire failing
    alerts too; ok→ok never alerts; error→ok fires exactly one recovery
    alert; a throwing notify is swallowed (tick still resolves, bookkeeping
    still persists); a still-failing job is throttled to ≤1 alert per window
  • daemon-job-alerts.test.ts (new): env-var resolution (unset / partial /
    both set); unreachable/non-telegram alert channel drops without
    throwing; a live TelegramTransport sends via reply() with the
    configured chat_id and a message containing job id / agent channel /
    cron / error / note path; a recovery event's message has no error/note
    detail; a reply() failure propagates out of notify (so the runner's
    guard is what's responsible for swallowing it, not the notify builder)

Not deployed, not merged to main, running daemon untouched.

A scheduled job's fire failing used to write lastStatus: "error: ..." on its
#agent/job note and nothing else happened -- every silent outage to date
(e.g. a rate-limited 4am weave with no report) was found by hand. The runner
now detects an ok<->error TRANSITION per job and calls an injected `notify`
hook: always on a fresh failure (or a job's first-ever fire failing), at most
once per 24h on a still-failing job (no spam), and once on recovery. The
daemon wires `notify` to the existing Telegram transport's reply() path,
gated on two new env vars (PARACHUTE_AGENT_ALERT_CHANNEL /
PARACHUTE_AGENT_ALERT_CHAT_ID) -- unconfigured is a silent no-op, and an
alert-delivery failure is caught so it never breaks the runner's tick.
unforced added a commit that referenced this pull request Jul 24, 2026
…erged); restore when #191 lands

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
unforced added a commit that referenced this pull request Jul 24, 2026
* docs: slim CLAUDE.md to purpose + gotchas (472 -> 157 lines)

Context refit per the workspace conventions (docs/process/context.md in
ParachuteComputer): CLAUDE.md carries purpose + non-derivable gotchas only;
derivable inventories (env-var table, MCP tool tables, discovery detail,
testing curl, state-dir listing) become pointers to the code that defines
them; the Stage-2 vault protocol and #agent/thread model compress to
trigger-level summaries linking their design docs.

Kept, compressed to their trigger: the =-binding flag-parser trap (#8), the
no-silent-message-loss high-water-mark rule, why-not-the-official-telegram-
plugin (anthropics/claude-code#38098), the step-up PIN hard rules (agent#80),
and the fail-closed allowInChats semantics. Post-merge-hygiene section
deleted (the workspace CLAUDE.md owns it).

Docs-only; no version bump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB

* docs: drop the alert env-var pairing line — it documents PR #191 (unmerged); restore when #191 lands

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB

* docs: clean the dangling sentence from the alert-line removal

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant