feat(observability): worker-unreachable watchdog (escalation #329)#4358
Merged
Conversation
Polls http://worker.process.<app>.internal:8080/internal/jobs from web every 60s and emits logger.error after 3 consecutive failures. logger.error auto-routes to #admin-errors via the existing posthog notifier, so the alert lands in Slack without any extra wiring. Closes the silent-death gap behind escalation #329: the worker crashlooped for 6 days. Every scheduled job (compliance heartbeat, escalation triage, weekly digest, announcement handlers, etc.) silently stopped firing and the only signal a user got was Evgeny's stale "comply re-runner" status. workerUnreachable was already in the /api/admin/jobs response shape but nothing was polling it. Fire-once semantics: alert when streak crosses threshold, info-level recovery line on the first successful tick after an alert. Without these a flapping worker would page on every tick. 5 vitest cases cover threshold-crossing, single-alert dedup, recovery transition, non-2xx as failure, and streak reset on transient success. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 18, 2026
bokelley
added a commit
that referenced
this pull request
May 18, 2026
PR #3374's `auto_start_machines = true` + `min_machines_running = 1` on the worker `[[services]]` block are inert without a `[[services.ports]]` entry — Fly only enforces those flags via fly-proxy, and a service with no ports isn't routed through fly-proxy. Rolling deploys that leave the worker `stopped` had no recovery path; the watchdog from PR #4358 observed the failure but couldn't act on it. Today's 04:31 UTC alert required a manual `fly machine start`. On probe failure the watchdog now calls the Fly Machines API, finds any worker machines in `stopped` state, and starts them. The recovered machine surfaces as a successful probe on the next tick. If start didn't help (real crashloop), failures keep climbing and the alert still fires at the threshold. Requires the `FLY_API_TOKEN` secret on the app — without it, recovery is a no-op and behavior matches PR #4358 exactly. Mint with `fly tokens create deploy -a adcp-docs` and set via `fly secrets set`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
added a commit
that referenced
this pull request
May 18, 2026
* docs(fly): correct misleading worker autostart comment (#4723) The fly.toml block for the worker process group claimed that adding `auto_start_machines = true` + `min_machines_running = 1` opted workers into Fly's autostart lifecycle. It does not — per Fly docs, those flags are enforced through fly-proxy, and a `[[services]]` block without `[[services.ports]]` doesn't get fly-proxy routing. The settings are inert; the block still earns its keep by attaching the http_check. The actual recovery gap (rolling deploys can leave the worker `stopped` with no automatic restart) is tracked in #4723. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(watchdog): active recovery for stopped workers (#4723) PR #3374's `auto_start_machines = true` + `min_machines_running = 1` on the worker `[[services]]` block are inert without a `[[services.ports]]` entry — Fly only enforces those flags via fly-proxy, and a service with no ports isn't routed through fly-proxy. Rolling deploys that leave the worker `stopped` had no recovery path; the watchdog from PR #4358 observed the failure but couldn't act on it. Today's 04:31 UTC alert required a manual `fly machine start`. On probe failure the watchdog now calls the Fly Machines API, finds any worker machines in `stopped` state, and starts them. The recovered machine surfaces as a successful probe on the next tick. If start didn't help (real crashloop), failures keep climbing and the alert still fires at the threshold. Requires the `FLY_API_TOKEN` secret on the app — without it, recovery is a no-op and behavior matches PR #4358 exactly. Mint with `fly tokens create deploy -a adcp-docs` and set via `fly secrets set`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Web-side watchdog that polls
/internal/jobson the worker every 60s and emitslogger.errorafter 3 consecutive failures.logger.errorauto-routes to #admin-errors via the existing posthog notifier, so the alert reaches Slack without any extra wiring.Why
Escalation #329 surfaced that the worker had crashlooped for 6 days (machine created 2026-05-04, recovered 2026-05-10). Every scheduled job — compliance heartbeat, escalation triage, weekly digest, announcement handlers, conversation insights, knowledge staleness, ~40 others — silently stopped firing. The only user-visible signal was Evgeny's stale "comply re-runner" status; the response from
/api/admin/jobsalready carriedworkerUnreachable: truebut nothing was polling it.Design
worker.process.<app>.internal:8080lookup the existing/api/admin/jobsproxy uses (http.ts:2256), so the alert reflects the exact reachability that the admin UI does.Tests
5 vitest cases:
Follow-up (separate PR)
The actual root cause of the 6-day crashloop —
app.listen()racing the training-agent tenant warmup, killing the worker via health-check timeout during the 30-60s warmup window — is being addressed by re-landing PR #4062 (the boot gate that was reverted on May 4). Tracking separately.🤖 Generated with Claude Code