feat(watchdog): active recovery for stopped workers (#4723)#4724
Merged
Conversation
The fly.toml block for the worker process group claimed that adding `auto_start_machines = true` + `min_machines_running = 1` opted workers into Fly's autostart lifecycle. It does not — per Fly docs, those flags are enforced through fly-proxy, and a `[[services]]` block without `[[services.ports]]` doesn't get fly-proxy routing. The settings are inert; the block still earns its keep by attaching the http_check. The actual recovery gap (rolling deploys can leave the worker `stopped` with no automatic restart) is tracked in #4723. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3374's `auto_start_machines = true` + `min_machines_running = 1` on the worker `[[services]]` block are inert without a `[[services.ports]]` entry — Fly only enforces those flags via fly-proxy, and a service with no ports isn't routed through fly-proxy. Rolling deploys that leave the worker `stopped` had no recovery path; the watchdog from PR #4358 observed the failure but couldn't act on it. Today's 04:31 UTC alert required a manual `fly machine start`. On probe failure the watchdog now calls the Fly Machines API, finds any worker machines in `stopped` state, and starts them. The recovered machine surfaces as a successful probe on the next tick. If start didn't help (real crashloop), failures keep climbing and the alert still fires at the threshold. Requires the `FLY_API_TOKEN` secret on the app — without it, recovery is a no-op and behavior matches PR #4358 exactly. Mint with `fly tokens create deploy -a adcp-docs` and set via `fly secrets set`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Watchdog now actively recovers a stopped worker via the Fly Machines API, closing the gap behind today's 04:31 UTC alert (issue #4723).
Root cause
PR #3374 added
[[services]]withauto_start_machines = true+min_machines_running = 1for the worker process group, claiming this would prevent rolling deploys from leaving workersstopped. Per Fly docs, those flags are only enforced via fly-proxy, and a[[services]]block without[[services.ports]]isn't routed through fly-proxy — so both settings are inert. The watchdog from PR #4358 observed the failure but couldn't recover it.Today's incident: v2738 rolling deploy. Worker
6832693c334dd8event log:pending(launch) → created(launch) → stopped(update)— nostartevent. Watchdog tripped at 3 misses (~180s); manualfly machine startwas required.Fix
On probe failure the watchdog calls
GET /v1/apps/{app}/machines, filters tofly_process_group == worker && state == stopped, and issuesPOST /v1/apps/{app}/machines/{id}/startfor each. Recovered machines surface as successful probes on the next tick. If start didn't help (real crashloop), the failure streak still climbs and the alert fires at threshold as before.Recovery requires
FLY_API_TOKENsecret on the app. Without it the watchdog gracefully degrades — behavior matches PR #4358 exactly. Failures during the recovery call are logged atwarn(noterror, so they don't re-route to #admin-errors and stack on top of the real alert).Setup
After merge, mint and set the token (one time):
Until that secret is set, the watchdog behaves identically to today.
Test plan
stoppedshould self-heal within one watchdog tick; alert should NOT fireRefs
🤖 Generated with Claude Code