fix(fly): auto_start workers — rolling deploys were leaving them stopped#3374
Merged
Conversation
…ers stopped) The worker process group has no auto_start_machines path. http_service covers web, but the bare [[checks]] block for worker-health only keeps running workers from being killed — it doesn't start a stopped worker. Symptom: rolling deploys create new worker machines but never start them. Both worker machines on adcp-docs sat in 'stopped' state with check status "the machine hasn't started" since the readers PR deploy (2026-04-28 06:51 UTC), which means the periodic crawler that calls publisherDb.upsertAdagentsCache (every 30 min for the catalog queue, every 6h for buying agents) never ran in production after the PR 4b chain merged. The full property-registry-unification chain (#3274/#3314/#3312/#3352) was therefore unexercised in prod until a manual /api/registry/crawl-request triggered it. Replace the bare [[checks]] block with a [[services]] block targeting processes=["worker"] with auto_start_machines=true and min_machines_running=2 — same lifecycle hooks http_service uses for web. No [[services.ports]] block, so workers stay private. The http_check inside the services block replaces the prior bare check. Refs #3177 (this is the gap that prevented prod verification of the 4b chain). EOF
This was referenced May 18, 2026
bokelley
added a commit
that referenced
this pull request
May 18, 2026
PR #3374's `auto_start_machines = true` + `min_machines_running = 1` on the worker `[[services]]` block are inert without a `[[services.ports]]` entry — Fly only enforces those flags via fly-proxy, and a service with no ports isn't routed through fly-proxy. Rolling deploys that leave the worker `stopped` had no recovery path; the watchdog from PR #4358 observed the failure but couldn't act on it. Today's 04:31 UTC alert required a manual `fly machine start`. On probe failure the watchdog now calls the Fly Machines API, finds any worker machines in `stopped` state, and starts them. The recovered machine surfaces as a successful probe on the next tick. If start didn't help (real crashloop), failures keep climbing and the alert still fires at the threshold. Requires the `FLY_API_TOKEN` secret on the app — without it, recovery is a no-op and behavior matches PR #4358 exactly. Mint with `fly tokens create deploy -a adcp-docs` and set via `fly secrets set`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
added a commit
that referenced
this pull request
May 18, 2026
* docs(fly): correct misleading worker autostart comment (#4723) The fly.toml block for the worker process group claimed that adding `auto_start_machines = true` + `min_machines_running = 1` opted workers into Fly's autostart lifecycle. It does not — per Fly docs, those flags are enforced through fly-proxy, and a `[[services]]` block without `[[services.ports]]` doesn't get fly-proxy routing. The settings are inert; the block still earns its keep by attaching the http_check. The actual recovery gap (rolling deploys can leave the worker `stopped` with no automatic restart) is tracked in #4723. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(watchdog): active recovery for stopped workers (#4723) PR #3374's `auto_start_machines = true` + `min_machines_running = 1` on the worker `[[services]]` block are inert without a `[[services.ports]]` entry — Fly only enforces those flags via fly-proxy, and a service with no ports isn't routed through fly-proxy. Rolling deploys that leave the worker `stopped` had no recovery path; the watchdog from PR #4358 observed the failure but couldn't act on it. Today's 04:31 UTC alert required a manual `fly machine start`. On probe failure the watchdog now calls the Fly Machines API, finds any worker machines in `stopped` state, and starts them. The recovered machine surfaces as a successful probe on the next tick. If start didn't help (real crashloop), failures keep climbing and the alert still fires at the threshold. Requires the `FLY_API_TOKEN` secret on the app — without it, recovery is a no-op and behavior matches PR #4358 exactly. Mint with `fly tokens create deploy -a adcp-docs` and set via `fly secrets set`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production workers on adcp-docs have been sitting in
stoppedstate with check status\"the machine hasn't started\"since the readers PR (#3352) deploy. The worker process group's lifecycle was never opted into Fly's auto_start_machines path the way[http_service]is for web — so a rolling deploy creates new worker machines but never starts them.This is what prevented the full property-registry-unification chain (#3274/#3314/#3312/#3352) from being exercised in production: the periodic crawler that calls
PublisherDatabase.upsertAdagentsCacheonly runs on workers (every 30 min for the catalog queue, every 6h for buying agents), and workers were dead.Verified by manually firing
POST /api/registry/crawl-requestformamamia.com.au: writer projected one row, trigger emittedauthorization.granted, UNION reader returned the catalog row through/api/registry/lookup/agent/.../domains. Full chain works — workers just weren't running it.Change
Replace the bare
[[checks]]block for worker-health with a[[services]]block:processes = [\"worker\"]— scopes to the worker process groupauto_start_machines = true— same lifecycle hook[http_service]uses for webmin_machines_running = 2— Fly will keep two workers alive[[services.ports]]block — no external traffic; the internal_port is only used by the http_check[[services.http_checks]]replaces the prior bare checkfly config validatepasses (warning about missing ports is stale documentation about a Feb 2024 deprecation that didn't actually hard-fail).Test plan
fly machine startneeded)./api/registry/feed?types=authorization.*shows new events from the queue draining.stopped.Rollback
Revert this PR; workers fall back to manual start. Either run
fly machine start <id>for each, orfly scale count worker=2 --process=worker.🤖 Generated with Claude Code