ci(deploy): /health fallback + forensics capture for transient Fly API timeouts (closes #4780)#4785
Merged
Merged
Conversation
…I timeouts Closes #4780. After today's outage (PRs #4769 / #4773), v2760's release ledger entry was marked `failed` even though the app was healthy. flyctl deploy timed out at 5min polling the third web machine — Fly machines API was rate-limited from the earlier crashloop storm. The deploy job exited 1 and the post-deploy verification gates never ran. Distinguish two failure modes: - Real app failure: image is broken. Hard-fail (exit 1). - Fly API timeout: app may be healthy on the new image already. Probe https://adcontextprotocol.org/health. If 200, treat as fallback-success and continue to the HTTP-based gates (tenant smoke). Skip the machine-image-consistency gate because it also goes through the Fly machines API. Independently: capture flyctl status / machines list / releases / last 500 lines of logs on any job failure, upload as a workflow artifact with 30-day retention. Fly log retention is short; today we lost the v2760 third-machine boot trace before forensics started. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…signal Self-review: adcontextprotocol.org is Cloudflare-fronted, and CF can cache /health responses. A stale 200 from CF would mask a real outage and incorrectly downgrade hard-fail to fallback-success. adcp-docs.fly.dev bypasses CDN and reflects actual app state on the machine. Verified during today's outage: the Fly hostname returned 200 the moment v2760's first web machine was healthy, before CF caches refreshed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes from independent review: 1. Replace single sleep+probe with a 4×5s retry loop. Catches slower rolling restarts without lengthening the happy path (the loop only runs when flyctl already exited non-zero). 2. Add 'set -eo pipefail' to "Verify all machines on same image" so a silent flyctl failure (empty $images → count=0) surfaces as a real failure instead of being misread as the "no running machines" branch. 3. Redact common secret shapes before persisting flyctl logs to the 30-day forensics artifact. Boot logs have historically contained JWT fragments and SDK init lines that echo env-derived config; the filter covers Bearer/Authorization/sk-/wos_/fly_/JWT-prefix patterns. Best-effort — not a substitute for not logging secrets in the first place — but the artifact's blast radius is now bounded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #4780. Surfaced by today's outage post-mortem (PRs #4769 hotfix, #4773 structural fix).
v2760 (the hotfix release) was marked `failed` in Fly's release ledger because `flyctl deploy` timed out at 5 minutes polling the third web machine's health (`net/http: request canceled` against `api.machines.dev`). The Fly machines API was rate-limited from the earlier crashloop storm. The app itself was healthy — `adcontextprotocol.org` served 200 throughout — but the deploy job exited 1 and the post-deploy verification gates (machine-image-consistency, training-agent tenant smoke, stale-console cleanup) never ran.
Changes
Test plan
Risk
Bounded to the deploy workflow. Doesn't affect prod runtime. The conservative direction: if /health is genuinely down AND flyctl timed out, we still hard-fail.
🤖 Generated with Claude Code