Skip to content

ci(deploy): /health fallback + forensics capture for transient Fly API timeouts (closes #4780)#4785

Merged
bokelley merged 3 commits into
mainfrom
fix/deploy-yml-resilience
May 19, 2026
Merged

ci(deploy): /health fallback + forensics capture for transient Fly API timeouts (closes #4780)#4785
bokelley merged 3 commits into
mainfrom
fix/deploy-yml-resilience

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

Closes #4780. Surfaced by today's outage post-mortem (PRs #4769 hotfix, #4773 structural fix).

v2760 (the hotfix release) was marked `failed` in Fly's release ledger because `flyctl deploy` timed out at 5 minutes polling the third web machine's health (`net/http: request canceled` against `api.machines.dev`). The Fly machines API was rate-limited from the earlier crashloop storm. The app itself was healthy — `adcontextprotocol.org` served 200 throughout — but the deploy job exited 1 and the post-deploy verification gates (machine-image-consistency, training-agent tenant smoke, stale-console cleanup) never ran.

Changes

  1. `Deploy` step captures `flyctl deploy` exit code instead of letting it short-circuit the job. On non-zero, probes `https://adcontextprotocol.org/health\` and stores outcome in step output `deploy_outcome`:
    • `success` — flyctl exited 0
    • `fallback-success` — flyctl failed but `/health=200`. Warning, continue.
    • `hard-fail` — flyctl failed and `/health!=200`. Exit 1.
  2. `Verify all app machines are on same image` now skips on `fallback-success` — that gate also goes through the Fly machines API (the thing that timed out in the first place).
  3. `Smoke training-agent tenants` unchanged. Uses external HTTP, independent of Fly API.
  4. Forensics capture on any failure: `flyctl status`, `machines list --json`, `releases`, last 500 lines of `flyctl logs`, plus a summary file with `deploy_outcome`, SHA, run ID. Uploaded as artifact `fly-forensics-{run_id}` with 30-day retention. Today's incident cost a debug cycle because Fly log retention is short.

Test plan

  • CI green
  • On merge: next `flyctl deploy` happy path completes as `success` (deploy_outcome=success), all gates run, no artifact uploaded (failure() didn't trigger)
  • Eventually: when next transient Fly API issue happens, the workflow logs `fallback-success` and the tenant smoke still runs

Risk

Bounded to the deploy workflow. Doesn't affect prod runtime. The conservative direction: if /health is genuinely down AND flyctl timed out, we still hard-fail.

🤖 Generated with Claude Code

bokelley and others added 3 commits May 19, 2026 07:18
…I timeouts

Closes #4780.

After today's outage (PRs #4769 / #4773), v2760's release ledger entry
was marked `failed` even though the app was healthy. flyctl deploy
timed out at 5min polling the third web machine — Fly machines API
was rate-limited from the earlier crashloop storm. The deploy job
exited 1 and the post-deploy verification gates never ran.

Distinguish two failure modes:
- Real app failure: image is broken. Hard-fail (exit 1).
- Fly API timeout: app may be healthy on the new image already.
  Probe https://adcontextprotocol.org/health. If 200, treat as
  fallback-success and continue to the HTTP-based gates (tenant
  smoke). Skip the machine-image-consistency gate because it also
  goes through the Fly machines API.

Independently: capture flyctl status / machines list / releases /
last 500 lines of logs on any job failure, upload as a workflow
artifact with 30-day retention. Fly log retention is short; today
we lost the v2760 third-machine boot trace before forensics started.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…signal

Self-review: adcontextprotocol.org is Cloudflare-fronted, and CF can
cache /health responses. A stale 200 from CF would mask a real
outage and incorrectly downgrade hard-fail to fallback-success.

adcp-docs.fly.dev bypasses CDN and reflects actual app state on the
machine. Verified during today's outage: the Fly hostname returned
200 the moment v2760's first web machine was healthy, before CF
caches refreshed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes from independent review:

1. Replace single sleep+probe with a 4×5s retry loop. Catches slower
   rolling restarts without lengthening the happy path (the loop only
   runs when flyctl already exited non-zero).

2. Add 'set -eo pipefail' to "Verify all machines on same image" so
   a silent flyctl failure (empty $images → count=0) surfaces as a
   real failure instead of being misread as the "no running machines"
   branch.

3. Redact common secret shapes before persisting flyctl logs to the
   30-day forensics artifact. Boot logs have historically contained
   JWT fragments and SDK init lines that echo env-derived config; the
   filter covers Bearer/Authorization/sk-/wos_/fly_/JWT-prefix patterns.
   Best-effort — not a substitute for not logging secrets in the first
   place — but the artifact's blast radius is now bounded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit 09cfc8e into main May 19, 2026
13 checks passed
@bokelley bokelley deleted the fix/deploy-yml-resilience branch May 19, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deploy.yml: add /health fallback + forensics capture when flyctl deploy times out

1 participant