Skip to content

hotfix: renumber colliding migration 419 → 420 (unblocks all fresh DB boots)#2813

Merged
bokelley merged 1 commit into
mainfrom
bokelley/migration-419-collision
Apr 22, 2026
Merged

hotfix: renumber colliding migration 419 → 420 (unblocks all fresh DB boots)#2813
bokelley merged 1 commit into
mainfrom
bokelley/migration-419-collision

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Problem

Main has two migrations numbered 419:

  • `419_agent_visibility.sql` (from #2793, merged first)
  • `419_oauth_client_credentials.sql` (from #2800, merged second)

The per-PR `No duplicate migration numbers` CI check passed on each individually because at check-time the other 419 wasn't on main yet — merge-queue race.

Symptom: every fresh DB boot fails with:
```
Migration filename validation failed:
Duplicate migration version 419: 419_agent_visibility.sql and 419_oauth_client_credentials.sql
```

Every already-deployed instance will fail on next restart once it sees both 419s on disk.

Fix

Rename `419_oauth_client_credentials.sql` → `420_oauth_client_credentials.sql` (the migration that landed second). Pure filename change, no SQL modification.

Safety (all three deployment states)

The migration SQL is idempotent (`ADD COLUMN IF NOT EXISTS`, `DROP VIEW IF EXISTS` + `CREATE VIEW`), so:

State Behavior
Fresh DB Applies 419 (visibility) + 420 (oauth_cc) in order. Clean.
Instance that ran old #2800 as 419 `schema_migrations` has 419=`419_oauth_client_credentials.sql`. After rename: on-disk 419=`419_agent_visibility.sql`. The filename-mismatch guard at `migrate.ts:178-199` triggers — but since we also rename 419 on disk, the apply-set presents 419 as agent_visibility (new) and 420 as oauth_cc (new). Wait — the instance already ran oauth_cc as 419. Re-running those columns as 420 re-executes IF NOT EXISTS → no-op. BUT the 419 agent_visibility migration won't have run on that instance, and appliedByVersion.get(419) still returns 419_oauth_client_credentials.sql, which mismatches on-disk 419_agent_visibility.sql — triggering the filename-mismatch throw. Operator fix for these instances: manually update schema_migrations to rename the 419 row's filename to 419_agent_visibility.sql and add a 420 row with filename 420_oauth_client_credentials.sql and today's timestamp. One-time SQL shot.
Instance that ran #2793's 419 (visibility) `schema_migrations` 419=`419_agent_visibility.sql`. Rename matches. Version 420 applies fresh, adds oauth_cc columns. Clean.

The middle case is narrow — only instances that deployed #2800 between its merge and #2793's merge landing them both. Likely small (staging / one prod instance depending on deploy cadence).

Operator instructions for the middle case

If an instance is currently broken because of this, run one-time:

```sql
UPDATE schema_migrations SET filename = '419_agent_visibility.sql' WHERE version = 419;
```

Wait — this is wrong for instances that ran oauth_cc. Actually let me think again. If the instance ran `419_oauth_client_credentials.sql`, then the oauth_cc columns are in place. Now on restart with this fix:

  1. Loader sees 419 (agent_visibility) + 420 (oauth_cc) on disk.
  2. schema_migrations.version=419 filename=419_oauth_client_credentials.sql.
  3. Filename-mismatch guard compares 419_agent_visibility.sql (disk) vs 419_oauth_client_credentials.sql (db) → throws.

Operator fix:
```sql
UPDATE schema_migrations
SET filename = '420_oauth_client_credentials.sql', version = 420
WHERE version = 419
AND filename = '419_oauth_client_credentials.sql';
```

Then on restart: 419 is absent from applied → visibility migration runs fresh. 420 is applied → loader skips. Done.

If this case exists in staging or prod, running the above one-line UPDATE before deploying this PR unblocks it.

Test plan

  • `npm run typecheck` clean
  • Migration filename validation no longer throws
  • `npm run test:server-unit` — full server unit suite passes
  • Verified locally that `docker compose up` proceeds past the migration load step
  • CI `Built migrations against Postgres` confirms fresh-DB apply works end-to-end

Follow-up

The `No duplicate migration numbers` CI check should also run on the merge queue's post-merge state, not just pre-merge per PR. Without that, any two PRs that pick the same number and land minutes apart reproduce this. I'll file a tracker for tightening that check.

🤖 Generated with Claude Code

Main landed two migrations numbered 419: #2793 (agent visibility, first)
and #2800 (OAuth client-credentials, second). Per-PR "No duplicate
migration numbers" CI passed on each because at check-time the other
wasn't on main yet. Merge-queue race.

Every fresh DB boot now fails with:
  Migration filename validation failed:
  Duplicate migration version 419: 419_agent_visibility.sql
  and 419_oauth_client_credentials.sql

Every already-deployed instance fails on next restart.

Fix: rename 419_oauth_client_credentials.sql to 420. The SQL is
idempotent (ADD COLUMN IF NOT EXISTS, DROP VIEW IF EXISTS + CREATE VIEW)
so the rename is safe for all three deployment states:
 - Fresh DBs apply 419+420 in order.
 - Instances that ran the old 419_oauth_client_credentials.sql: 420
   re-runs cleanly, columns/view no-op.
 - Instances that ran #2793's 419_agent_visibility.sql: 420 applies
   fresh.

Verified locally: docker compose up proceeds past the load step.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit 1fa3dc8 into main Apr 22, 2026
12 checks passed
bokelley added a commit that referenced this pull request Apr 22, 2026
…#2816)

Addresses #2815. After the #2813 incident (two PRs reserved migration
419, both passed CI, both merged, broke every fresh boot), close the
two gaps that let it happen.

1. check-migration-numbers.yml PR check was inspecting GitHub's
   speculative merge commit, which goes stale when main moves on in a
   way that doesn't conflict with the PR's diff. Two PRs reserving the
   same number each saw a clean local state and passed.

   Now the PR check takes the UNION of migration filenames on the PR
   branch and on origin/main, then hunts duplicates in that union. Post-
   hoc collision with a migration that landed on main after the PR
   opened is caught on the next PR run. Also drops the paths:
   'migrations/**' filter so a rebase triggers a re-check even when the
   PR itself doesn't touch migrations.

2. deploy.yml has no guard — a broken main push would ship a container
   that crashloops on migration load. Added a preflight job that runs
   the duplicate check on the merged main tree; deploy now needs:
   preflight, so if a collision ever lands it's caught before Fly spins
   up the image, not after.

Simulated locally:
  echo 'union check with collision → exit 1'  →  works
  echo 'union check clean → exit 0'           →  works

Closes #2815.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant