DB reliability: full-inventory drift detection (check:drift), DR rehearsal, expand/contract migration policy

[BigSimmo]

Summary

Database reliability workstream: standing full-inventory drift detection, a rehearsed disaster-recovery runbook, and an expand/contract migration policy for retrieval tables. The live project (sjrfecxgysukkwxsowpy) was read-only throughout — nothing was applied; the two new migrations ship prepared-only.

1. Automated drift detection (npm run check:drift)

Generalizes search_schema_health()'s single-function-hash approach into a comparison of every live function (comment/whitespace-stripped pg_get_functiondef md5 + sorted ACLs), index, RLS policy, table shape (columns sorted by name), constraint, trigger, view, extension, and storage bucket against supabase/schema.sql:

• Migration 20260706200000_schema_drift_snapshot.sql — schema_drift_snapshot(), a service-role-only RPC returning the normalized live inventory (mirrored in schema.sql; byte-parity is test-enforced). Prepared, NOT applied to live — check:drift needs it applied first.
• supabase/drift-manifest.json — expected state, generated by npm run drift:manifest: replays schema.sql from scratch into a disposable supabase/postgres:17.6.1.127 container and snapshots it. Embeds schema.sql's sha256; the freshness test in verify:cheap fails if schema.sql changes without regeneration — which also makes every regeneration a from-scratch replay proof.
• supabase/drift-allowlist.json — 124 measured, classified, reasoned entries for today's known divergence (pending migrations, legacy index estate, grant-posture decision, codification backlog). Anything not listed exits 1; stale entries are reported.
• Noise sources handled by design: whitespace/comment deltas, search_path-dependent rendering (pinned to ''), column ordinal drift, ACL append order, platform-provisioned extensions, legacy index names (machine-verified alias entries), and duplicate migration-history versions (history is never compared — see below for why).
• Offline halves in tests/drift-detection.test.ts (10 tests): manifest freshness, migration↔schema.sql snapshot parity, allowlist hygiene, comparison-engine unit tests.

2. Baseline three-way audit findings (live vs schema.sql replay vs migration-chain replay)

166 divergent keys, fully classified in docs/database-drift-detection.md. Highlights:

• supabase/schema.sql did not replay from scratch (document_index_units declared after its first validating reference). Fixed by reordering.
• 15 worker-written columns existed only on live (document_images ×7, document_index_quality ×6, ingestion_job_stages ×2) — a branch/preview DB restored from the repo had broken ingestion. Codified along with 3 NOT VALID content checks, autovacuum reloptions, 4 live-only functions and 2 triggers in 20260707000000_codify_live_observed_drift.sql (idempotent; byte-aligned with live so applying it there is a no-op).
• 20260703030000 is recorded as applied in live history but its effects are absent — the original incident class, recurring. This is why check:drift compares object state, never history.
• schema.sql text realigned to the migration-chain truth for 4 functions + 6 policies (operand order); 11 remaining function-body drifts are explained by pending migrations (20260705210000, 20260706010000, 20260706130000) and clear on the next approved db push.
• Open decisions surfaced (allowlisted + backlogged): live revoked authenticated Data-API grants on 17 tables (owner-read RLS policies are currently dead on live); 4 security-invoker functions retain PUBLIC execute; ~80 index-estate items.

3. DR rehearsal (docs/disaster-recovery-runbook.md)

• Schema restore measured at ~19 s (container start → scaffold → full schema.sql replay), repeatedly; migration-chain replay 48 s/102 files (needed the new committed storage scaffold scripts/sql/drift-replay-scaffold.sql — the chain is not self-sufficient on bare Postgres either).
• Restored copy verified: search_schema_health() → ok:true; all four hybrid retrieval RPCs + lexical + doc-query paths return seeded vectors correctly (similarity 1.000 / hybrid 0.760).
• Golden retrieval eval documented as the acceptance test for a data restore (needs the real corpus + keys; impossible against schema-only restore — that itself is a documented RPO/RTO finding).
• What did NOT survive restore: data, storage files, auth users, pg_cron schedules, Vault secrets, custom GUCs, edge functions, dashboard config — each with its re-entry step.

4. Migration discipline

Expand/contract policy for retrieval tables written into docs/supabase-migration-reconciliation.md, grounded in what the rehearsal exposed ("history presence is not effect presence"; NOT VALID + VALIDATE; side-by-side RPC versions; drops ship separately and later).

Verification

• npm run verify:cheap — pass (includes the 10 new drift tests; 41/41 existing schema tests still green after the schema.sql reconciliation).
• npm run check:supabase-project — pass (targets sjrfecxgysukkwxsowpy).
• npm run check:indexing — not runnable in this environment (requires SUPABASE_SERVICE_ROLE_KEY + OPENAI_API_KEY, not present on this machine); its DB-side core was exercised directly: search_schema_health() returned ok:true on live (via MCP, read-only) and on both replay containers.
• npm run drift:manifest — run end-to-end (its output is the committed manifest).
• Comparison engine validated against the migration-chain snapshot as pseudo-live (finds exactly the 13 known chain-vs-schema.sql fidelity gaps).
• Clinical governance preflight: no ingestion/answer-generation/search-ranking/clinical-output behavior changes; retrieval RPC texts in schema.sql were aligned to what live already runs, and the golden-eval gate applies to the (separate, approval-gated) live applies, not to this PR.

Operator follow-ups (need your explicit approval — nothing applied)

1. supabase db push the pending stack: 20260705210000, 20260706010000, 20260706130000, 20260706200000 (enables check:drift), 20260707000000.
2. Re-apply 20260703030000's statements under a new version (recorded-but-ineffective).
3. Then run npm run check:drift and start burning down the allowlist per the backlog in docs/database-drift-detection.md.

🤖 Generated with Claude Code

[supabase]

Updates to Preview Branch (claude/db-reliability) ↗︎

Deployments	Status	Updated
Database	✅	Tue, 07 Jul 2026 03:20:16 UTC
Services	✅	Tue, 07 Jul 2026 03:20:16 UTC
APIs	✅	Tue, 07 Jul 2026 03:20:16 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks	Status	Updated
Configurations	✅	Tue, 07 Jul 2026 03:20:17 UTC
Migrations	✅	Tue, 07 Jul 2026 03:20:19 UTC
Seeding	✅	Tue, 07 Jul 2026 03:20:20 UTC
Edge Functions	✅	Tue, 07 Jul 2026 03:20:21 UTC

---

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.