Skip to content

DB reliability: full-inventory drift detection (check:drift), DR rehearsal, expand/contract migration policy #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 3 commits into from
Jul 7, 2026
Merged

DB reliability: full-inventory drift detection (check:drift), DR rehearsal, expand/contract migration policy #342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eee3112
Add full-inventory schema drift detection, DR rehearsal, and migratio…
BigSimmo Jul 6, 2026
dd672bf
Merge remote-tracking branch 'origin/main' into claude/db-reliability
cursoragent Jul 7, 2026
e3cea6f
ci: retrigger checks after billing fix
cursoragent Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ scratch/
# Generated by `supabase gen types`; keep the generator's formatting so
# regeneration stays churn-free.
src/lib/supabase/database.types.ts
supabase/drift-manifest.json
134 changes: 134 additions & 0 deletions docs/database-drift-detection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Database drift detection (`npm run check:drift`)

Last updated: 2026-07-07

This repo's worst operational incidents were live-vs-repo schema drift: hybrid
retrieval RPCs silently broken on live for an unknown period, and migrations
recorded as applied whose objects were absent. `search_schema_health()` guards
a curated subset (signatures, 22 required indexes, execution smoke).
`check:drift` generalizes that into a full-inventory comparison of **every**
application-owned object against `supabase/schema.sql`.

## How it works

Three committed artifacts:

| Artifact | Role |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `supabase/migrations/20260706200000_schema_drift_snapshot.sql` | `public.schema_drift_snapshot()` — service-role-only RPC returning the normalized live inventory (also declared in `supabase/schema.sql`; a test enforces byte parity). |
| `supabase/drift-manifest.json` | The expected state: the same snapshot captured from a **from-scratch replay of `supabase/schema.sql`** into a disposable `supabase/postgres` Docker container (`npm run drift:manifest`). Embeds the sha256 of the schema.sql it came from. |
| `supabase/drift-allowlist.json` | Known, documented divergence (each entry has a `reason`). Reported as warnings; anything not listed fails the check. |

`npm run check:drift` (needs live service-role env) verifies the project ref,
fails fast if the manifest is stale, calls the RPC, diffs, applies the
allowlist, and exits 1 on unallowlisted divergence. The offline half runs in
`tests/drift-detection.test.ts` under `verify:cheap`: manifest freshness
(sha256), migration↔schema.sql parity for the snapshot function, allowlist
hygiene, and unit tests of the comparison engine.

Inventory coverage: functions (comment/whitespace-stripped `pg_get_functiondef`
md5 + sorted ACLs), indexes (normalized `pg_get_indexdef`), RLS policies
(permissive/roles/cmd/qual/with_check), table shapes (columns sorted by name,
RLS flags, reloptions, ACLs), constraints, triggers, views, extensions, and
storage bucket rows + storage.objects policies.

### Noise sources handled by design

- **Whitespace/comments in function bodies** — `prosrc` is stored verbatim, so
migration text vs schema.sql text differ trivially; both are stripped before
hashing (the same trick `20260701140631` used to validate byte-equivalence).
- **Rendering search_path** — the snapshot pins `search_path = ''` so
`pg_get_expr`/`pg_get_indexdef`/policy quals render fully qualified and
identically on live and replay.
- **Column ordinal drift** — live tables grew via `ALTER TABLE ADD COLUMN`;
columns compare sorted by name, not `attnum`.
- **ACL append order** — aclitem arrays are sorted.
- **Duplicate migration-history versions** — history is _not_ compared at all;
the check compares actual object state (history presence proved unreliable:
see `20260703030000` below).
- **Platform-provisioned extensions** (pg_net, pgsodium, pgmq, …) — extra live
extensions are informational; missing schema.sql-declared ones fail.
- **Legacy index names** — `alias` allowlist entries assert the live database
carries the _identical_ name-stripped index definition under a legacy name
(the machine-checked version of `search_schema_health()`'s `index_aliases`).

### Workflow

- Change `supabase/schema.sql` → run `npm run drift:manifest` (Docker) in the
same PR. The freshness test fails otherwise. This also continuously proves
schema.sql replays from scratch — which it did **not** before 2026-07-07
(`document_index_units` was declared after its first validating reference).
- Live drifts (check:drift red) → either codify live state (migration +
schema.sql + manifest regen) or fix live **through an approved migration**.
Never raw SQL against live; that is how this incident class started.
- New known-divergence → allowlist entry with a reason and a backlog line here.
- After the pending-migration backlog lands, delete the matching allowlist
entries; check:drift reports stale entries so they cannot silently linger.

## 2026-07-07 baseline audit (three-way: live vs schema.sql vs migration chain)

Both repo lineages were replayed into scratch containers and compared with the
live inventory. 166 divergent keys, fully classified:

- **Reconciled in this PR (schema.sql/migrations only, live untouched):**
replay-order fix; `20260707000000_codify_live_observed_drift.sql` codifying
15 live-only columns (`document_images` ×7, `document_index_quality` ×6,
`ingestion_job_stages` ×2 — worker-written, branch DBs broke without them),
3 `content_not_blank` NOT VALID checks, autovacuum reloptions on 5 RAG
tables, `content_hash` nullability alignment, 4 live-only functions
(`set_owner_id_from_auth_uid` + rag_queries/misses triggers,
`purge_expired_rag_queries`, `correct_clinical_query_terms`,
`invoke_ingestion_worker`) and 2 ACL tightenings; schema.sql function/policy
text realigned to the migration-chain truth for `analyze_rag_tables`,
`claim_indexing_v3_agent_jobs`, `is_committed_artifact_generation(uuid,jsonb)`,
`match_document_memory_cards_hybrid`, and 6 owner-read policies (operand
order only).
- **Allowlisted (124 entries)** — see `supabase/drift-allowlist.json`; backlog
below.

## Reconciliation backlog

Ordered; each item removes allowlist entries when it lands. Items touching the
live project need explicit operator approval.

1. **Apply the pending migrations** (`supabase db push` after review):
`20260705210000` (owner-sentinel rewrite — 8 live function bodies stale),
`20260706010000` (search_schema_health M13 guard), `20260706130000`
(embedding-fields-text sentinel), `20260706200000` (drift snapshot RPC —
required before check:drift can run at all), `20260707000000` (codification
wave; no-op on live by construction).
2. **`20260703030000` is recorded as applied but its effects are absent on
live** (storage_cleanup_jobs still has the legacy index names and the
non-partial status index). Recorded-but-ineffective is the exact original
incident class, recurring. Re-apply its idempotent statements under a new
version with approval; do not trust history presence.
3. **Codify the remaining live-only functions**: `get_visual_evidence_cards`,
`repair_enrichment_quality_batch`, `run_all_visual_eval_cases`,
`run_visual_eval_case` (same pattern as `20260707000000`).
4. **Authenticated-grant posture decision**: live revoked the authenticated
Data API grants on 17 tables (fail-closed hardening; the owner-read RLS
policies are currently dead on live) while schema.sql still declares them.
Either codify the revokes (schema.sql + tests + migration) or restore the
grants live.
5. **PUBLIC-execute revokes**: 4 security-invoker functions retain default
PUBLIC execute on live (`detect_legacy_ivfflat_indexes`,
`document_summary_text`, `search_document_chunks`,
`set_document_embedding_field_content_hash`).
6. **`document_label_metadata` direction**: schema.sql is AHEAD (hidden-label
filtering added without a migration). Ship the migration or revert.
7. **Index estate**: rename 10 legacy-named live indexes to schema.sql names;
decide the 24 schema.sql-declared indexes absent on live (recreate vs
remove — includes `documents_search_idx`, `document_chunks_anchor_idx`,
`documents_owner_content_hash_unique_idx`); drop ~45 live-only duplicate
indexes after `pg_stat_user_indexes` scan verification; reshape 3
(`import_batches_status_created_idx`, `ingestion_jobs_document_status_idx`,
`ingestion_jobs_status_next_run_idx`).
8. **Constraints**: add `ingestion_job_stages_job_id_fkey` to live; align the
`rag_visual_eval_*` document FK definitions.
9. **`invoke_ingestion_worker`** hardcodes the project URL — migrate to the
GUC pattern (`20260702160000` precedent).
10. **Migration-chain fidelity** (affects Supabase Preview/branches, not
live): 13 keys where the chain diverges from schema.sql — buckets are only
created by schema.sql, `documents`/`ingestion_jobs` updated_at trigger
variants, post-legacy-drop embedding-fields index set,
`document_chunks_content_trgm_idx` shape, `rag_visual_eval_*` shapes.
126 changes: 126 additions & 0 deletions docs/disaster-recovery-runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Disaster recovery runbook — Clinical KB Database

Last rehearsed: 2026-07-07 (schema restore rehearsal against a local Supabase
Postgres container; the live project `sjrfecxgysukkwxsowpy` was read-only
throughout).

## What a recovery is made of

A full recovery of this system has four independent layers. **Only the first
is covered by the repo**; the rest are Supabase-platform or operator state:

| Layer | Source of truth | RPO | RTO (measured/estimated) |
| --------------------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Schema (tables, RPCs, indexes, RLS, buckets config) | `supabase/schema.sql` in git | 0 (git) | **~19 s** into a local container (measured, repeatedly); minutes on a hosted project |
| Data (documents, 69k+ chunks, embeddings, jobs, logs) | Supabase managed backups | Daily backups → up to 24 h; with PITR enabled → ~2 min granularity. **Check the dashboard (Database → Backups) for the actual plan setting.** | Platform restore; not directly measurable without executing one. For the ~8.6 GB database expect tens of minutes to hours. Practice target: restore to a **new** project, never in place. |
| Storage objects (`clinical-documents`, `clinical-images` files) | Supabase Storage (S3) — separate from DB backups | Platform-managed | Bucket **rows** are recreated by schema.sql; the **files** are not in a DB restore. A DB-only restore leaves `documents.storage_path` pointing at objects that must still exist in Storage. |
| Config & secrets | Nowhere in the repo (deliberate) | n/a | Manual re-entry, see the checklist below |

## Schema restore procedure (rehearsed)

Works on any machine with Docker; identical mechanics on a fresh hosted
project via psql. This is exactly what `npm run drift:manifest` automates
(container start → scaffold → replay → snapshot → destroy), so **schema
restorability is re-proven every time the drift manifest is regenerated**.

```sh
# 1. Scratch Supabase Postgres matching live (17.6.1.127)
docker run -d --name kb-restore -e POSTGRES_PASSWORD=postgres -p 56543:5432 supabase/postgres:17.6.1.127
docker exec kb-restore pg_isready -U postgres # wait until ready

# 2. Storage scaffold (bare image ships an empty storage schema; the hosted
# platform provisions the real one). Run as supabase_admin.
docker cp scripts/sql/drift-replay-scaffold.sql kb-restore:/tmp/scaffold.sql
docker exec kb-restore psql -U supabase_admin -d postgres -v ON_ERROR_STOP=1 -f /tmp/scaffold.sql

# 3. Replay the canonical schema as postgres (matches how live is administered)
docker cp supabase/schema.sql kb-restore:/tmp/schema.sql
docker exec kb-restore psql -U postgres -d postgres -v ON_ERROR_STOP=1 -f /tmp/schema.sql

# 4. Verify
docker exec kb-restore psql -U postgres -d postgres -tAc "select (public.search_schema_health())->>'ok'" # must print: true
```

On a hosted target, prefer the migration chain (`supabase db push` /
`supabase db reset --linked` semantics); measured chain replay: **48 s** for
102 migrations into the same container. Note the chain has known fidelity gaps
vs schema.sql (buckets, two triggers, a handful of indexes — item 10 in
[database-drift-detection.md](database-drift-detection.md#reconciliation-backlog)).

## Retrieval verification on the restored copy (rehearsed 2026-07-07)

1. `search_schema_health()` → `ok: true, missing: []` on the restored copy.
2. `schema_drift_snapshot()` → full inventory captured; this became
`supabase/drift-manifest.json`.
3. Seeded a synthetic document + chunk + embedding field + index unit + memory
card (1536-dim unit vectors) and invoked the full retrieval surface:
`match_document_chunks_hybrid`, `match_document_embedding_fields_hybrid`,
`match_document_index_units_hybrid`, `match_document_memory_cards_hybrid`,
`match_document_chunks_text`, `match_documents_for_query`. All returned the
seeded rows (similarity 1.000 on the identity vector, hybrid score 0.760) —
the lexical, vector, and hybrid scoring paths all execute correctly on a
from-scratch restore.
4. **Golden retrieval eval (`npm run eval:retrieval:quality`) cannot run
against a schema-only restore** — it needs the real ~2,065-document corpus
plus `OPENAI_API_KEY` and service-role env. After a _data_ restore, point
`.env.local` at the restored project (update `NEXT_PUBLIC_SUPABASE_URL`,
`SUPABASE_PROJECT_REF`, keys), run `npm run check:supabase-project`, then
`npm run eval:retrieval:quality` and require the 23/23 pass before serving
traffic. This is the acceptance test for a real recovery.

## What did NOT survive the schema restore (verified)

Everything below must come from a data restore or manual re-entry — plan for
it before you need it:

- **All data** — documents, chunks, embeddings (re-embedding the corpus from
scratch costs real OpenAI spend and hours of worker time; the backup is the
only cheap path), jobs, caches, logs, registry records.
- **Storage object files** — schema.sql recreates the two private bucket rows
(`clinical-documents`, `clinical-images`) but no files.
- **Auth users** — `auth.users` is platform state; owner-scoped rows restored
from backup reference user ids that must exist again (same project restore
preserves them; a new project does not).
- **12 platform-provisioned extensions** (pg_net, pgsodium, pgmq, pg_cron,
pg_graphql, vault, …) — present on hosted projects, absent in the bare
image; schema.sql only declares vector/pg_trgm/uuid-ossp.
- **pg_cron schedules** — the invoked functions (`invoke_ingestion_worker`,
`invoke_indexing_v3_agent`) are codified, but the `cron.schedule(...)` rows
themselves are live-only. After restore, re-create the cron jobs.
- **Vault secrets** — `cron_ingestion_jwt` (and any siblings) must be re-added
before the cron→edge-function chain works.
- **Custom GUCs** — `20260702160000` reads the agent URL from a database GUC;
re-set it (`alter database ... set ...`) on the restored project.
- **Edge functions** — deploy `indexing-v3-agent` (and the ingestion worker
function) separately via the CLI.
- **Dashboard config** — auth providers (magic link, Google/Microsoft SSO
redirect URLs), connection-pool caps (the documented 10-connection auth cap
is dashboard-only), API keys (publishable + service role are per-project;
every consumer needs the new values), `E2E_USER_*` test accounts.
- **Role settings** — e.g. `alter role authenticator set
idle_in_transaction_session_timeout` (in the migration chain, so a chain
replay restores it; a schema.sql-only replay does not).

## Rehearsal findings that changed the repo

- `supabase/schema.sql` did **not** replay from scratch before 2026-07-07
(`document_index_units` was created ~900 lines after its first validating
reference — the strict-gate view). Fixed by reordering; now continuously
re-proven by `npm run drift:manifest` + the manifest freshness test.
- The bare `supabase/postgres` image lacks the storage schema objects; the
committed scaffold `scripts/sql/drift-replay-scaffold.sql` fills the gap.
- The first migration (`20260527000000`) dies at its storage-policy section
without that scaffold — the chain is **not** self-sufficient on a bare
Postgres either.
- Worker-written columns existed only on live (see
`20260707000000_codify_live_observed_drift.sql`): a branch/preview database
restored from the repo would have broken ingestion writes. Codified.

## Standing cadence

- Every `drift:manifest` regeneration = a schema-restore rehearsal (free).
- Run `npm run check:drift` after any live apply and on the operational
cadence alongside `check:indexing`.
- Re-rehearse the **data** layer (platform restore to a scratch project +
golden eval) before any risky bulk operation (re-index, mass migration), and
record the measured restore time here.
Loading
Loading