Staging-aware Supabase identity guard + staging setup runbook

docs/staging-setup.md

@@ -0,0 +1,115 @@
+# Staging Environment Setup
+
+Turnkey runbook for standing up a staging environment. Companion to
+`docs/deployment-architecture.md` §5 (why staging is a dedicated project, not a
+prod branch) and `docs/capacity-review.md` §4 (the soak test that validates it).
+
+Staging is two independent tiers: a **staging Supabase project** (data) and a
+**staging app host** (compute). Do the data tier first — the app needs it.
+
+The identity guard is already staging-aware (`src/lib/supabase/project.ts`): it
+accepts a second project **only** when you explicitly declare it via
+`SUPABASE_STAGING_PROJECT_REF` + `SUPABASE_STAGING_PROJECT_NAME`, and it refuses
+a staging ref that collides with the production or stale ref. So no code change
+is needed to activate staging — just env. Production behavior is unchanged when
+those vars are unset.
+
+## A. Staging Supabase project (data tier)
+
+1. **Create the project.** Same org (`BigSimmo's Org`), region
+   **ap-southeast-2 (Sydney)** — a _separate project_, not a branch of
+   production. Name it e.g. `Clinical KB Staging`. (Can be done via the
+   Supabase MCP `create_project` after a `confirm_cost` step — a new project is
+   ~$10/month — or from the dashboard.) Record the new project ref
+   (`<staging-ref>`) and generate a DB password.
+
+2. **Apply the schema.** From a checkout linked to the staging project:
+
+   ```bash
+   supabase link --project-ref <staging-ref>
+   supabase db push          # applies supabase/migrations/* → matches schema.sql
+   ```
+
+   Then confirm health: `npm run check:indexing` (runs `search_schema_health()`
+   over the hybrid RPCs) should report ok.
+
+3. **Seed a small corpus (~50 docs).** Use synthetic/public content only — do
+   **not** copy clinical production documents into staging.
+
+   ```bash
+   npm run samples                 # generate synthetic sample documents
+   npm run import:docs             # queue them for ingestion
+   # (run the worker — section B4 — to process the queue)
+   npm run registry:seed -- --owner-id <owner> --write --confirm
+   npm run differentials:seed
+   npm run medications:seed
+   ```
+
+4. **Capture the keys** for the staging project (dashboard → API): the
+   publishable key (`sb_publishable_…`) and the service-role secret
+   (`sb_secret_…`).
+
+## B. Staging app host (compute tier)
+
+Recommended host: **Fly.io `syd`** (Sydney region; no MCP available here, so
+this is an operator step). Google Cloud Run `australia-southeast2` is the
+equivalent. Railway is unsuitable — no Sydney region.
+
+1. **Build the image** from the committed `Dockerfile`, passing the _staging_
+   publishable key (it inlines into the client bundle):
+
+   ```bash
+   docker build \
+     --build-arg NEXT_PUBLIC_SUPABASE_URL=https://<staging-ref>.supabase.co \
+     --build-arg NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=<staging sb_publishable_…> \
+     -t clinical-kb-app:staging .
+   ```
+
+   Note: local Docker Desktop can OOM on the 8 GiB Next build heap; prefer the
+   CI image-build workflow (builds on GitHub runners) if the local build wedges.
+
+2. **Runtime secrets** (injected at deploy, never baked into the image) — all
+   with **staging** values, distinct from production:
+
+   | Var                             | Value                             |
+   | ------------------------------- | --------------------------------- |
+   | `SUPABASE_SERVICE_ROLE_KEY`     | staging `sb_secret_…`             |
+   | `OPENAI_API_KEY`                | an OpenAI key (staging or shared) |
+   | `SUPABASE_PROJECT_REF`          | `<staging-ref>`                   |
+   | `SUPABASE_PROJECT_NAME`         | `Clinical KB Staging`             |
+   | `SUPABASE_STAGING_PROJECT_REF`  | `<staging-ref>`                   |
+   | `SUPABASE_STAGING_PROJECT_NAME` | `Clinical KB Staging`             |
+   | `RAG_QUERY_HASH_SECRET`         | a staging secret                  |
+   | `RAG_PROVIDER_MODE`             | `auto`                            |
+
+   The two `SUPABASE_STAGING_PROJECT_*` vars are what make the identity guard
+   accept the staging project. Setting `NEXT_PUBLIC_SUPABASE_URL` +
+   `SUPABASE_PROJECT_REF` to the staging ref **without** them will (correctly)
+   fail `check:supabase-project` — that's the deliberate speed bump.
+
+3. **Host config:** bind `0.0.0.0:$PORT` (the Dockerfile already does),
+   health check `/api/health`, `min_machines_running=1`, no scale-to-zero.
+
+4. **Worker (optional, for ingestion in staging):** build `Dockerfile.worker`
+   and run one instance with the same staging secrets. Required to process the
+   seed queue from A3.
+
+## C. Validate staging
+
+1. Boot check: `GET /api/health` → `{"status":"ok"}` with the staging project.
+2. Load check — the soak test is hard-guarded against production:
+
+   ```bash
+   npx tsx scripts/soak-test.ts --target https://<staging-host> \
+     --confirm-staging --users 30 --duration-s 600 --ramp-s 120
+   ```
+
+   Success targets are in `docs/capacity-review.md` §4 (search p95 ≤ 3 s,
+   answer p95 ≤ 25 s, non-429 error rate < 1 %).
+
+## What is operator-only (cannot be scripted here)
+
+- Creating the Supabase project (billable) and its DB password.
+- Opening the Fly.io/host account and setting the runtime secrets.
+- Any change to the **production** project's settings (e.g. auth
+  percentage-based connection allocation — see `docs/capacity-review.md` §3).

scripts/check-supabase-project.ts

@@ -1,9 +1,5 @@
 import { loadEnvConfig } from "@next/env";
-import {
-  checkSupabaseProjectConfig,
-  expectedSupabaseProject,
-  formatSupabaseProjectCheck,
-} from "@/lib/supabase/project";
+import { checkSupabaseProjectConfig, formatSupabaseProjectCheck } from "@/lib/supabase/project";
 
 loadEnvConfig(process.cwd());
 
@@ -12,12 +8,16 @@ const check = checkSupabaseProjectConfig(
     NEXT_PUBLIC_SUPABASE_URL: process.env.NEXT_PUBLIC_SUPABASE_URL,
     SUPABASE_PROJECT_REF: process.env.SUPABASE_PROJECT_REF,
     SUPABASE_PROJECT_NAME: process.env.SUPABASE_PROJECT_NAME,
+    SUPABASE_STAGING_PROJECT_REF: process.env.SUPABASE_STAGING_PROJECT_REF,
+    SUPABASE_STAGING_PROJECT_NAME: process.env.SUPABASE_STAGING_PROJECT_NAME,
   },
   { requireMetadata: true },
 );
 
-console.log(`Expected Supabase project: ${expectedSupabaseProject.name} (${expectedSupabaseProject.ref})`);
-console.log(`Expected Supabase URL: ${expectedSupabaseProject.url}`);
+console.log(
+  `Expected Supabase project: ${check.expected.name} (${check.expected.ref}) [${check.observed.environment}]`,
+);
+console.log(`Expected Supabase URL: ${check.expected.url}`);
 console.log(`Configured URL ref: ${check.observed.urlRef ?? "not set or not recognized"}`);
 console.log(`Configured SUPABASE_PROJECT_REF: ${check.observed.configuredRef ?? "not set"}`);
 console.log(`Configured SUPABASE_PROJECT_NAME: ${check.observed.configuredName ?? "not set"}`);

src/lib/env.ts

@@ -6,6 +6,11 @@ const envSchema = z.object({
   NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY: z.string().optional(),
   SUPABASE_PROJECT_REF: z.string().optional(),
   SUPABASE_PROJECT_NAME: z.string().optional(),
+  // Optional: declares a second accepted (staging) Supabase project so the
+  // identity guard accepts it. Both must be set; the ref must differ from
+  // production. See docs/staging-setup.md and src/lib/supabase/project.ts.
+  SUPABASE_STAGING_PROJECT_REF: z.string().optional(),
+  SUPABASE_STAGING_PROJECT_NAME: z.string().optional(),
   SUPABASE_SERVICE_ROLE_KEY: z.string().optional(),
   SUPABASE_DB_URL: z.string().url().optional(),
   HEALTH_DEEP_PROBE_SECRET: z.string().min(16).optional(),

src/lib/supabase/project.ts

@@ -13,10 +13,23 @@ export const staleSupabaseProjects = [
   },
 ] as const;
 
+export type ExpectedSupabaseProject = {
+  name: string;
+  ref: string;
+  url: string;
+  region: string;
+};
+
 export type SupabaseProjectConfig = {
   NEXT_PUBLIC_SUPABASE_URL?: string | null;
   SUPABASE_PROJECT_REF?: string | null;
   SUPABASE_PROJECT_NAME?: string | null;
+  // Staging is declared explicitly, never inferred. Both must be set to enable
+  // a second accepted project; the ref must be a valid Supabase ref that is
+  // NOT the production or a stale ref (that would be the silent-point-at-prod
+  // footgun this guard exists to prevent). See docs/staging-setup.md.
+  SUPABASE_STAGING_PROJECT_REF?: string | null;
+  SUPABASE_STAGING_PROJECT_NAME?: string | null;
 };
 
 export type SupabaseProjectCheckStatus = "ready" | "missing" | "mismatch" | "warning";
@@ -27,12 +40,13 @@ type SupabaseProjectCheckOptions = {
 
 export type SupabaseProjectCheck = {
   status: SupabaseProjectCheckStatus;
-  expected: typeof expectedSupabaseProject;
+  expected: ExpectedSupabaseProject;
   observed: {
     url: string | null;
     urlRef: string | null;
     configuredRef: string | null;
     configuredName: string | null;
+    environment: "production" | "staging";
   };
   staleProject: (typeof staleSupabaseProjects)[number] | null;
   problems: string[];
@@ -63,6 +77,45 @@ function unique(values: Array<string | null>) {
   return [...new Set(values.filter((value): value is string => Boolean(value)))];
 }
 
+const reservedProjectRefs = new Set<string>([
+  expectedSupabaseProject.ref,
+  ...staleSupabaseProjects.map((project) => project.ref),
+]);
+
+/**
+ * Resolve an optional staging project declared via env. Returns the project
+ * when the declaration is valid, or a problem string when staging vars are
+ * partially/incorrectly set (so a broken staging declaration fails loud rather
+ * than silently falling through to the production guard).
+ */
+function resolveStagingProject(config: SupabaseProjectConfig): {
+  project: ExpectedSupabaseProject | null;
+  problem: string | null;
+} {
+  const stagingRef = trimmed(config.SUPABASE_STAGING_PROJECT_REF);
+  const stagingName = trimmed(config.SUPABASE_STAGING_PROJECT_NAME);
+  if (!stagingRef && !stagingName) return { project: null, problem: null };
+  if (!stagingRef || !stagingName) {
+    return {
+      project: null,
+      problem: "Set BOTH SUPABASE_STAGING_PROJECT_REF and SUPABASE_STAGING_PROJECT_NAME to enable the staging project.",
+    };
+  }
+  if (!/^[a-z0-9]{20}$/.test(stagingRef)) {
+    return { project: null, problem: `SUPABASE_STAGING_PROJECT_REF "${stagingRef}" is not a valid Supabase ref.` };
+  }
+  if (reservedProjectRefs.has(stagingRef)) {
+    return {
+      project: null,
+      problem: `SUPABASE_STAGING_PROJECT_REF ${stagingRef} collides with the production/stale project; staging must be a distinct project.`,
+    };
+  }
+  return {
+    project: { name: stagingName, ref: stagingRef, url: `https://${stagingRef}.supabase.co`, region: "ap-southeast-2" },
+    problem: null,
+  };
+}
+
 export function checkSupabaseProjectConfig(
   config: SupabaseProjectConfig,
   options: SupabaseProjectCheckOptions = {},
@@ -76,33 +129,42 @@ export function checkSupabaseProjectConfig(
   const problems: string[] = [];
   const warnings: string[] = [];
 
+  const { project: stagingProject, problem: stagingProblem } = resolveStagingProject(config);
+  // Pick which accepted project this config is targeting. Staging is only
+  // matched when the observed ref equals the explicitly-declared staging ref;
+  // everything else resolves to production, so production behavior is
+  // unchanged when no staging project is declared.
+  const expected: ExpectedSupabaseProject =
+    stagingProject && observedRefs.includes(stagingProject.ref) ? stagingProject : expectedSupabaseProject;
+  const environment: "production" | "staging" = expected === stagingProject ? "staging" : "production";
+
+  if (stagingProblem) problems.push(stagingProblem);
+
   if (!url) {
     return {
       status: "missing",
-      expected: expectedSupabaseProject,
-      observed: { url, urlRef, configuredRef, configuredName },
+      expected,
+      observed: { url, urlRef, configuredRef, configuredName, environment },
       staleProject,
       problems,
       warnings,
     };
   }
 
   if (!urlRef) {
-    problems.push(`NEXT_PUBLIC_SUPABASE_URL must be a Supabase project URL for ${expectedSupabaseProject.name}.`);
-  } else if (urlRef !== expectedSupabaseProject.ref) {
+    problems.push(`NEXT_PUBLIC_SUPABASE_URL must be a Supabase project URL for ${expected.name}.`);
+  } else if (urlRef !== expected.ref) {
     problems.push(
-      `NEXT_PUBLIC_SUPABASE_URL points to Supabase ref ${urlRef}; expected ${expectedSupabaseProject.ref} (${expectedSupabaseProject.name}).`,
+      `NEXT_PUBLIC_SUPABASE_URL points to Supabase ref ${urlRef}; expected ${expected.ref} (${expected.name}).`,
     );
   }
 
-  if (configuredRef && configuredRef !== expectedSupabaseProject.ref) {
-    problems.push(
-      `SUPABASE_PROJECT_REF is ${configuredRef}; expected ${expectedSupabaseProject.ref} (${expectedSupabaseProject.name}).`,
-    );
+  if (configuredRef && configuredRef !== expected.ref) {
+    problems.push(`SUPABASE_PROJECT_REF is ${configuredRef}; expected ${expected.ref} (${expected.name}).`);
   }
 
-  if (configuredName && configuredName !== expectedSupabaseProject.name) {
-    problems.push(`SUPABASE_PROJECT_NAME is "${configuredName}"; expected "${expectedSupabaseProject.name}".`);
+  if (configuredName && configuredName !== expected.name) {
+    problems.push(`SUPABASE_PROJECT_NAME is "${configuredName}"; expected "${expected.name}".`);
   }
 
   if (staleProject) {
@@ -111,19 +173,19 @@ export function checkSupabaseProjectConfig(
     );
   }
 
-  if (options.requireMetadata && urlRef === expectedSupabaseProject.ref && problems.length === 0) {
+  if (options.requireMetadata && urlRef === expected.ref && problems.length === 0) {
     if (!configuredRef) {
-      warnings.push(`Set SUPABASE_PROJECT_REF=${expectedSupabaseProject.ref} in .env.local.`);
+      warnings.push(`Set SUPABASE_PROJECT_REF=${expected.ref} in .env.local.`);
     }
     if (!configuredName) {
-      warnings.push(`Set SUPABASE_PROJECT_NAME="${expectedSupabaseProject.name}" in .env.local.`);
+      warnings.push(`Set SUPABASE_PROJECT_NAME="${expected.name}" in .env.local.`);
     }
   }
 
   return {
     status: problems.length > 0 ? "mismatch" : warnings.length > 0 ? "warning" : "ready",
-    expected: expectedSupabaseProject,
-    observed: { url, urlRef, configuredRef, configuredName },
+    expected,
+    observed: { url, urlRef, configuredRef, configuredName, environment },
     staleProject,
     problems,
     warnings,

tests/supabase-project.test.ts

@@ -62,6 +62,71 @@ describe("Supabase project guard", () => {
     expect(formatSupabaseProjectCheck(check)).toContain("older unused project");
   });
 
+  const stagingRef = "abcdefghijklmnopqrst";
+  const stagingProject = {
+    url: `https://${stagingRef}.supabase.co`,
+    ref: stagingRef,
+    name: "Clinical KB Staging",
+  };
+
+  it("accepts an explicitly declared staging project", () => {
+    const check = checkSupabaseProjectConfig(
+      {
+        NEXT_PUBLIC_SUPABASE_URL: stagingProject.url,
+        SUPABASE_PROJECT_REF: stagingProject.ref,
+        SUPABASE_PROJECT_NAME: stagingProject.name,
+        SUPABASE_STAGING_PROJECT_REF: stagingProject.ref,
+        SUPABASE_STAGING_PROJECT_NAME: stagingProject.name,
+      },
+      { requireMetadata: true },
+    );
+
+    expect(check.status).toBe("ready");
+    expect(check.observed.environment).toBe("staging");
+    expect(check.expected.ref).toBe(stagingProject.ref);
+    expect(formatSupabaseProjectCheck(check)).toContain(stagingProject.name);
+  });
+
+  it("keeps production behavior unchanged even when a staging project is declared", () => {
+    const check = checkSupabaseProjectConfig(
+      {
+        NEXT_PUBLIC_SUPABASE_URL: expectedSupabaseProject.url,
+        SUPABASE_PROJECT_REF: expectedSupabaseProject.ref,
+        SUPABASE_PROJECT_NAME: expectedSupabaseProject.name,
+        SUPABASE_STAGING_PROJECT_REF: stagingProject.ref,
+        SUPABASE_STAGING_PROJECT_NAME: stagingProject.name,
+      },
+      { requireMetadata: true },
+    );
+
+    expect(check.status).toBe("ready");
+    expect(check.observed.environment).toBe("production");
+    expect(check.expected.ref).toBe(expectedSupabaseProject.ref);
+  });
+
+  it("rejects a staging declaration that collides with the production ref (silent-point-at-prod footgun)", () => {
+    const check = checkSupabaseProjectConfig({
+      NEXT_PUBLIC_SUPABASE_URL: expectedSupabaseProject.url,
+      SUPABASE_STAGING_PROJECT_REF: expectedSupabaseProject.ref,
+      SUPABASE_STAGING_PROJECT_NAME: "Clinical KB Staging",
+    });
+
+    expect(check.status).toBe("mismatch");
+    expect(check.problems.join(" ")).toContain("collides with the production");
+  });
+
+  it("rejects a partial staging declaration (only one of the two vars set)", () => {
+    const check = checkSupabaseProjectConfig({
+      NEXT_PUBLIC_SUPABASE_URL: expectedSupabaseProject.url,
+      SUPABASE_PROJECT_REF: expectedSupabaseProject.ref,
+      SUPABASE_PROJECT_NAME: expectedSupabaseProject.name,
+      SUPABASE_STAGING_PROJECT_REF: stagingRef,
+    });
+
+    expect(check.status).toBe("mismatch");
+    expect(check.problems.join(" ")).toContain("BOTH");
+  });
+
   it("blocks server env when configured for a stale project ref", async () => {
     vi.stubEnv("NEXT_PUBLIC_SUPABASE_URL", staleProject.url);
     vi.stubEnv("SUPABASE_PROJECT_REF", staleProject.ref);