fix: harden Railway operations and staging checks

.github/workflows/ops-digest.yml

@@ -2,18 +2,16 @@
 # hit-rate, answer spend, degraded/truncation rates) into a one-screen summary
 # and keeps it in a rolling GitHub issue, commenting only when something is off.
 #
-# SHIPPED DISABLED: only workflow_dispatch is active. To enable the daily cadence:
-#   1. Add repo variable  PROD_HEALTH_URL          (e.g. https://<app-host>)
-#   2. Add repo secret     HEALTH_DEEP_PROBE_SECRET (same value as the deployment)
-#   3. Uncomment the `schedule:` block below.
-# Run one workflow_dispatch first and confirm the digest issue looks right.
+# The repo variable and matching deployment/GitHub secret are provisioned.
+# Keep workflow_dispatch for operator verification and the daily schedule for
+# the normal morning digest.
 name: Ops Digest
 
 on:
   workflow_dispatch: {}
-  # schedule:
-  #   # 20:00 UTC = 06:00 Australia/Sydney — a morning digest.
-  #   - cron: "0 20 * * *"
+  schedule:
+    # 20:00 UTC = 06:00 Australia/Sydney — a morning digest.
+    - cron: "0 20 * * *"
 
 concurrency:
   group: ops-digest

docs/deployment-architecture.md

@@ -232,7 +232,9 @@ check and watch patterns rather than relying on dashboard defaults.
 Ship the existing worker as a container (`Dockerfile.worker`: Node 24 + a
 prebuilt esbuild bundle over production-only `node_modules` +
 Tesseract + a Python venv with `worker/python/requirements.txt`) and run **one
-always-on worker instance** co-located in Railway Singapore (`worker` service).
+always-on worker instance** co-located in Railway Singapore (`worker` service),
+using Railway's `ALWAYS` restart policy so repeated bootstrap failures cannot
+exhaust a finite retry allowance and leave the queue undrained.
 The `indexing-v3-agent` Edge Function **stays** in its current role as the
 cron-triggered completion/repair gate — the two are complementary, not
 alternatives. The worker service selects its Dockerfile via the
@@ -301,7 +303,7 @@ Operational rules that follow:
 - **Worker death costs at most one stale window of latency** for the in-flight
   job and zero data loss: all artifact writes are idempotent per
   generation/chunk-key, and completion is gated by the strict completion RPCs
-  plus the edge agent. Railway's restart-on-failure brings the worker back and
+  plus the edge agent. Railway's always-restart policy brings the worker back and
   it reclaims stale jobs automatically.
 - **Backlog improvement (not in this change):** a heartbeat that refreshes
   `locked_at` could ride the existing throttled progress updates
@@ -342,7 +344,8 @@ Rules:
 ## 5. Staging environment
 
 - **A second, dedicated Supabase project** (same org, ap-southeast-2) — not a
-  branch of production. Rationale: staging must absorb soak tests, destructive
+  branch of production. `Clinical KB Staging` was provisioned and migrated on
+  2026-07-19. Rationale: staging must absorb soak tests, destructive
   ingestion experiments, and migration rehearsal without any shared compute,
   pooling, or the production auth 10-connection cap; per-environment keys fall
   out naturally.
@@ -351,11 +354,12 @@ Rules:
   synthetic/public corpus. `public/demo-documents/` plus generated samples
   (`npm run samples`) are sufficient for load-shape realism; do not copy
   clinical production documents into staging.
-- One staging `app` container + one staging `worker` container from the _same_
-  images, different env. On Railway this is naturally a **second environment in
-  the `clinical-kb` project** (e.g. a `staging` environment) or a separate
-  project, with `RAG_PROVIDER_MODE=auto` and the staging OpenAI key. See
-  `docs/staging-setup.md` for the turnkey runbook.
+- One staging `app` container and **no staging worker**. The active Railway
+  `Database` project has a `staging` environment pinned to Singapore with
+  `RAG_PROVIDER_MODE=offline`, isolated Supabase credentials, and no OpenAI key.
+  This keeps release proofs deterministic and prevents staging ingestion from
+  draining or mutating production data. See `docs/staging-setup.md` for the
+  turnkey runbook.
 - `src/lib/supabase/project.ts` is staging-aware only when both
   `SUPABASE_STAGING_PROJECT_REF` and `SUPABASE_STAGING_PROJECT_NAME` are set.
   The declared staging ref must differ from production and every stale project;

docs/operator-backlog.md

@@ -23,7 +23,7 @@ Findings inventory for handover: [audit-handover-2026-07-14.md](audit-handover-2
 | Apply drift-codify forward migration (step 1h)                             | ✅ done    | —                                                                                                                                                                                              | Applied and drift/readiness verified 2026-07-13; verify only unless new reviewed drift is found             | [database-drift-detection.md](database-drift-detection.md)                                                                                                                      |
 | Apply repo-ahead migrations to live (post-2026-07-13)                      | ⏳ pending | Queue includes `20260717120000`, `20260717170000`, and `20260717171000`; a forward migration to purge private/non-indexed `document_title_words` rows must be merged and reviewed before apply | Zero unsafe title-word rows; `npm run check:drift`; then `eval:retrieval:quality` (36/36) for the corrector | [deploy-corrector-public-titles.md](deploy-corrector-public-titles.md) · [operator-apply-performance-latency-remediation.md](operator-apply-performance-latency-remediation.md) |
 | Full release gate (bounded OpenAI spend)                                   | ⏳ pending | migrations 1 applied                                                                                                                                                                           | `npm run verify:release`; `npm run eval:quality -- --rag-only`                                              | [launch-operator-runbook.md §2](launch-operator-runbook.md)                                                                                                                     |
-| Provision staging Supabase project (`Clinical KB Staging`, ap-southeast-2) | ⏳ pending | —                                                                                                                                                                                              | `npm run check:indexing` after `db push`                                                                    | [staging-setup.md](staging-setup.md)                                                                                                                                            |
+| Provision staging Supabase project (`Clinical KB Staging`, ap-southeast-2) | ✅ done    | —                                                                                                                                                                                              | Migrated and deployed app-only Railway staging on 2026-07-19; health checks returned ok                     | [staging-setup.md](staging-setup.md)                                                                                                                                            |
 | Staging soak + rollback rehearsal on Railway                               | ⏳ pending | staging provisioned                                                                                                                                                                            | `scripts/soak-test.ts --confirm-staging` (answer p95 ≤ 25 s)                                                | [launch-operator-runbook.md §4](launch-operator-runbook.md) · [capacity-review.md](capacity-review.md)                                                                          |
 | Production deploy to Railway                                               | ✅ done    | —                                                                                                                                                                                              | App deployment recorded live 2026-07-14; re-verify with `GET /api/health` and deployment readiness          | [deployment-architecture.md](deployment-architecture.md)                                                                                                                        |
 
@@ -41,13 +41,13 @@ Findings inventory for handover: [audit-handover-2026-07-14.md](audit-handover-2
 Each environment gets **separate** service-role + OpenAI keys (per-env blast radius). Placement is a
 dashboard/CLI action, never committed.
 
-| Secret / config                            | Status     | Where                                | Notes                                                                                                                                                                                                       |
-| ------------------------------------------ | ---------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `RAG_QUERY_HASH_SECRET` (prod)             | 🔎 verify  | Railway runtime secret               | GitHub repo secret present since 2026-07-10 (CI green); confirm the SAME value is set in Railway runtime. PIA-2 fail-closed guard requires it at boot (min 16 chars)                                        |
-| `HEALTH_DEEP_PROBE_SECRET` (prod + GitHub) | ⏳ pending | Railway runtime + GitHub repo secret | Needed to enable ops-digest. NOT currently a repo secret. Must be set in BOTH Railway (so the app authorizes the deep probe) AND as a GitHub secret with the same value, then uncomment the ops-digest cron |
-| `SUPABASE_SERVICE_ROLE_KEY` (per env)      | ⏳ pending | Railway runtime secret               | accepts the `sb_secret_…` key                                                                                                                                                                               |
-| `OPENAI_API_KEY` (per env)                 | ⏳ pending | Railway runtime secret               | `RAG_PROVIDER_MODE=auto`                                                                                                                                                                                    |
-| OpenAI DPA / ZDR execution                 | ⏳ pending | OpenAI account + legal               | app endpoints are ZDR-eligible; execution is operator + legal — see [openai-cross-border-basis.md](openai-cross-border-basis.md)                                                                            |
+| Secret / config                            | Status     | Where                                | Notes                                                                                                                                                                |
+| ------------------------------------------ | ---------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `RAG_QUERY_HASH_SECRET` (prod)             | 🔎 verify  | Railway runtime secret               | GitHub repo secret present since 2026-07-10 (CI green); confirm the SAME value is set in Railway runtime. PIA-2 fail-closed guard requires it at boot (min 16 chars) |
+| `HEALTH_DEEP_PROBE_SECRET` (prod + GitHub) | ✅ done    | Railway runtime + GitHub repo secret | Matching values were set and the authorized production deep probe returned healthy on 2026-07-19; the ops-digest schedule is enabled.                                |
+| `SUPABASE_SERVICE_ROLE_KEY` (per env)      | ⏳ pending | Railway runtime secret               | accepts the `sb_secret_…` key                                                                                                                                        |
+| `OPENAI_API_KEY` (per env)                 | ⏳ pending | Railway runtime secret               | `RAG_PROVIDER_MODE=auto`                                                                                                                                             |
+| OpenAI DPA / ZDR execution                 | ⏳ pending | OpenAI account + legal               | app endpoints are ZDR-eligible; execution is operator + legal — see [openai-cross-border-basis.md](openai-cross-border-basis.md)                                     |
 
 ## Disaster-recovery re-creation (does NOT survive a schema restore)
 

docs/staging-setup.md

@@ -46,7 +46,7 @@ those vars are unset.
 ## B. Staging app host (compute tier)
 
 Host: **Railway**, same as production (see `docs/deployment-architecture.md` §2).
-Stand staging up as a **second environment** in the `clinical-kb` Railway project,
+Stand staging up as a **second environment** in the active `Database` Railway project,
 with one `app` service pinned to **Southeast Asia
 (`asia-southeast1-eqsg3a`, Singapore)** — the closest region to the staging
 Supabase project in Sydney. Do not create a worker for this release profile.

railway.worker.json

@@ -25,8 +25,7 @@
     ]
   },
   "deploy": {
-    "restartPolicyType": "ON_FAILURE",
-    "restartPolicyMaxRetries": 10,
+    "restartPolicyType": "ALWAYS",
     "multiRegionConfig": {
       "asia-southeast1-eqsg3a": {
         "numReplicas": 1

scripts/check-env-parity.mjs

@@ -26,7 +26,7 @@ const root = path.join(path.dirname(fileURLToPath(import.meta.url)), "..");
 // Vars that MUST be supplied as deployment/CI secrets (never committed). Each is
 // asserted to exist in the canonical name set below, so this list cannot silently
 // drift from the schema.
-const EXPECTED_SECRETS = [
+export const EXPECTED_GITHUB_SECRETS = [
   "SUPABASE_SERVICE_ROLE_KEY",
   "OPENAI_API_KEY",
   "RAG_QUERY_HASH_SECRET",
@@ -35,6 +35,13 @@ const EXPECTED_SECRETS = [
   "E2E_USER_PASSWORD",
 ];
 
+export const EXPECTED_RAILWAY_SECRETS = [
+  "SUPABASE_SERVICE_ROLE_KEY",
+  "OPENAI_API_KEY",
+  "RAG_QUERY_HASH_SECRET",
+  "HEALTH_DEEP_PROBE_SECRET",
+];
+
 /** Zod schema keys from env.ts: lines shaped like `  NAME: z.…`. */
 export function parseEnvSchemaNames(envTsText) {
   return [...envTsText.matchAll(/^\s*([A-Z][A-Z0-9_]*)\s*:\s*z\./gm)].map((m) => m[1]);
@@ -58,16 +65,23 @@ export function computeParity({ canonical, liveNames, expectedSecrets }) {
   };
 }
 
+/** Extract Railway variable names from the CLI's JSON object without exposing values. */
+export function parseRailwayVariableNames(raw) {
+  const parsed = JSON.parse(raw);
+  if (!parsed || Array.isArray(parsed) || typeof parsed !== "object") {
+    throw new Error("Railway variable JSON must be an object");
+  }
+  return Object.keys(parsed);
+}
+
 function ghSecretNames() {
   const raw = execFileSync("gh", ["secret", "list", "--json", "name"], { encoding: "utf8" });
   return JSON.parse(raw).map((s) => s.name);
 }
 
 function railwayVarNames() {
-  // `railway variables` prints a table; --json/-k vary by CLI version, so parse
-  // UPPER_SNAKE tokens defensively. Names only.
-  const raw = execFileSync("railway", ["variables"], { encoding: "utf8" });
-  return [...new Set([...raw.matchAll(/\b([A-Z][A-Z0-9_]{2,})\b/g)].map((m) => m[1]))];
+  const raw = execFileSync("railway", ["variable", "list", "--json"], { encoding: "utf8" });
+  return parseRailwayVariableNames(raw);
 }
 
 function main() {
@@ -81,19 +95,21 @@ function main() {
   const problems = [];
 
   // Self-consistency: every expected secret must be a name the app/CI actually knows.
-  const unknownExpected = EXPECTED_SECRETS.filter((name) => !canonical.has(name));
+  const expectedSecrets = new Set([...EXPECTED_GITHUB_SECRETS, ...EXPECTED_RAILWAY_SECRETS]);
+  const unknownExpected = [...expectedSecrets].filter((name) => !canonical.has(name));
   if (unknownExpected.length > 0) {
     problems.push(
       `Expected-secret names not found in env.ts/check-ci-env (typo or drift): ${unknownExpected.join(", ")}`,
     );
   }
 
   console.log(`Known env names: ${canonical.size} (env.ts schema + check-ci-env).`);
-  console.log(`Expected secrets: ${EXPECTED_SECRETS.join(", ")}`);
+  console.log(`Expected GitHub secrets: ${EXPECTED_GITHUB_SECRETS.join(", ")}`);
+  console.log(`Expected Railway secrets: ${EXPECTED_RAILWAY_SECRETS.join(", ")}`);
 
-  for (const [flag, enabled, label, getter] of [
-    ["--gh", useGh, "GitHub secrets", ghSecretNames],
-    ["--railway", useRailway, "Railway variables", railwayVarNames],
+  for (const [flag, enabled, label, getter, sourceExpectedSecrets] of [
+    ["--gh", useGh, "GitHub secrets", ghSecretNames, EXPECTED_GITHUB_SECRETS],
+    ["--railway", useRailway, "Railway variables", railwayVarNames, EXPECTED_RAILWAY_SECRETS],
   ]) {
     if (!enabled) {
       console.log(`(${label}: skipped — pass ${flag} to check; names only, no values)`);
@@ -109,7 +125,7 @@ function main() {
     const { missingSecrets, unknownLive } = computeParity({
       canonical: [...canonical],
       liveNames,
-      expectedSecrets: EXPECTED_SECRETS,
+      expectedSecrets: sourceExpectedSecrets,
     });
     console.log(`\n${label}: ${liveNames.length} names.`);
     if (missingSecrets.length > 0) problems.push(`${label}: missing expected secret(s): ${missingSecrets.join(", ")}`);

scripts/check-indexing.ts

@@ -361,7 +361,7 @@ async function main() {
   ]);
 
   requireServerEnv();
-  requireOpenAIEnv();
+  if (env.RAG_PROVIDER_MODE !== "offline") requireOpenAIEnv();
 
   // IDX-C2 fail-fast: verify the configured embedding dimension matches the schema's
   // vector(N) columns before any indexing/DB work, so a model/dimension misconfiguration

scripts/production-readiness.ts

@@ -277,9 +277,11 @@ async function main() {
     NEXT_PUBLIC_SUPABASE_URL: process.env.NEXT_PUBLIC_SUPABASE_URL,
     SUPABASE_PROJECT_REF: process.env.SUPABASE_PROJECT_REF,
     SUPABASE_PROJECT_NAME: process.env.SUPABASE_PROJECT_NAME,
+    SUPABASE_STAGING_PROJECT_REF: process.env.SUPABASE_STAGING_PROJECT_REF,
+    SUPABASE_STAGING_PROJECT_NAME: process.env.SUPABASE_STAGING_PROJECT_NAME,
   });
   if (supabaseCheck.status === "ready") {
-    result.passes.push("Supabase project config points to Clinical KB Database.");
+    result.passes.push(`Supabase project config points to ${supabaseCheck.expected.name}.`);
   } else if (supabaseCheck.status === "warning") {
     if (supabaseCheck.warnings.length) {
       result.warnings.push(...supabaseCheck.warnings);

tests/check-indexing-contract.test.ts

@@ -10,6 +10,10 @@ const source = readFileSync(new URL("../scripts/check-indexing.ts", import.meta.
 const smartBackfillSource = readFileSync(new URL("../scripts/backfill-smart-index.ts", import.meta.url), "utf8");
 
 describe("indexing health scan", () => {
+  it("does not require an OpenAI key for the staging offline profile", () => {
+    expect(source).toContain('if (env.RAG_PROVIDER_MODE !== "offline") requireOpenAIEnv();');
+  });
+
   it("uses bounded document batches large enough for the production corpus", () => {
     const batchSize = Number(source.match(/const documentIdBatchSize = (\d+);/)?.[1]);
     expect(batchSize).toBeGreaterThanOrEqual(50);

tests/production-readiness-offline.test.ts

@@ -1,8 +1,15 @@
+import { readFileSync } from "node:fs";
 import { describe, expect, it } from "vitest";
 
 import { openAIReadinessPolicy } from "../scripts/production-readiness";
 
 describe("production readiness provider policy", () => {
+  it("passes the explicit staging declaration to the shared project guard", () => {
+    const source = readFileSync(new URL("../scripts/production-readiness.ts", import.meta.url), "utf8");
+    expect(source).toContain("SUPABASE_STAGING_PROJECT_REF: process.env.SUPABASE_STAGING_PROJECT_REF");
+    expect(source).toContain("SUPABASE_STAGING_PROJECT_NAME: process.env.SUPABASE_STAGING_PROJECT_NAME");
+  });
+
   it("requires an OpenAI key for auto and openai modes", () => {
     expect(openAIReadinessPolicy("auto")).toEqual({ required: true, ready: false });
     expect(openAIReadinessPolicy("openai")).toEqual({ required: true, ready: false });