Skip to content

fix: harden Railway operations and staging checks#918

Merged
BigSimmo merged 1 commit into
mainfrom
codex/railway-ops-staging
Jul 19, 2026
Merged

fix: harden Railway operations and staging checks#918
BigSimmo merged 1 commit into
mainfrom
codex/railway-ops-staging

Conversation

@BigSimmo

@BigSimmo BigSimmo commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • Separate GitHub/CI secret expectations from Railway runtime expectations so CI-only E2E credentials are not required in production.
  • Keep the queue-draining Railway worker on the durable ALWAYS restart policy.
  • Enable the daily ops digest after provisioning and proving its matching production secret and URL.
  • Document the active app-only, offline Railway/Supabase staging environment.
  • Make indexing and production-readiness checks support the documented offline staging profile.
  • Parse Railway variable names from JSON keys only, avoiding value-bearing output.

Verification

  • npm exec -- vitest run tests/repo-hygiene.test.ts tests/railway-config.test.ts tests/ops-digest.test.ts — 52 passed
  • npm exec -- vitest run tests/check-indexing-contract.test.ts — 5 passed
  • npm exec -- vitest run tests/production-readiness-offline.test.ts tests/supabase-project.test.ts — 12 passed
  • scoped ESLint and Prettier — passed
  • npm run check:github-actions and npm run check:runtime — passed
  • npm run check:env-parity -- --gh --railway against production — passed, names only
  • staging npm run check:supabase-project, npm run check:indexing, and npm run check:production-readiness — passed
  • staging /api/health/ready and /api/health — both ok
  • GitHub Ops Digest run 29678689328 and Staging tenancy isolation run 29678690340 — success
  • npm run verify:cheap

Verification not run: npm run verify:cheap was attempted twice but both attempts were rejected by the repository-wide heavy-command lock while other worktrees ran verify:pr-local / verify:cheap:internal; required PR CI remains the full gate.

Risk and rollout

  • Risk: medium; this changes production worker restart behavior and enables a scheduled provider-backed operations workflow, while adding staging-aware verification paths.
  • Rollback: revert this commit to disable the schedule and restore the prior code/config contracts; if provider rollback is required, disable the GitHub schedule first, then unlink or pause the isolated Railway staging app without copying or modifying production data.
  • Provider or production effects: Railway production E2E variables were removed, the deep-probe secret was matched to GitHub, and the live worker policy reads ALWAYS with no retry cap. A dedicated Sydney Supabase project and app-only Singapore Railway staging environment were provisioned. Staging is offline, has no worker or OpenAI key, and contains no production clinical data; only two synthetic tenancy-test users were created.

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use
  • No patient-identifiable document workflow was introduced or expanded without explicit governance approval
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative
  • Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

  • The checked-in check:default-acl runner currently fails before execution because its top-level await is transformed to CommonJS on main; direct RPC verification was used instead.
  • Staging postgres default ACLs verify safe. supabase_admin defaults still require the six reviewed statements from migration 20260713102000 in the Supabase SQL editor. CLI and Management API correctly reject that role-owned operation, and the available dashboard session had expired.

Separate CI-only secrets from Railway parity, make worker restarts durable, enable the verified ops digest, and support the isolated offline staging profile. Focused tests, live staging checks, and manual GitHub workflow proofs passed; the broad local gate was blocked by another worktree's heavy-test lock.
@supabase

supabase Bot commented Jul 19, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

@coderabbitai

coderabbitai Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The pull request updates worker restart behavior, scheduled operations, staging deployment documentation, provider-specific environment parity checks, and offline staging readiness validation.

Changes

Operations and readiness

Layer / File(s) Summary
Deployment and staging configuration
.github/workflows/ops-digest.yml, railway.worker.json, docs/deployment-architecture.md, docs/operator-backlog.md, docs/staging-setup.md, tests/railway-config.test.ts
Enables the daily Ops Digest schedule, configures the worker with an ALWAYS restart policy, updates staging topology and project guidance, records completed operational checks, and tests the worker configuration.
Provider-specific environment parity
scripts/check-env-parity.mjs, tests/repo-hygiene.test.ts
Separates GitHub and Railway secret expectations, parses Railway JSON variable output, passes provider-specific expectations to parity checks, and tests names and parsing behavior.
Offline staging readiness
scripts/check-indexing.ts, scripts/production-readiness.ts, tests/check-indexing-contract.test.ts, tests/production-readiness-offline.test.ts
Allows offline indexing without OpenAI variables, forwards staging Supabase identifiers to readiness checks, reports the expected project name dynamically, and adds source-contract assertions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately summarizes the main change: hardening Railway ops and staging checks.
Description check ✅ Passed The description follows the template with Summary, Verification, Risk and rollout, Clinical Governance Preflight, and Notes, and is mostly complete.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/railway-ops-staging

Comment @coderabbitai help to get the list of available commands.

@BigSimmo
BigSimmo merged commit 61b00ce into main Jul 19, 2026
17 of 19 checks passed
@BigSimmo
BigSimmo deleted the codex/railway-ops-staging branch July 19, 2026 08:02

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b01140aec6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// UPPER_SNAKE tokens defensively. Names only.
const raw = execFileSync("railway", ["variables"], { encoding: "utf8" });
return [...new Set([...raw.matchAll(/\b([A-Z][A-Z0-9_]{2,})\b/g)].map((m) => m[1]))];
const raw = execFileSync("railway", ["variable", "list", "--json"], { encoding: "utf8" });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid value-bearing Railway JSON output

When --railway is used, this command is not consuming a names-only listing: Railway's CLI source for variable list --json documents that JSON output "includes raw values" and serializes the key/value map (see railwayapp/cli src/commands/variable.rs). That means npm run check:env-parity -- --railway reads every runtime secret into the Node process despite this script's names-only/no-values contract, so any future debugging or capture around this command can expose production secrets; use a genuinely name-only source or avoid the value-bearing JSON path.

Useful? React with 👍 / 👎.

Comment on lines +357 to +360
- One staging `app` container and **no staging worker**. The active Railway
`Database` project has a `staging` environment pinned to Singapore with
`RAG_PROVIDER_MODE=offline`, isolated Supabase credentials, and no OpenAI key.
This keeps release proofs deterministic and prevents staging ingestion from

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep staging runbooks aligned on offline mode

This new offline/no-key staging profile is still contradicted by docs/launch-operator-runbook.md:121-125, which tells operators to set OPENAI_API_KEY and RAG_PROVIDER_MODE=auto. When staging is recreated from that launch runbook, it will re-enable provider-backed paths and invalidate the deterministic/no-spend staging checks this change adds; update that runbook or point it at this offline profile.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant