Simplify operational tooling and runbooks#62
Conversation
- recover-ingestion-queue.ts: replace JSON output with human-readable summary; add interactive confirmation prompt (--yes / --apply to skip); extract confirm() to shared cli-utils.ts - supabase-recovery-status.ts: replace JSON output with readable dashboard - scripts/reindex.ts: new consolidated pipeline command that encapsulates the full reindex runbook (health check, recovery, worker, repeat) - package.json: add 'reindex' npm script - docs/reindex-runbook.md: document the new npm run reindex command
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c559343bf0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7393deefb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6735eea7fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const child = spawn("npx", ["tsx", "worker/index.ts", "--once"], { | ||
| stdio: "inherit", | ||
| shell: false, | ||
| }); |
| const staleAfterCutoff = new Date(staleCutoff); | ||
| const jobs = (data ?? []) | ||
| .map((job: RawJobRow) => ({ | ||
| ...job, | ||
| documents: Array.isArray(job.documents) | ||
| ? (job.documents[0] as RecoveryDocument | undefined) | ||
| : (job.documents as RecoveryDocument | undefined), | ||
| })) | ||
| .filter((job) => { | ||
| if (job.status === "failed") { | ||
| return true; | ||
| } | ||
| if (job.status !== "processing") { | ||
| return false; | ||
| } | ||
| if (job.locked_at === null) { | ||
| return true; | ||
| } | ||
| return new Date(job.locked_at) <= staleAfterCutoff; | ||
| }); |
| rl.question(`${question} (y/N) `, (answer: string) => { | ||
| rl.close(); | ||
| resolve(answer.trim().toLowerCase() === "y"); | ||
| }); |
| ".claude/**", | ||
| "**/.claude/**", | ||
| ".tmp-visual/**", | ||
| "**/.tmp-visual/**", |
| 3. Apply any pending local migration only after the probe succeeds. | ||
| 4. Run `npm run reindex:health`. | ||
| 5. If stale or failed jobs exist, run `npm run recover:ingestion -- --apply --limit 20`. | ||
| 5. If stale or failed jobs exist, run `npm run recover:ingestion` and confirm the prompt. | ||
| 6. Run `npm run worker:once` with the conservative defaults from `.env.example`. |
| const args = parseArgs(process.argv.slice(2)); | ||
| const staleAfterMinutes = env.WORKER_STALE_AFTER_MINUTES; | ||
| const limit = args.limit; | ||
|
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7d3b464dc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (hasActiveProcessingJobs) { | ||
| console.log( | ||
| "\n Active processing jobs detected; skipping worker run to avoid concurrency spikes.", | ||
| ); | ||
| continue; |
There was a problem hiding this comment.
When a fresh processing job exists, this new guard immediately continues into the next loop iteration and counts it against --max-rounds without sleeping or running a worker. Because the runbook documents --max-rounds as the maximum worker iterations, npm run reindex -- --yes --max-rounds 5 can burn all five rounds in seconds and exit 1 while another worker is still making normal progress. Return with an operator message, or wait/back off without consuming a worker round.
Useful? React with 👍 / 👎.
| ## Safe sequence | ||
| ## Consolidated pipeline command | ||
|
|
||
| The quickest way to run a full reindex cycle is the consolidated pipeline command: |
There was a problem hiding this comment.
In the common release case where all documents are already indexed and there are no open ingestion_jobs, scripts/reindex.ts hits its queue-clear check and exits successfully without enqueueing any document for the new indexing code. Since this runbook is specifically for applying RAG indexing changes, describing npm run reindex as a full reindex cycle can leave the corpus on the old chunks/enrichment while operators proceed to evals; either add an enqueue step/command or label this as draining an already queued reindex.
Useful? React with 👍 / 👎.
| const child = spawn("npx", ["tsx", "worker/index.ts", "--once"], { | ||
| stdio: "inherit", | ||
| shell: false, | ||
| }); |
There was a problem hiding this comment.
When WORKER_CONCURRENCY is not set in the environment, the worker spawned here falls back to the code default of 8 and processes the claimed jobs concurrently, while the recovery runbook/.env.example expect concurrency 1 for this operation. In that environment, npm run reindex -- --yes can fan out eight document reindexes during a supposedly conservative recovery run; set a safe env override for this child process or fail until conservative worker settings are present.
Useful? React with 👍 / 👎.
Recovery scripts emitted raw JSON, required manual
--applyflags with no interactive guidance, and the reindex runbook demanded 8+ hand-executed steps in strict order — all friction that increases operational error risk.Changes
scripts/cli-utils.ts(new)Shared
confirm()utility using Node.jsreadline— single source for interactive yes/no prompts.scripts/recover-ingestion-queue.ts--applyis absent--yesskips the prompt for CI/piped use;--applyremains backward-compatiblescripts/supabase-recovery-status.tsReplaces JSON dump with a readable dashboard: status, checked-at, count table, any read errors, and a plain-English recommendation line.
scripts/reindex.ts+npm run reindex(new)Consolidated pipeline that replaces the 8-step manual runbook:
worker:once--max-roundsis reachedFlags:
--yes(non-interactive),--max-rounds N(default 10),--limit N(recovery actions/round, default 20).docs/reindex-runbook.mdDocuments
npm run reindexas the primary path with a flags table; retains the manual sequence for reference.Verification
npm run verify:cheapnpm run verify:uiwhen UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changednpm run verify:releasebefore release or handoff confidence claimsnpm run format:checknpm run check:production-readinesswhen clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changedClinical Governance Preflight
Complete this section when the change touches ingestion, answer generation, search/ranking, source rendering, document access, privacy, production env, or clinical output.
Clinical KB Database(sjrfecxgysukkwxsowpy)Notes
RawJobRowtype is kept inline per-script (it maps a Supabase query shape, not a domain type) rather than promoted to@/lib/ingestion-recovery