feat(ops): ingestion autopilot, flake ledger, and CI failure triage by BigSimmo · Pull Request #588 · BigSimmo/Database

BigSimmo · 2026-07-13T14:18:20Z

What

Wave 2c — three linked reliability tools. Both live/CI workflows ship inert (nothing runs until you enable it).

Ingestion autopilot (self-heals silent queue stalls)

scripts/ingestion-autopilot.ts — pure assessIngestionHealth() flags a stuck queue (failed jobs, or a processing job locked past a stale threshold) from the read-only reindex-health JSON. Only with --apply does it run the existing recover-ingestion-queue path. Alerts (non-zero) only on unreachable Supabase or failed recovery. Dry-run by default. (Decision is based purely on reindex-health — check-indexing is intentionally not chained, as it needs the OpenAI+Python stack and is a strict corpus gate.)
.github/workflows/ingestion-autopilot.yml — SHIPPED DISABLED (dispatch-only; schedule commented). Apply is double-gated: dispatch input apply=true AND repo var INGESTION_AUTOPILOT_APPLY=true. check:supabase-project guard, pinned actions.

Flake ledger (stop re-diagnosing known flakes)

tests/flake-ledger.json — the 4 known Playwright flakes with reasons/refs.
scripts/flake-ledger.mjs — loader + isKnownFlake() + --self-test/--list/--match.

CI failure triage

.github/workflows/ci-triage.yml — SHIPPED INERT (no-op until repo var CI_TRIAGE_ENABLED=true). On PR CI failure, posts ONE comment classifying each failed job as main-side (also failing on the latest main run — CI merges with current main, which has cost debugging time before), possible known flake (UI job → flake ledger), or needs investigation. Reads only job metadata + the trusted default-branch flake ledger; never runs PR code; least-privilege token.

Verification

9/9 vitest (autopilot assessment + flake ledger), flake-ledger --self-test, check:github-actions, tsc --noEmit, eslint, prettier — all green. No live provider/API calls in CI.

Enabling (operator steps)

Ingestion autopilot: confirm SUPABASE_SERVICE_ROLE_KEY secret, run one dispatch dry-run, then uncomment schedule: (and set INGESTION_AUTOPILOT_APPLY=true to allow recovery).
CI triage: set repo var CI_TRIAGE_ENABLED=true.

🤖 Generated with Claude Code

Wave 2c. Three linked reliability tools; the two live/CI workflows ship inert. Ingestion autopilot (self-heals silent queue stalls): - scripts/ingestion-autopilot.ts: pure assessIngestionHealth() flags a stuck queue (failed jobs, or a processing job locked past a stale threshold) from the read-only reindex-health JSON; only with --apply does it run the existing recover-ingestion-queue path. Alerts (non-zero) only on unreachable Supabase or failed recovery. Dry-run by default. - .github/workflows/ingestion-autopilot.yml: SHIPPED DISABLED (dispatch-only, schedule commented). Apply is double-gated (dispatch input AND repo var INGESTION_AUTOPILOT_APPLY). check:supabase-project guard, pinned actions. Flake ledger (stop re-diagnosing known flakes): - tests/flake-ledger.json: the 4 known Playwright flakes with reasons/refs. - scripts/flake-ledger.mjs: loader + isKnownFlake() + --self-test/--list/--match. CI failure triage: - .github/workflows/ci-triage.yml: SHIPPED INERT (no-op until repo var CI_TRIAGE_ENABLED=true). On PR CI failure, posts ONE comment classifying each failed job as main-side (also failing on latest main — CI merges with current main), possible-known-flake (UI job → flake ledger), or needs-investigation. Reads only job metadata + the trusted default-branch flake ledger; never runs PR code; least-privilege token. Verified: 9/9 vitest, flake-ledger --self-test, check:github-actions, tsc, eslint, prettier — all green. No live provider/API calls in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

supabase · 2026-07-13T14:18:27Z

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

coderabbitai · 2026-07-13T14:19:25Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aa151cfa-9819-4f3e-a7d2-3f482b58d87d

📥 Commits

Reviewing files that changed from the base of the PR and between e4ff21c and 9122fee.

📒 Files selected for processing (8)

.github/workflows/ci-triage.yml
.github/workflows/ingestion-autopilot.yml
package.json
scripts/flake-ledger.mjs
scripts/ingestion-autopilot.ts
tests/flake-ledger.json
tests/flake-ledger.test.ts
tests/ingestion-autopilot.test.ts

📝 Walkthrough

Summary by CodeRabbit

New Features
- Added automated CI failure triage that categorizes failures and maintains a single status comment on pull requests.
- Added an ingestion queue autopilot with safe dry-run behavior and optional recovery execution.
- Added detection and tracking for known flaky browser tests.
- Added health assessment for failed jobs, stale processing locks, and service availability.
Tests
- Added coverage for flake detection and ingestion health assessment scenarios.

Walkthrough

Adds CI failure triage with flake-ledger classification and pull-request comments, plus an ingestion autopilot CLI and GitHub Actions workflow for health checks, dry-run recovery, apply-mode recovery, and failure alerts.

Changes

CI failure triage

Layer / File(s)	Summary
CI triage workflow and comment flow `.github/workflows/ci-triage.yml`	Runs after failed pull-request CI workflows, classifies failed jobs, and creates or updates a marked pull-request comment.
Flake ledger contract and validation `tests/flake-ledger.json`, `scripts/flake-ledger.mjs`, `package.json`, `tests/flake-ledger.test.ts`	Adds the ledger format, matching utilities, CLI commands, npm script, and Vitest coverage.

Ingestion queue autopilot

Layer / File(s)	Summary
Ingestion health assessment `scripts/ingestion-autopilot.ts`, `tests/ingestion-autopilot.test.ts`	Defines health assessment types, detects unavailable or stuck ingestion states, and tests failed-job and stale-lock handling.
Autopilot probe and recovery orchestration `scripts/ingestion-autopilot.ts`, `package.json`	Runs health probes, handles assessment outcomes, and optionally invokes queue recovery with propagated flags.
Manual workflow and failure alerting `.github/workflows/ingestion-autopilot.yml`	Adds manual dry-run/apply execution, Supabase validation, concurrency controls, and conditional issue alerts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkflowRun
  participant CIJobsAPI
  participant FlakeLedger
  participant PullRequest
  WorkflowRun->>CIJobsAPI: list failed jobs
  CIJobsAPI->>FlakeLedger: read ledger entries
  FlakeLedger-->>CIJobsAPI: return matching flake data
  CIJobsAPI->>PullRequest: create or update triage comment

sequenceDiagram
  participant WorkflowDispatch
  participant IngestionAutopilot
  participant ReindexHealth
  participant RecoverIngestionQueue
  WorkflowDispatch->>IngestionAutopilot: start with execution flags
  IngestionAutopilot->>ReindexHealth: request health probe
  ReindexHealth-->>IngestionAutopilot: return health JSON
  IngestionAutopilot->>RecoverIngestionQueue: run recovery when apply is enabled

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/ingestion-autopilot

✨ Simplify code

Create PR with simplified code
Commit simplified code in branch claude/ingestion-autopilot

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9122feef29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-13T14:21:24Z

+          APPLY_REQUESTED: ${{ github.event.inputs.apply }}
+          APPLY_ALLOWED: ${{ vars.INGESTION_AUTOPILOT_APPLY }}
+        run: |
+          if [ "$APPLY_REQUESTED" = "true" ] && [ "$APPLY_ALLOWED" = "true" ]; then


Allow scheduled autopilot runs to apply recovery

When the commented schedule is enabled, scheduled events do not have github.event.inputs.apply, so APPLY_REQUESTED is empty and this condition can never take the apply path even if INGESTION_AUTOPILOT_APPLY=true. A stuck queue would only run the dry-run command, exit 0, and skip the alert issue, leaving the promised self-healing cadence inert; gate scheduled recovery on the repo var separately from the manual dispatch input.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-13T14:21:24Z

+                repo: context.repo.repo,
+                event: "push",
+                branch: context.payload.repository.default_branch,
+                per_page: 1,


Query enough main runs to find the CI workflow

This fetches only the single latest default-branch workflow run across the repository, then later looks for one whose name matches CI. If any other workflow ran on main more recently than CI, latestMain is undefined and mainFailingJobs stays empty, so PR failures caused by an already-broken main are mislabeled as flakes or needing investigation. Query the specific workflow or fetch enough runs before selecting the latest matching CI run.

Useful? React with 👍 / 👎.

BigSimmo enabled auto-merge (squash) July 13, 2026 14:18

BigSimmo merged commit 5f220f9 into main Jul 13, 2026
14 of 17 checks passed

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

BigSimmo deleted the claude/ingestion-autopilot branch July 13, 2026 16:37

This was referenced Jul 17, 2026

test: harden reliable test execution #699

Closed

test: harden reliable test execution #705

Merged

ci: enforce safer pull request evidence #716

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ops): ingestion autopilot, flake ledger, and CI failure triage#588

feat(ops): ingestion autopilot, flake ledger, and CI failure triage#588
BigSimmo merged 1 commit into
mainfrom
claude/ingestion-autopilot

BigSimmo commented Jul 13, 2026

Uh oh!

supabase Bot commented Jul 13, 2026

Uh oh!

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BigSimmo commented Jul 13, 2026

What

Ingestion autopilot (self-heals silent queue stalls)

Flake ledger (stop re-diagnosing known flakes)

CI failure triage

Verification

Enabling (operator steps)

Uh oh!

supabase Bot commented Jul 13, 2026

Uh oh!

coderabbitai Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading