fix: surface and recover stranded uploads

.github/workflows/ingestion-autopilot.yml

@@ -1,12 +1,14 @@
 # Ingestion autopilot. Probes the live ingestion queue (reindex-health) and, when
 # a stuck-queue signal is present, runs the existing recovery path. Alerts (opens/
-# updates an issue) only when Supabase is unreachable or recovery fails.
+# updates an issue) when recoverable work is detected, Supabase is unreachable,
+# or an explicitly requested recovery fails.
 #
 # ENABLED IN DRY-RUN: the 6-hourly schedule is active, but recovery only APPLIES when
 # BOTH the dispatch input `apply=true` AND repo variable INGESTION_AUTOPILOT_APPLY ==
 # "true". The variable is intentionally unset, so scheduled runs are read-only probes
-# that alert on unreachable Supabase or (with apply) failed recovery. To allow real
-# recovery after observing a clean dry-run cycle, set INGESTION_AUTOPILOT_APPLY=true.
+# that alert on detected stuck/stranded work, unreachable Supabase, or (with apply)
+# failed recovery. To allow real recovery after reviewing a dry-run, set
+# INGESTION_AUTOPILOT_APPLY=true and dispatch with apply=true.
 name: Ingestion Autopilot
 
 on:
@@ -75,6 +77,9 @@ jobs:
           if [ "$APPLY_REQUESTED" = "true" ] && [ "$APPLY_ALLOWED" = "true" ]; then
             echo "Applying recovery when stuck."
             npm run ingestion:autopilot -- --apply
+          elif [ "$GITHUB_EVENT_NAME" = "schedule" ]; then
+            echo "Scheduled dry-run; alert if recoverable work is detected."
+            npm run ingestion:autopilot -- --alert-on-stuck
           else
             echo "Dry-run (set repo var INGESTION_AUTOPILOT_APPLY=true and dispatch apply=true to recover)."
             npm run ingestion:autopilot
@@ -92,9 +97,10 @@ jobs:
               "",
               `Run: ${runUrl}`,
               "",
-              "Either Supabase was unreachable or recovery failed. Triage:",
-              "`npm run reindex:health`, then `npm run recover:ingestion -- --apply` if a stuck",
-              "queue is confirmed. Do not assume corruption — a transient outage looks the same.",
+              "Either recoverable work was detected, Supabase was unreachable, or recovery failed. Triage:",
+              "`npm run reindex:health`, then `npm run recover:ingestion -- --include-stranded-queued --apply`",
+              "if a stuck queue or aged queued-without-job document is confirmed. Do not assume corruption —",
+              "a transient outage looks the same.",
             ].join("\n");
             const { data: existing } = await github.rest.issues.listForRepo({
               owner: context.repo.owner,

docs/outstanding-issues.md

@@ -151,8 +151,8 @@ Move resolved rows here with the resolution date and a one-line outcome. Keep th
 | #073 | issue | Presentation compare dock CTA is a self-link no-op               | Fixed in PR #1135: dock shows non-link "Comparing (N)" status while already comparing.                                                                                                                                                                                                                                                                                                                                                                           | 2026-07-24 |
 | #074 | issue | Mode-action popup hard-reloads internal clinical routes          | Fixed in PR #1135: `master-search-header` uses `router.push` for DSM/Specifiers/Formulation actions and mode href fallback.                                                                                                                                                                                                                                                                                                                                      | 2026-07-24 |
 | #068 | task  | Regenerate full drift-manifest snapshot after schema hygiene     | Full Docker `npm run drift:manifest` replay succeeded on a Docker-capable host; `supabase/drift-manifest.json` now carries live `def_hash` values for the plpgsql table-facts body (offline generator_note removed).                                                                                                                                                                                                                                             | 2026-07-24 |
-| #052 | issue | Reindex can overlap a fresh agent-enrichment pass                | Full/retry single and bulk reindex now preflight `listDocumentsWithActiveAgentEnrichment` / `hasActiveAgentEnrichmentJob`; enrichment mode unchanged; stale leases still do not block. Focused lease-gate tests plus private-access reindex coverage prove the 409 path.                                                                                                                                                                                         | 2026-07-24 |
-| #062 | issue | Upload crash can strand a queued document without a job          | Added stranded-row reproducer and owner-scoped aged `queued`-without-open-job recovery via `recover:ingestion --include-stranded-queued`, reusing idempotent `enqueueDocumentReindexJob`. Hosted apply of any future atomic-enqueue RPC remains optional; sweep is the smallest recovery path.                                                                                                                                                                   | 2026-07-24 |
+| #052 | issue | Reindex can overlap a fresh agent-enrichment pass                | PR #1143 retained the friendly full/retry preflight and closed its check-then-enqueue race with an owner-scoped transactional RPC. Reindex enqueue and the agent claim path serialize on the document row; disposable PostgreSQL proved both interleavings, and exact-head migration replay/unit/build/Chromium/policy/security checks passed.                                                                                                                   | 2026-07-24 |
+| #062 | issue | Upload crash can strand a queued document without a job          | Aged owner-scoped `queued`-without-open-job rows are detected by `reindex:health`; the six-hour autopilot raises a durable alert, and guarded recovery uses PR #1143's transactional RPC so enqueue is owner-scoped, idempotent and atomic. `recover:ingestion --include-stranded-queued` remains dry-run/confirmation-first; scheduled production mutation is not enabled.                                                                                      | 2026-07-24 |
 | #060 | issue | Safety Plan Generator contradicted the privacy contract          | PR #1119 removed patient identifier entry, leaves the post-export name line blank, and aligned tool, privacy and PIA copy. DOM/privacy tests and Chromium copy/print/network coverage prove working content remains in React memory with no fetch/XHR; hosted Production UI, build, unit, policy, safety, static-analysis and secret checks passed. Support-contact details remain classified as sensitive local-only working content.                           | 2026-07-24 |
 | #061 | issue | Missing answer relevance metadata was treated as source-backed   | PR #1125 now requires explicit source-backed relevance for trusted/grounded presentation and prevents visual tables, clinical-note sections and quotes, and comparison metadata from bypassing the render model. Three actionable P2 review paths were fixed; focused policy/DOM tests, offline RAG, production-readiness, build, unit, static, security, and Production UI gates passed. No retrieval, ranking, generation, provider, or data behavior changed. | 2026-07-24 |
 | #034 | issue | Answer cache can serve stale governance metadata                 | Current-source verification found direct route coverage already asserts RAG-cache invalidation on document PATCH, source review, label, bulk, and reindex mutation paths. The residual test recommendation is already met; changing the protected cache key is unnecessary.                                                                                                                                                                                      | 2026-07-24 |

docs/reindex-runbook.md

@@ -56,7 +56,7 @@ If the response includes `safety.safeToRun: false`, do not retry repeatedly. Use
 - `supabase_unavailable`: pause reindexing and rerun `npm run supabase:recovery-status`.
 - `active_jobs`: wait for pending or processing jobs to finish before retrying.
 - `stale_processing_jobs`: run `npm run recover:ingestion -- --apply --limit 20`, then rerun `npm run reindex:health`.
-- `queued` documents with no open job (upload crash between document insert and job insert): dry-run `npm run recover:ingestion -- --include-stranded-queued`, then re-run with `--apply` after reviewing the candidate list. Optional `--owner-id` and `--stranded-min-age-minutes` keep the sweep bounded.
+- `queued` documents with no open job (upload crash between document insert and job insert): dry-run `npm run recover:ingestion -- --include-stranded-queued`, then re-run with `--apply` after reviewing the candidate list. Optional `--owner-id` and `--stranded-min-age-minutes` keep the sweep bounded. The six-hour ingestion-autopilot probe detects aged owner-scoped rows and opens/updates the operator alert issue; it never applies this recovery unless the guarded manual apply dispatch is used.
 
 The preflight response also includes active job counts and job metadata for operator review.
 

docs/webhooks.md

@@ -189,8 +189,10 @@ missing or the POST fails (it just does nothing, mirroring the receiver's inert
 > an orphan, but a process crash/timeout/kill _between_ the two inserts can still
 > strand the same `queued`-without-job row nothing sweeps. The only true
 > must-never-drop guarantees are an atomic enqueue (document + job in one
-> transaction/RPC) or a scheduled queued-without-job sweep. Operator recovery
-> for aged stranded rows is available via
+> transaction/RPC) or a scheduled queued-without-job sweep. The six-hour
+> ingestion-autopilot probe now detects aged owner-scoped rows and raises a
+> durable operator alert without mutating production automatically. Recovery
+> uses the transactional reindex RPC and is available via
 > `npm run recover:ingestion -- --include-stranded-queued` (optional
 > `--owner-id`, `--stranded-min-age-minutes`, `--apply`). This trigger is a
 > low-latency optimisation, not a delivery guarantee.

scripts/ingestion-autopilot.ts

@@ -39,6 +39,7 @@ export type IngestionAssessment = {
   reasons: string[];
   failedJobs: number;
   staleProcessingJobs: number;
+  strandedQueuedDocuments: number;
 };
 
 /**
@@ -55,7 +56,14 @@ export function assessIngestionHealth(
   const now = options.now ?? Date.now();
 
   if (health.ok === false || health.status === "supabase_unavailable") {
-    return { available: false, stuck: false, reasons: ["supabase unavailable"], failedJobs: 0, staleProcessingJobs: 0 };
+    return {
+      available: false,
+      stuck: false,
+      reasons: ["supabase unavailable"],
+      failedJobs: 0,
+      staleProcessingJobs: 0,
+      strandedQueuedDocuments: 0,
+    };
   }
 
   const reasons: string[] = [];
@@ -75,13 +83,18 @@ export function assessIngestionHealth(
   if (staleProcessingJobs > 0) {
     reasons.push(`${staleProcessingJobs} processing job(s) locked > ${staleAfterMinutes}m (stale worker)`);
   }
+  const strandedQueuedDocuments = Number(health.counts?.documents_stranded_queued ?? 0) || 0;
+  if (strandedQueuedDocuments > 0) {
+    reasons.push(`${strandedQueuedDocuments} stranded queued document(s)`);
+  }
 
   return {
     available: true,
     stuck: reasons.length > 0,
     reasons,
     failedJobs,
     staleProcessingJobs,
+    strandedQueuedDocuments,
   };
 }
 
@@ -101,6 +114,7 @@ function runTsxScript(script: string, args: string[] = []) {
 
 function main() {
   const apply = process.argv.includes("--apply");
+  const alertOnStuck = process.argv.includes("--alert-on-stuck");
   const staleAfterMinutes = parseIntFlag("--stale-after-minutes", 30);
   const limit = parseIntFlag("--limit", 20);
 
@@ -134,16 +148,21 @@ function main() {
   if (!apply) {
     console.log(
       `[autopilot] DRY RUN — would run: npm run recover:ingestion -- --apply --yes ` +
-        `--stale-after-minutes ${staleAfterMinutes} --limit ${limit}\n` +
+        `--include-stranded-queued --stale-after-minutes ${staleAfterMinutes} --limit ${limit}\n` +
         `[autopilot] pass --apply to recover.`,
     );
+    if (alertOnStuck) {
+      console.error("[autopilot] scheduled probe detected recoverable ingestion work — alerting.");
+      process.exitCode = 2;
+    }
     return;
   }
 
   console.log("[autopilot] applying recovery…");
   const recover = runTsxScript("scripts/recover-ingestion-queue.ts", [
     "--apply",
     "--yes",
+    "--include-stranded-queued",
     "--stale-after-minutes",
     String(staleAfterMinutes),
     "--limit",

scripts/reindex-health.ts

@@ -24,10 +24,23 @@ async function safeCount(label: string, query: PromiseLike<CountResult>) {
   }
 }
 
+async function safeDerivedCount(label: string, load: () => Promise<unknown[]>) {
+  try {
+    return { label, count: (await load()).length, error: null };
+  } catch (error) {
+    return { label, count: null, error: error instanceof Error ? error.message : String(error) };
+  }
+}
+
 async function main() {
-  const [{ requireServerEnv }, { createAdminClient }] = await Promise.all([
+  const [
+    { requireServerEnv },
+    { createAdminClient },
+    { listStrandedQueuedDocuments, STRANDED_QUEUED_DEFAULT_LIMIT, STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES },
+  ] = await Promise.all([
     import("@/lib/env"),
     import("@/lib/supabase/admin"),
+    import("@/lib/stranded-queued-recovery"),
   ]);
   requireServerEnv();
   const supabase = createAdminClient();
@@ -58,6 +71,7 @@ async function main() {
     pendingJobs,
     processingJobs,
     failedJobs,
+    strandedQueued,
     chunksWithSynopsis,
   ] = await Promise.all([
     safeCount(
@@ -88,6 +102,13 @@ async function main() {
       "jobs_failed",
       supabase.from("ingestion_jobs").select("id", { count: "exact", head: true }).eq("status", "failed"),
     ),
+    safeDerivedCount("documents_stranded_queued", () =>
+      listStrandedQueuedDocuments({
+        supabase,
+        minAgeMinutes: STRANDED_QUEUED_DEFAULT_MIN_AGE_MINUTES,
+        limit: STRANDED_QUEUED_DEFAULT_LIMIT,
+      }),
+    ),
     safeCount(
       "chunks_with_retrieval_synopsis",
       supabase
@@ -113,6 +134,7 @@ async function main() {
     pendingJobs,
     processingJobs,
     failedJobs,
+    strandedQueued,
     chunksWithSynopsis,
   ];
   const countErrors = counts.filter((item) => item.error);
@@ -131,7 +153,7 @@ async function main() {
         openJobs: openJobs ?? [],
         recommendation:
           countErrors.length === 0 && !jobsError
-            ? "If no jobs are processing unexpectedly, run npm run recover:ingestion -- --apply before resuming conservative worker:once."
+            ? "If no jobs are processing unexpectedly, run npm run recover:ingestion -- --include-stranded-queued --apply before resuming conservative worker:once."
             : "Fix the reported read errors before running workers or evals.",
       },
       null,