Finish RAG registry re-embed and quality routing

docs/rag-hybrid-findings-and-todo.md

@@ -257,15 +257,18 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
     vocabulary table, never the raw query, so RET-H4 holds). Remaining: terms OUTSIDE the
     curated vocabulary still cannot be captured without a privacy review; promotion tooling
     from `candidate_aliases` → `rag_aliases` is still manual.
-18. ⏳ **`document_index_units` vector recall** — no HNSW index (dropped 2026-07-02) and hosted
-    Supabase denies `ALTER FUNCTION … SET hnsw.ef_search` for the `language sql` hybrid RPCs, so
-    only `match_document_memory_cards_hybrid` pins `ef_search=100`. Concrete measurement plan
-    (needs live keys, ~1 hour): run `eval:retrieval:quality` twice with `--force-embedding`
-    (bypasses lexical fast paths, exercising vectors directly) — once as-is and once after
-    `create index concurrently` on `document_index_units.embedding` in a Supabase branch — and
-    compare doc-recall@5 + p90 latency. If recall gain < 1 case, close as not-worth-4.4GB. The
-    ef_search half can be retested via the plpgsql-wrapper trick that memory_cards already uses
-    (wrap the `language sql` RPC in a plpgsql shim that SETs it).
+18. ✅ **`document_index_units` vector recall / HNSW measurement — CLOSED as not worth adding
+    (2026-07-09).** Measured with live keys under the explicit eval budget. Production baseline
+    `eval:retrieval:quality -- --force-embedding` produced `document_recall_at_5=0.9306`,
+    `content_recall_at_5=1`, `top_k_hit_rate=0.9444`, `force_embedding_failure_count=0`,
+    `p90_latency_ms=23293`, and `index_units_layer_count=0`. A data-cloned Supabase preview branch
+    (`rag-index-units-hnsw-20260709`) was created, branch-only HNSW index
+    `document_index_units_embedding_hnsw_idx` was applied and confirmed, then the same eval was run
+    against the branch. Candidate result: recall unchanged, failed cases unchanged, `index_units` still
+    unused, median latency worsened by `+3698ms`, p90 worsened by `+13963ms`. The preview branch was
+    deleted. **Do not add this HNSW index to production.** Revisit only if the retrieval path is changed
+    to actually use `document_index_units.embedding` and a new eval shows at least one recall win without
+    material p90 regression.
 19. ✅ **Demo fallback can mask live retrieval failures in non-prod — DONE (2026-07-06).**
     `nonProductionSupabaseDemoFallbackReason` (the shared choke point for /api/search,
     /api/answer, and /api/answer/stream) now emits a loud `console.warn` naming the env vars to
@@ -300,21 +303,25 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
     answer-confidence label — "high" requires a genuine-cosine citation; synthetic-origin
     evidence caps at "medium" (strictly tightening, ordering/routing untouched, unit-tested in
     tests/rag-score.test.ts).
-22. ⏳ **Registry-to-corpus embedding (universal search Phase 5).** Medications/services/forms/
-    differentials are federated into `/api/search/universal` but are not retrieval-corpus
-    entities, so Answer mode cannot cite them. Concrete implementation spec (in order):
-    1. Flag `RAG_REGISTRY_CORPUS_EMBEDDING` (default off) in `src/lib/env.ts`.
-    2. Ingestion script `scripts/embed-registry-records.ts`: map each registry record to a
-       synthetic "document" (`metadata.source_kind = 'registry_record'`, title = record title,
-       one chunk per record from the record's search text, embedded with the standard
-       `text-embedding-3-small` path) so the existing chunk pipeline/RPCs need no schema change.
-    3. Re-embed on registry edit: hook `ensureRegistrySeeded` / record-update routes to enqueue
-       re-embedding for the changed slug only.
-    4. Answer-surface labelling: `sourceGovernanceWarnings` must label registry-backed
-       citations distinctly (registry records are curated summaries, not source documents).
-    5. Gates before enabling anywhere real: `eval:retrieval:quality` 23/23 with the flag ON,
-       plus invented-term controls ("florbizone syndrome management") still refusing — registry
-       rows must not become a fabrication surface for unsupported topics.
+22. 🔶 **Registry-to-corpus embedding (universal search Phase 5) — implementation restored and live
+    owner embedded (2026-07-09).** Medications/services/forms/differentials are federated into
+    `/api/search/universal` but were not retrieval-corpus entities, so Answer mode could not cite them.
+    Implemented pieces: `RAG_REGISTRY_CORPUS_EMBEDDING` default-off flag,
+    `scripts/embed-registry-records.ts` dry-run/write/list-owner tool, synthetic document/chunk mapping
+    with `metadata.source_kind = 'registry_record'`, source-governance labelling, comparator tooling,
+    and registry corpus tests. The sentinel-owner dry-run
+    (`00000000-0000-0000-0000-000000000000`) correctly found zero rows. `--list-owners` found the real
+    registry owner `4f1b3c19-3c39-4597-b9df-168c8e6007ff` with 739 eligible rows; guarded write with
+    `RAG_REGISTRY_CORPUS_EMBEDDING=true --write --confirm` upserted 739 synthetic registry corpus
+    chunks. Post-write retrieval eval passed with `document_recall_at_5=1`, `content_recall_at_5=1`,
+    `top_k_hit_rate=1`, `force_embedding_failure_count=0`, `failed_cases=[]`. Post-write
+    `eval:quality -- --rag-only` completed under budget; invented-term controls still refused and
+    numeric grounding failure rate was `0`. Remaining blockers are not registry regressions:
+    citation failure rate `0.0227` and RAG latency thresholds (`p95=44847ms`; route p95 extractive
+    `46745ms`, fast `27131ms`, strong `63613ms`). Remaining implementation before treating this as
+    fully productized: re-embed-on-edit hooks for registry updates and a dedicated answer-mode UX check
+    that registry-backed citations are labelled as curated registry records rather than primary source
+    documents.
 23. ⏳ **Finding #11 full fix (RAG optimisation Phase 2)** — the classifier-verdict memo (shipped
     2026-07-06) makes zero-result behaviour deterministic per query but does not close the gap:
     the deterministic analyzer still cannot tell in-corpus topics from out-of-corpus ones.

scripts/embed-registry-records.ts

@@ -15,6 +15,7 @@ type Args = {
   slug?: string;
   write: boolean;
   confirmed: boolean;
+  listOwners: boolean;
 };
 
 function parseArgs(argv: string[]): Args {
@@ -23,10 +24,15 @@ function parseArgs(argv: string[]): Args {
     kind: "all",
     write: false,
     confirmed: false,
+    listOwners: false,
   };
 
   for (let index = 0; index < argv.length; index += 1) {
     const token = argv[index];
+    if (token === "--list-owners") {
+      args.listOwners = true;
+      continue;
+    }
     if (token === "--write") {
       args.write = true;
       continue;
@@ -97,10 +103,85 @@ async function loadDifferentialRows(
   return (data ?? []) as DifferentialRecordRow[];
 }
 
+type OwnerCounts = {
+  ownerId: string;
+  service: number;
+  form: number;
+  medication: number;
+  differential: number;
+};
+
+function ensureOwnerCount(counts: Map<string, OwnerCounts>, ownerId: string) {
+  let count = counts.get(ownerId);
+  if (!count) {
+    count = { ownerId, service: 0, form: 0, medication: 0, differential: 0 };
+    counts.set(ownerId, count);
+  }
+  return count;
+}
+
+async function listEligibleOwnerCounts(supabase: Awaited<ReturnType<typeof loadAdminClient>>) {
+  const counts = new Map<string, OwnerCounts>();
+
+  const { data: registryRows, error: registryError } = await supabase
+    .from("clinical_registry_records")
+    .select("owner_id, kind");
+  if (registryError) throw new Error(`Could not load registry owner counts: ${registryError.message}`);
+  for (const row of registryRows ?? []) {
+    const ownerId = typeof row.owner_id === "string" ? row.owner_id : null;
+    if (!ownerId) continue;
+    const count = ensureOwnerCount(counts, ownerId);
+    if (row.kind === "form") count.form += 1;
+    else count.service += 1;
+  }
+
+  const { data: medicationRows, error: medicationError } = await supabase.from("medication_records").select("owner_id");
+  if (medicationError) throw new Error(`Could not load medication owner counts: ${medicationError.message}`);
+  for (const row of medicationRows ?? []) {
+    const ownerId = typeof row.owner_id === "string" ? row.owner_id : null;
+    if (!ownerId) continue;
+    ensureOwnerCount(counts, ownerId).medication += 1;
+  }
+
+  const { data: differentialRows, error: differentialError } = await supabase
+    .from("differential_records")
+    .select("owner_id");
+  if (differentialError) throw new Error(`Could not load differential owner counts: ${differentialError.message}`);
+  for (const row of differentialRows ?? []) {
+    const ownerId = typeof row.owner_id === "string" ? row.owner_id : null;
+    if (!ownerId) continue;
+    ensureOwnerCount(counts, ownerId).differential += 1;
+  }
+
+  return [...counts.values()].sort((left, right) => {
+    const leftTotal = left.service + left.form + left.medication + left.differential;
+    const rightTotal = right.service + right.form + right.medication + right.differential;
+    return rightTotal - leftTotal || left.ownerId.localeCompare(right.ownerId);
+  });
+}
+
 async function main() {
   const args = parseArgs(process.argv.slice(2));
+  const supabase = await loadAdminClient();
+
+  if (args.listOwners) {
+    const ownerCounts = await listEligibleOwnerCounts(supabase);
+    console.log("[registry:embed] eligible owner counts");
+    if (ownerCounts.length === 0) {
+      console.log("  No registry, medication, or differential rows found.");
+      return;
+    }
+    for (const count of ownerCounts) {
+      const total = count.service + count.form + count.medication + count.differential;
+      console.log(
+        `  ${count.ownerId} total=${total} service=${count.service} form=${count.form} medication=${count.medication} differential=${count.differential}`,
+      );
+    }
+    return;
+  }
+
   const ownerId = args.ownerId;
-  if (!ownerId) throw new Error("No owner id. Pass --owner-id <uuid> or set LOCAL_NO_AUTH_OWNER_ID.");
+  if (!ownerId) throw new Error("No owner id. Pass --owner-id <uuid>, --list-owners, or set LOCAL_NO_AUTH_OWNER_ID.");
 
   const {
     clinicalRegistryRowsToCorpusEntries,
@@ -112,7 +193,6 @@ async function main() {
     registryCorpusEmbeddingEnabled,
   } = await import("@/lib/registry-corpus");
 
-  const supabase = await loadAdminClient();
   const [registryRows, medicationRows, differentialRows] = await Promise.all([
     loadRegistryRows(supabase, args, ownerId),
     loadMedicationRows(supabase, args, ownerId),
@@ -133,6 +213,11 @@ async function main() {
   if (total > 20) console.log(`  ... ${total - 20} more`);
 
   if (!args.write) {
+    if (total === 0) {
+      console.log(
+        "[registry:embed] No eligible rows found for this owner. Re-run with --list-owners to find owners with registry rows.",
+      );
+    }
     console.log(
       "[registry:embed] Dry run. Re-run with --write --confirm and RAG_REGISTRY_CORPUS_EMBEDDING=true to write embeddings.",
     );

src/lib/clinical-search.ts

@@ -1113,7 +1113,7 @@ export function buildClinicalTextSearchQuery(query: string) {
       "po",
     );
   } else if (/\badmission\b/i.test(query) && /\bcommunity patients?\b/i.test(query)) {
-    normalizedTokens.unshift("admission", "community", "pts");
+    normalizedTokens.unshift("admission", "community", "patients", "pts");
   } else if (/\bdischarge\b/i.test(query) && /\b(?:summari[sz]e|summary|guidance|documentation?)\b/i.test(query)) {
     normalizedTokens.splice(0, normalizedTokens.length, "mental", "health", "discharge");
   } else if (
@@ -1143,7 +1143,7 @@ export function buildClinicalTextSearchQuery(query: string) {
   } else if (/\b(?:risk matrix|red zone)\b/i.test(query)) {
     normalizedTokens.push("high", "visual", "alert");
   } else if (/\badmission\b/i.test(query) && /\bdischarge\b/i.test(query)) {
-    normalizedTokens.push("community", "pts");
+    normalizedTokens.push("community", "patients", "pts");
   } else if (/\bcommunity patients?\b/i.test(query) && normalizedTokens.includes("community")) {
     normalizedTokens.push("pts");
   }

src/lib/rag-extractive-answer.ts

@@ -284,6 +284,26 @@ function answerIntentEvidencePattern(intent: AnswerIntent) {
   }
 }
 
+function requiresBloodCountEvidence(query: string) {
+  return /\b(?:anc|fbc|wbc|wcc|neutrophil|neutrophils)\b/i.test(query);
+}
+
+function asksForWithholdAction(query: string) {
+  return /\b(?:withhold|withheld|withholding|cease|stop|stopped|discontinue|discontinued)\b/i.test(query);
+}
+
+function hasBloodCountEvidence(text: string) {
+  return /\b(?:anc|fbc|full blood count|wbc|wcc|white blood cell|white cell|neutrophil|neutrophils|blood count)\b/i.test(
+    text,
+  );
+}
+
+function hasWithholdActionEvidence(text: string) {
+  return /\b(?:withhold|withheld|withholding|cease|stop|stopped|discontinue|discontinued|red range|amber range|red|amber)\b/i.test(
+    text,
+  );
+}
+
 function resultCoversAnswerIntent(result: SearchResult, query: string, intent: AnswerIntent) {
   if (intent === "unsupported") return false;
   const text = evidenceTextForGate(result);
@@ -298,6 +318,8 @@ function resultCoversAnswerIntent(result: SearchResult, query: string, intent: A
   const intentCoverage = answerIntentEvidencePattern(intent).test(text);
   if (!intentCoverage) return false;
   if (intentTokens.length > 0 && !intentTokens.some((token) => queryTokenMatchesText(token, text))) return false;
+  if (intent === "red_result_action" && requiresBloodCountEvidence(query) && !hasBloodCountEvidence(text)) return false;
+  if (intent === "red_result_action" && asksForWithholdAction(query) && !hasWithholdActionEvidence(text)) return false;
   if (/\brenal\b/i.test(query) && !/\b(?:renal|kidney|eGFR|creatinine)\b/i.test(text)) return false;
   if (/\bmax(?:imum)?\b/i.test(query) && !/\b(?:max(?:imum)?|\d+(?:\.\d+)?\s?(?:mg|mcg))\b/i.test(text)) {
     return false;
@@ -559,6 +581,8 @@ function factSupportsAnswerIntent(
       );
     case "red_result_action":
       if (kind !== "threshold_action" && kind !== "caveat") return false;
+      if (requiresBloodCountEvidence(query) && !hasBloodCountEvidence(text)) return false;
+      if (asksForWithholdAction(query) && !hasWithholdActionEvidence(text)) return false;
       return (
         /\b(?:withhold|cease|stop|discontinue|discontinued|contact|urgent|repeat|review|call for help|escalat\w*|monitor|toxicity|rash)\b/i.test(
           text,

src/lib/rag-retrieval-variants.ts

@@ -166,11 +166,18 @@ export function buildRetrievalQueryVariants(
   };
 
   addVariant(buildClinicalTextSearchQuery(query));
+  if (/\badmission\b/i.test(query) && /\bcommunity patients?\b/i.test(query)) {
+    addVariant("admission of community patients");
+    addVariant("admission community patients");
+  }
   aliasExpansions.slice(0, 2).forEach(addVariant);
   if (/\bpatient property\b/i.test(query)) {
     addVariant("patient property");
   }
   if (/\bclozapine\b/i.test(query) && /\b(?:anc|fbc|wbc|neutrophil|white cell)\b/i.test(query)) {
+    if (/\b(?:threshold|cut[\s-]?off|withhold|withheld|withholding|cease|stop|stopped|discontinue)\b/i.test(query)) {
+      addVariant("clozapine blood results amber red range");
+    }
     addVariant("clozapine anc fbc");
     addVariant("clozapine monitoring");
   }