refactor(rag): extract the rag cluster into src/lib/rag/ (maturity X2)

docs/codebase-index.md

@@ -94,6 +94,10 @@ Structured map for AI agents and onboarding. For live routes, see `docs/site-map
 
 ### RAG, retrieval, answers
 
+The `rag.ts` orchestrator and its `rag-*` cluster live in **`src/lib/rag/`** (the first
+domain-extracted directory; imported as `@/lib/rag/rag*`). Other modules below remain flat in
+`src/lib/`.
+
 | Module                                                                                                                  | Role                                              |
 | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
 | `rag.ts`                                                                                                                | Main answer pipeline orchestrator                 |

docs/deployment-architecture.md

@@ -112,7 +112,7 @@ one.
 
 - **In-memory coalescing and caches are load-bearing.** The answer pipeline
   coalesces identical in-flight questions (`answer_inflight_coalesced` in
-  `src/lib/rag.ts`) and holds LRU answer/search caches
+  `src/lib/rag/rag.ts`) and holds LRU answer/search caches
   (`RAG_ANSWER_CACHE_TTL_MS`/`RAG_ANSWER_CACHE_SIZE`). Serverless isolates get
   one request each, so coalescing never fires and every duplicate ward-round
   question pays the full ~6-RPC fan-out plus an OpenAI generation.

docs/maturity-backlog-workorders.md

@@ -61,22 +61,23 @@ confirmation) · `SATISFIED` (already true in the repo; no work needed).
 - **Verification:** `npm run lint` reports **0** new errors; add a deliberately-wrong import in a
   scratch file to confirm the rule fires, then remove it.
 
-### X2 · `src/lib` domain-directory extraction — rag pilot — `OPEN`
+### X2 · `src/lib` domain-directory extraction — rag pilot — `DONE`
 
 - **Outcome:** the first real domain directory; unblocks directory-scoped boundary rules for the
   rest of `src/lib` (197 flat files).
-- **Approach:** `git mv` the `rag*.ts` cluster (~23 files) into a new `rag/` directory under
-  `src/lib/`; codemod `@/lib/rag*` importers; update the per-file paths in
-  `scripts/check-maintainability-budgets.mjs`.
-- **Files:** ~23 `src/lib/rag*.ts` + every importer + the budgets script + `docs/codebase-index.md`.
-- **Risk:** HIGH — broad import churn; keep it isolated with no behaviour change.
-- **Verification:** `npm run typecheck` && `npm run test`; diff must be pure moves + import-path
-  rewrites (no logic changes).
+- **Landed:** `git mv` the 22-file `rag` cluster (`rag.ts` + 21 `rag-*.ts`) into `src/lib/rag/`;
+  codemod every `@/lib/rag*` and `../src/lib/rag*` importer to `.../rag/rag*`; updated the budgets
+  key, the client-bundle boundary + worker-deploy test fixtures, `docs/codebase-index.md`, and
+  the rag path references across 13 maintained docs. **Pure moves + path rewrites, no logic
+  change.**
+- **Verification:** `typecheck`, full `test` suite (only the pre-existing container-only
+  `pdf-extraction-budget` flake fails — confirmed identical on `origin/main`), `lint`,
+  `docs:check-index`, `docs:check-links`, and maintainability budgets all pass.
 
 ### X3 · Decompose the monoliths — `OPEN`
 
 - **Outcome:** shrink the three files the maintainability ratchet caps but never reduces:
-  `src/lib/rag.ts` (5,143), `src/components/ClinicalDashboard.tsx` (4,270),
+  `src/lib/rag/rag.ts` (5,143), `src/components/ClinicalDashboard.tsx` (4,270),
   `src/components/DocumentViewer.tsx` (3,166).
 - **Approach:** extract cohesive units behind the existing budgets; `rag.ts` is the natural seam
   once X2 lands (its ~23 siblings already exist).
@@ -182,7 +183,7 @@ collaborators join — `AGENTS.md` + the PR template already carry that load.
 | N1 Dependabot grouping         | Now      | **DONE** (this PR)                             |
 | N2 Dependency-report decision  | Now      | OPEN (recommend enable)                        |
 | X1 Import-boundary linter      | Next     | READY (mockup rule; service-role rule dropped) |
-| X2 `src/lib` rag extraction    | Next     | OPEN (isolated PR)                             |
+| X2 `src/lib` rag extraction    | Next     | **DONE** (this PR)                             |
 | X3 Monolith decomposition      | Next     | OPEN                                           |
 | X4 SAST-blocking on parser     | Next     | PROVIDER-GATED (triage-first)                  |
 | X5 ACL-migration consolidation | Next     | PROVIDER-GATED (DB owner)                      |

docs/observability-slos.md

@@ -195,7 +195,7 @@ header (same operator gate as the Supabase probe) — returns three counter bloc
 - **`cache`** — `cacheMetricsSnapshot` (`src/lib/observability/cache-metrics.ts`)
   reports `{ lookups, hits, misses, hitRate }` for the retrieval search cache,
   incremented in-process at the two-layer cache orchestration in
-  `searchChunksWithTelemetry` (`src/lib/rag.ts`). A request served by **either**
+  `searchChunksWithTelemetry` (`src/lib/rag/rag.ts`). A request served by **either**
   the process-local or the shared (`rag_response_cache`) layer counts as a hit,
   so a cold process falling through to a warm shared cache is not miscounted as a
   miss; disabled/skipped lookups are recorded as neither. These are **cumulative

docs/openai-cross-border-basis.md

@@ -39,10 +39,10 @@ The code-side controls cannot _by themselves_ discharge APP 8 — the "reasonabl
 
 ## 2. What actually crosses the border
 
-| Egress    | Payload                                                          | Endpoint                                                      | Reference                                                                           |
-| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
-| Embedding | Raw query text (normalized)                                      | `POST /v1/embeddings` (`text-embedding-3-small`)              | [openai.ts embedText](../src/lib/openai.ts)                                         |
-| Answer    | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (Terra fast / Sol strong, `store:false`) | [rag.ts](../src/lib/rag.ts) · [rag-source-block.ts](../src/lib/rag-source-block.ts) |
+| Egress    | Payload                                                          | Endpoint                                                      | Reference                                                                                   |
+| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
+| Embedding | Raw query text (normalized)                                      | `POST /v1/embeddings` (`text-embedding-3-small`)              | [openai.ts embedText](../src/lib/openai.ts)                                                 |
+| Answer    | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (Terra fast / Sol strong, `store:false`) | [rag.ts](../src/lib/rag/rag.ts) · [rag-source-block.ts](../src/lib/rag/rag-source-block.ts) |
 
 The app **adds no raw patient or owner identifiers** and stores queries only as a keyed hash locally.
 When configured, authenticated Responses requests include a stable HMAC-SHA256

docs/privacy-impact-assessment.md

@@ -69,7 +69,7 @@ material.
 | Operational telemetry                                                     | `rag_retrieval_logs`, ingestion job tables                                                                                                             | Low–Medium                                 | Redacted query text; per-owner.                                                                                                    |
 
 **Deployment context (from code):** the answer system prompt positions the assistant as _"an
-experienced psychiatrist in Perth"_ ([src/lib/rag.ts:7053](src/lib/rag.ts)) — i.e. a **WA psychiatry**
+experienced psychiatrist in Perth"_ ([src/lib/rag/rag.ts:7053](src/lib/rag/rag.ts)) — i.e. a **WA psychiatry**
 use case. Psychiatric context raises the sensitivity ceiling: mental-health information is squarely
 "sensitive information" and "health information" under the _Privacy Act 1988_ (Cth).
 
@@ -88,7 +88,7 @@ Clinician browser
    │  • rate-limit bucket "answer"                                       :83
    │  • resolveSearchScope() → owner-scoped candidate document set       :93
    ▼
-[RAG pipeline]  answerQuestionWithScope()  src/lib/rag.ts
+[RAG pipeline]  answerQuestionWithScope()  src/lib/rag/rag.ts
    │
    ├──►(A) QUERY EMBEDDING ─────────────────────────────────────────────┐
    │      raw query text → OpenAI embeddings (text-embedding-3-small)    │
@@ -137,9 +137,9 @@ OpenAI as the only cross-border flow.
 | Payload         | Content                                                                                                                                       | Reference                                                      |
 | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
 | Embedding input | **Raw query text**, verbatim (normalized whitespace/case only)                                                                                | [src/lib/openai.ts:498](src/lib/openai.ts) → `embedTexts` :423 |
-| Answer input    | **Raw query verbatim** (`Question:\n${args.query}`)                                                                                           | [src/lib/rag.ts:7144](src/lib/rag.ts)                          |
-| Answer input    | **Retrieved chunk text** (content, capped ~1800 chars, plus title/page/section/table-facts/captions)                                          | [src/lib/rag.ts:6306-6325](src/lib/rag.ts)                     |
-| Instructions    | Static system prompt ("experienced psychiatrist in Perth…")                                                                                   | [src/lib/rag.ts:7053](src/lib/rag.ts)                          |
+| Answer input    | **Raw query verbatim** (`Question:\n${args.query}`)                                                                                           | [src/lib/rag/rag.ts:7144](src/lib/rag/rag.ts)                  |
+| Answer input    | **Retrieved chunk text** (content, capped ~1800 chars, plus title/page/section/table-facts/captions)                                          | [src/lib/rag/rag.ts:6306-6325](src/lib/rag/rag.ts)             |
+| Instructions    | Static system prompt ("experienced psychiatrist in Perth…")                                                                                   | [src/lib/rag/rag.ts:7053](src/lib/rag/rag.ts)                  |
 | Metadata        | `{ operation }`; when configured, `safety_identifier` is an HMAC-SHA256 pseudonym of the authenticated owner. The raw owner id is never sent. | [src/lib/openai.ts](src/lib/openai.ts)                         |
 
 The app never _adds_ patient identifiers, but it does not scrub them either: **any PHI the clinician
@@ -402,7 +402,7 @@ remaining items are compliance-posture and PHI-minimisation gaps.
   cross-tenant) and purged at 30 days, but it was un-redacted PHI-derived content at rest.
 - **Fix (shipped):** Answer-text persistence in the durable log is gated behind a dedicated
   `RAG_PERSIST_ANSWER_TEXT` flag (default **off**), applied centrally in `insertRagQuery` via
-  `answerTextForStorage` ([query-privacy.ts](src/lib/query-privacy.ts), [rag.ts](src/lib/rag.ts)) so
+  `answerTextForStorage` ([query-privacy.ts](src/lib/query-privacy.ts), [rag.ts](src/lib/rag/rag.ts)) so
   every `logRagQuery` caller is covered, and at the promoted-eval-case write in
   [eval-cases/route.ts](src/app/api/eval-cases/route.ts). With the flag off the column is written as
   `null` and each row records `metadata.answer_retained = false`. The offline eval/quality pipeline
@@ -412,7 +412,7 @@ remaining items are compliance-posture and PHI-minimisation gaps.
   affect eval — confirming the pipeline has no real dependency on stored answer text. The flag is
   additionally blocked in a production-like environment by `npm run check:production-readiness`.
 - **Residual cache copy:** The answer also lands in `rag_response_cache.payload`
-  ([rag-cache.ts](src/lib/rag-cache.ts)). Its `expires_at` TTL (`RAG_ANSWER_CACHE_TTL_MS`, default
+  ([rag-cache.ts](src/lib/rag/rag-cache.ts)). Its `expires_at` TTL (`RAG_ANSWER_CACHE_TTL_MS`, default
   5 min) only gates **reads** — `sharedCacheSelector` filters on `expires_at`, while
   `replaceSharedCacheRow` deletes only the _same_ cache key before inserting. Migration
   `20260713201542_consolidate_rag_response_cache_retention.sql` unschedules the duplicate unbounded

docs/process-hardening.md

@@ -188,7 +188,7 @@ passes `p_worker_id`. Ordered apply steps, R17 manual `CONCURRENTLY` index, and
 ## Nondeterministic "unsupported" retrieval — finding #11 needs Phase 2 (2026-07-03)
 
 - **Symptom (rag-hybrid-findings finding #11):** valid clinical topics phrased as bare low-confidence queries ("bipolar disorder", "anorexia management") intermittently return 0 results (`unsupported_short_circuit`) instead of an answer.
-- **Root cause (confirmed live):** the soft-tail branch of `shouldShortCircuitUnsupportedSearch` (`src/lib/rag.ts`) fires on `analysis.queryClass === "unsupported_or_general"`, and that class is set by `analyzeQueryWithClassifierFallback`, which calls a **generative LLM classifier** (6s timeout, `reasoningEffort:"low"`, uncached) for low-confidence queries. That call is nondeterministic: it reclassifies "bipolar disorder" to a supported class on some runs (→ answered) and declines/times-out on others (→ short-circuited to 0). Reproduced with a same-process ×N probe.
+- **Root cause (confirmed live):** the soft-tail branch of `shouldShortCircuitUnsupportedSearch` (`src/lib/rag/rag.ts`) fires on `analysis.queryClass === "unsupported_or_general"`, and that class is set by `analyzeQueryWithClassifierFallback`, which calls a **generative LLM classifier** (6s timeout, `reasoningEffort:"low"`, uncached) for low-confidence queries. That call is nondeterministic: it reclassifies "bipolar disorder" to a supported class on some runs (→ answered) and declines/times-out on others (→ short-circuited to 0). Reproduced with a same-process ×N probe.
 - **Why a clean Phase-1 fix does NOT exist:** the deterministic analyzer carries **no signal** distinguishing in-corpus topics from out-of-corpus (bipolar, anorexia, gout, DKA all produce identical `unsupported_or_general`, confidence ≈ 0.40, `canonicalTerms` = query tokens). Only the corpus can tell them apart. Removing the soft-tail short-circuit fixes the valid topics but **regresses `unsupported_correct_rate` 1.0 → 0.79** on `eval:quality --rag-only`: a lexical distinctive-term relevance gate in `chooseAnswerRoute` either over-refuses legitimate semantic/vector matches (whose exact term is not in the retrieved text — e.g. the `rag-routing.test.ts` "admission" fixtures) or under-refuses invented terms ("florbizone syndrome") that score strongly on generic scaffolding words ("syndrome"/"management"). The corpus is broad (gout 13 / crohn 49 / appendicitis 30 / angioplasty 32 / bipolar 719 chunks), so almost no common medical term is truly absent — grounded low-confidence answers, not fabrication, are the realistic worst case for real terms; only genuinely invented terms should refuse.
 - **Decision (2026-07-03):** the risky change (soft-tail removal + relevance gate + invented-term eval controls) was **reverted**; only a safe, independent hardening was kept — `fetchEnabledRagAliases` no longer caches `[]` on a transient `rag_aliases` read error (which would suppress alias expansion for the whole TTL). Finding #11 is **re-scoped into RAG optimisation Phase 2** ("fit retrieval to content"), where it should be fixed with corpus-grounded relevance — IDF/corpus-frequency weighting of query terms and/or the semantic relevance model, plus the data-driven query-understanding vocabulary (RC6) so the deterministic classifier recognises in-corpus topics up front. Any gate must keep `eval:quality --rag-only` `unsupported_correct_rate` at 1.0 (add invented-term controls like "florbizone syndrome management" / "quxbyria disorder treatment" once it can pass them) while letting valid bare topics answer. A cheaper interim option worth measuring first: **classifier-verdict memoization** to at least make the current behaviour deterministic per query.
 - **RESOLVED (2026-07-07, `claude/retrieval-correctness`):** the Phase-2 corpus-grounded fix shipped. `corpus_topic_term_stats` (migration `20260707100000`, applied live) reports per-term title-topic membership (with a measured 5% genericity ceiling — "management"/"guideline" headline ~18–20% of titles and are scaffolding; real topics ≤3%) and chunk-level presence, scoped with the exact retrieval `owner_filter`. `src/lib/corpus-grounding.ts` + the `analyzeQueryWithClassifierFallback` hook classify only the soft-tail branch (pattern-guarded refusals and higher-confidence classes untouched): in-corpus bare topics deterministically reclassify to `broad_summary` and answer; corpus-absent queries skip the LLM and refuse deterministically (trigram correction still runs); inconclusive keeps memoized-LLM behaviour; DB errors fail open. Verified live: "bipolar disorder" / "anorexia management" answer 4/4 runs with the right document at rank 1 (new golden cases `bare-topic-bipolar` / `bare-topic-anorexia`); "florbizone syndrome management" / "quxbyria disorder treatment" refuse 4/4 runs (new `ragEvalCases` controls); golden eval 36/36.

docs/rag-hybrid-findings-and-todo.md

@@ -41,7 +41,7 @@ measurements are evidence, not instructions to repeat provider-backed work witho
 ## P0 — correctness / observability ✅ DONE (2026-07-01)
 
 1. ✅ **App silently swallows hybrid-RPC failures — FIXED.** Added `recordHybridRpcError` in
-   `src/lib/rag.ts` (structured `logger.error("hybrid_rpc_failed", …)` + new
+   `src/lib/rag/rag.ts` (structured `logger.error("hybrid_rpc_failed", …)` + new
    `SearchTelemetry.hybrid_rpc_errors` map surfaced in `rag_retrieval_logs`), threaded through
    `searchEmbeddingFieldCandidates` / `searchIndexUnitCandidates` / the chunks call, and a matching
    `logger.error` at the memory-card call in `src/lib/deep-memory.ts`. A dead hybrid layer now logs +