BigSimmo · BigSimmo · Jul 20, 2026 · Jul 20, 2026 · Jul 20, 2026
@@ -94,13 +94,15 @@ jobs:
               .write();
 
             // Advisory metadata nudges (title/summary/verification/UI/risk) are
-            // surfaced but never fail the check. Only a clinical-risk PR with an
-            // incomplete Clinical Governance Preflight blocks the merge.
+            // surfaced but never fail the check. Two conditions block the merge: a
+            // clinical-risk PR with an incomplete Clinical Governance Preflight, and a
+            // RAG-ranking-surface PR without an explicit `RAG impact:` declaration
+            // (docs/rag-behaviour/safeguards.md).
             for (const warning of warnings) core.warning(warning);
 
             if (!result.ok) {
               for (const error of result.errors) core.error(error);
               core.setFailed(
-                `PR policy found ${result.errors.length} blocking clinical-governance issue(s). Complete the Clinical Governance Preflight and rerun.`,
+                `PR policy found ${result.errors.length} blocking issue(s) — see the errors above (Clinical Governance Preflight and/or the RAG impact declaration), fix the PR body, and rerun.`,
               );
             }
@@ -185,6 +185,34 @@ When a branch or PR review completes, record the reviewed branch/ref, HEAD SHA,
 
 <!-- END:supabase-project-safety -->
 
+<!-- BEGIN:rag-ranking-protection -->
+
+# RAG ranking protection
+
+Retrieval/ranking behaviour is live-validated and safeguarded. Before touching any protected
+surface, read `docs/rag-behaviour/` (README → behaviour-map → refuted-approaches → safeguards).
+
+- **Flag it.** Any task that will touch `src/lib/rag/**`, clinical-search, retrieval-selection,
+  released-search-order, ranking-config, evidence/result-sort/answer-ranking, the eval harness
+  (`scripts/eval-retrieval.ts`, `scripts/lib/clinical-aliases.ts`, ranking-tuning/snapshot
+  tooling), the golden fixture/snapshot, or the retrieval RPCs must say so to the user BEFORE
+  editing, even when the change looks incidental (refactor, rename, "just a comment").
+- **PR gate.** PRs touching those surfaces fail `pr-policy` without an explicit `RAG impact:`
+  line in the body — either `RAG impact: no retrieval behaviour change — <reason>` or
+  `RAG impact: behaviour change — canary pair <baseline> -> <post>`. The source-pin contract
+  test (`tests/rag-imputation-contract.test.ts`) additionally goes red on any edit to the
+  imputation formulas or release-comparator key order.
+- **Canary for behaviour.** Any retrieval/ranking/ordering behaviour change requires a live
+  eval-canary before/after pair (doc/content recall pinned 1.0, zero per-case rr regressions)
+  before it is trusted; regression → immediate single-commit revert + confirmation run.
+  Dispatches are provider-backed (~$1–2) and always need explicit user approval.
+- **Never** insert a comparator key above the relevance score, bulk-merge the wide
+  captured-case alias tier into the strict golden tier, relax the clamped-score contract, or
+  adopt tuner recommendations without a measured live gain. Offline-green + review-approved
+  was proven insufficient for this surface on 2026-07-20 (see refuted-approaches).
+
+<!-- END:rag-ranking-protection -->
+
 <!-- BEGIN:railway-project-safety -->
 
 # Railway project safety

@@ -657,3 +657,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: Phase C saturation-tail primaries) | 7572c7f | ADDENDUM 4 Phase C (user-authorized): per-candidate discriminative primaries for saturated fast-path ties — design-agent planned (consumer map + dead-band envelope proof), red-proven, tie-conservation guarded | Mechanism: min(text_rank,1) collapses all tr≥1 candidates to byte-identical imputed primaries; ordering fell to chunk id at release. Fix: saturationTailUnit (pure, monotone, SET-INDEPENDENT — rejected per-query min-max + rank-tier designs for set-dependence/#118 authority risk; rejected full-range log rescale for moving sub-knee values across the 0.62-0.82 gate ladder) scales the excess into DEAD cap bands only: S2 table-fact similarity (0.92, 0.94) with hybrid byte-identical (gates/triggers/selection provably unchanged; similarity = the release tie-break key), S1 lexical-chunk hybrid (0.48, 0.5) behind the truthful-contract signature (sub-0.5 bars hold). Sub-knee byte-identical (fixtures now DERIVE from the helper; 0.45→0.755/0.795 pinned). Discriminating test verified RED on old formulas (2 fail: discriminating + envelope) → green with tail; equal-tr tie-conservation pins the #987 coverage comparator; second-stage-engaged pools documented out of scope (position-derived releaseRankScore sorts first there) — matches live evidence that non-engaged pools (patient-safety, opioid, flowchart) are where id-order decided. S3/S4 = C-PR-2 candidates, evidence-gated on the post-merge canary vs #54 baseline (doc/content recall MUST stay 1.0, zero per-case regressions; success signal = rr lift on the headroom cases). Rollback: single revert (helpers + 2 expression sites + 1 map call; no schema/config/cache surface). | Targeted vitest 121/121 (fast-path 11/11 incl. 6 new, retrieval-selection, rag-routing, rag-answer-fallback, ranking-tuning, second-stage); npm run test 3025 passed / 1 known container pdf-budget artifact; lint + typecheck + prettier clean; red-proof executed and recorded; live validation = post-merge canary dispatch (~$1-2) |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: revert Phase C pending review) | revert of f96217c | User-requested revert of #1004 (Phase C saturation-tail primaries): auto-merge fired before the adversarial rag-retrieval-reviewer pass landed; retrieval code returns to the last reviewed state until that verdict is in | Clean single-commit revert (the documented rollback path — helpers + 2 expression sites + 1 map call; no schema/config/cache surface). Ledger history rows from f96217c retained (docs are append-only record, not behavior). The 19:44Z post-merge canary dispatched on f96217c completes regardless and stands as Phase C's live validation datapoint; re-land decision = reviewer verdict + that pair result together. Offline state of the reverted change remains fully proven (red-proof + 121/121 + envelope tests). | Revert verified by vitest fast-path suite returning to pre-C 5/5 shape expected in CI; ladder on the revert = hosted pr-required |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (Phase C live verdict — no code change; main stays reverted at 0a498e6) | canary run 29773198933 (#55, on f96217c) | ADDENDUM 4 Phase C LIVE-REFUTED: canary on the merged tail code FAILED 3/36 (doc_recall 1.0→0.9167, mrr@10 0.8921→0.8138) — the user-ordered revert (#1005) was correct and STANDS | Failures: patient-safety-plan-include (PtSafetyPlan out of top-5, rr 0.33→0.14), patient-property-visual-table (rr 1.00→0.11) and schizophrenia-overview (rr 1.00→0.14) — two previously rank-1 cases destroyed. Root cause (post-hoc): the S1 lexical-chunk lift spreads hybrid_score, which is the PRIMARY release sort key — inside the dead (0.48,0.5) band it still PREEMPTS every downstream key, so raw ts_rank order overrode the boost/title/subject-aware relevance order that previously decided all-tied-at-0.48 pools. Lexically-loud chunks leapfrogged title-boosted correct documents = the #118 mechanism, reproduced live. The offline S1 test used identical-content candidates (coverage tie) and could not see it. The S2 similarity tail (tie-break-position key) remains reviewer-verified safe in isolation — retrieval reviewer verdict on the full diff: APPROVE-WITH-NITS, P3 only (0.49-lowering proven unreachable via the SQL 0.48 cap; no gate crossings in (0.92,0.94); hardening nits recorded). DISPOSITION: no re-land as-is. Any retry = S2-only + S1 redesigned to a key BELOW relevance in the comparator chain, new design + fresh pair. The staged-rollback discipline (canary pair + instant revert) worked exactly as designed. Spend ≈$4-8 of ≤$20. | Canary #55 read from job log (3 FAIL lines + summary); reviewer verdict from subagent report; main verified reverted (0 saturationTailUnit refs at 0a498e6) |
+| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: RAG memory + safeguards) | see PR head | User-directed close-out: durable RAG behaviour memory (docs/rag-behaviour/, 4 files) + AGENTS.md standing protection rules + ENFORCED safeguards — pr-policy blocking `RAG impact:` gate on protected surfaces (self-tested: undeclared/vague blocked, no-change/canary declarations pass) + source-pin contract test on imputation formulas and release comparator key order (red-proven vs a mutated constant). Confirmation canary #56 (29774459706, reverted main 0a498e6): SUCCESS — 36/36 restored, closing the #55-regression→revert→restore arc live. Phase D complete (D-1 ref input #1003, D-2 eval:trend #1006, D-3 policy §3.1 #1006, D-4 latency in trend rows); remaining documented plans: word-boundary matcher (own PR), irrelevant@10 labeling audit, Phase E (separate approval). Spend ≈$5-10 of ≤$20. | pr-policy self-test + workflow guard PASS; contract test 4/4 + red-proof; npm run test 3025 passed / 1 known container pdf-budget artifact; lint+typecheck+prettier clean; docs:check-links 1030 refs PASS; check:github-actions PASS |
@@ -0,0 +1,32 @@
+# RAG behaviour memory
+
+Durable, evidence-backed knowledge about how this repo's retrieval/ranking stack actually
+behaves — created 2026-07-20 after a full measure → tune → structural-fix → refute cycle
+(ADDENDUM 4; canary runs #49–#56). Read this BEFORE touching any retrieval, ranking,
+selection, release-ordering, or eval-ground-truth surface.
+
+| File                    | What it holds                                                                                                                                                                      |
+| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `behaviour-map.md`      | The verified mechanics: score imputation sites, the release comparator chains, the gate/threshold ladder, second-stage engagement rules, and which live cases exercise which path. |
+| `refuted-approaches.md` | The two live-refuted improvement attempts with their numbers, the root-cause post-mortems, and the binding constraints any third attempt must satisfy.                             |
+| `safeguards.md`         | The protection stack: protected-surface list, the pr-policy `RAG impact:` gate, the source-pin contract test, the canary-pair protocol, and the regeneration procedures.           |
+
+Standing rules (mirrored in `AGENTS.md` so every agent session inherits them):
+
+1. **Flag RAG impact.** Any task touching a protected surface (list in `safeguards.md`) must say
+   so explicitly before editing, and its PR must carry a `RAG impact:` line (enforced by
+   `scripts/pr-policy.mjs` — the check fails without it).
+2. **Canary for behaviour changes.** Any retrieval/ranking/ordering behaviour change requires a
+   live eval-canary pair — baseline + post — with doc/content recall pinned at 1.0 and zero
+   per-case regressions, before the change is trusted. `workflow_dispatch` on
+   `eval-canary.yml` (provider-backed → explicit user approval per run).
+3. **Offline green is necessary, never sufficient.** The Phase C regression passed 121/121
+   offline tests and an adversarial code review, then failed 3/36 live within one run. The live
+   corpus is the only authority on ordering behaviour.
+4. **Ties are load-bearing.** Pools of equal scores are resolved by the boost/title/subject-aware
+   relevance rank — that resolution is correct behaviour, not noise. Never insert a new
+   comparator key above the relevance score.
+
+Related: `docs/observability-slos.md` §3.1 (boundary-case & metric-interpretation policy),
+`docs/branch-review-ledger.md` (2026-07-20 rows = the full audit trail),
+`docs/rag-hybrid-findings-and-todo.md` (older findings; RC-numbered history).
@@ -0,0 +1,86 @@
+# RAG behaviour map (verified 2026-07-20)
+
+Everything below was verified against live canary evidence (runs #49–#56) and direct source
+inspection during the ADDENDUM-4 cycle. Line numbers drift with refactors — search by symbol.
+
+## 1. Score imputation on the embedding-free text fast path
+
+When a query resolves without embeddings (`strategy=text_fast_path` and friends), candidates
+carry **imputed** primaries derived only from Postgres `text_rank`:
+
+| Site               | Where                                                 | Formula (current, reviewed state)                                                                                                                   | Saturation                          |
+| ------------------ | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
+| S1 text chunks     | SQL `match_document_chunks_text(_v2)`                 | `similarity = 0` (truthful contract — no fabricated cosine); `hybrid = least(0.5, 0.18 + least(text_rank,1)·0.3)`; honest signal in `lexical_score` | tr ≥ 1 → hybrid exactly 0.48        |
+| S2 table facts     | `rag-candidate-sources.ts` (`runTableFacts` grouping) | `similarity = min(0.94, 0.62 + min(tr,1)·0.3)`; `hybrid = min(0.97, 0.66 + min(tr,1)·0.3)`                                                          | tr ≥ 1 → byte-identical 0.92 / 0.96 |
+| S3 document lookup | `rag-candidate-sources.ts` (documentScore path)       | partial discrimination via `chunkScore·0.08`; doc rank clamps at 0.34, sim cap 0.92 reachable                                                       | high-tr + high-sim pools tie        |
+| S4 memory cards    | `rag-candidate-sources.ts`                            | `sim = min(0.92, 0.58 + confidence·0.28)`                                                                                                           | equal-confidence cards tie          |
+
+`text_rank` itself is unbounded above 1 (ts_rank_cd + title-weighted + trigram/term bonuses)
+and the SQL orders by it — so the RPCs discriminate candidates that the `min(tr,1)` clamps then
+collapse to byte-identical app-side primaries. **This saturation is why unrelated documents
+matching the same terms tie exactly.**
+
+## 2. What resolves ties (the comparator chains)
+
+- **Selection** (`retrieval-selection.ts`): clamped score → lexicalScore → rerankScore →
+  contentCoverageScore (#987) → chunkId. The clamped-score contract is sacred (measured
+  golden doc-recall 1.0→0.76 when violated — the #118 lesson). Coverage is carried ONLY as a
+  late tie-break, never added to score.
+- **Release without second stage** (`released-search-order.ts`): hybrid → similarity →
+  relevance.score → id.
+- **Release with second stage**: releaseRankScore (= max(hybrid, finalScore + position
+  adjustment)) → similarity → relevance.score → id. Position adjustments launder the selection
+  order into release order, so for engaged pools the selection comparator is what matters.
+- **rankClinicalResults** (`clinical-search.ts`): unbounded rankScore → similarity → id.
+
+**The critical property: in all-saturated pools, hybrid AND similarity tie, so ordering falls
+to `relevance.score` — the boost/title/subject-aware clinical rank. That fallback is doing
+correct clinical work.** (Proven live: spreading similarity/hybrid above it caused the Phase C
+regression — see `refuted-approaches.md`.)
+
+## 3. Second-stage engagement (`shouldUseSecondStageRerank`)
+
+- `table_threshold` / `medication_dose_risk`: engages on visual evidence OR `topScoresClose`
+  (|top1−top2| ≤ 0.04) — saturated pools always engage.
+- `comparison`: engages on overflow or closeness.
+- Everything else: needs closeness AND visual evidence — plain lexical pools do NOT engage.
+
+Live mapping (canary #54): engaged — clozapine, alcohol-ciwa-threshold; not engaged —
+patient-safety-plan, opioid-withdrawal, flowchart-next-step. Fixes that only touch release-time
+tie-breaks can never move engaged pools.
+
+## 4. The gate/threshold ladder (why score bands matter)
+
+All functional gates read `max(hybrid ?? similarity)` and live at ≤ 0.82: fast-path acceptance
+0.62/0.64/0.66, coverage gates 0.48–0.62, answer routing 0.32/0.48/0.64/0.76, confidence bars
+0.5/0.68, conflict/high-confidence 0.82. The bands (0.92, 0.94] (table-fact sim), (0.96, 0.97]
+(table-fact hybrid) and (0.48, 0.5) (lexical hybrid) contain **no gates** — but they are NOT
+free real estate for ordering keys (see §2's critical property).
+
+## 5. Live case ↔ path map (canary #53/#54 evidence)
+
+- Perfect rank-1 under current behaviour: 27+ of 36 cases.
+- Known rank-depth headroom, all `text_fast_path`, all still PASSING their gates:
+  flowchart-next-step rr 0.20, alcohol-ciwa-threshold 0.25 (second-stage-engaged),
+  patient-safety-plan 0.33, opioid-withdrawal 0.33.
+- `lithium-therapy-monitoring` rr was a hardcoded 0.00 until 2026-07-20 (no document
+  expectation); now gated on `["Lithium"]` and measuring 1.0. Treat 2026-07-20 as an mrr@10
+  baseline step (+~0.028 from de-noising).
+- `irrelevant_source_rate@10` ≈ 0.108 is dominated by broad/vector cases pulling topically
+  adjacent sibling guidelines — audit labels before treating as ranking debt
+  (`docs/observability-slos.md` §3.1).
+
+## 6. Eval measurement mechanics
+
+- Golden gates are zero-tolerance top-5 per-case checks; `--fail-on-threshold` fails the run on
+  any miss. The canary log's per-case lines + human summary are what humans and the
+  failure-issue analyzer read; `--json-out` writes the machine artifact independently.
+- `textContainsClinicalTerm` is whitespace-delimited: it cannot match punctuation-joined tokens
+  (`CIWA-Ar`, `treatment,`, `(opioid`). Sanctioned aliases (`scripts/lib/clinical-aliases.ts`,
+  the STRICT tier) absorb known cases; a word-boundary matcher change is a proposed,
+  separately-reviewed follow-up. The WIDER captured-case tier in
+  `src/lib/eval-document-matching.ts` must never be bulk-merged into the strict tier.
+- Fixture and ranking snapshot move in lockstep (test-pinned); the snapshot carries
+  `generatedAt` provenance with an active 30-day freshness gate; regenerate from the latest
+  `eval-canary-output` artifact via `npm run build:ranking-snapshot`.
+- Run-over-run trends: `npm run eval:trend -- <artifact.json...>`.