fix(rag): break saturated selection ties by query-term coverage, not rank score

docs/branch-review-ledger.md

@@ -646,3 +646,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-20 | Credentialed release-gate checkpoint closeout (workflow_dispatch on main `7ec25d9`; user-authorized ≤$10, single dispatch each) | 7ec25d9675dea13635fa4a895af88c93da694a42 | Credentialed half of the release gate: CI dispatch + live eval canary | CI dispatch: 9/10 jobs green (unit coverage, build, Chromium production journeys, migration replay on local Supabase emulator, production-readiness CI-safe, policy self-tests, static/safety/scope). `release-browser-matrix` (WebKit/Firefox) CANCELLED twice by main-churn: ci.yml `concurrency: CI-${ref}, cancel-in-progress: true` kills in-flight dispatch runs on every main push and this repo merges every few minutes — livelock confirmed at the 2-attempt cap; WebKit/iOS verification remains outstanding with three human options (quiet-window dispatch, the weekly scheduled run, or a one-line dedicated concurrency group for the matrix job — operational-risk change, not applied). Eval canary: golden retrieval eval FAILED 4/36 (document_recall@5 0.944, ndcg@10 0.923, force_embedding_failure_count 0, no 429s — vector layer healthy, NOT the documented vector-ptsd transient class); the July 17 dispatch PASSED this step pre-#901, so the regression window implicates #901's deterministic semantic reranking (lithium-therapy-monitoring shows three unrelated documents with byte-identical rerank scores burying the lithium guideline; two other failures add fixture-vs-corpus identity components; answer-quality subset — the July 17 failure — never ran). NO canary re-run per plan (deterministic, not transient); retrieval is clinical-path and deferred to a human decision. Provider spend ≈ $1–2 (one canary run's embeddings); matrix/CI runs $0. | actions_run_trigger dispatches + rerun_failed_jobs (attempt 2); job-level conclusions and log excerpts from runs 29675875530 (both attempts), 29675878737 (July 19 canary), and 29567502452 (July 17 baseline). No local provider calls; secrets never left GitHub Actions. |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: ci.yml concurrency) | 79aaacf04e64f753c480dd957a81ac9be6acbb43 | CI workflow concurrency: stop main churn cancelling dispatch/schedule runs | One-line group expression: workflow_dispatch/schedule events now get a per-run concurrency group (github.run_id) while push/PR keep the shared ref group with cancel-in-progress — fixes the release-browser-matrix livelock (cancelled twice on 2026-07-19/20 by main merges mid-run; the weekly Sunday 18:00 UTC scheduled run was subject to the same cancellation). release-browser-matrix is not a required branch-protection check; pin/scope checkers do not constrain the concurrency block. Accepted side effect: deliberate runs can overlap push runs. Rollback: plain revert. | check:github-actions PASS; check:ci-scope PASS; format:check PASS (repo files; local-only .claude/settings.local.json warning is gitignored) |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: #901 rerank regression closeout) | fd5dcd1582e1c7aa8bf92ff097575e0c68aa930d | #901 rerank regression: baseline re-eval + offline guard net + saturated-tie fix | Baseline eval-canary dispatch on remediated main b3ae061 (run 29731533081) PASSED — golden retrieval 36/36 + answer-quality green, confirming #913-#926 closed the live 4/36 regression (July 19 failing run tested the raw #901 state; fixes landed 1-8h after it, never re-evaled until now). Residual defect found by new offline repro and fixed: with byte-identical imputed fast-path primaries, selection's clamped keys tie exactly and the chunk-id fallback decides, which secondStageScore's position adjustment launders into releaseRankScore (buried the CIWA answer doc in repro); fix = contentRankScore carried on RetrievalCandidate as tie-break between clamped rerankScore and chunk id, never added to scores. Amended #901's own never-live-validated saturated-tie pin to content-rank-then-id. Guards: tests/rag-fast-path-ordering.test.ts (4 end-to-end eval-shape repros) + ranking-tuning gates (4 golden-mapped hard negatives at production weights + full-snapshot high-risk). Fixture aliases NOT changed (Step C not triggered — identity cases pass live). | Targeted 4-suite vitest 37/37; npm run test 2981 passed (sole fail = container-only pdf-extraction-budget artifact); verify:cheap green through all 18 pre-test checks incl. lint+typecheck; check:production-readiness PASS on substantive checks (2 FAILs = documented missing-secret class in this container); build + client-bundle scan + check:rag:fixtures PASS |
+| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: coverage tie-break follow-up) | 57ec880b306a8e2b31c5f20dacc47256fc93b4e2 | Post-merge live-eval finding on #982 + corrective fix: saturated-tie key rankScore → query-term coverage | Post-#982 golden dispatch (eval-canary run #50, 29735004222, main b9057f0 + deps) came back 35/36: the three verifiable July-19 failures (lithium-therapy-monitoring, clozapine-anc-threshold, patient-safety-plan-include) all PASS live, but alcohol-ciwa-threshold flipped pass→FAIL vs the same-morning pre-#982 run #49 (29731533081, 36/36) — failing top-3 ordered by descending rankScore (1.85/1.75/1.53, all finalScore-saturated, releaseRankScore 1.09/1.086/1.07), i.e. #982's tie-break let generic clinicalSignalBoost stacking outvote the ciwa/score/threshold-bearing chunk; #982 is the only retrieval-path delta in the window. Fix: contentRankScore → contentCoverageScore sourced from lexicalCoverageScore (query-term coverage, immune to boost stacking; ties still fall to chunk id); saturated-tie contract test re-pinned so coverage beats a HIGHER rankScore (discriminating — old key fails it); fast-path CIWA guard gains the run-#50 screening-chunk shape + content-term assertion. Live validation: tonight's 18:00 UTC scheduled canary (dispatch cap 2/2 spent ≈$2-4). Separately: ci.yml dispatch 4012 survived 30+ min of main churn under #979's per-run concurrency group (fix working); duplicate dispatch 4017 cancelled. | Targeted vitest 38/38; npm run test 3012 passed / 1 known container-only pdf-budget artifact; verify:cheap green to the same artifact; build + client-bundle scan + check:rag:fixtures PASS; check:production-readiness expected missing-secret FAILs only (no secrets in container) |

src/lib/retrieval-selection.ts

@@ -493,16 +493,13 @@ export function buildRetrievalCandidates(
       // Carried ONLY as the last tie-break before chunk id: on the embedding-free fast path,
       // imputed primaries make clamped score/lexical/rerank ties routine, and a chunk-id tie-break
       // is arbitrary — the second stage's position-based adjustment then launders that arbitrary
-      // winner into the released order. Never added to score. Note this rankScore is the full
-      // clinical rank, which includes the bounded source-governance metadata terms — deliberate
-      // for a tie-break: among otherwise-indistinguishable candidates it prefers the current,
-      // well-extracted document (the conservative direction), unlike governance WEIGHTING of the
-      // primary score, which the NOTE below still forbids.
-      contentRankScore:
-        result.score_explanation?.rankScore ??
-        result.score_explanation?.preClampFinalScore ??
-        result.score_explanation?.finalScore ??
-        result.hybrid_score,
+      // winner into the released order. The key is the clinical rank's QUERY-TERM COVERAGE, not
+      // the boost-laden rankScore: the 2026-07-20 live golden run (eval-canary #50) showed that
+      // breaking saturated ties by rankScore lets generic clinicalSignalBoost stacking outvote the
+      // chunk that actually contains the queried terms (alcohol-ciwa-threshold regressed to FAIL),
+      // which is the same failure mode the clamped-score contract below exists to prevent.
+      // Never added to score.
+      contentCoverageScore: result.score_explanation?.lexicalCoverageScore,
       matchedSignals: [],
       sourceHref: documentCitationHref(citationFromResult(result)),
     };
@@ -616,11 +613,12 @@ export function selectRetrievalEvidence(args: {
       return (right.lexicalScore ?? 0) - (left.lexicalScore ?? 0);
     if ((right.rerankScore ?? 0) !== (left.rerankScore ?? 0)) return (right.rerankScore ?? 0) - (left.rerankScore ?? 0);
     // Exact tie on every clamped key (routine on the embedding-free fast path, where imputed
-    // primaries are byte-identical and confidences saturate at 1.0): prefer the content-aware
-    // clinical rank over an arbitrary chunk-id ordering. This never raises any score — it only
-    // orders otherwise-indistinguishable candidates, so the measured clamped-score contract holds.
-    if ((right.contentRankScore ?? 0) !== (left.contentRankScore ?? 0))
-      return (right.contentRankScore ?? 0) - (left.contentRankScore ?? 0);
+    // primaries are byte-identical and confidences saturate at 1.0): prefer the candidate whose
+    // content actually covers the query's terms over an arbitrary chunk-id ordering. This never
+    // raises any score — it only orders otherwise-indistinguishable candidates, so the measured
+    // clamped-score contract holds, and generic boost stacking cannot outvote query relevance.
+    if ((right.contentCoverageScore ?? 0) !== (left.contentCoverageScore ?? 0))
+      return (right.contentCoverageScore ?? 0) - (left.contentCoverageScore ?? 0);
     return left.chunkId.localeCompare(right.chunkId);
   });
   const selectedCandidates: RetrievalCandidate[] = [];

src/lib/types.ts

@@ -179,7 +179,7 @@ export type RetrievalCandidate = {
   lexicalScore?: number;
   semanticScore?: number;
   rerankScore?: number;
-  contentRankScore?: number;
+  contentCoverageScore?: number;
   matchedSignals: string[];
   sourceHref?: string;
 };

tests/rag-fast-path-ordering.test.ts

@@ -13,7 +13,7 @@ import type { SearchResult } from "../src/lib/types";
 // documents matching the same query terms collapse to byte-identical vector/text/hybrid
 // scores and only content-aware ranking can separate them. Each test replays one failed
 // eval case's shape through the production ordering pipeline. Guard surface: the CIWA
-// case is the one whose top-1 assertion binds the selection contentRankScore tie-break
+// case is the one whose top-1 assertion binds the selection contentCoverageScore tie-break
 // (its candidates tie on every clamped key); the lithium/clozapine/safety-plan cases win
 // earlier (subject boosts, table-type boosts, hybrid+id release order) and guard the
 // remediation stack end-to-end rather than the tie-break itself.
@@ -258,14 +258,29 @@ describe("text-fast-path ordering under imputed identical primaries", () => {
       content:
         "Assessment of intoxication, toxicity and withdrawal for alcohol and other drugs, with supportive care and monitoring principles.",
     });
+    // Models the chunk that took #1 in the 2026-07-20 live golden run (eval-canary #50) when the
+    // tie-break keyed on the boost-laden rankScore: screening/monitoring vocabulary earns strong
+    // generic clinicalSignalBoost, its content covers several query terms (alcohol, withdrawal,
+    // treatment, drug) — but none of the answer terms (ciwa/score/threshold). Query-term coverage
+    // must still rank the answer-bearing scale chunk above it.
+    const screeningChunk = imputedResult({
+      id: "aod-screening-p1",
+      document_id: "aod-doc",
+      title: "Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal",
+      file_name: "Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal (FSH).pdf",
+      page_number: 1,
+      section_heading: "Screening and treatment outcomes monitoring",
+      content:
+        "Use the screening tool to initially screen all patients presenting to the service, and monitor treatment outcomes for alcohol and other drug withdrawal management.",
+    });
 
-    // Offline this pins the pairwise decision between the two live contenders; the live
-    // corpus depth (the expected document sat at rank ~8 on 2026-07-19) is covered by the
-    // live golden eval, not this fixture.
+    // Offline this pins the pairwise decision among the live contenders; the live corpus depth
+    // (the expected document sat at rank ~8 on 2026-07-19) is covered by the live golden eval,
+    // not this fixture.
     const released = runTextFastPathOrdering({
       query,
       queryClass,
-      candidates: [aodOverview, ciwaDoc],
+      candidates: [aodOverview, screeningChunk, ciwaDoc],
       topK: 12,
     });
 
@@ -274,6 +289,8 @@ describe("text-fast-path ordering under imputed identical primaries", () => {
       .slice(0, 5)
       .some((result) => normalizedDocumentName(`${result.title} ${result.file_name}`).includes("alcohol withdrawal"));
     expect(docGateSatisfied).toBe(true);
+    // The eval's content gate for this case: at least one top-5 chunk must carry an answer term.
+    expect(released.slice(0, 5).some((result) => /ciwa|score|threshold/i.test(result.content))).toBe(true);
   });
 
   it("ranks the pinned safety-plan document above the policy duplicate for document lookups (patient-safety-plan-include)", () => {

tests/retrieval-selection.test.ts

@@ -659,11 +659,14 @@ describe("retrieval source selection", () => {
 });
 
 describe("saturated-score tie-breaking", () => {
-  function saturatedExplanation(preClampFinalScore: number): NonNullable<SearchResult["score_explanation"]> {
+  function saturatedExplanation(
+    preClampFinalScore: number,
+    lexicalCoverageScore = 0.5,
+  ): NonNullable<SearchResult["score_explanation"]> {
     return {
       vectorScore: 0.9,
       textRank: 0.3,
-      lexicalCoverageScore: 0.5,
+      lexicalCoverageScore,
       metadataMatchScore: 0.2,
       sectionTitleMatchBoost: 0.1,
       freshnessRecencyBoost: 0,
@@ -682,30 +685,32 @@ describe("saturated-score tie-breaking", () => {
     };
   }
 
-  it("breaks saturated-score ties by content-aware clinical rank, then chunk id", () => {
-    // Amended from #901's chunk-id-only pin (which was never live-eval-validated — the only
-    // golden eval on the #901 state failed 4/36): among candidates whose clamped score, lexical
-    // coverage, and clamped rerank confidence all tie exactly, the content-aware clinical rank
-    // now decides before the chunk-id fallback. On the embedding-free fast path this is what
-    // stops an arbitrary id ordering from burying the answer-bearing document
-    // (alcohol-ciwa-threshold class; see tests/rag-fast-path-ordering.test.ts).
-    const higherPreClamp = source({
+  it("breaks saturated-score ties by query-term coverage, then chunk id", () => {
+    // Amended from #901's chunk-id-only pin (never live-eval-validated — the only golden eval on
+    // the #901 state failed 4/36): among candidates whose clamped score, lexical signal score, and
+    // clamped rerank confidence all tie exactly, the clinical rank's QUERY-TERM COVERAGE decides
+    // before the chunk-id fallback. Coverage — not the boost-laden rankScore — is deliberate:
+    // the 2026-07-20 live golden run (eval-canary #50) proved a rankScore tie-break lets generic
+    // clinicalSignalBoost stacking outvote the chunk that contains the queried terms
+    // (alcohol-ciwa-threshold regressed to FAIL). Note the chunk-b fixture carries the LOWER
+    // rankScore: boost magnitude must not win a saturated tie, coverage must.
+    const higherCoverage = source({
       id: "chunk-b",
       hybrid_score: 1,
       similarity: 0.9,
-      score_explanation: saturatedExplanation(1.8),
+      score_explanation: saturatedExplanation(1.2, 0.9),
     });
-    const lowerPreClamp = source({
+    const lowerCoverage = source({
       id: "chunk-a",
       hybrid_score: 1,
       similarity: 0.9,
-      score_explanation: saturatedExplanation(1.2),
+      score_explanation: saturatedExplanation(1.8, 0.5),
     });
 
     const selection = selectRetrievalEvidence({
       query: "clinical guidance",
       queryClass: "broad_summary",
-      results: [lowerPreClamp, higherPreClamp],
+      results: [lowerCoverage, higherCoverage],
       topK: 2,
       maxResultsPerDocument: 2,
     });
@@ -714,12 +719,12 @@ describe("saturated-score tie-breaking", () => {
     // The prior recall fix remains: selection never lowers the raw hybrid score.
     expect(selection.results.map((item) => item.hybrid_score)).toEqual([1, 1]);
 
-    // Identical content rank still falls back to the stable chunk-id order.
+    // Identical coverage still falls back to the stable chunk-id order, regardless of rankScore.
     const tiedSelection = selectRetrievalEvidence({
       query: "clinical guidance",
       queryClass: "broad_summary",
       results: [
-        source({ id: "chunk-d", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.4) }),
+        source({ id: "chunk-d", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.6) }),
         source({ id: "chunk-c", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.4) }),
       ],
       topK: 2,