fix(rag): break saturated selection ties by query-term coverage, not rank score

[BigSimmo]

Summary

• Corrects fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982's saturated-tie key after a live finding. The post-merge golden dispatch (eval-canary run Save Codex local changes #50, main b9057f0) came back 35/36: the three verifiable July-19 regressions (lithium-therapy-monitoring, clozapine-anc-threshold, patient-safety-plan-include) now all pass live, but alcohol-ciwa-threshold flipped pass→FAIL versus the same-morning pre-fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982 run chore(deps-dev): bump @vitest/coverage-v8 from 4.1.8 to 4.1.9 #49 (36/36). The failing top-3 was ordered by descending boost-laden rankScore (1.85/1.75/1.53, all finalScore-saturated) — fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982's tie-break let generic clinicalSignalBoost stacking (screening/monitoring vocabulary) outvote the chunk that actually contains the queried terms (ciwa/score/threshold). fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982 was the only retrieval-path change in the window, so this is our regression to fix, and it is precisely the failure mode the clamped-score contract exists to prevent.
• Fix (src/lib/retrieval-selection.ts, src/lib/types.ts): the tie-break carried on RetrievalCandidate is renamed contentRankScore → contentCoverageScore and sourced from the clinical rank's lexicalCoverageScore — query-term coverage, which is exactly what the golden eval's content gate measures and is immune to boost stacking. It remains ONLY a tie-break between the clamped rerankScore and the chunk-id fallback; scores are never modified; coverage ties still fall back to the stable chunk-id order.
• Tests: the saturated-tie contract test is re-pinned so the winning chunk carries the LOWER rankScore but higher coverage — the old fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982 key fails this test, so the pin discriminates. The fast-path CIWA guard (tests/rag-fast-path-ordering.test.ts) gains a third candidate modeling run Save Codex local changes #50's winning screening chunk plus the eval's content-term assertion.
• Ledger row records the run-Save Codex local changes #50 evidence chain, plus two side observations: ci.yml dispatch run 4012 survived 30+ minutes of main churn under fix(ci): keep dispatched and scheduled CI runs alive through main churn #979's per-run concurrency group (fix confirmed working), and my duplicate dispatch 4017 was cancelled to save runners.

Verification

• Targeted vitest (rag-fast-path-ordering, rag-second-stage-ranking, retrieval-selection, ranking-tuning) — 38/38
• npm run test — 3012 passed; sole failure is the long-baselined container-only pdf-extraction-budget artifact (hosted-CI-green through feat(pwa): retirement kill-switch, offline-version binding guard, and dev teardown flag #826–fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982)
• npm run verify:cheap — all static checks, lint, typecheck green; same single artifact in the test step
• npm run build + client-bundle secret scan + npm run check:rag:fixtures — pass (36 golden cases, 21 suites)
• npm run check:production-readiness — substantive checks pass; the two FAILs are the documented missing-secret class (no live secrets in this container)
• npm run eval:retrieval:quality (must stay 36/36) — cannot run locally (needs live keys, which exist only in Actions). Live validation lands via tonight's Sunday 18:00 UTC scheduled eval-canary on post-merge main; the authorized dispatch budget (2 runs, ≈$2–4) is spent. Both today's run logs (chore(deps-dev): bump @vitest/coverage-v8 from 4.1.8 to 4.1.9 #49 pass / Save Codex local changes #50 fail) are the paste-equivalent evidence and are cited in the ledger row.

Risk and rollout

• Risk: low and narrower than fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982 — the tie-break only orders candidates whose clamped score, lexical signal score, and clamped rerank confidence all tie exactly; the key changes from a boost-laden composite to the pure query-term-coverage component. Offline, all four failure-signature guards plus the full suite pass; the discriminating contract test proves the old key is rejected.
• Rollback: plain revert of the content commit restores fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982's behavior (or reverting both restores the pre-fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982 chunk-id-only tie).
• Provider or production effects: None from merging. Live confirmation arrives via the already-scheduled weekly canary (no new provider calls initiated by this PR).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources (new fixtures are test-only synthetic text)
• Source metadata, review status, and outdated/unknown-source behavior remain conservative (governance penalties in clamped keys are untouched; the tie-break cannot promote a candidate past any governance demotion)
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed (ordering-only refinement within existing retrieval behavior; no new clinical output surface)

Notes

• Chain of evidence: July-19 eval failed 4/36 on raw rag: add deterministic and opt-in semantic reranking #901 → Fix repository regression findings #913–Fix residual audit findings #926 remediated → run chore(deps-dev): bump @vitest/coverage-v8 from 4.1.8 to 4.1.9 #49 (pre-fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982) 36/36 → run Save Codex local changes #50 (post-fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class #982) 35/36 with the CIWA flip → this PR replaces the tie-break key. Tonight's scheduled canary is the closing validation; if it fails on this case again the next lever is the fixture-alias route documented in the plan, not further ranking changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved retrieval result ordering when candidates have identical top-level scores.
  • Prioritizes content coverage and query-term relevance in saturated ties, with stable ID ordering as a final fallback.
  • Improved fast-path results to ensure relevant threshold-related content appears in the top results.

• Documentation

  • Added a review record documenting the fix and validation results.

[coderabbitai]

📝 Walkthrough

Walkthrough

The retrieval candidate tie-break changed from rank score to lexical coverage for saturated primary-score ties. Candidate types, selection tests, fast-path ordering fixtures, and the review ledger were updated to reflect and validate coverage-first ordering.

Changes

Retrieval coverage tie-break

Layer / File(s)	Summary
Coverage tie-break contract src/lib/types.ts, src/lib/retrieval-selection.ts	RetrievalCandidate now carries contentCoverageScore, sourced from lexical coverage and used as the final descending tie-break.
Saturated-tie regression tests tests/retrieval-selection.test.ts	Tests verify coverage-first ordering and stable chunk-id ordering when coverage is identical.
Fast-path validation and ledger tests/rag-fast-path-ordering.test.ts, docs/branch-review-ledger.md	Fast-path fixtures and assertions validate answer-bearing top-five results, while the ledger records the follow-up evaluation and correction.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• BigSimmo/Database#982: Both changes update saturated-tie retrieval ordering and its associated candidate metadata and tests.

Suggested reviewers: claude

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	It clearly summarizes the main change: using query-term coverage for saturated tie-breaking instead of rank score.
Description check	✅ Passed	The description is well structured and covers summary, verification, risk, governance, and notes; the required retrieval eval is acknowledged as not run.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.