fix(rag): content-aware saturated-tie ordering + offline guards for the #901 eval regression class

[BigSimmo]

Summary

• Closes out the rag: add deterministic and opt-in semantic reranking #901 rerank regression with evidence. The 2026-07-19 live golden retrieval eval failed 4/36 (lithium-therapy-monitoring, clozapine-anc-threshold, alcohol-ciwa-threshold, patient-safety-plan-include) on the raw rag: add deterministic and opt-in semantic reranking #901 ordering. A fresh baseline dispatch of eval-canary on remediated main b3ae061 this morning (run 29731533081) passed — golden retrieval 36/36 and the answer-quality subset green — confirming the same-day remediation (Fix repository regression findings #913–Fix residual audit findings #926) closed the live regression. That failing July-19 run had tested the rag: add deterministic and opt-in semantic reranking #901 state before any of those fixes landed; nobody had re-evaled since.
• Fixes the one reproducible defect that survived remediation (found by the new offline guards): on the embedding-free text fast path, candidates carry imputed primaries that are byte-identical across unrelated documents, so selection's clamped sort keys (score, lexicalScore, clamped rerankScore) all tie exactly and the chunk-id fallback decides the order arbitrarily — and secondStageScore's position-based adjustment then launders that arbitrary winner into releaseRankScore and the released order (in the repro this buried the CIWA answer document below a generic overview). Fix: RetrievalCandidate now carries contentRankScore (the content-aware clinical rank) used only as the tie-break between the clamped rerank confidence and the chunk-id fallback (src/lib/retrieval-selection.ts). It is never added to any score, so the measured clamped-score contract ("golden doc-recall@5 regressed 1.0 → 0.76 with unbounded boosts") is untouched — selection membership, per-document capping, and all non-tied orderings are unchanged.
• Adds the offline guard net this failure class never had (the rag: add deterministic and opt-in semantic reranking #901 regression shipped with zero offline coverage able to catch it): tests/rag-fast-path-ordering.test.ts replays all four failed eval cases' shapes end-to-end through the production ordering pipeline (rankClinicalResults → selectRetrievalEvidence → applySecondStageRerankIfNeeded → stabilizeReleasedSearchOrder), including tag-wiring assertions for the fix(rag): restore bounded live retrieval reranking #919 clinical_subject guard and an input-order-independence check; tests/ranking-tuning.test.ts gains a gate holding the four golden-mapped hard negatives below the first relevant candidate at production weights, plus a full-snapshot high-risk assertion.
• Amends rag: add deterministic and opt-in semantic reranking #901's own saturated-tie pin test: the "eval-validated stable order" test was introduced by the regressing PR itself (a871dd7) and was never validated by any live eval (the only golden eval on rag: add deterministic and opt-in semantic reranking #901's state failed 4/36). It pinned exactly the chunk-id arbitrariness fixed here. Amended to the content-rank-then-id contract, with a new case proving identical content ranks still fall back to the stable chunk-id order.
• Fixture ground truth (expectedDocumentSubstrings, expectedContentTerms, expectTableEvidence, clinicalDocumentAliases) is deliberately not touched — the passing baseline shows the identity-pinned cases resolve on the live corpus without alias changes.

Verification

• Targeted suites: npm run test -- tests/rag-fast-path-ordering.test.ts tests/rag-second-stage-ranking.test.ts tests/retrieval-selection.test.ts tests/ranking-tuning.test.ts — 37/37
• npm run test — 2981 passed; sole failure is the long-baselined container-only tests/pdf-extraction-budget.test.ts child-process artifact (hosted-CI-green through feat(pwa): retirement kill-switch, offline-version binding guard, and dev teardown flag #826–fix(ci): keep dispatched and scheduled CI runs alive through main churn #979)
• npm run verify:cheap — all 18 pre-test checks green (incl. lint, typecheck, knip, pr-policy/ci-scope/actions-pin self-tests); test step carries only the artifact above
• npm run check:production-readiness — substantive checks pass; the two FAILs are the documented missing-secret class in this secretless container (no Supabase/OpenAI env)
• npm run build + client-bundle secret scan + npm run check:rag:fixtures (36 golden cases, 21 suites) — pass
• Live golden retrieval eval on remediated main: PASSED 36/36 (eval-canary run 29731533081, dispatched as the pre-change baseline; user-authorized, ~$1–2). Tonight's scheduled 18:00 UTC canary re-verifies post-merge — protected from main-churn cancellation by fix(ci): keep dispatched and scheduled CI runs alive through main churn #979.

UI verification not run: no UI surface changed (lib + tests only).

Risk and rollout

• Risk: low. The single production change orders only exact-tie groups (all clamped keys equal) that previously fell to chunk-id lexicographic order; every non-tied comparison is byte-for-byte unchanged. Selection membership and per-document caps are unaffected (the comparator change reorders candidates before the same cap loop; ties previously ordered arbitrarily). The change is validated by four end-to-end pipeline guards plus the amended contract test with an id-fallback proof.
• Rollback: plain revert of fd5dcd1 (the ledger commit is documentation-only). No cache, schema, provider, or config surface is touched.
• Provider or production effects: None from the code. The baseline eval dispatch (already run, user-authorized) wrote only rag_queries telemetry rows.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — answer/source-governance layers untouched
• No patient-identifiable document workflow was introduced or expanded — test fixtures use synthetic guideline snippets only
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — no env/config change
• Service-role keys and private document access remain server-only — no access-path change
• Demo/synthetic content remains clearly separated from real clinical sources — new fixtures live in tests/ only
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — governance penalties flow into the content rank itself, so the tie-break cannot resurrect a demoted document above an undemoted equal
• Deployment classification/TGA SaMD impact was checked — retrieval ordering refinement within existing decision-support behavior; no new clinical capability

Notes

• Review context: the branch reuses claude/clinical-kb-pwa-review-asi3wb serially after fix(ci): keep dispatched and scheduled CI runs alive through main churn #979 (ci.yml concurrency fix) merged; both changes belong to the same authorized closeout of the two parked release-gate items.
• docs/branch-review-ledger.md carries the full review row including the baseline-run forensics.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved deterministic retrieval evidence ordering when candidate scores collapse, using an additional content-derived tie-break to keep clinically relevant results prioritized.
  • Updated saturated-score tie behavior to ensure stable, clinically-aware ordering with clear chunk-id fallback.
  • Refined results for medication monitoring, threshold, safety-plan, and policy-related searches to better preserve useful supporting content.
• Quality Improvements
  • Added/expanded regression tests to guarantee fast-path ordering stability and ranking-tuner expectations across golden cases.
• Documentation
  • Updated the review records with the associated regression closeout details and verification outcomes.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 828fd636-4589-42a7-9036-449b0ef853b3

📥 Commits

Reviewing files that changed from the base of the PR and between 53af06a and c19715d.

📒 Files selected for processing (3)

• src/lib/retrieval-selection.ts
• tests/rag-fast-path-ordering.test.ts
• tests/ranking-tuning.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• src/lib/retrieval-selection.ts
• tests/ranking-tuning.test.ts

---

📝 Walkthrough

Walkthrough

The retrieval pipeline now preserves contentRankScore for deterministic saturated-score tie-breaking. New tests cover five text fast-path scenarios, ranking-tuning gates, and the associated rerank regression closeout record.

Changes

Retrieval ranking correction

Layer / File(s)	Summary
Content-rank tie-break selection src/lib/types.ts, src/lib/retrieval-selection.ts, tests/retrieval-selection.test.ts	Candidates carry content rank separately from clamped primary scores; selection compares it before chunkId, with updated saturated-tie expectations.
Text fast-path ordering coverage tests/rag-fast-path-ordering.test.ts	A production-like fast-path harness validates medication, table-threshold, CIWA-Ar, document-lookup, and governance ordering scenarios.
Ranking gates and regression record tests/ranking-tuning.test.ts, docs/branch-review-ledger.md	Golden-regression and neutral-weight checks are added, and the rerank closeout is recorded with verification results.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TextFastPath
  participant buildRetrievalCandidates
  participant selectRetrievalEvidence
  participant SecondStageRerank
  participant ReleasedOrder
  TextFastPath->>buildRetrievalCandidates: build candidates with contentRankScore
  buildRetrievalCandidates->>selectRetrievalEvidence: provide candidates and clamped scores
  selectRetrievalEvidence->>SecondStageRerank: pass selected evidence
  SecondStageRerank->>ReleasedOrder: stabilize reranked results

Loading

Possibly related PRs

• BigSimmo/Database#901: Introduced the ranking-score and deterministic ordering behavior extended by this change.

Suggested labels: codex

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: content-aware saturated-tie ordering plus offline regression guards for #901.
Description check	✅ Passed	The description follows the required template and includes summary, verification, risk, governance, and notes with concrete details.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.