diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 9c638bf1b..65d57120e 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -646,3 +646,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-20 | Credentialed release-gate checkpoint closeout (workflow_dispatch on main `7ec25d9`; user-authorized ≤$10, single dispatch each) | 7ec25d9675dea13635fa4a895af88c93da694a42 | Credentialed half of the release gate: CI dispatch + live eval canary | CI dispatch: 9/10 jobs green (unit coverage, build, Chromium production journeys, migration replay on local Supabase emulator, production-readiness CI-safe, policy self-tests, static/safety/scope). `release-browser-matrix` (WebKit/Firefox) CANCELLED twice by main-churn: ci.yml `concurrency: CI-${ref}, cancel-in-progress: true` kills in-flight dispatch runs on every main push and this repo merges every few minutes — livelock confirmed at the 2-attempt cap; WebKit/iOS verification remains outstanding with three human options (quiet-window dispatch, the weekly scheduled run, or a one-line dedicated concurrency group for the matrix job — operational-risk change, not applied). Eval canary: golden retrieval eval FAILED 4/36 (document_recall@5 0.944, ndcg@10 0.923, force_embedding_failure_count 0, no 429s — vector layer healthy, NOT the documented vector-ptsd transient class); the July 17 dispatch PASSED this step pre-#901, so the regression window implicates #901's deterministic semantic reranking (lithium-therapy-monitoring shows three unrelated documents with byte-identical rerank scores burying the lithium guideline; two other failures add fixture-vs-corpus identity components; answer-quality subset — the July 17 failure — never ran). NO canary re-run per plan (deterministic, not transient); retrieval is clinical-path and deferred to a human decision. Provider spend ≈ $1–2 (one canary run's embeddings); matrix/CI runs $0. | actions_run_trigger dispatches + rerun_failed_jobs (attempt 2); job-level conclusions and log excerpts from runs 29675875530 (both attempts), 29675878737 (July 19 canary), and 29567502452 (July 17 baseline). No local provider calls; secrets never left GitHub Actions. | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: ci.yml concurrency) | 79aaacf04e64f753c480dd957a81ac9be6acbb43 | CI workflow concurrency: stop main churn cancelling dispatch/schedule runs | One-line group expression: workflow_dispatch/schedule events now get a per-run concurrency group (github.run_id) while push/PR keep the shared ref group with cancel-in-progress — fixes the release-browser-matrix livelock (cancelled twice on 2026-07-19/20 by main merges mid-run; the weekly Sunday 18:00 UTC scheduled run was subject to the same cancellation). release-browser-matrix is not a required branch-protection check; pin/scope checkers do not constrain the concurrency block. Accepted side effect: deliberate runs can overlap push runs. Rollback: plain revert. | check:github-actions PASS; check:ci-scope PASS; format:check PASS (repo files; local-only .claude/settings.local.json warning is gitignored) | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: #901 rerank regression closeout) | fd5dcd1582e1c7aa8bf92ff097575e0c68aa930d | #901 rerank regression: baseline re-eval + offline guard net + saturated-tie fix | Baseline eval-canary dispatch on remediated main b3ae061 (run 29731533081) PASSED — golden retrieval 36/36 + answer-quality green, confirming #913-#926 closed the live 4/36 regression (July 19 failing run tested the raw #901 state; fixes landed 1-8h after it, never re-evaled until now). Residual defect found by new offline repro and fixed: with byte-identical imputed fast-path primaries, selection's clamped keys tie exactly and the chunk-id fallback decides, which secondStageScore's position adjustment launders into releaseRankScore (buried the CIWA answer doc in repro); fix = contentRankScore carried on RetrievalCandidate as tie-break between clamped rerankScore and chunk id, never added to scores. Amended #901's own never-live-validated saturated-tie pin to content-rank-then-id. Guards: tests/rag-fast-path-ordering.test.ts (4 end-to-end eval-shape repros) + ranking-tuning gates (4 golden-mapped hard negatives at production weights + full-snapshot high-risk). Fixture aliases NOT changed (Step C not triggered — identity cases pass live). | Targeted 4-suite vitest 37/37; npm run test 2981 passed (sole fail = container-only pdf-extraction-budget artifact); verify:cheap green through all 18 pre-test checks incl. lint+typecheck; check:production-readiness PASS on substantive checks (2 FAILs = documented missing-secret class in this container); build + client-bundle scan + check:rag:fixtures PASS | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: coverage tie-break follow-up) | 57ec880b306a8e2b31c5f20dacc47256fc93b4e2 | Post-merge live-eval finding on #982 + corrective fix: saturated-tie key rankScore → query-term coverage | Post-#982 golden dispatch (eval-canary run #50, 29735004222, main b9057f0 + deps) came back 35/36: the three verifiable July-19 failures (lithium-therapy-monitoring, clozapine-anc-threshold, patient-safety-plan-include) all PASS live, but alcohol-ciwa-threshold flipped pass→FAIL vs the same-morning pre-#982 run #49 (29731533081, 36/36) — failing top-3 ordered by descending rankScore (1.85/1.75/1.53, all finalScore-saturated, releaseRankScore 1.09/1.086/1.07), i.e. #982's tie-break let generic clinicalSignalBoost stacking outvote the ciwa/score/threshold-bearing chunk; #982 is the only retrieval-path delta in the window. Fix: contentRankScore → contentCoverageScore sourced from lexicalCoverageScore (query-term coverage, immune to boost stacking; ties still fall to chunk id); saturated-tie contract test re-pinned so coverage beats a HIGHER rankScore (discriminating — old key fails it); fast-path CIWA guard gains the run-#50 screening-chunk shape + content-term assertion. Live validation: tonight's 18:00 UTC scheduled canary (dispatch cap 2/2 spent ≈$2-4). Separately: ci.yml dispatch 4012 survived 30+ min of main churn under #979's per-run concurrency group (fix working); duplicate dispatch 4017 cancelled. | Targeted vitest 38/38; npm run test 3012 passed / 1 known container-only pdf-budget artifact; verify:cheap green to the same artifact; build + client-bundle scan + check:rag:fixtures PASS; check:production-readiness expected missing-secret FAILs only (no secrets in container) | diff --git a/src/lib/retrieval-selection.ts b/src/lib/retrieval-selection.ts index 90aabfdbb..816721150 100644 --- a/src/lib/retrieval-selection.ts +++ b/src/lib/retrieval-selection.ts @@ -493,16 +493,13 @@ export function buildRetrievalCandidates( // Carried ONLY as the last tie-break before chunk id: on the embedding-free fast path, // imputed primaries make clamped score/lexical/rerank ties routine, and a chunk-id tie-break // is arbitrary — the second stage's position-based adjustment then launders that arbitrary - // winner into the released order. Never added to score. Note this rankScore is the full - // clinical rank, which includes the bounded source-governance metadata terms — deliberate - // for a tie-break: among otherwise-indistinguishable candidates it prefers the current, - // well-extracted document (the conservative direction), unlike governance WEIGHTING of the - // primary score, which the NOTE below still forbids. - contentRankScore: - result.score_explanation?.rankScore ?? - result.score_explanation?.preClampFinalScore ?? - result.score_explanation?.finalScore ?? - result.hybrid_score, + // winner into the released order. The key is the clinical rank's QUERY-TERM COVERAGE, not + // the boost-laden rankScore: the 2026-07-20 live golden run (eval-canary #50) showed that + // breaking saturated ties by rankScore lets generic clinicalSignalBoost stacking outvote the + // chunk that actually contains the queried terms (alcohol-ciwa-threshold regressed to FAIL), + // which is the same failure mode the clamped-score contract below exists to prevent. + // Never added to score. + contentCoverageScore: result.score_explanation?.lexicalCoverageScore, matchedSignals: [], sourceHref: documentCitationHref(citationFromResult(result)), }; @@ -616,11 +613,12 @@ export function selectRetrievalEvidence(args: { return (right.lexicalScore ?? 0) - (left.lexicalScore ?? 0); if ((right.rerankScore ?? 0) !== (left.rerankScore ?? 0)) return (right.rerankScore ?? 0) - (left.rerankScore ?? 0); // Exact tie on every clamped key (routine on the embedding-free fast path, where imputed - // primaries are byte-identical and confidences saturate at 1.0): prefer the content-aware - // clinical rank over an arbitrary chunk-id ordering. This never raises any score — it only - // orders otherwise-indistinguishable candidates, so the measured clamped-score contract holds. - if ((right.contentRankScore ?? 0) !== (left.contentRankScore ?? 0)) - return (right.contentRankScore ?? 0) - (left.contentRankScore ?? 0); + // primaries are byte-identical and confidences saturate at 1.0): prefer the candidate whose + // content actually covers the query's terms over an arbitrary chunk-id ordering. This never + // raises any score — it only orders otherwise-indistinguishable candidates, so the measured + // clamped-score contract holds, and generic boost stacking cannot outvote query relevance. + if ((right.contentCoverageScore ?? 0) !== (left.contentCoverageScore ?? 0)) + return (right.contentCoverageScore ?? 0) - (left.contentCoverageScore ?? 0); return left.chunkId.localeCompare(right.chunkId); }); const selectedCandidates: RetrievalCandidate[] = []; diff --git a/src/lib/types.ts b/src/lib/types.ts index 0299b278c..353720173 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -179,7 +179,7 @@ export type RetrievalCandidate = { lexicalScore?: number; semanticScore?: number; rerankScore?: number; - contentRankScore?: number; + contentCoverageScore?: number; matchedSignals: string[]; sourceHref?: string; }; diff --git a/tests/rag-fast-path-ordering.test.ts b/tests/rag-fast-path-ordering.test.ts index a0d5cf007..f04915fa1 100644 --- a/tests/rag-fast-path-ordering.test.ts +++ b/tests/rag-fast-path-ordering.test.ts @@ -13,7 +13,7 @@ import type { SearchResult } from "../src/lib/types"; // documents matching the same query terms collapse to byte-identical vector/text/hybrid // scores and only content-aware ranking can separate them. Each test replays one failed // eval case's shape through the production ordering pipeline. Guard surface: the CIWA -// case is the one whose top-1 assertion binds the selection contentRankScore tie-break +// case is the one whose top-1 assertion binds the selection contentCoverageScore tie-break // (its candidates tie on every clamped key); the lithium/clozapine/safety-plan cases win // earlier (subject boosts, table-type boosts, hybrid+id release order) and guard the // remediation stack end-to-end rather than the tie-break itself. @@ -258,14 +258,29 @@ describe("text-fast-path ordering under imputed identical primaries", () => { content: "Assessment of intoxication, toxicity and withdrawal for alcohol and other drugs, with supportive care and monitoring principles.", }); + // Models the chunk that took #1 in the 2026-07-20 live golden run (eval-canary #50) when the + // tie-break keyed on the boost-laden rankScore: screening/monitoring vocabulary earns strong + // generic clinicalSignalBoost, its content covers several query terms (alcohol, withdrawal, + // treatment, drug) — but none of the answer terms (ciwa/score/threshold). Query-term coverage + // must still rank the answer-bearing scale chunk above it. + const screeningChunk = imputedResult({ + id: "aod-screening-p1", + document_id: "aod-doc", + title: "Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal", + file_name: "Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal (FSH).pdf", + page_number: 1, + section_heading: "Screening and treatment outcomes monitoring", + content: + "Use the screening tool to initially screen all patients presenting to the service, and monitor treatment outcomes for alcohol and other drug withdrawal management.", + }); - // Offline this pins the pairwise decision between the two live contenders; the live - // corpus depth (the expected document sat at rank ~8 on 2026-07-19) is covered by the - // live golden eval, not this fixture. + // Offline this pins the pairwise decision among the live contenders; the live corpus depth + // (the expected document sat at rank ~8 on 2026-07-19) is covered by the live golden eval, + // not this fixture. const released = runTextFastPathOrdering({ query, queryClass, - candidates: [aodOverview, ciwaDoc], + candidates: [aodOverview, screeningChunk, ciwaDoc], topK: 12, }); @@ -274,6 +289,8 @@ describe("text-fast-path ordering under imputed identical primaries", () => { .slice(0, 5) .some((result) => normalizedDocumentName(`${result.title} ${result.file_name}`).includes("alcohol withdrawal")); expect(docGateSatisfied).toBe(true); + // The eval's content gate for this case: at least one top-5 chunk must carry an answer term. + expect(released.slice(0, 5).some((result) => /ciwa|score|threshold/i.test(result.content))).toBe(true); }); it("ranks the pinned safety-plan document above the policy duplicate for document lookups (patient-safety-plan-include)", () => { diff --git a/tests/retrieval-selection.test.ts b/tests/retrieval-selection.test.ts index 2e4b6230a..febf313ca 100644 --- a/tests/retrieval-selection.test.ts +++ b/tests/retrieval-selection.test.ts @@ -659,11 +659,14 @@ describe("retrieval source selection", () => { }); describe("saturated-score tie-breaking", () => { - function saturatedExplanation(preClampFinalScore: number): NonNullable { + function saturatedExplanation( + preClampFinalScore: number, + lexicalCoverageScore = 0.5, + ): NonNullable { return { vectorScore: 0.9, textRank: 0.3, - lexicalCoverageScore: 0.5, + lexicalCoverageScore, metadataMatchScore: 0.2, sectionTitleMatchBoost: 0.1, freshnessRecencyBoost: 0, @@ -682,30 +685,32 @@ describe("saturated-score tie-breaking", () => { }; } - it("breaks saturated-score ties by content-aware clinical rank, then chunk id", () => { - // Amended from #901's chunk-id-only pin (which was never live-eval-validated — the only - // golden eval on the #901 state failed 4/36): among candidates whose clamped score, lexical - // coverage, and clamped rerank confidence all tie exactly, the content-aware clinical rank - // now decides before the chunk-id fallback. On the embedding-free fast path this is what - // stops an arbitrary id ordering from burying the answer-bearing document - // (alcohol-ciwa-threshold class; see tests/rag-fast-path-ordering.test.ts). - const higherPreClamp = source({ + it("breaks saturated-score ties by query-term coverage, then chunk id", () => { + // Amended from #901's chunk-id-only pin (never live-eval-validated — the only golden eval on + // the #901 state failed 4/36): among candidates whose clamped score, lexical signal score, and + // clamped rerank confidence all tie exactly, the clinical rank's QUERY-TERM COVERAGE decides + // before the chunk-id fallback. Coverage — not the boost-laden rankScore — is deliberate: + // the 2026-07-20 live golden run (eval-canary #50) proved a rankScore tie-break lets generic + // clinicalSignalBoost stacking outvote the chunk that contains the queried terms + // (alcohol-ciwa-threshold regressed to FAIL). Note the chunk-b fixture carries the LOWER + // rankScore: boost magnitude must not win a saturated tie, coverage must. + const higherCoverage = source({ id: "chunk-b", hybrid_score: 1, similarity: 0.9, - score_explanation: saturatedExplanation(1.8), + score_explanation: saturatedExplanation(1.2, 0.9), }); - const lowerPreClamp = source({ + const lowerCoverage = source({ id: "chunk-a", hybrid_score: 1, similarity: 0.9, - score_explanation: saturatedExplanation(1.2), + score_explanation: saturatedExplanation(1.8, 0.5), }); const selection = selectRetrievalEvidence({ query: "clinical guidance", queryClass: "broad_summary", - results: [lowerPreClamp, higherPreClamp], + results: [lowerCoverage, higherCoverage], topK: 2, maxResultsPerDocument: 2, }); @@ -714,12 +719,12 @@ describe("saturated-score tie-breaking", () => { // The prior recall fix remains: selection never lowers the raw hybrid score. expect(selection.results.map((item) => item.hybrid_score)).toEqual([1, 1]); - // Identical content rank still falls back to the stable chunk-id order. + // Identical coverage still falls back to the stable chunk-id order, regardless of rankScore. const tiedSelection = selectRetrievalEvidence({ query: "clinical guidance", queryClass: "broad_summary", results: [ - source({ id: "chunk-d", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.4) }), + source({ id: "chunk-d", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.6) }), source({ id: "chunk-c", hybrid_score: 1, similarity: 0.9, score_explanation: saturatedExplanation(1.4) }), ], topK: 2,