From 82426c73777a54cea6e926584166b8660024afdb Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 19:45:35 +0000 Subject: [PATCH 1/2] Revert "feat(rag): per-candidate saturation-tail primaries on the text fast path (ADDENDUM 4 Phase C) (#1004)" This reverts commit f96217c6213649d4760533ebff039edfe9d294ce. --- docs/branch-review-ledger.md | 2 - docs/rag-hybrid-findings-and-todo.md | 13 +- src/lib/rag/rag-candidate-sources.ts | 66 +-------- src/lib/rag/rag.ts | 8 +- tests/rag-fast-path-ordering.test.ts | 198 +-------------------------- 5 files changed, 13 insertions(+), 274 deletions(-) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index ae4096166..6f0a2c433 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -653,5 +653,3 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: eval measurement floor, ADDENDUM 4 A-PR-1) | 81ab9696da4b330ca0b2e5519891a9942f90421b | Measurement floor for evidence-gated ranking tuning: canary artifact emission, alias-aware snapshot builder, snapshot provenance/freshness — no ranking behavior change | Closes the three gaps blocking safe tuning (Phase B): (1) eval-canary's golden step now writes the per-case JSON artifact (--json-out decoupled from --json so the tee'd log keeps the human-readable lines the failure-issue analyzer parses) and uploads .local/eval-canary/ via pinned upload-artifact (30-day retention, include-hidden-files for the dot-dir, contents = same class as the already-public step logs: titles/telemetry/220-char previews of the all-public corpus) — snapshot regeneration stops costing a paid dispatch; (2) clinicalDocumentAliases/clinicalContentAliases moved verbatim to shared scripts/lib/clinical-aliases.ts and the snapshot builder grades documentMatch/contentMatch through them (discriminating tests: EMHS agitation title and spelled-out "absolute neutrophil count" grade as hits only via aliases — raw labelMatches pinned false), ending tuner ground truth disagreeing with the live gates; (3) snapshots carry generatedAt + optional sourceRunId, validator accepts them, exactly-36 relaxed to at-least-36 (floor still rejects truncated artifacts; sourceCaseCount + per-case candidate minimums unchanged), and a 30-day freshness test (activates on first regeneration) blocks silent corpus drift. Builder smoke-verified end-to-end on a synthetic 36-case artifact (alias grading + provenance stamped + validator green). Static hyphen audit of all 36 cases' terms: no currently-blocked term (canary #52 = 36/36); residual risk classes documented for the A-PR-2 artifact-grounded pass — punctuation-joined tokens (IM/PO, schizo-affective, post-natal) and inert stem entries (obsess/compuls/hyperactiv/impuls can never match whole-token) that currently ride on whole-word OR-alternates. | Targeted vitest 52/52 (ranking-tuning + eval-retrieval + eval-quality); npm run test 3019 passed / 1 known container-only pdf-budget artifact; lint + typecheck clean; check:github-actions + check:ci-scope PASS; prettier clean; check:production-readiness expected missing-secret FAILs only (demo-mode container); no provider calls — live validation = tonight's scheduled canary emits the first artifact at $0 | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR #1001: A-PR-2 measurement-floor completion) | 3d8f798 + 5b664f8 + e58c827 | First provenance-stamped snapshot regeneration from a live canary artifact + alias tiering docs + fixture-length pin + lithium doc-gate — Phase A of ADDENDUM 4 functionally complete | Artifact chain: user-authorized paid dispatch (canary #53, run 29763761133, 36/36 green, doc_recall 1.0 / content_recall 1.0 — first perfect content recall, ciwa alias confirmed live; mrr@10 0.8644, irrelevant@10 0.1083) emitted the first eval-canary-output artifact (51787 bytes, sha256 5af5b802… verified byte-identical after user transfer into the sandbox — this container cannot download run artifacts). Work: (1) two-tier alias documentation — investigation of the governance-review P3 showed src/lib/eval-document-matching.ts is a deliberately WIDER captured-case tier (e.g. "Clozapine GP Shared Care"); bulk-merge would loosen golden ground truth, so both files now carry cross-referencing do-not-merge headers instead; (2) snapshot case count pinned to live golden fixture length (regeneration instructions in failure message) — closes the coarse-floor P3; (3) snapshot regenerated via the alias-aware builder with --source-run-id provenance: agitation-im-po-options 0→5 graded positives (EMHS alias working on real data), flowchart-next-step confirmed sole zero-positive case; generatedAt promoted to validator-REQUIRED (closes the hand-edit P3); two stale data pins updated (missing-positives 2→1; broad_summary defaults-equality pin dropped — defaults' provenance was the retired snapshot, fresh recommendations are Phase B input); (4) artifact-grounded punctuation audit: 7 joined-token occurrences in top-5 previews — 3 ciwa-ar (alias-covered, incl. line-broken "ciwa- ar"), 4 ORDINARY-PROSE punctuation ("treatment," / "mood," / "(opioid" / "ptsd.[35]") — matcher word-boundary change proposed as its OWN reviewed follow-up per plan (systemic class, not bundled); (5) lithium-therapy-monitoring was the ONLY ungated case (rr@10 hardcoded 0.00 = measurement noise): expectedDocumentSubstrings ["Lithium"] added from live evidence (deliberately broad across the corpus's multiple legitimate lithium guidelines), snapshot rebuilt in lockstep from the same artifact, measured mrr@10 +~0.028 from de-noising. Deferred with reasons: NEW-query fixture cases (saturated-tie shapes, captured rag_query_misses) need live validation before they may gate — unlocked by Phase D-1 branch-eval dispatch or a dedicated validation dispatch; real ordering headroom for Phase B = flowchart 0.20, alcohol-ciwa 0.25, patient-safety 0.33, opioid 0.33, all text_fast_path. | Targeted vitest 76/76 ×3 (after each stage); npm run test 3019 passed / 1 known container-only pdf-budget artifact; prettier clean; freshness gate ACTIVE and green; no provider calls beyond the user-authorized dispatch (~$1-2, ADDENDUM 4 spend now ~$1-2 of ≤$10) | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: B-PR-1 canary dispatch inputs) | see PR head | eval-canary.yml workflow_dispatch gains rag_ranking_config (staged Phase B weight evals, loud malformed-JSON validation to prevent silent fallback corrupting pair comparisons) + ref (Phase D-1 pre-merge branch evals) — schedule runs unaffected (empty inputs = today's behavior) | Tuner (offline, $0) on the A-PR-2 regenerated snapshot recommends 3 constrained per-class improvements (document_lookup titleSectionRelevance→0.9; table_threshold clinicalEvidence→0.95; comparison hybridRelevance→0.95 with proxy mrr 0.833→1.0), all with recall non-regression + zero high-risk hard-negative failures; medication_dose_risk + broad_summary stay neutral. Staged config JSON banked for the live pair; baseline = canary #53 re-gated (mrr@10 ≈0.8922 after lithium de-noising). Targets per approved plan: mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions. Budget: user raised cap to ≤$20 (spent ≈$1-2). Security note: ref input runs branch code with eval secrets — dispatch requires repo write access (sole trusted collaborator), documented in PR risk. | check:github-actions PASS; check:ci-scope PASS; prettier clean; no provider calls (workflow change only) | -| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (Phase B live pair — no code change; recorded for eval-history completeness) | canary runs 29763761133 (#53 baseline) + 29769050798 (#54 tuned) | ADDENDUM 4 Phase B verdict: staged 3-class tuner weights produced ZERO live movement — NOT adopted per the measured-gain rule | Offline tuner recommendations (document_lookup titleSection→0.9, table_threshold clinicalEvidence→0.95, comparison hybridRelevance→0.95; proxy comparison-mrr 0.833→1.0) staged via the B-PR-1 rag_ranking_config input; #54 log proves the override active (validation step echo). Result vs re-gated baseline: mrr@10 0.8921 vs 0.8922, irrelevant@10 0.1083 vs 0.1083, recalls 1.0/1.0 both, 36/36 both, all four headroom cases byte-identical rr (flowchart 0.20, ciwa-threshold 0.25, patient-safety 0.33, opioid 0.33). Learning: the 5-candidate linear proxy saturates; real headroom lives in fast-path saturated-tie structure (Phase C trigger condition mrr < 0.90 formally met; user authorized C). Nothing to roll back (per-run override). Spend ≈$2-4 of user-raised ≤$20 cap. | Both canary runs green (36/36 + answer-quality); override-active proof in #54 job log; no adoption = no code diff | -| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: Phase C saturation-tail primaries) | 7572c7f | ADDENDUM 4 Phase C (user-authorized): per-candidate discriminative primaries for saturated fast-path ties — design-agent planned (consumer map + dead-band envelope proof), red-proven, tie-conservation guarded | Mechanism: min(text_rank,1) collapses all tr≥1 candidates to byte-identical imputed primaries; ordering fell to chunk id at release. Fix: saturationTailUnit (pure, monotone, SET-INDEPENDENT — rejected per-query min-max + rank-tier designs for set-dependence/#118 authority risk; rejected full-range log rescale for moving sub-knee values across the 0.62-0.82 gate ladder) scales the excess into DEAD cap bands only: S2 table-fact similarity (0.92, 0.94) with hybrid byte-identical (gates/triggers/selection provably unchanged; similarity = the release tie-break key), S1 lexical-chunk hybrid (0.48, 0.5) behind the truthful-contract signature (sub-0.5 bars hold). Sub-knee byte-identical (fixtures now DERIVE from the helper; 0.45→0.755/0.795 pinned). Discriminating test verified RED on old formulas (2 fail: discriminating + envelope) → green with tail; equal-tr tie-conservation pins the #987 coverage comparator; second-stage-engaged pools documented out of scope (position-derived releaseRankScore sorts first there) — matches live evidence that non-engaged pools (patient-safety, opioid, flowchart) are where id-order decided. S3/S4 = C-PR-2 candidates, evidence-gated on the post-merge canary vs #54 baseline (doc/content recall MUST stay 1.0, zero per-case regressions; success signal = rr lift on the headroom cases). Rollback: single revert (helpers + 2 expression sites + 1 map call; no schema/config/cache surface). | Targeted vitest 121/121 (fast-path 11/11 incl. 6 new, retrieval-selection, rag-routing, rag-answer-fallback, ranking-tuning, second-stage); npm run test 3025 passed / 1 known container pdf-budget artifact; lint + typecheck + prettier clean; red-proof executed and recorded; live validation = post-merge canary dispatch (~$1-2) | diff --git a/docs/rag-hybrid-findings-and-todo.md b/docs/rag-hybrid-findings-and-todo.md index 4d93f91ff..1a780aaf0 100644 --- a/docs/rag-hybrid-findings-and-todo.md +++ b/docs/rag-hybrid-findings-and-todo.md @@ -297,15 +297,10 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows audit on `claude/retrieval-correctness`: the headline `least(0.95, 0.56 + text_rank*0.39)` proxy NO LONGER EXISTS — `match_document_chunks_text` already returns `similarity = 0` with hybrid capped at 0.5 and the lexical signal isolated in `lexical_score` (codified in - schema.sql with the "do not fabricate" comment). Since ADDENDUM-4 Phase C, rows whose - text_rank saturates the SQL clamp additionally get a per-candidate hybrid spread inside the - dead (0.48, 0.5) band (`liftSaturatedLexicalChunkHybrid`) so exact ties resolve by raw text - relevance instead of chunk-id order — the sub-0.5 contract holds. What remains synthetic are - three app-side fabricators, all tagged `similarity_origin: "synthetic_text"`: the - document-lookup fast path (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader - (0.58 + confidence·0.28, hybrid ≤ 0.89), and table-fact signal matches (whose imputed - similarity likewise carries a Phase C tail into its dead (0.92, 0.94) band via - `imputedTableFactPrimaries`; hybrid unchanged). Their consumers: + schema.sql with the "do not fabricate" comment). What remains synthetic are three app-side + fabricators, all tagged `similarity_origin: "synthetic_text"`: the document-lookup fast path + (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader (0.58 + confidence·0.28, + hybrid ≤ 0.89), and table-fact signal matches. Their consumers: `evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` / `shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is deliberately load-bearing there, always paired with structural checks like diff --git a/src/lib/rag/rag-candidate-sources.ts b/src/lib/rag/rag-candidate-sources.ts index 7def62ac9..d5208f98b 100644 --- a/src/lib/rag/rag-candidate-sources.ts +++ b/src/lib/rag/rag-candidate-sources.ts @@ -211,12 +211,7 @@ export async function searchTextChunkCandidates(args: { // most-terminal lexical layer surfaces in hybrid_rpc_errors telemetry // instead of silently degrading to zero candidates. Return value unchanged. if (error) recordHybridRpcError(args.telemetry, "match_document_chunks_text", error); - // Phase C: single choke point for every lexical-chunk variant (strict, corrected, - // OR-relaxed, legacy-merged) so saturated rows gain the discriminative hybrid lift - // before mergeSearchResults sees them. - return error || !data?.length - ? ([] as SearchResult[]) - : (data as SearchResult[]).map(liftSaturatedLexicalChunkHybrid); + return error || !data?.length ? ([] as SearchResult[]) : (data as SearchResult[]); }; const variants = args.queryVariants.slice(0, maxTextRpcQueryVariants); @@ -319,60 +314,6 @@ type DocumentLookupChunkRow = { text_rank?: number | null; }; -/** - * ADDENDUM-4 Phase C: per-candidate discrimination for saturated imputed primaries. - * - * The embedding-free text fast path imputes primaries from text_rank through min(text_rank, 1), - * so every candidate with text_rank >= 1 collapses to byte-identical values and unrelated - * documents matching the same terms tie exactly — ordering then falls to chunk-id, which is - * arbitrary (live evidence: canary #53/#54 identical rr@10 on the four fast-path headroom cases). - * saturationTailUnit maps the raw rank's excess above the knee into [0, 1) — pure, strictly - * monotone, set-independent (one candidate's value never depends on the rest of the pool) — and - * the callers scale it into DEAD score bands above the pre-existing caps' functional range - * (table-fact similarity (0.92, 0.94); lexical-chunk hybrid (0.48, 0.5)), where no retrieval - * gate, routing threshold, or confidence bar exists. Below the knee the imputation is - * byte-identical to the historical formulas, and genuinely equal text_ranks still tie, so the - * coverage tie-break contract (retrieval-selection) is preserved. See the Phase C design in the - * PR that introduced this for the full consumer map and envelope proof. - */ -export function saturationTailUnit(textRank: number): number { - if (!Number.isFinite(textRank) || textRank <= 1) return 0; - return (textRank - 1) / (textRank - 1 + 0.5); -} - -/** - * Imputed table-fact primaries (S2). similarity carries the discriminative tail into its dead - * cap band; hybridScore stays byte-identical to the historical formula so selection scores, - * fast-path/coverage gates, and the second-stage trigger are provably unchanged — similarity is - * exactly the tie-break key after hybrid/releaseRankScore in the released-order comparators. - */ -export function imputedTableFactPrimaries(textRank: number): { similarity: number; hybridScore: number } { - return { - similarity: Math.min(0.94, 0.62 + Math.min(textRank, 1) * 0.3 + 0.02 * saturationTailUnit(textRank)), - hybridScore: Math.min(0.97, 0.66 + Math.min(textRank, 1) * 0.3), - }; -} - -/** - * Lexical text-chunk lift (S1). Chunk rows keep similarity === 0 (the no-fabricated-cosine - * contract), so their saturation spread goes into hybrid's dead (0.48, 0.5) band instead — - * still strictly below the 0.5 answer-confidence and 0.64 strong-evidence bars. Guarded by the - * truthful-contract signature, never a float-equality test on the saturated value. - */ -export function liftSaturatedLexicalChunkHybrid(row: SearchResult): SearchResult { - const textRank = typeof row.text_rank === "number" ? row.text_rank : Number(row.text_rank ?? 0); - if ( - row.similarity !== 0 || - typeof row.lexical_score !== "number" || - !Number.isFinite(textRank) || - textRank <= 1 || - typeof row.hybrid_score !== "number" - ) { - return row; - } - return { ...row, hybrid_score: Math.min(0.4999, 0.18 + 0.3 + 0.02 * saturationTailUnit(textRank)) }; -} - export type ChunkSignalMatch = { chunkId: string; similarity: number; @@ -1050,12 +991,11 @@ export async function searchTableFactCandidates(args: { const textRank = Number(fact.text_rank ?? 0); const current = grouped.get(fact.source_chunk_id); const tableFact = { ...fact, text_rank: textRank, match_reason: fact.match_reason ?? "table_row" }; - const imputed = imputedTableFactPrimaries(textRank); const next: ChunkSignalMatch = { chunkId: fact.source_chunk_id, - similarity: imputed.similarity, + similarity: Math.min(0.94, 0.62 + Math.min(textRank, 1) * 0.3), textRank, - hybridScore: imputed.hybridScore, + hybridScore: Math.min(0.97, 0.66 + Math.min(textRank, 1) * 0.3), reason: fact.match_reason ?? "table_row", tableFacts: [...(current?.tableFacts ?? []), tableFact].slice(0, 5), }; diff --git a/src/lib/rag/rag.ts b/src/lib/rag/rag.ts index 335df7cc4..6425d2439 100644 --- a/src/lib/rag/rag.ts +++ b/src/lib/rag/rag.ts @@ -831,11 +831,9 @@ function buildRetrievalDiagnostics(args: { }) { // Lexical-only retrieval rows carry a truthful score contract since migration // 20260713062107_restore_text_fallback_lexical_score: similarity is 0 (no vector - // ran) and hybrid_score stays strictly below 0.5 — 0.48 from SQL, plus a sub-0.5 - // per-candidate saturation spread (liftSaturatedLexicalChunkHybrid, Phase C) that - // only disambiguates rows whose text_rank saturated the SQL clamp — so a keyword - // hit can never masquerade as a moderate/strong cosine match downstream. The honest - // lexical signal lives in lexical_score (0.4..0.99). This gate must therefore read + // ran) and hybrid_score is deliberately capped at 0.48 so a keyword hit can never + // masquerade as a moderate/strong cosine match downstream. The honest lexical + // signal lives in lexical_score (0.4..0.99). This gate must therefore read // max(scoreValue, lexical_score) — reading the capped hybrid_score alone makes // topScore < 0.5 unconditional for every text-fast-path answer, refusing // well-supported documentation lookups whose expected document is at rank 1. diff --git a/tests/rag-fast-path-ordering.test.ts b/tests/rag-fast-path-ordering.test.ts index 654f2c735..f1ed951db 100644 --- a/tests/rag-fast-path-ordering.test.ts +++ b/tests/rag-fast-path-ordering.test.ts @@ -1,10 +1,5 @@ import { describe, expect, it } from "vitest"; import { applySecondStageRerankIfNeeded } from "../src/lib/rag/rag"; -import { - imputedTableFactPrimaries, - liftSaturatedLexicalChunkHybrid, - saturationTailUnit, -} from "../src/lib/rag/rag-candidate-sources"; import { classifyRagQuery, rankClinicalResults } from "../src/lib/clinical-search"; import { selectRetrievalEvidence } from "../src/lib/retrieval-selection"; import { resultsHaveReleaseRankScore, stabilizeReleasedSearchOrder } from "../src/lib/released-search-order"; @@ -25,25 +20,18 @@ import type { SearchResult } from "../src/lib/types"; type QueryClass = ReturnType["queryClass"]; -// Fixtures derive their primaries from the production imputation helper so they stay -// byte-accurate to whatever the fast path actually emits. At the historical default -// text_rank 0.45 (sub-knee) this yields exactly the former hardcoded pair -// similarity 0.755 / hybrid 0.795, keeping every pre-Phase-C guard's -// "byte-identical primaries" premise intact. function imputedResult( overrides: Partial & Pick, ): SearchResult { - const textRank = typeof overrides.text_rank === "number" ? overrides.text_rank : 0.45; - const imputed = imputedTableFactPrimaries(textRank); return { page_number: 5, chunk_index: 0, section_heading: null, image_ids: [], images: [], - similarity: imputed.similarity, - text_rank: textRank, - hybrid_score: imputed.hybridScore, + similarity: 0.755, + text_rank: 0.45, + hybrid_score: 0.795, ...overrides, }; } @@ -395,183 +383,3 @@ describe("text-fast-path ordering under imputed identical primaries", () => { expect(released.map((result) => result.document_id)).toContain("archived-doc"); }); }); - -describe("Phase C saturation-tail primaries", () => { - it("keeps sub-knee imputation byte-identical to the historical formulas", () => { - for (const textRank of [0, 0.1, 0.25, 0.45, 0.7, 0.999, 1]) { - const { similarity, hybridScore } = imputedTableFactPrimaries(textRank); - expect(similarity).toBe(Math.min(0.94, 0.62 + Math.min(textRank, 1) * 0.3)); - expect(hybridScore).toBe(Math.min(0.97, 0.66 + Math.min(textRank, 1) * 0.3)); - expect(saturationTailUnit(textRank)).toBe(0); - } - // The historical hardcoded fixture pair reproduces exactly at the default text_rank. - expect(imputedTableFactPrimaries(0.45)).toEqual({ similarity: 0.755, hybridScore: 0.795 }); - }); - - it("confines the saturated similarity tail to the dead (0.92, 0.94) band and leaves hybrid untouched", () => { - let previous = 0.92; - for (const textRank of [1.0001, 1.05, 1.5, 3, 50]) { - const { similarity, hybridScore } = imputedTableFactPrimaries(textRank); - expect(similarity).toBeGreaterThan(0.92); - expect(similarity).toBeLessThan(0.94); - // hybrid must stay byte-identical so gates, triggers, and selection scores cannot move. - expect(hybridScore).toBe(0.96); - // strict monotonicity in raw text relevance - expect(similarity).toBeGreaterThan(previous); - previous = similarity; - expect(saturationTailUnit(textRank)).toBeGreaterThan(0); - expect(saturationTailUnit(textRank)).toBeLessThan(1); - } - }); - - it("lifts saturated lexical chunks only under the truthful-contract signature and stays below 0.5", () => { - const base = imputedResult({ - id: "chunk-a", - document_id: "doc-a", - title: "Reference Compendium", - file_name: "ReferenceCompendium.pdf", - content: "content", - }); - const lexicalRow: SearchResult = { ...base, similarity: 0, lexical_score: 0.99, hybrid_score: 0.48 }; - - const lifted = liftSaturatedLexicalChunkHybrid({ ...lexicalRow, text_rank: 1.9 }); - expect(lifted.hybrid_score).toBeGreaterThan(0.48); - expect(lifted.hybrid_score).toBeLessThan(0.5); - const liftedLower = liftSaturatedLexicalChunkHybrid({ ...lexicalRow, text_rank: 1.1 }); - expect(liftedLower.hybrid_score).toBeGreaterThan(0.48); - expect(lifted.hybrid_score!).toBeGreaterThan(liftedLower.hybrid_score!); - - // Non-matching signatures pass through unmodified: real-vector rows, sub-knee rows, - // rows without the lexical contract. - expect(liftSaturatedLexicalChunkHybrid({ ...lexicalRow, similarity: 0.5, text_rank: 1.9 }).hybrid_score).toBe(0.48); - expect(liftSaturatedLexicalChunkHybrid({ ...lexicalRow, text_rank: 0.9 }).hybrid_score).toBe(0.48); - expect( - liftSaturatedLexicalChunkHybrid({ ...lexicalRow, lexical_score: undefined, text_rank: 1.9 }).hybrid_score, - ).toBe(0.48); - }); - - it("resolves a saturated tie by raw text_rank instead of chunk-id order (the discriminating case)", () => { - // Both candidates saturate the historical clamp (text_rank >= 1) and tie on every - // pre-Phase-C key: identical content (identical coverage and boost signals), no table - // facts or images, query-token-free titles, no governance metadata — and ids chosen so - // the chunk-id fallback would seat the DISTRACTOR first. Under the pre-Phase-C - // byte-identical primaries this ordering was arbitrary (id order); the saturation tail - // makes the strictly more text-relevant document win. Verified red on the old formulas. - // - // Scope note (matches the live headroom evidence): the tail resolves ties at the - // release comparator, which is consulted when the second-stage rerank does NOT engage — - // exactly the pools where canary #53/#54 show id-order deciding (patient-safety, opioid, - // flowchart). A table_threshold pool with tied top scores always engages the second - // stage, whose position-derived releaseRankScore sorts first, so this case deliberately - // uses a visual-evidence-free document_lookup pool and pins the non-engaged premise. - const query = "What should a patient safety plan include?"; - const { queryClass } = classifyRagQuery(query); - expect(queryClass).toBe("document_lookup"); - const sharedContent = - "A patient safety plan should include warning signs, coping strategies, and emergency contacts."; - - const distractor = imputedResult({ - id: "aa-distractor", - document_id: "aa-generic-doc", - title: "Service Reference Compendium", - file_name: "ServiceReferenceCompendium (NMHS).pdf", - content: sharedContent, - text_rank: 1.05, - }); - const correct = imputedResult({ - id: "zz-correct", - document_id: "zz-relevant-doc", - title: "Ward Reference Compendium", - file_name: "WardReferenceCompendium (SMHS).pdf", - content: sharedContent, - text_rank: 1.8, - }); - - const released = runTextFastPathOrdering({ - query, - queryClass, - candidates: [distractor, correct], - topK: 8, - }); - // Non-engaged premise: no visual evidence in the pool, so release order falls through - // hybrid (tied) to the Phase C similarity spread. - expect(resultsHaveReleaseRankScore(released)).toBe(false); - expect(released[0]?.document_id).toBe("zz-relevant-doc"); - expect(released.map((result) => result.document_id)).toContain("aa-generic-doc"); - - // Arrival order must not matter either. - const releasedReversed = runTextFastPathOrdering({ - query, - queryClass, - candidates: [correct, distractor], - topK: 8, - }); - expect(releasedReversed[0]?.document_id).toBe("zz-relevant-doc"); - }); - - it("still ties genuinely equal text_ranks so the coverage comparator keeps deciding (#987 contract)", () => { - // Identical text_rank -> identical primaries under the tail too; the coverage-richer - // candidate must win regardless of adversarial id order, proving the tail cannot - // starve the content-aware tie-break. - const query = "What CIWA-Ar score threshold requires drug treatment in alcohol withdrawal?"; - const { queryClass } = classifyRagQuery(query); - - const coveragePoor = imputedResult({ - id: "aa-poor-coverage", - document_id: "aa-poor-doc", - title: "Withdrawal Care Overview", - file_name: "WithdrawalCareOverview (NMHS).pdf", - content: "General supportive care principles for patients in alcohol withdrawal.", - text_rank: 1.6, - }); - const coverageRich = imputedResult({ - id: "zz-rich-coverage", - document_id: "zz-rich-doc", - title: "Withdrawal Scale Reference", - file_name: "WithdrawalScaleReference (SMHS).pdf", - content: "A CIWA-Ar score of 10 or more is the threshold that requires drug treatment in alcohol withdrawal.", - text_rank: 1.6, - }); - - expect(imputedTableFactPrimaries(1.6)).toEqual(imputedTableFactPrimaries(1.6)); - const released = runTextFastPathOrdering({ - query, - queryClass, - candidates: [coveragePoor, coverageRich], - topK: 8, - }); - expect(released[0]?.document_id).toBe("zz-rich-doc"); - }); - - it("orders saturated lexical text chunks by raw text_rank through the release pipeline", () => { - const query = "What ANC threshold requires review?"; - const { queryClass } = classifyRagQuery(query); - const sharedContent = "The ANC threshold requires review whenever results fall in the flagged range."; - const lexicalChunk = (id: string, documentId: string, title: string, textRank: number): SearchResult => - liftSaturatedLexicalChunkHybrid({ - ...imputedResult({ - id, - document_id: documentId, - title, - file_name: `${title.replace(/\s+/g, "")}.pdf`, - content: sharedContent, - text_rank: textRank, - }), - similarity: 0, - lexical_score: 0.99, - hybrid_score: 0.48, - }); - - const released = runTextFastPathOrdering({ - query, - queryClass, - candidates: [ - lexicalChunk("aa-low", "aa-low-doc", "Service Reference Compendium", 1.1), - lexicalChunk("zz-high", "zz-high-doc", "Ward Reference Compendium", 1.9), - ], - topK: 8, - }); - expect(released[0]?.document_id).toBe("zz-high-doc"); - expect(released.map((result) => result.document_id)).toContain("aa-low-doc"); - }); -}); From b5c4eb9af0ac60bd838ed868a3778a0715582ee4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 19:45:36 +0000 Subject: [PATCH 2/2] docs(ledger): retain history rows + record the pending-review revert Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- docs/branch-review-ledger.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 6f0a2c433..cb2245c7d 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -653,3 +653,6 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: eval measurement floor, ADDENDUM 4 A-PR-1) | 81ab9696da4b330ca0b2e5519891a9942f90421b | Measurement floor for evidence-gated ranking tuning: canary artifact emission, alias-aware snapshot builder, snapshot provenance/freshness — no ranking behavior change | Closes the three gaps blocking safe tuning (Phase B): (1) eval-canary's golden step now writes the per-case JSON artifact (--json-out decoupled from --json so the tee'd log keeps the human-readable lines the failure-issue analyzer parses) and uploads .local/eval-canary/ via pinned upload-artifact (30-day retention, include-hidden-files for the dot-dir, contents = same class as the already-public step logs: titles/telemetry/220-char previews of the all-public corpus) — snapshot regeneration stops costing a paid dispatch; (2) clinicalDocumentAliases/clinicalContentAliases moved verbatim to shared scripts/lib/clinical-aliases.ts and the snapshot builder grades documentMatch/contentMatch through them (discriminating tests: EMHS agitation title and spelled-out "absolute neutrophil count" grade as hits only via aliases — raw labelMatches pinned false), ending tuner ground truth disagreeing with the live gates; (3) snapshots carry generatedAt + optional sourceRunId, validator accepts them, exactly-36 relaxed to at-least-36 (floor still rejects truncated artifacts; sourceCaseCount + per-case candidate minimums unchanged), and a 30-day freshness test (activates on first regeneration) blocks silent corpus drift. Builder smoke-verified end-to-end on a synthetic 36-case artifact (alias grading + provenance stamped + validator green). Static hyphen audit of all 36 cases' terms: no currently-blocked term (canary #52 = 36/36); residual risk classes documented for the A-PR-2 artifact-grounded pass — punctuation-joined tokens (IM/PO, schizo-affective, post-natal) and inert stem entries (obsess/compuls/hyperactiv/impuls can never match whole-token) that currently ride on whole-word OR-alternates. | Targeted vitest 52/52 (ranking-tuning + eval-retrieval + eval-quality); npm run test 3019 passed / 1 known container-only pdf-budget artifact; lint + typecheck clean; check:github-actions + check:ci-scope PASS; prettier clean; check:production-readiness expected missing-secret FAILs only (demo-mode container); no provider calls — live validation = tonight's scheduled canary emits the first artifact at $0 | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR #1001: A-PR-2 measurement-floor completion) | 3d8f798 + 5b664f8 + e58c827 | First provenance-stamped snapshot regeneration from a live canary artifact + alias tiering docs + fixture-length pin + lithium doc-gate — Phase A of ADDENDUM 4 functionally complete | Artifact chain: user-authorized paid dispatch (canary #53, run 29763761133, 36/36 green, doc_recall 1.0 / content_recall 1.0 — first perfect content recall, ciwa alias confirmed live; mrr@10 0.8644, irrelevant@10 0.1083) emitted the first eval-canary-output artifact (51787 bytes, sha256 5af5b802… verified byte-identical after user transfer into the sandbox — this container cannot download run artifacts). Work: (1) two-tier alias documentation — investigation of the governance-review P3 showed src/lib/eval-document-matching.ts is a deliberately WIDER captured-case tier (e.g. "Clozapine GP Shared Care"); bulk-merge would loosen golden ground truth, so both files now carry cross-referencing do-not-merge headers instead; (2) snapshot case count pinned to live golden fixture length (regeneration instructions in failure message) — closes the coarse-floor P3; (3) snapshot regenerated via the alias-aware builder with --source-run-id provenance: agitation-im-po-options 0→5 graded positives (EMHS alias working on real data), flowchart-next-step confirmed sole zero-positive case; generatedAt promoted to validator-REQUIRED (closes the hand-edit P3); two stale data pins updated (missing-positives 2→1; broad_summary defaults-equality pin dropped — defaults' provenance was the retired snapshot, fresh recommendations are Phase B input); (4) artifact-grounded punctuation audit: 7 joined-token occurrences in top-5 previews — 3 ciwa-ar (alias-covered, incl. line-broken "ciwa- ar"), 4 ORDINARY-PROSE punctuation ("treatment," / "mood," / "(opioid" / "ptsd.[35]") — matcher word-boundary change proposed as its OWN reviewed follow-up per plan (systemic class, not bundled); (5) lithium-therapy-monitoring was the ONLY ungated case (rr@10 hardcoded 0.00 = measurement noise): expectedDocumentSubstrings ["Lithium"] added from live evidence (deliberately broad across the corpus's multiple legitimate lithium guidelines), snapshot rebuilt in lockstep from the same artifact, measured mrr@10 +~0.028 from de-noising. Deferred with reasons: NEW-query fixture cases (saturated-tie shapes, captured rag_query_misses) need live validation before they may gate — unlocked by Phase D-1 branch-eval dispatch or a dedicated validation dispatch; real ordering headroom for Phase B = flowchart 0.20, alcohol-ciwa 0.25, patient-safety 0.33, opioid 0.33, all text_fast_path. | Targeted vitest 76/76 ×3 (after each stage); npm run test 3019 passed / 1 known container-only pdf-budget artifact; prettier clean; freshness gate ACTIVE and green; no provider calls beyond the user-authorized dispatch (~$1-2, ADDENDUM 4 spend now ~$1-2 of ≤$10) | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: B-PR-1 canary dispatch inputs) | see PR head | eval-canary.yml workflow_dispatch gains rag_ranking_config (staged Phase B weight evals, loud malformed-JSON validation to prevent silent fallback corrupting pair comparisons) + ref (Phase D-1 pre-merge branch evals) — schedule runs unaffected (empty inputs = today's behavior) | Tuner (offline, $0) on the A-PR-2 regenerated snapshot recommends 3 constrained per-class improvements (document_lookup titleSectionRelevance→0.9; table_threshold clinicalEvidence→0.95; comparison hybridRelevance→0.95 with proxy mrr 0.833→1.0), all with recall non-regression + zero high-risk hard-negative failures; medication_dose_risk + broad_summary stay neutral. Staged config JSON banked for the live pair; baseline = canary #53 re-gated (mrr@10 ≈0.8922 after lithium de-noising). Targets per approved plan: mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions. Budget: user raised cap to ≤$20 (spent ≈$1-2). Security note: ref input runs branch code with eval secrets — dispatch requires repo write access (sole trusted collaborator), documented in PR risk. | check:github-actions PASS; check:ci-scope PASS; prettier clean; no provider calls (workflow change only) | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (Phase B live pair — no code change; recorded for eval-history completeness) | canary runs 29763761133 (#53 baseline) + 29769050798 (#54 tuned) | ADDENDUM 4 Phase B verdict: staged 3-class tuner weights produced ZERO live movement — NOT adopted per the measured-gain rule | Offline tuner recommendations (document_lookup titleSection→0.9, table_threshold clinicalEvidence→0.95, comparison hybridRelevance→0.95; proxy comparison-mrr 0.833→1.0) staged via the B-PR-1 rag_ranking_config input; #54 log proves the override active (validation step echo). Result vs re-gated baseline: mrr@10 0.8921 vs 0.8922, irrelevant@10 0.1083 vs 0.1083, recalls 1.0/1.0 both, 36/36 both, all four headroom cases byte-identical rr (flowchart 0.20, ciwa-threshold 0.25, patient-safety 0.33, opioid 0.33). Learning: the 5-candidate linear proxy saturates; real headroom lives in fast-path saturated-tie structure (Phase C trigger condition mrr < 0.90 formally met; user authorized C). Nothing to roll back (per-run override). Spend ≈$2-4 of user-raised ≤$20 cap. | Both canary runs green (36/36 + answer-quality); override-active proof in #54 job log; no adoption = no code diff | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: Phase C saturation-tail primaries) | 7572c7f | ADDENDUM 4 Phase C (user-authorized): per-candidate discriminative primaries for saturated fast-path ties — design-agent planned (consumer map + dead-band envelope proof), red-proven, tie-conservation guarded | Mechanism: min(text_rank,1) collapses all tr≥1 candidates to byte-identical imputed primaries; ordering fell to chunk id at release. Fix: saturationTailUnit (pure, monotone, SET-INDEPENDENT — rejected per-query min-max + rank-tier designs for set-dependence/#118 authority risk; rejected full-range log rescale for moving sub-knee values across the 0.62-0.82 gate ladder) scales the excess into DEAD cap bands only: S2 table-fact similarity (0.92, 0.94) with hybrid byte-identical (gates/triggers/selection provably unchanged; similarity = the release tie-break key), S1 lexical-chunk hybrid (0.48, 0.5) behind the truthful-contract signature (sub-0.5 bars hold). Sub-knee byte-identical (fixtures now DERIVE from the helper; 0.45→0.755/0.795 pinned). Discriminating test verified RED on old formulas (2 fail: discriminating + envelope) → green with tail; equal-tr tie-conservation pins the #987 coverage comparator; second-stage-engaged pools documented out of scope (position-derived releaseRankScore sorts first there) — matches live evidence that non-engaged pools (patient-safety, opioid, flowchart) are where id-order decided. S3/S4 = C-PR-2 candidates, evidence-gated on the post-merge canary vs #54 baseline (doc/content recall MUST stay 1.0, zero per-case regressions; success signal = rr lift on the headroom cases). Rollback: single revert (helpers + 2 expression sites + 1 map call; no schema/config/cache surface). | Targeted vitest 121/121 (fast-path 11/11 incl. 6 new, retrieval-selection, rag-routing, rag-answer-fallback, ranking-tuning, second-stage); npm run test 3025 passed / 1 known container pdf-budget artifact; lint + typecheck + prettier clean; red-proof executed and recorded; live validation = post-merge canary dispatch (~$1-2) | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: revert Phase C pending review) | revert of f96217c | User-requested revert of #1004 (Phase C saturation-tail primaries): auto-merge fired before the adversarial rag-retrieval-reviewer pass landed; retrieval code returns to the last reviewed state until that verdict is in | Clean single-commit revert (the documented rollback path — helpers + 2 expression sites + 1 map call; no schema/config/cache surface). Ledger history rows from f96217c retained (docs are append-only record, not behavior). The 19:44Z post-merge canary dispatched on f96217c completes regardless and stands as Phase C's live validation datapoint; re-land decision = reviewer verdict + that pair result together. Offline state of the reverted change remains fully proven (red-proof + 121/121 + envelope tests). | Revert verified by vitest fast-path suite returning to pre-C 5/5 shape expected in CI; ladder on the revert = hosted pr-required |