Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/branch-review-ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -655,3 +655,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: B-PR-1 canary dispatch inputs) | see PR head | eval-canary.yml workflow_dispatch gains rag_ranking_config (staged Phase B weight evals, loud malformed-JSON validation to prevent silent fallback corrupting pair comparisons) + ref (Phase D-1 pre-merge branch evals) — schedule runs unaffected (empty inputs = today's behavior) | Tuner (offline, $0) on the A-PR-2 regenerated snapshot recommends 3 constrained per-class improvements (document_lookup titleSectionRelevance→0.9; table_threshold clinicalEvidence→0.95; comparison hybridRelevance→0.95 with proxy mrr 0.833→1.0), all with recall non-regression + zero high-risk hard-negative failures; medication_dose_risk + broad_summary stay neutral. Staged config JSON banked for the live pair; baseline = canary #53 re-gated (mrr@10 ≈0.8922 after lithium de-noising). Targets per approved plan: mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions. Budget: user raised cap to ≤$20 (spent ≈$1-2). Security note: ref input runs branch code with eval secrets — dispatch requires repo write access (sole trusted collaborator), documented in PR risk. | check:github-actions PASS; check:ci-scope PASS; prettier clean; no provider calls (workflow change only) |
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (Phase B live pair — no code change; recorded for eval-history completeness) | canary runs 29763761133 (#53 baseline) + 29769050798 (#54 tuned) | ADDENDUM 4 Phase B verdict: staged 3-class tuner weights produced ZERO live movement — NOT adopted per the measured-gain rule | Offline tuner recommendations (document_lookup titleSection→0.9, table_threshold clinicalEvidence→0.95, comparison hybridRelevance→0.95; proxy comparison-mrr 0.833→1.0) staged via the B-PR-1 rag_ranking_config input; #54 log proves the override active (validation step echo). Result vs re-gated baseline: mrr@10 0.8921 vs 0.8922, irrelevant@10 0.1083 vs 0.1083, recalls 1.0/1.0 both, 36/36 both, all four headroom cases byte-identical rr (flowchart 0.20, ciwa-threshold 0.25, patient-safety 0.33, opioid 0.33). Learning: the 5-candidate linear proxy saturates; real headroom lives in fast-path saturated-tie structure (Phase C trigger condition mrr < 0.90 formally met; user authorized C). Nothing to roll back (per-run override). Spend ≈$2-4 of user-raised ≤$20 cap. | Both canary runs green (36/36 + answer-quality); override-active proof in #54 job log; no adoption = no code diff |
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: Phase C saturation-tail primaries) | 7572c7f | ADDENDUM 4 Phase C (user-authorized): per-candidate discriminative primaries for saturated fast-path ties — design-agent planned (consumer map + dead-band envelope proof), red-proven, tie-conservation guarded | Mechanism: min(text_rank,1) collapses all tr≥1 candidates to byte-identical imputed primaries; ordering fell to chunk id at release. Fix: saturationTailUnit (pure, monotone, SET-INDEPENDENT — rejected per-query min-max + rank-tier designs for set-dependence/#118 authority risk; rejected full-range log rescale for moving sub-knee values across the 0.62-0.82 gate ladder) scales the excess into DEAD cap bands only: S2 table-fact similarity (0.92, 0.94) with hybrid byte-identical (gates/triggers/selection provably unchanged; similarity = the release tie-break key), S1 lexical-chunk hybrid (0.48, 0.5) behind the truthful-contract signature (sub-0.5 bars hold). Sub-knee byte-identical (fixtures now DERIVE from the helper; 0.45→0.755/0.795 pinned). Discriminating test verified RED on old formulas (2 fail: discriminating + envelope) → green with tail; equal-tr tie-conservation pins the #987 coverage comparator; second-stage-engaged pools documented out of scope (position-derived releaseRankScore sorts first there) — matches live evidence that non-engaged pools (patient-safety, opioid, flowchart) are where id-order decided. S3/S4 = C-PR-2 candidates, evidence-gated on the post-merge canary vs #54 baseline (doc/content recall MUST stay 1.0, zero per-case regressions; success signal = rr lift on the headroom cases). Rollback: single revert (helpers + 2 expression sites + 1 map call; no schema/config/cache surface). | Targeted vitest 121/121 (fast-path 11/11 incl. 6 new, retrieval-selection, rag-routing, rag-answer-fallback, ranking-tuning, second-stage); npm run test 3025 passed / 1 known container pdf-budget artifact; lint + typecheck + prettier clean; red-proof executed and recorded; live validation = post-merge canary dispatch (~$1-2) |
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: revert Phase C pending review) | revert of f96217c | User-requested revert of #1004 (Phase C saturation-tail primaries): auto-merge fired before the adversarial rag-retrieval-reviewer pass landed; retrieval code returns to the last reviewed state until that verdict is in | Clean single-commit revert (the documented rollback path — helpers + 2 expression sites + 1 map call; no schema/config/cache surface). Ledger history rows from f96217c retained (docs are append-only record, not behavior). The 19:44Z post-merge canary dispatched on f96217c completes regardless and stands as Phase C's live validation datapoint; re-land decision = reviewer verdict + that pair result together. Offline state of the reverted change remains fully proven (red-proof + 121/121 + envelope tests). | Revert verified by vitest fast-path suite returning to pre-C 5/5 shape expected in CI; ladder on the revert = hosted pr-required |
13 changes: 4 additions & 9 deletions docs/rag-hybrid-findings-and-todo.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,15 +297,10 @@ denied to set parameter`)** — the RC11 blocker. The only method hosted allows
audit on `claude/retrieval-correctness`: the headline `least(0.95, 0.56 + text_rank*0.39)`
proxy NO LONGER EXISTS — `match_document_chunks_text` already returns `similarity = 0` with
hybrid capped at 0.5 and the lexical signal isolated in `lexical_score` (codified in
schema.sql with the "do not fabricate" comment). Since ADDENDUM-4 Phase C, rows whose
text_rank saturates the SQL clamp additionally get a per-candidate hybrid spread inside the
dead (0.48, 0.5) band (`liftSaturatedLexicalChunkHybrid`) so exact ties resolve by raw text
relevance instead of chunk-id order — the sub-0.5 contract holds. What remains synthetic are
three app-side fabricators, all tagged `similarity_origin: "synthetic_text"`: the
document-lookup fast path (0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader
(0.58 + confidence·0.28, hybrid ≤ 0.89), and table-fact signal matches (whose imputed
similarity likewise carries a Phase C tail into its dead (0.92, 0.94) band via
`imputedTableFactPrimaries`; hybrid unchanged). Their consumers:
schema.sql with the "do not fabricate" comment). What remains synthetic are three app-side
fabricators, all tagged `similarity_origin: "synthetic_text"`: the document-lookup fast path
(0.58 + documentScore, hybrid ≤ 0.94), memory-card chunk loader (0.58 + confidence·0.28,
hybrid ≤ 0.89), and table-fact signal matches. Their consumers:
`evaluateEvidenceCoverageGate` / `shouldReturnTextFastPath` / `chooseAnswerRoute` /
`shouldUseExtractiveAnswer` (thresholds 0.32–0.76 — the fabricated 0.58 floor is
deliberately load-bearing there, always paired with structural checks like
Expand Down
66 changes: 3 additions & 63 deletions src/lib/rag/rag-candidate-sources.ts
Original file line number Diff line number Diff line change
Expand Up @@ -211,12 +211,7 @@ export async function searchTextChunkCandidates(args: {
// most-terminal lexical layer surfaces in hybrid_rpc_errors telemetry
// instead of silently degrading to zero candidates. Return value unchanged.
if (error) recordHybridRpcError(args.telemetry, "match_document_chunks_text", error);
// Phase C: single choke point for every lexical-chunk variant (strict, corrected,
// OR-relaxed, legacy-merged) so saturated rows gain the discriminative hybrid lift
// before mergeSearchResults sees them.
return error || !data?.length
? ([] as SearchResult[])
: (data as SearchResult[]).map(liftSaturatedLexicalChunkHybrid);
return error || !data?.length ? ([] as SearchResult[]) : (data as SearchResult[]);
};

const variants = args.queryVariants.slice(0, maxTextRpcQueryVariants);
Expand Down Expand Up @@ -319,60 +314,6 @@ type DocumentLookupChunkRow = {
text_rank?: number | null;
};

/**
* ADDENDUM-4 Phase C: per-candidate discrimination for saturated imputed primaries.
*
* The embedding-free text fast path imputes primaries from text_rank through min(text_rank, 1),
* so every candidate with text_rank >= 1 collapses to byte-identical values and unrelated
* documents matching the same terms tie exactly — ordering then falls to chunk-id, which is
* arbitrary (live evidence: canary #53/#54 identical rr@10 on the four fast-path headroom cases).
* saturationTailUnit maps the raw rank's excess above the knee into [0, 1) — pure, strictly
* monotone, set-independent (one candidate's value never depends on the rest of the pool) — and
* the callers scale it into DEAD score bands above the pre-existing caps' functional range
* (table-fact similarity (0.92, 0.94); lexical-chunk hybrid (0.48, 0.5)), where no retrieval
* gate, routing threshold, or confidence bar exists. Below the knee the imputation is
* byte-identical to the historical formulas, and genuinely equal text_ranks still tie, so the
* coverage tie-break contract (retrieval-selection) is preserved. See the Phase C design in the
* PR that introduced this for the full consumer map and envelope proof.
*/
export function saturationTailUnit(textRank: number): number {
if (!Number.isFinite(textRank) || textRank <= 1) return 0;
return (textRank - 1) / (textRank - 1 + 0.5);
}

/**
* Imputed table-fact primaries (S2). similarity carries the discriminative tail into its dead
* cap band; hybridScore stays byte-identical to the historical formula so selection scores,
* fast-path/coverage gates, and the second-stage trigger are provably unchanged — similarity is
* exactly the tie-break key after hybrid/releaseRankScore in the released-order comparators.
*/
export function imputedTableFactPrimaries(textRank: number): { similarity: number; hybridScore: number } {
return {
similarity: Math.min(0.94, 0.62 + Math.min(textRank, 1) * 0.3 + 0.02 * saturationTailUnit(textRank)),
hybridScore: Math.min(0.97, 0.66 + Math.min(textRank, 1) * 0.3),
};
}

/**
* Lexical text-chunk lift (S1). Chunk rows keep similarity === 0 (the no-fabricated-cosine
* contract), so their saturation spread goes into hybrid's dead (0.48, 0.5) band instead —
* still strictly below the 0.5 answer-confidence and 0.64 strong-evidence bars. Guarded by the
* truthful-contract signature, never a float-equality test on the saturated value.
*/
export function liftSaturatedLexicalChunkHybrid(row: SearchResult): SearchResult {
const textRank = typeof row.text_rank === "number" ? row.text_rank : Number(row.text_rank ?? 0);
if (
row.similarity !== 0 ||
typeof row.lexical_score !== "number" ||
!Number.isFinite(textRank) ||
textRank <= 1 ||
typeof row.hybrid_score !== "number"
) {
return row;
}
return { ...row, hybrid_score: Math.min(0.4999, 0.18 + 0.3 + 0.02 * saturationTailUnit(textRank)) };
}

export type ChunkSignalMatch = {
chunkId: string;
similarity: number;
Expand Down Expand Up @@ -1050,12 +991,11 @@ export async function searchTableFactCandidates(args: {
const textRank = Number(fact.text_rank ?? 0);
const current = grouped.get(fact.source_chunk_id);
const tableFact = { ...fact, text_rank: textRank, match_reason: fact.match_reason ?? "table_row" };
const imputed = imputedTableFactPrimaries(textRank);
const next: ChunkSignalMatch = {
chunkId: fact.source_chunk_id,
similarity: imputed.similarity,
similarity: Math.min(0.94, 0.62 + Math.min(textRank, 1) * 0.3),
textRank,
hybridScore: imputed.hybridScore,
hybridScore: Math.min(0.97, 0.66 + Math.min(textRank, 1) * 0.3),
reason: fact.match_reason ?? "table_row",
tableFacts: [...(current?.tableFacts ?? []), tableFact].slice(0, 5),
};
Expand Down
8 changes: 3 additions & 5 deletions src/lib/rag/rag.ts
Original file line number Diff line number Diff line change
Expand Up @@ -831,11 +831,9 @@ function buildRetrievalDiagnostics(args: {
}) {
// Lexical-only retrieval rows carry a truthful score contract since migration
// 20260713062107_restore_text_fallback_lexical_score: similarity is 0 (no vector
// ran) and hybrid_score stays strictly below 0.5 — 0.48 from SQL, plus a sub-0.5
// per-candidate saturation spread (liftSaturatedLexicalChunkHybrid, Phase C) that
// only disambiguates rows whose text_rank saturated the SQL clamp — so a keyword
// hit can never masquerade as a moderate/strong cosine match downstream. The honest
// lexical signal lives in lexical_score (0.4..0.99). This gate must therefore read
// ran) and hybrid_score is deliberately capped at 0.48 so a keyword hit can never
// masquerade as a moderate/strong cosine match downstream. The honest lexical
// signal lives in lexical_score (0.4..0.99). This gate must therefore read
// max(scoreValue, lexical_score) — reading the capped hybrid_score alone makes
// topScore < 0.5 unconditional for every text-fast-path answer, refusing
// well-supported documentation lookups whose expected document is at rank 1.
Expand Down
Loading
Loading