From 3fafad7424339ac08377a820a9703b2725755406 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 12:10:24 +0000 Subject: [PATCH 1/2] =?UTF-8?q?docs(ledger):=20Option=20A=20wave=20ADOPTED?= =?UTF-8?q?=20=E2=80=94=20run=20#61=20first=20fully-green=2044-case=20cana?= =?UTF-8?q?ry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- docs/branch-review-ledger.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 3c1b13d5..157eca0d 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -678,3 +678,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (parity commit review) | 1aebf02 (fix landed a3b9a54) | rag-retrieval-reviewer on the monitoring evidence-gate parity commit: REQUEST-CHANGES (soft) — P2 reproduced: inflected monitoring kind tokens (monitor\w*/annual(?:ly)?/blood tests?/ecgs?/lfts?) steal sole-dose-value sentences from the dose arm; dose-intent answers then reject the monitoring-kind fact ("Quetiapine is monitored at a dose of 200 mg daily" flipped grounded true→false, source-gap — fails CLOSED, never a wrong dose). P3: monitoring figure escape lacked the dose escape's multi-drug bare-row guard. Clean: over-admission bounded (broad vocab lives in gate/filter only, promotion still corpus-guarded, claim-support unchanged); regex cost negligible; mismatched-unit test relaxation legitimate (synopsis is corpus-verbatim; weeks pin enforced by atom identity + adjacent_context exclusion from both gate and claim corpora). | BOTH FINDINGS FIXED in a3b9a54: kind arm classifies legacy tokens byte-identically and new-inflection-only sentences fall through to the dose arm when they carry a clinicalDoseValuePattern value (both repro sentences pinned as dose-intent tests); multi-drug bare-row guard extended to monitoring_schedule with a discriminating test — red-proven both directions. Reviewer checks: formatting 38/38, focused 644/644; targeting eval deferred to the wave's live canary pair. | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (parity commit review) | 1aebf02 (fix landed a3b9a54) | clinical-governance-reviewer on the same commit: APPROVE-WITH-NITS. P2 (independently converged with the retrieval reviewer's P3): monitoring figure-escape lacked the dose-path multi-drug cross-entity guard — a bare wrong-drug schedule/level row in a multi-drug chunk could be entity-prefixed for a named-drug monitoring query; downstream gates verify text-vs-source presence, never attribution (worked lithium/valproate LFT path traced through finalize). FIXED in a3b9a54 exactly as its smallest-fix prescribed (guard at the :872-881 site now fires for monitoring_schedule; negative multi-drug test added, red-proven). Clean: unsupported figures impossible (admission-only change; promotion corpus guard + numeric verification + claim support all byte-unchanged); conservative failure intact (figure-bearing-only escape, schedule-free refusal pinned); adjacent-context safety held (sourceEvidenceText excludes adjacent_context; weeks refusal confirmed by probe); no PHI/provider/ranking surface. P3s: RAG impact line (present in the PR body — behaviour-change form, correct since the PR also carries the Option A retrieval change); multi-drug negative test (landed in a3b9a54). | Offline guard-chain trace + targeted vitest probes (named-drug guard, bare-figure admission, conservative gap, adjacent refusal). Provider/release gates deferred per confirmation boundary. | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A rescue review) | 0abf3c9 | clinical-governance-reviewer on the S3 escalation rescue: APPROVE-WITH-NITS, no P0/P1. P2 = the mandated live canary pair itself (process gate, declared in the PR body; offline-green + review-approved proven insufficient for this surface 2026-07-20). P3s: multi-drug escalation-query recall edge (titled drug + untitled drug — fast-path return can skip the vector leg; recall limitation, not misattribution, mirrors the pre-existing allowlisted-class tradeoff); reviewer probe files must stay uncommitted (relocated to scratchpad). All six clinical concerns verified safe: wrong-document impossible (alias phrases must appear in the query; per-document alias groups, no cross-drug conflation), conservative availability (S3 purely additive via keyed-union merge; sibling retention test-pinned), live expansion acceptable (title-named correct-entity SOP in every firing shape), fail-closed double layer (adversarial short-circuit precedes the predicate; injection-forge case intent=protocol cannot fire — executed), governance metadata unbypassed (same attachDocumentRankingMetadata + status=indexed + access-scope filters), no PHI/provider/schema surface. | Reviewer checks: escalation-rescue suite 8/8, injection-forge intent derivation executed, static trace of the full S3 chain. Live canary pair = the trust gate, dispatched post-merge. | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A wave verdict — no code change; #1040 merged as cde6c5c) | canary run 29827012719 (#61, main cde6c5c) vs banked #60 (29800029819) | OPTION A WAVE ADOPTED — FIRST FULLY-GREEN 44-CASE CANARY IN PROGRAM HISTORY (Blocking failures: None). (1) Option A payoff EXCEEDED: citation_failure_rate 0.0227→0; the neuroleptic-side-effect-escalation case flipped from wrong-doc→failed-generation→1-citation-fallback to **strong route, successful gpt-5.6-sol generation, passed in 15.4s with no fallback marker** — the rescued S3 retrieval fixed generation itself, not just the citation count; expected_source_hit 0.6364→0.6591. (2) Golden held exactly as the blast-radius analysis promised: 36/36 PASS, content_recall 1.0, mrr@10 0.8921 BYTE-IDENTICAL to the pre-wave baseline (zero ordering movement — no golden case fires the predicate), irrelevant@10 0.1083→0.0917 (slightly better). (3) Parity payoff PARTIAL: monitoring targeting 1/5→2/5 (olanzapine-lai flipped — previously called a retrieval-depth residual; quetiapine-dose also flipped on the dose side); lithium-range (232ch) + metabolic (73ch, byte-identical answer to #60) did NOT flip despite offline-proven fixes — their live chunk sets evidently contain no admissible schedule sentence even under the widened gate → reclassified as retrieval-depth/live-content residuals joining adhd; below the ≥3/5 target but strictly improved, no regression anywhere. Dose 2/5 vs 2/4: same passing count, applicable set grew (new quality-metformin-renal-dosing miss = eval-set churn, not regression). (4) No-worse EXCEEDED: relevance 0.5333→0.6 (the two-step watch-item slide FULLY REVERSED to the #58 level), targeting_rate 0.6667→0.6957, fail_closed 0.9 held, readability/artifact_leaks 1.0, route ceilings 0, grounded 1.0, unsupported_correct 1.0, numeric 0, p95 22.8s, red_result 3/3. Adoption per the measured-gain rule: primary goal achieved, three case flips, relevance recovered, zero regressions. Residual queue: monitoring retrieval-depth trio (lithium-range/metabolic/adhd), E-3d H2 discards, weekly ANSWER_CASE_LIMIT 8→44 raise now unblocked (gate would be green), comparison-class coverage. Wave spend +~$2-4 → Phase E + Option A total ~$12-20 of ≤$20. | Evidence: run #61 job log read (Threshold Status: None; Answer Metrics; neuroleptic diagnostics row; targeting metric_rates + 6-miss list; golden 36 PASS lines + summary). Revert drill NOT triggered. | From fceb444affd055392cf7e9aeab05a054c9f05f64 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 12:12:21 +0000 Subject: [PATCH 2/2] feat(ci): weekly canary guards the full 44-case answer set (I9) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scheduled Sunday canary's ANSWER_CASE_LIMIT default rises 8→44 now that run #61 proved the citation gate green on the full set — both historical blocking reds lived outside the first 8 cases, leaving the weekly gate blind to them. Dispatch input override unchanged; contract test pin updated in lockstep. Est +$1-2/week, plan-authorized (ADDENDUM 5 I9, post-green). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- .github/workflows/eval-canary.yml | 4 ++-- docs/branch-review-ledger.md | 1 + tests/eval-canary-workflow.test.ts | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/eval-canary.yml b/.github/workflows/eval-canary.yml index 2a179c4e..bf75b24c 100644 --- a/.github/workflows/eval-canary.yml +++ b/.github/workflows/eval-canary.yml @@ -17,7 +17,7 @@ on: answer_case_limit: description: "Number of answer-quality cases to run (--limit)" required: false - default: "8" + default: "44" # Staged ranking-weight evaluation (ADDENDUM 4 Phase B): a RAG_RANKING_CONFIG JSON # override evaluated live WITHOUT touching code defaults. Empty = production weights. # Malformed JSON fails the run loudly (validation step below) instead of silently @@ -158,7 +158,7 @@ jobs: - name: Answer-quality subset (live generation) id: answer_quality env: - ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }} + ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '44' }} # Canary-only latency allowance: cross-region runner -> Sydney Supabase # + OpenAI. Release/local eval:quality keeps the strict default gates. EVAL_LATENCY_CONTEXT: cross-region-runner diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 157eca0d..79a7c6c7 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -679,3 +679,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (parity commit review) | 1aebf02 (fix landed a3b9a54) | clinical-governance-reviewer on the same commit: APPROVE-WITH-NITS. P2 (independently converged with the retrieval reviewer's P3): monitoring figure-escape lacked the dose-path multi-drug cross-entity guard — a bare wrong-drug schedule/level row in a multi-drug chunk could be entity-prefixed for a named-drug monitoring query; downstream gates verify text-vs-source presence, never attribution (worked lithium/valproate LFT path traced through finalize). FIXED in a3b9a54 exactly as its smallest-fix prescribed (guard at the :872-881 site now fires for monitoring_schedule; negative multi-drug test added, red-proven). Clean: unsupported figures impossible (admission-only change; promotion corpus guard + numeric verification + claim support all byte-unchanged); conservative failure intact (figure-bearing-only escape, schedule-free refusal pinned); adjacent-context safety held (sourceEvidenceText excludes adjacent_context; weeks refusal confirmed by probe); no PHI/provider/ranking surface. P3s: RAG impact line (present in the PR body — behaviour-change form, correct since the PR also carries the Option A retrieval change); multi-drug negative test (landed in a3b9a54). | Offline guard-chain trace + targeted vitest probes (named-drug guard, bare-figure admission, conservative gap, adjacent refusal). Provider/release gates deferred per confirmation boundary. | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A rescue review) | 0abf3c9 | clinical-governance-reviewer on the S3 escalation rescue: APPROVE-WITH-NITS, no P0/P1. P2 = the mandated live canary pair itself (process gate, declared in the PR body; offline-green + review-approved proven insufficient for this surface 2026-07-20). P3s: multi-drug escalation-query recall edge (titled drug + untitled drug — fast-path return can skip the vector leg; recall limitation, not misattribution, mirrors the pre-existing allowlisted-class tradeoff); reviewer probe files must stay uncommitted (relocated to scratchpad). All six clinical concerns verified safe: wrong-document impossible (alias phrases must appear in the query; per-document alias groups, no cross-drug conflation), conservative availability (S3 purely additive via keyed-union merge; sibling retention test-pinned), live expansion acceptable (title-named correct-entity SOP in every firing shape), fail-closed double layer (adversarial short-circuit precedes the predicate; injection-forge case intent=protocol cannot fire — executed), governance metadata unbypassed (same attachDocumentRankingMetadata + status=indexed + access-scope filters), no PHI/provider/schema surface. | Reviewer checks: escalation-rescue suite 8/8, injection-forge intent derivation executed, static trace of the full S3 chain. Live canary pair = the trust gate, dispatched post-merge. | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A wave verdict — no code change; #1040 merged as cde6c5c) | canary run 29827012719 (#61, main cde6c5c) vs banked #60 (29800029819) | OPTION A WAVE ADOPTED — FIRST FULLY-GREEN 44-CASE CANARY IN PROGRAM HISTORY (Blocking failures: None). (1) Option A payoff EXCEEDED: citation_failure_rate 0.0227→0; the neuroleptic-side-effect-escalation case flipped from wrong-doc→failed-generation→1-citation-fallback to **strong route, successful gpt-5.6-sol generation, passed in 15.4s with no fallback marker** — the rescued S3 retrieval fixed generation itself, not just the citation count; expected_source_hit 0.6364→0.6591. (2) Golden held exactly as the blast-radius analysis promised: 36/36 PASS, content_recall 1.0, mrr@10 0.8921 BYTE-IDENTICAL to the pre-wave baseline (zero ordering movement — no golden case fires the predicate), irrelevant@10 0.1083→0.0917 (slightly better). (3) Parity payoff PARTIAL: monitoring targeting 1/5→2/5 (olanzapine-lai flipped — previously called a retrieval-depth residual; quetiapine-dose also flipped on the dose side); lithium-range (232ch) + metabolic (73ch, byte-identical answer to #60) did NOT flip despite offline-proven fixes — their live chunk sets evidently contain no admissible schedule sentence even under the widened gate → reclassified as retrieval-depth/live-content residuals joining adhd; below the ≥3/5 target but strictly improved, no regression anywhere. Dose 2/5 vs 2/4: same passing count, applicable set grew (new quality-metformin-renal-dosing miss = eval-set churn, not regression). (4) No-worse EXCEEDED: relevance 0.5333→0.6 (the two-step watch-item slide FULLY REVERSED to the #58 level), targeting_rate 0.6667→0.6957, fail_closed 0.9 held, readability/artifact_leaks 1.0, route ceilings 0, grounded 1.0, unsupported_correct 1.0, numeric 0, p95 22.8s, red_result 3/3. Adoption per the measured-gain rule: primary goal achieved, three case flips, relevance recovered, zero regressions. Residual queue: monitoring retrieval-depth trio (lithium-range/metabolic/adhd), E-3d H2 discards, weekly ANSWER_CASE_LIMIT 8→44 raise now unblocked (gate would be green), comparison-class coverage. Wave spend +~$2-4 → Phase E + Option A total ~$12-20 of ≤$20. | Evidence: run #61 job log read (Threshold Status: None; Answer Metrics; neuroleptic diagnostics row; targeting metric_rates + 6-miss list; golden 36 PASS lines + summary). Revert drill NOT triggered. | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: I9 weekly coverage raise) | see PR head | ADDENDUM 5 post-green item I9 (plan-authorized "after reds fixed"): weekly scheduled canary ANSWER_CASE_LIMIT default 8→44 — the Sunday 18:00 UTC cron now guards the FULL answer-quality case set instead of the first 8 (both #57 blocking reds historically lived OUTSIDE the first 8, leaving the weekly gate blind to them). Unblocked by run #61 proving the citation gate green on the full 44. Cost: est +$1-2/week (user-authorized in the plan). Contract test pin updated in lockstep (eval-canary-workflow.test.ts). Dispatch shapes unchanged (input override still wins); operational-risk diff, plain-revert rollback. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS (20/20); eval-canary-workflow contract 4/4; prettier clean; no provider calls | diff --git a/tests/eval-canary-workflow.test.ts b/tests/eval-canary-workflow.test.ts index 69b40d2d..ca97a943 100644 --- a/tests/eval-canary-workflow.test.ts +++ b/tests/eval-canary-workflow.test.ts @@ -11,7 +11,7 @@ describe("eval canary workflow input", () => { }); it("validates the dispatch limit outside shell source and passes it as one quoted argument", () => { - expect(workflow).toContain("ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }}"); + expect(workflow).toContain("ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '44' }}"); expect(workflow).toContain('[[ ! "$ANSWER_CASE_LIMIT" =~ ^[0-9]+$ ]]'); expect(workflow).toContain("ANSWER_CASE_LIMIT < 1 || ANSWER_CASE_LIMIT > 100"); expect(workflow).toContain('--limit "$ANSWER_CASE_LIMIT"');