BigSimmo · BigSimmo · Jul 20, 2026 · Jul 20, 2026 · Jul 20, 2026
@@ -18,6 +18,21 @@ on:
         description: "Number of answer-quality cases to run (--limit)"
         required: false
         default: "8"
+      # Staged ranking-weight evaluation (ADDENDUM 4 Phase B): a RAG_RANKING_CONFIG JSON
+      # override evaluated live WITHOUT touching code defaults. Empty = production weights.
+      # Malformed JSON fails the run loudly (validation step below) instead of silently
+      # reverting to defaults and corrupting a baseline/post comparison.
+      rag_ranking_config:
+        description: "Optional RAG_RANKING_CONFIG JSON override (staged ranking weights); empty = production defaults"
+        required: false
+        default: ""
+      # Pre-merge branch evals (Phase D-1): run the golden eval from a branch so ranking PRs
+      # can gather live proof BEFORE merging. Dispatch requires repo write access, so refs are
+      # limited to trusted collaborators' branches. Empty = the dispatched/default branch.
+      ref:
+        description: "Optional git ref (branch/SHA) to evaluate; empty = default branch"
+        required: false
+        default: ""
   schedule:
     # Sunday 18:00 UTC = Monday 02:00 Australia/Perth — off-peak for clinicians.
     - cron: "0 18 * * 0"
@@ -39,6 +54,9 @@ env:
   NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY: placeholder-ci-anon-key
   SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }}
   OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+  # Empty for schedule runs and default dispatches (resolveRankingConfig treats empty as
+  # unset); non-empty only for staged ranking-weight evals via the dispatch input.
+  RAG_RANKING_CONFIG: ${{ github.event.inputs.rag_ranking_config || '' }}
   # Pace golden retrieval so forced-vector probes do not trip transient 429s
   # after a long text-fast-path run (see vector-ptsd history in eval-canary).
   RAG_EVAL_CASE_DELAY_MS: 2000
@@ -65,6 +83,24 @@ jobs:
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
         with:
           persist-credentials: false
+          # Empty ref = checkout's default (the triggering sha) — schedule runs unaffected.
+          ref: ${{ github.event.inputs.ref || '' }}
+
+      # Fail loudly on a malformed override: resolveRankingConfig (src/lib/ranking-config.ts)
+      # silently falls back to production defaults on bad JSON, which would turn a staged-weights
+      # eval into an accidental baseline and corrupt the pair comparison.
+      - name: Validate ranking-config override
+        id: validate_override
+        if: github.event.inputs.rag_ranking_config != ''
+        run: |
+          node -e '
+            const raw = process.env.RAG_RANKING_CONFIG;
+            const parsed = JSON.parse(raw);
+            if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
+              throw new Error("rag_ranking_config must be a JSON object");
+            }
+            console.log("RAG_RANKING_CONFIG override active:", JSON.stringify(parsed));
+          '
 
       - name: Preflight required secrets
         id: preflight

@@ -652,3 +652,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: matrix-remainder fixes) | ec070ccb20b9a067216e64e8399d9cd795027024 | Run-4012 remainder: 7 root-caused fixes (SW navigation hijack, WebKit focus/forced-colors/stale-style, Firefox tab order, CIWA alias, helper extraction) | Agent-diagnosed with probe evidence: (1) production-build PWA worker registers in EVERY matrix test (pwa-lifecycle.tsx:287 NODE_ENV gate), clients.claim()s the page and serves all navigations — Chromium probe proved SW-served reloads bypass page.route entirely (routeSawNav=[]); Playwright-Firefox wedges on its only two reloads (ui-smoke 2206/2932, both deterministic) → playwright.config.ts serviceWorkers:'block' + ui-pwa 'allow' opt-in (offline/CacheStorage journey verified passing locally under the opt-in). (2) Mode-menu dismissal relied solely on wrapper focusout; WebKit Tab navigation can move focus nowhere (links excluded) or wrap into the menu → keydown Tab-close in handleModeTriggerKeyDown (agent verified no test depends on old forward-Tab behavior). (3) WebKit has no forced-colors implementation → capability skip on the token-remap test. (4) WebKit stale :disabled computed style feeds axe a phantom 1.93 contrast for the re-enabled Previous button (blend arithmetic exact: 0.4×#475467+0.6×#fff=#b5bbc2) → toBeEnabled+opacity-1 pin before the scan, failure-at-pin = direct proof. (5) Firefox includes scrollable containers in tab order (sheet body scrollHeight 1125 vs 707 measured) → conditional step-over. (6) canary #50/#51 CIWA failure root cause: whitespace-delimited textContainsClinicalTerm can never match hyphenated 'CIWA-Ar' though the dosing-table region ranks top-5 → clinicalContentAliases ciwa:[ciwa, ciwa-ar] (plan-authorized fixture-alias route; content substance unchanged). (7) tests/helpers/zero-touch.ts shared helper replaces six #995 inline stubs (CodeRabbit follow-up). NOT claimed: ui-stress 409 overflow, ui-tools 2087 navigation race, webkit answer-flow subset — next matrix run (post-merge dispatch) measures these vs the 28-failure baseline; ui-smoke 2932 webkit-side is a distinct mock-data class deliberately deferred pending that run. | Targeted vitest 25/25 (eval-retrieval + search-command-surface); npm run test 3012 passed / 1 known container pdf-budget artifact; verify:cheap green to same artifact; build + client-bundle scan PASS; prettier/eslint/typecheck clean; chromium ui-pwa under new SW config: offline/CacheStorage privacy journey PASS, installability = known container in-incognito artifact; UI verification not run locally beyond that: chromium ui suites carry documented container artifacts — hosted ui-critical/ui-advisory + next matrix authoritative |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: eval measurement floor, ADDENDUM 4 A-PR-1) | 81ab9696da4b330ca0b2e5519891a9942f90421b | Measurement floor for evidence-gated ranking tuning: canary artifact emission, alias-aware snapshot builder, snapshot provenance/freshness — no ranking behavior change | Closes the three gaps blocking safe tuning (Phase B): (1) eval-canary's golden step now writes the per-case JSON artifact (--json-out decoupled from --json so the tee'd log keeps the human-readable lines the failure-issue analyzer parses) and uploads .local/eval-canary/ via pinned upload-artifact (30-day retention, include-hidden-files for the dot-dir, contents = same class as the already-public step logs: titles/telemetry/220-char previews of the all-public corpus) — snapshot regeneration stops costing a paid dispatch; (2) clinicalDocumentAliases/clinicalContentAliases moved verbatim to shared scripts/lib/clinical-aliases.ts and the snapshot builder grades documentMatch/contentMatch through them (discriminating tests: EMHS agitation title and spelled-out "absolute neutrophil count" grade as hits only via aliases — raw labelMatches pinned false), ending tuner ground truth disagreeing with the live gates; (3) snapshots carry generatedAt + optional sourceRunId, validator accepts them, exactly-36 relaxed to at-least-36 (floor still rejects truncated artifacts; sourceCaseCount + per-case candidate minimums unchanged), and a 30-day freshness test (activates on first regeneration) blocks silent corpus drift. Builder smoke-verified end-to-end on a synthetic 36-case artifact (alias grading + provenance stamped + validator green). Static hyphen audit of all 36 cases' terms: no currently-blocked term (canary #52 = 36/36); residual risk classes documented for the A-PR-2 artifact-grounded pass — punctuation-joined tokens (IM/PO, schizo-affective, post-natal) and inert stem entries (obsess/compuls/hyperactiv/impuls can never match whole-token) that currently ride on whole-word OR-alternates. | Targeted vitest 52/52 (ranking-tuning + eval-retrieval + eval-quality); npm run test 3019 passed / 1 known container-only pdf-budget artifact; lint + typecheck clean; check:github-actions + check:ci-scope PASS; prettier clean; check:production-readiness expected missing-secret FAILs only (demo-mode container); no provider calls — live validation = tonight's scheduled canary emits the first artifact at $0 |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR #1001: A-PR-2 measurement-floor completion) | 3d8f798 + 5b664f8 + e58c827 | First provenance-stamped snapshot regeneration from a live canary artifact + alias tiering docs + fixture-length pin + lithium doc-gate — Phase A of ADDENDUM 4 functionally complete | Artifact chain: user-authorized paid dispatch (canary #53, run 29763761133, 36/36 green, doc_recall 1.0 / content_recall 1.0 — first perfect content recall, ciwa alias confirmed live; mrr@10 0.8644, irrelevant@10 0.1083) emitted the first eval-canary-output artifact (51787 bytes, sha256 5af5b802… verified byte-identical after user transfer into the sandbox — this container cannot download run artifacts). Work: (1) two-tier alias documentation — investigation of the governance-review P3 showed src/lib/eval-document-matching.ts is a deliberately WIDER captured-case tier (e.g. "Clozapine GP Shared Care"); bulk-merge would loosen golden ground truth, so both files now carry cross-referencing do-not-merge headers instead; (2) snapshot case count pinned to live golden fixture length (regeneration instructions in failure message) — closes the coarse-floor P3; (3) snapshot regenerated via the alias-aware builder with --source-run-id provenance: agitation-im-po-options 0→5 graded positives (EMHS alias working on real data), flowchart-next-step confirmed sole zero-positive case; generatedAt promoted to validator-REQUIRED (closes the hand-edit P3); two stale data pins updated (missing-positives 2→1; broad_summary defaults-equality pin dropped — defaults' provenance was the retired snapshot, fresh recommendations are Phase B input); (4) artifact-grounded punctuation audit: 7 joined-token occurrences in top-5 previews — 3 ciwa-ar (alias-covered, incl. line-broken "ciwa- ar"), 4 ORDINARY-PROSE punctuation ("treatment," / "mood," / "(opioid" / "ptsd.[35]") — matcher word-boundary change proposed as its OWN reviewed follow-up per plan (systemic class, not bundled); (5) lithium-therapy-monitoring was the ONLY ungated case (rr@10 hardcoded 0.00 = measurement noise): expectedDocumentSubstrings ["Lithium"] added from live evidence (deliberately broad across the corpus's multiple legitimate lithium guidelines), snapshot rebuilt in lockstep from the same artifact, measured mrr@10 +~0.028 from de-noising. Deferred with reasons: NEW-query fixture cases (saturated-tie shapes, captured rag_query_misses) need live validation before they may gate — unlocked by Phase D-1 branch-eval dispatch or a dedicated validation dispatch; real ordering headroom for Phase B = flowchart 0.20, alcohol-ciwa 0.25, patient-safety 0.33, opioid 0.33, all text_fast_path. | Targeted vitest 76/76 ×3 (after each stage); npm run test 3019 passed / 1 known container-only pdf-budget artifact; prettier clean; freshness gate ACTIVE and green; no provider calls beyond the user-authorized dispatch (~$1-2, ADDENDUM 4 spend now ~$1-2 of ≤$10) |
+| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted; PR: B-PR-1 canary dispatch inputs) | see PR head | eval-canary.yml workflow_dispatch gains rag_ranking_config (staged Phase B weight evals, loud malformed-JSON validation to prevent silent fallback corrupting pair comparisons) + ref (Phase D-1 pre-merge branch evals) — schedule runs unaffected (empty inputs = today's behavior) | Tuner (offline, $0) on the A-PR-2 regenerated snapshot recommends 3 constrained per-class improvements (document_lookup titleSectionRelevance→0.9; table_threshold clinicalEvidence→0.95; comparison hybridRelevance→0.95 with proxy mrr 0.833→1.0), all with recall non-regression + zero high-risk hard-negative failures; medication_dose_risk + broad_summary stay neutral. Staged config JSON banked for the live pair; baseline = canary #53 re-gated (mrr@10 ≈0.8922 after lithium de-noising). Targets per approved plan: mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions. Budget: user raised cap to ≤$20 (spent ≈$1-2). Security note: ref input runs branch code with eval secrets — dispatch requires repo write access (sole trusted collaborator), documented in PR risk. | check:github-actions PASS; check:ci-scope PASS; prettier clean; no provider calls (workflow change only) |