BigSimmo · BigSimmo · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026
@@ -72,6 +72,14 @@ env:
   RAG_EVAL_PROVIDER_RETRY_ATTEMPTS: 6
   RAG_EVAL_PROVIDER_RETRY_INITIAL_MS: 8000
   RAG_EVAL_PROVIDER_RETRY_MAX_MS: 120000
+  # Cost self-reporting (estimated_cost_usd in eval summaries; was "n/a" with these unset).
+  # Rates = gpt-5.6-terra standard tier per 1M tokens (verified from the OpenAI pricing page
+  # 2026-07-21). The strong model (gpt-5.6-sol) costs 2x these rates and the estimator
+  # applies ONE rate set to all usage, so estimated_cost_usd is a LOWER BOUND on runs where
+  # strong retries fire. Public list prices, not secrets.
+  RAG_EVAL_INPUT_USD_PER_MILLION: "2.50"
+  RAG_EVAL_CACHED_INPUT_USD_PER_MILLION: "0.25"
+  RAG_EVAL_OUTPUT_USD_PER_MILLION: "15.00"
   # The live corpus is all-public (owner_id = NULL). Owner-scoped retrieval
   # fail-closes on a null owner, so the eval must run as the public-owner
   # sentinel. resolveEvalOwnerId (scripts/eval-utils.ts) defaults to that

@@ -664,3 +664,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-21 | claude/x4-sast-gate (PR #1012) | 0d4985e63 | Maturity X4: blocking SAST gate on the untrusted-document parsing surface | Triage-first per workorder: CI-pinned semgrep/semgrep:1.168.0 over worker/**, src/lib/ingestion*.ts, src/lib/extractors, src/app/api/{ingestion,upload} = 0 ERROR findings (24 TS rules/17 files; 55 Python rules/3 files) — gate starts green with no suppressions. Shipped `semgrep-ingestion-gate` job (no continue-on-error; container digest-pinned to the triage-verified 1.168.0 image) with p/python added for the worker OCR stack; repo-wide advisory job untouched. check-github-action-pins.mjs now enforces both policy halves fail-closed (advisory repo-wide / blocking-and-scoped gate / digest-pinned gate container). Residuals: registry-pack mutability accepted for the narrow surface; making the workflow a branch-protection required check is an operator decision outside this PR. | Exact gate command exit 0 in pinned container; check:github-actions (new assertions verified fail-closed); check:ci-scope; yaml-contract vitest 1/1; verify:cheap 3031 tests green. No provider calls (local Docker only). |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from f33890a; PR: Phase E instrument input) | see PR head + follow-up SHA pin | Phase E kickoff (user-approved with budget): eval-canary gains opt-in `answer_quality_eval` dispatch input running the 30-case eval:answer-quality fixture (5 quality metrics + per-intent targeting) — the instrument was previously never run in CI. Dispatch-only, default-off, informational (exit 0, no --targeting-floor): scheduled runs and existing dispatch shapes byte-identical; gates stay owned by eval:quality. Follows the #1003 input-only precedent. | E-1 recon (all $0/read-only) recorded here: canary answer gate samples only 8 of 44 eval:quality cases (ANSWER_CASE_LIMIT default); eval:quality --provider-mode offline is a real $0 harness (provider deleted, deterministic source-only path) usable as an E-3 regression guard; live 30-day answer telemetry (34 answer-path rows): 25 full answers, misses = evidence_gap ×4 (avg 26s spent before gap), provider_incomplete_max_output_tokens ×2 (avg 82s wasted then discarded), provider_generation_failed ×1, retrieval_gap_or_conflict ×1, source_only_no_api ×1; success latency avg 13.1s / p90 25.2s. Headroom classes for E-3: truncation waste (rag.ts:4265 self-heal insufficient live), late evidence-gap detection, p90 latency, thin CI answer coverage. Next: E-2 baseline dispatch (answer_case_limit=44 + answer_quality_eval=true, est $3-8 of user-authorized ≤$20 Phase E envelope) after this merges. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; no provider calls this PR (workflow change only); RAG impact: none — retrieval steps untouched |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from 5b4098d; PR: E-2 baseline record + instrument fix) | see PR head | ADDENDUM Phase E-2 BASELINE BANKED (canary dispatch #57, run 29786560936, main 5b4098d, answer_case_limit=44): golden retrieval 36/36 green in-run (no-regression net held); eval:quality full-44 RED on exactly two gates — citation_failure_rate 0.0227 (1/44: neuroleptic-side-effect-escalation — expected doc never retrieved, generation quality-failed, extractive fallback with 1 citation) and route_ceiling_failure_count 2 (clozapine-anc-withhold-threshold: 13.3s pure retrieval vs 12s extractive budget, RPC 9.3s, zero generation; agitation-arousal-typo-dosing: 25054ms vs 25000ms after provider_timeout ate 22.6s pre-recovery). Green gates: grounded_supported 1.0, unsupported_correct 1.0 (all 14 refusals incl. both prompt-injection probes at 2ms), numeric grounding failures 0, governance danger 0, p95 17.4s. Non-blocking signals: expected_source_hit 0.6136 (both admission-discharge cases miss MHSP.AdmissionCommunityPts.pdf to sibling NMHS/RKPG policies — labeling-vs-ranking question, §3.1 class), source governance warning rate 0.8182 (metadata debt, waivable class). SYSTEMIC E-3 TARGET: ~9 of 19 generation attempts discarded (fast output fails quality gate → extractive fallback wins; ~7 cases carry fast_quality_retry_strong→extractive reasons) = ~half of generation latency+spend wasted. Instrument defect found+fixed this PR: targeting step was skipped after the red gate (GitHub failure-skip semantics) → if: gains !cancelled() so baselines observe red gates. Spend: est ~$1-2 actual (19 OpenAI-request cases; cost rates unset in CI so report shows n/a) of ≤$20 Phase E envelope. Next: cheap re-dispatch (default limit 8 + answer_quality_eval=true) to bank the skipped 30-case targeting baseline. | Gates this PR: check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean. Baseline evidence: job log run 29786560936 (5 failing-case diagnostics + Answer Metrics table read in full) |
+| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from 87815f4; PR: E-3a cost self-reporting) | see PR head + follow-up pin | Phase E-3a wave (all $0 diagnostics complete, recorded here) + I8 fix: eval-canary env gains the three RAG_EVAL_*_USD_PER_MILLION rates (gpt-5.6-terra standard tier, verified from the live OpenAI pricing page 2026-07-21; documented as a LOWER BOUND since gpt-5.6-sol strong retries cost 2x and the estimator applies one rate set) so estimated_cost_usd stops reading n/a in CI. E-3a findings: (1) discarded-generation histogram from run #57 — fast_quality_retry_strong→extractive x6, generation_quality_failed x4, provider_timeout x2; gate sub-reasons fragment_like x2, bad_final_answer_quality x1, ungrounded_extractive_fallback x1 → E-3c targets the fast-attempt-fails-extractive-wins shape first. (2) I2 resolved: routeCeilingExceeded (eval-quality.ts:157/171) keys on the RUNTIME's own route_budget_ms + deadline flag, NOT the cross-region-widened eval gates — clozapine case = 13.3s retrieval vs the runtime's 12s extractive budget, geography-amplified; design → E-3b. (3) FIXTURE-CORPUS DRIFT (major): the live corpus contains ZERO MHSP*-named files; the 44-case fixture's expectedFiles are all MHSP.*/CG.MHSP.* and pass only via the eval-document-matching alias tier. All three expected-doc failures triaged: neuroleptic-side-effect-escalation = REAL retrieval-coverage gap (alias exists; Neuroleptic Side Effects (AKG).pdf indexed 11 chunks, 3 contain escalat%, yet answer retrieval returned a single off-point source) → protected-path item awaiting user go-ahead; admission-discharge x2 = top-5 preference for sibling NMHS/RKPG policies over the existing aliased AKG doc → labeling-vs-ranking decision presented to user. No fixture/alias edits made (ground truth requires sign-off). | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; Supabase access read-only (documents/document_chunks SELECTs); OpenAI pricing page read via WebFetch |