From 2e0040ffed79edffd3e33cfd4faeadb4f6517c3b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 23:58:23 +0000 Subject: [PATCH 1/2] docs(issues): capture post-program follow-ups #012-#017 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- docs/outstanding-issues.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index fbda62d1..44980535 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -27,7 +27,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope - Resolving an item moves its row to **Resolved / archive** with the date and a one-line outcome — rows are archived, not deleted, so the history stays auditable. - + ## Open items @@ -42,6 +42,12 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | | #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | | #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | +| #012 | P2 | task | Monitoring retrieval-depth trio misses schedules | quality-lithium-monitoring-range, quality-antipsychotic-metabolic-monitoring, quality-adhd-medication-monitoring still fail targeting ("no schedule/interval") after the #1040 gate parity — their live chunk sets apparently surface no admissible schedule sentence. Diagnose via read-only chunk queries + run #61 dump artifact (30d retention) to split corpus-content vs retrieval-depth vs fact-extraction; fix shape TBD after diagnosis. A retrieval-side fix is a protected surface (canary pair, fresh ~$2-4 authorization). | run #61 targeting misses; branch-review-ledger 2026-07-21 | 2026-07-21 | +| #013 | P2 | task | Comparison-class coverage: admission doc never in top-5 | admission-discharge-comparison misses its expected admission-side doc on every canary (#57-#61) — top-5 carries discharge-planning + unrelated docs only. Same "why doesn't retrieval surface X" species as #012; diagnose jointly. Ground-truth already widened (#1020); the residual is genuine comparison-class retrieval coverage. | runs #57-#61; PR #1020 replay verdict | 2026-07-21 | +| #014 | P3 | task | eval:quality cost readout stuck at n/a | RAG_EVAL_*_USD_PER_MILLION envs are set in eval-canary.yml (PR #1017) and the targeting step consumes them, but the 44-case eval:quality report still prints "Estimated cost USD: n/a" — the estimator does not consume the rates on that path. Wire it so every canary self-reports spend. | run #60/#61 Answer Metrics tables; scripts/eval-quality.ts | 2026-07-21 | +| #015 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | ~~6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (~~$2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | +| #016 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | +| #017 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage:312, firefox ui-formulation:91 — diagnose only if reproduced), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | ## Resolved / archive From 2ed9ce125d5d20b581d32b50a47daa61e89d6175 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Jul 2026 00:00:55 +0000 Subject: [PATCH 2/2] fix(eval): zero-usage cases cost $0, not n/a, when rates are configured (#014) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 44-case eval:quality report printed 'Estimated cost USD: n/a' even with the RAG_EVAL_*_USD_PER_MILLION rates set (PR #1017): every extractive or unsupported case makes no provider call, its per-case estimate was null, and the all-or-nothing total nulled on any null case. A no-usage case costs exactly $0 when rates are configured — null stays reserved for rates-unconfigured, so the total keeps meaning 'cannot estimate'. Aggregation pin added. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- scripts/eval-quality.ts | 7 ++++++- tests/eval-quality.test.ts | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/scripts/eval-quality.ts b/scripts/eval-quality.ts index 47dd71e8..6d829f89 100644 --- a/scripts/eval-quality.ts +++ b/scripts/eval-quality.ts @@ -12,6 +12,7 @@ import { type GoldenRetrievalResult, } from "./eval-retrieval"; import { + configuredCostRates, estimateCostUsd, loadAdminClient, percentile, @@ -1137,7 +1138,11 @@ async function runRagQualityCases(args: { routingReason: answer.routingReason, timings, routeCeilingExceeded, - estimatedCostUsd: hasOpenAIUsage ? estimateCostUsd(openAIUsage) : null, + // A case with no provider call costs exactly $0 when rates are + // configured; null stays reserved for rates-unconfigured so the + // all-or-nothing total keeps meaning "cannot estimate" instead of being + // tripped by every extractive/unsupported case (issue #014). + estimatedCostUsd: hasOpenAIUsage ? estimateCostUsd(openAIUsage) : configuredCostRates() ? 0 : null, openAIRequestIds: openAIRequestIds.length > 0 ? openAIRequestIds : undefined, openAIUsage: hasOpenAIUsage ? openAIUsage : undefined, }); diff --git a/tests/eval-quality.test.ts b/tests/eval-quality.test.ts index 2161ede7..9f4c311e 100644 --- a/tests/eval-quality.test.ts +++ b/tests/eval-quality.test.ts @@ -658,6 +658,21 @@ describe("eval quality reporting", () => { expect(report.blocking_threshold_failures).toContain("RAG route_ceiling_failure_count 1 above 0"); }); + it("sums estimated cost across priced and zero-cost cases, going n/a only when a case cannot be estimated", () => { + // Issue #014: extractive/unsupported cases make no provider call and cost + // exactly $0 when rates are configured — they must not null the run total. + // null stays reserved for "rates unconfigured" (cannot estimate). + const priced = ragResult({ estimatedCostUsd: 0.002 }); + const zeroCost = ragResult({ estimatedCostUsd: 0 }); + const unpriced = ragResult({ estimatedCostUsd: null }); + + const summed = buildEvalQualityReport({ retrievalResults: [], ragResults: [priced, zeroCost] }); + expect(summed.rag.summary.estimated_cost_usd).toBe(0.002); + + const unknown = buildEvalQualityReport({ retrievalResults: [], ragResults: [priced, zeroCost, unpriced] }); + expect(unknown.rag.summary.estimated_cost_usd).toBeNull(); + }); + it("accepts an offline source-only result only when expected sources are retrieved and cited", () => { const sourceOnly = ragResult({ grounded: false,