diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index c6bcc96b..daedab10 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -27,7 +27,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope - Resolving an item moves its row to **Resolved / archive** with the date and a one-line outcome — rows are archived, not deleted, so the history stays auditable. - + ## Open items @@ -47,6 +47,12 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #015 | P3 | task | Content-first fallback regression tests | CodeRabbit nitpick on #1024: add focused tests for the medication + registry (services/forms) content-first transitions — fallback content during loading, live record swap-in, error/empty states, and that **no authoritative verification badge shows before live governance reconciles** (the neutralized fixture flag). | #1024 CodeRabbit review; `src/components/clinical-dashboard/medication-record-page.tsx`; `src/components/registry-record-loader.tsx` | 2026-07-21 | | #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | | #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | +| #018 | P2 | task | Monitoring retrieval-depth trio misses schedules | quality-lithium-monitoring-range, quality-antipsychotic-metabolic-monitoring, quality-adhd-medication-monitoring still fail targeting ("no schedule/interval") after the #1040 gate parity — their live chunk sets apparently surface no admissible schedule sentence. Diagnose via read-only chunk queries + run #61 dump artifact (30d retention) to split corpus-content vs retrieval-depth vs fact-extraction; fix shape TBD after diagnosis. A retrieval-side fix is a protected surface (canary pair, fresh ~$2-4 authorization). | run #61 targeting misses; branch-review-ledger 2026-07-21 | 2026-07-21 | +| #019 | P2 | task | Comparison-class coverage: admission doc never in top-5 | admission-discharge-comparison misses its expected admission-side doc on every canary (#57-#61) — top-5 carries discharge-planning + unrelated docs only. Same "why doesn't retrieval surface X" species as #018; diagnose jointly. Ground-truth already widened (#1020); the residual is genuine comparison-class retrieval coverage. | runs #57-#61; PR #1020 replay verdict | 2026-07-21 | +| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | +| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | +| #022 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | +| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage:312, firefox ui-formulation:91 — diagnose only if reproduced), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | ## Resolved / archive diff --git a/scripts/eval-quality.ts b/scripts/eval-quality.ts index 47dd71e8..150f8620 100644 --- a/scripts/eval-quality.ts +++ b/scripts/eval-quality.ts @@ -12,6 +12,7 @@ import { type GoldenRetrievalResult, } from "./eval-retrieval"; import { + configuredCostRates, estimateCostUsd, loadAdminClient, percentile, @@ -1137,7 +1138,19 @@ async function runRagQualityCases(args: { routingReason: answer.routingReason, timings, routeCeilingExceeded, - estimatedCostUsd: hasOpenAIUsage ? estimateCostUsd(openAIUsage) : null, + // A case that never touched the provider costs exactly $0 when rates + // are configured; null stays reserved for "cannot estimate" — either + // rates unconfigured, or a provider call was attempted (request id, + // model, or generation time recorded) without usage metadata (e.g. a + // provider timeout), where real-but-unmeasured tokens must not be + // reported as $0 (issue #014 + review findings on PR #1050). + estimatedCostUsd: hasOpenAIUsage + ? estimateCostUsd(openAIUsage) + : openAIRequestIds.length > 0 || answer.modelUsed || generationLatencyMs > 0 + ? null + : configuredCostRates() + ? 0 + : null, openAIRequestIds: openAIRequestIds.length > 0 ? openAIRequestIds : undefined, openAIUsage: hasOpenAIUsage ? openAIUsage : undefined, }); diff --git a/tests/eval-quality.test.ts b/tests/eval-quality.test.ts index 2161ede7..9f4c311e 100644 --- a/tests/eval-quality.test.ts +++ b/tests/eval-quality.test.ts @@ -658,6 +658,21 @@ describe("eval quality reporting", () => { expect(report.blocking_threshold_failures).toContain("RAG route_ceiling_failure_count 1 above 0"); }); + it("sums estimated cost across priced and zero-cost cases, going n/a only when a case cannot be estimated", () => { + // Issue #014: extractive/unsupported cases make no provider call and cost + // exactly $0 when rates are configured — they must not null the run total. + // null stays reserved for "rates unconfigured" (cannot estimate). + const priced = ragResult({ estimatedCostUsd: 0.002 }); + const zeroCost = ragResult({ estimatedCostUsd: 0 }); + const unpriced = ragResult({ estimatedCostUsd: null }); + + const summed = buildEvalQualityReport({ retrievalResults: [], ragResults: [priced, zeroCost] }); + expect(summed.rag.summary.estimated_cost_usd).toBe(0.002); + + const unknown = buildEvalQualityReport({ retrievalResults: [], ragResults: [priced, zeroCost, unpriced] }); + expect(unknown.rag.summary.estimated_cost_usd).toBeNull(); + }); + it("accepts an offline source-only result only when expected sources are retrieved and cited", () => { const sourceOnly = ragResult({ grounded: false,