Eval canary regression: nightly golden eval failed

[github-actions]

Nightly eval canary failed on 2026-07-10T19:21:15.976Z.

Run: https://github.com/BigSimmo/Database/actions/runs/29117275232

Triage order (docs/observability-slos.md §3): rerun via workflow_dispatch,
check hybrid_rpc_errors and npm run check:indexing, then bisect code.
A failure can be corpus-state-dependent — confirm before reverting anything.

[github-actions]

Nightly eval canary failed on 2026-07-11T18:50:25.696Z.

Run: https://github.com/BigSimmo/Database/actions/runs/29163955713

Triage order (docs/observability-slos.md §3): rerun via workflow_dispatch,
check hybrid_rpc_errors and npm run check:indexing, then bisect code.
A failure can be corpus-state-dependent — confirm before reverting anything.

[BigSimmo]

Triage 2026-07-13 (dispatch run 29249365270, main @ cd25178)

The original retrieval regression is RESOLVED. The failing case in every prior red run was opioid-withdrawal-doses (dose-route evidence gate letting unrelated numeric tables outrank subject-matched guidance). #571 fixed it; this dispatch's retrieval golden eval is green (failed_cases=0), and a local full run confirms 36/36 with opioid-withdrawal-doses at docRecall@5=1.00 / contentRR@10=1.00, content_mrr@10=0.9111.

The canary is still red, but for a different, fully-pinned reason: answer-quality thresholds are no longer attainable on the tightened eval. The blocking failures are grounded_supported_rate 0.25 < 0.9, citation_failure_rate 0.75 > 0, numeric_grounding_failure_rate 0.125 > 0 on the 8-case live-generation subset. A full 44-case run on the same main commit measures grounded 0.2333 / citation 0.5 / numeric 0.1591 — i.e. main itself cannot meet these thresholds since the acceptSourceOnly eval tightening (#562/#564). Every ungrounded case is a confidence-gate refusal with the correct documents retrieved (the documented documentation-lookup false-positive family — e.g. discharge-documentation); none are generation truncations or timeouts.

Also ruled out: PR #577 (reasoning-token starvation fix) was A/B'd against this same main baseline on the live corpus — byte-identical failing-case sets, no metric drop, zero max_output_tokens/provider_timeout fallbacks on either side.

Recommended next step (needs a product decision): either recalibrate the canary answer-quality thresholds to post-#562/#564 reality, or fix the acceptSourceOnly/confidence-gate interplay so grounded documentation-lookup answers pass the gate. The retrieval half of this issue can be considered closed.

[BigSimmo]

Root cause found and fixed — correction to my earlier triage

The earlier comment attributed the answer-quality failures to the #562/#564 eval tightening. Deeper investigation disproved that: the flips are two product-side gate regressions, fixed in #606.

Family A (5 of the 8 subset cases): the confidence gate refuses all lexical-only answers. Migration 20260713062107_restore_text_fallback_lexical_score (in #552, applied live ~06:21Z on 7/13 — between the last green canary and the first answer-quality red) made the lexical text-fallback RPC honest: similarity = 0, hybrid_score hard-capped at 0.48, real signal in the new lexical_score column. The app's confidence gate still read only the capped hybrid, so topScore < 0.5 became unconditionally true for every text-fast-path answer → confidence_gate_blocked. Live repro: "What does the metabolic screening document require?" refused with the correct document at rank 1 and every chunk scored exactly 0.48. This also affects production users right now until #606 deploys. Fix: the gate's evidence score is max(scoreValue, lexical_score); ranking/ordering unchanged (golden retrieval eval 36/36, content_mrr@10=0.9111 re-proven on the fix branch).

Family B (4 golden cases): the numeric-faithfulness gate required each figure in EVERY cited chunk. #553's verifyAnswerNumbers used intersection over cited chunks, so stitched multi-source answers could never verify (live repro: the only flagged token, "60 minutes", is verbatim in a cited chunk). Fix: union over cited chunks, preserving fail-closed behaviour for unmapped citations and fabricated figures.

Results (live corpus): canary-equivalent 8-case subset grounded 0.25 → 0.875; full 44-case run grounded 0.2333 → 0.8333, numeric grounding failures 0.1591 → 0, citation failures 0.5 → 0.1136, with unsupported_correct_rate staying 1.0 and danger failures 0.

Residual (not blocking this fix): long-acting-injectables flakes when its fast route retries onto the strong route (a route the eval case forbids) and the strong answer cites chunks that don't contain its quoted figures — the gate flagging that is intended fail-closed behaviour. The strong-route/claim-support family (claim_support_high_risk_gap, final_quality_gate:ungrounded_unsupported_answer) from #553 is deliberately untouched. If the canary still flakes on these after #606, the decision is eval-side (allow strong for that case / relax the 0.9 grounded threshold slightly), not another gate change.

Will re-dispatch the canary once #606 merges.

[BigSimmo]

Post-#606 canary status: quality fully recovered; only latency SLOs remain red

Dispatch run 29262119701 (main with #606):

• Golden retrieval eval: green (failed_cases=0).
• Answer-quality subset: every quality metric perfect — grounded supported rate 1.0, citation failure 0, numeric grounding failure 0, expected source hit 1.0, danger failure 0, failing cases none.
• Only blocking failures: p95_latency_ms 48256 above 25000 and route extractive p95_latency_ms 48256 above 12000.

The latency breach is partly a consequence of the fix: confidence_gate_blocked refusals return in ~1–2s, so the pre-#606 canary measured fast wrong answers. Real grounded answers pay full generation time, and the timeout→extractive-fallback chain costs 30s+ per affected case from the GitHub runner's cross-region path to the Sydney Supabase project + OpenAI. This is the already-documented host/Sydney latency workstream, not an answer-quality regression.

Decision needed (product/infra, not another gate change):

1. Recalibrate the canary latency thresholds to the post-fix reality of answering (e.g. p95 25000 → ~60000 from GitHub runners, or measure route-level thresholds excluding provider-timeout fallback chains), or
2. Prioritise the cross-region latency work (host placement / Sydney) that memory and prior triage already flagged as out of scope for gate fixes.

The regression this issue was opened for (retrieval: opioid-withdrawal-doses, fixed by #571; answer-quality collapse: fixed by #606) is resolved. Leaving the issue open only for the latency-threshold decision — if you recalibrate, the next dispatch should be fully green and this can close.

[BigSimmo]

Post-#624 proof run 29275370553 for the record: every executed case PASSED — the previously 17s lexical cases now run at 1.4–2.7s from the runner after the #582 index-friendly rewrite — and the run aborted only on OpenAI account quota exhaustion mid-eval (retrieval:vector-ptsd, 6 rate-limit retries then quota exhausted), an account/billing condition from today's heavy eval volume, not a regression. Operator note: check OpenAI usage limits before the next scheduled canary (18:00 UTC daily); a genuine scheduled failure re-opens automatically.

🤖 Generated with Claude Code

[BigSimmo]

Recalibration merged (#622); final green blocked only on OpenAI billing quota

Dispatch run 29276446601 (main with #606 + #622 + #624's context gating): zero eval failures — 25/25 golden retrieval cases passed (including opioid-withdrawal-doses), then the run died at the vector-ptsd forced-embedding case with OpenAI quota is exhausted after six 429 retries. Verified directly against the OpenAI API: insufficient_quota (billing), not rate-limiting.

Consequences right now:

• The canary cannot go green until the OpenAI account is topped up (platform.openai.com → billing). Today's eval-heavy verification work plus concurrent sessions drained the quota.
• Production is affected too: the app degrades gracefully to source-only answers (real citations, no LLM synthesis) while quota is out — the fix(rag): read truthful lexical evidence in the confidence gate; verify numerics against the union of cited chunks #606 gate fixes keep retrieval-grounded answers working, but generated prose is unavailable.

Everything code-side is done and live:

• Fix clinical-subject dose retrieval #571 (dose retrieval) + fix(rag): read truthful lexical evidence in the confidence gate; verify numerics against the union of cited chunks #606 (confidence gate reads lexical_score; numeric verification union-over-cited-chunks) + fix(eval): recalibrate canary latency budgets to the post-#606 honest baseline #622 (context-gated latency budgets: strict 25s/12s/35s near-region, canary opts into 60s/35s/60s via EVAL_LATENCY_CONTEXT, dedicated 50s fallback bucket) merged; fix(eval): calibrate canary latency gates to the runner's measurement context #624 reconciled.
• Railway app + worker deployed from main@84cebf38f at 18:26Z; live retest confirmed previously-refused documentation lookups now answer grounded with citations, and the fix(rag): stop reasoning-token starvation that discarded answers as "unsupported" #580 waste modes (max_output_tokens, provider_timeout) are absent from telemetry post-deploy.

To close this issue: top up OpenAI quota, then either wait for the nightly canary or dispatch once — with quality metrics already perfect (grounded 1.0 / citation 0 / numeric 0 on the previous run) and budgets now honest, it should be fully green.