fix(rag): stop reasoning-token starvation that discarded answers as "unsupported"

[BigSimmo]

Summary

Production answers were degrading to "unsupported" after 60–90s on high-value clinical queries. Root cause (confirmed via live rag_queries telemetry): gpt-5.5 draws reasoning tokens from the same max_output_tokens budget as the visible answer. The strong route ran high reasoning effort against a 4000-token cap, so reasoning exhausted the budget and returned incomplete: max_output_tokens before writing the answer — the answer was discarded. The same high effort also overran the 30s timeout (provider_timeout). It's a class of bug: every reasoning-model call sized its token budget without reserving reasoning headroom, including the ingestion enrichment/extraction paths that ran silently over the whole corpus.

• env: OPENAI_MAX_OUTPUT_TOKENS 4000→16000; OPENAI_STRONG_REASONING_EFFORT high→medium. medium resolves every query class — including the safety-critical medication_dose_risk/table_threshold, which strongReasoningEffortForQueryClass kept at the full configured effort and which starved first.
• openai: reasoningHeadroomFloor(effort) floors max_output_tokens by effort in responseBody, so no call site can under-provision reasoning headroom. Math.max only ever raises a budget, and the budget is a ceiling (billed per token used) — free unless hit. One change fixes every call site.
• rag: truncation self-heal — strong escalations retry at a boosted cap (2×, ≥24000) instead of re-truncating on the same budget; a cumulative generation wall-clock budget skips the third (polish) generation to bound tail latency, keeping the valid strong answer rather than risking a truncation→unsupported tail.
• ingestion: document-enrichment + model-index-extraction now observe truncation and warn loudly (with document identity) instead of silently cutting off enrichment.
• observability: answer-slo adds truncation/timeout fallback counters to the /api/health deep probe.

Host/region change (the separate cross-region retrieval latency) is intentionally out of scope.

Verification

• npm run verify:cheap — green (1673 tests, 1 skipped, 179 files) + typecheck + prettier clean. (Full verify:pr-local build/bundle scan left to CI.)
• npm run eval:retrieval:quality — 36/36, document_recall@5=1.0, content_recall@5=1.0, content_mrr@10=0.9414 (≥ 0.9255 baseline). Retrieval untouched (change is generation-only).
• npm run eval:quality -- --rag-only (live) — the max_output_tokens and provider_timeout fallbacks are eliminated. Ran an A/B baseline (pre-fix vs fix on the same live corpus): every grounding/citation/failure metric is byte-identical (grounded 0.4667=0.4667, citation_failure 0.2955=0.2955, failure_category_counts identical, zero grounded cases lost) → zero quality regression, plus a ~14s p95 latency win (75.9s→61.5s) and one timeout fallback removed. The grounded-rate threshold miss is proven pre-existing (documentation-lookup confidence-gate cases, e.g. discharge-documentation).
• npm run check:production-readiness — not run (provider-backed / confirmation-gated per repo policy). No clinical-decision, privacy, Supabase, or source-governance behavior changed by this PR.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — unchanged
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — untouched
• Service-role keys and private document access remain server-only — untouched
• Demo/synthetic content remains clearly separated from real clinical sources — untouched
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — untouched
• Deployment classification/TGA SaMD impact checked — no change to clinical decision-support behavior: the A/B baseline shows answer content/grounding is byte-identical; this PR only prevents token-starvation truncation and lowers latency.

Notes

• Change was developed off a diverged branch; this PR is a clean re-application of the 10-file change onto current main (only tests/answer-slo.test.ts needed a trivial merge with main's new event_type scoping — the truncation/timeout counters inherit that filter via the shared base()).

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 678dec22-81f3-4670-8f61-a9f34e7e6135

📥 Commits

Reviewing files that changed from the base of the PR and between 7ecade0 and 5874814.

📒 Files selected for processing (10)

• src/lib/document-enrichment.ts
• src/lib/env.ts
• src/lib/model-index-extraction.ts
• src/lib/observability/answer-slo.ts
• src/lib/openai.ts
• src/lib/rag.ts
• tests/answer-slo.test.ts
• tests/document-enrichment.test.ts
• tests/model-index-extraction.test.ts
• tests/openai-cache.test.ts

---

📝 Walkthrough

Summary by CodeRabbit

• Improvements
  • Improved AI-generated answers by providing additional output capacity for complex reasoning.
  • Enhanced recovery when responses are incomplete, including smarter retries and safeguards against excessive retry delays.
  • Improved document enrichment and model-generated profiles when AI output is partially incomplete.
  • Added more detailed monitoring for answers affected by output limits or timeouts.
• Bug Fixes
  • Reduced cases where reasoning consumed too much of the available answer space.

Walkthrough

Generation now preserves structured-result metadata, warns on truncation, reserves reasoning headroom, expands strong retry budgets, limits cumulative retry time, and tracks truncation and timeout fallbacks in answer SLO metrics.

Changes

Generation resilience

Layer / File(s)	Summary
Token headroom defaults and request flooring src/lib/env.ts, src/lib/openai.ts, tests/openai-cache.test.ts	Generation defaults, reasoning effort, and OpenAI request budgets now apply reasoning headroom flooring with corresponding tests.
Strong retry budgeting src/lib/rag.ts	Strong retries can use larger output-token budgets and are skipped when the cumulative generation time budget is exhausted.
Structured output truncation handling src/lib/document-enrichment.ts, src/lib/model-index-extraction.ts, tests/document-enrichment.test.ts, tests/model-index-extraction.test.ts	Structured generation consumers inspect result metadata, warn on truncation, and parse result.text; mocks now provide object-shaped results.
Fallback SLO dimensions src/lib/observability/answer-slo.ts, tests/answer-slo.test.ts	Answer SLO snapshots now count and rate truncation and timeout fallback cases using additional PostgREST filters.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant generateWithModel
  participant OpenAIResponsesAPI
  participant strongQualityRepair
  generateWithModel->>OpenAIResponsesAPI: Request structured output
  OpenAIResponsesAPI-->>generateWithModel: Return text and truncation status
  generateWithModel->>strongQualityRepair: Evaluate answer quality
  strongQualityRepair->>OpenAIResponsesAPI: Retry with larger output budget
  OpenAIResponsesAPI-->>strongQualityRepair: Return retry result

Loading

Possibly related PRs

• BigSimmo/Database#443: Both changes modify RAG fast/strong retry control flow.
• BigSimmo/Database#513: Both changes extend answer SLO fallback query dimensions.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/generation-token-starvation-fix

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/generation-token-starvation-fix

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5874814cd3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reserve time before starting the quality retry

When the fast/strong path has already spent just under 2 * OPENAI_ANSWER_TIMEOUT_MS, this >= check still allows the strong_quality_retry below to start with a full per-call timeout, so two 29s generations under the default 30s timeout can still kick off a third 30s call and recreate the ~90s tail this guard is meant to prevent. Reserve the next call's timeout in this guard, or pass only the remaining budget into the retry, so the new wall-clock budget actually bounds the chain.

Useful? React with 👍 / 👎.