diff --git a/.env.example b/.env.example index f478f3e65..f1ee74c74 100644 --- a/.env.example +++ b/.env.example @@ -38,16 +38,19 @@ OPENAI_API_KEY=replace-with-openai-api-key OPENAI_EMBEDDING_MODEL=text-embedding-3-small # Must match vector(N) in supabase/schema.sql. Do not change without a migration. EMBEDDING_DIMENSIONS=1536 -OPENAI_ANSWER_MODEL=gpt-5.5 -OPENAI_FAST_ANSWER_MODEL=gpt-5.5 -# Strong tier stays on the standard (non-pro) model; fast vs strong differ by reasoning effort. -OPENAI_STRONG_ANSWER_MODEL=gpt-5.5 +OPENAI_ANSWER_MODEL=gpt-5.6-terra +OPENAI_FAST_ANSWER_MODEL=gpt-5.6-terra +OPENAI_STRONG_ANSWER_MODEL=gpt-5.6-sol +# Workload-specific models can be canaried independently of clinical synthesis. +OPENAI_QUERY_CLASSIFIER_MODEL=gpt-5.6-luna +OPENAI_SUMMARY_MODEL=gpt-5.6-terra +OPENAI_INDEXING_MODEL=gpt-5.6-terra OPENAI_MAX_OUTPUT_TOKENS=4000 OPENAI_QUERY_CACHE_SIZE=200 # Max inputs per embeddings request (OpenAI caps a request at 2048 inputs / ~300k tokens). # embedTexts splits a full-corpus re-embed into batches of this size. OPENAI_EMBEDDING_BATCH_SIZE=256 -OPENAI_VISION_MODEL=gpt-5.5 +OPENAI_VISION_MODEL=gpt-5.6-terra OPENAI_VISION_IMAGE_DETAIL=auto OPENAI_REQUEST_TIMEOUT_MS=45000 # Answer-generation budget. Kept generous so a strong reasoning model can finish a @@ -55,7 +58,13 @@ OPENAI_REQUEST_TIMEOUT_MS=45000 OPENAI_ANSWER_TIMEOUT_MS=30000 OPENAI_MAX_RETRIES=2 OPENAI_GENERATION_MAX_RETRIES=0 +# Pre-5.6 compatibility only. GPT-5.6 never receives prompt_cache_retention. OPENAI_PROMPT_CACHE_RETENTION=24h +# GPT-5.6 prompt cache option: 30m | off. "off" omits the extended TTL option. +OPENAI_PROMPT_CACHE_TTL=30m +# Optional HMAC secret (min 32 chars) for a stable pseudonymous OpenAI safety_identifier. +# Raw owner IDs are never sent. Review derivation/retention with privacy governance before enabling. +#OPENAI_SAFETY_IDENTIFIER_SECRET= OPENAI_STORE_RESPONSES=false OPENAI_FAST_REASONING_EFFORT=low OPENAI_STRONG_REASONING_EFFORT=high diff --git a/.githooks/pre-push b/.githooks/pre-push old mode 100644 new mode 100755 diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 5c30d165b..18fd426aa 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -59,6 +59,11 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-13 | origin/main (detached review worktree) | c523cabeae4b68ebdf569ecbc18d9f5a7b5afbf1 | repo-wide audit | Changes requested: one P1 clinical-answer trust cluster; three P2 tenancy/reindex guardrail issues; three P3 information-disclosure, dead-code, and transitive-deprecation cleanup items. No P0 found. | `npm run verify:cheap` (1,721 passed, 1 skipped); `npm run test:coverage` (thresholds passed); `npm run build`; `npm run verify:ui` (137/137); `npm run eval:rag:offline` (36 fixtures, 60 tests); format, Edge Function, Codex workflow, import/secret/dead-reference scans. Provider-backed checks skipped. | | 2026-07-13 | HEAD / origin/main (detached worktree) | 04c1d0b036cae8af4dabfc692055c7aab93d5888 | OpenAI-facing API and integration review | Read-only review found one P1 clinical-streaming defect and four P2 reliability/API-contract issues: provisional clinical prose is exposed before validation; mid-stream failure can silently trigger a second buffered generation; answer caches are not model/prompt fingerprinted; OpenAI access/model errors are under-classified; and table-fact route IDs are not validated before database access. GPT-5.6 migration also requires replacing the legacy prompt-cache parameter. No application code was changed. | Static call-flow, prompt, model, schema, streaming, cache, error, route, test, and governance inspection; official OpenAI model/Responses/structured-output/streaming/prompt-cache guidance reviewed. Provider-backed checks were not run. Local tests were not run because this worktree has no installed dependencies (`openai`, `vitest`). | +| 2026-07-13 | codex/openai-gpt56-rag-upgrade | 4fa4c35e98d60fc104639089494b271a5f1951fd | OpenAI and RAG review remediation | Remediated all recorded findings: clinical SSE is final-only across mixed-version deployments; buffered generation cannot silently replace a partial stream; answer caches are generation/retrieval fingerprinted; GPT-5.6 model, prompt-cache, workload routing, parsed-output, usage, safety-identifier, and error handling are capability-aware; and table-fact UUIDs fail with the shared 400 contract. Added rollout and governance documentation. Independent final review found no remaining high-confidence issue after the mixed-version client guard was added. | Replaced the external `node_modules` junction with a clean `npm ci`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (211 files passed, 1 skipped; 1,941 tests passed, 1 skipped); focused cache/stream tests, offline RAG preflight, production-readiness CI, changed-file Prettier/ESLint, and `git diff --check` passed before the current `origin/main` integration. Provider and post-merge checks are recorded separately when complete. | +| 2026-07-14 | codex/openai-gpt56-rag-upgrade | f83d7d49b32fb9bb8b8c2ede99e93675574c88d8 | PR review follow-up | Confirmed and remediated five scoped review findings: answer-quality JSON no longer retains owner-scoped answer previews; valid tablet/mmol dose phrases are accepted; review-fallback routing uses a shared token; the classifier sends only supported structural schema constraints and validates bounds after parsing; and equivalent maximum-dose wording remains eligible. Replaced the redundant dashboard source-string guard with the existing mixed-version browser behavior regression. | Focused Vitest 123/123; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (234 files passed, 1 skipped; 2,229 tests passed, 1 skipped); `git diff --check`. Provider-backed retrieval remains separately blocked by exhausted OpenAI embedding quota after case 24/36. | +| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | 1c6c780f30f33d47e7dfb57dce56d5fd0501d46e | PR review follow-up | Confirmed and remediated three remaining scoped findings: empty `max_output_tokens` responses now reach the RAG retry path; equivalent maximum-dose wording satisfies the maximum-dose intent gate without requiring literal `maximum` or `dose` tokens; and long-form clinical dose units are accepted consistently by evidence extraction and final-answer validation. | Focused Vitest 99/99; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (235 files passed, 1 skipped; 2,238 tests passed, 1 skipped); `npm run build` generated 636 static pages and passed the client-bundle secret scan; configured production readiness was READY with three documented warnings; `git diff --check`. Provider-backed retrieval remains separately blocked by exhausted OpenAI embedding quota after case 24/36. | +| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | 0301d59ab8d6afeecd340bb08b40b98ff6b95c6c | PR review follow-up | Confirmed and fixed one additional maximum-dose wording gap discovered after the current-main merge: `not to exceed` and `not more than` numeric limits now reach the same validated extractive path as the existing equivalent phrases. | Focused Vitest 84/84; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; focused Prettier and `git diff --check`. The immediately preceding merged head also passed `npm run verify:cheap` (236 files passed, 1 skipped; 2,242 tests passed, 1 skipped), production build, and all hosted required checks. Provider-backed retrieval remains blocked by exhausted OpenAI embedding quota after case 24/36. | +| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | f2652021c1f14533f17c4a9724ef6dff17dd43f5 | PR review follow-up | Confirmed and fixed the two final scoped findings: GPT-5.6-and-later model families now use the TTL prompt-cache request shape, and unrelated prose containing `maximum` can no longer pass the maximum-dose evidence gate without a clinical dose value or explicit dose wording. | Focused Vitest 99/99 before current-main integration and 163/163 after it; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed after integrating `origin/main` (239 files passed, 1 skipped; 2,248 tests passed, 1 skipped); focused Prettier and `git diff --check`. The reviewed head passed every hosted required check before these narrow fixes. Provider-backed retrieval remains blocked by exhausted OpenAI embedding quota after case 24/36. | | 2026-07-13 | claude/audit-ci-browser-gate-2026-07-13 | 65a8a0c9e7c3a165b09a1ad79e893af8c2c6973b | branch-cleanup | Retained because the branch is checked out in an active or protected worktree. | Fresh worktree, status, lock, and process activity scan. | | 2026-07-13 | claude/beautiful-hamilton-5df54c | 5e2e90f0a3af4039c7e15515151569228476a60c | branch-cleanup | Redundant: the exact HEAD is an ancestor of `origin/main`; eligible for deletion when no worktree or open PR references it. | `git merge-base --is-ancestor 5e2e90f0a3af4039c7e15515151569228476a60c origin/main`. | | 2026-07-13 | claude/beautiful-williamson-57d144 | 072434cd5f59b1441dd114958489028972c42d34 | branch-cleanup | Redundant: exact source HEAD was squash-merged by PR #474; eligible for deletion when unreferenced. | GitHub merged-PR source HEAD matched exactly. | diff --git a/docs/capacity-review.md b/docs/capacity-review.md index 29eb479f7..6f23f5dd5 100644 --- a/docs/capacity-review.md +++ b/docs/capacity-review.md @@ -72,8 +72,9 @@ moment load is highest. This only works while the app is a **single process** Per answer: 1 embedding call (unless the lexical fast path skips it) + 1–2 generations (fast route, escalation to strong). 15 answers/min with grounded prompts (~5–15k tokens each) lands in the low hundreds of thousands of -tokens/min at worst — within Tier-2+ gpt-5.5 TPM limits, but _bursts_ of -simultaneous strong-route generations can trip request-per-minute limits. +tokens/min at worst. Re-baseline the production project's current Terra/Sol +token and request limits before rollout; _bursts_ of simultaneous strong-route +generations can still trip request-per-minute limits. Existing dampers: coalescing (duplicate questions never reach OpenAI), the answer cache, `OPENAI_MAX_RETRIES`, and graceful degradation to source-only answers (which must stay _visible_ — see the degraded-rate SLO). diff --git a/docs/codex-prompt-playbook.md b/docs/codex-prompt-playbook.md index 32127c6de..d7f50ad38 100644 --- a/docs/codex-prompt-playbook.md +++ b/docs/codex-prompt-playbook.md @@ -705,7 +705,7 @@ Audit and harden structured output contracts for this repo. Focus on places where model or model-like output is parsed, displayed, stored, or graded: - src/lib/rag.ts -- src/lib/answer-stream-extractor.ts +- src/lib/answer-stream-contract.ts - src/lib/answer-verification.ts - src/lib/answer-render-policy.ts - src/app/api/answer/route.ts diff --git a/docs/openai-cross-border-basis.md b/docs/openai-cross-border-basis.md index b7ffdced5..c3df4d9b8 100644 --- a/docs/openai-cross-border-basis.md +++ b/docs/openai-cross-border-basis.md @@ -17,10 +17,11 @@ This document covers the model-provider leg: query text + retrieved excerpts sent to OpenAI in the United States for embedding and answer synthesis (PIA §3–4; verified still true in code — [src/lib/openai.ts:75-79](../src/lib/openai.ts) builds a plain `new OpenAI({ apiKey, timeout, maxRetries })` -with no `baseURL`/ZDR header, `store:false` by default, and `prompt_cache_retention` forced to `"24h"` -for gpt-5.5 at [openai.ts:174](../src/lib/openai.ts)). Railway application and worker processing in -Singapore is the separate overseas processor leg recorded in the PIA; the APP 8 record must cover -both providers. +with no `baseURL`/ZDR header and `store:false` by default. GPT-5.6-and-later requests use +`prompt_cache_options.ttl="30m"`; gpt-5.5 requests force the legacy +`prompt_cache_retention="24h"` field ([openai.ts](../src/lib/openai.ts)). Railway application and +worker processing in Singapore is the separate overseas processor leg recorded in the PIA; the APP +8 record must cover both providers. Two obligations attach to that flow: @@ -38,14 +39,16 @@ The code-side controls cannot _by themselves_ discharge APP 8 — the "reasonabl ## 2. What actually crosses the border -| Egress | Payload | Endpoint | Reference | -| --------- | ---------------------------------------------------------------- | ------------------------------------------------ | -------------------------------------------------------------------------------------------- | -| Embedding | Raw query text (normalized) | `POST /v1/embeddings` (`text-embedding-3-small`) | [openai.ts embedText](../src/lib/openai.ts) | -| Answer | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (`gpt-5.5`, `store:false`) | [rag.ts:4220](../src/lib/rag.ts) · [rag-source-block.ts:180](../src/lib/rag-source-block.ts) | +| Egress | Payload | Endpoint | Reference | +| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| Embedding | Raw query text (normalized) | `POST /v1/embeddings` (`text-embedding-3-small`) | [openai.ts embedText](../src/lib/openai.ts) | +| Answer | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (Terra fast / Sol strong, `store:false`) | [rag.ts](../src/lib/rag.ts) · [rag-source-block.ts](../src/lib/rag-source-block.ts) | -The app **adds no patient identifiers** and stores queries only as a keyed hash locally; but it does -**not scrub** PHI a clinician types. Everything else (documents, embeddings, logs, auth) stays at rest -in **Sydney — AWS `ap-southeast-2`** (PIA §7). +The app **adds no raw patient or owner identifiers** and stores queries only as a keyed hash locally. +When configured, authenticated Responses requests include a stable HMAC-SHA256 +`safety_identifier`; anonymous and background requests omit it. The app does **not scrub** PHI a +clinician types. Everything else (documents, embeddings, logs, auth) stays at rest in +**Sydney — AWS `ap-southeast-2`** (PIA §7). ## 3. OpenAI's current terms (verified 2026-07-13) @@ -60,7 +63,7 @@ these terms change; the PIA (2026-07-06) already predates the Australia data-res | **Zero Data Retention (ZDR)** | Removes the 30-day abuse-monitoring retention; **not self-serve** — prior approval by OpenAI, configured per **project**. Apply via the account/sales team. | [Data controls](https://developers.openai.com/api/docs/guides/your-data) | | **Data residency** | API data residency now covers **Australia** (among US, Europe, UK, Canada, Japan, Korea, Singapore, India, UAE). Enabled by creating a **new Project** and selecting the country; eligibility via sales. **Australia = storage at rest only** — regional _processing/inference_ is US/Europe/UAE only. ~10% uplift for models released from 5 Mar 2026. | [Data residency (API)](https://help.openai.com/en/articles/10503543-data-residency-for-the-openai-api) · [Announcement](https://openai.com/index/expanding-data-residency-access-to-business-customers-worldwide/) | | **Sub-processors** | Published list of sub-processors that may process Customer Data. Review for the APP 8 accountability chain. | [Sub-processor list](https://openai.com/policies/sub-processor-list/) · [platform](https://platform.openai.com/subprocessors) | -| **Prompt caching** | For gpt-5.5, `prompt_cache_retention` cannot be `in_memory`; app forces `"24h"` ([openai.ts:174](../src/lib/openai.ts)). Whether ZDR also zeroes the prompt cache must be **confirmed in writing** (see §6, PIA-6). | [Data controls](https://developers.openai.com/api/docs/guides/your-data) | +| **Prompt caching** | GPT-5.6 requests `prompt_cache_options.ttl="30m"` by default and never receives the deprecated retention field. The TTL is a minimum, not a guaranteed deletion deadline. Explicit pre-5.6 deployments retain the legacy retention behavior. ZDR interaction must be **confirmed in writing** (see §6, PIA-6). | [Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) · [Data controls](https://developers.openai.com/api/docs/guides/your-data) | ## 4. This app's endpoints are ZDR-eligible @@ -90,9 +93,10 @@ Items 1–2 (plus documenting 4–5) are what turn PIA-1 from open to closed. It ## 6. Open question to pin with OpenAI -**Does ZDR eliminate the forced 24h gpt-5.5 prompt cache**, or does an encrypted ≤24h cache persist -regardless? Get this in writing — it determines whether **PIA-6** is fully resolved by ZDR or merely -mitigated. Record the answer in the status block. +**What is the effective prompt-cache deletion behavior under ZDR for GPT-5.6 requests that specify +the 30-minute TTL, and for requests where the app omits the extended TTL option?** Get this in +writing — it determines whether **PIA-6** is fully resolved by ZDR or merely mitigated. Record the +answer in the status block. ## 7. Consistency with the shipped user-facing notice @@ -166,9 +170,8 @@ in the PIA. No automated action in this review accepted either provider's terms. These touch the OpenAI request path — do them **only after** the legal decision, and treat them as provider-path changes (confirm before running against live). -- **ZDR granted:** no code change strictly required (ZDR is account/project-side). Optionally revisit - the forced `"24h"` prompt-cache retention ([openai.ts:174](../src/lib/openai.ts)) depending on the §6 - answer, and note the resolution against **PIA-6**. +- **ZDR granted:** no code change strictly required (ZDR is account/project-side). Revisit + `OPENAI_PROMPT_CACHE_TTL` depending on the §6 answer and note the resolution against **PIA-6**. - **Australia data residency adopted:** the client currently has no `baseURL` override ([openai.ts:75-79](../src/lib/openai.ts)). Data-residency Projects route via the standard API with a region-scoped project key; confirm whether a `baseURL`/project-key change is needed and wire an diff --git a/docs/openai-rag-operations.md b/docs/openai-rag-operations.md new file mode 100644 index 000000000..6da0c0d40 --- /dev/null +++ b/docs/openai-rag-operations.md @@ -0,0 +1,85 @@ +# OpenAI and RAG operations + +## Supported architecture + +The app uses the OpenAI Responses API for stateless structured generation and +multimodal image inputs, plus the embeddings API for owner-scoped Supabase retrieval. It does not +use Chat Completions, Assistants, the Agents SDK, Realtime, built-in file search, or persisted +OpenAI conversation state. + +The clinical request path is: + +`ClinicalDashboard -> /api/answer/stream -> retrieval/classification -> Responses API -> deterministic quality and source-governance gates -> final SSE event` + +The SSE connection emits progress events and heartbeat comments while work is running. Clinical +answer prose is sent only in the validated `final` event. The browser deliberately ignores legacy +`token` and `revising` events so a mixed-version deployment cannot restore provisional prose. + +## Workload models + +| Workload | Environment variable | Documented rollout value | +| -------------------------------- | ------------------------------- | ------------------------------------------- | +| Fast clinical synthesis | `OPENAI_FAST_ANSWER_MODEL` | `gpt-5.6-terra` | +| Strong clinical synthesis | `OPENAI_STRONG_ANSWER_MODEL` | `gpt-5.6-sol` | +| Query classifier | `OPENAI_QUERY_CLASSIFIER_MODEL` | `gpt-5.6-luna` | +| Document summaries | `OPENAI_SUMMARY_MODEL` | `gpt-5.6-terra` | +| Enrichment/index profiles | `OPENAI_INDEXING_MODEL` | `gpt-5.6-terra` | +| Vision classification/captioning | `OPENAI_VISION_MODEL` | `gpt-5.6-terra` | +| Embeddings | `OPENAI_EMBEDDING_MODEL` | `text-embedding-3-small` at 1536 dimensions | + +If a workload-specific variable is omitted, the query classifier follows the fast-answer model, +summaries follow the answer model, and indexing follows the strong-answer model. Existing +deployments with explicit GPT-5.5 variables remain pinned until their environment is changed. + +OpenAI recommends the Responses API for current general-purpose multimodal and tool-capable +workflows. See the [Responses migration guide](https://developers.openai.com/api/docs/guides/migrate-to-responses), +[latest-model guide](https://developers.openai.com/api/docs/guides/latest-model), and +[model catalog](https://developers.openai.com/api/docs/models). + +## Compatibility and safety controls + +- GPT-5.6 requests use `prompt_cache_options.ttl`; they never receive the deprecated + `prompt_cache_retention` field. `OPENAI_PROMPT_CACHE_TTL=off` omits the extended TTL option. + Pre-5.6 models retain the legacy retention configuration. +- Answer caches and in-flight coalescing include a fingerprint of models, reasoning effort, + provider mode, answer prompt/schema versions, retrieval version, and indexing prompt version. +- Static query-classifier output uses `responses.parse` with a strict Zod schema. The dynamic + clinical answer schema remains strict JSON Schema because its evidence-ID enums are generated + from the retrieved source set. +- Generation retries remain disabled at the SDK layer. The RAG pipeline performs explicit, + state-aware fast-to-strong and deterministic quality retries and fails closed to a labelled + source-only answer. +- `response.failed`, content-filtered, and empty/absent outputs are explicit provider failures. + Timeout, key, access, missing-model, quota, rate-limit, invalid-request, and service failures use + distinct public-safe error codes. +- When `OPENAI_SAFETY_IDENTIFIER_SECRET` is configured, authenticated Responses requests use an + HMAC-SHA256 pseudonym. Raw owner IDs are never sent; anonymous/background requests omit it. +- Usage telemetry includes input, output, total, cached-input, cache-write, and reasoning tokens. + +## Rollout sequence + +Change one workload at a time: classifier, indexing/enrichment, summaries, vision, fast answers, +then strong answers. Do not change prompts and models in the same experiment. A model or prompt +change automatically misses the prior answer cache because the generation fingerprint changes. + +Before provider-backed rollout, manually confirm model access, project/org permissions, pricing, +rate limits, DPA/ZDR posture, prompt-cache handling, and the privacy basis in +[openai-cross-border-basis.md](openai-cross-border-basis.md). + +## Local validation + +Keep provider variables cleared for local/static/mocked checks: + +```powershell +$env:OPENAI_API_KEY=$null +$env:OPENAI_ORG_ID=$null +$env:OPENAI_PROJECT_ID=$null + +npm run test -- tests/openai-cache.test.ts tests/openai-error-mapping.test.ts tests/openai-safety-identifier.test.ts tests/rag-generation-fingerprint.test.ts tests/rag-classifier-memo.test.ts tests/rag-answer-fallback.test.ts tests/private-rag-access.test.ts tests/document-mutation-routes.test.ts +npm run eval:rag:offline +npm run verify:cheap +``` + +Provider-backed evaluation requires explicit approval. Run classifier/vision/answer canaries +separately and compare citation validity, unsupported-number rate, source-gap behavior, fallback +rate, p50/p95 latency, output/reasoning tokens, cache reads/writes, and cost per accepted answer. diff --git a/docs/privacy-impact-assessment.md b/docs/privacy-impact-assessment.md index c9dfb74b2..70c8df18e 100644 --- a/docs/privacy-impact-assessment.md +++ b/docs/privacy-impact-assessment.md @@ -45,15 +45,15 @@ material. **Top gaps (full register in §10):** -| ID | Risk | One-line | -| ----- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| PIA-1 | High | Overseas processing occurs in Railway Singapore and OpenAI US; the applicable processor/APP 8 basis and final notice wording require governance approval. | -| PIA-2 | High | Production now fails closed without `RAG_QUERY_HASH_SECRET`; the operator must still place the secret in the deploy host. | -| PIA-3 | Mitigated | Generated answer text is omitted from `rag_queries` by default. `RAG_PERSIST_ANSWER_TEXT=true` is explicit opt-in and blocked by production readiness. | -| PIA-4 | Mitigated | Query-miss and bounded response-cache purges were verified active live on 2026-07-14; the duplicate unbounded cache job was removed. | -| PIA-5 | Medium | Draft point-of-entry collection notices and a `/privacy` data-processing page ship, but no governance-approved final privacy policy exists (APP 1, APP 5). | -| PIA-6 | Low-Med | OpenAI **prompt-cache retention is forced to 24h** for gpt-5.5 regardless of config — query + retrieved excerpts persist ≤24h at OpenAI. | -| PIA-7 | Low | `RAG_PERSIST_RAW_QUERY_TEXT=true` would store raw PHI query text with no secondary safeguard beyond the 30-day purge. | +| ID | Risk | One-line | +| ----- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| PIA-1 | High | Overseas processing occurs in Railway Singapore and OpenAI US; the applicable processor/APP 8 basis and final notice wording require governance approval. | +| PIA-2 | High | Production now fails closed without `RAG_QUERY_HASH_SECRET`; the operator must still place the secret in the deploy host. | +| PIA-3 | Mitigated | Generated answer text is omitted from `rag_queries` by default. `RAG_PERSIST_ANSWER_TEXT=true` is explicit opt-in and blocked by production readiness. | +| PIA-4 | Mitigated | Query-miss and bounded response-cache purges were verified active live on 2026-07-14; the duplicate unbounded cache job was removed. | +| PIA-5 | Medium | Draft point-of-entry collection notices and a `/privacy` data-processing page ship, but no governance-approved final privacy policy exists (APP 1, APP 5). | +| PIA-6 | Low-Med | GPT-5.6-and-later models use `prompt_cache_options.ttl="30m"` by default; gpt-5.5 forces the legacy 24h field. Provider controls may retain cached data longer than the configured minimum. | +| PIA-7 | Low | `RAG_PERSIST_RAW_QUERY_TEXT=true` would store raw PHI query text with no secondary safeguard beyond the 30-day purge. | --- @@ -101,8 +101,8 @@ Clinician browser │ **raw query verbatim** ("Question:\n${query}", rag.ts:7144) │ │ + **retrieved chunk text** (buildRagSourceBlock, rag.ts:6306) │ │ + system instructions (rag.ts:7053) │ - │ → OpenAI Responses API (gpt-5.5) openai.ts:384 ─┘ - │ store:false (openai.ts:220); prompt_cache_retention:24h (:168) + │ → OpenAI Responses API (Terra fast / Sol strong) ─┘ + │ store:false; GPT-5.6 prompt_cache_options.ttl:30m │ ├──►(D) LOCAL LOGGING (Supabase, Sydney, owner-stamped) │ insertRagQuery(): rag.ts:1983 @@ -134,26 +134,34 @@ OpenAI as the only cross-border flow. ### 4.1 What is sent -| Payload | Content | Reference | -| --------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -| Embedding input | **Raw query text**, verbatim (normalized whitespace/case only) | [src/lib/openai.ts:498](src/lib/openai.ts) → `embedTexts` :423 | -| Answer input | **Raw query verbatim** (`Question:\n${args.query}`) | [src/lib/rag.ts:7144](src/lib/rag.ts) | -| Answer input | **Retrieved chunk text** (content, capped ~1800 chars, plus title/page/section/table-facts/captions) | [src/lib/rag.ts:6306-6325](src/lib/rag.ts) | -| Instructions | Static system prompt ("experienced psychiatrist in Perth…") | [src/lib/rag.ts:7053](src/lib/rag.ts) | -| Metadata | `{ operation }` only — **no** owner id, **no** patient identifiers added by the app | [src/lib/openai.ts:223](src/lib/openai.ts) | +| Payload | Content | Reference | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | +| Embedding input | **Raw query text**, verbatim (normalized whitespace/case only) | [src/lib/openai.ts:498](src/lib/openai.ts) → `embedTexts` :423 | +| Answer input | **Raw query verbatim** (`Question:\n${args.query}`) | [src/lib/rag.ts:7144](src/lib/rag.ts) | +| Answer input | **Retrieved chunk text** (content, capped ~1800 chars, plus title/page/section/table-facts/captions) | [src/lib/rag.ts:6306-6325](src/lib/rag.ts) | +| Instructions | Static system prompt ("experienced psychiatrist in Perth…") | [src/lib/rag.ts:7053](src/lib/rag.ts) | +| Metadata | `{ operation }`; when configured, `safety_identifier` is an HMAC-SHA256 pseudonym of the authenticated owner. The raw owner id is never sent. | [src/lib/openai.ts](src/lib/openai.ts) | The app never _adds_ patient identifiers, but it does not scrub them either: **any PHI the clinician types into the query, or that exists in an indexed excerpt, is transmitted to OpenAI.** ### 4.2 Handling controls on the OpenAI request -- **Model:** `gpt-5.5` for answers, `text-embedding-3-small` for embeddings ([src/lib/env.ts:18-27](src/lib/env.ts)). +- **Models:** `gpt-5.6-terra` for fast synthesis, summaries, indexing, and vision; + `gpt-5.6-sol` for strong synthesis; `gpt-5.6-luna` is the documented query-classifier + rollout target; `text-embedding-3-small` remains the embedding model + ([src/lib/env.ts](src/lib/env.ts), [.env.example](../.env.example)). Existing deployments + with explicit model variables remain pinned until their configuration is changed. - **`store: false`** by default — responses are not retained in OpenAI's dashboard/store ([src/lib/openai.ts:220](src/lib/openai.ts), [src/lib/env.ts:55-58](src/lib/env.ts)). -- **`prompt_cache_retention: "24h"`** — **forced on for gpt-5.5** regardless of the - `OPENAI_PROMPT_CACHE_RETENTION` env value ([src/lib/openai.ts:168, 208, 221-222](src/lib/openai.ts)). - Prompt prefixes (which include retrieved excerpts and can include the query) are cacheable at OpenAI - for up to 24 hours. See PIA-6. +- **GPT-5.6 prompt caching:** the app sends `prompt_cache_options: { ttl: "30m" }` + unless `OPENAI_PROMPT_CACHE_TTL=off`; it never sends the deprecated + `prompt_cache_retention` field to GPT-5.6. Explicit pre-5.6 deployments retain the legacy + `OPENAI_PROMPT_CACHE_RETENTION` behavior ([src/lib/openai.ts](src/lib/openai.ts)). The 30-minute + value is a minimum cache lifetime, not a guaranteed deletion deadline. See PIA-6. +- **Safety identifier:** when `OPENAI_SAFETY_IDENTIFIER_SECRET` is configured, authenticated + Responses requests carry a stable HMAC-SHA256 pseudonym. Anonymous and background requests omit + it, and raw owner identifiers are never sent. Production readiness warns when the secret is absent. - **No `baseURL` override and no zero-data-retention (ZDR) header** are set in code — the client is a plain `new OpenAI({ apiKey, timeout, maxRetries })` ([src/lib/openai.ts:69-73](src/lib/openai.ts)), so traffic goes to `api.openai.com` (US) under whatever data-processing terms attach to the API @@ -161,14 +169,15 @@ types into the query, or that exists in an indexed excerpt, is transmitted to Op ### 4.3 Data-processing terms — what code can and cannot tell us -The code shows the _technical_ posture (US endpoint, `store:false`, 24h prompt cache, no ZDR header). +The code shows the _technical_ posture (US endpoint, `store:false`, model-aware prompt-cache +configuration, optional HMAC safety identifier, no ZDR header). It **cannot** tell us the contractual posture. The following are **operator/legal actions**, not code facts, and must be confirmed: - Whether a **Data Processing Addendum (DPA)** / OpenAI Business/Enterprise agreement is in place for the account behind `OPENAI_API_KEY`. -- Whether **Zero Data Retention (ZDR)** has been granted for the org (which would also remove the 24h - prompt-cache window). +- Whether **Zero Data Retention (ZDR)** has been granted for the org and how it applies to the + configured prompt-cache lifetime. - OpenAI's standard API commitment (no training on API data by default; limited abuse-monitoring retention) — this needs to be pinned to the specific contract, not assumed. @@ -434,13 +443,15 @@ remaining items are compliance-posture and PHI-minimisation gaps. APP privacy policy and retain the point-of-entry links/notices. Broader retention and breach-response documentation also remains outstanding. No legal approval is claimed here. -### PIA-6 — OpenAI prompt-cache retention forced to 24h **(Low-Medium)** +### PIA-6 — OpenAI prompt-cache lifetime requires contractual confirmation **(Low-Medium)** -- **Risk:** Query + retrieved excerpts persist at OpenAI for ≤24h via prompt caching even with - `store:false`; not operator-tunable for gpt-5.5. -- **Evidence:** [openai.ts:168, 208, 221-222](src/lib/openai.ts). -- **Fix:** Resolve via **ZDR** (removes the window) as part of PIA-1; document the 24h window in the - meantime. If a shorter window becomes configurable, expose it. +- **Risk:** Query + retrieved excerpts can enter OpenAI prompt caches even with `store:false`. + GPT-5.6 requests a 30-minute TTL by default, but that value is a minimum and is not a contractual + deletion deadline. Explicit pre-5.6 models can still request the legacy 24-hour retention mode. +- **Evidence:** [openai.ts](src/lib/openai.ts), [.env.example](../.env.example). +- **Fix:** Confirm the effective cache/deletion behavior under the production project's **ZDR** and + data-residency terms. Keep `OPENAI_PROMPT_CACHE_TTL=off` available when governance requires the app + to omit the extended GPT-5.6 TTL option; document that provider-default caching policy still applies. ### PIA-7 — `RAG_PERSIST_RAW_QUERY_TEXT=true` stores raw PHI query text **(Low, config-gated)** diff --git a/docs/search-rag-master-context.md b/docs/search-rag-master-context.md index 64f9492a6..a5b969f88 100644 --- a/docs/search-rag-master-context.md +++ b/docs/search-rag-master-context.md @@ -238,7 +238,7 @@ Manual parsing and clamping should move into shared utilities only. - Preserve answer in-flight coalescing for duplicate requests where available. - Avoid repeating expensive generation after cancellation or client retry. - Keep prompt/cache versioning explicit when schema changes. -- Track model route, retry path, usage, request IDs, latency, cached-input tokens, and fallback reason. +- Track model route, retry path, usage, request IDs, latency, cached-input/cache-write tokens, and fallback reason. - Keep answer-generation timeout bounded separately from the global OpenAI request timeout. - Keep explicit source/table/document lookup paths model-free when retrieval support is strong enough. - Cap UI supplemental block counts to reduce render noise. diff --git a/docs/worker-deploy-runbook.md b/docs/worker-deploy-runbook.md index 8fec7783d..49d1ddb71 100644 --- a/docs/worker-deploy-runbook.md +++ b/docs/worker-deploy-runbook.md @@ -148,7 +148,8 @@ the client publishable key (build-time, app bundle only) or - Models/dimensions: `OPENAI_EMBEDDING_MODEL=text-embedding-3-small`, `EMBEDDING_DIMENSIONS=1536` (must match `vector(N)` in `supabase/schema.sql` — a mismatch is caught by the startup dimension probe), - `OPENAI_VISION_MODEL=gpt-5.5`. + `OPENAI_VISION_MODEL=gpt-5.6-terra` and + `OPENAI_INDEXING_MODEL=gpt-5.6-terra`. - `RAG_PROVIDER_MODE=auto` (OpenAI with graceful source-only fallback). - Worker knobs (all defaulted): `WORKER_POLL_MS=30000`, `WORKER_BATCH_SIZE=3`, `WORKER_CONCURRENCY=1`, `WORKER_MAX_ATTEMPTS=3`, diff --git a/scripts/check-client-bundle-secrets.mjs b/scripts/check-client-bundle-secrets.mjs index 89e44b74c..9319a103f 100644 --- a/scripts/check-client-bundle-secrets.mjs +++ b/scripts/check-client-bundle-secrets.mjs @@ -7,6 +7,7 @@ const textExtensions = new Set([".css", ".html", ".js", ".json", ".map", ".md", const forbiddenMarkers = [ "SUPABASE_SERVICE_ROLE_KEY", "OPENAI_API_KEY", + "OPENAI_SAFETY_IDENTIFIER_SECRET", "OPENAI_ORG_ID", "OPENAI_PROJECT_ID", "RAG_QUERY_HASH_SECRET", diff --git a/scripts/eval-answer-quality.ts b/scripts/eval-answer-quality.ts index f7419f238..6ebda9a9e 100644 --- a/scripts/eval-answer-quality.ts +++ b/scripts/eval-answer-quality.ts @@ -79,14 +79,16 @@ async function main() { const targetingByIntent = new Map(); let targetingApplicable = 0; let targetingHit = 0; - const targetingMisses: Array<{ id: string; intent: string; reason: string; answer: string }> = []; + const targetingMisses: Array<{ id: string; intent: string; reason: string; answer_length: number }> = []; + const caseResults: Array> = []; for (const testCase of cases) { const answer = (await withProviderBackoff(`answer-quality:${testCase.id}`, () => answerQuestionWithScope({ query: testCase.question, ownerId, logQuery: false, skipCache: true }), )) as RagAnswer; - for (const score of scoreAnswerQualityEvalCase(testCase, answer)) { + const metricScores = scoreAnswerQualityEvalCase(testCase, answer); + for (const score of metricScores) { metricTotals[score.metric] += score.score; } @@ -103,11 +105,26 @@ async function main() { id: testCase.id, intent: testCase.expectedIntent, reason: targeting.reason, - answer: (answer.answer ?? "").replace(/\s+/g, " ").slice(0, 160), + answer_length: answer.answer?.length ?? 0, }); } } targetingByIntent.set(testCase.expectedIntent, bucket); + caseResults.push({ + id: testCase.id, + intent: testCase.expectedIntent, + grounded: answer.grounded, + confidence: answer.confidence, + route: answer.routingMode, + query_class: answer.queryClass ?? null, + model: answer.modelUsed ?? null, + citation_count: answer.citations.length, + routing_reason: answer.routingReason ?? null, + metrics: Object.fromEntries(metricScores.map((score) => [score.metric, score.score])), + targeting: targeting.applicable ? targeting.score : null, + targeting_reason: targeting.reason, + answer_length: answer.answer?.length ?? 0, + }); } const caseCount = cases.length; @@ -131,6 +148,7 @@ async function main() { targeting_rate: targetingRate, targeting_by_intent: targetingByIntentRates, targeting_misses: targetingMisses, + case_results: caseResults, }; if (args.json) { @@ -147,7 +165,7 @@ async function main() { if (targetingMisses.length) { console.log(" targeting_misses:"); for (const miss of targetingMisses) { - console.log(` [${miss.intent}] ${miss.id}: ${miss.reason} :: "${miss.answer}"`); + console.log(` [${miss.intent}] ${miss.id}: ${miss.reason} :: answer_length=${miss.answer_length}`); } } } diff --git a/scripts/production-readiness.ts b/scripts/production-readiness.ts index d033e847a..4d643c96a 100644 --- a/scripts/production-readiness.ts +++ b/scripts/production-readiness.ts @@ -229,6 +229,14 @@ async function main() { } } + if (envModule.env.OPENAI_API_KEY && !envModule.env.OPENAI_SAFETY_IDENTIFIER_SECRET) { + result.warnings.push( + "OPENAI_SAFETY_IDENTIFIER_SECRET is not set; authenticated Responses requests omit the privacy-preserving safety identifier.", + ); + } else if (envModule.env.OPENAI_SAFETY_IDENTIFIER_SECRET) { + result.passes.push("OpenAI safety identifiers use a deployment-secret HMAC; raw owner IDs are not sent."); + } + // Exercise the real boot guard so this check tracks its behaviour instead of // re-encoding the env rule (mirrors requireServerEnv/requireOpenAIEnv above). A // present secret passes in any environment; a missing one fails closed only in a diff --git a/src/app/api/answer/route.ts b/src/app/api/answer/route.ts index de3cb580c..43f6071fa 100644 --- a/src/app/api/answer/route.ts +++ b/src/app/api/answer/route.ts @@ -13,8 +13,8 @@ import { import { publicAccessContext } from "@/lib/public-api-access"; import { classifyRagQuery } from "@/lib/clinical-search"; import { buildSmartRagApiPlan } from "@/lib/smart-rag-api"; -import { clinicalQueryModeSchema, queryClassForClinicalMode, queryForClinicalMode } from "@/lib/clinical-query-mode"; -import { resolveSearchScope, searchScopeFiltersSchema } from "@/lib/search-scope"; +import { queryClassForClinicalMode, queryForClinicalMode } from "@/lib/clinical-query-mode"; +import { resolveSearchScope } from "@/lib/search-scope"; import { resolveRetrievalAccessScope } from "@/lib/owner-scope"; import { sourceGovernanceWarnings } from "@/lib/source-governance"; import { parseJsonBody } from "@/lib/validation/body"; @@ -29,19 +29,11 @@ import { captureServerException } from "@/lib/observability/error-capture"; import { logAnswerDiagnostics } from "@/lib/answer-telemetry"; import { nonProductionSupabaseDemoFallbackReason } from "@/lib/supabase/errors"; import * as serverAuth from "@/lib/supabase/auth"; +import { answerRequestSchema, type AnswerRequestBody } from "@/lib/validation/answer-request"; import { answerFeedbackMetadata } from "@/lib/answer-feedback-token"; export const runtime = "nodejs"; -const answerSchema = z.object({ - query: z.string().trim().min(1).max(2000), - documentId: z.string().uuid().optional(), - documentIds: z.array(z.string().uuid()).max(25).optional(), - filters: searchScopeFiltersSchema.optional(), - queryMode: clinicalQueryModeSchema.optional().default("auto"), -}); - -type AnswerRequestBody = z.infer; const emptyScopeAnswer = "The selected filters did not match any indexed documents, so I cannot generate an answer for that scope."; @@ -70,7 +62,7 @@ export async function POST(request: Request) { const routeStartedAt = Date.now(); let body: AnswerRequestBody | null = null; try { - const answerBody = await parseJsonBody(request, answerSchema, "Invalid answer request."); + const answerBody = await parseJsonBody(request, answerRequestSchema, "Invalid answer request."); body = answerBody; if (isDemoMode()) { return NextResponse.json({ ...buildDemoAnswerPayload(answerBody), interactionId }); diff --git a/src/app/api/answer/stream/route.ts b/src/app/api/answer/stream/route.ts index f5a86d491..52416b8c3 100644 --- a/src/app/api/answer/stream/route.ts +++ b/src/app/api/answer/stream/route.ts @@ -19,8 +19,8 @@ import { answerQuestionWithScope, summarizeDocument, type AnswerProgressEvent } import { classifyRagQuery } from "@/lib/clinical-search"; import { annotateSearchResults, buildEvidenceRelevance } from "@/lib/evidence-relevance"; import { buildSmartRagApiPlan } from "@/lib/smart-rag-api"; -import { clinicalQueryModeSchema, queryClassForClinicalMode, queryForClinicalMode } from "@/lib/clinical-query-mode"; -import { resolveSearchScope, searchScopeFiltersSchema } from "@/lib/search-scope"; +import { queryClassForClinicalMode, queryForClinicalMode } from "@/lib/clinical-query-mode"; +import { resolveSearchScope } from "@/lib/search-scope"; import { resolveRetrievalAccessScope, type RetrievalAccessScope } from "@/lib/owner-scope"; import { sourceGovernanceWarnings } from "@/lib/source-governance"; import { createAdminClient } from "@/lib/supabase/admin"; @@ -32,31 +32,13 @@ import { logger } from "@/lib/logger"; import { safeErrorLogDetails } from "@/lib/privacy"; import { startSseHeartbeat } from "@/lib/sse-heartbeat"; import { parseJsonBody } from "@/lib/validation/body"; +import { answerRequestSchema, type AnswerRequestBody } from "@/lib/validation/answer-request"; +import type { AnswerStreamEventMap, AnswerStreamEventName } from "@/lib/answer-stream-contract"; import { toPublicAnswerProgressEvent } from "@/lib/answer-progress-public"; import { answerFeedbackMetadata } from "@/lib/answer-feedback-token"; export const runtime = "nodejs"; -const answerSchema = z - .object({ - query: z.string().trim().min(1).max(2000), - documentId: z.string().uuid().optional(), - documentIds: z.array(z.string().uuid()).max(25).optional(), - filters: searchScopeFiltersSchema.optional(), - queryMode: clinicalQueryModeSchema.optional().default("auto"), - summaryMode: z.boolean().optional().default(false), - }) - .superRefine((value, context) => { - if (value.summaryMode && !value.documentId) { - context.addIssue({ - code: "custom", - path: ["documentId"], - message: "Document summary mode requires a document id.", - }); - } - }); - -type AnswerBody = z.infer; const emptyScopeAnswer = "The selected filters did not match any indexed documents, so I cannot generate an answer for that scope."; @@ -120,7 +102,7 @@ function logStreamError(error: unknown, signal?: AbortSignal) { void captureServerException(error, { route: "api/answer/stream", source: "stream" }); } -function buildDemoStreamAnswer(body: AnswerBody, fallbackReason?: string) { +function buildDemoStreamAnswer(body: AnswerRequestBody, fallbackReason?: string) { const demo = body.summaryMode && body.documentId ? demoSummary(body.documentId) @@ -146,7 +128,7 @@ function buildDemoStreamAnswer(body: AnswerBody, fallbackReason?: string) { ); } -function streamAnswer(body: AnswerBody, accessScope: RetrievalAccessScope, signal?: AbortSignal) { +function streamAnswer(body: AnswerRequestBody, accessScope: RetrievalAccessScope, signal?: AbortSignal) { const ownerId = accessScope.ownerId; const encoder = new TextEncoder(); const interactionId = randomUUID(); @@ -156,7 +138,7 @@ function streamAnswer(body: AnswerBody, accessScope: RetrievalAccessScope, signa async start(controller) { const streamStartedAt = Date.now(); let completionSent = false; - const send = (event: string, data: unknown) => { + const send = (event: Name, data: AnswerStreamEventMap[Name]) => { try { controller.enqueue(encoder.encode(encodeSse(event, data))); } catch { @@ -173,20 +155,15 @@ function streamAnswer(body: AnswerBody, accessScope: RetrievalAccessScope, signa const sendComplete = () => { sendProgress({ stage: "complete", elapsedMs: Date.now() - streamStartedAt }); }; - const sendFinal = (data: unknown) => { + const sendFinal = (data: AnswerStreamEventMap["final"]) => { sendComplete(); send("final", data); }; - // Generation can go silent for long stretches (strong-route reasoning - // before the first output token); heartbeat comments keep the - // connection visibly alive for proxies and the client's stall watchdog. + // Generation can go silent for long stretches while the model reasons + // and deterministic gates run; heartbeat comments keep the connection + // visibly alive without exposing provisional clinical prose. const stopHeartbeat = startSseHeartbeat((frame) => controller.enqueue(encoder.encode(frame))); const onProgress = (event: AnswerProgressEvent) => sendProgress(event); - // Stream the answer prose as it generates (content-preserving) and signal a reset when a - // provisional answer is being revised by the quality gates. - const onToken = (delta: string) => send("token", { delta }); - const onRevising = () => send("revising", {}); - try { sendProgress({ stage: "scoping" }); const scope = isDemoMode() @@ -238,8 +215,6 @@ function streamAnswer(body: AnswerBody, accessScope: RetrievalAccessScope, signa allowGlobalSearch: !ownerId, queryMode: body.queryMode, onProgress, - onToken, - onRevising, signal, }); const governedResponse = buildGovernedAnswerClientResponse(answer); @@ -292,7 +267,7 @@ function streamAnswer(body: AnswerBody, accessScope: RetrievalAccessScope, signa export async function POST(request: Request) { try { - const body = await parseJsonBody(request, answerSchema, "Invalid answer request."); + const body = await parseJsonBody(request, answerRequestSchema, "Invalid answer request."); if (isDemoMode()) return streamAnswer(body, resolveRetrievalAccessScope(), request.signal); const supabase = createAdminClient(); diff --git a/src/app/api/documents/[id]/table-facts/route.ts b/src/app/api/documents/[id]/table-facts/route.ts index f146315f7..18653deb5 100644 --- a/src/app/api/documents/[id]/table-facts/route.ts +++ b/src/app/api/documents/[id]/table-facts/route.ts @@ -8,12 +8,14 @@ import { createAdminClient } from "@/lib/supabase/admin"; import { AuthenticationError, requireAuthenticatedUser, unauthorizedResponse } from "@/lib/supabase/auth"; import { tableReviewMetadata, tableReviewSchema } from "@/lib/table-review"; import { parseJsonBody } from "@/lib/validation/body"; +import { parseRouteParams } from "@/lib/validation/params"; export const runtime = "nodejs"; const updateSchema = tableReviewSchema.extend({ factId: z.string().uuid(), }); +const tableFactsRouteParamsSchema = z.object({ id: z.string().uuid() }); function metadataRecord(value: unknown) { return value && typeof value === "object" && !Array.isArray(value) ? { ...(value as Record) } : {}; @@ -36,7 +38,8 @@ async function loadOwnedDocument(args: { export async function GET(request: Request, { params }: { params: Promise<{ id: string }> }) { try { - const { id } = await params; + const { id: rawId } = await params; + const { id } = parseRouteParams({ id: rawId }, tableFactsRouteParamsSchema, "Invalid document id."); if (isDemoMode()) return NextResponse.json({ tableFacts: [], demoMode: true }); const supabase = createAdminClient(); @@ -67,7 +70,8 @@ export async function GET(request: Request, { params }: { params: Promise<{ id: export async function PATCH(request: Request, { params }: { params: Promise<{ id: string }> }) { try { - const { id } = await params; + const { id: rawId } = await params; + const { id } = parseRouteParams({ id: rawId }, tableFactsRouteParamsSchema, "Invalid document id."); if (isDemoMode()) return NextResponse.json({ error: "Table review is unavailable in demo mode." }, { status: 400 }); const parsed = await parseJsonBody(request, updateSchema, "Table review payload is invalid."); diff --git a/src/components/ClinicalDashboard.tsx b/src/components/ClinicalDashboard.tsx index 6b0151e04..016dc9373 100644 --- a/src/components/ClinicalDashboard.tsx +++ b/src/components/ClinicalDashboard.tsx @@ -330,34 +330,6 @@ function isAbortError(error: unknown): boolean { return error instanceof DOMException && error.name === "AbortError"; } -// Provisional view shown while an answer streams in. The prose is content-preserving (the same -// text the final payload will carry); the caret conveys that generation is still in flight. On a -// quality-gate escalation the pipeline sends a `revising` signal and this switches to a neutral -// "revising for accuracy" state so a clinician never acts on soon-to-be-replaced text. -function StreamingAnswerPreview({ text, revising }: { text: string; revising: boolean }) { - if (revising) { - return ( -
-
- - Revising for accuracy… -
-
- ); - } - return ( -
-

- {text} - -

-
- ); -} - function normalizeNavigationHash(hash: string) { return navigationHashes.includes(hash as (typeof navigationHashes)[number]) ? hash : "#search"; } @@ -377,7 +349,7 @@ const maxVisiblePriorTurns = 10; // Non-retryable so an aborted request does not immediately re-fetch against the // already-aborted signal; the user re-submits to try again. Raised by the // stall watchdog (see createAnswerRequestWatchdog): a live stream that keeps -// delivering progress/token/heartbeat bytes is never aborted, no matter how +// delivering progress/heartbeat bytes is never aborted, no matter how // long a fast->strong escalation takes, so this now only appears when the // stream genuinely went silent or hit the absolute ceiling. function answerTimedOutError() { @@ -746,11 +718,6 @@ export function ClinicalDashboard({ const [answerProgress, setAnswerProgress] = useState(null); const [answerProgressEvents, setAnswerProgressEvents] = useState([]); const [answerProgressStartedAt, setAnswerProgressStartedAt] = useState(null); - // In-progress streamed answer prose (content-preserving — the final committed answer still comes - // from the parsed `final` payload). null between searches; `{ text, revising }` while generating. - // `revising` = the quality gates dropped a provisional answer and are re-generating, so a - // "revising for accuracy" state shows instead of stale text. - const [streamingAnswer, setStreamingAnswer] = useState<{ text: string; revising: boolean } | null>(null); const [answerLifecycle, dispatchAnswerLifecycle] = useReducer(answerLifecycleReducer, initialAnswerLifecycle); const [error, setError] = useState(null); // Companion state for `error`, used to pick the right recovery UI (retry vs. @@ -881,7 +848,6 @@ export function ClinicalDashboard({ resetAnswerThread(); setAnswer(null); setSources([]); - setStreamingAnswer(null); setDocuments([]); setDocumentsPagination(null); setJobs([]); @@ -1877,7 +1843,6 @@ export function ClinicalDashboard({ signal?: AbortSignal, onStreamActivity?: () => void, ) { - setStreamingAnswer(null); let response: Response; try { response = await fetch("/api/answer/stream", { @@ -1910,19 +1875,7 @@ export function ClinicalDashboard({ let payload: AnswerPayload; try { - payload = await readAnswerStream( - response, - onProgress, - (delta) => { - dispatchAnswerLifecycle({ type: "stream" }); - setStreamingAnswer((prev) => ({ text: (prev?.text ?? "") + delta, revising: false })); - }, - () => { - dispatchAnswerLifecycle({ type: "revise" }); - setStreamingAnswer({ text: "", revising: true }); - }, - onStreamActivity, - ); + payload = await readAnswerStream(response, onProgress, onStreamActivity); } catch (error) { if (answerTimedOutRef.current) throw answerTimedOutError(); if (isAbortError(error)) throw error; @@ -1981,7 +1934,6 @@ export function ClinicalDashboard({ searchRequestSeqRef.current += 1; searchAbortRef.current?.abort(); searchAbortRef.current = null; - setStreamingAnswer(null); setLoading(false); setAnswerProgress(null); setAnswerProgressEvents([]); @@ -3845,11 +3797,7 @@ export function ClinicalDashboard({ ) ) : showAnswerPending ? ( - streamingAnswer && (streamingAnswer.text || streamingAnswer.revising) ? ( - - ) : ( - - ) + ) : answer && answerRenderModel ? ( stagedDashboardExtraction.answerSurface ? ( <> diff --git a/src/lib/answer-stream-contract.ts b/src/lib/answer-stream-contract.ts new file mode 100644 index 000000000..97a8b6dc5 --- /dev/null +++ b/src/lib/answer-stream-contract.ts @@ -0,0 +1,25 @@ +import type { PublicAnswerProgressEvent } from "@/lib/answer-progress-public"; + +export type AnswerStreamEventMap = { + progress: PublicAnswerProgressEvent; + final: unknown; + error: { + error: string; + status?: number; + details?: { code?: string; message?: string }; + }; +}; + +export type AnswerStreamEventName = keyof AnswerStreamEventMap; +export type AnswerStreamEvent = { + [Name in AnswerStreamEventName]: { event: Name; data: AnswerStreamEventMap[Name] }; +}[AnswerStreamEventName]; + +// Deliberately excludes the legacy `token` and `revising` event names. A new +// client can be routed to an older server during a rolling deployment, so +// accepting those events would re-expose unvalidated clinical prose. +const answerStreamEventNames = new Set(["progress", "final", "error"]); + +export function isAnswerStreamEventName(value: string): value is AnswerStreamEventName { + return answerStreamEventNames.has(value as AnswerStreamEventName); +} diff --git a/src/lib/answer-stream-extractor.ts b/src/lib/answer-stream-extractor.ts deleted file mode 100644 index d7ad2d5de..000000000 --- a/src/lib/answer-stream-extractor.ts +++ /dev/null @@ -1,76 +0,0 @@ -// Incremental extraction of the `answer` prose field from a streaming structured-JSON response. -// -// Answer generation uses a strict json_schema output, so the model streams raw JSON text -// (`{"answer":"…prose…","confidence":"high",…}`) rather than plain prose. To surface the answer -// as it streams — without changing what is generated (the final parsed answer is byte-identical -// to the non-streaming path) — we scan the accumulated raw buffer for the `answer` field's string -// value and emit only the newly-decoded characters since the last call. -// -// The scan stops at a safe boundary (an unescaped closing quote, or before an incomplete trailing -// escape), so JSON.parse of the captured fragment never fails on a mid-escape cut. If a fragment -// still cannot be decoded (e.g. a Unicode escape split across chunk boundaries), we emit nothing -// this round; the next, longer buffer completes it. Delivery is best-effort and lossless in -// aggregate — the authoritative answer always comes from parsing the final full payload. - -const answerFieldOpening = /"answer"\s*:\s*"/; - -/** Decode the JSON-escaped answer content captured up to a safe (non-mid-escape) boundary. */ -function decodeCapturedAnswer(rawEscaped: string): string | null { - try { - return JSON.parse(`"${rawEscaped}"`) as string; - } catch { - return null; - } -} - -/** - * Stateful extractor over a growing raw-JSON buffer. Call `push(fullBufferSoFar)` on each streamed - * chunk (passing the entire accumulated buffer, not just the new bytes); it returns the decoded - * answer-prose delta to append to the UI, or "" when there is nothing new yet. - */ -export function createStreamingAnswerExtractor() { - let emitted = 0; - - return { - push(rawBuffer: string): string { - const opening = answerFieldOpening.exec(rawBuffer); - if (!opening) return ""; - const valueStart = opening.index + opening[0].length; - - let escaped = ""; - let i = valueStart; - while (i < rawBuffer.length) { - const ch = rawBuffer[i]; - if (ch === "\\") { - const next = rawBuffer[i + 1]; - // Incomplete trailing escape (buffer cut right after a backslash) — stop before it. - if (next === undefined) break; - if (next === "u") { - // \uXXXX needs four hex digits; if the buffer cuts inside them, hold back the whole - // escape and emit only the safe prefix decoded so far. - if (i + 6 > rawBuffer.length) break; - escaped += rawBuffer.slice(i, i + 6); - i += 6; - continue; - } - escaped += rawBuffer[i] + next; - i += 2; - continue; - } - if (ch === '"') break; // unescaped closing quote — the answer value is complete - escaped += ch; - i += 1; - } - - const decoded = decodeCapturedAnswer(escaped); - if (decoded === null || decoded.length <= emitted) return ""; - const delta = decoded.slice(emitted); - emitted = decoded.length; - return delta; - }, - /** Total decoded prose length emitted so far (for tests / diagnostics). */ - get emittedLength() { - return emitted; - }, - }; -} diff --git a/src/lib/document-enrichment.ts b/src/lib/document-enrichment.ts index 0332cddb1..3f7089839 100644 --- a/src/lib/document-enrichment.ts +++ b/src/lib/document-enrichment.ts @@ -559,7 +559,7 @@ export async function generateDocumentEnrichment(args: { buildEnrichmentPrompt({ ...args, images: args.images ?? [] }), summarySchema, { - model: env.OPENAI_STRONG_ANSWER_MODEL || env.OPENAI_FAST_ANSWER_MODEL, + model: env.OPENAI_INDEXING_MODEL, // Answer-size budget; responseBody() floors the effective cap by reasoning effort so // medium-effort reasoning cannot starve the enrichment JSON (reasoningHeadroomFloor). maxOutputTokens: 2400, @@ -653,7 +653,7 @@ export async function upsertDocumentEnrichment(args: { clinical_specifics: enrichment.clinical_specifics, source_chunk_ids: sourceChunkIds, source_image_ids: sourceImageIds, - model: env.OPENAI_STRONG_ANSWER_MODEL || env.OPENAI_FAST_ANSWER_MODEL, + model: env.OPENAI_INDEXING_MODEL, metadata: { ...generatedMetadata, ...coverageMetadata, label_count: enrichment.labels.length }, generated_at: enrichedAt, updated_at: enrichedAt, diff --git a/src/lib/env.ts b/src/lib/env.ts index 5e9d30a86..3cfa77427 100644 --- a/src/lib/env.ts +++ b/src/lib/env.ts @@ -36,11 +36,14 @@ const envSchema = z.object({ // Must match the vector(N) dimension in supabase/schema.sql. Changing the embedding // model without updating this (and the schema) silently corrupts ingestion (IDX-C2). EMBEDDING_DIMENSIONS: z.coerce.number().int().positive().default(1536), - OPENAI_ANSWER_MODEL: z.string().default("gpt-5.5"), - OPENAI_FAST_ANSWER_MODEL: z.string().default("gpt-5.5"), - // Strong tier intentionally stays on the standard (non-"pro") model. Fast vs strong - // is differentiated by reasoning effort (OPENAI_*_REASONING_EFFORT), not model tier. - OPENAI_STRONG_ANSWER_MODEL: z.string().default("gpt-5.5"), + OPENAI_ANSWER_MODEL: z.string().default("gpt-5.6-terra"), + OPENAI_FAST_ANSWER_MODEL: z.string().default("gpt-5.6-terra"), + OPENAI_STRONG_ANSWER_MODEL: z.string().default("gpt-5.6-sol"), + // Workload-specific overrides keep rollout experiments isolated. When omitted, + // the resolved values below preserve the existing answer-tier behaviour. + OPENAI_QUERY_CLASSIFIER_MODEL: z.string().optional(), + OPENAI_SUMMARY_MODEL: z.string().optional(), + OPENAI_INDEXING_MODEL: z.string().optional(), // Reasoning models (gpt-5*) draw reasoning tokens from this SAME budget as the // visible answer, so a low cap makes medium/high-effort reasoning consume the whole // budget *thinking* and return `incomplete: max_output_tokens` BEFORE it writes the @@ -73,7 +76,7 @@ const envSchema = z.object({ // batches of this size. 256 keeps total tokens well under the ceiling even for the // largest (narrative-profile) chunks while staying far below the 2048 input cap. OPENAI_EMBEDDING_BATCH_SIZE: z.coerce.number().int().positive().max(2048).default(256), - OPENAI_VISION_MODEL: z.string().default("gpt-5.5"), + OPENAI_VISION_MODEL: z.string().default("gpt-5.6-terra"), OPENAI_VISION_IMAGE_DETAIL: z.enum(["auto", "low", "high"]).default("auto"), OPENAI_REQUEST_TIMEOUT_MS: z.coerce.number().int().positive().default(45000), // Answer generation has a source-backed fallback path, but a too-tight budget @@ -87,7 +90,13 @@ const envSchema = z.object({ OPENAI_ANSWER_TIMEOUT_MS: z.coerce.number().int().positive().default(30000), OPENAI_MAX_RETRIES: z.coerce.number().int().nonnegative().default(2), OPENAI_GENERATION_MAX_RETRIES: z.coerce.number().int().nonnegative().default(0), + // Legacy Responses prompt-cache retention for pre-5.6 models. GPT-5.6 uses + // OPENAI_PROMPT_CACHE_TTL and never receives this deprecated field. OPENAI_PROMPT_CACHE_RETENTION: z.enum(["off", "in_memory", "24h"]).default("24h"), + OPENAI_PROMPT_CACHE_TTL: z.enum(["off", "30m"]).optional(), + // Optional deployment-secret HMAC key for privacy-preserving Responses API + // safety identifiers. Raw owner/user identifiers are never sent to OpenAI. + OPENAI_SAFETY_IDENTIFIER_SECRET: z.string().min(32).optional(), OPENAI_STORE_RESPONSES: z .enum(["true", "false"]) .default("false") @@ -206,7 +215,7 @@ const envSchema = z.object({ }); const parsedEnv = envSchema.parse(process.env); -const nonProAnswerModelFallback = "gpt-5.5"; +const nonProAnswerModelFallback = "gpt-5.6-terra"; function isProAnswerModel(model: string) { return /(?:^|[-_])pro(?:$|[-_])/i.test(model); @@ -227,6 +236,11 @@ export const env = { OPENAI_ANSWER_MODEL: runtimeAnswerModel(parsedEnv.OPENAI_ANSWER_MODEL), OPENAI_FAST_ANSWER_MODEL: runtimeAnswerModel(parsedEnv.OPENAI_FAST_ANSWER_MODEL), OPENAI_STRONG_ANSWER_MODEL: runtimeAnswerModel(parsedEnv.OPENAI_STRONG_ANSWER_MODEL), + OPENAI_QUERY_CLASSIFIER_MODEL: runtimeAnswerModel( + parsedEnv.OPENAI_QUERY_CLASSIFIER_MODEL ?? parsedEnv.OPENAI_FAST_ANSWER_MODEL, + ), + OPENAI_SUMMARY_MODEL: runtimeAnswerModel(parsedEnv.OPENAI_SUMMARY_MODEL ?? parsedEnv.OPENAI_ANSWER_MODEL), + OPENAI_INDEXING_MODEL: runtimeAnswerModel(parsedEnv.OPENAI_INDEXING_MODEL ?? parsedEnv.OPENAI_STRONG_ANSWER_MODEL), } satisfies typeof parsedEnv; export function requireServerEnv(): { diff --git a/src/lib/eval-document-matching.ts b/src/lib/eval-document-matching.ts index afec16d2d..185e0b0e6 100644 --- a/src/lib/eval-document-matching.ts +++ b/src/lib/eval-document-matching.ts @@ -17,6 +17,7 @@ export function normalizedDocumentName(value: string) { } const clinicalDocumentAliases: Record = { + Acamprosate: ["Acamprosate"], ActiveCommunityPtED: [ "Active Community Patients in the Emergency Department", "Active Community Patients Emergency Department", @@ -29,6 +30,7 @@ const clinicalDocumentAliases: Record = { "Mental Health Pharmacological Management of Agitation and Arousal", ], AssessmentDocumentation: ["Assessment Documentation", "Clinical Assessment", "Mental Health Assessment"], + ADHD: ["ADHD", "Attention Deficit Hyperactivity Disorder"], BestPracticePrescription: ["Best Practice Prescription", "Best Practice Prescribing", "Prescription"], ClozapinePresAdminMonitor: [ "Clozapine Prescribing Administration Monitoring", @@ -65,11 +67,15 @@ const clinicalDocumentAliases: Record = { "Olanzapine LAI", "Long Acting Injectable Antipsychotic", ], + Lithium: ["Lithium", "Lithium Clinical Guideline", "Lithium CAMHS"], + Metformin: ["Metformin"], MetabolicScreening: ["Metabolic Screening", "Metabolic Monitoring", "Physical Health Monitoring"], MHATMHCTTreatmentTeamProcess: ["Mental Health Treatment Team Process", "Treatment Team Process", "MHAT", "MHCT"], NeurolepticSideEffect: ["Neuroleptic Side Effects", "Neuroleptic Side Effect", "Neuroleptic Effects"], NOCC: ["NOCC", "National Outcomes and Casemix Collection", "Outcome Measures Completion"], + Naltrexone: ["Naltrexone"], PtSafetyPlan: ["Patient Safety Plan", "Safety Planning", "Safety Plan"], + Sertraline: ["Sertraline"], }; export function documentExpectationAlternatives(expectation: string) { diff --git a/src/lib/model-index-extraction.ts b/src/lib/model-index-extraction.ts index f2b70f8df..33422a728 100644 --- a/src/lib/model-index-extraction.ts +++ b/src/lib/model-index-extraction.ts @@ -367,7 +367,7 @@ export async function generateModelIndexProfile(args: { images?: ModelIndexImage[]; }) { if (args.chunks.length === 0) return emptyProfile(); - const model = env.OPENAI_STRONG_ANSWER_MODEL || env.OPENAI_ANSWER_MODEL; + const model = env.OPENAI_INDEXING_MODEL; const result = await generateStructuredTextResult(buildPrompt({ ...args, images: args.images ?? [] }), schema, { model, // Answer-size budget; responseBody() floors the effective cap by reasoning effort so diff --git a/src/lib/openai.ts b/src/lib/openai.ts index 0c736d6cd..d09231c46 100644 --- a/src/lib/openai.ts +++ b/src/lib/openai.ts @@ -1,4 +1,7 @@ +import { createHmac } from "node:crypto"; import OpenAI from "openai"; +import { zodTextFormat } from "openai/helpers/zod"; +import type { ZodType } from "zod"; import { env, requireOpenAIEnv } from "@/lib/env"; import { assessClinicalImageUse } from "@/lib/image-filtering"; import { PublicApiError } from "@/lib/http"; @@ -29,12 +32,7 @@ type TextGenerationOptions = { timeoutMs?: number; maxRetries?: number; signal?: AbortSignal; - // When set, the request is streamed and this is invoked with each raw output-text delta as it - // arrives. The returned OpenAITextResult is identical to the non-streaming path (same accumulated - // text, usage, and completion status), so streaming never changes what is generated — only when - // the caller sees it. Retries are disabled for streamed requests (a half-streamed retry would - // double-emit), so callers must tolerate that. - onOutputTextDelta?: (delta: string) => void; + safetyIdentifier?: string; }; type ResolvedTextGenerationOptions = Required> & @@ -59,6 +57,8 @@ export type OpenAITextResult = { incompleteReason?: string; }; +export type OpenAIParsedTextResult = OpenAITextResult & { parsed: T }; + let openAIClient: OpenAI | null = null; const queryEmbeddingCache = new Map(); const queryEmbeddingInflight = new Map>(); @@ -166,11 +166,16 @@ const reasoningEfforts = new Set(["low", "medium", "high" function openAIModelCapabilities(model: string) { const normalized = model.toLowerCase(); const isGpt5 = /^gpt-5(?:[.-]|$)/.test(normalized); + const gptVersionMatch = normalized.match(/^gpt-(\d+)(?:\.(\d+))?(?:[.-]|$)/); + const gptMajorVersion = Number(gptVersionMatch?.[1] ?? 0); + const gptMinorVersion = Number(gptVersionMatch?.[2] ?? 0); + const usesPromptCacheOptions = gptMajorVersion > 5 || (gptMajorVersion === 5 && gptMinorVersion >= 6); const isReasoningModel = isGpt5 || /^o\d(?:[.-]|$)/.test(normalized); return { supportsReasoning: isReasoningModel, supportsTextVerbosity: isGpt5, + usesPromptCacheOptions, requiredPromptCacheRetention: /^gpt-5\.5(?:[.-]|$)/.test(normalized) ? "24h" : undefined, allowedReasoningEfforts: isReasoningModel ? reasoningEfforts : new Set(), }; @@ -236,6 +241,12 @@ function responseBody( const configuredPromptCacheRetention = env.OPENAI_PROMPT_CACHE_RETENTION === "off" ? undefined : env.OPENAI_PROMPT_CACHE_RETENTION; const promptCacheRetention = capabilities.requiredPromptCacheRetention ?? configuredPromptCacheRetention; + const promptCacheTtl = + capabilities.usesPromptCacheOptions && + env.OPENAI_PROMPT_CACHE_TTL !== "off" && + !(env.OPENAI_PROMPT_CACHE_TTL === undefined && env.OPENAI_PROMPT_CACHE_RETENTION === "off") + ? (env.OPENAI_PROMPT_CACHE_TTL ?? "30m") + : undefined; if (format) textConfig.format = format; if (capabilities.supportsTextVerbosity) { @@ -249,7 +260,14 @@ function responseBody( max_output_tokens: Math.max(resolved.maxOutputTokens, reasoningHeadroomFloor(resolvedReasoningEffort)), store: env.OPENAI_STORE_RESPONSES, prompt_cache_key: resolved.promptCacheKey ?? promptCacheKeyFor(operation), - ...(promptCacheRetention ? { prompt_cache_retention: promptCacheRetention } : {}), + ...(capabilities.usesPromptCacheOptions + ? promptCacheTtl + ? { prompt_cache_options: { ttl: promptCacheTtl } } + : {} + : promptCacheRetention + ? { prompt_cache_retention: promptCacheRetention } + : {}), + ...(resolved.safetyIdentifier ? { safety_identifier: resolved.safetyIdentifier } : {}), metadata: { operation }, ...(resolvedReasoningEffort !== "none" ? { reasoning: { effort: resolvedReasoningEffort } } : {}), ...(Object.keys(textConfig).length > 0 ? { text: textConfig } : {}), @@ -297,6 +315,8 @@ function extractUsage(response: unknown): OpenAITokenUsage | undefined { output_tokens: typeof usage.output_tokens === "number" ? usage.output_tokens : undefined, total_tokens: typeof usage.total_tokens === "number" ? usage.total_tokens : undefined, cached_input_tokens: typeof inputDetails?.cached_tokens === "number" ? inputDetails.cached_tokens : undefined, + cache_write_tokens: + typeof inputDetails?.cache_write_tokens === "number" ? inputDetails.cache_write_tokens : undefined, reasoning_output_tokens: typeof outputDetails?.reasoning_tokens === "number" ? outputDetails.reasoning_tokens : undefined, }; @@ -348,14 +368,28 @@ export function mapOpenAIError(error: unknown, operation: OpenAIOperation) { if (isTimeoutError(error) || status === 408) { return new PublicApiError("OpenAI timed out. Trying source-only fallback response.", 504, { - code, + code: "openai_timeout", requestId, }); } - if (status === 401 || status === 403) { + if (status === 401) { return new PublicApiError("OpenAI authentication failed. Check the server API key configuration.", 500, { - code, + code: "openai_invalid_api_key", + requestId, + }); + } + + if (code === "model_not_found" || status === 404) { + return new PublicApiError("The configured OpenAI model is unavailable or this project cannot access it.", 500, { + code: "openai_model_not_found", + requestId, + }); + } + + if (status === 403) { + return new PublicApiError("OpenAI denied access. Check project, organization, and model permissions.", 500, { + code: "openai_access_denied", requestId, }); } @@ -371,7 +405,10 @@ export function mapOpenAIError(error: unknown, operation: OpenAIOperation) { } if (status === 429 || code === "rate_limit_exceeded") { - return new PublicApiError("OpenAI is rate limited. Retry in a moment.", 429, { code, requestId }); + return new PublicApiError("OpenAI is rate limited. Retry in a moment.", 429, { + code: "rate_limit_exceeded", + requestId, + }); } if (code.startsWith("invalid_image") || code === "image_too_large" || code === "unsupported_image_media_type") { @@ -387,101 +424,106 @@ export function mapOpenAIError(error: unknown, operation: OpenAIOperation) { if (status === 400) { return new PublicApiError("OpenAI rejected the request. Check the model, schema, and input configuration.", 502, { - code, + code: "openai_invalid_request", requestId, }); } if (status && status >= 500) { - return new PublicApiError("OpenAI service error. Retry shortly.", 502, { code, requestId }); + return new PublicApiError("OpenAI service error. Retry shortly.", 502, { + code: "openai_service_error", + requestId, + }); } - return new PublicApiError(`OpenAI ${operation.replaceAll("_", " ")} request failed.`, 502, { code, requestId }); + return new PublicApiError(`OpenAI ${operation.replaceAll("_", " ")} request failed.`, 502, { + code: "openai_request_failed", + requestId, + }); } async function createTextResult( input: OpenAIResponseInput, options: ResolvedTextGenerationOptions, format?: Record, -): Promise { + parseResponse = false, +): Promise { const operation = options.operation ?? "text_generation"; const startedAt = Date.now(); - let streamedRequestId: string | null = null; + let responseRequestId: string | null = null; // Buffered (non-streaming) request — the baseline behaviour. The `responses.create` overloads are // incompatible with our generic call-site; the double-cast works around the SDK type mismatch. async function requestBuffered(client: ReturnType): Promise { - const request = client.responses.create(responseBody(input, options, format) as never, requestOptions(options)); + const body = responseBody(input, options, format) as never; + const request = parseResponse + ? client.responses.parse(body, requestOptions(options)) + : client.responses.create(body, requestOptions(options)); const { data: responseData, request_id } = await unwrapOpenAIResponse( request as unknown as APIPromiseLike, ); - streamedRequestId = request_id ?? null; + responseRequestId = request_id ?? null; return responseData; } try { const client = createOpenAIClient(); - let data: unknown; - if (options.onOutputTextDelta) { - try { - data = await streamResponseData(client, input, options, format); - } catch { - // A streaming-specific failure must never make the answer worse than the buffered baseline: - // fall back to a normal generation so the caller still gets a complete answer. A genuine - // API failure (quota/auth) will also throw here and propagate exactly as it would today. - data = await requestBuffered(client); - } - } else { - data = await requestBuffered(client); - } + const data = await requestBuffered(client); const completion = extractCompletionStatus(data); + const outputText = extractOutputText(data); + const parsed = parseResponse ? (data as { output_parsed?: unknown }).output_parsed : undefined; + const requestId = responseRequestId ?? getRequestId(data); + + if (completion.status === "failed") { + throw new PublicApiError("OpenAI failed to complete the response.", 502, { + code: "openai_response_failed", + requestId, + }); + } + if (completion.incompleteReason === "content_filter") { + throw new PublicApiError("OpenAI could not complete the response because it was filtered.", 502, { + code: "openai_content_filtered", + requestId, + }); + } + const retryableEmptyTruncation = completion.truncated && completion.incompleteReason === "max_output_tokens"; + if (!outputText && !retryableEmptyTruncation) { + throw new PublicApiError("OpenAI returned no usable output.", 502, { + code: "openai_missing_output", + requestId, + }); + } + if (parseResponse && parsed == null) { + throw new PublicApiError("OpenAI returned no schema-valid parsed output.", 502, { + code: "openai_missing_parsed_output", + requestId, + }); + } + return { - text: extractOutputText(data), + text: outputText, model: options.model, operation, latencyMs: Date.now() - startedAt, - requestId: streamedRequestId ?? getRequestId(data), + requestId, usage: extractUsage(data), status: completion.status, truncated: completion.truncated, incompleteReason: completion.incompleteReason, + ...(parseResponse ? { parsed } : {}), }; } catch (error) { - throw mapOpenAIError(error, operation); - } - - // Streamed Responses request: iterate the event stream, forwarding each output-text delta to the - // caller, and return the final full Response object (identical shape to the non-streaming path) - // captured from the terminal completed/incomplete/failed event. maxRetries is forced to 0 because - // a retry mid-stream would re-emit already-delivered deltas. - async function streamResponseData( - client: ReturnType, - streamInput: OpenAIResponseInput, - streamOptions: ResolvedTextGenerationOptions, - streamFormat?: Record, - ): Promise { - const stream = (await client.responses.create( - { ...responseBody(streamInput, streamOptions, streamFormat), stream: true } as never, - { ...requestOptions(streamOptions), maxRetries: 0 }, - )) as unknown as AsyncIterable; - let finalResponse: unknown = null; - for await (const event of stream) { - if (event.type === "response.output_text.delta" && typeof event.delta === "string") { - streamOptions.onOutputTextDelta?.(event.delta); - } else if ( - event.type === "response.completed" || - event.type === "response.incomplete" || - event.type === "response.failed" - ) { - finalResponse = event.response; - } + if (options.signal?.aborted) { + throw options.signal.reason ?? new DOMException("The operation was aborted.", "AbortError"); } - if (!finalResponse) throw new Error("OpenAI stream ended without a final response."); - return finalResponse; + throw mapOpenAIError(error, operation); } } -type StreamEvent = { type: string; delta?: string; response?: unknown }; +export function openAISafetyIdentifier(ownerId: string | null | undefined) { + if (!ownerId || !env.OPENAI_SAFETY_IDENTIFIER_SECRET) return undefined; + return createHmac("sha256", env.OPENAI_SAFETY_IDENTIFIER_SECRET).update(ownerId, "utf8").digest("hex"); +} export function clearOpenAICaches() { queryEmbeddingCache.clear(); @@ -653,6 +695,20 @@ export async function generateStructuredTextResponse( return result.text; } +export async function generateParsedTextResult( + input: OpenAIResponseInput, + schema: ZodType, + options: string | TextGenerationOptions = env.OPENAI_ANSWER_MODEL, +): Promise> { + const resolved = resolveTextGenerationOptions(options, env.OPENAI_ANSWER_MODEL); + const format = zodTextFormat(schema, resolved.schemaName ?? "structured_output") as unknown as Record< + string, + unknown + >; + const result = await createTextResult(input, resolved, format, true); + return { ...result, parsed: result.parsed as T }; +} + const imageCaptionInstructions = "Generate a concise, clinically useful caption for an extracted guideline image. " + "Mention visible table or figure purpose, key labels, and any medication, risk, or monitoring details. " + diff --git a/src/lib/rag-answer-text.ts b/src/lib/rag-answer-text.ts index 7996b4370..d7fef497b 100644 --- a/src/lib/rag-answer-text.ts +++ b/src/lib/rag-answer-text.ts @@ -33,7 +33,11 @@ const orphanSourceHeadingPattern = const sourceInventoryWordingPattern = /\b(?:the\s+(?:strongest\s+)?retrieved\s+(?:source|sources|passages|excerpts)\s+(?:support|supports|show|shows|indicate|indicates)|retrieved\s+(?:source|sources|passages|excerpts)|indexed\s+source\s+passages\s+matched|no\s+concise\s+source\s+sentence|source-backed|based\s+on\s+(?:the\s+)?(?:provided\s+)?(?:sources|excerpts|passages|retrieved\s+sources)|dose evidence|monitoring evidence|table evidence|direct source-backed answer)\b/i; const clippedClinicalFragmentPattern = - /\b(?:stabili[sz]e\s+the\s+do|the\s+do\b|liver\s+functi\b|respiratio\b|if\s+a\s+60%\s+decrease\s+in\s+b\b)\b/i; + /\b(?:stabili[sz]e\s+the\s+do|the\s+do\b|liver\s+functi\b|respiratio\b|if\s+a\s+60%\s+decrease\s+in\s+b\b|guidance(?:\s+for\s+[^.!?]{1,60})?\s+is\s+that\s+(?:adjust|monitoring|higher\s+doses?\s+than|lower\s+doses?\s+than))\b/i; +const danglingClinicalClausePattern = + /(?:^|[.!?]\s+)[^.!?]*\b(?:before|after|with|than|for|to|of|and|or|when\s+compared\s+with)\s*[.!?](?:\s|$)/i; +const extractiveAnswerArtifactPattern = + /^(?:»|›|→)|(?:^|[.!?]\s+)(?:mg|mcg|micrograms?|g|ml|units?|iu)\b|\b(?:Best Uses|Bottom Line|clinical Focus)\b/i; const genericMedicationCasePatterns: Array<[RegExp, string]> = [ [/\bLithium Carbonate\b/g, "lithium carbonate"], [/\bClozapine\b/g, "clozapine"], @@ -286,15 +290,20 @@ export function hasClinicalAnswerQualityIssue(value: string) { sourceFormCodePattern.lastIndex = 0; bracketedCitationMarkerPattern.lastIndex = 0; clinicalAbbreviationCitationDigitPattern.lastIndex = 0; + const bareDoseFigurePattern = + /\b(?:maximum(?:\s+(?:recommended|daily))?(?:\s+dose)?|dose(?:\s+(?:of|is|to))?|increase(?:d)?\s+to)\s*:?\s*\d+(?:\.\d+)?(?!\d|\.\d|\s*(?:mg|milligrams?|mcg|micrograms?|g|grams?|kg|kilograms?|ml|millilit(?:er|re)s?|l|lit(?:er|re)s?|international\s+units?|units?|iu|mmol(?:\/l)?|millimoles?(?:\s+per\s+lit(?:er|re))?|meq|milliequivalents?|tablets?|capsules?|puffs?|drops?|sprays?|patch(?:es)?)\b)/i; return ( sourceInventoryWordingPattern.test(normalized) || clippedClinicalFragmentPattern.test(normalized) || + danglingClinicalClausePattern.test(normalized) || + extractiveAnswerArtifactPattern.test(normalized) || productCatalogueFragmentPattern.test(normalized) || brandOrFormularyFragmentPattern.test(normalized) || allCapsSourceHeadingPattern.test(normalized) || sourceFormCodePattern.test(normalized) || bracketedCitationMarkerPattern.test(normalized) || clinicalAbbreviationCitationDigitPattern.test(normalized) || + bareDoseFigurePattern.test(normalized) || /(?<=[a-z)])\d+(?=\.?(?:\s|$))/.test(normalized) ); } diff --git a/src/lib/rag-cache.ts b/src/lib/rag-cache.ts index 51bb512b3..90ee32774 100644 --- a/src/lib/rag-cache.ts +++ b/src/lib/rag-cache.ts @@ -13,6 +13,12 @@ import { normalizeSourceMetadata } from "@/lib/source-metadata"; import { retrievalPlanForQueryClass, type SearchChunksArgs, type SearchTelemetry } from "@/lib/rag-contracts"; import type { Json } from "@/lib/supabase/database.types"; import type { RagAnswer, RagQueryClass, SearchResult } from "@/lib/types"; +import { + ragAnswerPromptVersion, + ragAnswerSchemaVersion, + ragIndexingPromptVersion, + ragQueryClassifierPromptVersion, +} from "@/lib/rag-versioning"; const answerCache = new Map(); export const answerInflight = new Map>(); @@ -20,7 +26,7 @@ const searchCache = new Map< string, { expiresAt: number; results: SearchResult[]; telemetry: SearchTelemetry; indexingVersion: string } >(); -const ragCacheDependencyVersion = "rag-cache-v12"; +const ragCacheDependencyVersion = "rag-cache-v13"; const cacheIndexingVersionTtlMs = 5000; const cacheIndexingVersionMaxEntries = 512; const cacheIndexingVersionCache = new Map(); @@ -52,6 +58,57 @@ function modeKey(args: Pick) { return args.queryMode ?? "auto"; } +export type AnswerGenerationFingerprintInput = { + answerModel: string; + fastModel: string; + strongModel: string; + classifierModel: string; + indexingModel: string; + embeddingModel: string; + embeddingDimensions: number; + fastReasoningEffort: string; + strongReasoningEffort: string; + answerVerbosity: string; + maxOutputTokens: number; + providerMode: string; + promptVersion: string; + schemaVersion: string; + classifierPromptVersion: string; + retrievalVersion: string; + indexingPromptVersion: string; +}; + +export function buildAnswerGenerationFingerprint(input: AnswerGenerationFingerprintInput) { + return createHash("sha256").update(JSON.stringify(input)).digest("hex").slice(0, 20); +} + +export function answerGenerationFingerprint() { + return buildAnswerGenerationFingerprint({ + answerModel: env.OPENAI_ANSWER_MODEL, + fastModel: env.OPENAI_FAST_ANSWER_MODEL, + strongModel: env.OPENAI_STRONG_ANSWER_MODEL, + classifierModel: env.OPENAI_QUERY_CLASSIFIER_MODEL, + indexingModel: env.OPENAI_INDEXING_MODEL, + embeddingModel: env.OPENAI_EMBEDDING_MODEL, + embeddingDimensions: env.EMBEDDING_DIMENSIONS, + fastReasoningEffort: env.OPENAI_FAST_REASONING_EFFORT, + strongReasoningEffort: env.OPENAI_STRONG_REASONING_EFFORT, + answerVerbosity: env.OPENAI_TEXT_VERBOSITY, + maxOutputTokens: env.OPENAI_MAX_OUTPUT_TOKENS, + providerMode: env.RAG_PROVIDER_MODE, + promptVersion: ragAnswerPromptVersion, + schemaVersion: ragAnswerSchemaVersion, + classifierPromptVersion: ragQueryClassifierPromptVersion, + retrievalVersion: ragDeepMemoryVersion, + indexingPromptVersion: ragIndexingPromptVersion, + }); +} + +function sharedAnswerNormalizedQuery(args: Pick) { + const query = normalizedCacheQuery(`${modeKey(args)} ${args.query}`); + return queryCacheKeyForStorage(`${query}|generation:${answerGenerationFingerprint()}`); +} + export function scopedAnswerCacheKey( args: Pick, ) { @@ -59,6 +116,7 @@ export function scopedAnswerCacheKey( ragCacheDependencyVersion, scopeKey(args), modeKey(args), + `generation:${answerGenerationFingerprint()}`, args.query.trim().toLowerCase().replace(/\s+/g, " "), ].join("|"); } @@ -443,6 +501,7 @@ export async function getSharedCachedAnswer( "answer", args, indexingVersion, + sharedAnswerNormalizedQuery(args), ).maybeSingle(); if (error || !data?.payload) return null; const answer = cloneAnswer((data.payload as { answer: RagAnswer }).answer); @@ -527,7 +586,13 @@ function setSharedCachedAnswer( answer: RagAnswer, ) { if (!answerCacheAllowedForOwner(args.ownerId) || args.skipCache || env.RAG_ANSWER_CACHE_TTL_MS <= 0) return; - void replaceSharedCacheRow("answer", args, { answer: cloneAnswer(answer) }, env.RAG_ANSWER_CACHE_TTL_MS); + void replaceSharedCacheRow( + "answer", + args, + { answer: cloneAnswer(answer) }, + env.RAG_ANSWER_CACHE_TTL_MS, + sharedAnswerNormalizedQuery(args), + ); } export function invalidateRagCachesForOwner(ownerId?: string | null) { diff --git a/src/lib/rag-claim-support.ts b/src/lib/rag-claim-support.ts index b0607d5df..7ced082bd 100644 --- a/src/lib/rag-claim-support.ts +++ b/src/lib/rag-claim-support.ts @@ -1,4 +1,5 @@ import { extractClinicalValueAtoms, type ClinicalValueAtom } from "@/lib/answer-verification"; +import { SOURCE_BACKED_REVIEW_FALLBACK_REASON } from "@/lib/rag-routing"; import type { CitationProvenance, EvidenceAssessment, RagAnswer, SearchResult, SupportedClaim } from "@/lib/types"; const acceptedProvenance = new Set([ @@ -52,9 +53,14 @@ function cleanText(value: string) { } function splitClaims(value: string) { - return cleanText(value) - .split(/(?<=[.!?])\s+|\n+/) - .map((claim) => claim.trim()) + // Preserve model-authored line boundaries until after splitting. Calling + // cleanText first collapses newlines, which can merge independently cited + // bullet claims into one compound claim that no single chunk can support. + return value + .replace(/\r\n?/g, "\n") + .replace(/[*_`#>]/g, "") + .split(/(?<=[.!?])(?:[ \t]+|\n+)|\n+/) + .map(cleanText) .filter((claim) => claim.length >= 8) .slice(0, 24); } @@ -389,8 +395,11 @@ export function assessClaimSupport(answer: RagAnswer) { (answer.preformatted && (answer.answerSections?.length ?? 0) > 0 && (answer.answerSections ?? []).every((section) => section.kind === "documentation")); + const sourceBackedReviewAnswer = (answer.routingReason ?? "").includes(SOURCE_BACKED_REVIEW_FALLBACK_REASON); const inputs = claimInputs(answer); - const claims = inputs.map((input, index) => claimAssessment(input, index, sourceById, Boolean(documentLookupAnswer))); + const claims = inputs.map((input, index) => + claimAssessment(input, index, sourceById, Boolean(documentLookupAnswer || sourceBackedReviewAnswer)), + ); const evidenceAssessments = Object.fromEntries( answer.sources.map((source) => [source.id, evidenceAssessment(source, claims, inputs)]), ); diff --git a/src/lib/rag-eval-cases.ts b/src/lib/rag-eval-cases.ts index 705d4613a..771187dac 100644 --- a/src/lib/rag-eval-cases.ts +++ b/src/lib/rag-eval-cases.ts @@ -423,7 +423,7 @@ export const answerQualityEvalCases: AnswerQualityEvalCase[] = [ id: "quality-lithium-monitoring-range", question: "What lithium level range is used for maintenance monitoring?", expectedIntent: "monitoring_schedule", - expectedQueryClass: "medication_dose_risk", + expectedQueryClass: "table_threshold", expectedFiles: ["CG.MHSP.Lithium.pdf"], mustContainAny: ["lithium", "level", "mmol"], }, @@ -432,7 +432,9 @@ export const answerQualityEvalCases: AnswerQualityEvalCase[] = [ id: "quality-lithium-monitoring-documents", question: "What documents support lithium monitoring?", expectedIntent: "document_lookup", - expectedQueryClass: "document_lookup", + // Retrieval intentionally remains medication-focused for this wording; the + // answer intent separately switches to the deterministic document list. + expectedQueryClass: "medication_dose_risk", expectedFiles: ["CG.MHSP.Lithium.pdf"], mustContainAny: ["document", "lithium"], }, diff --git a/src/lib/rag-extractive-answer.ts b/src/lib/rag-extractive-answer.ts index 6bd41febf..cdfeaf7ae 100644 --- a/src/lib/rag-extractive-answer.ts +++ b/src/lib/rag-extractive-answer.ts @@ -335,7 +335,7 @@ function queryIntentTokens(query: string, intent: AnswerIntent) { function answerIntentEvidencePattern(intent: AnswerIntent) { switch (intent) { case "dose": - return /\b(?:doses?|dosing|dosage|max(?:imum)?|mg|mcg|microgram|micrograms|mmol\/l|eGFR|renal|creatinine|daily|bd|tds|mane|nocte)\b/i; + return doseIntentEvidencePattern; case "contraindication": return /\b(?:contraindicat\w*|avoid|must not|do not|should not|not use|opioid[-\s]?free|withdrawal|precipitat\w*)\b/i; case "monitoring_schedule": @@ -351,6 +351,22 @@ function answerIntentEvidencePattern(intent: AnswerIntent) { } } +const clinicalDoseUnitSource = String.raw`(?:mg|milligrams?|mcg|micrograms?|g|grams?|kg|kilograms?|ml|millilit(?:er|re)s?|l|lit(?:er|re)s?|international\s+units?|units?|iu|mmol(?:\/l)?|millimoles?(?:\s+per\s+lit(?:er|re))?|meq|milliequivalents?|tablets?|capsules?|puffs?|drops?|sprays?|patch(?:es)?)`; +const clinicalDoseValueSource = String.raw`\d+(?:\.\d+)?\s*${clinicalDoseUnitSource}`; +const clinicalDoseValuePattern = new RegExp(String.raw`\b${clinicalDoseValueSource}\b`, "i"); +const maximumDoseEquivalentPattern = new RegExp( + String.raw`\b(?:up\s+to|(?:do\s+)?not\s+exceed|not\s+to\s+exceed|(?:no|not)\s+more\s+than|at\s+most|limit(?:ed)?(?:\s+the\s+dose)?\s+to)\s+${clinicalDoseValueSource}\b`, + "i", +); +const explicitMaximumDosePattern = new RegExp( + String.raw`(?:\bmax(?:imum)?(?:\s+\w+){0,3}\s+doses?\b|\bdoses?(?:\s+\w+){0,3}\s+max(?:imum)?\b|\bmax(?:imum)?(?:\s+\w+){0,3}\s+${clinicalDoseValueSource}\b|\b${clinicalDoseValueSource}(?:\s+\w+){0,3}\s+max(?:imum)?\b)`, + "i", +); +const doseIntentEvidencePattern = new RegExp( + String.raw`\b(?:doses?|dosing|dosage|${clinicalDoseUnitSource}|eGFR|renal|creatinine|daily|bd|tds|mane|nocte)\b`, + "i", +); + /** Requires blood count evidence. */ function requiresBloodCountEvidence(query: string) { return /\b(?:anc|fbc|full blood count|blood count|wbc|wcc|white blood cells?|white cells?|neutrophils?)\b/i.test( @@ -377,6 +393,11 @@ function hasWithholdActionEvidence(text: string) { ); } +/** Has explicit or equivalent maximum-dose evidence. */ +export function hasMaximumDoseEvidence(text: string) { + return explicitMaximumDosePattern.test(text) || maximumDoseEquivalentPattern.test(text); +} + /** Result covers answer intent. */ function resultCoversAnswerIntent(result: SearchResult, query: string, intent: AnswerIntent) { if (intent === "unsupported") return false; @@ -389,13 +410,21 @@ function resultCoversAnswerIntent(result: SearchResult, query: string, intent: A (/\bect\b/i.test(query) && /\b(?:ect|electroconvulsive)\b/i.test(text)); if (!entityCoverage) return false; if (intent === "general") return true; - const intentCoverage = answerIntentEvidencePattern(intent).test(text); + const asksForMaximumDose = intent === "dose" && /\bmax(?:imum)?\b/i.test(query); + const maximumDoseCoverage = asksForMaximumDose && hasMaximumDoseEvidence(text); + const intentCoverage = answerIntentEvidencePattern(intent).test(text) || maximumDoseCoverage; if (!intentCoverage) return false; - if (intentTokens.length > 0 && !intentTokens.some((token) => queryTokenMatchesText(token, text))) return false; + if ( + intentTokens.length > 0 && + !intentTokens.some((token) => queryTokenMatchesText(token, text)) && + !maximumDoseCoverage + ) { + return false; + } if (intent === "red_result_action" && requiresBloodCountEvidence(query) && !hasBloodCountEvidence(text)) return false; if (intent === "red_result_action" && asksForWithholdAction(query) && !hasWithholdActionEvidence(text)) return false; if (/\brenal\b/i.test(query) && !/\b(?:renal|kidney|eGFR|creatinine)\b/i.test(text)) return false; - if (/\bmax(?:imum)?\b/i.test(query) && !/\b(?:max(?:imum)?|\d+(?:\.\d+)?\s?(?:mg|mcg))\b/i.test(text)) { + if (asksForMaximumDose && !maximumDoseCoverage) { return false; } return true; @@ -410,10 +439,12 @@ const extractiveHeadingOnlyPattern = /^(?:dosage?|dosing|monitoring|baseline tests?|therapy|source|section|table|guideline|referral criteria|criteria)(?:\s*\([^)]{1,80}\))?\.?$/i; const extractiveAllowedLowercaseStarterPattern = /^(?:if|when|for|in|avoid|do|must|withhold|cease|stop|monitor|check|reduce|increase|adjust|start|commence|begin|use|target|baseline|serum|therapy|dosing|titrate|arrange|refer|review|prescribe|record|complete|continue|discontinue|escalate)\b/i; -const extractiveConcreteDosePattern = - /\b(?:\d+(?:\.\d+)?\s?(?:mg|mcg|microgram|micrograms|mmol\/?l)|mmol\/l|daily|bd|tds|mane|nocte|target|range|serum|levels?|titration|titrate|titrated|adjust(?:ed|ment)?|dose\s+(?:adjust|reduc|increas)|reduce(?:d)?\s+doses?|doses?\s+(?:in|for|when|with|based|according)|max(?:imum)?|renal|eGFR|CrCl|creatinine|elderly|impairment|conventional tablets?)\b/i; +const extractiveConcreteDosePattern = new RegExp( + String.raw`\b(?:${clinicalDoseValueSource}|mmol\/l|daily|bd|tds|mane|nocte|target|range|serum|levels?|titration|titrate|titrated|adjust(?:ed|ment)?|dose\s+(?:adjust|reduc|increas)|reduce(?:d)?\s+doses?|doses?\s+(?:in|for|when|with|based|according)|max(?:imum)?|renal|eGFR|CrCl|creatinine|elderly|impairment|conventional tablets?)\b`, + "i", +); const extractiveMedicationEntityPattern = - /\b(?:acamprosate|aripiprazole|baclofen|citalopram|clozapine|diazepam|disulfiram|droperidol|escitalopram|fluoxetine|haloperidol|lithium|lorazepam|naltrexone|olanzapine|promethazine|quetiapine|risperidone|sertraline|valproate)\b/gi; + /\b(?:acamprosate|aripiprazole|baclofen|benzodiazepine|citalopram|clozapine|diazepam|disulfiram|droperidol|escitalopram|fluoxetine|haloperidol|lithium|lorazepam|naltrexone|olanzapine|promethazine|quetiapine|risperidone|sertraline|valproate)\b/gi; /** Extractive query tokens. */ function extractiveQueryTokens(query: string) { @@ -650,7 +681,8 @@ function factKindForSentence(sentence: string, query: string, intent: AnswerInte return "monitoring"; } if ( - /\b(?:doses?|dosing|dosage|max(?:imum)?|\d+(?:\.\d+)?\s?(?:mg|mcg)|daily|bd|tds|mane|nocte|mmol\/l)\b/i.test(text) + /\b(?:doses?|dosing|dosage|daily|bd|tds|mane|nocte|mmol\/l)\b/i.test(text) || + clinicalDoseValuePattern.test(text) ) { return "dose"; } @@ -687,7 +719,7 @@ function factSupportsAnswerIntent( } } if (/\brenal\b/i.test(query) && !/\b(?:renal|kidney|eGFR|creatinine|CrCl)\b/i.test(text)) return false; - if (/\bmax(?:imum)?\b/i.test(query) && !/\b(?:max(?:imum)?|\d+(?:\.\d+)?\s?(?:mg|mcg))\b/i.test(text)) { + if (/\bmax(?:imum)?\b/i.test(query) && !hasMaximumDoseEvidence(text)) { return false; } return extractiveConcreteDosePattern.test(text); @@ -759,6 +791,19 @@ function factSentenceMatchesQueryFromResult( const resultText = evidenceTextForGate(result); const entityTokens = queryEntityTokens(query, intent); + const queryMedicationEntities = medicationEntitiesInText(query); + const sentenceMedicationEntities = medicationEntitiesInText(sentence); + const resultMedicationEntities = medicationEntitiesInText(resultText); + if ( + intent === "dose" && + queryMedicationEntities.length > 0 && + sentenceMedicationEntities.length === 0 && + resultMedicationEntities.length > 1 + ) { + // A bare dose row from a multi-drug table cannot safely inherit the query's + // medication/class label. Require the row itself to name its medication. + return false; + } const entityCoveredByResult = entityTokens.length === 0 || entityTokens.some((token) => queryTokenMatchesText(token, resultText)); if (!entityCoveredByResult) return false; @@ -899,6 +944,7 @@ export function sentenceFromFact( /** Lower first. */ function lowerFirst(value: string) { if (!value) return value; + if (/^[A-Z][A-Z0-9&+-]{1,}\b/.test(value)) return value; return `${value.charAt(0).toLowerCase()}${value.slice(1)}`; } @@ -1344,8 +1390,13 @@ function sourceBackedFallbackSubject(query: string) { .replace(/^summari[sz]e\s+(?:the\s+)?/i, "") .replace(/^what\s+(?:is|are)\s+(?:the\s+)?(?:process|requirements?)\s+for\s+/i, "") .replace(/^what\s+(?:is|are)\s+required\s+(?:for|when)\s+/i, "") + .replace(/^what\s+(.+?)\s+should\s+((?:withhold|cease|stop)\s+.+)$/i, "$1 for the decision to $2") + .replace(/^what\s+(.+?)\s+(?:is|are)\s+(?:used|required|recommended|needed)\s+for\s+(.+)$/i, "$1 for $2") + .replace(/^what\s+(.+?)\s+(?:apply|applies)$/i, "$1") .replace(/^what\s+(.+?)\s+is\s+required$/i, "$1") .replace(/^what\s+does\s+(?:the\s+)?/i, "") + .replace(/^what\s+(?:is|are)\s+(?:the\s+)?/i, "") + .replace(/^what\s+/i, "") .replace(/\s+(?:document|procedure|guideline)\s+require$/i, "") .replace(/^how\s+(?:is|are)\s+/i, "") .replace(/\s+managed$/i, " management") @@ -1358,7 +1409,7 @@ function sourceBackedFallbackSubject(query: string) { /** Source backed generation timeout answer. */ export function sourceBackedGenerationTimeoutAnswer(query: string) { const subject = sourceBackedFallbackSubject(query); - return `The uploaded documents contain relevant guidance on ${subject}, but a full written answer could not be completed just now. The key source passages are cited below — please review them directly.`; + return `The uploaded documents contain relevant guidance on ${subject}, but a full written answer could not be completed just now. Relevant document passages are cited below — please review them directly.`; } const reasoningEffortRank: Record = { @@ -1648,6 +1699,15 @@ export function generatedAnswerQualityFailureReason(answer: RagAnswer, query: st return "missing_query_overlap"; } if (hasInvalidModelEvidenceIds(answer)) return "invalid_model_evidence_ids"; + const broadDocumentCoverageRequested = + queryClass === "document_lookup" && + /(?:\b(?:what|which)\b.{0,100}\b(?:include|included|require|required|requirements?)\b|\b(?:process|procedure)\b|\bhow\b.{0,80}\b(?:handled|managed|performed|completed)\b)/i.test( + query, + ); + const distinctAvailableSources = new Set((answer.sources ?? []).map((source) => source.id)).size; + if (broadDocumentCoverageRequested && distinctAvailableSources >= 2 && answer.citations.length < 2) { + return "insufficient_broad_citation_coverage"; + } if (isUnusableGeneratedAnswer(answer)) return "unusable_generated_answer"; if (isTemplateLikeGeneratedAnswer(answer)) return "template_like_answer"; if (isOverExpandedSimpleGeneratedAnswer(query, queryClass, answer)) return "overexpanded_simple_answer"; diff --git a/src/lib/rag-routing.ts b/src/lib/rag-routing.ts index 1bb8779d0..812d36d83 100644 --- a/src/lib/rag-routing.ts +++ b/src/lib/rag-routing.ts @@ -12,6 +12,8 @@ export type AnswerRoute = { documentCount: number; }; +export const SOURCE_BACKED_REVIEW_FALLBACK_REASON = "source_backed_review_fallback"; + const unsupportedSimilarityThreshold = 0.32; const strongRetrievalThreshold = 0.64; const extractiveRetrievalThreshold = 0.76; diff --git a/src/lib/rag-versioning.ts b/src/lib/rag-versioning.ts new file mode 100644 index 000000000..c0ec01e66 --- /dev/null +++ b/src/lib/rag-versioning.ts @@ -0,0 +1,5 @@ +export const ragAnswerPromptVersion = "clinical-rag-answer-v18"; +export const ragAnswerSchemaVersion = "clinical-rag-answer-schema-v3"; +export const ragQueryClassifierPromptVersion = "clinical-rag-query-classifier-v1"; +export const ragSummaryPromptVersion = "clinical-document-summary-v2"; +export const ragIndexingPromptVersion = "clinical-indexing-prompts-v1"; diff --git a/src/lib/rag.ts b/src/lib/rag.ts index 0e44cf08e..d4d6a2592 100644 --- a/src/lib/rag.ts +++ b/src/lib/rag.ts @@ -8,8 +8,13 @@ import { } from "@/lib/owner-scope"; import { classifyCorpusGrounding } from "@/lib/corpus-grounding"; import type { Database, Json } from "@/lib/supabase/database.types"; -import { createStreamingAnswerExtractor } from "@/lib/answer-stream-extractor"; -import { embedTextWithTelemetry, generateStructuredTextResult, type OpenAITextResult } from "@/lib/openai"; +import { + embedTextWithTelemetry, + generateParsedTextResult, + generateStructuredTextResult, + openAISafetyIdentifier, + type OpenAITextResult, +} from "@/lib/openai"; import { SOURCE_ONLY_EMBEDDING_SKIP_REASON, allowsAutoDegrade, @@ -125,6 +130,7 @@ import { zoneContextPatternsForQuery, } from "@/lib/clinical-search"; import { env, requestedOpenAIAnswerModels } from "@/lib/env"; +import { ragAnswerPromptVersion, ragQueryClassifierPromptVersion, ragSummaryPromptVersion } from "@/lib/rag-versioning"; import { logger } from "@/lib/logger"; import { captureServerEvent } from "@/lib/observability/error-capture"; import { @@ -137,6 +143,7 @@ import { normalizeSourceMetadata } from "@/lib/source-metadata"; import { isReviewedTablePromotable } from "@/lib/table-review"; import { isClinicalImageEvidence, normalizeImageBbox } from "@/lib/image-filtering"; import { + SOURCE_BACKED_REVIEW_FALLBACK_REASON, chooseAnswerRoute, hasAdversarialManipulationIntent, hasDirectTitleSupport, @@ -439,13 +446,6 @@ export type AnswerProgressEvent = { type AnswerQuestionWithScopeArgs = SearchChunksArgs & { logQuery?: boolean; onProgress?: (event: AnswerProgressEvent) => void | Promise; - // Streaming hooks. onToken receives the answer prose as it generates (content-preserving — the - // same text arrives incrementally). onRevising fires when a generated answer fails a quality - // gate and the pipeline re-generates with the strong model, so the UI can clear the provisional - // text and show a "revising for accuracy" state before the corrected answer streams in. Only the - // streaming route sets these; the non-streaming path leaves them undefined. - onToken?: (delta: string) => void; - onRevising?: (reason: string) => void; signal?: AbortSignal; }; @@ -1213,6 +1213,7 @@ function addOpenAIUsage(total: OpenAITokenUsage, usage?: OpenAITokenUsage) { output_tokens: (total.output_tokens ?? 0) + (usage.output_tokens ?? 0), total_tokens: (total.total_tokens ?? 0) + (usage.total_tokens ?? 0), cached_input_tokens: (total.cached_input_tokens ?? 0) + (usage.cached_input_tokens ?? 0), + cache_write_tokens: (total.cache_write_tokens ?? 0) + (usage.cache_write_tokens ?? 0), reasoning_output_tokens: (total.reasoning_output_tokens ?? 0) + (usage.reasoning_output_tokens ?? 0), }; } @@ -1222,53 +1223,26 @@ function hasOpenAIUsage(usage: OpenAITokenUsage) { return Object.values(usage).some((value) => typeof value === "number" && value > 0); } -const queryClassifierOutputSchema = { - type: "object", - description: "Low-cost query classification fallback for clinical RAG retrieval only.", - additionalProperties: false, - properties: { - queryClass: { - type: "string", - enum: [ - "document_lookup", - "table_threshold", - "medication_dose_risk", - "comparison", - "broad_summary", - "unsupported_or_general", - ], - }, - confidence: { - type: "number", - minimum: 0, - maximum: 1, - }, - reasons: { - type: "array", - maxItems: 4, - items: { type: "string", maxLength: 80 }, - }, - expandedTerms: { - type: "array", - maxItems: 10, - items: { type: "string", maxLength: 60 }, - }, - }, - required: ["queryClass", "confidence", "reasons", "expandedTerms"], -}; - -const queryClassifierParseSchema = z.object({ - queryClass: z.enum([ - "document_lookup", - "table_threshold", - "medication_dose_risk", - "comparison", - "broad_summary", - "unsupported_or_general", - ]), +const queryClassifierParseSchema = z + .object({ + queryClass: z.enum([ + "document_lookup", + "table_threshold", + "medication_dose_risk", + "comparison", + "broad_summary", + "unsupported_or_general", + ]), + confidence: z.number(), + reasons: z.array(z.string()), + expandedTerms: z.array(z.string()), + }) + .strict(); + +const queryClassifierVerdictSchema = queryClassifierParseSchema.extend({ confidence: z.number().min(0).max(1), - reasons: z.array(z.string()).max(4), - expandedTerms: z.array(z.string()).max(10), + reasons: z.array(z.string().max(80)).max(4), + expandedTerms: z.array(z.string().max(60)).max(10), }); /** Unique text values. */ @@ -1287,7 +1261,7 @@ function uniqueTextValues(values: Array, limit = 32) return output; } -type ClassifierVerdict = z.infer; +type ClassifierVerdict = z.infer; // Finding #11 interim fix (docs/process-hardening.md): the LLM classifier verdict flips // run-to-run for the same query, so the unsupported short-circuit downstream intermittently @@ -1307,7 +1281,13 @@ function classifierVerdictMemoKey(query: string, analysis: ClinicalQueryAnalysis const normalizedQuery = query.normalize("NFKC").toLowerCase().replace(/\s+/g, " ").trim(); // The deterministic class + confidence bucket are part of the key so a deterministic-analyzer // change invalidates stale verdicts instead of replaying them against a different baseline. - return `${normalizedQuery}::${analysis.queryClass}::${analysis.confidence.toFixed(2)}`; + return [ + env.OPENAI_QUERY_CLASSIFIER_MODEL, + ragQueryClassifierPromptVersion, + normalizedQuery, + analysis.queryClass, + analysis.confidence.toFixed(2), + ].join("::"); } /** Store classifier verdict memo. */ @@ -1326,8 +1306,12 @@ export function resetClassifierVerdictMemoForTests() { } /** Request classifier verdict. */ -async function requestClassifierVerdict(query: string, analysis: ClinicalQueryAnalysis): Promise { - const result = await generateStructuredTextResult( +async function requestClassifierVerdict( + query: string, + analysis: ClinicalQueryAnalysis, + ownerId?: string | null, +): Promise { + const result = await generateParsedTextResult( [ { role: "user", @@ -1344,9 +1328,9 @@ async function requestClassifierVerdict(query: string, analysis: ClinicalQueryAn ], }, ], - queryClassifierOutputSchema, + queryClassifierParseSchema, { - model: env.OPENAI_FAST_ANSWER_MODEL, + model: env.OPENAI_QUERY_CLASSIFIER_MODEL, maxOutputTokens: 220, operation: "text_generation", instructions: @@ -1354,11 +1338,12 @@ async function requestClassifierVerdict(query: string, analysis: ClinicalQueryAn reasoningEffort: "low", textVerbosity: "low", schemaName: "clinical_rag_query_classifier", - promptCacheKey: "clinical-rag-query-classifier-v1", + promptCacheKey: ragQueryClassifierPromptVersion, timeoutMs: 6000, + safetyIdentifier: env.OPENAI_SAFETY_IDENTIFIER_SECRET ? openAISafetyIdentifier(ownerId) : undefined, }, ); - return queryClassifierParseSchema.parse(JSON.parse(result.text)); + return queryClassifierVerdictSchema.parse(result.parsed); } /** Apply classifier verdict. */ @@ -1397,6 +1382,7 @@ export async function analyzeQueryWithClassifierFallback( // against the corpus BEFORE the nondeterministic LLM classifier. Scoped with the exact // owner_filter retrieval will use so grounding can never see documents retrieval cannot. corpusGrounding?: { supabase: ReturnType; ownerFilter: string | null }; + ownerId?: string | null; }, ) { if ( @@ -1461,7 +1447,7 @@ export async function analyzeQueryWithClassifierFallback( let pending = classifierVerdictInflight.get(memoKey); if (!pending) { - pending = requestClassifierVerdict(query, analysis).finally(() => { + pending = requestClassifierVerdict(query, analysis, opts?.ownerId).finally(() => { classifierVerdictInflight.delete(memoKey); }); classifierVerdictInflight.set(memoKey, pending); @@ -3361,6 +3347,7 @@ export async function searchChunksWithTelemetry(args: SearchChunksArgs) { })(); const queryAnalysis = await analyzeQueryWithClassifierFallback(retrievalQuery, analyzeClinicalQuery(retrievalQuery), { corpusGrounding: corpusGroundingScope, + ownerId: args.ownerId, }); throwIfAborted(args.signal); if (modeQueryClass) queryAnalysis.queryClass = modeQueryClass; @@ -4100,7 +4087,7 @@ export function isCacheableGroundedGenerationFallback( answer.citations.length > 0 && (answer.unverifiedNumericTokens?.length ?? 0) === 0 && /(?:source_backed_extractive_fallback|comparison_source_safe_fallback)/.test(answer.routingReason ?? "") && - !/source_backed_review_fallback/.test(answer.routingReason ?? "") + !(answer.routingReason ?? "").includes(SOURCE_BACKED_REVIEW_FALLBACK_REASON) ); } @@ -4618,40 +4605,43 @@ async function answerQuestionWithScopeUncoalesced( generation_latency_ms: 0, total_latency_ms: Date.now() - startedAt, }; - const answer: RagAnswer = annotateAnswerWithDiagnostics( + const sourceSafeExtractiveAnswer = buildExtractiveAnswer({ + query: args.query, + queryClass, + results: answerInputResults, + quoteCards, + documentBreakdown, + evidenceSummary, + sourceCoverage, + conflictsOrGaps, + visualEvidence, + bestSource, + smartPanel: { ...smartPanel, relevance, bestSource, relatedDocuments }, + relatedDocuments, + routeReason: + queryClass === "comparison" ? `${route.reason}; comparison_source_extractive_fallback` : route.reason, + timings: extractiveTimings, + }); + const sourceSafeComparisonAnswer = queryClass === "comparison" ? (buildComparisonAnswer({ query: args.query, results: answerInputResults, - selectedDocuments: explicitlySelectedComparisonDocuments, routeReason: route.reason, + selectedDocuments: explicitlySelectedComparisonDocuments, timings: extractiveTimings, }) ?? - buildComparisonEvidenceGapAnswer({ - query: args.query, - results: answerInputResults, - selectedDocuments: explicitlySelectedComparisonDocuments, - routeReason: `${route.reason}; comparison_evidence_gap`, - timings: extractiveTimings, - })) - : buildExtractiveAnswer({ - query: args.query, - queryClass, - results: answerInputResults, - quoteCards, - documentBreakdown, - evidenceSummary, - sourceCoverage, - conflictsOrGaps, - visualEvidence, - bestSource, - smartPanel: { ...smartPanel, relevance, bestSource, relatedDocuments }, - relatedDocuments, - routeReason: route.reason, - timings: extractiveTimings, - }), - retrievalDiagnostics, - ); + (sourceSafeExtractiveAnswer.grounded + ? sourceSafeExtractiveAnswer + : buildComparisonEvidenceGapAnswer({ + query: args.query, + results: answerInputResults, + selectedDocuments: explicitlySelectedComparisonDocuments, + routeReason: `${route.reason}; comparison_evidence_gap`, + timings: extractiveTimings, + }))) + : sourceSafeExtractiveAnswer; + const answer: RagAnswer = annotateAnswerWithDiagnostics(sourceSafeComparisonAnswer, retrievalDiagnostics); answer.quoteCards ??= quoteCards; answer.documentBreakdown ??= documentBreakdown; answer.evidenceSummary ??= evidenceSummary; @@ -4666,7 +4656,32 @@ async function answerQuestionWithScopeUncoalesced( answer.smartPanel = answer.smartPanel ? { ...answer.smartPanel, relevance } : answer.smartPanel; answer.smartApiPlan = smartApiPlan; answer.scoreExplanations = answerScoreExplanations; - const finalizedAnswer = finalizeRagAnswerQuality(answer, args.query, queryClass); + let finalizedAnswer = finalizeRagAnswerQuality(answer, args.query, queryClass); + const extractiveReviewCitations = answer.citations.length + ? answer.citations + : compactCitations(answerInputResults, 5, "deterministic_support"); + const extractiveNeedsReviewFallback = !finalizedAnswer.grounded && extractiveReviewCitations.length > 0; + if (extractiveNeedsReviewFallback) { + const reviewRouteReason = `${answer.routingReason ?? route.reason}; ${SOURCE_BACKED_REVIEW_FALLBACK_REASON}`; + const reviewPlan = buildCurrentSmartApiPlan("extractive", reviewRouteReason); + finalizedAnswer = finalizeRagAnswerQuality( + { + ...answer, + answer: boldHighYieldClinicalText(sourceBackedGenerationTimeoutAnswer(args.query), args.query), + grounded: true, + confidence: deriveConfidence(answerInputResults, extractiveReviewCitations), + citations: extractiveReviewCitations, + modelUsed: null, + routingMode: "extractive", + routingReason: reviewRouteReason, + responseMode: reviewPlan.displayMode, + smartApiPlan: reviewPlan, + answerSections: [], + }, + args.query, + queryClass, + ); + } if (args.logQuery !== false) await logRagQuery({ @@ -4752,7 +4767,8 @@ async function answerQuestionWithScopeUncoalesced( - Never state unsupported numbers, doses, frequencies, thresholds, routes, or medication names. If a number or dose is not clearly in the evidence, leave it out. - Copy every dose, level, threshold, cut-off, frequency, and duration EXACTLY as written in a cited excerpt — digit for digit, with its unit. Never supply a number from general clinical knowledge (including "typical" therapeutic levels or well-known reference ranges) that is not verbatim in the excerpts, and never round, infer, or complete a partial figure. - Do not merge separate values into a range. If the excerpts list discrete dose steps (for example 0.25 mg, 0.5 mg, 1 mg), present them as discrete steps — never as "0.25–1 mg" or any range the excerpt does not itself state. -- Use only citation_chunk_id values from the supplied source block — never invent, transform, abbreviate, or reuse IDs from outside the retrieved evidence. Cite only the strongest 3-5, not every source. +- Use only citation_chunk_id values from the supplied source block — never invent, transform, abbreviate, or reuse IDs from outside the retrieved evidence. Cite only the strongest 3-5 that collectively support every claim you retain; if five chunks cannot cover a lower-priority claim, omit that claim instead of leaving it uncited. +- For every number, dose, threshold, frequency, or timing, include the exact supporting chunk ID in the containing section's citation_chunk_ids (and in the top-level citations when the figure appears in the answer field). Do not combine independently sourced requirements into one sentence unless all supporting chunk IDs are attached to that sentence's citation scope. - If the excerpts contain only headings, partial table fragments, or disconnected text that cannot support a logical answer, say the uploaded documents do not contain enough information — do not fill from general knowledge. - Integrate relevant sources: merge overlapping guidance once; when several documents are relevant, synthesise by clinical theme/action and reconcile conflicts explicitly rather than silently choosing one; call out weak, nearby-only, or missing support when the evidence is partial or conflicting. Prefer Australian or WA-specific guidance when present. Sources are ordered by relevance — prioritise earlier ones unless a later source resolves a conflict or gap. The fused source brief and structured memory lines are orientation only; verify every claim against the raw excerpts below them and cite the original chunks. - Do not give patient-specific medical advice. @@ -4887,11 +4903,6 @@ Quality retry instruction: ${qualityRetryInstruction}` : buildAnswerInput(contextResults); const generationStartedAt = Date.now(); - // Fresh per-generation extractor: each generateWithModel call is a separate JSON stream, so the - // answer prose restarts from zero. Deltas are forwarded only for the answer field; the rest of - // the structured JSON (sections, citations) is delivered in the final payload as before. - const streamExtractor = args.onToken ? createStreamingAnswerExtractor() : null; - let streamRawBuffer = ""; try { const result = await generateStructuredTextResult(input, answerJsonOutputSchemaForResults(contextResults), { model, @@ -4899,21 +4910,14 @@ ${qualityRetryInstruction}` operation: "answer", schemaName: "clinical_rag_answer", instructions: answerInstructions, - promptCacheKey: "clinical-rag-answer-v18", + promptCacheKey: ragAnswerPromptVersion, timeoutMs: env.OPENAI_ANSWER_TIMEOUT_MS, maxRetries: 0, reasoningEffort: useStrongReasoning ? strongReasoningEffortForQueryClass(queryClass, env.OPENAI_STRONG_REASONING_EFFORT) : env.OPENAI_FAST_REASONING_EFFORT, signal: args.signal, - onOutputTextDelta: - streamExtractor && args.onToken - ? (delta) => { - streamRawBuffer += delta; - const prose = streamExtractor.push(streamRawBuffer); - if (prose) args.onToken?.(prose); - } - : undefined, + safetyIdentifier: env.OPENAI_SAFETY_IDENTIFIER_SECRET ? openAISafetyIdentifier(args.ownerId) : undefined, }); openAIUsage = addOpenAIUsage(openAIUsage, result.usage); if (result.requestId) openAIRequestIds.push(result.requestId); @@ -5126,9 +5130,6 @@ ${qualityRetryInstruction}` model: env.OPENAI_STRONG_ANSWER_MODEL, reason: routingReason, }); - // Any prose already streamed to the client is now provisional — tell the UI to clear it and - // show a "revising for accuracy" state before the corrected strong answer streams in. - args.onRevising?.(retryReason); // Widen the retry context from the trimmed fast set to the full result set, but keep the P9 // per-document crowding cap — the strong-initial route is capped, so the retry must be too. packedContextResults = await packContextForGeneration(strongRetryContextResults); @@ -5211,7 +5212,6 @@ ${qualityRetryInstruction}` model: env.OPENAI_STRONG_ANSWER_MODEL, reason: routingReason, }); - args.onRevising?.(retryReason); // Same as the truncation retry above: widen but keep the P9 per-document crowding cap. packedContextResults = await packContextForGeneration(strongRetryContextResults); // Strong spends more reasoning tokens than the fast attempt it is replacing, so it needs @@ -5239,6 +5239,12 @@ ${qualityRetryInstruction}` const strongQualityFailureReason = usedStrongModel ? generatedAnswerQualityFailureReason(answer, args.query, queryClass) : null; + if (route.mode === "strong" && queryClass === "comparison" && strongQualityFailureReason) { + // A second strong-model pass is expensive and pushes comparison requests beyond the + // latency target. The catch path can rebuild these answers deterministically from the + // same attributed sources, so prefer that bounded recovery over another generation. + throw new Error(`OpenAI generation quality gate failed: ${strongQualityFailureReason}`); + } const answerNeedsStrongQualityRepair = usedStrongModel && Boolean(strongQualityFailureReason); if (answerNeedsStrongQualityRepair && generationLatencyMs >= generationTotalBudgetMs) { // A4 tail-latency guard: out of the cumulative generation time budget, so keep the @@ -5256,7 +5262,6 @@ ${qualityRetryInstruction}` model: env.OPENAI_STRONG_ANSWER_MODEL, reason: routingReason, }); - args.onRevising?.("strong_quality_retry"); generated = await generateWithModel(env.OPENAI_STRONG_ANSWER_MODEL, packedContextResults, { strong: true, maxOutputTokensOverride: strongRetryMaxOutputTokens, @@ -5374,6 +5379,22 @@ ${qualityRetryInstruction}` }); answer = finalizeRagAnswerQuality(answer, args.query, queryClass, numericVerificationSources); + // A provider response can be schema-valid yet still fail the deterministic claim/numeric + // provenance gates after its citations are scoped to individual claims. Reuse the existing + // source-safe comparison/extractive recovery path instead of returning an empty unsupported + // model answer. The fallback is finalized through the same gates in the catch block, so weak + // or unsafe source evidence still fails closed. + const sourceSafeFallbackReason = answer.routingReason?.includes("claim_support_high_risk_gap") + ? "claim_support_high_risk_gap" + : answer.routingReason?.includes("material_source_governance_gap") + ? "material_source_governance_gap" + : answer.unverifiedNumericTokens?.length + ? "numeric_faithfulness_gap" + : null; + if (sourceSafeFallbackReason) { + throw new Error(`OpenAI generation quality gate failed: ${sourceSafeFallbackReason}`); + } + if (args.logQuery !== false) await logRagQuery({ owner_id: args.ownerId ?? null, @@ -5472,23 +5493,53 @@ ${qualityRetryInstruction}` queryClass, routeMode: route.mode ?? "unknown", }); - const comparisonFallbackAnswer = + const comparisonMatrixFallbackAnswer = queryClass === "comparison" - ? (buildComparisonAnswer({ + ? buildComparisonAnswer({ query: args.query, results: generationFallbackResults, selectedDocuments: explicitlySelectedComparisonDocuments, routeReason: `${route.reason}; generation_fallback:${sanitizedReason}; comparison_source_safe_fallback`, timings: baseFallbackAnswer.latencyTimings, - }) ?? - buildComparisonEvidenceGapAnswer({ + }) + : null; + const comparisonExtractiveFallbackAnswer = + queryClass === "comparison" && !comparisonMatrixFallbackAnswer + ? buildExtractiveAnswer({ query: args.query, + queryClass, results: generationFallbackResults, - selectedDocuments: explicitlySelectedComparisonDocuments, - routeReason: `${route.reason}; generation_fallback:${sanitizedReason}; comparison_evidence_gap`, + quoteCards: generationFallbackArtifacts.quoteCards, + documentBreakdown: generationFallbackArtifacts.documentBreakdown, + evidenceSummary: generationFallbackArtifacts.evidenceSummary, + sourceCoverage: generationFallbackArtifacts.sourceCoverage, + conflictsOrGaps: generationFallbackArtifacts.conflictsOrGaps, + visualEvidence: generationFallbackArtifacts.visualEvidence, + bestSource: generationFallbackArtifacts.bestSource, + smartPanel: { + ...generationFallbackArtifacts.smartPanel, + relevance: generationFallbackArtifacts.relevance, + bestSource: generationFallbackArtifacts.bestSource, + relatedDocuments, + }, + relatedDocuments, + routeReason: `${route.reason}; generation_fallback:${sanitizedReason}; comparison_source_extractive_fallback`, timings: baseFallbackAnswer.latencyTimings, - })) + }) : null; + const comparisonFallbackAnswer = + comparisonMatrixFallbackAnswer ?? + (comparisonExtractiveFallbackAnswer?.grounded + ? comparisonExtractiveFallbackAnswer + : queryClass === "comparison" + ? buildComparisonEvidenceGapAnswer({ + query: args.query, + results: generationFallbackResults, + selectedDocuments: explicitlySelectedComparisonDocuments, + routeReason: `${route.reason}; generation_fallback:${sanitizedReason}; comparison_evidence_gap`, + timings: baseFallbackAnswer.latencyTimings, + }) + : null); const canRecoverGenerationErrorExtractively = queryClass !== "comparison" && generationFallbackResults.length > 0 && baseFallbackAnswer.citations.length > 0; const extractiveFallbackRouteReason = `${route.reason}; generation_fallback:${sanitizedReason}; source_backed_extractive_fallback`; @@ -5600,7 +5651,7 @@ ${qualityRetryInstruction}` const reviewRouteReason = [ route.reason, `generation_fallback:${sanitizedReason}`, - "source_backed_review_fallback", + SOURCE_BACKED_REVIEW_FALLBACK_REASON, `extractive_quality_gate:${sourceBackedReviewReason}`, ].join("; "); const reviewPlan = buildCurrentSmartApiPlan("extractive", reviewRouteReason); @@ -5620,11 +5671,46 @@ ${qualityRetryInstruction}` } satisfies RagAnswer; })() : (extractiveFallbackAnswer ?? baseFallbackAnswer); - const fallbackAnswer = finalizeRagAnswerQuality( + let fallbackAnswer = finalizeRagAnswerQuality( annotateAnswerWithDiagnostics(generationFallbackAnswer, retrievalDiagnostics), args.query, queryClass, ); + const finalizedFallbackNeedsReview = + fallbackAnswer.responseMode === "evidence_gap" && + /(?:claim_support_high_risk_gap|material_source_governance_gap)/.test(fallbackAnswer.routingReason ?? "") && + baseFallbackAnswer.citations.length > 0; + if (finalizedFallbackNeedsReview) { + const reviewRouteReason = [ + route.reason, + `generation_fallback:${sanitizedReason}`, + SOURCE_BACKED_REVIEW_FALLBACK_REASON, + "post_generation_claim_quality_gate", + ].join("; "); + const reviewPlan = buildCurrentSmartApiPlan("extractive", reviewRouteReason); + fallbackAnswer = finalizeRagAnswerQuality( + annotateAnswerWithDiagnostics( + { + ...baseFallbackAnswer, + answer: boldHighYieldClinicalText(sourceBackedGenerationTimeoutAnswer(args.query), args.query), + grounded: true, + confidence: deriveConfidence(generationFallbackResults, baseFallbackAnswer.citations), + modelUsed: null, + routingMode: "extractive", + routingReason: reviewRouteReason, + responseMode: reviewPlan.displayMode, + smartApiPlan: reviewPlan, + answerSections: [], + queryAnalysis, + relevance: generationFallbackArtifacts.relevance, + scoreExplanations: generationFallbackArtifacts.scoreExplanations, + }, + retrievalDiagnostics, + ), + args.query, + queryClass, + ); + } await args.onProgress?.({ stage: "verifying", message: "Checking citations and source metadata." }); if (args.logQuery !== false) await logRagQuery({ @@ -5763,13 +5849,14 @@ Sources: ${buildRagSourceBlock(results)}`; const generated = await generateStructuredTextResult(summaryInput, answerJsonOutputSchemaForResults(results), { - model: env.OPENAI_ANSWER_MODEL, + model: env.OPENAI_SUMMARY_MODEL, maxOutputTokens: env.OPENAI_MAX_OUTPUT_TOKENS, operation: "summary", schemaName: "clinical_document_summary", instructions: summaryInstructions, - promptCacheKey: "clinical-document-summary-v2", + promptCacheKey: ragSummaryPromptVersion, reasoningEffort: env.OPENAI_SUMMARY_REASONING_EFFORT, + safetyIdentifier: env.OPENAI_SAFETY_IDENTIFIER_SECRET ? openAISafetyIdentifier(ownerId) : undefined, signal: options?.signal, }); const answer = parseAnswerJson(generated.text, results, "summary"); @@ -5782,7 +5869,7 @@ ${buildRagSourceBlock(results)}`; answer.visualEvidence = buildVisualEvidence(results); answer.bestSource = selectBestSourceRecommendation(results, answer.quoteCards); answer.smartPanel = { ...buildSmartPanel("summary", results), bestSource: answer.bestSource }; - answer.modelUsed = env.OPENAI_ANSWER_MODEL; + answer.modelUsed = env.OPENAI_SUMMARY_MODEL; answer.openAIRequestIds = generated.requestId ? [generated.requestId] : []; answer.openAIUsage = generated.usage; answer.latencyTimings = { diff --git a/src/lib/sse-heartbeat.ts b/src/lib/sse-heartbeat.ts index 03fd916dc..2a0528031 100644 --- a/src/lib/sse-heartbeat.ts +++ b/src/lib/sse-heartbeat.ts @@ -1,7 +1,6 @@ // SSE liveness for long-running answer streams. Generation legitimately goes -// silent for stretches (e.g. strong-route reasoning before the first output -// token, bounded by OPENAI_ANSWER_TIMEOUT_MS), during which neither progress -// events nor token deltas are emitted. A periodic comment line keeps +// silent for stretches (e.g. strong-route reasoning and deterministic quality +// gates, bounded by OPENAI_ANSWER_TIMEOUT_MS). A periodic comment line keeps // intermediaries from idling out the connection and gives the client's stall // watchdog a byte-level liveness signal. Comment lines (leading ":") are // ignored by SSE parsers, so clients need no changes to tolerate them. diff --git a/src/lib/types.ts b/src/lib/types.ts index af1997813..6eaa09a5b 100644 --- a/src/lib/types.ts +++ b/src/lib/types.ts @@ -806,6 +806,7 @@ export type OpenAITokenUsage = { output_tokens?: number; total_tokens?: number; cached_input_tokens?: number; + cache_write_tokens?: number; reasoning_output_tokens?: number; }; diff --git a/src/lib/validation/answer-request.ts b/src/lib/validation/answer-request.ts new file mode 100644 index 000000000..c3805af7e --- /dev/null +++ b/src/lib/validation/answer-request.ts @@ -0,0 +1,24 @@ +import { z } from "zod"; +import { clinicalQueryModeSchema } from "@/lib/clinical-query-mode"; +import { searchScopeFiltersSchema } from "@/lib/search-scope"; + +export const answerRequestSchema = z + .object({ + query: z.string().trim().min(1).max(2000), + documentId: z.string().uuid().optional(), + documentIds: z.array(z.string().uuid()).max(25).optional(), + filters: searchScopeFiltersSchema.optional(), + queryMode: clinicalQueryModeSchema.optional().default("auto"), + summaryMode: z.boolean().optional().default(false), + }) + .superRefine((value, context) => { + if (value.summaryMode && !value.documentId) { + context.addIssue({ + code: "custom", + path: ["documentId"], + message: "Document summary mode requires a document id.", + }); + } + }); + +export type AnswerRequestBody = z.infer; diff --git a/tests/answer-progress-ui-smoke.spec.ts b/tests/answer-progress-ui-smoke.spec.ts index a2ce149b1..f113141e1 100644 --- a/tests/answer-progress-ui-smoke.spec.ts +++ b/tests/answer-progress-ui-smoke.spec.ts @@ -207,11 +207,12 @@ test("answer progress remains user-safe through fallback and keeps a compact com timeout: 3_000, }); - await expect(page.getByTestId("answer-streaming")).toContainText("Provisional lithium draft", { - timeout: 4_000, - }); await expect(progress).toContainText("Building a source-backed answer", { timeout: 5_000 }); - await expect(page.getByTestId("answer-streaming-revising")).toBeVisible(); + // Rolling deployments may still route a new client to an older server that + // emits provisional token/revising frames. The client must ignore both so + // unvalidated clinical prose never reaches the page before the final event. + await expect(page.getByTestId("answer-streaming")).toHaveCount(0); + await expect(page.getByTestId("answer-streaming-revising")).toHaveCount(0); await expect(page.getByText("Provisional lithium draft")).toHaveCount(0); await expect(progress).toHaveAttribute("data-progress-state", "complete", { timeout: 6_000 }); diff --git a/tests/answer-responsiveness-gate.test.ts b/tests/answer-responsiveness-gate.test.ts index ef9e05923..af2cbbcf3 100644 --- a/tests/answer-responsiveness-gate.test.ts +++ b/tests/answer-responsiveness-gate.test.ts @@ -70,6 +70,38 @@ describe("responsiveness gate — model-answer core-term overlap (P3)", () => { }); }); +describe("broad document-answer citation coverage", () => { + it("rejects a one-citation synthesis when a broad requirements question has multiple source chunks", () => { + const reason = generatedAnswerQualityFailureReason( + modelAnswer({ + answer: + "A patient safety plan should identify warning signs, agreed crisis actions, and the people responsible for those actions.", + citations: [{ chunk_id: "safety-plan-actions" }] as RagAnswer["citations"], + sources: [{ id: "safety-plan-actions" }, { id: "safety-plan-review" }] as RagAnswer["sources"], + }), + "What should a patient safety plan include?", + "document_lookup" satisfies RagQueryClass, + ); + + expect(reason).toBe("insufficient_broad_citation_coverage"); + }); + + it("allows a single directly supporting chunk when no second source chunk is available", () => { + const reason = generatedAnswerQualityFailureReason( + modelAnswer({ + answer: + "A patient safety plan should identify warning signs, agreed crisis actions, and the people responsible for those actions.", + citations: [{ chunk_id: "safety-plan-actions" }] as RagAnswer["citations"], + sources: [{ id: "safety-plan-actions" }] as RagAnswer["sources"], + }), + "What should a patient safety plan include?", + "document_lookup" satisfies RagQueryClass, + ); + + expect(reason).toBeNull(); + }); +}); + describe("isBareDefinitionQuestion (P3 guard)", () => { it("recognises definitional phrasings", () => { expect(isBareDefinitionQuestion("What is akathisia?")).toBe(true); @@ -90,6 +122,7 @@ describe("generation-timeout fallback wording (P2)", () => { const text = sourceBackedGenerationTimeoutAnswer("What is the clozapine ANC threshold?"); expect(text).toContain("cited below"); expect(text).toMatch(/review them directly/i); + expect(text).toContain("document passages"); // The wording the task explicitly wants eliminated. expect(text).not.toMatch(/source status/i); expect(text).not.toMatch(/source-backed/i); diff --git a/tests/answer-stream-contract.test.ts b/tests/answer-stream-contract.test.ts new file mode 100644 index 000000000..52534fd3b --- /dev/null +++ b/tests/answer-stream-contract.test.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from "vitest"; +import { isAnswerStreamEventName } from "../src/lib/answer-stream-contract"; + +describe("answer stream client safety contract", () => { + it("accepts only final-only clinical stream events", () => { + expect(isAnswerStreamEventName("progress")).toBe(true); + expect(isAnswerStreamEventName("final")).toBe(true); + expect(isAnswerStreamEventName("error")).toBe(true); + expect(isAnswerStreamEventName("token")).toBe(false); + expect(isAnswerStreamEventName("revising")).toBe(false); + }); +}); diff --git a/tests/answer-stream-extractor.test.ts b/tests/answer-stream-extractor.test.ts deleted file mode 100644 index 087fe5e08..000000000 --- a/tests/answer-stream-extractor.test.ts +++ /dev/null @@ -1,73 +0,0 @@ -import { describe, expect, it } from "vitest"; - -import { createStreamingAnswerExtractor } from "@/lib/answer-stream-extractor"; - -// Feed a full JSON payload one character at a time, collecting the emitted prose deltas. This -// mirrors the worst-case streaming granularity (single-char chunks) the real extractor must survive. -function streamCharByChar(json: string): string { - const extractor = createStreamingAnswerExtractor(); - let buffer = ""; - let out = ""; - for (const ch of json) { - buffer += ch; - out += extractor.push(buffer); - } - return out; -} - -describe("createStreamingAnswerExtractor", () => { - it("emits the answer prose incrementally and losslessly, char by char", () => { - const answer = "Give 500 mg orally now, then reassess in 30 minutes."; - const payload = JSON.stringify({ answer, confidence: "high", answerSections: [] }); - expect(streamCharByChar(payload)).toBe(answer); - }); - - it("returns nothing until the answer field opens", () => { - const extractor = createStreamingAnswerExtractor(); - expect(extractor.push('{"confidence":"hi')).toBe(""); - expect(extractor.push('{"confidence":"high","answer":"Hel')).toBe("Hel"); - }); - - it("never re-emits already-emitted prose across chunk boundaries", () => { - const extractor = createStreamingAnswerExtractor(); - expect(extractor.push('{"answer":"Hello')).toBe("Hello"); - expect(extractor.push('{"answer":"Hello wor')).toBe(" wor"); - expect(extractor.push('{"answer":"Hello world"')).toBe("ld"); - expect(extractor.emittedLength).toBe("Hello world".length); - }); - - it("decodes JSON escapes (quotes, newlines) without leaking raw escape sequences", () => { - const answer = 'Use the "PRN" order.\nThen document the dose.'; - const payload = JSON.stringify({ answer, confidence: "high" }); - expect(streamCharByChar(payload)).toBe(answer); - }); - - it("does not treat an escaped quote as the end of the value", () => { - const extractor = createStreamingAnswerExtractor(); - // The \" is an escaped quote inside the value, not the closing quote. - expect(extractor.push('{"answer":"say \\"hi\\" now')).toBe('say "hi" now'); - }); - - it("holds back an incomplete trailing escape until the next chunk completes it", () => { - const extractor = createStreamingAnswerExtractor(); - // Buffer cut right after the backslash — must not emit a dangling escape. - expect(extractor.push('{"answer":"line one\\')).toBe("line one"); - expect(extractor.push('{"answer":"line one\\n')).toBe("\n"); - }); - - it("handles a unicode escape split across chunk boundaries", () => { - const answer = "café"; // é as é in JSON.stringify? No — stringify keeps é literal. - const payload = JSON.stringify({ answer }); - expect(streamCharByChar(payload)).toBe(answer); - // Explicit split unicode escape: - const extractor = createStreamingAnswerExtractor(); - expect(extractor.push('{"answer":"A\\u00')).toBe("A"); // incomplete escape held back - expect(extractor.push('{"answer":"A\\u00e9')).toBe("é"); - }); - - it("emits nothing extra once the value is closed", () => { - const extractor = createStreamingAnswerExtractor(); - extractor.push('{"answer":"done"'); - expect(extractor.push('{"answer":"done","confidence":"high"}')).toBe(""); - }); -}); diff --git a/tests/client-secret-surface.test.ts b/tests/client-secret-surface.test.ts index 6f0ee9f1d..fff4b038e 100644 --- a/tests/client-secret-surface.test.ts +++ b/tests/client-secret-surface.test.ts @@ -133,7 +133,9 @@ describe("client environment isolation", () => { const text = readFileSync(PUBLIC_ENV, "utf8"); const referencedKeys = [...text.matchAll(/process\.env\.([A-Z0-9_]+)/g)].map((match) => match[1]!); expect(new Set(referencedKeys)).toEqual(PUBLIC_ENV_KEYS); - expect(text).not.toMatch(/SUPABASE_SERVICE_ROLE_KEY|OPENAI_API_KEY|RAG_QUERY_HASH_SECRET/); + expect(text).not.toMatch( + /SUPABASE_SERVICE_ROLE_KEY|OPENAI_API_KEY|OPENAI_SAFETY_IDENTIFIER_SECRET|RAG_QUERY_HASH_SECRET/, + ); }); it("scans public assets and generated client chunks without printing surrounding content", () => { diff --git a/tests/corpus-grounding.test.ts b/tests/corpus-grounding.test.ts index a54b1cc90..b045696e3 100644 --- a/tests/corpus-grounding.test.ts +++ b/tests/corpus-grounding.test.ts @@ -210,7 +210,7 @@ describe("analyzeQueryWithClassifierFallback corpus grounding", () => { }); vi.doMock("@/lib/openai", async (importOriginal) => { const actual = await importOriginal(); - return { ...actual, generateStructuredTextResult: classifierMock }; + return { ...actual, generateParsedTextResult: classifierMock }; }); const rag = await import("../src/lib/rag"); const corpusGrounding = await import("../src/lib/corpus-grounding"); @@ -281,12 +281,12 @@ describe("analyzeQueryWithClassifierFallback corpus grounding", () => { it("falls through to the LLM classifier when grounding is inconclusive", async () => { const classifierMock = vi.fn(async () => ({ - text: JSON.stringify({ + parsed: { queryClass: "broad_summary", confidence: 0.9, reasons: ["classifier_test"], expandedTerms: [], - }), + }, })); const { rag, analyzeClinicalQuery, opts } = await loadRag({ classifierMock, @@ -316,12 +316,12 @@ describe("analyzeQueryWithClassifierFallback corpus grounding", () => { it("keeps legacy behaviour when no corpus grounding scope is provided", async () => { const classifierMock = vi.fn(async () => ({ - text: JSON.stringify({ + parsed: { queryClass: "broad_summary", confidence: 0.9, reasons: ["classifier_test"], expandedTerms: [], - }), + }, })); const { rag, analyzeClinicalQuery } = await loadRag({ classifierMock, rows: [] }); const analysis = analyzeClinicalQuery("bipolar disorder"); diff --git a/tests/document-enrichment.test.ts b/tests/document-enrichment.test.ts index 216047c2d..19e676493 100644 --- a/tests/document-enrichment.test.ts +++ b/tests/document-enrichment.test.ts @@ -6,7 +6,7 @@ const mocks = vi.hoisted(() => ({ vi.mock("@/lib/env", () => ({ env: { - OPENAI_FAST_ANSWER_MODEL: "gpt-fast-test", + OPENAI_INDEXING_MODEL: "gpt-indexing-test", }, })); diff --git a/tests/document-mutation-routes.test.ts b/tests/document-mutation-routes.test.ts index 039154c5f..8cf92b404 100644 --- a/tests/document-mutation-routes.test.ts +++ b/tests/document-mutation-routes.test.ts @@ -200,6 +200,23 @@ describe("/api/documents/bulk", () => { }); describe("/api/documents/[id]/table-facts", () => { + it("rejects an invalid document UUID before querying UUID columns", async () => { + const supabase = createSupabaseMock(() => ({ data: null, error: null })); + mockRouteRuntime(supabase.client); + const { GET } = await import("../src/app/api/documents/[id]/table-facts/route"); + + const response = await GET(new Request("http://localhost/api/documents/not-a-uuid/table-facts"), { + params: Promise.resolve({ id: "not-a-uuid" }), + }); + + expect(response.status).toBe(400); + await expect(response.json()).resolves.toMatchObject({ + error: "Invalid document id.", + code: "invalid_route_params", + }); + expect(supabase.calls).toHaveLength(0); + }); + it("updates a committed table fact and its linked source image", async () => { const supabase = createSupabaseMock((call) => { if (call.table === "documents") { diff --git a/tests/extractive-answer-formatting.test.ts b/tests/extractive-answer-formatting.test.ts index 69d792983..c944a2077 100644 --- a/tests/extractive-answer-formatting.test.ts +++ b/tests/extractive-answer-formatting.test.ts @@ -1,11 +1,30 @@ import { describe, expect, it } from "vitest"; import { buildExtractiveAnswer, + hasMaximumDoseEvidence, sentenceFromFact, splitClinicalEvidenceSentences, } from "../src/lib/rag-extractive-answer"; import type { RagAnswer, SearchResult } from "../src/lib/types"; +describe("maximum-dose evidence", () => { + it("accepts equivalent numeric limit wording without accepting an unrelated dose", () => { + expect(hasMaximumDoseEvidence("Olanzapine may be increased up to 20 mg daily.")).toBe(true); + expect(hasMaximumDoseEvidence("The total daily dose must not exceed 20 mg.")).toBe(true); + expect(hasMaximumDoseEvidence("The dose is not to exceed 200 mg/day.")).toBe(true); + expect(hasMaximumDoseEvidence("Use not more than 200 mg daily.")).toBe(true); + expect(hasMaximumDoseEvidence("Use no more than 2 tablets daily.")).toBe(true); + expect(hasMaximumDoseEvidence("The total daily dose must not exceed 20 milligrams.")).toBe(true); + expect(hasMaximumDoseEvidence("Use at most 500 micrograms daily.")).toBe(true); + expect(hasMaximumDoseEvidence("Limit the dose to 5 millilitres daily.")).toBe(true); + expect(hasMaximumDoseEvidence("Use no more than 10 international units daily.")).toBe(true); + expect(hasMaximumDoseEvidence("The maximum recommended daily dose is 20 mg.")).toBe(true); + expect(hasMaximumDoseEvidence("The maximum is 20 mg daily.")).toBe(true); + expect(hasMaximumDoseEvidence("The starting dose is 5 mg daily.")).toBe(false); + expect(hasMaximumDoseEvidence("Lithium has a maximum treatment duration of two years.")).toBe(false); + }); +}); + // Regression for the live source-only answer that rendered as: // "For lithium, twice daily dosing should be spaced by 12 hours. The guidance // is that for lithium, acute Mania: o IR product: 750 to 1000mg daily in 2 or @@ -229,6 +248,42 @@ describe("extractive sentence stitching", () => { }); describe("extractive answer end to end", () => { + it("uses equivalent maximum-dose wording even when the source omits maximum and dose tokens", () => { + const result = { + id: "olanzapine-chunk-1", + document_id: "olanzapine-doc", + title: "Olanzapine Prescribing Guideline", + file_name: "Olanzapine Prescribing Guideline.pdf", + page_number: 4, + chunk_index: 3, + section_heading: "Maintenance", + content: "Olanzapine may be increased up to 20 milligrams daily.", + image_ids: [], + similarity: 0.91, + hybrid_score: 0.95, + images: [], + } as unknown as SearchResult; + + const answer = buildExtractiveAnswer({ + query: "What is the maximum olanzapine dose?", + queryClass: "medication_dose_risk", + results: [result], + quoteCards: [], + documentBreakdown: [] as RagAnswer["documentBreakdown"], + evidenceSummary: undefined as unknown as RagAnswer["evidenceSummary"], + sourceCoverage: undefined as unknown as RagAnswer["sourceCoverage"], + conflictsOrGaps: [], + visualEvidence: [] as unknown as RagAnswer["visualEvidence"], + bestSource: undefined as unknown as RagAnswer["bestSource"], + smartPanel: undefined as unknown as RagAnswer["smartPanel"], + relatedDocuments: [] as unknown as RagAnswer["relatedDocuments"], + routeReason: "demo", + timings: undefined as unknown as RagAnswer["latencyTimings"], + }); + + expect((answer.answer ?? "").replace(/\*\*/g, "")).toMatch(/up to 20 milligrams daily/i); + }); + it("renders the lithium source-only case cleanly through buildExtractiveAnswer", () => { const result = { id: "lithium-chunk-1", diff --git a/tests/model-index-extraction.test.ts b/tests/model-index-extraction.test.ts index 566c60cea..a4090e1fb 100644 --- a/tests/model-index-extraction.test.ts +++ b/tests/model-index-extraction.test.ts @@ -6,8 +6,7 @@ const mocks = vi.hoisted(() => ({ vi.mock("@/lib/env", () => ({ env: { - OPENAI_ANSWER_MODEL: "gpt-test", - OPENAI_STRONG_ANSWER_MODEL: "gpt-strong-test", + OPENAI_INDEXING_MODEL: "gpt-indexing-test", }, })); diff --git a/tests/openai-cache.test.ts b/tests/openai-cache.test.ts index 3e4d09dc0..644e391f4 100644 --- a/tests/openai-cache.test.ts +++ b/tests/openai-cache.test.ts @@ -1,4 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { z } from "zod"; afterEach(() => { vi.unstubAllEnvs(); @@ -110,7 +111,7 @@ describe("OpenAI query embedding cache", () => { input_tokens: 100, output_tokens: 20, total_tokens: 120, - input_tokens_details: { cached_tokens: 64 }, + input_tokens_details: { cached_tokens: 64, cache_write_tokens: 36 }, output_tokens_details: { reasoning_tokens: 5 }, }, }, @@ -142,6 +143,7 @@ describe("OpenAI query embedding cache", () => { output_tokens: 20, total_tokens: 120, cached_input_tokens: 64, + cache_write_tokens: 36, reasoning_output_tokens: 5, }); expect(capturedOptions).toMatchObject({ timeout: 1234, maxRetries: 1, signal: controller.signal }); @@ -168,6 +170,197 @@ describe("OpenAI query embedding cache", () => { }); }); + it("uses GPT-5.6-and-later prompt cache options instead of the deprecated retention field", async () => { + const capturedBodies: Record[] = []; + + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.stubEnv("OPENAI_PROMPT_CACHE_RETENTION", "24h"); + vi.stubEnv("OPENAI_PROMPT_CACHE_TTL", "30m"); + + vi.doMock("openai", () => ({ + default: class MockOpenAI { + embeddings = { create: vi.fn() }; + responses = { + create: vi.fn((body: Record) => { + capturedBodies.push(body); + return { + withResponse: async () => ({ + data: { status: "completed", output_text: '{"answer":"ok"}' }, + request_id: "req_gpt56_cache", + }), + }; + }), + }; + }, + })); + + const { generateStructuredTextResult } = await import("../src/lib/openai"); + await generateStructuredTextResult( + "Question", + { type: "object", properties: {}, required: [] }, + { model: "gpt-5.6-terra", operation: "answer", schemaName: "clinical_test" }, + ); + await generateStructuredTextResult( + "Question", + { type: "object", properties: {}, required: [] }, + { model: "gpt-5.7-terra", operation: "answer", schemaName: "clinical_test" }, + ); + + expect(capturedBodies).toHaveLength(2); + for (const [index, model] of ["gpt-5.6-terra", "gpt-5.7-terra"].entries()) { + expect(capturedBodies[index]).toMatchObject({ + model, + prompt_cache_options: { ttl: "30m" }, + }); + expect(capturedBodies[index]).not.toHaveProperty("prompt_cache_retention"); + } + }); + + it("uses Responses parse for static Zod schemas and forwards a pseudonymous safety identifier", async () => { + let capturedBody: Record = {}; + + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.doMock("openai", () => ({ + default: class MockOpenAI { + embeddings = { create: vi.fn() }; + responses = { + create: vi.fn(), + parse: vi.fn((body: Record) => { + capturedBody = body; + return { + withResponse: async () => ({ + data: { + status: "completed", + output_text: '{"queryClass":"broad_summary","confidence":0.9}', + output_parsed: { queryClass: "broad_summary", confidence: 0.9 }, + }, + request_id: "req_parse", + }), + }; + }), + }; + }, + })); + + const { generateParsedTextResult } = await import("../src/lib/openai"); + const result = await generateParsedTextResult( + "Question", + z.object({ queryClass: z.string(), confidence: z.number() }).strict(), + { + model: "gpt-5.6-luna", + operation: "text_generation", + schemaName: "clinical_query_classifier", + safetyIdentifier: "a".repeat(64), + }, + ); + + expect(result.parsed).toEqual({ queryClass: "broad_summary", confidence: 0.9 }); + expect(capturedBody).toMatchObject({ + safety_identifier: "a".repeat(64), + text: { + format: expect.objectContaining({ type: "json_schema", name: "clinical_query_classifier", strict: true }), + }, + }); + }); + + it("omits GPT-5.6 prompt cache options when the TTL is disabled", async () => { + let capturedBody: Record = {}; + + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.stubEnv("OPENAI_PROMPT_CACHE_TTL", "off"); + + vi.doMock("openai", () => ({ + default: class MockOpenAI { + embeddings = { create: vi.fn() }; + responses = { + create: vi.fn((body: Record) => { + capturedBody = body; + return { + withResponse: async () => ({ + data: { status: "completed", output_text: '{"answer":"ok"}' }, + request_id: "req_gpt56_cache_off", + }), + }; + }), + }; + }, + })); + + const { generateStructuredTextResult } = await import("../src/lib/openai"); + await generateStructuredTextResult( + "Question", + { type: "object", properties: {}, required: [] }, + { model: "gpt-5.6-sol", operation: "answer", schemaName: "clinical_test" }, + ); + + expect(capturedBody).not.toHaveProperty("prompt_cache_options"); + expect(capturedBody).not.toHaveProperty("prompt_cache_retention"); + }); + + it("fails closed when a Responses request reports failure or no output", async () => { + const responseQueue = [ + { status: "failed", output_text: "" }, + { status: "completed", output_text: "" }, + { status: "incomplete", incomplete_details: { reason: "content_filter" }, output_text: "blocked" }, + ]; + + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.doMock("openai", () => ({ + default: class MockOpenAI { + embeddings = { create: vi.fn() }; + responses = { + create: vi.fn(() => ({ + withResponse: async () => ({ data: responseQueue.shift(), request_id: "req_failed" }), + })), + }; + }, + })); + + const { generateStructuredTextResult } = await import("../src/lib/openai"); + const generate = () => + generateStructuredTextResult( + "Question", + { type: "object", properties: {}, required: [] }, + { model: "gpt-5.6-terra", operation: "answer", schemaName: "clinical_test" }, + ); + + await expect(generate()).rejects.toMatchObject({ details: { code: "openai_response_failed" } }); + await expect(generate()).rejects.toMatchObject({ details: { code: "openai_missing_output" } }); + await expect(generate()).rejects.toMatchObject({ details: { code: "openai_content_filtered" } }); + }); + + it("preserves caller cancellation instead of remapping it to a provider timeout", async () => { + const controller = new AbortController(); + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.doMock("openai", () => ({ + default: class MockOpenAI { + embeddings = { create: vi.fn() }; + responses = { + create: vi.fn(() => ({ + withResponse: async () => { + controller.abort(); + throw controller.signal.reason; + }, + })), + }; + }, + })); + + const { generateStructuredTextResult } = await import("../src/lib/openai"); + await expect( + generateStructuredTextResult( + "Question", + { type: "object", properties: {}, required: [] }, + { + model: "gpt-5.6-terra", + operation: "answer", + schemaName: "clinical_test", + signal: controller.signal, + }, + ), + ).rejects.toMatchObject({ name: "AbortError" }); + }); + it("applies model capability rules for reasoning and verbosity", async () => { const capturedBodies: Record[] = []; @@ -314,7 +507,7 @@ describe("OpenAI query embedding cache", () => { expect(capturedBodies[3]).toMatchObject({ max_output_tokens: 300 }); }); - it("flags truncated (incomplete) responses (GEN-C1)", async () => { + it("returns an empty max-token incomplete response so the caller can retry (GEN-C1)", async () => { vi.stubEnv("OPENAI_API_KEY", "test-key"); vi.doMock("openai", () => ({ @@ -326,7 +519,7 @@ describe("OpenAI query embedding cache", () => { data: { status: "incomplete", incomplete_details: { reason: "max_output_tokens" }, - output_text: '{"answer":"Withhold clozapine if ANC below', + output_text: "", }, request_id: "req_trunc", }), @@ -345,6 +538,7 @@ describe("OpenAI query embedding cache", () => { expect(result.truncated).toBe(true); expect(result.status).toBe("incomplete"); expect(result.incompleteReason).toBe("max_output_tokens"); + expect(result.text).toBe(""); }); it("marks a completed response as not truncated (GEN-C1)", async () => { diff --git a/tests/openai-error-mapping.test.ts b/tests/openai-error-mapping.test.ts index 1d337f682..c6aa1c19f 100644 --- a/tests/openai-error-mapping.test.ts +++ b/tests/openai-error-mapping.test.ts @@ -46,10 +46,31 @@ describe("mapOpenAIError quota vs rate-limit classification", () => { const mapped = mapOpenAIError(openAIError("Incorrect API key provided.", { status: 401 }), "answer"); expect(mapped.status).toBe(500); expect(mapped.message.toLowerCase()).toContain("authentication"); + expect(mapped.details?.code).toBe("openai_invalid_api_key"); + }); + + it("distinguishes access denial from an invalid API key", () => { + const mapped = mapOpenAIError( + openAIError("Project is not permitted to use this model.", { status: 403 }), + "answer", + ); + expect(mapped.status).toBe(500); + expect(mapped.details?.code).toBe("openai_access_denied"); + expect(mapped.message).toMatch(/project.*organization.*model/i); + }); + + it("maps an unavailable model to a stable operator-facing code", () => { + const mapped = mapOpenAIError( + openAIError("The model does not exist or you do not have access.", { status: 404, code: "model_not_found" }), + "answer", + ); + expect(mapped.status).toBe(500); + expect(mapped.details?.code).toBe("openai_model_not_found"); }); it("maps timeouts to a 504 source-only fallback signal", () => { const mapped = mapOpenAIError(openAIError("Request timed out.", { code: "ETIMEDOUT" }), "answer"); expect(mapped.status).toBe(504); + expect(mapped.details?.code).toBe("openai_timeout"); }); }); diff --git a/tests/openai-safety-identifier.test.ts b/tests/openai-safety-identifier.test.ts new file mode 100644 index 000000000..4a3259785 --- /dev/null +++ b/tests/openai-safety-identifier.test.ts @@ -0,0 +1,28 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +afterEach(() => { + vi.unstubAllEnvs(); + vi.resetModules(); +}); + +describe("OpenAI safety identifiers", () => { + it("returns a stable HMAC without exposing the owner ID", async () => { + vi.stubEnv("OPENAI_SAFETY_IDENTIFIER_SECRET", "test-secret-that-is-at-least-thirty-two-characters"); + const { openAISafetyIdentifier } = await import("../src/lib/openai"); + + const first = openAISafetyIdentifier("2dfb60cb-cc8b-48fd-865d-428227fbda89"); + const second = openAISafetyIdentifier("2dfb60cb-cc8b-48fd-865d-428227fbda89"); + const other = openAISafetyIdentifier("6f26191c-5ce1-42e1-8e59-00312dff0d99"); + + expect(first).toMatch(/^[a-f0-9]{64}$/); + expect(first).toBe(second); + expect(first).not.toBe(other); + expect(first).not.toContain("2dfb60cb"); + }); + + it("omits the identifier for anonymous requests or unconfigured deployments", async () => { + const { openAISafetyIdentifier } = await import("../src/lib/openai"); + expect(openAISafetyIdentifier(undefined)).toBeUndefined(); + expect(openAISafetyIdentifier("owner-id")).toBeUndefined(); + }); +}); diff --git a/tests/private-access-routes.test.ts b/tests/private-access-routes.test.ts index f14fd4d5d..f58484873 100644 --- a/tests/private-access-routes.test.ts +++ b/tests/private-access-routes.test.ts @@ -3860,40 +3860,34 @@ describe("private document API access", () => { }); it("streams only public progress details and exactly one completion before the final answer", async () => { - const answerQuestionWithScope = vi.fn( - async (args: { - onProgress?: (event: unknown) => void | Promise; - onRevising?: (reason: string) => void; - }) => { - await args.onProgress?.({ - stage: "routing", - message: "private-message-marker", - resultCount: 2, - visibleSourceCount: 1, - timingMs: 42, - selectedContextCount: 4.9, - australianSourceCount: 4, - waSourceCount: 3, - usedSupplementaryFallback: true, - model: "private-model-marker", - mode: "private-mode-marker", - reason: "private-reason-marker", - smartApiPlan: { marker: "private-plan-marker" }, - relevance: { marker: "private-relevance-marker" }, - privateMarker: "private-direct-marker", - }); - args.onRevising?.("private-revising-marker"); - await args.onProgress?.({ stage: "complete", message: "private-complete-marker" }); - await args.onProgress?.({ stage: "complete", message: "private-duplicate-complete-marker" }); - return { - answer: "Owned evidence.", - grounded: true, - confidence: "medium", - citations: [], - sources: [], - }; - }, - ); + const answerQuestionWithScope = vi.fn(async (args: { onProgress?: (event: unknown) => void | Promise }) => { + await args.onProgress?.({ + stage: "routing", + message: "private-message-marker", + resultCount: 2, + visibleSourceCount: 1, + timingMs: 42, + selectedContextCount: 4.9, + australianSourceCount: 4, + waSourceCount: 3, + usedSupplementaryFallback: true, + model: "private-model-marker", + mode: "private-mode-marker", + reason: "private-reason-marker", + smartApiPlan: { marker: "private-plan-marker" }, + relevance: { marker: "private-relevance-marker" }, + privateMarker: "private-direct-marker", + }); + await args.onProgress?.({ stage: "complete", message: "private-complete-marker" }); + await args.onProgress?.({ stage: "complete", message: "private-duplicate-complete-marker" }); + return { + answer: "Owned evidence.", + grounded: true, + confidence: "medium", + citations: [], + sources: [], + }; + }); const client = createSupabaseMock(); mockRuntime(client, { answerQuestionWithScope }); const { POST } = await import("../src/app/api/answer/stream/route"); @@ -3931,7 +3925,8 @@ describe("private document API access", () => { waSourceCount: 3, }); expect(body).not.toContain("usedSupplementaryFallback"); - expect(body).toContain("event: revising\ndata: {}"); + expect(body).not.toContain("event: token"); + expect(body).not.toContain("event: revising"); for (const privateMarker of [ "private-message-marker", "private-model-marker", @@ -3940,7 +3935,6 @@ describe("private document API access", () => { "private-plan-marker", "private-relevance-marker", "private-direct-marker", - "private-revising-marker", "private-complete-marker", "private-duplicate-complete-marker", ]) { diff --git a/tests/private-rag-access.test.ts b/tests/private-rag-access.test.ts index 7ad6b41ad..b548dfada 100644 --- a/tests/private-rag-access.test.ts +++ b/tests/private-rag-access.test.ts @@ -400,12 +400,18 @@ describe("private RAG API access", () => { const { POST } = await import("../src/app/api/answer/stream/route"); const response = await POST(jsonRequest("/api/answer/stream", { query: "clozapine monitoring" })); - await response.text(); + const stream = await response.text(); expect(response.status).toBe(200); + expect(stream).not.toContain("event: token"); + expect(stream).not.toContain("event: revising"); + expect(stream).toContain("event: final"); expect(mocks.answerQuestionWithScope).toHaveBeenCalledWith( expect.objectContaining({ ownerId: undefined, allowGlobalSearch: true, query: "clozapine monitoring" }), ); + expect(mocks.answerQuestionWithScope).toHaveBeenCalledWith( + expect.not.objectContaining({ onToken: expect.anything(), onRevising: expect.anything() }), + ); }); it("scopes authenticated real answer streams to the authenticated owner", async () => { diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index 4e5354eb5..59c18f288 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -187,6 +187,55 @@ describe("RAG structured-output fallback", () => { expect(answer.routingReason).not.toContain("source_backed_extractive_fallback"); }); + it("recovers a generated comparison with an uncited high-risk value through the source-safe fallback", async () => { + const comparisonFact = (documentId: string, chunkId: string, value: string) => ({ + id: `${documentId}-threshold`, + document_id: documentId, + source_chunk_id: chunkId, + source_image_id: null, + page_number: 2, + table_title: "ANC thresholds", + row_label: "Red range", + clinical_parameter: "ANC", + threshold_value: value, + action: "Withhold and repeat FBC", + }); + const answer = await answerFromTextSources( + "Compare and reconcile the clinical implications of these ANC thresholds", + [ + source({ + id: "chunk-a", + document_id: "doc-a", + title: "Protocol A", + table_facts: [comparisonFact("doc-a", "chunk-a", "below 1.5 x 10^9/L")], + }), + source({ + id: "chunk-b", + document_id: "doc-b", + title: "Protocol B", + table_facts: [comparisonFact("doc-b", "chunk-b", "below 1.0 x 10^9/L")], + }), + ], + { + answer: "Protocol A uses below 1.5 x 10^9/L, while Protocol B uses below 1.0 x 10^9/L; both require action.", + grounded: true, + confidence: "high", + answerSections: [], + citations: [{ chunk_id: "chunk-a" }], + quoteCards: [], + conflictsOrGaps: [], + }, + ); + + expect(answer.grounded).toBe(true); + expect(answer.routingMode).toBe("extractive"); + expect(answer.routingReason).toContain("generation_fallback:generation_quality_failed"); + expect(answer.routingReason).toContain("comparison_source_safe_fallback"); + expect(answer.answer).toContain("Protocol A: below 1.5 x 10^9/L"); + expect(answer.answer).toContain("Protocol B: below 1.0 x 10^9/L"); + expect(answer.unverifiedNumericTokens).toBeUndefined(); + }); + it("keeps table-threshold questions on fact synthesis instead of source lookup", async () => { const answer = await answerFromTextSources("What ANC threshold does the clozapine table show?", [ source({ @@ -726,21 +775,16 @@ describe("RAG structured-output fallback", () => { from: vi.fn(() => new EmptyQuery()), }), })); + const generateParsedTextResult = vi.fn(async () => ({ + parsed: { + queryClass: "unsupported_or_general", + confidence: 0.4, + reasons: ["direct definition question"], + expandedTerms: ["bulimia nervosa"], + }, + })); const generateStructuredTextResult = vi .fn() - .mockResolvedValueOnce({ - text: JSON.stringify({ - queryClass: "unsupported_or_general", - confidence: 0.4, - reasons: ["direct definition question"], - expandedTerms: ["bulimia nervosa"], - }), - model: "gpt-4.1-mini", - operation: "text_generation", - latencyMs: 6, - requestId: "req_classifier", - usage: { input_tokens: 80, output_tokens: 20, total_tokens: 100 }, - }) .mockResolvedValueOnce({ text: JSON.stringify({ answer: @@ -805,6 +849,7 @@ describe("RAG structured-output fallback", () => { vi.doMock("@/lib/openai", () => ({ embedTextWithTelemetry: vi.fn(), + generateParsedTextResult, generateStructuredTextResult, })); @@ -817,7 +862,8 @@ describe("RAG structured-output fallback", () => { skipCache: true, }); - expect(generateStructuredTextResult).toHaveBeenCalledTimes(3); + expect(generateParsedTextResult).toHaveBeenCalledTimes(1); + expect(generateStructuredTextResult).toHaveBeenCalledTimes(2); expect(answer.routingMode).toBe("strong"); expect(answer.routingReason).toContain("fast_overexpanded_simple_retry_strong"); expect(answer.openAIRequestIds ?? []).toEqual(["req_fast_overexpanded", "req_strong_concise"]); @@ -1505,21 +1551,16 @@ describe("RAG structured-output fallback", () => { if (retrievalRpcBaseName(name) === "get_related_document_metadata") return { data: [], error: null }; return { data: [], error: null }; }); + const generateParsedTextResult = vi.fn(async () => ({ + parsed: { + queryClass: "unsupported_or_general", + confidence: 0.4, + reasons: ["direct definition question"], + expandedTerms: ["bulimia nervosa"], + }, + })); const generateStructuredTextResult = vi .fn() - .mockResolvedValueOnce({ - text: JSON.stringify({ - queryClass: "unsupported_or_general", - confidence: 0.4, - reasons: ["direct definition question"], - expandedTerms: ["bulimia nervosa"], - }), - model: "gpt-5.4-mini", - operation: "text_generation", - latencyMs: 6, - requestId: "req_classifier_invalid_evidence", - usage: { input_tokens: 80, output_tokens: 20, total_tokens: 100 }, - }) .mockResolvedValueOnce({ text: JSON.stringify({ answer: @@ -1569,6 +1610,7 @@ describe("RAG structured-output fallback", () => { })); vi.doMock("@/lib/openai", () => ({ embedTextWithTelemetry: vi.fn(async () => ({ embedding: [0.1, 0.2, 0.3], cacheHit: false })), + generateParsedTextResult, generateStructuredTextResult, })); @@ -1580,7 +1622,8 @@ describe("RAG structured-output fallback", () => { skipCache: true, }); - expect(generateStructuredTextResult).toHaveBeenCalledTimes(3); + expect(generateParsedTextResult).toHaveBeenCalledTimes(1); + expect(generateStructuredTextResult).toHaveBeenCalledTimes(2); expect(answer.routingMode).toBe("strong"); expect(answer.routingReason).toContain("fast_invalid_evidence_retry_strong"); expect(answer.grounded).toBe(true); @@ -1728,7 +1771,7 @@ describe("RAG structured-output fallback", () => { }); let requestIndex = 0; const generateStructuredTextResult = vi.fn(async () => ({ - text: '{"answer":"Lithium dosing requires', + text: "", model: "gpt-5.4-mini", operation: "answer", latencyMs: 12, diff --git a/tests/rag-answer-text.test.ts b/tests/rag-answer-text.test.ts index ca7c46bfe..d3b66607a 100644 --- a/tests/rag-answer-text.test.ts +++ b/tests/rag-answer-text.test.ts @@ -109,6 +109,29 @@ describe("RAG answer text helpers", () => { expect(hasClinicalAnswerQualityIssue("Monitor for respiratio before discharge.")).toBe(true); }); + it("rejects dose figures with a missing unit while accepting a complete dose", () => { + expect(hasClinicalAnswerQualityIssue("For sertraline, increase according to response, maximum 60.")).toBe(true); + expect(hasClinicalAnswerQualityIssue("For sertraline, the maximum dose is 60 mg daily.")).toBe(false); + expect(hasClinicalAnswerQualityIssue("For sertraline, the maximum dose is 60 milligrams daily.")).toBe(false); + expect(hasClinicalAnswerQualityIssue("For lithium, the dose is 10 mmol daily.")).toBe(false); + expect(hasClinicalAnswerQualityIssue("For insulin, the dose is 10 international units daily.")).toBe(false); + expect(hasClinicalAnswerQualityIssue("For olanzapine, the maximum dose is 2 tablets daily.")).toBe(false); + }); + + it("rejects incomplete extractive guidance clauses", () => { + expect(hasClinicalAnswerQualityIssue("The guidance for sertraline is that higher doses than the maximum.")).toBe( + true, + ); + expect(hasClinicalAnswerQualityIssue("Check renal function before. Review the result when compared with.")).toBe( + true, + ); + expect(hasClinicalAnswerQualityIssue("» sertraline is 50 mg daily. mg daily, increase.")).toBe(true); + expect( + hasClinicalAnswerQualityIssue("Best Uses acamprosate requires renal review. Bottom Line continue care."), + ).toBe(true); + expect(hasClinicalAnswerQualityIssue("For acamprosate, clinical Focus Renal function is important.")).toBe(true); + }); + it("polishes cached answer display text without requiring regeneration", () => { expect( polishClinicalAnswerProse("Serum lithium concentrations should be monitored once every three months.1"), diff --git a/tests/rag-cache-utils.test.ts b/tests/rag-cache-utils.test.ts index 818cbc39b..42a5d8ed0 100644 --- a/tests/rag-cache-utils.test.ts +++ b/tests/rag-cache-utils.test.ts @@ -5,7 +5,7 @@ describe("ragCacheKeyMatchesOwner", () => { const ownerId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; it("matches versioned scoped cache keys", () => { - const key = `rag-cache-v12|${ownerId}|scope:all|plan:hybrid|class:dose`; + const key = `rag-cache-v13|${ownerId}|scope:all|plan:hybrid|class:dose`; expect(ragCacheKeyMatchesOwner(key, ownerId)).toBe(true); }); @@ -15,7 +15,7 @@ describe("ragCacheKeyMatchesOwner", () => { }); it("matches owner-plus-public cache keys", () => { - const key = `rag-cache-v12|owner:${ownerId}+public|scope:all|plan:hybrid`; + const key = `rag-cache-v13|owner:${ownerId}+public|scope:all|plan:hybrid`; expect(ragCacheKeyMatchesOwner(key, ownerId)).toBe(true); }); @@ -25,12 +25,12 @@ describe("ragCacheKeyMatchesOwner", () => { }); it("does not match a different owner", () => { - const key = `rag-cache-v12|bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb|scope:all`; + const key = `rag-cache-v13|bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb|scope:all`; expect(ragCacheKeyMatchesOwner(key, ownerId)).toBe(false); }); it("does not match an owner id prefix", () => { - const key = `rag-cache-v12|owner:${ownerId}0+public|scope:all`; + const key = `rag-cache-v13|owner:${ownerId}0+public|scope:all`; expect(ragCacheKeyMatchesOwner(key, ownerId)).toBe(false); }); }); diff --git a/tests/rag-claim-support.test.ts b/tests/rag-claim-support.test.ts index bab026677..4d09a8291 100644 --- a/tests/rag-claim-support.test.ts +++ b/tests/rag-claim-support.test.ts @@ -130,6 +130,29 @@ describe("deterministic claim support", () => { expect(result.confidence).toBe("medium"); }); + it("assesses newline-delimited claims independently instead of merging their evidence scopes", () => { + const admission = source("admission", "Admission requires referral and bed allocation."); + const followUp = source("follow-up", "Follow-up review should occur within 72 hours."); + const input = answer( + "Admission requires referral and bed allocation.\nFollow-up review should occur within 72 hours.", + [admission, followUp], + ); + + const assessment = assessClaimSupport(input); + expect(assessment.claims).toHaveLength(2); + expect(assessment.claims).toEqual( + expect.arrayContaining([ + expect.objectContaining({ text: "Admission requires referral and bed allocation.", supportStatus: "direct" }), + expect.objectContaining({ + text: "Follow-up review should occur within 72 hours.", + riskClass: "high_risk", + supportStatus: "direct", + }), + ]), + ); + expect(assessAndEnforceClaimSupport(input).responseMode).not.toBe("evidence_gap"); + }); + it("ignores incidental outdated or poor retrieval-only sources but fails closed when direct support is dangerous", () => { const direct = source("direct", "Stop clozapine below ANC 1.0 x10^9/L."); const incidental = source("incidental", "An old unrelated administrative note.", { diff --git a/tests/rag-classifier-memo.test.ts b/tests/rag-classifier-memo.test.ts index dc180b878..6a904a6e0 100644 --- a/tests/rag-classifier-memo.test.ts +++ b/tests/rag-classifier-memo.test.ts @@ -1,4 +1,6 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { zodTextFormat } from "openai/helpers/zod"; +import type { ZodType } from "zod"; // Finding #11 interim fix: the LLM classifier verdict must be deterministic per query for // the memo TTL window, so the unsupported short-circuit cannot flip run-to-run and return @@ -12,9 +14,10 @@ afterEach(() => { async function loadWithClassifierMock(mock: ReturnType) { vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.stubEnv("OPENAI_QUERY_CLASSIFIER_MODEL", "gpt-classifier-test"); vi.doMock("@/lib/openai", async (importOriginal) => { const actual = await importOriginal(); - return { ...actual, generateStructuredTextResult: mock }; + return { ...actual, generateParsedTextResult: mock }; }); const rag = await import("../src/lib/rag"); const { analyzeClinicalQuery } = await import("../src/lib/clinical-search"); @@ -24,13 +27,13 @@ async function loadWithClassifierMock(mock: ReturnType) { function classifierResponse(overrides: Record = {}) { return { - text: JSON.stringify({ + parsed: { queryClass: "broad_summary", confidence: 0.9, reasons: ["classifier_test"], expandedTerms: ["mood disorder"], ...overrides, - }), + }, }; } @@ -55,6 +58,14 @@ describe("classifier verdict memoization", () => { const second = await rag.analyzeQueryWithClassifierFallback(query, analysis); expect(mock).toHaveBeenCalledTimes(1); + expect(mock).toHaveBeenCalledWith( + expect.any(Array), + expect.any(Object), + expect.objectContaining({ + model: "gpt-classifier-test", + promptCacheKey: "clinical-rag-query-classifier-v1", + }), + ); expect(first.queryClass).toBe("broad_summary"); expect(second.queryClass).toBe("broad_summary"); expect(second).toEqual(first); @@ -74,6 +85,52 @@ describe("classifier verdict memoization", () => { expect(second).toBe(analysis); }); + it("sends only supported structural constraints to Structured Outputs", async () => { + const mock = vi.fn(async () => classifierResponse()); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + await rag.analyzeQueryWithClassifierFallback(query, analysis); + + const calls = mock.mock.calls as unknown as Array<[unknown, ZodType, unknown]>; + const format = zodTextFormat(calls[0]![1], "classifier_schema_probe"); + const schemaJson = JSON.stringify(format); + expect(schemaJson).not.toMatch(/"(?:minimum|maximum|maxItems|maxLength)"/); + }); + + it("rejects out-of-bounds classifier output after parsing and keeps it retryable", async () => { + const mock = vi.fn(async () => + classifierResponse({ + confidence: 1.1, + reasons: Array.from({ length: 5 }, (_, index) => `reason-${index}`), + expandedTerms: Array.from({ length: 11 }, (_, index) => `term-${index}`), + }), + ); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + const first = await rag.analyzeQueryWithClassifierFallback(query, analysis); + const second = await rag.analyzeQueryWithClassifierFallback(query, analysis); + + expect(mock).toHaveBeenCalledTimes(2); + expect(first).toBe(analysis); + expect(second).toBe(analysis); + }); + + it("threads a pseudonymous safety identifier for an authenticated classifier request", async () => { + vi.stubEnv("OPENAI_SAFETY_IDENTIFIER_SECRET", "test-secret-that-is-at-least-thirty-two-characters"); + const mock = vi.fn(async () => classifierResponse()); + const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); + const { query, analysis } = fallbackQueryAnalysis(analyzeClinicalQuery); + + await rag.analyzeQueryWithClassifierFallback(query, analysis, { ownerId: "owner-a" }); + + const calls = mock.mock.calls as unknown as Array<[unknown, unknown, { safetyIdentifier?: string }]>; + const options = calls[0]?.[2]; + expect(options?.safetyIdentifier).toMatch(/^[a-f0-9]{64}$/); + expect(options?.safetyIdentifier).not.toContain("owner-a"); + }); + it("does not memoize transport errors — the next request retries the classifier", async () => { const mock = vi.fn().mockRejectedValueOnce(new Error("timeout")).mockResolvedValueOnce(classifierResponse()); const { rag, analyzeClinicalQuery } = await loadWithClassifierMock(mock); @@ -88,10 +145,10 @@ describe("classifier verdict memoization", () => { }); it("deduplicates concurrent in-flight calls for the same query", async () => { - let resolveCall: ((value: { text: string }) => void) | undefined; + let resolveCall: ((value: ReturnType) => void) | undefined; const mock = vi.fn( () => - new Promise<{ text: string }>((resolve) => { + new Promise<{ parsed: ReturnType["parsed"] }>((resolve) => { resolveCall = resolve; }), ); diff --git a/tests/rag-generation-fingerprint.test.ts b/tests/rag-generation-fingerprint.test.ts new file mode 100644 index 000000000..1b52e50e0 --- /dev/null +++ b/tests/rag-generation-fingerprint.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from "vitest"; +import { + answerGenerationFingerprint, + buildAnswerGenerationFingerprint, + scopedAnswerCacheKey, + type AnswerGenerationFingerprintInput, +} from "../src/lib/rag-cache"; + +const baseline: AnswerGenerationFingerprintInput = { + answerModel: "gpt-5.6-terra", + fastModel: "gpt-5.6-terra", + strongModel: "gpt-5.6-sol", + classifierModel: "gpt-5.6-luna", + indexingModel: "gpt-5.6-terra", + embeddingModel: "text-embedding-3-small", + embeddingDimensions: 1536, + fastReasoningEffort: "low", + strongReasoningEffort: "high", + answerVerbosity: "medium", + maxOutputTokens: 4000, + providerMode: "auto", + promptVersion: "clinical-rag-answer-v18", + schemaVersion: "clinical-rag-answer-schema-v3", + classifierPromptVersion: "clinical-rag-query-classifier-v1", + retrievalVersion: "deep-memory-v1", + indexingPromptVersion: "clinical-indexing-prompts-v1", +}; + +describe("RAG answer generation fingerprints", () => { + it.each([ + ["fast model", { fastModel: "gpt-5.6-luna" }], + ["classifier model", { classifierModel: "gpt-5.6-terra" }], + ["indexing model", { indexingModel: "gpt-5.6-sol" }], + ["embedding model", { embeddingModel: "text-embedding-3-large" }], + ["embedding dimensions", { embeddingDimensions: 3072 }], + ["strong reasoning", { strongReasoningEffort: "medium" }], + ["answer verbosity", { answerVerbosity: "low" }], + ["maximum output tokens", { maxOutputTokens: 6000 }], + ["provider mode", { providerMode: "offline" }], + ["prompt version", { promptVersion: "clinical-rag-answer-v19" }], + ["schema version", { schemaVersion: "clinical-rag-answer-schema-v4" }], + ["classifier prompt version", { classifierPromptVersion: "clinical-rag-query-classifier-v2" }], + ["retrieval version", { retrievalVersion: "deep-memory-v2" }], + ["indexing prompt version", { indexingPromptVersion: "clinical-indexing-prompts-v2" }], + ])("changes when %s changes", (_label, override) => { + expect(buildAnswerGenerationFingerprint({ ...baseline, ...override })).not.toBe( + buildAnswerGenerationFingerprint(baseline), + ); + }); + + it("is stable for the same generation configuration", () => { + expect(buildAnswerGenerationFingerprint({ ...baseline })).toBe(buildAnswerGenerationFingerprint(baseline)); + }); + + it("is included in the scoped answer cache key", () => { + expect(scopedAnswerCacheKey({ query: "Clozapine monitoring", ownerId: "owner-a" })).toContain( + `generation:${answerGenerationFingerprint()}`, + ); + }); +}); diff --git a/tests/rag-offline-answer.test.ts b/tests/rag-offline-answer.test.ts index c670be6c6..c24e28968 100644 --- a/tests/rag-offline-answer.test.ts +++ b/tests/rag-offline-answer.test.ts @@ -131,8 +131,8 @@ describe("source-only / offline answers", () => { expect(answer.routingReason).toContain("source_only"); expect(answer).toMatchObject({ answer: expect.any(String), - grounded: false, - confidence: "unsupported", + grounded: true, + confidence: "medium", citations: [ expect.objectContaining({ chunk_id: "clozapine-chunk-1", @@ -158,6 +158,10 @@ describe("source-only / offline answers", () => { expect(new Set(answer.sources.map((item) => item.id))).toEqual( new Set(answer.citations.map((citation) => citation.chunk_id)), ); + expect(answer.answer.replaceAll("**", "")).toContain( + "guidance on ANC threshold for the decision to withhold clozapine", + ); + expect(answer.answer).toContain("review them directly"); // quality signalling for the UI disclosure expect(answer.answerQualityTier).toBe("source_only"); expect(answer.providerMode).toBe("offline"); diff --git a/tests/ui-smoke.spec.ts b/tests/ui-smoke.spec.ts index c22f2a837..5dd766501 100644 --- a/tests/ui-smoke.spec.ts +++ b/tests/ui-smoke.spec.ts @@ -1864,7 +1864,7 @@ test.describe("Clinical KB UI smoke coverage", () => { await expectNoPageHorizontalOverflow(page); }); - test("stopping generation removes provisional output and exposes a stable rerun action", async ({ page }) => { + test("stopping generation exposes a stable rerun action without answer output", async ({ page }) => { await page.setViewportSize({ width: 390, height: 844 }); await mockDemoApi(page, { answerDelayMs: 1500 }); const question = "What monitoring is required for clozapine?";