fix(rag): harden GPT-5.6 answer generation

.env.example

@@ -38,24 +38,33 @@ OPENAI_API_KEY=replace-with-openai-api-key
 OPENAI_EMBEDDING_MODEL=text-embedding-3-small
 # Must match vector(N) in supabase/schema.sql. Do not change without a migration.
 EMBEDDING_DIMENSIONS=1536
-OPENAI_ANSWER_MODEL=gpt-5.5
-OPENAI_FAST_ANSWER_MODEL=gpt-5.5
-# Strong tier stays on the standard (non-pro) model; fast vs strong differ by reasoning effort.
-OPENAI_STRONG_ANSWER_MODEL=gpt-5.5
+OPENAI_ANSWER_MODEL=gpt-5.6-terra
+OPENAI_FAST_ANSWER_MODEL=gpt-5.6-terra
+OPENAI_STRONG_ANSWER_MODEL=gpt-5.6-sol
+# Workload-specific models can be canaried independently of clinical synthesis.
+OPENAI_QUERY_CLASSIFIER_MODEL=gpt-5.6-luna
+OPENAI_SUMMARY_MODEL=gpt-5.6-terra
+OPENAI_INDEXING_MODEL=gpt-5.6-terra
 OPENAI_MAX_OUTPUT_TOKENS=4000
 OPENAI_QUERY_CACHE_SIZE=200
 # Max inputs per embeddings request (OpenAI caps a request at 2048 inputs / ~300k tokens).
 # embedTexts splits a full-corpus re-embed into batches of this size.
 OPENAI_EMBEDDING_BATCH_SIZE=256
-OPENAI_VISION_MODEL=gpt-5.5
+OPENAI_VISION_MODEL=gpt-5.6-terra
 OPENAI_VISION_IMAGE_DETAIL=auto
 OPENAI_REQUEST_TIMEOUT_MS=45000
 # Answer-generation budget. Kept generous so a strong reasoning model can finish a
 # natural, model-written answer instead of timing out into stitched extractive prose.
 OPENAI_ANSWER_TIMEOUT_MS=30000
 OPENAI_MAX_RETRIES=2
 OPENAI_GENERATION_MAX_RETRIES=0
+# Pre-5.6 compatibility only. GPT-5.6 never receives prompt_cache_retention.
 OPENAI_PROMPT_CACHE_RETENTION=24h
+# GPT-5.6 prompt cache option: 30m | off. "off" omits the extended TTL option.
+OPENAI_PROMPT_CACHE_TTL=30m
+# Optional HMAC secret (min 32 chars) for a stable pseudonymous OpenAI safety_identifier.
+# Raw owner IDs are never sent. Review derivation/retention with privacy governance before enabling.
+#OPENAI_SAFETY_IDENTIFIER_SECRET=
 OPENAI_STORE_RESPONSES=false
 OPENAI_FAST_REASONING_EFFORT=low
 OPENAI_STRONG_REASONING_EFFORT=high

docs/branch-review-ledger.md

@@ -59,6 +59,11 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 
 | 2026-07-13 | origin/main (detached review worktree) | c523cabeae4b68ebdf569ecbc18d9f5a7b5afbf1 | repo-wide audit | Changes requested: one P1 clinical-answer trust cluster; three P2 tenancy/reindex guardrail issues; three P3 information-disclosure, dead-code, and transitive-deprecation cleanup items. No P0 found. | `npm run verify:cheap` (1,721 passed, 1 skipped); `npm run test:coverage` (thresholds passed); `npm run build`; `npm run verify:ui` (137/137); `npm run eval:rag:offline` (36 fixtures, 60 tests); format, Edge Function, Codex workflow, import/secret/dead-reference scans. Provider-backed checks skipped. |
 | 2026-07-13 | HEAD / origin/main (detached worktree) | 04c1d0b036cae8af4dabfc692055c7aab93d5888 | OpenAI-facing API and integration review | Read-only review found one P1 clinical-streaming defect and four P2 reliability/API-contract issues: provisional clinical prose is exposed before validation; mid-stream failure can silently trigger a second buffered generation; answer caches are not model/prompt fingerprinted; OpenAI access/model errors are under-classified; and table-fact route IDs are not validated before database access. GPT-5.6 migration also requires replacing the legacy prompt-cache parameter. No application code was changed. | Static call-flow, prompt, model, schema, streaming, cache, error, route, test, and governance inspection; official OpenAI model/Responses/structured-output/streaming/prompt-cache guidance reviewed. Provider-backed checks were not run. Local tests were not run because this worktree has no installed dependencies (`openai`, `vitest`). |
+| 2026-07-13 | codex/openai-gpt56-rag-upgrade | 4fa4c35e98d60fc104639089494b271a5f1951fd | OpenAI and RAG review remediation | Remediated all recorded findings: clinical SSE is final-only across mixed-version deployments; buffered generation cannot silently replace a partial stream; answer caches are generation/retrieval fingerprinted; GPT-5.6 model, prompt-cache, workload routing, parsed-output, usage, safety-identifier, and error handling are capability-aware; and table-fact UUIDs fail with the shared 400 contract. Added rollout and governance documentation. Independent final review found no remaining high-confidence issue after the mixed-version client guard was added. | Replaced the external `node_modules` junction with a clean `npm ci`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (211 files passed, 1 skipped; 1,941 tests passed, 1 skipped); focused cache/stream tests, offline RAG preflight, production-readiness CI, changed-file Prettier/ESLint, and `git diff --check` passed before the current `origin/main` integration. Provider and post-merge checks are recorded separately when complete. |
+| 2026-07-14 | codex/openai-gpt56-rag-upgrade | f83d7d49b32fb9bb8b8c2ede99e93675574c88d8 | PR review follow-up | Confirmed and remediated five scoped review findings: answer-quality JSON no longer retains owner-scoped answer previews; valid tablet/mmol dose phrases are accepted; review-fallback routing uses a shared token; the classifier sends only supported structural schema constraints and validates bounds after parsing; and equivalent maximum-dose wording remains eligible. Replaced the redundant dashboard source-string guard with the existing mixed-version browser behavior regression. | Focused Vitest 123/123; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (234 files passed, 1 skipped; 2,229 tests passed, 1 skipped); `git diff --check`. Provider-backed retrieval remains separately blocked by exhausted OpenAI embedding quota after case 24/36. |
+| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | 1c6c780f30f33d47e7dfb57dce56d5fd0501d46e | PR review follow-up | Confirmed and remediated three remaining scoped findings: empty `max_output_tokens` responses now reach the RAG retry path; equivalent maximum-dose wording satisfies the maximum-dose intent gate without requiring literal `maximum` or `dose` tokens; and long-form clinical dose units are accepted consistently by evidence extraction and final-answer validation. | Focused Vitest 99/99; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed runtime/policy/lint/typecheck and full Vitest (235 files passed, 1 skipped; 2,238 tests passed, 1 skipped); `npm run build` generated 636 static pages and passed the client-bundle secret scan; configured production readiness was READY with three documented warnings; `git diff --check`. Provider-backed retrieval remains separately blocked by exhausted OpenAI embedding quota after case 24/36. |
+| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | 0301d59ab8d6afeecd340bb08b40b98ff6b95c6c | PR review follow-up | Confirmed and fixed one additional maximum-dose wording gap discovered after the current-main merge: `not to exceed` and `not more than` numeric limits now reach the same validated extractive path as the existing equivalent phrases. | Focused Vitest 84/84; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; focused Prettier and `git diff --check`. The immediately preceding merged head also passed `npm run verify:cheap` (236 files passed, 1 skipped; 2,242 tests passed, 1 skipped), production build, and all hosted required checks. Provider-backed retrieval remains blocked by exhausted OpenAI embedding quota after case 24/36. |
+| 2026-07-14 | PR #626 / codex/openai-gpt56-rag-upgrade | f2652021c1f14533f17c4a9724ef6dff17dd43f5 | PR review follow-up | Confirmed and fixed the two final scoped findings: GPT-5.6-and-later model families now use the TTL prompt-cache request shape, and unrelated prose containing `maximum` can no longer pass the maximum-dose evidence gate without a clinical dose value or explicit dose wording. | Focused Vitest 99/99 before current-main integration and 163/163 after it; offline RAG 36 fixtures and 279/279 tests; `npm run typecheck`; `npm run verify:cheap` passed after integrating `origin/main` (239 files passed, 1 skipped; 2,248 tests passed, 1 skipped); focused Prettier and `git diff --check`. The reviewed head passed every hosted required check before these narrow fixes. Provider-backed retrieval remains blocked by exhausted OpenAI embedding quota after case 24/36. |
 | 2026-07-13 | claude/audit-ci-browser-gate-2026-07-13 | 65a8a0c9e7c3a165b09a1ad79e893af8c2c6973b | branch-cleanup | Retained because the branch is checked out in an active or protected worktree. | Fresh worktree, status, lock, and process activity scan. |
 | 2026-07-13 | claude/beautiful-hamilton-5df54c | 5e2e90f0a3af4039c7e15515151569228476a60c | branch-cleanup | Redundant: the exact HEAD is an ancestor of `origin/main`; eligible for deletion when no worktree or open PR references it. | `git merge-base --is-ancestor 5e2e90f0a3af4039c7e15515151569228476a60c origin/main`. |
 | 2026-07-13 | claude/beautiful-williamson-57d144 | 072434cd5f59b1441dd114958489028972c42d34 | branch-cleanup | Redundant: exact source HEAD was squash-merged by PR #474; eligible for deletion when unreferenced. | GitHub merged-PR source HEAD matched exactly. |

docs/capacity-review.md

@@ -72,8 +72,9 @@ moment load is highest. This only works while the app is a **single process**
 Per answer: 1 embedding call (unless the lexical fast path skips it) + 1–2
 generations (fast route, escalation to strong). 15 answers/min with grounded
 prompts (~5–15k tokens each) lands in the low hundreds of thousands of
-tokens/min at worst — within Tier-2+ gpt-5.5 TPM limits, but _bursts_ of
-simultaneous strong-route generations can trip request-per-minute limits.
+tokens/min at worst. Re-baseline the production project's current Terra/Sol
+token and request limits before rollout; _bursts_ of simultaneous strong-route
+generations can still trip request-per-minute limits.
 Existing dampers: coalescing (duplicate questions never reach OpenAI), the
 answer cache, `OPENAI_MAX_RETRIES`, and graceful degradation to source-only
 answers (which must stay _visible_ — see the degraded-rate SLO).

docs/codex-prompt-playbook.md

@@ -705,7 +705,7 @@ Audit and harden structured output contracts for this repo.
 Focus on places where model or model-like output is parsed, displayed, stored,
 or graded:
 - src/lib/rag.ts
-- src/lib/answer-stream-extractor.ts
+- src/lib/answer-stream-contract.ts
 - src/lib/answer-verification.ts
 - src/lib/answer-render-policy.ts
 - src/app/api/answer/route.ts

docs/openai-cross-border-basis.md

@@ -17,10 +17,11 @@
 This document covers the model-provider leg: query text + retrieved excerpts sent to OpenAI in the
 United States for embedding and answer synthesis (PIA §3–4; verified still true in code —
 [src/lib/openai.ts:75-79](../src/lib/openai.ts) builds a plain `new OpenAI({ apiKey, timeout, maxRetries })`
-with no `baseURL`/ZDR header, `store:false` by default, and `prompt_cache_retention` forced to `"24h"`
-for gpt-5.5 at [openai.ts:174](../src/lib/openai.ts)). Railway application and worker processing in
-Singapore is the separate overseas processor leg recorded in the PIA; the APP 8 record must cover
-both providers.
+with no `baseURL`/ZDR header and `store:false` by default. GPT-5.6-and-later requests use
+`prompt_cache_options.ttl="30m"`; gpt-5.5 requests force the legacy
+`prompt_cache_retention="24h"` field ([openai.ts](../src/lib/openai.ts)). Railway application and
+worker processing in Singapore is the separate overseas processor leg recorded in the PIA; the APP
+8 record must cover both providers.
 
 Two obligations attach to that flow:
 
@@ -38,14 +39,16 @@ The code-side controls cannot _by themselves_ discharge APP 8 — the "reasonabl
 
 ## 2. What actually crosses the border
 
-| Egress    | Payload                                                          | Endpoint                                         | Reference                                                                                    |
-| --------- | ---------------------------------------------------------------- | ------------------------------------------------ | -------------------------------------------------------------------------------------------- |
-| Embedding | Raw query text (normalized)                                      | `POST /v1/embeddings` (`text-embedding-3-small`) | [openai.ts embedText](../src/lib/openai.ts)                                                  |
-| Answer    | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (`gpt-5.5`, `store:false`)  | [rag.ts:4220](../src/lib/rag.ts) · [rag-source-block.ts:180](../src/lib/rag-source-block.ts) |
+| Egress    | Payload                                                          | Endpoint                                                      | Reference                                                                           |
+| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
+| Embedding | Raw query text (normalized)                                      | `POST /v1/embeddings` (`text-embedding-3-small`)              | [openai.ts embedText](../src/lib/openai.ts)                                         |
+| Answer    | Raw query verbatim + retrieved chunk text + static system prompt | `POST /v1/responses` (Terra fast / Sol strong, `store:false`) | [rag.ts](../src/lib/rag.ts) · [rag-source-block.ts](../src/lib/rag-source-block.ts) |
 
-The app **adds no patient identifiers** and stores queries only as a keyed hash locally; but it does
-**not scrub** PHI a clinician types. Everything else (documents, embeddings, logs, auth) stays at rest
-in **Sydney — AWS `ap-southeast-2`** (PIA §7).
+The app **adds no raw patient or owner identifiers** and stores queries only as a keyed hash locally.
+When configured, authenticated Responses requests include a stable HMAC-SHA256
+`safety_identifier`; anonymous and background requests omit it. The app does **not scrub** PHI a
+clinician types. Everything else (documents, embeddings, logs, auth) stays at rest in
+**Sydney — AWS `ap-southeast-2`** (PIA §7).
 
 ## 3. OpenAI's current terms (verified 2026-07-13)
 
@@ -60,7 +63,7 @@ these terms change; the PIA (2026-07-06) already predates the Australia data-res
 | **Zero Data Retention (ZDR)** | Removes the 30-day abuse-monitoring retention; **not self-serve** — prior approval by OpenAI, configured per **project**. Apply via the account/sales team.                                                                                                                                                                                             | [Data controls](https://developers.openai.com/api/docs/guides/your-data)                                                                                                                                           |
 | **Data residency**            | API data residency now covers **Australia** (among US, Europe, UK, Canada, Japan, Korea, Singapore, India, UAE). Enabled by creating a **new Project** and selecting the country; eligibility via sales. **Australia = storage at rest only** — regional _processing/inference_ is US/Europe/UAE only. ~10% uplift for models released from 5 Mar 2026. | [Data residency (API)](https://help.openai.com/en/articles/10503543-data-residency-for-the-openai-api) · [Announcement](https://openai.com/index/expanding-data-residency-access-to-business-customers-worldwide/) |
 | **Sub-processors**            | Published list of sub-processors that may process Customer Data. Review for the APP 8 accountability chain.                                                                                                                                                                                                                                             | [Sub-processor list](https://openai.com/policies/sub-processor-list/) · [platform](https://platform.openai.com/subprocessors)                                                                                      |
-| **Prompt caching**            | For gpt-5.5, `prompt_cache_retention` cannot be `in_memory`; app forces `"24h"` ([openai.ts:174](../src/lib/openai.ts)). Whether ZDR also zeroes the prompt cache must be **confirmed in writing** (see §6, PIA-6).                                                                                                                                     | [Data controls](https://developers.openai.com/api/docs/guides/your-data)                                                                                                                                           |
+| **Prompt caching**            | GPT-5.6 requests `prompt_cache_options.ttl="30m"` by default and never receives the deprecated retention field. The TTL is a minimum, not a guaranteed deletion deadline. Explicit pre-5.6 deployments retain the legacy retention behavior. ZDR interaction must be **confirmed in writing** (see §6, PIA-6).                                          | [Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) · [Data controls](https://developers.openai.com/api/docs/guides/your-data)                                                          |
 
 ## 4. This app's endpoints are ZDR-eligible
 
@@ -90,9 +93,10 @@ Items 1–2 (plus documenting 4–5) are what turn PIA-1 from open to closed. It
 
 ## 6. Open question to pin with OpenAI
 
-**Does ZDR eliminate the forced 24h gpt-5.5 prompt cache**, or does an encrypted ≤24h cache persist
-regardless? Get this in writing — it determines whether **PIA-6** is fully resolved by ZDR or merely
-mitigated. Record the answer in the status block.
+**What is the effective prompt-cache deletion behavior under ZDR for GPT-5.6 requests that specify
+the 30-minute TTL, and for requests where the app omits the extended TTL option?** Get this in
+writing — it determines whether **PIA-6** is fully resolved by ZDR or merely mitigated. Record the
+answer in the status block.
 
 ## 7. Consistency with the shipped user-facing notice
 
@@ -166,9 +170,8 @@ in the PIA. No automated action in this review accepted either provider's terms.
 These touch the OpenAI request path — do them **only after** the legal decision, and treat them as
 provider-path changes (confirm before running against live).
 
-- **ZDR granted:** no code change strictly required (ZDR is account/project-side). Optionally revisit
-  the forced `"24h"` prompt-cache retention ([openai.ts:174](../src/lib/openai.ts)) depending on the §6
-  answer, and note the resolution against **PIA-6**.
+- **ZDR granted:** no code change strictly required (ZDR is account/project-side). Revisit
+  `OPENAI_PROMPT_CACHE_TTL` depending on the §6 answer and note the resolution against **PIA-6**.
 - **Australia data residency adopted:** the client currently has no `baseURL` override
   ([openai.ts:75-79](../src/lib/openai.ts)). Data-residency Projects route via the standard API with a
   region-scoped project key; confirm whether a `baseURL`/project-key change is needed and wire an