BigSimmo · BigSimmo · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026
@@ -666,3 +666,5 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from 5b4098d; PR: E-2 baseline record + instrument fix) | see PR head | ADDENDUM Phase E-2 BASELINE BANKED (canary dispatch #57, run 29786560936, main 5b4098d, answer_case_limit=44): golden retrieval 36/36 green in-run (no-regression net held); eval:quality full-44 RED on exactly two gates — citation_failure_rate 0.0227 (1/44: neuroleptic-side-effect-escalation — expected doc never retrieved, generation quality-failed, extractive fallback with 1 citation) and route_ceiling_failure_count 2 (clozapine-anc-withhold-threshold: 13.3s pure retrieval vs 12s extractive budget, RPC 9.3s, zero generation; agitation-arousal-typo-dosing: 25054ms vs 25000ms after provider_timeout ate 22.6s pre-recovery). Green gates: grounded_supported 1.0, unsupported_correct 1.0 (all 14 refusals incl. both prompt-injection probes at 2ms), numeric grounding failures 0, governance danger 0, p95 17.4s. Non-blocking signals: expected_source_hit 0.6136 (both admission-discharge cases miss MHSP.AdmissionCommunityPts.pdf to sibling NMHS/RKPG policies — labeling-vs-ranking question, §3.1 class), source governance warning rate 0.8182 (metadata debt, waivable class). SYSTEMIC E-3 TARGET: ~9 of 19 generation attempts discarded (fast output fails quality gate → extractive fallback wins; ~7 cases carry fast_quality_retry_strong→extractive reasons) = ~half of generation latency+spend wasted. Instrument defect found+fixed this PR: targeting step was skipped after the red gate (GitHub failure-skip semantics) → if: gains !cancelled() so baselines observe red gates. Spend: est ~$1-2 actual (19 OpenAI-request cases; cost rates unset in CI so report shows n/a) of ≤$20 Phase E envelope. Next: cheap re-dispatch (default limit 8 + answer_quality_eval=true) to bank the skipped 30-case targeting baseline. | Gates this PR: check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean. Baseline evidence: job log run 29786560936 (5 failing-case diagnostics + Answer Metrics table read in full) |
 | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from 87815f4; PR: E-3a cost self-reporting) | see PR head + follow-up pin | Phase E-3a wave (all $0 diagnostics complete, recorded here) + I8 fix: eval-canary env gains the three RAG_EVAL_*_USD_PER_MILLION rates (gpt-5.6-terra standard tier, verified from the live OpenAI pricing page 2026-07-21; documented as a LOWER BOUND since gpt-5.6-sol strong retries cost 2x and the estimator applies one rate set) so estimated_cost_usd stops reading n/a in CI. E-3a findings: (1) discarded-generation histogram from run #57 — fast_quality_retry_strong→extractive x6, generation_quality_failed x4, provider_timeout x2; gate sub-reasons fragment_like x2, bad_final_answer_quality x1, ungrounded_extractive_fallback x1 → E-3c targets the fast-attempt-fails-extractive-wins shape first. (2) I2 resolved: routeCeilingExceeded (eval-quality.ts:157/171) keys on the RUNTIME's own route_budget_ms + deadline flag, NOT the cross-region-widened eval gates — clozapine case = 13.3s retrieval vs the runtime's 12s extractive budget, geography-amplified; design → E-3b. (3) FIXTURE-CORPUS DRIFT (major): the live corpus contains ZERO MHSP*-named files; the 44-case fixture's expectedFiles are all MHSP.*/CG.MHSP.* and pass only via the eval-document-matching alias tier. All three expected-doc failures triaged: neuroleptic-side-effect-escalation = REAL retrieval-coverage gap (alias exists; Neuroleptic Side Effects (AKG).pdf indexed 11 chunks, 3 contain escalat%, yet answer retrieval returned a single off-point source) → protected-path item awaiting user go-ahead; admission-discharge x2 = top-5 preference for sibling NMHS/RKPG policies over the existing aliased AKG doc → labeling-vs-ranking decision presented to user. No fixture/alias edits made (ground truth requires sign-off). | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; Supabase access read-only (documents/document_chunks SELECTs); OpenAI pricing page read via WebFetch |
 | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from c308fdc; PR: labeling widen) | see PR head | USER-APPROVED ground-truth widening (AskUserQuestion 2026-07-21: "Widen to accept siblings") for the two admission-discharge cases: the WIDER alias tier's AdmissionCommunityPts entry gains the two NMHS "Admission to Discharge" titles (Community Mental Health + Mental Health Inpatients). Deliberate exclusions documented in-code: discharge-only docs must not satisfy the admission slot; the MHHITH programme policy is too narrow. STRICT golden tier untouched (bulk-merge prohibition respected). Verified by replaying run #57's actual top-5 lists through expectedFileCoverage: admission-discharge-coverage-paraphrase now passes; admission-discharge-comparison STILL FAILS honestly (its top-5 carries no admission-side doc at all — dup discharge-planning + Falls Prevention) and is retained as a genuine comparison-class retrieval-coverage signal, folded into the same investigation as the neuroleptic case. | typecheck PASS; prettier clean; probe script replay recorded (paraphrase allHit true / comparison allHit false, missing admission slot); no retrieval code touched; protected-file edit (eval-document-matching.ts) per explicit user authorization |
+| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f | Clinical-governance review of E-3b diff `git diff origin/main...HEAD` (commits 1078264 + 314d03f): answer-side generation timing/gating + telemetry — reserve-aware generation timeout (`generationRequestTimeoutMs`), truncation self-heal budget gate (`deadlineAllowsGenerationRetry`), `route_budget_exhausted_by_retrieval` telemetry, and the cross-region eval carve-out in scripts/eval-quality.ts. | APPROVE-WITH-NITS. No P0/P1/P2. (a) Conservative failure preserved: reserve-aware timeout fires ~2s early producing the SAME error type — an internal SDK timeout is mapped by mapOpenAIError to PublicApiError(openai_timeout) (openai.ts:525-529), NOT a bare DOMException, so the rag.ts:4636 re-throw guard is not tripped and the existing source-backed fallback/extractive recovery is reached; truncation-skip falls through to the terminal throw (rag.ts:4309-4313) into the same catch. No new answer-producing path. (b) Safety gates intact: all recovery answers finalize through finalizeAnswer→finalizeRagAnswerQuality and isSafeExtractiveFallbackCandidate (grounded/confidence/quality/numeric) — none touched. (c) Eval carve-out env-gated: crossRegionRunner && budgetExhaustedByRetrieval && generationMs===0; EVAL_LATENCY_CONTEXT set only in eval-canary.yml:164, prod caller (eval-quality.ts:1095) passes no options → inert in release/local; generationMs===0 requirement means a generation-side failure (generationMs>0) is never suppressed. (d)/(e) Telemetry additions non-PHI (boolean + mechanical retry-reason strings); no privacy/query-privacy/cross-border/verification source files touched; no new provider call. Nits (P3, non-blocking): carve-out also excuses fast/strong routes when generationMs===0 (sound — provably no generation ran); `requestTimeoutMs` now prod-dead (test-only); report label "retrieval-exhausted" (routeDeadlineExceeded && flag) is broader than actual gate suppression (audit label only, gate stays strict). | 88 offline unit tests PASS (tests/rag-route-budget.test.ts, tests/eval-quality.test.ts, tests/rag-offline-answer.test.ts, tests/rag-answer-fallback.test.ts); no provider/Supabase/OpenAI calls; no files mutated except this ledger row |
+| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f (+eb08ec5 review row) | ADDENDUM 5 wave E-3b implemented per the design-agent plan: generation attempts clamped to route budget minus a measured 2s recovery reserve (single call-site, all four attempt kinds); truncation self-heal gated on retry viability (reserve+5s floor) with observable truncation_retry_skipped_budget_reserve marker; additive route_budget_exhausted_by_retrieval runtime flag; eval route-ceiling gains the triple-condition cross-region carve-out (context + runtime flag + zero generation) with retrieval-exhausted audit cells — local/release gates provably strict. Fixes I3 (54ms budget overrun after 22.6s provider timeout), I5 (82s truncation waste class), resolves I2 (clozapine 13.3s retrieval vs 12s runtime budget = geography, now suppressed ONLY in the sanctioned cross-region context with full auditability). Reviewer verdicts: rag-retrieval-reviewer APPROVE-WITH-NITS (2 P3: prod-dead requestTimeoutMs retained for symmetry; report-cell coupling cosmetic; cached-replay invariant PROVEN — budget-exhausted answers never cached, carve-out unreachable via replay; marker isolation proven — SLO counters key on fallback_reason not answer_retry_reasons); clinical-governance-reviewer APPROVE-WITH-NITS (prior row) — internal-timeout→PublicApiError→existing-fallback path verified, all safety gates still applied to recovery answers. ALSO BANKED — E-2 targeting baseline (canary run #58, 29788404357, all-green incl. first execution of the !cancelled()-fixed instrument, ~$1-2): metric_rates relevance 0.6 / readability 1.0 / artifact_leaks 1.0 / intent_coverage 0.9333 / fail_closed 0.9; targeting_rate 0.5909 (13/22); by intent: document_lookup 5/5, red_result_action 3/3, contraindication 2/2, dose 1/5, monitoring_schedule 1/5, pathway_referral 1/2; all 9 misses = missing dose figure/schedule-interval (answer lengths 73-232 chars) → E-3c co-primary target alongside the wasted-generation class. Phase E spend ≈$3-6 of ≤$20. | Red-proofs: reserve pinned 3 independent ways (exact 23000ms grant, deadline flag clear, total under budget); self-heal skip pins exact marker + single provider call; offline flag pinned true/false. Focused: route-budget 9/9, eval-quality 27/27, fallback+offline 52/52, parser/abort regressions 15/15. Full suite 3043 passed / 1 known container pdf artifact. typecheck+lint+prettier clean. No provider calls; live proof = E-4 paired run |
@@ -89,6 +89,7 @@ export type RagQualityResult = {
     totalMs: number;
     routeBudgetMs: number;
     routeDeadlineExceeded: boolean;
+    budgetExhaustedByRetrieval?: boolean;
   };
   routeCeilingExceeded?: boolean;
   estimatedCostUsd: number | null;
@@ -150,14 +151,25 @@ export function deliveredGroundedAfterSourceGovernancePolicy(
 
 const crossRegionRunnerLatencyContext = process.env.EVAL_LATENCY_CONTEXT === "cross-region-runner";
 
-export function ragAnswerTimingDiagnostics(answer: Pick<RagAnswer, "latencyTimings" | "routingMode">) {
+export function ragAnswerTimingDiagnostics(
+  answer: Pick<RagAnswer, "latencyTimings" | "routingMode">,
+  options?: { crossRegionRunner?: boolean },
+) {
   const latencyTimings = answer.latencyTimings;
   const answerRoute = answer.routingMode ?? "unsupported";
   const defaultRouteBudgetMs = answerRouteBudgetMs[answerRoute as AnswerRouteMode] ?? 0;
   const routeBudgetMs = latencyTimings?.route_budget_ms ?? defaultRouteBudgetMs;
   const totalMs = latencyTimings?.total_latency_ms ?? 0;
   const generationMs = latencyTimings?.generation_latency_ms ?? 0;
   const routeDeadlineExceeded = latencyTimings?.route_deadline_exceeded ?? false;
+  const budgetExhaustedByRetrieval = latencyTimings?.route_budget_exhausted_by_retrieval ?? false;
+  const crossRegionRunner = options?.crossRegionRunner ?? crossRegionRunnerLatencyContext;
+  // Cross-region carve-out (E-3b): when GitHub's US runner spends the whole route budget on
+  // retrieval alone and the runtime then behaves optimally (zero generation), the ceiling is
+  // runner geography, not a runtime regression. Requires ALL THREE: the sanctioned cross-region
+  // context, the runtime's own authoritative flag, and no generation time. Local/release
+  // contexts (EVAL_LATENCY_CONTEXT unset) are unaffected — the gate is not weakened there.
+  const ceilingExcusedByCrossRegionRetrieval = crossRegionRunner && budgetExhaustedByRetrieval && generationMs === 0;
   return {
     timings: {
       retrievalMs: latencyTimings?.retrieval_latency_ms ?? latencyTimings?.search_latency_ms ?? 0,
@@ -167,8 +179,11 @@ export function ragAnswerTimingDiagnostics(answer: Pick<RagAnswer, "latencyTimin
       totalMs,
       routeBudgetMs,
       routeDeadlineExceeded,
+      budgetExhaustedByRetrieval,
     },
-    routeCeilingExceeded: routeDeadlineExceeded || (routeBudgetMs === 0 ? generationMs > 0 : totalMs > routeBudgetMs),
+    routeCeilingExceeded: ceilingExcusedByCrossRegionRetrieval
+      ? false
+      : routeDeadlineExceeded || (routeBudgetMs === 0 ? generationMs > 0 : totalMs > routeBudgetMs),
   };
 }
 
@@ -744,7 +759,13 @@ function ragCaseDiagnosticsTable(results: RagQualityResult[]) {
         result.rpcLatencyMs,
         result.embeddingLatencyMs,
         result.timings?.routeBudgetMs,
-        result.timings?.routeDeadlineExceeded ? "yes" : "no",
+        // "retrieval-exhausted" keeps cross-region-suppressed ceilings auditable in
+        // run-over-run comparisons even though they no longer fail the gate.
+        result.timings?.routeDeadlineExceeded
+          ? result.timings?.budgetExhaustedByRetrieval
+            ? "retrieval-exhausted"
+            : "yes"
+          : "no",
         result.model,
         result.failures.length > 0 ? `failed (${result.failures.length})` : "passed",
       ]
@@ -841,9 +862,13 @@ export function renderEvalQualityMarkdown(report: EvalQualityReport) {
           item.timings?.generationMs ?? "n/a"
         }ms verification=${item.timings?.verificationMs ?? "n/a"}ms total=${
           item.timings?.totalMs ?? item.latencyMs
-        }ms budget=${
-          item.timings?.routeBudgetMs ?? "n/a"
-        }ms deadline=${item.timings?.routeDeadlineExceeded ? "yes" : "no"}`,
+        }ms budget=${item.timings?.routeBudgetMs ?? "n/a"}ms deadline=${
+          item.timings?.routeDeadlineExceeded
+            ? item.timings?.budgetExhaustedByRetrieval
+              ? "retrieval-exhausted"
+              : "yes"
+            : "no"
+        }`,
     )
     .join("\n");
   return `# Retrieval Quality Report

@@ -7,6 +7,16 @@ export const answerRouteBudgetMs = {
   strong: 35_000,
 } as const satisfies Record<AnswerRouteMode, number>;
 
+// A generation attempt must never spend the recovery path's share of the route budget.
+// Fixed (not a fraction): recovery cost is O(1) — measured 26ms post-retrieval extractive
+// (eval run #57 clozapine case) and ~100ms-1s for fallback + verification + logging
+// cross-region — so 2s is ~2x the worst observed while costing only 8% of the fast budget.
+export const generationRecoveryReserveMs = 2_000;
+// A truncation self-heal retry needs at least this much wall time to be worth attempting:
+// truncated attempts measure ~20s+ before hitting max_output_tokens, and the strong retry
+// spends MORE reasoning under a boosted cap, so anything shorter is a guaranteed-discard.
+export const minimumGenerationRetryMs = 5_000;
+
 export class AnswerRouteDeadlineExceededError extends Error {
   readonly routeMode: AnswerRouteMode;
   readonly budgetMs: number;
@@ -25,10 +35,20 @@ export type AnswerRouteDeadline = {
   readonly deadlineExceeded: boolean;
   remainingMs(): number;
   requestTimeoutMs(maximumMs: number): number;
+  /** Like requestTimeoutMs, but holds back generationRecoveryReserveMs so the
+   * source-backed recovery path always fits inside the route budget. */
+  generationRequestTimeoutMs(maximumMs: number): number;
   race<T>(promise: Promise<T>): Promise<T>;
   dispose(): void;
 };
 
+/** True when enough budget remains for a generation retry to plausibly complete
+ * AND still leave the recovery reserve. Zero-budget routes always return false. */
+export function deadlineAllowsGenerationRetry(deadline: Pick<AnswerRouteDeadline, "remainingMs" | "budgetMs">) {
+  if (deadline.budgetMs <= 0) return false;
+  return deadline.remainingMs() >= generationRecoveryReserveMs + minimumGenerationRetryMs;
+}
+
 export function answerRouteResultCanBeCached(deadline: Pick<AnswerRouteDeadline, "deadlineExceeded">) {
   return !deadline.deadlineExceeded;
 }
@@ -87,6 +107,12 @@ export function createAnswerRouteDeadline(args: {
       throwIfAborted();
       return Math.max(1, Math.min(maximumMs, remainingMs()));
     },
+    generationRequestTimeoutMs(maximumMs) {
+      throwIfAborted();
+      // Structurally identical to requestTimeoutMs (same 1ms degenerate floor, same
+      // zero-budget behavior) minus the recovery reserve.
+      return Math.max(1, Math.min(maximumMs, remainingMs() - generationRecoveryReserveMs));
+    },
     race<T>(promise: Promise<T>) {
       try {
         throwIfAborted();

@@ -167,6 +167,7 @@ import {
 import {
   answerRouteResultCanBeCached,
   createAnswerRouteDeadline,
+  deadlineAllowsGenerationRetry,
   isAnswerRouteDeadlineExceeded,
 } from "@/lib/rag/rag-route-budget";
 import { fetchRelatedDocumentMetadata, fetchRelatedDocuments } from "@/lib/document-enrichment";
@@ -3570,11 +3571,15 @@ async function answerQuestionWithScopeUncoalesced(
     callerSignal: args.signal,
     startedAt,
   });
+  // True exactly when pre-answer work (dominated by retrieval) consumed the whole route
+  // budget before generation could start. Additive telemetry; deadlineExceeded unchanged.
+  const routeBudgetExhaustedByRetrieval = routeDeadline.budgetMs > 0 && routeDeadline.remainingMs() <= 0;
   const routeTimingDiagnostics = () => ({
     retrieval_latency_ms: searchLatencyMs,
     routing_latency_ms: routingLatencyMs,
     route_budget_ms: routeDeadline.budgetMs,
     route_deadline_exceeded: routeDeadline.deadlineExceeded,
+    route_budget_exhausted_by_retrieval: routeBudgetExhaustedByRetrieval,
   });
   const finalizeAnswer = (answer: RagAnswer, numericVerificationSources?: SearchResult[]) => {
     const verificationStartedAt = Date.now();
@@ -4059,7 +4064,8 @@ ${qualityRetryInstruction}`
           schemaName: "clinical_rag_answer",
           instructions: answerInstructions,
           promptCacheKey: ragAnswerPromptVersion,
-          timeoutMs: routeDeadline.requestTimeoutMs(env.OPENAI_ANSWER_TIMEOUT_MS),
+          // Reserve-aware: never spend the recovery path's share of the route budget.
+          timeoutMs: routeDeadline.generationRequestTimeoutMs(env.OPENAI_ANSWER_TIMEOUT_MS),
           maxRetries: 0,
           reasoningEffort: useStrongReasoning
             ? strongReasoningEffortForQueryClass(queryClass, env.OPENAI_STRONG_REASONING_EFFORT)
@@ -4261,8 +4267,13 @@ ${qualityRetryInstruction}`
     });
     // Adopted from main: retry truncation once for BOTH fast- and strong-routed first attempts
     // (previously fast-only), keyed on route.mode rather than model identity so it stays correct
-    // when the tiers share a model.
-    if (generated.truncated && !retriedWithStrong) {
+    // when the tiers share a model. Budget-gated: a retry into a nearly-spent budget is a
+    // guaranteed-discard — skip it and let the existing source-backed recovery deliver.
+    if (generated.truncated && !retriedWithStrong && !deadlineAllowsGenerationRetry(routeDeadline)) {
+      answerRetryReasons.push(
+        `truncation_retry_skipped_budget_reserve:${generationRetryReason(route.mode === "fast" ? "fast" : "strong", generated)}`,
+      );
+    } else if (generated.truncated && !retriedWithStrong) {
       const retryPrefix = route.mode === "fast" ? "fast" : "strong";
       const retryReason = `${generationRetryReason(retryPrefix, generated)}_retry_strong`;
       answerRetryCount += 1;

@@ -1009,6 +1009,9 @@ export type RagAnswer = {
     verification_latency_ms?: number;
     route_budget_ms?: number;
     route_deadline_exceeded?: boolean;
+    /** Pre-answer work (dominated by retrieval) consumed the entire route budget before
+     * generation could start. Additive; optional for mixed-version telemetry compat. */
+    route_budget_exhausted_by_retrieval?: boolean;
     total_latency_ms?: number;
   };
   openAIRequestIds?: string[];