feat(rag): deferred injection hardening + answer telemetry

docs/rag-injection-threat-model.md

@@ -154,8 +154,20 @@ The prompt-level mitigations are now implemented. Note the code has since been d
 
 **Already present before this change (verified):** JSON-schema citation constraint to retrieved chunk IDs (`answerJsonOutputSchemaForResults`, an enum over retrieved `id`s applied to `citations`, `quoteCards`, `answerSections.citation_chunk_ids`, and `conflictsOrGaps.source_chunk_ids`) — the master plan's "constrain citations where practical" gap is closed. Query-side manipulation refusal (`hasAdversarialManipulationIntent`) and the strong-route reasoning-effort latency cap also already existed.
 
-**Deferred (unchanged here; tracked as follow-ups):** widening the neutralizer vocabulary (mitigation #8 — high false-positive risk on clinical prose, and the provenance boundary already covers the class); per-chunk numeric scoping (#9); cross-source dose-disagreement flag (#10); the seeded-chunk generation eval harness (#11) needed for the answer-text-level INJ cases (INJ-1/2/5/9/10/11/14/15); and the source-authority tier / prose-entailment / structural-role-separation items (#12–14).
+## 7. Follow-ups implemented on `claude/deferred-rag-improvements`
+
+Three of the deferred items had a low-risk, offline-verifiable "simplest safe" form and are now implemented:
+
+1. **Conservative neutralizer widening (mitigation #8 — defense-in-depth only).** `neutralizePromptInstructions` ([`source-text-sanitizer.ts:407`](../src/lib/source-text-sanitizer.ts)) gains two patterns chosen for **near-zero clinical-prose false-positive risk** because they explicitly address the AI: (a) AI/assistant-addressee markers (`note/attention/message/instruction/directive/override/reminder to|for the ai/assistant/model/llm/bot`), catching `NOTE TO AI` (INJ-1) and `override for assistants` (INJ-13); and (b) a scoped `from now on … recommend/say/state/answer/respond/reply/report/mention/claim` directive (INJ-1). It **deliberately does not** widen to `additional/new instructions:`, `disregard the previous guidance`, or bare `do not mention/exceed/administer`, which occur legitimately in dosing leaflets and superseding guidelines — the provenance boundary (§6.1) remains the durable defense for the broader class. Guarded by false-positive tests in [`tests/rag-injection.test.ts`](../tests/rag-injection.test.ts).
+2. **Cross-source withholding-threshold disagreement (mitigation #10, targeted variant — INJ-10).** `detectConflictsOrGaps` ([`evidence.ts:572`](../src/lib/evidence.ts)) now emits a `{type:"conflict"}` when two **different documents** tie a **withholding action** (cease/withhold/stop) for the same haematological parameter (ANC/WBC/platelet) to **different threshold values** — the exact faithful-but-wrong / OCR-corruption case where numeric verification launders a poisoned cutoff. Scoping to the withholding action (not every number) keeps legitimate red/amber monitoring bands and titration schedules from tripping it; requiring two distinct documents keeps a single self-listing document quiet. Covered by unit + false-positive tests in [`tests/evidence.test.ts`](../tests/evidence.test.ts).
+3. **Per-answer observability (the tracked gap).** The answer paths ([`/api/answer`](../src/app/api/answer/route.ts) and [`/api/answer/stream`](../src/app/api/answer/stream/route.ts)) previously wrote **no** telemetry — an answer's generation route/model/token usage was only visible via `eval:rag`. They now write one `rag_retrieval_logs` row per answered request (fire-and-forget, mirroring `logRetrievalDiagnostics`) via the pure, unit-tested `buildAnswerLogRow` ([`answer-telemetry.ts`](../src/lib/answer-telemetry.ts)), carrying route/model/provider/quality/latency and raw token counts (input/output/cached/reasoning — USD cost is `tokens × per-model price`, derived downstream) in the freeform `metadata.answer` object and tagged `metadata.answer.log_source = "answer"`. Using the existing table + `metadata` JSONB makes this a **zero-migration, zero-schema-drift** change. The fuller Phase-7 design — a dedicated `rag_answer_logs` table with typed cost columns, its own retention cron, RLS, and drift registration — remains a separate owner-approved follow-up.
+
+**Still deferred (genuinely larger; unchanged):**
+
+- **#9 per-chunk numeric scoping** — already scoped at **section** granularity (each `answerSections` entry verifies against its own `citation_chunk_ids`, [`answer-verification.ts:252`](../src/lib/answer-verification.ts)). True per-**sentence** scoping (INJ-14: drug A's number verifying a claim about drug B when both are cited in one section) needs sentence→chunk attribution the model does not reliably emit; it is high-effort, not a simple fix.
+- **#11 seeded-chunk generation eval harness** — the **deterministic** render-time INJ cases are already unit-tested (§6.5 + the two suites above). The remaining answer-**text**-level cases (INJ-1/2/5/9/10/11/14/15) require live model generation, so a harness cannot be verified in an offline/quota-limited environment; it stays a follow-up.
+- **#12–14 (source-authority tier, per-claim prose entailment, structural role separation)** — architectural ("High effort" in §4); no faithful "simplest" form. Left as designed follow-ups.
 
 ---
 
-_Originally prepared as an analysis artifact on branch `claude/safety-analysis`; every BLOCK/MISS verdict was verified against the code at the cited `file:line`. Section 6 records the mitigations subsequently implemented on `claude/llm-pipeline-review`._
+_Originally prepared as an analysis artifact on branch `claude/safety-analysis`; every BLOCK/MISS verdict was verified against the code at the cited `file:line`. Section 6 records the mitigations subsequently implemented on `claude/llm-pipeline-review`; section 7 records the follow-ups implemented on `claude/deferred-rag-improvements`._

src/app/api/answer/route.ts

@@ -21,6 +21,7 @@ import {
 } from "@/lib/source-governance";
 import { parseJsonBody } from "@/lib/validation/body";
 import { createAdminClient } from "@/lib/supabase/admin";
+import { logAnswerDiagnostics } from "@/lib/answer-telemetry";
 import { nonProductionSupabaseDemoFallbackReason } from "@/lib/supabase/errors";
 import * as serverAuth from "@/lib/supabase/auth";
 import type { RagAnswer } from "@/lib/types";
@@ -151,6 +152,8 @@ export async function POST(request: Request) {
       });
     }
 
+    logAnswerDiagnostics({ supabase, query: answerBody.query, ownerId: access.ownerId, answer });
+
     return NextResponse.json({
       ...answer,
       degradedMode: answerDegradedModeSignal(answer),

src/app/api/answer/stream/route.ts

@@ -20,6 +20,7 @@ import {
   sourceGovernanceWarnings,
 } from "@/lib/source-governance";
 import { createAdminClient } from "@/lib/supabase/admin";
+import { logAnswerDiagnostics } from "@/lib/answer-telemetry";
 import { isSupabaseApiKeyConfigurationError, nonProductionSupabaseDemoFallbackReason } from "@/lib/supabase/errors";
 import { AuthenticationError, unauthorizedResponse } from "@/lib/supabase/auth";
 import { logger } from "@/lib/logger";
@@ -218,6 +219,10 @@ function streamAnswer(body: AnswerBody, ownerId?: string, signal?: AbortSignal,
             return;
           }
 
+          if (!isDemoMode()) {
+            logAnswerDiagnostics({ supabase: createAdminClient(), query: body.query, ownerId, answer });
+          }
+
           send("final", {
             ...answer,
             degradedMode: answerDegradedModeSignal(answer),

src/lib/answer-telemetry.ts

@@ -0,0 +1,150 @@
+import { normalizedQueryTextForStorage, queryTextForStorage } from "@/lib/query-privacy";
+import type { createAdminClient } from "@/lib/supabase/admin";
+import type { Json } from "@/lib/supabase/database.types";
+import type { RagAnswer } from "@/lib/types";
+
+// Per-answer observability (threat-model §7 follow-up).
+//
+// The retrieval-side /api/search path writes rag_retrieval_logs, but the answer
+// paths (/api/answer and /api/answer/stream) wrote NOTHING — so an answer's
+// generation route, model, token usage, and cost were only observable via
+// eval:rag, never in production. This records one rag_retrieval_logs row per
+// answered request (the answer path DOES retrieve, so a retrieval-log row is
+// the right home) and carries the answer-side telemetry in the freeform
+// `metadata.answer` object. Using the existing table + `metadata` JSONB keeps
+// this a zero-migration, zero-schema-drift change; a dedicated `rag_answer_logs`
+// table with typed cost columns is the fuller Phase-7 design.
+//
+// Rows are tagged `metadata.answer.log_source = "answer"` so answer-path
+// retrievals are filterable apart from the search-path retrieval telemetry that
+// shares the table.
+
+const UUID_PATTERN = /^[0-9a-f-]{36}$/i;
+
+// The subset of a generated answer this telemetry reads. Kept as a Pick so the
+// builder is decoupled from the full RagAnswer surface and trivially testable.
+export type AnswerTelemetrySource = Pick<
+  RagAnswer,
+  | "grounded"
+  | "confidence"
+  | "sources"
+  | "queryClass"
+  | "modelUsed"
+  | "routingMode"
+  | "routingReason"
+  | "providerMode"
+  | "answerQualityTier"
+  | "responseMode"
+  | "fallbackReason"
+  | "degradedMode"
+  | "openAIUsage"
+  | "openAIRequestIds"
+  | "latencyTimings"
+  | "retrievalDiagnostics"
+>;
+
+function finiteOrNull(value: number | null | undefined): number | null {
+  return typeof value === "number" && Number.isFinite(value) ? value : null;
+}
+
+function meanHybridScore(sources: AnswerTelemetrySource["sources"]): number | null {
+  const scores = (sources ?? []).slice(0, 20).map((source) => source.hybrid_score ?? source.similarity ?? 0);
+  if (scores.length === 0) return null;
+  const mean = scores.reduce((total, score) => total + score, 0) / scores.length;
+  return mean || null;
+}
+
+// Build the rag_retrieval_logs insert row for an answered request. Pure and
+// synchronous so it can be unit-tested without a database.
+export function buildAnswerLogRow(args: { query: string; ownerId?: string | null; answer: AnswerTelemetrySource }) {
+  const { answer } = args;
+  const sources = answer.sources ?? [];
+  const topSource = sources[0] ?? null;
+  const topScore = topSource ? (topSource.hybrid_score ?? topSource.similarity ?? 0) : 0;
+  const timings = answer.latencyTimings ?? {};
+  const usage = answer.openAIUsage ?? {};
+  const isMiss = answer.grounded === false || answer.confidence === "unsupported";
+
+  const answerTelemetry = {
+    log_source: "answer",
+    route: answer.routingMode ?? answer.retrievalDiagnostics?.routeMode ?? null,
+    model: answer.modelUsed ?? null,
+    provider_mode: answer.providerMode ?? null,
+    quality_tier: answer.answerQualityTier ?? null,
+    confidence: answer.confidence,
+    grounded: answer.grounded,
+    response_mode: answer.responseMode ?? null,
+    routing_reason: answer.routingReason ?? null,
+    fallback_reason: answer.fallbackReason ?? null,
+    degraded: answer.degradedMode?.active ?? false,
+    generation_latency_ms: finiteOrNull(timings.generation_latency_ms),
+    search_latency_ms: finiteOrNull(timings.search_latency_ms),
+    answer_retry_count: finiteOrNull(timings.answer_retry_count),
+    request_ids: answer.openAIRequestIds ?? [],
+    // Raw token counts are the durable primitive — USD cost = tokens × per-model
+    // price is derived downstream from a pricing table, not baked into the row.
+    tokens: {
+      input: finiteOrNull(usage.input_tokens),
+      output: finiteOrNull(usage.output_tokens),
+      total: finiteOrNull(usage.total_tokens),
+      cached_input: finiteOrNull(usage.cached_input_tokens),
+      reasoning_output: finiteOrNull(usage.reasoning_output_tokens),
+    },
+  };
+
+  return {
+    owner_id: args.ownerId ?? null,
+    query: queryTextForStorage(args.query),
+    normalized_query: normalizedQueryTextForStorage(args.query),
+    query_class: answer.queryClass ?? answer.retrievalDiagnostics?.queryClass ?? null,
+    candidate_count: sources.length,
+    top_similarity: finiteOrNull(topSource?.similarity),
+    top_hybrid_score: finiteOrNull(topScore),
+    mean_hybrid_score: meanHybridScore(sources),
+    selected_chunk_ids: sources
+      .slice(0, 12)
+      .map((source) => source.id)
+      .filter((id) => UUID_PATTERN.test(id)),
+    selected_document_ids: [...new Set(sources.slice(0, 12).map((source) => source.document_id))].filter((id) =>
+      UUID_PATTERN.test(id),
+    ),
+    selected_count: Math.min(sources.length, 12),
+    embedding_latency_ms: finiteOrNull(timings.embedding_latency_ms),
+    rpc_latency_ms: finiteOrNull(timings.supabase_rpc_latency_ms),
+    rerank_latency_ms: finiteOrNull(timings.rerank_latency_ms),
+    total_latency_ms: finiteOrNull(timings.total_latency_ms),
+    vector_candidate_count: finiteOrNull(timings.vector_candidate_count),
+    text_candidate_count: finiteOrNull(timings.text_candidate_count),
+    embedding_field_count: finiteOrNull(timings.embedding_field_count),
+    embedding_cache_hit: typeof timings.embedding_cache_hit === "boolean" ? timings.embedding_cache_hit : null,
+    is_miss: isMiss,
+    miss_reason: isMiss ? (answer.fallbackReason ?? answer.responseMode ?? "unsupported") : null,
+    metadata: { answer: answerTelemetry } as unknown as Json,
+  };
+}
+
+// Fire-and-forget writer, mirroring logRetrievalDiagnostics in /api/search: a
+// logging failure must never affect the answer response, so the insert is
+// detached and its error swallowed (with a throttled warning).
+let answerLogFailureCount = 0;
+
+export function logAnswerDiagnostics(args: {
+  supabase: ReturnType<typeof createAdminClient>;
+  query: string;
+  ownerId?: string | null;
+  answer: AnswerTelemetrySource;
+}) {
+  void (async () => {
+    try {
+      await args.supabase.from("rag_retrieval_logs").insert(buildAnswerLogRow(args));
+    } catch (error) {
+      answerLogFailureCount += 1;
+      if (answerLogFailureCount <= 3 || answerLogFailureCount % 25 === 0) {
+        console.warn("rag_retrieval_logs answer insert failed", {
+          failures: answerLogFailureCount,
+          message: error instanceof Error ? error.message : "unknown answer logging error",
+        });
+      }
+    }
+  })();
+}