perf: reduce search and document latency

bundle-budget.json

@@ -1,6 +1,7 @@
 {
-  "$comment": "Client JS bundle-size budget. Warn-only until a baseline is captured: after a known-good `npm run build`, run `npm run check:bundle-budget -- --update` to set totalGzipBytes, then flip enforce to true so CI fails on >tolerancePct growth. See scripts/check-bundle-budget.mjs.",
-  "enforce": false,
+  "$comment": "Client JS bundle-size budget captured from a known-good production build. CI fails when total gzip size grows beyond tolerancePct; refresh intentionally with `npm run check:bundle-budget -- --update`.",
+  "enforce": true,
   "tolerancePct": 10,
-  "totalGzipBytes": null
+  "totalGzipBytes": 1363382,
+  "updatedAt": "2026-07-17T12:11:55.267Z"
 }

docs/operator-apply-performance-latency-remediation.md

@@ -0,0 +1,25 @@
+# Operator apply — performance and latency remediation
+
+This worktree does not apply migrations. Production rollout remains a separate,
+explicitly authorized operation after local replay, review, and backups.
+
+## Registry projection index on a busy database
+
+`20260717170000_registry_projection_cleanup.sql` creates
+`documents_registry_projection_lookup_idx` transactionally so clean local
+replay remains deterministic. On a busy production database, pre-create the
+exact index outside a transaction:
+
+```sql
+create index concurrently if not exists documents_registry_projection_lookup_idx
+  on public.documents (
+    (metadata->>'registry_record_kind'),
+    (metadata->>'registry_record_id')
+  )
+  where metadata->>'source_kind' = 'registry_record';
+```
+
+After the index is valid and ready, the migration's `create index if not
+exists` is a no-op. Do not mark the migration applied merely because the index
+exists: the cleanup function, hardened privileges, and three lifecycle triggers
+must still be installed through the normal authorized migration rollout.

docs/process-hardening.md

@@ -320,7 +320,7 @@ hybrid:10}`, all 10 forced-embedding vector cases passed (`force_embedding_failu
 ## 2026-07-13 audit remediation batch (branch claude/audit-remediation-2026-07-13)
 
 - **Lexical retrieval rewrite (audit finding 1):** `20260713100000_index_friendly_lexical_retrieval.sql` splits `match_document_chunks_text`'s OR-across-relations candidate search into two GIN-index probes unioned by chunk id (same contract, same scores; the `_v2` wrapper inherits the speedup). Parity + plan + timing harness: `scripts/sql/lexical-rpc-parity-check.sql` (scratch databases only; run it against the drift-manifest container kept with `--keep`). Re-run it whenever either lexical body changes.
-- **supabase_admin default privileges (audit finding 7):** `supabase/roles.sql` establishes and verifies the intended defaults before migrations on fresh databases. Supabase CLI reads that file as the unprivileged local `postgres` role, so the migration-replay job starts an empty emulator without migrations, applies the file once as `supabase_admin`, and then applies every migration with `supabase migration up --local` in the same fresh database. This preserves the database-scoped defaults without granting reserved role membership. `20260717161000_assert_supabase_admin_default_privileges.sql` evaluates effective defaults from `pg_default_acl` plus `acldefault`, so a missing catalog row cannot hide PostgreSQL's built-in `PUBLIC EXECUTE` on functions. The migration attempts the hardening statements and then fails closed if the effective postcondition is unsafe, returning the exact six-statement operator remediation in its error hint. Existing hosted databases do not receive `roles.sql` through ordinary migration deployment; after an authorized hosted apply, run the read-only provider check with `npm run check:default-acl -- --confirm-provider-read`. This command contacts the live Supabase project and therefore requires explicit provider approval. The future-object probe in `20260713102000_revoke_supabase_admin_default_privileges.sql` remains useful in environments that can assume `supabase_admin`.
+- **supabase_admin default privileges (audit finding 7):** `supabase/roles.sql` establishes and verifies the intended defaults before migrations on fresh databases. Supabase CLI reads that file as the unprivileged local `postgres` role, so the migration-replay job starts an empty emulator without migrations, applies the file once as `supabase_admin`, and then applies every migration with `supabase migration up --local` in the same fresh database. This preserves the database-scoped defaults without granting reserved role membership. `20260717161000_assert_supabase_admin_default_privileges.sql` evaluates effective defaults from `pg_default_acl` plus `acldefault`, so a missing catalog row cannot hide PostgreSQL's built-in `PUBLIC EXECUTE` on functions. `20260717173000_reassert_supabase_admin_default_privileges.sql` repeats that fail-closed postcondition after the later performance migrations and must remain the final migration. Existing hosted databases do not receive `roles.sql` through ordinary migration deployment; after an authorized hosted apply, run the read-only provider check with `npm run check:default-acl -- --confirm-provider-read`. This command contacts the live Supabase project and therefore requires explicit provider approval. The future-object probe in `20260713102000_revoke_supabase_admin_default_privileges.sql` remains useful in environments that can assume `supabase_admin`.
 - **Legacy rag query text scrub (audit finding 5):** `20260713103000_scrub_legacy_rag_query_text.sql` performs four redaction/deletion operations for pre-HMAC plaintext query text: (1) scrubs `rag_queries.query` rows not matching `redacted-query:%`, replacing them with salted `redacted-query:legacy:` placeholders; (2) scrubs both `rag_query_misses.query` and `rag_query_misses.normalized_query` not matching `redacted-query:%`; (3) scrubs both `rag_retrieval_logs.query` and `rag_retrieval_logs.normalized_query` not matching `redacted-query:%` (nullable); (4) deletes `rag_response_cache` rows where `normalized_query` does not match `redacted-cache:%` (cache entries, not re-keyed). **Operator verification after live apply:** for each affected table/operation, count rows not matching the expected redacted pattern (expect 0 unless `RAG_PERSIST_RAW_QUERY_TEXT` is deliberately enabled): `select count(*) from rag_queries where query not like 'redacted-query:%';` (expect 0), `select count(*) from rag_query_misses where query not like 'redacted-query:%' or normalized_query not like 'redacted-query:%';` (expect 0), `select count(*) from rag_retrieval_logs where query not like 'redacted-query:%' or (normalized_query is not null and normalized_query not like 'redacted-query:%');` (expect 0), `select count(*) from rag_response_cache where normalized_query not like 'redacted-cache:%';` (expect 0). The migration includes a post-apply assertion block that enforces these exact checks and fails the migration if any unscrubbed/undeleted rows remain.
 - Remaining operator items from the 2026-07-13 audit (confirmation-required, not automated): apply this batch's migrations live, re-run Supabase advisors, trigger the Eval Canary and require two consecutive green scheduled runs, repair the invalid `storage.idx_objects_bucket_id_name_lower` index via dashboard/support, and dedupe response-cache purge cron jobs if `cron.job` shows duplicates.
 

scripts/check-bundle-budget.mjs

@@ -25,6 +25,30 @@ import { fileURLToPath, pathToFileURL } from "node:url";
 const root = path.join(path.dirname(fileURLToPath(import.meta.url)), "..");
 const CHUNKS_DIR = path.join(root, ".next", "static", "chunks");
 const BUDGET_PATH = path.join(root, "bundle-budget.json");
+const APP_BUILD_MANIFEST_PATH = path.join(root, ".next", "app-build-manifest.json");
+const BUILD_MANIFEST_PATH = path.join(root, ".next", "build-manifest.json");
+const ROOT_PAGE_CLIENT_REFERENCE_MANIFEST_PATH = path.join(
+  root,
+  ".next",
+  "server",
+  "app",
+  "page_client-reference-manifest.js",
+);
+
+const fixtureSnapshotMarkerGroups = [
+  {
+    name: "services snapshot",
+    markers: ["deep_research_citation_tokens", "canonical_name_key", "source_table_lines"],
+  },
+  {
+    name: "forms fixture catalogue",
+    markers: ["transport-crisis-form", "extension-transport-order", "detention-examination-movement"],
+  },
+  {
+    name: "differentials snapshot",
+    markers: ["redFlagFlows", "searchAliases", "exportedAt"],
+  },
+];
 
 function walkJsFiles(dir) {
   const out = [];
@@ -74,6 +98,45 @@ export function compareToBudget(current, budget) {
   };
 }
 
+/** Resolve the JavaScript chunks required by the root App Router dashboard. */
+export function initialDashboardChunkNames(appBuildManifest, pageClientReferenceManifest) {
+  const pages = appBuildManifest?.pages ?? {};
+  const pageClientChunks = Object.values(pageClientReferenceManifest?.clientModules ?? {}).flatMap((module) =>
+    Array.isArray(module?.chunks) ? module.chunks : [],
+  );
+  const names = new Set([
+    ...(appBuildManifest?.rootMainFiles ?? []),
+    ...(pages["/layout"] ?? []),
+    ...(pages["/page"] ?? []),
+    ...pageClientChunks,
+  ]);
+  return [...names]
+    .filter((name) => typeof name === "string" && name.endsWith(".js"))
+    .map((name) => name.replace(/^\/?static\/chunks\//, ""));
+}
+
+function loadRootPageClientReferenceManifest() {
+  if (!existsSync(ROOT_PAGE_CLIENT_REFERENCE_MANIFEST_PATH)) return null;
+  const source = readFileSync(ROOT_PAGE_CLIENT_REFERENCE_MANIFEST_PATH, "utf8");
+  const marker = 'globalThis.__RSC_MANIFEST["/page"]=';
+  const start = source.indexOf(marker);
+  if (start < 0) return null;
+  const jsonStart = start + marker.length;
+  const jsonEnd = source.lastIndexOf(";");
+  if (jsonEnd <= jsonStart) return null;
+  return JSON.parse(source.slice(jsonStart, jsonEnd));
+}
+
+/** Identify large fixture payloads from stable groups of serialized keys/slugs.
+ * Requiring every marker in a group avoids failing on ordinary UI copy that
+ * happens to mention one fixture term. */
+export function findFixtureSnapshotsInChunks(files) {
+  const content = files.map(({ buffer }) => buffer.toString("utf8")).join("\n");
+  return fixtureSnapshotMarkerGroups
+    .filter((group) => group.markers.every((marker) => content.includes(marker)))
+    .map((group) => group.name);
+}
+
 function kb(bytes) {
   return `${(bytes / 1024).toFixed(1)} KiB`;
 }
@@ -101,6 +164,30 @@ function main() {
     name: path.relative(CHUNKS_DIR, full),
     buffer: readFileSync(full),
   }));
+  const manifestPath = existsSync(APP_BUILD_MANIFEST_PATH)
+    ? APP_BUILD_MANIFEST_PATH
+    : existsSync(BUILD_MANIFEST_PATH)
+      ? BUILD_MANIFEST_PATH
+      : null;
+  if (!manifestPath) {
+    console.error("[bundle-budget] FAIL — no build manifest is available; cannot verify initial dashboard chunks.");
+    process.exit(1);
+  }
+  const appBuildManifest = JSON.parse(readFileSync(manifestPath, "utf8"));
+  const pageClientReferenceManifest = loadRootPageClientReferenceManifest();
+  const initialChunkNames = new Set(initialDashboardChunkNames(appBuildManifest, pageClientReferenceManifest));
+  const initialDashboardChunks = files.filter((file) => initialChunkNames.has(file.name.replace(/\\/g, "/")));
+  if (initialDashboardChunks.length === 0) {
+    console.error("[bundle-budget] FAIL — no root dashboard JavaScript chunks were resolved from the build manifest.");
+    process.exit(1);
+  }
+  const fixtureViolations = findFixtureSnapshotsInChunks(initialDashboardChunks);
+  if (fixtureViolations.length > 0) {
+    console.error(
+      `[bundle-budget] FAIL — initial dashboard chunks contain fixture payloads: ${fixtureViolations.join(", ")}.`,
+    );
+    process.exit(1);
+  }
   const current = measureChunks(files);
   const budget = loadBudget();
 
@@ -113,7 +200,7 @@ function main() {
 
   const verdict = compareToBudget(current, budget);
   if (asJson) {
-    console.log(JSON.stringify({ current, verdict }, null, 2));
+    console.log(JSON.stringify({ current, verdict, initialDashboardChunks: initialDashboardChunks.length }, null, 2));
   } else {
     console.log(
       `[bundle-budget] client chunks: ${current.files} files, ${kb(current.totalGzipBytes)} gzip (${kb(current.totalRawBytes)} raw).`,
@@ -122,6 +209,9 @@ function main() {
       console.log(`[bundle-budget] baseline ${kb(verdict.baseline)} gzip; ${verdict.reason}.`);
     console.log("[bundle-budget] largest chunks (gzip):");
     for (const c of current.largest) console.log(`  ${kb(c.gzipBytes).padStart(12)}  ${c.name}`);
+    console.log(
+      `[bundle-budget] initial dashboard fixture assertion passed (${initialDashboardChunks.length} chunks).`,
+    );
   }
 
   if (verdict.status === "fail") {

scripts/dev-free-port.mjs

@@ -1,5 +1,6 @@
 #!/usr/bin/env node
 import { spawn } from "node:child_process";
+import fs from "node:fs";
 import net from "node:net";
 import path from "node:path";
 import { fileURLToPath } from "node:url";
@@ -60,6 +61,24 @@ function removePortArgs(args) {
   return cleaned;
 }
 
+function dependencyBundlerArgs(command, args) {
+  if (command !== "dev" || args.some((arg) => ["--webpack", "--turbopack", "--turbo"].includes(arg))) {
+    return [];
+  }
+
+  try {
+    const dependenciesPath = fs.realpathSync(path.join(projectRoot, "node_modules"));
+    const relativeDependenciesPath = path.relative(projectRoot, dependenciesPath);
+    const dependenciesAreExternal =
+      relativeDependenciesPath === ".." ||
+      relativeDependenciesPath.startsWith(`..${path.sep}`) ||
+      path.isAbsolute(relativeDependenciesPath);
+    return dependenciesAreExternal ? ["--webpack"] : [];
+  } catch {
+    return [];
+  }
+}
+
 function canListenOnHost(port, host) {
   return new Promise((resolve) => {
     const server = net.createServer();
@@ -140,6 +159,7 @@ const child = spawn(
     "--port",
     String(freePort),
     ...removePortArgs(forwardedArgs),
+    ...dependencyBundlerArgs(parsedCommand.command, forwardedArgs),
   ],
   {
     cwd: projectRoot,

src/app/api/answer/stream/route.ts

@@ -5,6 +5,7 @@ import { isDemoMode } from "@/lib/env";
 import { PublicApiError, jsonError } from "@/lib/http";
 import {
   allowRateLimitInMemoryFallbackOnUnavailable,
+  consumeSummaryRateLimits,
   consumeSubjectApiRateLimit,
   rateLimitJsonResponse,
   type ApiRateLimitResult,
@@ -272,24 +273,24 @@ export async function POST(request: Request) {
     const supabase = createAdminClient();
     const access = await publicAccessContext(request, supabase);
 
-    const rateLimit = await consumeSubjectApiRateLimit({
-      supabase,
-      subject: access.rateLimitSubject,
-      bucket: "answer",
-      allowInMemoryFallbackOnUnavailable: allowRateLimitInMemoryFallbackOnUnavailable(),
-    });
-    if (rateLimit.limited) return rateLimitStream(rateLimit);
-
     if (body.summaryMode) {
-      // Streamed full-document summaries use the same paid provider path as the
-      // legacy summary endpoint. Preserve the general answer ceiling, then also
-      // enforce the stricter summary quota before the SSE stream can start.
-      const summaryRateLimit = await consumeSubjectApiRateLimit({
+      const decision = await consumeSummaryRateLimits({
+        supabase,
+        subject: access.rateLimitSubject,
+      });
+      if (decision.rateLimit.limited) {
+        return decision.bucket === "document_summarize"
+          ? documentSummaryRateLimitStream(decision.rateLimit)
+          : rateLimitStream(decision.rateLimit);
+      }
+    } else {
+      const rateLimit = await consumeSubjectApiRateLimit({
         supabase,
         subject: access.rateLimitSubject,
-        bucket: "document_summarize",
+        bucket: "answer",
+        allowInMemoryFallbackOnUnavailable: allowRateLimitInMemoryFallbackOnUnavailable(),
       });
-      if (summaryRateLimit.limited) return documentSummaryRateLimitStream(summaryRateLimit);
+      if (rateLimit.limited) return rateLimitStream(rateLimit);
     }
 
     return streamAnswer(body, resolveRetrievalAccessScope(access.ownerId), request.signal);

src/app/api/differentials/[slug]/route.ts

@@ -17,6 +17,7 @@ import {
 import { ensureDifferentialsSeeded, loadDifferentialSnapshot } from "@/lib/differential-seed";
 import { getDifferentialDetailContext, getDifferentialRecord, getPresentationWorkflow } from "@/lib/differentials";
 import { isDemoMode, isLocalNoAuthMode } from "@/lib/env";
+import { fixtureResponseHeaders } from "@/lib/fixture-response-cache";
 import { jsonError } from "@/lib/http";
 import { safeErrorLogDetails } from "@/lib/privacy";
 import { publicAccessContext } from "@/lib/public-api-access";
@@ -30,10 +31,13 @@ const differentialDetailQuerySchema = z.object({
   kind: z.enum(["presentation", "diagnosis"]).optional().default("diagnosis"),
 });
 
-function differentialResponse(payload: Record<string, unknown>, init?: { status?: number }) {
+function differentialResponse(
+  payload: Record<string, unknown>,
+  init: { status?: number; request?: Request; fixture?: boolean } = {},
+) {
   return NextResponse.json(payload, {
-    status: init?.status ?? 200,
-    headers: { "Cache-Control": "private, no-store" },
+    status: init.status ?? 200,
+    headers: fixtureResponseHeaders(init.request, init),
   });
 }
 
@@ -53,20 +57,26 @@ export async function GET(request: Request, context: { params: Promise<{ slug: s
       if (kind === "presentation") {
         const workflow = getPresentationWorkflow(normalizedSlug);
         if (!workflow) return notFoundResponse(normalizedSlug);
-        return differentialResponse({
-          workflow,
-          governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
-          demoMode: true,
-        });
+        return differentialResponse(
+          {
+            workflow,
+            governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
+            demoMode: true,
+          },
+          { request, fixture: true },
+        );
       }
       const record = getDifferentialRecord(normalizedSlug);
       if (!record) return notFoundResponse(normalizedSlug);
-      return differentialResponse({
-        record,
-        detailContext: getDifferentialDetailContext(record),
-        governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
-        demoMode: true,
-      });
+      return differentialResponse(
+        {
+          record,
+          detailContext: getDifferentialDetailContext(record),
+          governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
+          demoMode: true,
+        },
+        { request, fixture: true },
+      );
     }
 
     // Anonymous callers still resolve access + rate limit before we serve the seed detail:
@@ -91,20 +101,26 @@ export async function GET(request: Request, context: { params: Promise<{ slug: s
       if (kind === "presentation") {
         const workflow = getPresentationWorkflow(normalizedSlug);
         if (!workflow) return notFoundResponse(normalizedSlug);
-        return differentialResponse({
-          workflow,
-          governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
-          publicAccess: true,
-        });
+        return differentialResponse(
+          {
+            workflow,
+            governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
+            publicAccess: true,
+          },
+          { request, fixture: true },
+        );
       }
       const record = getDifferentialRecord(normalizedSlug);
       if (!record) return notFoundResponse(normalizedSlug);
-      return differentialResponse({
-        record,
-        detailContext: getDifferentialDetailContext(record),
-        governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
-        publicAccess: true,
-      });
+      return differentialResponse(
+        {
+          record,
+          detailContext: getDifferentialDetailContext(record),
+          governance: { sourceStatus: governance.source_status, validationStatus: governance.validation_status },
+          publicAccess: true,
+        },
+        { request, fixture: true },
+      );
     }
 
     const fetchRecord = async () => {