Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ SUPABASE_IMAGE_BUCKET=clinical-images
# Conservative local-first defaults. Raise only after testing your worker machine,
# Supabase plan limits, and confidentiality policy.
MAX_UPLOAD_MB=150
# Next Proxy has a fixed 151 MiB transport envelope (150 MiB file plus
# multipart framing), so values above 150 are intentionally rejected.
MAX_IMPORT_JOBS_PER_RUN=5
MAX_IMPORT_BYTES_PER_RUN=157286400
CHUNK_SIZE=2000
Expand Down
9 changes: 8 additions & 1 deletion .github/workflows/eval-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,14 @@ jobs:
run: npm run eval:retrieval:quality -- --fail-on-threshold

- name: Answer-quality subset (live generation)
run: npm run eval:quality -- --rag-only --limit ${{ github.event.inputs.answer_case_limit || '8' }} --fail-on-threshold
env:
ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }}
run: |
if [[ ! "$ANSWER_CASE_LIMIT" =~ ^[0-9]+$ ]] || (( ANSWER_CASE_LIMIT < 1 || ANSWER_CASE_LIMIT > 100 )); then
echo "answer_case_limit must be an integer from 1 to 100" >&2
exit 2
fi
npm run eval:quality -- --rag-only --limit "$ANSWER_CASE_LIMIT" --fail-on-threshold

- name: Open or update regression issue
if: failure() && github.event_name == 'schedule'
Expand Down
62 changes: 32 additions & 30 deletions docs/branch-review-ledger.md

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions docs/deployment-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,11 @@ comparable (~200 ms) from Singapore or Sydney and does not favour either host.
`/api/health`.
- No secret is ever baked into a layer. `SUPABASE_SERVICE_ROLE_KEY`,
`OPENAI_API_KEY`, etc. are injected at run time by Railway's variable store.
- Request bodies are bounded twice: Next Proxy buffers at most 151 MiB, and
`/api/upload` rejects declared multipart bodies above `MAX_UPLOAD_MB` plus
1 MiB framing overhead before authentication or `request.formData()`. Keep
the managed host's request-body limit at 151 MiB or lower as a third ingress
fence; `MAX_UPLOAD_MB` is capped at 150 MiB by environment validation.

### Config as code (`railway.app.json`)

Expand Down
8 changes: 6 additions & 2 deletions docs/forward-codify-retrieval-rpcs-workorder.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# Work-order — forward-codify live-ahead retrieval RPC bodies (drift backlog #0)

**Status: staged, not executed.** This is plan task 1.2. It is deliberately **not** shipped as a
migration yet because the two safety preconditions were unavailable at authoring time (2026-07-12):
**Status: reconciled in source on 2026-07-13; live apply remains pending explicit write approval.**
The production-current definitions were captured read-only, preserved under their actual migration
versions, replayed successfully in scratch PostgreSQL, and removed from the drift allowlist. The
four forward remediation migrations remain unapplied to live.

Historical blockers at authoring time (2026-07-12):

1. **Byte-faithful Docker-replay validation is required and Docker was down.** The established method
(see [supabase-migration-reconciliation.md](supabase-migration-reconciliation.md) and the
Expand Down
5 changes: 5 additions & 0 deletions next.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ const nextConfig: NextConfig = {
experimental: {
cpus: 1,
optimizePackageImports: ["lucide-react"],
// Proxy is on every API route. Bound its buffered client body so a
// chunked multipart upload cannot grow without limit before route code
// reaches request.formData(). MAX_UPLOAD_MB is capped at 150 below this
// 151 MiB transport envelope (1 MiB reserved for multipart framing).
proxyClientMaxBodySize: "151mb",
},
poweredByHeader: false,
turbopack: {
Expand Down
20 changes: 18 additions & 2 deletions scripts/check-drift.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { readFileSync, writeFileSync } from "node:fs";
import { loadEnvConfig } from "@next/env";

loadEnvConfig(process.cwd());
Expand Down Expand Up @@ -229,7 +229,9 @@ async function main() {
postgres_image: string;
snapshot: Snapshot;
};
const allowlistFile = JSON.parse(read("supabase/drift-allowlist.json")) as { entries: AllowlistEntry[] };
const allowlistFile = JSON.parse(read("supabase/drift-allowlist.json")) as Record<string, unknown> & {
entries: AllowlistEntry[];
};
const allowlist = allowlistFile.entries ?? [];

const schemaSha = normalizedSchemaSha256(read("supabase/schema.sql"));
Expand Down Expand Up @@ -261,6 +263,20 @@ async function main() {

const { findings: remaining, allowed, staleEntries, infos } = compareDriftSnapshots(expected, live, allowlist);

if (process.argv.includes("--prune-stale") && remaining.length === 0 && staleEntries.length > 0) {
const staleSet = new Set(staleEntries);
const nextAllowlist = {
...allowlistFile,
entries: allowlist.filter((entry) => !staleSet.has(entry)),
};
writeFileSync(
new URL("../supabase/drift-allowlist.json", import.meta.url),
`${JSON.stringify(nextAllowlist, null, 2)}\n`,
"utf8",
);
console.log(`Pruned ${staleEntries.length} stale allowlist entr${staleEntries.length === 1 ? "y" : "ies"}.`);
}

console.log(`Drift manifest: generated ${manifestFile.generated_at} from schema.sql ${schemaSha.slice(0, 12)}…`);
console.log(
`Compared ${Object.keys(categoryKeys)
Expand Down
42 changes: 13 additions & 29 deletions scripts/sql/capture-live-retrieval-rpcs.sql
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
-- capture-live-retrieval-rpcs.sql (READ-ONLY)
--
-- Captures the VERBATIM live bodies of the retrieval RPCs that are still flagged
-- "LIVE IS AHEAD" in supabase/drift-allowlist.json, so their definitions can be
-- Captures the VERBATIM live bodies of retrieval RPCs still flagged
-- "LIVE IS AHEAD" in supabase/drift-allowlist.json. The target set is currently
-- empty because the 20260712 live-ahead definitions have been forward-codified.
-- If drift reappears, add only the newly allowlisted signatures between the
-- guard markers below so the test and capture query stay synchronized.
--
-- Definitions are forward-codified into a migration + supabase/schema.sql without
-- forward-codified into a migration + supabase/schema.sql WITHOUT hand-authoring
-- them. These bodies are complex, have diverged in both directions (e.g. the
-- fail-closed retrieval_owner_matches predicate was applied to live while live
Expand All @@ -27,11 +32,7 @@
-- pg_proc. Run it via the Supabase Dashboard SQL editor, an approved
-- service-role SQL path, or the Supabase MCP execute_sql tool.
--
-- Expect EXACTLY 7 rows back (one per target). A target that no longer resolves
-- on live (renamed, dropped, or already reconciled) is filtered out and simply
-- yields FEWER rows — the query does not error. If you get fewer than 7, find
-- which with the "missing signatures" snippet in the work-order and reconcile
-- the target set + allowlist before continuing.
-- Expect zero rows while the drift allowlist has no LIVE IS AHEAD targets.
--
-- search_path is pinned to '' so oid::regprocedure renders fully-qualified and
-- matches the allowlist keys / schema_drift_snapshot() signatures exactly, and
Expand All @@ -43,25 +44,8 @@

set search_path = '';

select
r.signature,
pg_get_functiondef(r.proc) as definition
from (
select t.signature, to_regprocedure(t.signature) as proc
from (
-- >>> forward-codify targets >>> (kept in lockstep with the "LIVE IS AHEAD"
-- retrieval entries in supabase/drift-allowlist.json — see
-- tests/forward-codify-retrieval-targets.test.ts)
values
('public.get_related_document_metadata(uuid[],uuid)'),
('public.match_document_chunks(extensions.vector,integer,double precision,uuid,uuid)'),
('public.match_document_chunks_hybrid(extensions.vector,text,integer,double precision,uuid[],uuid)'),
('public.match_document_chunks_text(text,integer,uuid[],uuid)'),
('public.match_document_table_facts_text(text,integer,uuid[],uuid)'),
('public.match_documents_for_query(text,integer,uuid)'),
('public.repair_strict_enrichment_gate_batch(integer)')
-- <<< forward-codify targets <<<
) as t(signature)
) as r
where r.proc is not null
order by r.signature;
-- >>> forward-codify targets >>> (kept in lockstep with the LIVE IS AHEAD
-- retrieval entries in supabase/drift-allowlist.json; currently empty)
-- <<< forward-codify targets <<<
select null::text as signature, null::text as definition
where false;
14 changes: 14 additions & 0 deletions src/app/api/answer/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,20 @@ export async function POST(request: Request) {
// (refused) sources/smartPanel/smartApiPlan would still reach the client and
// defeat the refusal. Keep only the safe "unsupported" contract fields, matching
// the empty-scope branch above.
void logAnswerDiagnostics({
supabase,
query: answerBody.query,
ownerId: access.ownerId,
answer: {
...answer,
grounded: false,
confidence: "unsupported",
sources: [],
responseMode: "evidence_gap",
fallbackReason: "source_governance_refusal",
routingReason: [answer.routingReason, "source_governance_refusal"].filter(Boolean).join("; "),
},
});
return NextResponse.json({
answer: sourceGovernanceRefusalAnswer,
grounded: false,
Expand Down
16 changes: 16 additions & 0 deletions src/app/api/answer/stream/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,22 @@ function streamAnswer(body: AnswerBody, ownerId?: string, signal?: AbortSignal,
if (shouldUseSourceGovernanceRefusal && hasDangerSourceGovernanceWarning(warnings)) {
// Explicit refusal payload — do not spread ...answer (see /api/answer):
// the refused sources/smartPanel/smartApiPlan must not reach the client.
if (!isDemoMode()) {
void logAnswerDiagnostics({
supabase: createAdminClient(),
query: body.query,
ownerId,
answer: {
...answer,
grounded: false,
confidence: "unsupported",
sources: [],
responseMode: "evidence_gap",
fallbackReason: "source_governance_refusal",
routingReason: [answer.routingReason, "source_governance_refusal"].filter(Boolean).join("; "),
},
});
}
send("final", {
answer: sourceGovernanceRefusalAnswer,
grounded: false,
Expand Down
142 changes: 7 additions & 135 deletions src/app/api/documents/[id]/reindex/route.ts
Original file line number Diff line number Diff line change
@@ -1,33 +1,24 @@
import { NextResponse } from "next/server";
import type { SupabaseClient } from "@supabase/supabase-js";
import { z } from "zod";
import { env, isDemoMode } from "@/lib/env";
import { upsertDocumentEnrichment } from "@/lib/document-enrichment";
import { upsertDocumentDeepMemory } from "@/lib/deep-memory";
import { jsonError } from "@/lib/http";
import {
activeIngestionJobColumns,
buildActiveJobsSafetyResult,
checkIngestionMutationSafety,
hasActiveAgentEnrichmentJob,
ingestionMutationSafetyPayload,
ingestionRollbackFenceStamp,
type IngestionJobRow,
} from "@/lib/ingestion-mutation-safety";
import { consumeApiRateLimit, rateLimitJsonResponse } from "@/lib/api-rate-limit";
import {
committedIndexGeneration,
isAtomicReindexCandidate,
isCommittedGenerationMetadata,
} from "@/lib/reindex-pipeline";
import { isAtomicReindexCandidate } from "@/lib/reindex-pipeline";
import { createAdminClient } from "@/lib/supabase/admin";
import { AuthenticationError, requireAuthenticatedUser, unauthorizedResponse } from "@/lib/supabase/auth";
import { parseJsonBodyOrDefault } from "@/lib/validation/body";
import { parseRouteParams } from "@/lib/validation/params";

export const runtime = "nodejs";

const reindexPageSize = 1000;
const reindexModeSchema = z
.object({
mode: z.preprocess((value) => (value === "enrichment" ? "enrichment" : "full"), z.enum(["full", "enrichment"])),
Expand All @@ -37,73 +28,11 @@ const reindexRouteParamsSchema = z.object({
id: z.string().uuid(),
});

type ReindexChunk = {
id: string;
document_id: string;
page_number: number | null;
chunk_index: number;
section_heading: string | null;
content: string;
image_ids?: string[] | null;
metadata?: Record<string, unknown> | null;
};

type ReindexImage = {
id: string;
page_number: number | null;
caption: string | null;
image_type: string | null;
labels?: string[] | null;
source_kind?: string | null;
clinical_relevance_score?: number | null;
metadata?: Record<string, unknown> | null;
};

function committedReindexRows<T extends { metadata?: unknown }>(document: { metadata?: unknown }, rows: T[]) {
const committedGeneration = committedIndexGeneration(document.metadata);
return rows.filter((row) =>
isCommittedGenerationMetadata({
rowMetadata: row.metadata,
committedGeneration,
}),
);
}

async function readMode(request: Request) {
const parsed = await parseJsonBodyOrDefault(request, reindexModeSchema, { mode: "full" });
return parsed.mode;
}

async function selectReindexRowsInPages<T>(args: {
supabase: ReturnType<typeof createAdminClient>;
table: "document_chunks" | "document_images";
select: string;
documentId: string;
searchableOnly?: boolean;
}) {
const rows: T[] = [];
if (args.searchableOnly && args.table !== "document_images") {
throw new Error("searchableOnly reindex paging only supports the document_images table.");
}
for (let offset = 0; ; offset += reindexPageSize) {
const dynamicSupabase = args.supabase as unknown as SupabaseClient;
const query = args.searchableOnly
? dynamicSupabase
.from("document_images")
.select(args.select)
.eq("document_id", args.documentId)
.eq("searchable", true)
: dynamicSupabase.from(args.table).select(args.select).eq("document_id", args.documentId);
const { data, error } = await query.range(offset, offset + reindexPageSize - 1);
if (error) throw new Error(error.message);

const page = (data ?? []) as T[];
rows.push(...page);
if (page.length < reindexPageSize) break;
}
return rows;
}

export async function POST(request: Request, { params }: { params: Promise<{ id: string }> }) {
try {
if (isDemoMode()) return NextResponse.json({ error: "Reindex is unavailable in demo mode." }, { status: 400 });
Expand Down Expand Up @@ -139,74 +68,17 @@ export async function POST(request: Request, { params }: { params: Promise<{ id:
if (!safety.ok) return NextResponse.json(ingestionMutationSafetyPayload(safety), { status: safety.status });

if (mode === "enrichment") {
// Audit R24d: route-mode enrichment and the enrichment agent both
// delete-then-insert the same artifact families with no shared lock. If
// the agent is mid-pass, the interleaved deletes can strand a
// "completed/good" document with zero enrichment artifacts (no repair
// path exists). Refuse to run while a live agent pass holds the document.
const agentBusy = await hasActiveAgentEnrichmentJob({
supabase,
documentId: id,
staleAfterMinutes: env.WORKER_STALE_AFTER_MINUTES,
const { data: queued, error: queueError } = await supabase.rpc("request_indexing_v3_enrichment", {
p_document_id: id,
p_owner_id: user.id,
});
if (agentBusy) {
if (queueError) {
return NextResponse.json(
{
error:
"The enrichment agent is currently processing this document. Wait for it to finish before re-running enrichment.",
},
{ error: "Enrichment is already active or could not be queued safely." },
{ status: 409 },
);
}

const [chunks, images] = await Promise.all([
selectReindexRowsInPages<ReindexChunk>({
supabase,
table: "document_chunks",
select: "id,document_id,page_number,chunk_index,section_heading,content,image_ids,metadata",
documentId: id,
}),
selectReindexRowsInPages<ReindexImage>({
supabase,
table: "document_images",
select: "id,page_number,caption,image_type,labels,source_kind,clinical_relevance_score,metadata",
documentId: id,
searchableOnly: true,
}),
]);

const committedChunks = committedReindexRows(document, chunks);
const committedImages = committedReindexRows(document, images);

committedChunks.sort((a, b) => Number(a.chunk_index ?? 0) - Number(b.chunk_index ?? 0));
committedImages.sort((a, b) => Number(b.clinical_relevance_score ?? 0) - Number(a.clinical_relevance_score ?? 0));

if (!committedChunks.length) {
return NextResponse.json({ error: "Document has no indexed chunks to enrich." }, { status: 400 });
}

const enrichment = await upsertDocumentEnrichment({
supabase,
document: document as Parameters<typeof upsertDocumentEnrichment>[0]["document"],
chunks: committedChunks,
images: committedImages,
});
const deepMemory = await upsertDocumentDeepMemory({
supabase,
document: document as Parameters<typeof upsertDocumentDeepMemory>[0]["document"],
chunks: committedChunks,
images: committedImages,
summary: enrichment.summary.summary,
});
return NextResponse.json({
mode,
enrichment,
deepMemory: {
sectionCount: deepMemory.sections.length,
memoryCardCount: deepMemory.memoryCards.length,
indexUnitCount: deepMemory.indexUnits.length,
},
});
return NextResponse.json({ mode, queued }, { status: 202 });
}

const atomicReindex = isAtomicReindexCandidate(document);
Expand Down
Loading
Loading