Skip to content

perf(db): optimize scoped retrieval queries #714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 2 commits into from
Jul 17, 2026
Merged

perf(db): optimize scoped retrieval queries #714

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9f3cb96
perf(db): optimize scoped retrieval queries
BigSimmo Jul 17, 2026
1ef0fae
Merge remote-tracking branch 'origin/main' into codex/chat-supabase-m…
BigSimmo Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/branch-review-ledger.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,7 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
| 2026-07-17 | codex/pwa-privacy-safe-20260717 | 35fa8c929d44a9bd84b3f7f2b795354d3b6dae02 | privacy-safe PWA shell and merge-readiness review | No remaining high-confidence product defect in the changed scope. The pre-push browser gate found and fixed one P2 test defect: cleanup referenced `PWA_CACHE_PREFIX` without passing it into the browser context, and the cold installability flow now has a focused 120-second budget. The worker caches only the generic offline page and allow-listed public shell assets; navigations, APIs, auth, queries, documents, uploads, signed URLs, range requests, and cross-origin traffic remain network-only. | Current-main integration; focused Vitest 81/81; full uncached ESLint; TypeScript; scoped Prettier and diff checks; production Webpack build generated 1,043 pages and the client-bundle secret scan passed; full Vitest produced 2,506 passes plus six contention timeouts, with all affected files passing 24/24 serially; focused Chromium PWA 2/2. No Supabase/OpenAI/live-provider checks run. |
| 2026-07-17 | codex/historical-branch-cleanup-20260717 | e36ac0c6628264c7ed6c494a597a62d0214b68f6 | branch-cleanup and historical-content recovery | Completed the pending historical cleanup: deleted 55 exact-SHA remote refs and 20 redundant local refs, removed nine clean merged worktrees, preserved every dirty, active, open-PR, or patch-unique worktree, and recovered the still-useful governance incident runbook from `codex/domain-1-governance-remediation`. Historical code changes were either tied to merged PRs or reviewed as superseded by current implementations; open PRs #699, #700, #702, and #704 remain protected. | Fresh `git fetch --prune`; GitHub PR inventory and exact commit-to-merged-PR associations; exact remote SHA rechecks before deletion; cherry-pick-aware logs; two-dot tree and branch-only-file review; Codex task-to-worktree cross-check; focused documentation validation recorded in the cleanup PR. No OpenAI, Supabase, production-data, or live clinical workflow was run. |
| 2026-07-17 | PR #704 / codex/scroll-geometry-stability-20260717 | 35e74ddbd61bacc5b34f06efbd58091f092665fd | nested scroll-source review follow-up | Confirmed the outside-diff CodeRabbit finding: the standalone shell shared one intent history across main and descendant scroll containers, so a switch from a deep main offset to a near-zero nested offset could falsely reveal chrome. Scroll metrics now identify their source, source changes rebase direction and travel while preserving visibility, and unit/UI regressions cover the switch. No unresolved actionable review finding remains. | Focused Vitest 9/9; TypeScript; scoped ESLint; Prettier; `git diff --check`. Exact-head hosted CI and UI remain required after push. No Supabase/OpenAI/live-provider checks run. |
| 2026-07-17 | codex/chat-supabase-migration-preflight-b463 | f7c4e293ef35acc54f2b82bbccb2990d51289d5c | live production Supabase security, integrity, drift, and performance review plus remediation | Resolved and deployed the P2 retrieval-performance issue with `20260717160000_optimize_owner_public_retrieval`: owner/public filtering now happens in one scoped query and index-unit text/term candidates use separate GIN-friendly branches. Warm text retrieval improved from 1.269 seconds to 34 ms; warm index-unit retrieval completed in 36-39 ms (first cold run 2.376 seconds with 2,009 physical reads). No P0/P1 security, privacy, RLS, privilege, storage, migration-history, or integrity issue was found. | Isolated Docker schema replay; pre-deploy drift showed exactly four intended function changes; linked production push; post-deploy `No unexpected schema drift`; exact project and migration-history checks; security and performance advisors; live access-scope parity; bounded `EXPLAIN ANALYZE`; post-migration logs; focused Vitest 74/74; offline RAG fixtures 36/36 and contract tests 291/291; ESLint; TypeScript; function grants; production readiness. Full `verify:cheap` reached the 10-minute host timeout during broad Vitest and ended with EPIPE; focused and domain checks passed. No OpenAI calls, write load test, or backup/PITR restore test was performed. |
| 2026-07-17 | codex/chat-forms-import-6914 | e5caaa46cad5fb9a937f1dc43312723799b98abb + working-tree diff | shared Forms/Services catalogue access and LOCAL_NO_AUTH_OWNER_ID review | Fixed one P1 availability/design defect: authenticated reads materialized a private copy of the shared catalogue on first access, creating drift, unnecessary writes, and possible registry-corpus side effects. Forms/Services now merge the reviewed shared baseline with private owner overrides for list, detail, and universal search; older partial overrides retain missing shared metadata; no registry GET seeds or embeds. Private rows and linked documents remain owner-scoped. The ignored local owner setting was corrected to the verified sole live-owner UUID and source validation now requires a UUID. No remaining high-confidence defect in scope. | Focused registry/universal Vitest initially exposed four local expectation/count mismatches; corrected registry suite passed 17/17 and registry/logging suite passed 23/23. Full TypeScript passed. `verify:cheap` passed runtime, action pins, sitemap, brand, type/icon/function guards, full lint, and TypeScript; full Vitest reached 2,588 passing with one stale logging-guard failure, which was fixed and focused-verified. The final full-suite rerun was terminated by the 5-minute host timeout without a reported assertion failure. `git diff --check` passed. No Supabase/OpenAI/provider call or schema/RLS mutation was run for this review. |
| 2026-07-17 | codex/scrolling-cleanup-20260717 | ff77cd06c + latest origin/main sync | cross-page scrolling and interaction stability review | Fixed two confirmed P2 defects: desktop action-popup placement performed synchronous geometry work for every captured scroll event, and submitted differential searches with zero document matches fell back to the home state. Placement is now coalesced per animation frame with passive scroll listeners, and submitted empty-evidence results remain visible. Hardened three popup/navigation browser helpers that reproduced hydration timing failures. No other high-confidence defect remains in the scoped diff. | Scroll-focused Chromium 28/28; final affected Chromium 5/5; source regressions 2/2; scoped ESLint; Prettier; TypeScript and production build passed before the final upstream-only sync; `git diff --check`. The aggregate local Vitest/UI runs were affected by concurrent-worktree resource contention, so exact-head hosted CI remains required before merge. No Supabase/OpenAI/live-provider checks run. |
| 2026-07-17 | PR #713 / codex/chat-workflow-ideas-0916 | b52112df6aa36311d7420189064acd79dcf2c3f5 + reviewed follow-up diff | workflow toolkit review follow-up | Fixed all 14 actionable Codex and CodeRabbit threads: cross-platform path fixtures, installation-managed preflight guidance, complete Supabase-backed API database scoping, per-command approval boundaries, plugin-ignore narrowing, isolated CI-scope proof, remote-Git command guarding, repository-skill verification classification, `TypeError` diagnosis, strict CLI option values, machine-parseable JSON evidence output, and preservation of baseline database/clinical approval gates in the RAG lab. No unresolved actionable finding remains in the reviewed scope. | `npm run verify:cheap` passed with 273 files and 2,599 tests; focused toolkit Vitest 20/20; CI-scope self-test; plugin-ignore proof; `git diff --check`. Exact-head hosted CI remains required after the follow-up push. No Supabase, OpenAI, or other live product-provider command was run. |
22 changes: 12 additions & 10 deletions scripts/generate-drift-manifest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,12 @@ import { fileURLToPath } from "node:url";
* Flags:
* --keep leave the container running (for inspection / DR rehearsal)
* --port <n> host port for the scratch Postgres (default 56599)
* --container <name> override the scratch container name for concurrent worktrees
* --image <x> override the Postgres image tag
*/

const IMAGE_DEFAULT = "supabase/postgres:17.6.1.127";
const CONTAINER = "clinical-kb-drift-manifest";
const CONTAINER_DEFAULT = "clinical-kb-drift-manifest";

const repoUrl = (relative: string) => fileURLToPath(new URL(`../${relative}`, import.meta.url));

Expand All @@ -43,6 +44,7 @@ function docker(args: string[], input?: string): string {
async function main() {
const image = arg("--image") ?? IMAGE_DEFAULT;
const port = arg("--port") ?? "56599";
const container = arg("--container") ?? CONTAINER_DEFAULT;
const keep = process.argv.includes("--keep");

try {
Expand All @@ -57,15 +59,15 @@ async function main() {
const scaffoldSql = readFileSync(repoUrl("scripts/sql/drift-replay-scaffold.sql"), "utf8");
const { normalizedSchemaSha256 } = await import("./check-drift");

console.log(`Starting scratch container ${CONTAINER} (${image}) on port ${port}…`);
console.log(`Starting scratch container ${container} (${image}) on port ${port}…`);
try {
docker(["rm", "-f", CONTAINER]);
docker(["rm", "-f", container]);
} catch {
// not running — fine
}
const startedAt = Date.now();
const scratchPassword = randomBytes(16).toString("hex");
docker(["run", "-d", "--name", CONTAINER, "-e", `POSTGRES_PASSWORD=${scratchPassword}`, "-p", `${port}:5432`, image]);
docker(["run", "-d", "--name", container, "-e", `POSTGRES_PASSWORD=${scratchPassword}`, "-p", `${port}:5432`, image]);
Comment on lines +62 to +70

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Do not forcibly remove an arbitrary --container target.

A typo or reused name currently executes docker rm -f against an unrelated container. Let docker run fail on collision; cleanup will then only run after this invocation successfully starts its own container.

Proposed fix 
   console.log(`Starting scratch container ${container} (${image}) on port ${port}…`);
-  try {
-    docker(["rm", "-f", container]);
-  } catch {
-    // not running — fine
-  }
   const startedAt = Date.now();

As per coding guidelines, “do not kill or modify another project's server.”

Also applies to: 124-129

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import { execFileSync } from "node:child_process";
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(detect-child-process-typescript)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/generate-drift-manifest.ts` around lines 62 - 70, Remove the
preemptive docker(["rm", "-f", container]) cleanup around the scratch-container
startup, including the corresponding try/catch near the cleanup path. Let
docker(["run", ...]) fail naturally when the requested container name is already
in use, and ensure later cleanup executes only after this invocation
successfully starts its own container.

Source: Coding guidelines


try {
// The Supabase image briefly accepts connections before its init migrations
Expand All @@ -74,7 +76,7 @@ async function main() {
let consecutiveReadyChecks = 0;
for (let attempt = 0; attempt < 60; attempt += 1) {
try {
docker(["exec", CONTAINER, "pg_isready", "-U", "postgres", "-q"]);
docker(["exec", container, "pg_isready", "-U", "postgres", "-q"]);
consecutiveReadyChecks += 1;
if (consecutiveReadyChecks >= 5) break;
} catch {
Expand All @@ -88,7 +90,7 @@ async function main() {

const psql = (user: string, sql: string) =>
docker(
["exec", "-i", CONTAINER, "psql", "-U", user, "-d", "postgres", "-v", "ON_ERROR_STOP=1", "-q", "-f", "-"],
["exec", "-i", container, "psql", "-U", user, "-d", "postgres", "-v", "ON_ERROR_STOP=1", "-q", "-f", "-"],
sql,
);

Expand All @@ -100,7 +102,7 @@ async function main() {
console.log(`Replay complete in ${replaySeconds}s (container start included).`);

const raw = docker(
["exec", "-i", CONTAINER, "psql", "-U", "postgres", "-d", "postgres", "-tA", "-v", "ON_ERROR_STOP=1", "-f", "-"],
["exec", "-i", container, "psql", "-U", "postgres", "-d", "postgres", "-tA", "-v", "ON_ERROR_STOP=1", "-f", "-"],
"select public.schema_drift_snapshot()::text;",
).trim();
const snapshot = JSON.parse(raw) as Record<string, unknown>;
Expand All @@ -119,12 +121,12 @@ async function main() {
console.log("Next: run `npm run check:drift` against live (needs service-role env).");
} finally {
if (keep) {
console.log(`Container ${CONTAINER} kept running on port ${port} (--keep).`);
console.log(`Container ${container} kept running on port ${port} (--keep).`);
} else {
try {
docker(["rm", "-f", CONTAINER]);
docker(["rm", "-f", container]);
} catch {
console.warn(`Could not remove container ${CONTAINER}; remove it manually.`);
console.warn(`Could not remove container ${container}; remove it manually.`);
}
}
}
Expand Down
26 changes: 21 additions & 5 deletions supabase/drift-manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"generated_at": "2026-07-17T06:06:10.141Z",
"generated_at": "2026-07-17T11:30:04.370Z",
"generator": "scripts/generate-drift-manifest.ts",
"postgres_image": "supabase/postgres:17.6.1.127",
"schema_sha256": "b9faccf4dac098def5d9547ca1b4cf0bb24621f1b76e9aed1b57d2495b65d72b",
"replay_seconds": 60,
"schema_sha256": "59ee18d0723774416b931bea9f4fef96fa699eb99ba61d218b6870a621dde649",
"replay_seconds": 45,
"snapshot": {
"views": [
{
Expand Down Expand Up @@ -6481,7 +6481,15 @@
"postgres=X/postgres",
"service_role=X/postgres"
],
"def_hash": "698d24f90e3fa5f17ef5898e3ba2d8ff",
"def_hash": "2b890b6f6c617f78f41f06293c9f7ade",
"signature": "public.match_document_chunks_text_scoped(text,integer,uuid[],uuid,boolean)"
},
{
"acl": [
"postgres=X/postgres",
"service_role=X/postgres"
],
"def_hash": "4bf81d62c0953a8d11c80c37de6045e7",
"signature": "public.match_document_chunks_text_v2(text,integer,uuid[],uuid,boolean)"
},
{
Expand Down Expand Up @@ -6537,7 +6545,15 @@
"postgres=X/postgres",
"service_role=X/postgres"
],
"def_hash": "7e7e5a83bad724a62fdc66a0213b843b",
"def_hash": "e802785a1872f92ca4b748c08c57435d",
"signature": "public.match_document_index_units_hybrid_scoped(extensions.vector,text,integer,double precision,uuid[],uuid,boolean)"
},
{
"acl": [
"postgres=X/postgres",
"service_role=X/postgres"
],
"def_hash": "38282636c4c22a4f01974a943aa20d42",
"signature": "public.match_document_index_units_hybrid_v2(extensions.vector,text,integer,double precision,uuid[],uuid,boolean)"
},
{
Expand Down
Loading