perf: overlap query embedding with lexical retrieval + typeahead catalog cache (round 2, phase 5)#486
Conversation
…head catalogs Round-2 phase 5 (eval-gated; user-approved provider runs). - Speculative embedding: on the cold hybrid path the embedding round trip started only after every lexical layer had run, making it fully additive. It now starts alongside the lexical RPCs — but is consumed ONLY when expandedQuery is still byte-identical at the embed site, because the lexical layers can enrich the query with candidate metadata before embedding. Enriched queries take the serial path exactly as before, so retrieval results cannot change by construction. Cost: one wasted cached/coalesced embedding call when a lexical fast path wins (embedding_skipped_rate is ~0.72 on the golden set; embeddings are fractions of a cent). Never speculates for lexical-only (typeahead) or source-only retrieval. - Per-owner typeahead catalog cache (60s TTL): signed-in typeahead re-read the medications/services/forms catalogs from Supabase on every debounced keystroke; the mapped records are now cached per owner while ranking still runs per keystroke, so ranked output within the TTL is identical. - OPENAI_FAST_ANSWER_TIMEOUT_MS: optional tighter deadline for the fast answer tier only, so a hung fast generation can fall back / escalate before the full 30s answer timeout. Ships UNSET (behavior unchanged) — to be tuned in production against the Server-Timing/rag_queries generation-latency signal rather than local noise. Deliberately NOT done: trimming the per-keystroke lexical enrichment layers (ranking-metadata / memory-boost / visual-evidence) — memory boosting affects typeahead ORDERING, a real behavior change with poor risk/benefit after round 1 already removed the per-keystroke embedding. Eval gate (live Supabase + OpenAI, user-approved): - eval:retrieval baseline vs post-change: BIT-IDENTICAL correctness — 36 cases, document/content recall@5 = 1, mrr@10 0.8148, content_mrr@10 0.9279, identical retrieval strategy/plan/layer counts, failed_cases 0 (p90 latency 10407 -> 9658 ms). - eval:quality sanity (44 cases): citation failures 0, numeric grounding failures 0, governance danger failures 0, grounded-supported 0.9667, P95 16.7s. One case (clozapine-fbc-acronym-threshold) failed its 20s latency threshold on the strong route with correct retrieval — consistent with local-box latency noise; no generation behavior changes by default in this diff. Also: tsc, eslint, prettier, targeted vitest green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 901fae59ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const speculativeEmbedding = | ||
| sourceOnlyRetrieval || args.lexicalOnly ? null : embedTextWithTelemetry(speculativeEmbeddingQuery); |
There was a problem hiding this comment.
When an answer request is cancelled while the lexical Supabase work is still running, request.signal is checked after that work and the function exits, but this line has already started embedTextWithTelemetry() without any abort signal. The added .catch() only hides the rejection; it cannot cancel the OpenAI request, so abandoned answer/search requests can still spend embedding calls that the previous serial path would never have started. A focused abort test where the signal fires during searchTextChunkCandidates would catch this.
Useful? React with 👍 / 👎.
471099c
into
claude/perf-r2-plan-cache-migration
1 participant
Summary
Phase 5 of the round-2 performance initiative (stacked on #484). Retrieval-adjacent latency work, gated on the golden evals per
docs/capacity-review.md— provider runs were user-approved.expandedQuerywith candidate metadata before the embed site, so the speculative result is consumed only when the query is still byte-identical there; enriched queries take the serial path exactly as before. Results cannot change by construction. Cost: one wasted cached/coalesced embedding call when a lexical fast path wins (~72% of golden queries; fractions of a cent each). Never speculates for typeahead (lexical-only) or source-only retrieval.OPENAI_FAST_ANSWER_TIMEOUT_MS(optional): a tighter fast-tier-only deadline so a hung fast generation can fall back/escalate before the full 30s answer timeout. Ships unset — behavior unchanged. Tune in production against the Server-Timing /rag_queriesgeneration-latency signal (perf: bundle analyzer + Server-Timing measurement rail (perf round 2, phase 0) #458) rather than local noise.Deliberately not done: trimming the per-keystroke lexical enrichment layers (ranking-metadata / memory-boost / visual-evidence). Memory boosting affects typeahead ordering — a real behavior change with poor risk/benefit after round 1 already removed the per-keystroke embedding.
Clinical governance preflight
eval:retrieval(golden set, live) — baseline vs post-change BIT-IDENTICAL correctness: 36 cases, document/content recall@5 = 1, mrr@10 0.8148, content_mrr@10 0.9279, identical retrieval strategy/plan/layer counts,failed_cases=0(p90 latency 10,407 → 9,658 ms)eval:qualitysanity (44 cases, live): citation failures 0, numeric grounding failures 0, governance danger failures 0, grounded-supported 0.9667, P95 16.7s. One case (clozapine-fbc-acronym-threshold) failed its 20s latency threshold on the strong route with correct retrieval ("missing: none") — consistent with local-box latency noise; no generation behavior changes by default in this difftsc/eslint/prettier/ targeted vitest green🤖 Generated with Claude Code