test(eval): harden force-embedding flag and vector-path eval guards#264
Closed
BigSimmo wants to merge 2 commits into
Closed
test(eval): harden force-embedding flag and vector-path eval guards#264BigSimmo wants to merge 2 commits into
BigSimmo wants to merge 2 commits into
Conversation
The golden retrieval set was 100% lexical fast-path (embedding_skipped_rate=1.0), so it could not measure whether a re-index changes vector/embedding retrieval quality. - forceEmbedding option on searchChunksWithTelemetry (SearchChunksArgs): bypasses every lexical text-fast-path so retrieval always exercises the embedding/vector stage. Diagnostic/eval-only; folded into the search cache key; never set on production paths. - eval-retrieval.ts: per-case `forceEmbedding` field + a global `--force-embedding` flag. - 10 `vector-*` cases (psychiatric monographs: PTSD, OCD, panic, anorexia, GAD, Tourette, postnatal, bipolar, ADHD, opioid) with forceEmbedding=true. Each is a clinical query that must be answered by vector retrieval of the right monograph — verified live at document_recall@5=1.0, content_recall@5=1.0, all via strategy=hybrid (embedding used). Rationale: forcing embedding is the correct instrument for re-index measurement — you want to measure the vector index directly, not have a lexical shortcut mask a regression. Wording alone can't reliably force the vector path (the fast-path is driven by emergent lexical-match strength), so the flag makes these probes deterministic. Live golden eval: 34/34 pass (24 existing + 10 new), no regression. verify:cheap green (980). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire forceEmbedding through eval runners and retrieval cache keys, bypass coverage/lexical shortcuts when forced, and add golden-case failure metrics so vector regressions cannot hide behind text-fast-path or cache hits.
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
3 tasks
Owner
Author
|
Superseded by #269 (consolidated platform fixes branch). |
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
forceEmbeddingthrough eval runners, retrieval cache keys, and RAG search paths.forceEmbeddingis set.Verification
npm run test -- tests/eval-quality.test.ts tests/retrieval-query-variants.test.tsnpm run verify:cheapnpm run eval:retrieval:quality(retrieval/ranking/scoring behavior changed)npm run check:production-readinessClinical Governance Preflight
forceEmbeddingtest flag