Fix all Critical & High RAG pipeline issues (indexing, retrieval, generation)#38
Merged
Merged
Conversation
IDX-C1: reassemble embeddings by response item.index, not array order, so texts never get the wrong vector when OpenAI returns items out of order. IDX-C2: pass EMBEDDING_DIMENSIONS to the embeddings API, guard each returned vector against the configured dimension, add env + worker startup probe so a model/dimension mismatch fails loud instead of writing bad vectors. IDX-C3: retry route refuses (409) to re-queue a job a live worker still holds (status=processing + fresh lock), preventing two workers interleaving mixed-generation clinical chunks for one document. IDX-H1: stop reset-then-enqueue in retry/reindex/bulk-reindex; only re-queue so a previously-good index stays live until the worker commits a fresh one. IDX-H2: compute deep-memory embeddings before deleting existing cards so a failed embed never leaves a document with fewer cards than before. IDX-H3: JS PDF fallback marks image-only/low-text pages needs_ocr and the worker fails loud rather than marking an image-only PDF "indexed". IDX-H4: OCR triggers on text density / image coverage, not a flat char floor. IDX-H5: paragraph-path chunking now carries CHUNK_OVERLAP across boundaries. IDX-H6: tables are chunked atomically on row boundaries with the header repeated per chunk, never severing dose/threshold values from columns. Adds focused tests: embed-texts-integrity, chunking table/overlap cases, retry-guard route tests. typecheck/lint/test all pass (350 tests). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ET-C1,C2,H1..H5) RET-C1: include topK/minSimilarity in the shared rag_response_cache key so searches with different depth/threshold no longer collide on a stale entry. RET-C2: stop fabricating a cosine similarity for text-only retrieval. The text RPC now returns real lexical_score in its own column, leaves similarity at 0, and caps hybrid_score below the moderate threshold so keyword-only hits cannot masquerade as strong semantic evidence. New migration mirrors schema.sql. RET-H1: clamp the clinical composite score to [0,1]. RET-H2: cap stacked ranking penalties and exempt passages that hold numeric or table-backed dose/threshold evidence, so the passage with the actual dose is never demoted below drug-name boilerplate. RET-H3: bound the signed-url cache with LRU eviction, strict expiry, an expiry skew window, and a finite default TTL for payloads missing expiresAt. RET-H4: gate raw query persistence behind RAG_PERSIST_RAW_QUERY_TEXT (default off), store normalized text + a query hash for PHI safety, and carry owner_id on search interaction logging. RET-H5: new migration tokenizes per-document viewer search so any significant token can match instead of requiring the whole query as one trigram/LIKE unit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…(GEN-C1..C3,H1..H3) GEN-C1: Detect silently-truncated OpenAI responses (status "incomplete" / incomplete_details.reason "max_output_tokens"); surface a `truncated` flag, record the reason in log metadata, downgrade confidence, and append a "verify against source" caveat. Raise OPENAI_MAX_OUTPUT_TOKENS default. GEN-C2 / GEN-H2: Add shared numeric-faithfulness utility (src/lib/answer-verification.ts). Verify every dose/threshold/numeric token in the generated answer appears in a cited chunk; flag unverified tokens, downgrade confidence, and add a "verify against the source" gap. GEN-C3: When the model cites nothing, mark grounded:false / confidence:"unsupported" with routingReason "ungrounded_no_model_citation" instead of back-filling all retrieved chunks as fake citations. GEN-H1: Wrap each source excerpt in escaped fences the model is told never to treat as instructions; neutralize instruction-like control phrases and role reassignment in source content. GEN-H3: Make clinical table normalization conservative — preserve the raw grid and flag low-confidence (rendered with a "verify against source" note) when dose/threshold cell pairing is ambiguous, instead of risking a mis-merge. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
BigSimmo
added a commit
that referenced
this pull request
Jun 19, 2026
…(B1-B6)
B1: numeric verification now matches by exact normalized-token set membership
instead of substring, so "2.5 mg" no longer verifies against "12.5 mg"
(a 5x dose error). Source tokens are extracted with the same extractor.
B2: fold unicode superscripts (×10⁹/L) to ASCII and try the scientific-notation
branch first so ANC/WBC thresholds extract whole and match ASCII sources.
B3: remove the impossible trailing \b after % so percentages extract and a
percentage mismatch is flagged; fix the misleading doc comment.
B4: numeric gate now verifies answerSections[].body (e.g. medication_dose),
scoping each section to its own citation_chunk_ids when present.
B5: safeFallbackAnswer fails closed on parse failure — no citation back-fill,
grounded=false/unsupported — and runs the numeric gate over salvaged prose.
Extractive recovery and strong-retry gating updated to match.
B6: retry route reset is now a single conditional UPDATE guarded on
status/stale locked_at; 0 rows affected -> 409, closing the TOCTOU race.
N1: verifyAnswerNumbers fails closed when no citation maps to a known chunk.
Nit-1: drop the dead query param in applyNumericVerification.
Adds regression tests for each fix; updates rag-trust.test.ts to assert the
safe (ungrounded) parse-failure behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
BigSimmo
added a commit
that referenced
this pull request
Jun 19, 2026
…rged #38 (#41) Re-applies the 6 blocking clinical-safety fixes (B1-B6) on top of current main. PR #38 merged 22 RAG fixes before these blockers landed, and #39/#40 did not address them, so main still trusts wrong doses. - B1: numeric verification now matches by exact normalized token-set membership instead of substring, so "2.5 mg" no longer verifies against "12.5 mg" (5x dose error). Fails closed. - B2: fold unicode superscripts (×10⁹/L) to ASCII before extraction/match. - B3: percentage branch no longer ends in \b (which could never match), so percentages extract and percentage mismatches are flagged. - B4: numeric gate now scans answerSections[].body (medication_dose), scoped to each section's citation_chunk_ids. - B5: safeFallbackAnswer fails closed on parse failure (ungrounded, no citation back-fill) and still runs the numeric gate; extractive recovery and strong-retry routing updated accordingly. - B6: retry route reset is a single conditional UPDATE guarded on status/locked_at; 0 rows affected => 409, closing the TOCTOU race. - N1: verifyAnswerNumbers fails closed when no cited chunk maps to a result. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
BigSimmo
added a commit
that referenced
this pull request
Jun 19, 2026
…(B1-B6)
B1: numeric verification now matches by exact normalized-token set membership
instead of substring, so "2.5 mg" no longer verifies against "12.5 mg"
(a 5x dose error). Source tokens are extracted with the same extractor.
B2: fold unicode superscripts (×10⁹/L) to ASCII and try the scientific-notation
branch first so ANC/WBC thresholds extract whole and match ASCII sources.
B3: remove the impossible trailing \b after % so percentages extract and a
percentage mismatch is flagged; fix the misleading doc comment.
B4: numeric gate now verifies answerSections[].body (e.g. medication_dose),
scoping each section to its own citation_chunk_ids when present.
B5: safeFallbackAnswer fails closed on parse failure — no citation back-fill,
grounded=false/unsupported — and runs the numeric gate over salvaged prose.
Extractive recovery and strong-retry gating updated to match.
B6: retry route reset is now a single conditional UPDATE guarded on
status/stale locked_at; 0 rows affected -> 409, closing the TOCTOU race.
N1: verifyAnswerNumbers fails closed when no citation maps to a known chunk.
Nit-1: drop the dead query param in applyNumericVerification.
Adds regression tests for each fix; updates rag-trust.test.ts to assert the
safe (ungrounded) parse-failure behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
BigSimmo
added a commit
that referenced
this pull request
Jun 19, 2026
…(B1-B6)
B1: numeric verification now matches by exact normalized-token set membership
instead of substring, so "2.5 mg" no longer verifies against "12.5 mg"
(a 5x dose error). Source tokens are extracted with the same extractor.
B2: fold unicode superscripts (×10⁹/L) to ASCII and try the scientific-notation
branch first so ANC/WBC thresholds extract whole and match ASCII sources.
B3: remove the impossible trailing \b after % so percentages extract and a
percentage mismatch is flagged; fix the misleading doc comment.
B4: numeric gate now verifies answerSections[].body (e.g. medication_dose),
scoping each section to its own citation_chunk_ids when present.
B5: safeFallbackAnswer fails closed on parse failure — no citation back-fill,
grounded=false/unsupported — and runs the numeric gate over salvaged prose.
Extractive recovery and strong-retry gating updated to match.
B6: retry route reset is now a single conditional UPDATE guarded on
status/stale locked_at; 0 rows affected -> 409, closing the TOCTOU race.
N1: verifyAnswerNumbers fails closed when no citation maps to a known chunk.
Nit-1: drop the dead query param in applyNumericVerification.
Adds regression tests for each fix; updates rag-trust.test.ts to assert the
safe (ungrounded) parse-failure behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
End-to-end fix for all Critical and High severity issues across the three stages of the Clinical KB RAG pipeline. This branch stacks stages 1 + 2 + 3 — it contains every fix below in three commits and targets
main.Guiding principle throughout: fail loud / surface "verify against source" rather than silently degrade. No existing passing tests were weakened; focused tests were added per fix.
Stage 1 — Indexing / Ingestion (
IDX-*)indexfield on OpenAI embedding responses so embeddings are matched to the correct chunk instead of relying on array order.EMBEDDING_DIMENSIONSagainst vector/length mismatch so a mis-sized embedding fails loudly rather than corrupting the index.Stage 2 — Retrieval / Search (
RET-*)hasNumericOrTableEvidence()gating so dose/threshold queries only trust chunks that actually carry numeric/table evidence.Stage 3 — Answer / Response Generation (
GEN-*)status: "incomplete"/incomplete_details.reason: "max_output_tokens"): surface atruncatedflag, log the reason, downgrade confidence, append a verify-against-source caveat, and raise theOPENAI_MAX_OUTPUT_TOKENSdefault.src/lib/answer-verification.ts): every dose/threshold/numeric token in the answer must appear in a cited chunk; unverified tokens are flagged, confidence downgraded, and a "verify against the source" gap added.grounded:false/confidence:"unsupported"(routingReasonungrounded_no_model_citation) instead of back-filling all retrieved chunks as fake citations.Test plan
npm run typecheck— cleannpm run lint— cleannpm run test— 369 passed (55 files), including new focused tests:tests/answer-verification.test.ts(numeric token extraction + faithfulness verification)tests/openai-cache.test.tstests/rag-trust.test.ts🤖 Generated with Claude Code