eval: answer-targeting metric + retrieval-eval process gate + quote-truncation flag (Stage 1: measurement & guardrails)#243
Conversation
…uote truncation flag Stage 1 of the safe-execution plan — measurement and guardrails, zero live behavior change. P3 (measurement, additive, opt-in): - scoreAnswerTargeting: structural per-intent targeting check (dose→figure/regimen, red-result→withhold action, monitoring→schedule, contraindication→avoid, referral→criteria, doc-lookup→named/cited document) — stronger than the loose keyword cue; informational, never a hard gate; unsupported/fail-closed cases are n/a. Existing scoreAnswerQualityEvalCase and its 5-metric contract untouched. - New standalone scripts/eval-answer-quality.ts (npm run eval:answer-quality) runs the previously dormant 30-case answerQualityEvalCases fixture and reports the 5 metrics + per-intent targeting. Separate/opt-in so it adds no spend to any existing eval run. P2 (process gate, docs only): - PR template: require eval:retrieval:quality (23/23) for retrieval/ranking/selection/chunking PRs and eval:rag + eval:quality for answer-generation PRs (CI can't run them — needs live keys). - process-hardening.md: document the gate + the measured PR-118 regression (governance metadata weighting buried correct docs 1.0->0.76) and the relevance-first standing constraint. P4c (provenance, additive display): - QuoteCard.isTruncated (optional) set by bestQuoteFromContent when a quote is cut at 340 chars; quote card shows a "truncated — open the source" note so a value past the cut is never read as the complete passage. verify:cheap green (984 passed); format:check clean. New unit tests for scoreAnswerTargeting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fb1fe4f11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b364def046
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55b30e7f70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const hasMonitoringRange = | ||
| /\b\d+(?:\.\d+)?\s*(?:-|to|–)\s*\d+(?:\.\d+)?\s*(?:mmol\/l|mmol|mg\/l|microgram\/l|mcg\/l|ng\/ml)\b/i.test(text) || | ||
| /\b(?:mmol\/l|mmol|mg\/l|microgram\/l|mcg\/l|ng\/ml)\b/i.test(text); |
There was a problem hiding this comment.
For monitoring-range questions such as quality-lithium-monitoring-range, this bare-unit fallback makes hasMonitoringRange true for an answer that only says something like “lithium levels are reported in mmol/L” without giving the requested 0.4–0.8-style range. That lets generic answers pass the new targeting metric and masks the non-specific failures the metric is intended to surface; require a numeric level/range rather than a unit token alone.
Useful? React with 👍 / 👎.
| ? { score: 1, applicable: true, reason: "carries a dose figure/regimen" } | ||
| : { score: 0, applicable: true, reason: "no dose figure/regimen" }; | ||
| case "red_result_action": | ||
| return hasRedAction && (!asksForThreshold || hasNumber) |
There was a problem hiding this comment.
For threshold questions, this accepts any digit anywhere in the answer as the threshold signal, so an answer like “Step 1: withhold clozapine and arrange review” passes targeting for an ANC-threshold case without giving an ANC/WBC/count value. That hides the same generic/non-specific red-result failures this metric is supposed to expose; require a real threshold pattern instead of hasNumber.
Useful? React with 👍 / 👎.
| - [ ] `npm run verify:ui` when UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changed | ||
| - [ ] `npm run verify:release` before release or handoff confidence claims | ||
| - [ ] `npm run format:check` | ||
| - [ ] **`npm run eval:retrieval:quality` (must stay 23/23) when retrieval, ranking, selection, chunking, or scoring behavior changed** — CI cannot run it (needs live keys), so run it locally and paste the summary. A metadata/governance-weighting change once buried correct docs (recall 1.0→0.76) and only this eval caught it. |
There was a problem hiding this comment.
The checked-in scripts/fixtures/rag-retrieval-golden.json currently has 24 cases, and this PR's verification summary reports eval:retrieval:quality as 24/24, but the new PR checkbox tells reviewers the gate only needs to stay 23/23. For retrieval/ranking PRs this can make a one-case regression look acceptable; require zero failures or the current 24/24 count instead of the stale 23/23 number.
Useful? React with 👍 / 👎.
| alternative | ||
| .split(/\s+/) | ||
| .filter((token) => token.length > 2) | ||
| .every((token) => text.includes(token)), |
There was a problem hiding this comment.
For document-lookup cases where an alias is also the clinical topic, this fallback marks generic topic repetition as naming the expected document. For example CG.MHSP.PtSafetyPlan.pdf has the alias “patient safety plan”, so an answer that only says “the patient safety plan should be documented” passes without naming or citing the source document; require an expected citation or an explicit document/form/guideline title cue instead of token presence anywhere in the answer.
Useful? React with 👍 / 👎.
| if (token === "--limit") args.limit = Number.parseInt(value, 10); | ||
| if (token === "--targeting-floor") args.targetingFloor = Number.parseFloat(value); |
There was a problem hiding this comment.
When --limit or --targeting-floor is omitted, typoed, or provided in an unsupported form like --limit=5, these assignments store NaN. The limit is then ignored by the truthy check below, so the script runs all 30 live answer calls, and a NaN floor silently disables the requested gate; reject missing/non-positive numeric values before calling the provider.
Useful? React with 👍 / 👎.
1 participant
Summary
Stage 1 of the safe-execution plan — measurement and guardrails, zero live behavior change. Every item is additive/opt-in; no default answer or retrieval behavior changes; no re-index; nothing depends on the parked metadata backfill.
scoreAnswerTargeting: a structural per-intent targeting check (dose→figure/regimen, red-result→withhold action + threshold, monitoring→schedule, contraindication→avoid, referral→criteria, doc-lookup→named/cited document) — stronger than the loose keyword cue, informational (never a hard gate), n/a for unsupported/fail-closed cases. New standalonescripts/eval-answer-quality.ts(npm run eval:answer-quality) finally runs the previously-dormant 30-caseanswerQualityEvalCasesfixture and reports the 5 quality metrics + per-intent targeting. Separate/opt-in so it adds no spend to any existing eval. The existingscoreAnswerQualityEvalCase5-metric contract and its test are untouched.eval:retrieval:quality(23/23) for retrieval/ranking/selection/chunking PRs andeval:rag+eval:qualityfor answer-generation PRs (CI can't run them — needs live keys).process-hardening.mddocuments the rule and the measured PR-118 regression (governance-metadata weighting buried correct docs, recall 1.0→0.76) + the relevance-first standing constraint.QuoteCard.isTruncatedset bybestQuoteFromContentwhen a quote is cut at the 340-char cap; the quote card shows a "truncated — open the source" note so a value past the cut is never read as the complete passage.First run of the new tool already earned its keep: it surfaced that during a provider-slowness episode a chunk of answers correctly degraded to the labeled source-only timeout pointer (relevance 0.53, red-result targeting 0/3) — a transient OpenAI-latency signal, not a code regression (golden retrieval 24/24,
verify:cheapgreen). A clean baseline should be re-run when the provider is healthy.Verification
npm run verify:cheap(986 passed / lint + typecheck clean)npm run format:check(changed files prettier-clean)npm run eval:retrieval:quality— 24/24, document_recall@5 1.0, content_recall@5 1.0, top_k 1.0, mrr@10 0.75 (ran because this touchesevidence.ts; no retrieval regression)npm run eval:rag/eval:quality— n/a: no answer-generation or post-processing logic changed (quote-truncation is display-only; eval additions don't affect the answer path)npm run check:production-readiness— n/a: no clinical workflow / env / Supabase / source-governance behavior changedClinical Governance Preflight
Clinical KB Database)Notes
eval:answer-qualityuses the same live-key + owner-scope bootstrap aseval:rag; it does not run in CI or in any existing gate.🤖 Generated with Claude Code