fix(eval): require citation coverage (not prose) for acceptSourceOnly relevance

[BigSimmo]

Summary

Completes the #562 fix (Codex P2). The acceptSourceOnly relevance guard used citesOrNamesExpectedDocument, whose answer-text fallback matches bare topic tokens — documentExpectationAlternatives("MHSP.Duress.pdf") includes duress — so an uncited gap answer like "No current source for the duress procedure…" still scored relevant, defeating the retrieval-regression guard #562 was adding.

Fix: require citation coverage in the acceptSourceOnly branch via expectedFileCoverage(testCase.expectedFiles, answer.citations, …).anyHit instead of the cite-or-name helper. (citesOrNamesExpectedDocument is left intact — still used by the separate targeting metric at its other call site.)

Defense-in-depth as before: the core duress-procedure/discharge-documentation cases already enforce expectedHit && citations, so the regression was never hidden globally.

Verification

Offline only (pure eval-scoring logic):

• tests/rag-eval-cases.test.ts: strengthened — a source-only answer whose prose mentions the topic ("discharge") but cites nothing now scores 0; citing the expected doc scores 1. 24 pass.
• typecheck, prettier, eslint --max-warnings 0: clean

Clinical Governance Preflight

Eval-scoring-only; no ingestion/answer-generation/retrieval/document-access changes. Makes the source-only quality gate stricter (must cite expected docs).

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

📝 Walkthrough

Walkthrough

Source-only relevance scoring now requires expected-document citation coverage instead of prose or document-name matching. The corresponding test clarifies that topic mentions without the expected citation do not satisfy relevance.

Changes

Source-only relevance scoring

Layer / File(s)	Summary
Require expected-document citation coverage src/lib/rag-eval-cases.ts, tests/rag-eval-cases.test.ts	The relevance gate now checks expected-file citation coverage, and the test verifies that uncited topic mentions do not count as coverage.

Estimated code review effort: 2 (Simple) | ~10 minutes

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Verification Claims	❌ Error	PR notes claim verification (“typecheck, formatting, and lint checks pass”; “24 pass”) without naming exact commands/checks and results.	Rewrite each verification item as exact check + outcome, e.g. “Ran npm run verify:cheap: passed” or “Not run: ”.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: requiring citation coverage for acceptSourceOnly relevance.
Description check	✅ Passed	The description includes the required Summary, Verification, and Clinical Governance sections and is detailed enough despite not mirroring every template checkbox.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Generated And Sensitive Files	✅ Passed	PR changes only eval source/test files; no new .env, keys, logs, caches, build artifacts, or other sensitive/generated files were introduced.
Risky Git Or Deployment Actions	✅ Passed	Changed files only adjust eval scoring/tests; no force-push, reset --hard, destructive clean, rebasing, or deployment instructions appear.
Supabase Project And Schema Safety	✅ Passed	PR only changes eval scoring/tests; no Supabase project refs, migrations, RLS/policy, or destructive schema/data changes appear in the diff.
Runtime And Package Manager Integrity	✅ Passed	Diff only touches src/lib/rag-eval-cases.ts and tests; no package-manager/runtime files or flags changed, and repo still pins Node 24/npm 11 with engine-strict.
Api Route Failure Handling	✅ Passed	Only eval-scoring and tests changed; no API route/server action/provider/ingestion code was touched, and the new citation matcher is pure and deterministic.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/eval-citation-coverage-strict

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/eval-citation-coverage-strict

---

_{Comment @coderabbitai help to get the list of available commands.}