fix(eval): require expected-doc coverage for acceptSourceOnly quality pass

[BigSimmo]

Summary

Follow-up to #538 (Codex P2). scoreAnswerQualityEvalCase's acceptSourceOnly relevance branch accepted any source-only/unsupported answer with only (grounded || unsupported) && expectedClassOk — it never checked that the answer still surfaced the expected documents. Combined with the dropped mustContainAny cue, a source-only answer that stopped retrieving e.g. MHSP.Duress.pdf / MHSP.Discharge.pdf could still score relevant and intent-covered, hiding a retrieval regression in the quality canary (and contradicting the documented "expectedFiles keeps the retrieval guard" intent).

Fix: wire the existing citesOrNamesExpectedDocument helper into the acceptSourceOnly branch, so a source-only answer must still cite or name the expected doc to score relevant. This also tightens the already-merged discharge sibling.

Note: this is defense-in-depth — the core duress-procedure/discharge-documentation cases already enforce expectedHit && citations in the quality summary, so the regression was never hidden globally.

Verification

Offline only (pure eval-scoring logic; no answer-generation or provider behavior changed):

• tests/rag-eval-cases.test.ts updated — asserts a source-only answer scores relevant only when it cites the expected doc, and 0 when it does not; 24 pass
• eval-quality + eval-utils harness tests: green (35 pass across suites)
• typecheck, prettier, eslint --max-warnings 0: clean

Clinical Governance Preflight

Eval-scoring-only change; does not touch ingestion, answer generation, retrieval, or document access. It makes the source-only eval stricter (must still surface expected docs), improving regression detection.

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

📝 Walkthrough

Walkthrough

Diffuse-question relevance scoring now requires answers to cite or name the expected document, with tests covering both cited and uncited source-only answers.

Changes

Relevance scoring

Layer / File(s)	Summary
Enforce and test expected-document coverage src/lib/rag-eval-cases.ts, tests/rag-eval-cases.test.ts	The diffuse-question relevance condition now checks expected-document coverage, and tests expect scores of 1 with the citation and 0 without it.

Estimated code review effort: 2 (Simple) | ~10 minutes

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
Verification Claims	❌ Error	PR description claims verification passed (“24 pass”, “typecheck, formatting, lint… clean”) without naming exact commands/results, which the check forbids.	Rewrite verification notes as explicit entries, e.g. “Ran : passed” or “Not run: ”, and name each test/build/check separately.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 70.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (9 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main eval change: requiring expected-doc coverage for acceptSourceOnly passes.
Description check	✅ Passed	The description covers summary, verification, and governance sections and is mostly aligned with the template.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Generated And Sensitive Files	✅ Passed	Only two tracked TS files changed; no .env/log/cache/build artifacts or secret-like strings were present in the diff or file scan.
Risky Git Or Deployment Actions	✅ Passed	The PR only changes eval-scoring logic and tests; I found no added instructions, scripts, workflows, or docs recommending risky git/deployment actions.
Supabase Project And Schema Safety	✅ Passed	Only eval-scoring code and tests changed; no Supabase project refs, migrations, RLS, policies, or schema edits were touched.
Runtime And Package Manager Integrity	✅ Passed	PR only changes src/test files; package.json still pins npm@11.17.0, engines node 24/npm 11, and .npmrc keeps engine-strict=true. No lockfile/runtime changes.
Api Route Failure Handling	✅ Passed	PR only changes eval-scoring logic and tests; no API route/server action/provider integration code is touched, so failure-handling concerns don’t apply.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/quality-eval-expected-doc-coverage

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch claude/quality-eval-expected-doc-coverage

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4868031b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".