Skip to content

fix(eval): require citation coverage (not prose) for acceptSourceOnly relevance #564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 1 commit into from
Jul 13, 2026
+11 −5
Merged

fix(eval): require citation coverage (not prose) for acceptSourceOnly relevance #564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions src/lib/rag-eval-cases.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,14 @@ export function scoreAnswerQualityEvalCase(testCase: AnswerQualityEvalCase, answ
const relevanceOk = testCase.supported
? testCase.acceptSourceOnly
? // Diffuse question: a grounded synthesis OR a source-only/unsupported answer is acceptable,
// but it must still surface the expected documents (cite or name them). Without this a
// source-only answer that stopped retrieving the expected doc would still score relevant,
// hiding a retrieval regression in this canary.
(answer.grounded || unsupported) && expectedClassOk && citesOrNamesExpectedDocument(testCase, answer, text)
// but it must still CITE the expected documents. A prose mention is not enough — the doc-name
// alternatives include bare topic tokens (e.g. "duress"), so a gap answer that merely names
// the topic would slip through. Require citation coverage, otherwise a source-only answer
// that stopped retrieving the expected doc would still score relevant and hide a retrieval
// regression in this canary.
(answer.grounded || unsupported) &&
expectedClassOk &&
expectedFileCoverage(testCase.expectedFiles, answer.citations, answer.citations.length).anyHit
: answer.grounded && answer.citations.length >= testCase.minCitations && expectedClassOk
: unsupported;
const readabilityOk = wordCount >= 5 && wordCount <= 220 && !fragmentPattern.test(text);
Expand Down
4 changes: 3 additions & 1 deletion tests/rag-eval-cases.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,8 +276,10 @@ describe("captured RAG eval cases", () => {

// A source-only answer that no longer surfaces the expected document must NOT score relevant —
// otherwise a retrieval regression that stops returning MHSP.Discharge.pdf would hide here.
// A prose mention of the topic ("discharge") must NOT rescue an uncited answer — coverage is
// citation-based, not answer-text based (the doc-name alternatives include bare topic tokens).
const withoutCite = {
answer: "No current indexed document directly supporting this request was found.",
answer: "No current source with discharge documentation guidance was found.",
grounded: false,
confidence: "unsupported",
citations: [],
Expand Down