evals: contains_expected / word_overlap raise on empty or whitespace expected#134
Merged
Conversation
…expected PR #123 made both raise when expected_output is None, but the empty-string frontier kept the old vacuous behavior: - contains_expected with expected="": "" in output is true for any output, so every case passes. - word_overlap with empty or whitespace-only expected: expected_words is empty and it returned a fabricated overlap=1.0. Both are the same silent-pass family as #116/#117/#118: a result with no meaning that still comes back green. Extend the None guard to empty and whitespace-only expected (contains_expected: empty after strip; word_overlap: no words after split), with a clear per-case error. Docs updated. Closes #126
renaudcepre
added a commit
that referenced
this pull request
Jun 25, 2026
…expected (#134) PR #123 made both raise when expected_output is None, but the empty-string frontier kept the old vacuous behavior: - contains_expected with expected="": "" in output is true for any output, so every case passes. - word_overlap with empty or whitespace-only expected: expected_words is empty and it returned a fabricated overlap=1.0. Both are the same silent-pass family as #116/#117/#118: a result with no meaning that still comes back green. Extend the None guard to empty and whitespace-only expected (contains_expected: empty after strip; word_overlap: no words after split), with a clear per-case error. Docs updated. Closes #126 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #126.
PR #123 made
contains_expectedandword_overlapraise whenexpected_output is None, but the empty-string frontier kept the old vacuous behavior:contains_expectedwithexpected="":"" in outputis true for any output, so every case passes regardless of the result.word_overlapwithexpected=""or whitespace-only:expected_wordsis empty and it returned a fabricatedoverlap=1.0, poisoning the tracked metric.Both are the same silent-pass family as #116/#117/#118 — a result with no meaning that still comes back green.
expected=""technically satisfied "requiresexpected" while still being vacuous.Decision
Per discussion: fail-loud, consistent with #123 and the #116-118 family. The guard fires on the value that is actually vacuous for each evaluator:
contains_expected: raises whenexpectedis empty or whitespace-only (empty afterstrip()).word_overlap: raises whenexpectedhas no words aftersplit()(empty or whitespace-only).Each raises a clear
ValueErrornaming the case and pointing at the fix (set a non-empty expected, or attach the evaluator per-case for mixed case sets).Nonekeeps its existing dedicated message.Tests
tests/evals/test_e2e.py(next to the existing*_none_expected_raisestests): empty and whitespace-only expected now raise, for both evaluators. Docs (docs/evals.md) updated to state the non-empty requirement.Full suite: 1245 passed,
ruffclean,mypy --strictclean.Generated by Claude Code