fix(json): stop a skipped sub-score being averaged as a mismatch - #210
Open
Arthi Arumugam (arthi-arumugam-git) wants to merge 1 commit into
Open
Conversation
Score documents that "If the score is None, the evaluation is considered to be skipped", and json_diff filters those out before averaging. The two branches then disagreed about what to divide by. The dict branch divided by len(base_scores), the count after filtering, so a skipped key was excluded from both the numerator and the denominator and correctly ignored. The list branch divided by max(len(o1), len(o2)), which still counts the skipped element, so the same skip was averaged in as a zero. The result is that one skipped comparison scores 1.0 inside an object and 0.5 inside a two-element array, for identical values and an identical scorer. The dict branch also divided by len(base_scores) with no guard. An object whose every comparison is skipped leaves that list empty and raises ZeroDivisionError rather than reporting a skip. Both branches now drop skipped comparisons from the denominator and return None, propagating the skip, when nothing is left to average. The list denominator stays max(len(o1), len(o2)) minus the skips, so elements with no counterpart are still counted as a real difference; only the skips come out. Eight tests, four of which fail on main. The other four pin what must not move: missing elements are still penalised, and unskipped lists, dicts and empty containers score exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Scoredocuments the contract:JSONDiff.json_difffilters those out before averaging, but the two branches then divide bydifferent things:
In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.
The same skip therefore lands differently depending on the container:
main{"a": "skip", "b": "same"}["skip", "same"]Same values, same scorer, same skip.
This is reachable with any scorer that can abstain, which is the case the
Nonescore existsfor: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.
Second, smaller issue
The dict branch divides by
len(base_scores)with no guard. An object whose comparisons areall skipped leaves that list empty and raises
ZeroDivisionError. The empty-object case ishandled above it, so this needs a non-empty object rather than an edge case in the inputs.
The change
Both branches now drop skipped comparisons from the denominator, and return
Nonewhennothing is left to average, which propagates the skip upward instead of inventing a number or
raising.
The list denominator stays
max(len(o1), len(o2))minus the skips. That distinction isdeliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what
max(...)contributes over thezip. Only the skips come out.Tests
py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail onmain:The other four pin behaviour that must not move: missing elements are still penalised
(
["a"]vs["a", "b"]is still 0.5), and unskipped lists, dicts and empty containers scoreexactly as before.
py/autoevals/test_json.py: 4 passed. Acrosspy/autoevalsthe results are identical withand without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
blackandisortclean at line-length 119.