Skip to content

docs(results): document raw vs weighted winner fields#689

Merged
LinoGiger merged 3 commits into
mainfrom
docs(results)/document-raw-and-weighted-winners
Jul 21, 2026
Merged

docs(results): document raw vs weighted winner fields#689
LinoGiger merged 3 commits into
mainfrom
docs(results)/document-raw-and-weighted-winners

Conversation

@RapidPoseidon

Copy link
Copy Markdown
Contributor

Companion docs PR to RapidataAI/rapidata-evaluator#302, which adds explicit rawWinner/rawWinnerIndex and weightedWinner/weightedWinnerIndex to Compare/Ranking results.

Why

Results exposed raw counts, weighted user-score sums, and both ratios, but only one generic winner/winner_index — which follows the weighted result. On close votes the raw-majority and weighted winners can differ, so a consumer could unintentionally read the weighted winner as a raw majority (reported from SDK integration, Compare task).

What this PR does (docs only)

Updates docs/understanding_the_results.md:

  • Documents rawWinner/rawWinnerIndex and weightedWinner/weightedWinnerIndex — both winners, their shared index space (Both/Neither as trailing indexes), and the null-on-tie / no-clear-winner behavior.
  • Marks the generic winner/winner_index as deprecated aliases of the weighted fields. Because the result payload is a plain JSON dict there's no type-level deprecation hook, so docs are the deprecation channel (runtime typed accessors / warnings are the sibling stabilize result schema + typed accessors effort's job, reading these same field names).
  • Clarifies how summedUserScores is summed (per-voter aggregated userScore) and how summedUserScoresRatios normalizes it.
  • Adds the example fields + a note on the aggregator version floor (4.1.0) so readers know older result files fall back to winner/winner_index.

No SDK code changes here — the canonical fields are produced server-side by the evaluator (design decision: evaluator is the single source of truth).

🔗 Session: https://node-39981d64.poseidon.rapidata.internal/

The aggregator now emits explicit rawWinner/rawWinnerIndex and
weightedWinner/weightedWinnerIndex on Compare/Ranking datapoints
(rapidata-evaluator). Document them in the result reference:

- Describe both winners, their shared index space (Both/Neither as
  trailing indexes), and the null-on-tie / no-clear-winner behavior.
- Mark the generic winner/winner_index as deprecated aliases of the
  weighted fields (the result file is a plain dict, so this is the
  deprecation channel) and steer new code to the explicit names.
- Clarify how summedUserScores is summed (per-voter aggregated score)
  and how summedUserScoresRatios normalizes it.
- Note the aggregator version floor (4.1.0) so readers know old result
  files fall back to winner/winner_index.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
@claude

claude Bot commented Jul 20, 2026

Copy link
Copy Markdown

Reviewed — this is a docs-only change (docs/understanding_the_results.md), no SDK code touched, which matches the PR description. Overall the additions are clear and well-organized: the raw-vs-weighted distinction, shared index space (with Both/Neither trailing indexes), and the 4.1.0 version-floor note all read well and follow the existing !!! note admonition convention used elsewhere in the docs.

One thing worth double-checking before merge:

Possible inconsistency in the winner/winner_index description (lines ~112-116): they're called "deprecated aliases" of weightedWinner/weightedWinnerIndex, but the same bullet then says they're "never null (ties are broken toward the first option)" — while weightedWinner is documented as null-on-tie. If winner/winner_index truly alias the weighted fields, they should share the same tie behavior (including going null). If instead the legacy fields intentionally keep the old "break ties toward first option" behavior while the new explicit fields introduce null-on-tie as a behavior change, that's fine — but then "alias" is a bit misleading and it'd be worth a one-line clarification (e.g. "same value except on ties, where the legacy fields fall back to index 0 instead of null"). Since this doc is explicitly called out as the deprecation channel (no type-level enforcement possible), it's worth getting this precise since it's the only source of truth for consumers.

Minor/optional:

  • The summary.A_wins_total/B_wins_total section isn't updated to clarify whether it's derived from the raw or weighted winner — given the PR's whole point is that these can diverge, a short note there could preempt the same ambiguity the PR is fixing elsewhere (not necessary if summary predates and is out of scope).
  • The Both/Neither trailing-index behavior is mentioned but not shown in the example JSON (which only has 2 options) — not blocking, just noting a reader has to take it on faith.

No bugs, security, or test-coverage concerns — this is text-only and there's nothing to unit test. Nice, focused documentation update.

RapidPoseidon and others added 2 commits July 20, 2026 15:09
Match the evaluator rename (rapidata-evaluator#302): the reliability-
weighted winner is winner/winnerIndex, the raw vote-count winner is
majorityWinner/majorityWinnerIndex, both null when there is no clear
winner. winner_index is documented as a deprecated snake_case alias of
winnerIndex.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
Match the evaluator naming (rapidata-evaluator#302): winner/winnerIndex
is the raw vote-count winner, weightedWinner/weightedWinnerIndex is the
reliability-weighted winner, both null when there is no clear winner.
winner_index stays documented as a deprecated alias of winnerIndex.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
@LinoGiger
LinoGiger marked this pull request as ready for review July 21, 2026 09:01
@LinoGiger
LinoGiger self-requested a review as a code owner July 21, 2026 09:01
@LinoGiger
LinoGiger merged commit 501f91b into main Jul 21, 2026
1 check passed
@LinoGiger
LinoGiger deleted the docs(results)/document-raw-and-weighted-winners branch July 21, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants