feat(rag): eval-gate ON #213's ranking wins (diversity demotion + linear freshness)#242
Merged
Conversation
…on + linear freshness) #213 shipped documentDiversityPenalty and the linear freshness curve config-gated and default-OFF, to be turned on only after eval-gating. Live evals show a strict improvement with no regression on the safety bar, so both are now ON by default: - documentDiversityPenalty 0 -> 0.02 (gentle same-document crowding demotion, capped 0.12) - freshness.mode step -> linear (ramped decay instead of the harsh cliff) Also fixes resolveRankingConfig, which hardcoded the freshness `mode` fallback to "step" instead of deriving it from defaultRankingConfig, so changing the default alone would not have taken effect at runtime without an explicit env override. Eval evidence (golden retrieval + eval:quality --rag-only, live): - retrieval mrr@10 0.7283 -> 0.7569, document/content recall@5 stays 1.0, 0 failures - grounded_supported 0.967 -> 1.0, unsupported_correct unchanged, citation/numeric failure 0 - verify:cheap green (980/980) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#213 shipped two retrieval-time ranking improvements config-gated and default-OFF, to be flipped on only after eval-gating. This PR does the eval-gating and turns them on by default:
documentDiversityPenalty0 → 0.02 — gentle same-document crowding demotion (capped 0.12; RC7/CI-16).freshness.modestep → linear — ramped freshness decay instead of the harsh cliff (CI-17).It also fixes a latent bug in
resolveRankingConfig: the freshnessmodefallback was hardcoded to"step"instead of derived fromdefaultRankingConfig, so changing the default alone would not have taken effect at runtime without an explicitRAG_RANKING_CONFIGoverride. The override path is unchanged; settingdocumentDiversityPenalty:0+freshness.mode:"step"restores exact pre-flip behavior.Eval evidence (live)
mrr@10grounded_supported_rateunsupported_correct_rate*The one failing unsupported case (
unsupported-close-title-noise) fails identically on baseline — it's a pre-existing flaky false-positive control, not caused by this change. The flip also cleared the twoagitationgrounding cases in the after-run.verify:cheap: green, 980/980. Confirmed the flip takes effect by default (clean env): goldenmrr@10=0.7569.Clinical governance preflight
grounded_supportedimproves to 1.0;unsupported_correctunchanged (safety bar held); citation/numeric-grounding failure stay 0; no source-governance change.RAG_RANKING_CONFIGwithout a code change.🤖 Generated with Claude Code