Skip to content

fix(graduation): session-count-aware state for backfilled patterns#119

Merged
Gradata merged 1 commit into
mainfrom
fix/graduation-session-count-state
Apr 20, 2026
Merged

fix(graduation): session-count-aware state for backfilled patterns#119
Gradata merged 1 commit into
mainfrom
fix/graduation-session-count-state

Conversation

@Gradata

@Gradata Gradata commented Apr 20, 2026

Copy link
Copy Markdown
Owner

Summary

Cherry-picked from a local commit on the main worktree (40f33e5a, authored 2026-04-19 by Oliver) — promoting it to a reviewable PR so it can land cleanly before the v0.6.1 tag.

_patterns_to_graduated_lessons previously teleported every qualifying correction_pattern directly to RULE@0.92, bypassing the INSTINCT→PATTERN→RULE graduation gates. A 2-session pattern ended up with the same standing as a 50-session rule. Live brain showed 18 RULEs / 0 PATTERNs / 2 INSTINCTs — all 18 RULEs came from weak 2-session evidence.

Fix

_state_for_sessions now maps evidence → state:

  • 2 sessions → PATTERN @ 0.70
  • 3-4 → PATTERN @ 0.80
  • 5+ → RULE @ 0.92

Aligns backfill with live graduation thresholds (MIN_APPLICATIONS_FOR_PATTERN=3, MIN_APPLICATIONS_FOR_RULE=5). Backfill still filters single-session patterns (min_sessions=2).

Test plan

  • pytest tests/test_rule_pipeline.py — 26 passed in 0.56s
  • After merge, re-backfill live brain and confirm state distribution is no longer 18/0/2

Why now

Needed before cutting the v0.6.1 tag — without this, any brain backfilled on 0.6.1 will replicate the over-promotion bug. Blocker for #117.

Generated with Gradata

_patterns_to_graduated_lessons previously teleported every qualifying
correction_pattern directly to RULE@0.92, bypassing the INSTINCT→PATTERN
→RULE graduation gates. A 2-session pattern ended up with the same
standing as a 50-session rule. Result on live brain: 18 RULEs / 0
PATTERNs / 2 INSTINCTs — all 18 RULEs came from weak 2-session evidence.

_state_for_sessions now maps evidence to state:
  - 2 sessions  → PATTERN @ 0.70
  - 3-4         → PATTERN @ 0.80
  - 5+          → RULE    @ 0.92

Aligns backfill with live graduation thresholds (MIN_APPLICATIONS_FOR_
PATTERN=3, MIN_APPLICATIONS_FOR_RULE=5). Backfill still filters single-
session patterns (min_sessions=2) — those would be noise at the 2.1GB
transcript scale.

Co-Authored-By: Gradata <noreply@gradata.ai>

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

@coderabbitai

coderabbitai Bot commented Apr 20, 2026

Copy link
Copy Markdown

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Summary

  • Fixed over-promotion during backfill: Previously, all backfilled patterns were promoted to RULE state with 0.92 confidence regardless of session count, causing weak 2-session evidence to be treated identically to battle-tested 50-session rules.

  • Session-aware state mapping: Introduced _state_for_sessions() function that derives both LessonState and confidence based on distinct session counts:

    • 2 sessions → PATTERN @ 0.70
    • 3–4 sessions → PATTERN @ 0.80
    • 5+ sessions → RULE @ 0.92
  • Aligns with live graduation gates: Thresholds now match live system constants (MIN_APPLICATIONS_FOR_PATTERN=3, MIN_APPLICATIONS_FOR_RULE=5).

  • Updated _patterns_to_graduated_lessons(): Now computes distinct_sessions per candidate and calls _state_for_sessions() to populate Lesson.state and Lesson.confidence instead of hard-coding RULE/0.92.

  • Test coverage: Updated existing test expectations and added new comprehensive test validating all three state tiers map correctly based on session count.

  • No public API changes: Modified function is internal; no breaking changes to public interfaces.

  • Post-merge manual verification required: Re-backfill live brain to confirm state distribution no longer shows the over-promotion pattern (expected change from 18 RULEs / 0 PATTERNs / 2 INSTINCTs distribution).

Walkthrough

Updated the rule pipeline to derive lesson state and confidence based on the count of distinct sessions rather than using fixed values. Added a tiered threshold function that maps session counts to state transitions and session-aware confidence scores, replacing the previous hard-coded RULE state and 0.92 confidence.

Changes

Cohort / File(s) Summary
Session-Based State Determination
src/gradata/enhancements/rule_pipeline.py
Added _state_for_sessions() function implementing tiered thresholds (2→PATTERN@0.70, 3–4→PATTERN@0.80, 5+→RULE@0.92). Modified _patterns_to_graduated_lessons() to compute distinct sessions and delegate state/confidence assignment to the new function instead of hard-coding RULE with 0.92.
Test Updates
tests/test_rule_pipeline.py
Updated test assertions to expect LessonState.PATTERN with ~0.70 confidence for 2-session patterns instead of RULE with ≥0.90. Added new test validating session-count-driven state transitions and confidence outputs across all three threshold categories.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

bug

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding session-count awareness to determine lesson state during pattern graduation, replacing the previous hard-coded RULE@0.92 assignment.
Description check ✅ Passed The description provides clear context for the fix, explaining the prior bug, the new mapping logic, and the test coverage, all directly related to the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/graduation-session-count-state

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant