bench: PMR-100 procedural memory retention benchmark (rebased)

[Gradata]

Clean rebase of #148.

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7f14fe36-1123-467a-9c53-a97afa363cf7

📥 Commits

Reviewing files that changed from the base of the PR and between 5f5f87f and 7516950.

📒 Files selected for processing (4)

• Gradata/.gitignore
• Gradata/bench/README.md
• Gradata/bench/__init__.py
• Gradata/bench/pmr_100.py

---

📝 Walkthrough

• New benchmark: Adds PMR-100 (Procedural Memory Retention) benchmarking suite with 100 scripted sessions across 6 correction classes
• Metrics: Implements recall@1 and recall@3 scoring with per-class breakdowns; CLI invocation via python -m bench.pmr_100 [--quick] [-n N]
• New public API: Exports Scenario, SessionResult, and BenchResult dataclasses, plus run_benchmark() and main() functions
• Benchmark structure: Tests correction injection, distractor turns, probing, and keyword-based recall validation against expected outputs
• Baseline data: Initial 3-session run shows 0% rule extraction and 0% recall (expected behavior per FSRS design)
• Package setup: Enables bench/ as importable Python package
• Result persistence: Outputs benchmark results to JSON with timestamp, config, summary stats, and per-session data
• Documentation: Adds comprehensive README with workflow description, CLI usage, and scenario configuration guidance

Walkthrough

Introduces the PMR-100 "Procedural Memory Retention" benchmark suite. Adds a benchmarking script that evaluates a Brain system's ability to extract and recall procedural rules through correction injection, distractor turns, and recall scoring. Includes documentation, package setup, and CLI interface for running benchmarks with configurable parameters.

Changes

Cohort / File(s)	Summary
Package Setup Gradata/.gitignore, Gradata/bench/__init__.py	Adds ignore rule for bench/results/ directory and marks bench/ as a Python package to enable module execution.
Documentation Gradata/bench/README.md	Documents PMR-100 benchmark workflow, expected outputs, CLI commands, baseline results, and instructions for adding new scenarios.
Benchmark Implementation Gradata/bench/pmr_100.py	Implements complete benchmarking script with dataclasses for scenario, session, and result definitions. Includes run_one_session() to execute individual benchmark sessions with correction injection and distractor turns, run_benchmark() to aggregate results across multiple sessions, and main() CLI entrypoint with configurable parameters (session count, distractor count, seed, quick mode).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

feature, docs

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rebase/pmr-100-benchmark

---

_{Review rate limit: 1/5 review remaining, refill in 38 minutes and 9 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}