Add Quality Ratchet System

[frankbria]

Summary

Implement automated tracking of code quality metrics across AI conversation sessions to detect context window degradation.

Problem

As AI conversations grow longer:

• AI takes shortcuts that initially work
• Quality metrics gradually decline
• Coverage drops without notice
• Test pass rates decrease
• "Lazy" patterns reinforce themselves

Tasks

1. Quality Tracking Script

• Create tools/quality-ratchet.py
• Track metrics: coverage %, test pass rate, response count
• Store history in .claude/quality_history.json
• Implement degradation detection algorithm

2. Metrics Collection

• Parse pytest output for pass/fail counts
• Extract coverage percentage from reports
• Track conversation response count
• Timestamp each checkpoint

3. Degradation Detection

• Compare recent average to historical peak
• Flag >10% coverage drop
• Flag >10% pass rate drop
• Recommend context reset when triggered

4. CLI Interface

• quality-ratchet.py record --response-count N
• quality-ratchet.py check
• quality-ratchet.py stats
• quality-ratchet.py reset

5. Integration

• Add checkpoint calls to AI rules
• Update workflow documentation
• Create alerting for degradation
• Add to CI/CD for trending

Algorithm

recent_avg = avg(last_3_checkpoints)
peak_quality = max(all_previous_checkpoints)

if recent_avg < peak_quality - 10%:
    alert("Quality degradation detected")
    recommend("Reset AI context")

Data Format

{
  "history": [
    {
      "timestamp": "2025-11-09T10:30:00",
      "response_count": 5,
      "test_pass_rate": 100.0,
      "coverage": 85.5
    }
  ]
}

Success Criteria

• Automatically detects quality drops
• Provides clear visualizations of trends
• Recommends context resets at right time
• Integrates smoothly with development workflow

Dependencies

• Issue Add Claude Code GitHub Workflow #1 (enforcement foundation) must be complete
• Requires pytest and coverage.py

Estimated Effort: 4-6 hours

[traycerai]

Plan

Observations

The codebase has a well-established testing infrastructure with pytest, pytest-json-report, and coverage.py. The CLI uses Typer with Rich for output. Currently, there's no automated quality tracking system, and GitHub workflows don't run tests automatically. The quality ratchet system needs to integrate with existing patterns while filling this gap.

Approach

Create a standalone Python script tools/quality-ratchet.py that uses Typer CLI to track quality metrics (test pass rate, coverage %, response count) in .claude/quality_history.json. The script will parse pytest-json-report output and coverage.json to extract metrics, implement a degradation detection algorithm comparing recent averages to historical peaks, and provide commands for recording, checking, viewing stats, and resetting history. Integration points include CLAUDE.md documentation updates and a new GitHub Actions workflow for CI/CD trending.

Reasoning

I explored the repository structure, examined the testing infrastructure (pytest.ini, .coveragerc, pyproject.toml), analyzed existing CLI patterns in codeframe/cli.py, studied the TestRunner implementation to understand pytest-json-report usage, reviewed coverage.json structure, examined script patterns for JSON I/O and Rich console usage, and identified integration points in CLAUDE.md and GitHub workflows.

Proposed File Changes

📁 tools ^(NEW) 🔗

Create a new directory tools/ at the repository root to house the quality ratchet script and any future tooling scripts. This follows common Python project conventions for separating development tools from the main package code.

📄 tools/__init__.py ^(NEW) 🔗

Create an empty __init__.py file to make the tools directory a Python package. This allows for potential future imports and maintains Python package conventions.

📄 tools/quality-ratchet.py ^(NEW) 🔗

References

• 📄 codeframe/cli.py
• 📄 codeframe/core/config.py
• 📄 codeframe/testing/test_runner.py

Create the main quality ratchet script with the following structure:

Imports and Setup:

• Import json, subprocess, sys, datetime, statistics from stdlib
• Import Path from pathlib
• Import typer, Console and Table from rich (following patterns in codeframe/cli.py)
• Import Optional, List, Dict, Any from typing
• Create Typer app instance with name="quality-ratchet" and help text
• Create Rich Console instance for styled output

Constants:

• HISTORY_FILE = Path(".claude/quality_history.json")
• DEGRADATION_THRESHOLD = 10.0 # percentage
• RECENT_WINDOW = 3 # last N checkpoints for averaging

Data Models (using dict structures):

• QualityCheckpoint: timestamp, response_count, test_pass_rate, coverage
• QualityHistory: history list of checkpoints

Core Functions:

1. load_history() -> Dict: Load quality history from HISTORY_FILE, return empty structure if file doesn't exist, handle JSON decode errors gracefully

2. save_history(history: Dict): Ensure .claude directory exists using Path.mkdir(parents=True, exist_ok=True), write history to HISTORY_FILE with json.dump(indent=2) following pattern in codeframe/core/config.py

3. run_tests() -> Dict: Execute pytest with --json-report and --json-report-file flags (following pattern in codeframe/testing/test_runner.py), use subprocess.run with capture_output=True, parse JSON output to extract summary.total, summary.passed, summary.failed, calculate pass_rate = (passed / total * 100) if total > 0 else 0.0, return dict with total, passed, failed, pass_rate

4. get_coverage() -> float: Read coverage.json file (following structure discovered in Agent 2's analysis), extract totals.percent_covered from JSON (the numeric coverage percentage), handle FileNotFoundError if coverage.json doesn't exist, return 0.0 on error with warning message

5. detect_degradation(history: Dict) -> Dict: Implement the degradation detection algorithm from the ticket:

   • Extract last RECENT_WINDOW checkpoints from history
   • Calculate recent_avg_coverage and recent_avg_pass_rate using statistics.mean
   • Find peak_coverage and peak_pass_rate from all historical checkpoints using max
   • Check if recent_avg_coverage < peak_coverage - DEGRADATION_THRESHOLD
   • Check if recent_avg_pass_rate < peak_pass_rate - DEGRADATION_THRESHOLD
   • Return dict with degraded (bool), coverage_drop (float), pass_rate_drop (float), recommendations (list of strings)

6. format_timestamp(ts: str) -> str: Helper to format ISO timestamps for display using datetime.fromisoformat and strftime

CLI Commands (using @app.command() decorator):

1. record(response_count: int = typer.Option(..., "--response-count", "-r", help="Number of AI responses in conversation")):

   • Print "Running tests..." with console.print
   • Call run_tests() to get test metrics
   • Call get_coverage() to get coverage percentage
   • Create checkpoint dict with timestamp (datetime.now().isoformat()), response_count, test_pass_rate, coverage
   • Load history, append checkpoint, save history
   • Call detect_degradation and print results
   • If degraded, print warning in [yellow] with recommendations
   • Print success message with checkpoint summary in [green]
   • Error handling: wrap in try/except, print [red]Error:[/red] and raise typer.Exit(1)

2. check():

   • Load history
   • If history is empty, print info message and exit
   • Call detect_degradation
   • Print current status using Rich formatting
   • If degraded, print alert in [red] with details and recommendations
   • Else print [green] quality is stable message
   • Error handling: same pattern as record

3. stats():

   • Load history
   • If empty, print message and exit
   • Calculate and display statistics:
     • Total checkpoints
     • Coverage: current, peak, average, trend (last 5)
     • Pass rate: current, peak, average, trend (last 5)
     • Response count: average, min, max
   • Use Rich Table to display formatted statistics (following pattern in codeframe/cli.py where Table is imported)
   • Create table with columns: Metric, Current, Peak, Average
   • Add rows for coverage and pass rate with color coding ([green] for good, [yellow] for warning)
   • Print table with console.print
   • Error handling: same pattern

4. reset(confirm: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt")):

   • If not confirm, prompt user with typer.confirm("Are you sure you want to reset quality history?")
   • If confirmed, delete HISTORY_FILE if it exists using Path.unlink(missing_ok=True)
   • Print success message in [green]
   • Error handling: same pattern

Main Entry Point:

• Add if __name__ == "__main__": app() at the end

Error Handling Patterns:

• Follow the pattern from codeframe/cli.py: wrap command bodies in try/except Exception as e
• Use console.print with [red]Error:[/red] markup for errors
• Raise typer.Exit(1) on failures
• Handle specific exceptions (FileNotFoundError, JSONDecodeError, subprocess errors) with helpful messages

📄 .claude/quality_history.json ^(NEW) 🔗

Create an initial quality history JSON file with an empty history array. This file will be populated by the quality-ratchet.py script when checkpoints are recorded.

Structure:

• Root object with a "history" key containing an empty array
• This matches the data format specified in the GitHub issue

Note: The script will create this file automatically if it doesn't exist, but having an initial file with the correct structure serves as documentation and prevents confusion.

📄 CLAUDE.md ^(MODIFY) 🔗

Add a new "Quality Ratchet Checkpoints" section immediately after the "Commands" section (after line 43).

New Section Content:

• Section header: "## Quality Ratchet Checkpoints"
• Brief explanation: "Track code quality metrics across AI conversation sessions to detect context window degradation."
• Command reference: python tools/quality-ratchet.py record --response-count N (run after test suite)
• When to run checkpoints:
  • After completing a feature or bug fix
  • After running the full test suite with coverage
  • Before requesting code review
  • When conversation reaches 10+ responses
• What gets tracked: test pass rate, coverage percentage, response count
• Degradation alerts: System alerts when coverage or pass rate drops >10% from historical peak
• Recommendation: Reset AI context when degradation is detected
• Cross-reference to TESTING.md for quality gates and thresholds

Additional Update:

• In the "Commands" section, add a line for the quality ratchet tool:
  • python tools/quality-ratchet.py check # Check quality metrics

This integration point makes the quality ratchet system discoverable to both human developers and AI agents working with the codebase, following the pattern established in the existing CLAUDE.md structure.

📄 TESTING.md ^(MODIFY) 🔗

Add a new section "Quality Ratchet Integration" after the "Automated Test Suite Verification" section (after line 310).

New Section Content:

• Section header: "### Test 11: Quality Ratchet System"
• Subsection "11.1 Record Quality Checkpoint":
  • Run: python tools/quality-ratchet.py record --response-count 5
  • Verify: Tests execute automatically
  • Verify: Coverage is collected
  • Verify: Checkpoint saved to .claude/quality_history.json
  • Verify: Success message displayed with metrics
• Subsection "11.2 Check Quality Status":
  • Run: python tools/quality-ratchet.py check
  • Verify: Current quality status displayed
  • Verify: No degradation alerts (on first run)
• Subsection "11.3 View Statistics":
  • Run: python tools/quality-ratchet.py stats
  • Verify: Table displays with coverage and pass rate metrics
  • Verify: Shows current, peak, and average values
• Subsection "11.4 Degradation Detection":
  • Simulate degradation by temporarily commenting out tests
  • Run record command again
  • Verify: Degradation alert appears
  • Verify: Recommendations include context reset suggestion
  • Restore commented tests
• Subsection "11.5 Reset History":
  • Run: python tools/quality-ratchet.py reset --yes
  • Verify: History file cleared
  • Verify: Success message displayed

Update to Definition of Done:

• Add checkbox: "✅ Quality ratchet system tracks metrics across sessions"
• Add checkbox: "✅ Degradation detection alerts when quality drops >10%"

This provides comprehensive testing documentation for the quality ratchet system, following the established pattern in TESTING.md with detailed verification steps.

📄 .github/workflows/quality-check.yml ^(NEW) 🔗

References

• 📄 .github/workflows/claude-code-review.yml
• 📄 pytest.ini
• 📄 .coveragerc

Create a new GitHub Actions workflow for automated quality tracking and trending in CI/CD.

Workflow Configuration:

• name: "Quality Check"
• Triggers:
  • on.pull_request: types [opened, synchronize]
  • on.push: branches [main, develop]
  • on.schedule: cron for daily runs (e.g., "0 0 * * *" for midnight UTC)
  • on.workflow_dispatch: for manual triggers

Job: quality-check

• runs-on: ubuntu-latest
• permissions: contents: read, pull-requests: write (for posting comments)

Steps:

1. Checkout repository (actions/checkout@v4 with fetch-depth: 0 for history)

2. Set up Python (actions/setup-python@v4 with python-version: "3.11")

3. Install dependencies:

   • Run: pip install -e ".[dev]"
   • This installs pytest, pytest-cov, pytest-json-report, and all dev dependencies from pyproject.toml

4. Run tests with coverage:

   • Run: pytest --cov=codeframe --cov-report=json --cov-report=term --json-report --json-report-file=.report.json -v
   • Set ANTHROPIC_API_KEY: "test-key" in env (following pattern from .claude/settings.local.json)
   • This generates both coverage.json and .report.json files

5. Record quality checkpoint:

   • Run: python tools/quality-ratchet.py record --response-count 1
   • Note: Use response-count=1 for automated CI runs (not a conversation)
   • Capture output for later use

6. Check for degradation:

   • Run: python tools/quality-ratchet.py check
   • Continue on error (don't fail the build, just warn)
   • Capture output

7. Generate statistics:

   • Run: python tools/quality-ratchet.py stats
   • Capture output for PR comment

8. Post PR comment (if pull_request event):

   • Use actions/github-script@v7 or gh CLI to post comment
   • Include quality metrics summary
   • Include degradation warnings if any
   • Include link to full coverage report
   • Conditional: only run on pull_request events

9. Upload coverage artifacts:

   • Use actions/upload-artifact@v3
   • Upload htmlcov/ directory
   • Upload coverage.json
   • Upload .claude/quality_history.json for trending

10. Fail on critical degradation (optional):

    • Check if degradation detected
    • Exit with failure code if critical thresholds breached
    • Make this configurable via workflow input

Environment Variables:

• ANTHROPIC_API_KEY: Set to "test-key" for test runs (following existing pattern)

Artifacts:

• Coverage HTML report (htmlcov/)
• Coverage JSON (coverage.json)
• Quality history (.claude/quality_history.json)
• Test report (.report.json)

This workflow provides automated quality tracking on every PR and push to main, with daily trending data collection. It follows GitHub Actions best practices and integrates with the existing test infrastructure.

📄 README.md ^(MODIFY) 🔗

Add a new section about the Quality Ratchet System in the README to make it discoverable to new contributors.

Location: Add after the testing/development section (typically near the end of the README)

New Section Content:

• Section header: "## Quality Ratchet System"
• Brief description: "CodeFRAME includes an automated quality tracking system that monitors code quality metrics across development sessions to detect context window degradation."
• Key features bullet list:
  • Tracks test pass rate, coverage percentage, and conversation length
  • Detects quality degradation (>10% drops from historical peak)
  • Recommends context resets when degradation occurs
  • Integrates with CI/CD for automated trending
• Usage subsection:
  • Record checkpoint: python tools/quality-ratchet.py record --response-count N
  • Check status: python tools/quality-ratchet.py check
  • View statistics: python tools/quality-ratchet.py stats
  • Reset history: python tools/quality-ratchet.py reset --yes
• When to use:
  • After completing features or bug fixes
  • Before requesting code review
  • When AI conversations exceed 10 responses
  • Automatically in CI/CD on every PR
• Reference: "See TESTING.md for detailed testing procedures and CLAUDE.md for AI workflow integration."

This makes the quality ratchet system visible in the main project documentation, helping new contributors understand and use the tool effectively.

📄 .gitignore ^(MODIFY) 🔗

Add entries to .gitignore to exclude temporary test artifacts and quality ratchet working files that shouldn't be committed.

Add the following entries:

• .report.json - pytest-json-report output file (temporary test artifact)
• .pytest_cache/ - pytest cache directory (if not already present)
• htmlcov/ - coverage HTML report directory (if not already present)
• coverage.xml - coverage XML report (if not already present)

Note: Do NOT add .claude/quality_history.json to .gitignore - this file should be committed to track quality trends over time in the repository.

Verify that existing entries already cover:

• __pycache__/ directories
• *.pyc files
• .coverage file
• venv/ and env/ directories

This ensures that temporary test artifacts are not accidentally committed while preserving the quality history for team visibility and trending analysis.

Diagram

sequenceDiagram
    participant Dev as Developer/AI
    participant QR as quality-ratchet.py
    participant Tests as pytest + coverage
    participant History as .claude/quality_history.json
    participant CI as GitHub Actions

    Note over Dev,CI: Manual Workflow
    Dev->>QR: record --response-count N
    QR->>Tests: Run pytest with --json-report
    Tests-->>QR: Test results (pass/fail counts)
    QR->>Tests: Read coverage.json
    Tests-->>QR: Coverage percentage
    QR->>QR: Calculate metrics & detect degradation
    QR->>History: Append checkpoint
    QR->>Dev: Display status + alerts
    
    Note over Dev,CI: Check Status
    Dev->>QR: check
    QR->>History: Load history
    QR->>QR: Analyze last 3 vs peak
    QR->>Dev: Show degradation status
    
    Note over Dev,CI: CI/CD Integration
    CI->>Tests: Run pytest with coverage
    CI->>QR: record --response-count 1
    QR->>History: Update checkpoint
    QR->>CI: Return status
    CI->>CI: Post PR comment with metrics
    
    Note over Dev,CI: Degradation Detected
    QR->>Dev: ⚠️ Coverage dropped 12%
    QR->>Dev: 💡 Recommend: Reset AI context

Loading

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established testing infrastructure with pytest, pytest-json-report, and coverage.py. The CLI uses Typer with Rich for output. Currently, there's no automated quality tracking system, and GitHub workflows don't run tests automatically. The quality ratchet system needs to integrate with existing patterns while filling this gap.

### Approach

Create a standalone Python script `tools/quality-ratchet.py` that uses Typer CLI to track quality metrics (test pass rate, coverage %, response count) in `.claude/quality_history.json`. The script will parse pytest-json-report output and coverage.json to extract metrics, implement a degradation detection algorithm comparing recent averages to historical peaks, and provide commands for recording, checking, viewing stats, and resetting history. Integration points include CLAUDE.md documentation updates and a new GitHub Actions workflow for CI/CD trending.

### Reasoning

I explored the repository structure, examined the testing infrastructure (pytest.ini, .coveragerc, pyproject.toml), analyzed existing CLI patterns in codeframe/cli.py, studied the TestRunner implementation to understand pytest-json-report usage, reviewed coverage.json structure, examined script patterns for JSON I/O and Rich console usage, and identified integration points in CLAUDE.md and GitHub workflows.

## Mermaid Diagram

sequenceDiagram
    participant Dev as Developer/AI
    participant QR as quality-ratchet.py
    participant Tests as pytest + coverage
    participant History as .claude/quality_history.json
    participant CI as GitHub Actions

    Note over Dev,CI: Manual Workflow
    Dev->>QR: record --response-count N
    QR->>Tests: Run pytest with --json-report
    Tests-->>QR: Test results (pass/fail counts)
    QR->>Tests: Read coverage.json
    Tests-->>QR: Coverage percentage
    QR->>QR: Calculate metrics & detect degradation
    QR->>History: Append checkpoint
    QR->>Dev: Display status + alerts
    
    Note over Dev,CI: Check Status
    Dev->>QR: check
    QR->>History: Load history
    QR->>QR: Analyze last 3 vs peak
    QR->>Dev: Show degradation status
    
    Note over Dev,CI: CI/CD Integration
    CI->>Tests: Run pytest with coverage
    CI->>QR: record --response-count 1
    QR->>History: Update checkpoint
    QR->>CI: Return status
    CI->>CI: Post PR comment with metrics
    
    Note over Dev,CI: Degradation Detected
    QR->>Dev: ⚠️ Coverage dropped 12%
    QR->>Dev: 💡 Recommend: Reset AI context

## Proposed File Changes

### tools(NEW)

Create a new directory `tools/` at the repository root to house the quality ratchet script and any future tooling scripts. This follows common Python project conventions for separating development tools from the main package code.

### tools/__init__.py(NEW)

Create an empty `__init__.py` file to make the tools directory a Python package. This allows for potential future imports and maintains Python package conventions.

### tools/quality-ratchet.py(NEW)

References: 

- codeframe/cli.py
- codeframe/core/config.py
- codeframe/testing/test_runner.py

Create the main quality ratchet script with the following structure:

**Imports and Setup:**
- Import json, subprocess, sys, datetime, statistics from stdlib
- Import Path from pathlib
- Import typer, Console and Table from rich (following patterns in `codeframe/cli.py`)
- Import Optional, List, Dict, Any from typing
- Create Typer app instance with name="quality-ratchet" and help text
- Create Rich Console instance for styled output

**Constants:**
- HISTORY_FILE = Path(".claude/quality_history.json")
- DEGRADATION_THRESHOLD = 10.0  # percentage
- RECENT_WINDOW = 3  # last N checkpoints for averaging

**Data Models (using dict structures):**
- QualityCheckpoint: timestamp, response_count, test_pass_rate, coverage
- QualityHistory: history list of checkpoints

**Core Functions:**

1. `load_history() -> Dict`: Load quality history from HISTORY_FILE, return empty structure if file doesn't exist, handle JSON decode errors gracefully

2. `save_history(history: Dict)`: Ensure .claude directory exists using Path.mkdir(parents=True, exist_ok=True), write history to HISTORY_FILE with json.dump(indent=2) following pattern in `codeframe/core/config.py`

3. `run_tests() -> Dict`: Execute pytest with --json-report and --json-report-file flags (following pattern in `codeframe/testing/test_runner.py`), use subprocess.run with capture_output=True, parse JSON output to extract summary.total, summary.passed, summary.failed, calculate pass_rate = (passed / total * 100) if total > 0 else 0.0, return dict with total, passed, failed, pass_rate

4. `get_coverage() -> float`: Read coverage.json file (following structure discovered in Agent 2's analysis), extract totals.percent_covered from JSON (the numeric coverage percentage), handle FileNotFoundError if coverage.json doesn't exist, return 0.0 on error with warning message

5. `detect_degradation(history: Dict) -> Dict`: Implement the degradation detection algorithm from the ticket:
   - Extract last RECENT_WINDOW checkpoints from history
   - Calculate recent_avg_coverage and recent_avg_pass_rate using statistics.mean
   - Find peak_coverage and peak_pass_rate from all historical checkpoints using max
   - Check if recent_avg_coverage < peak_coverage - DEGRADATION_THRESHOLD
   - Check if recent_avg_pass_rate < peak_pass_rate - DEGRADATION_THRESHOLD
   - Return dict with degraded (bool), coverage_drop (float), pass_rate_drop (float), recommendations (list of strings)

6. `format_timestamp(ts: str) -> str`: Helper to format ISO timestamps for display using datetime.fromisoformat and strftime

**CLI Commands (using @app.command() decorator):**

1. `record(response_count: int = typer.Option(..., "--response-count", "-r", help="Number of AI responses in conversation"))`: 
   - Print "Running tests..." with console.print
   - Call run_tests() to get test metrics
   - Call get_coverage() to get coverage percentage
   - Create checkpoint dict with timestamp (datetime.now().isoformat()), response_count, test_pass_rate, coverage
   - Load history, append checkpoint, save history
   - Call detect_degradation and print results
   - If degraded, print warning in [yellow] with recommendations
   - Print success message with checkpoint summary in [green]
   - Error handling: wrap in try/except, print [red]Error:[/red] and raise typer.Exit(1)

2. `check()`: 
   - Load history
   - If history is empty, print info message and exit
   - Call detect_degradation
   - Print current status using Rich formatting
   - If degraded, print alert in [red] with details and recommendations
   - Else print [green] quality is stable message
   - Error handling: same pattern as record

3. `stats()`: 
   - Load history
   - If empty, print message and exit
   - Calculate and display statistics:
     - Total checkpoints
     - Coverage: current, peak, average, trend (last 5)
     - Pass rate: current, peak, average, trend (last 5)
     - Response count: average, min, max
   - Use Rich Table to display formatted statistics (following pattern in `codeframe/cli.py` where Table is imported)
   - Create table with columns: Metric, Current, Peak, Average
   - Add rows for coverage and pass rate with color coding ([green] for good, [yellow] for warning)
   - Print table with console.print
   - Error handling: same pattern

4. `reset(confirm: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"))`: 
   - If not confirm, prompt user with typer.confirm("Are you sure you want to reset quality history?")
   - If confirmed, delete HISTORY_FILE if it exists using Path.unlink(missing_ok=True)
   - Print success message in [green]
   - Error handling: same pattern

**Main Entry Point:**
- Add `if __name__ == "__main__": app()` at the end

**Error Handling Patterns:**
- Follow the pattern from `codeframe/cli.py`: wrap command bodies in try/except Exception as e
- Use console.print with [red]Error:[/red] markup for errors
- Raise typer.Exit(1) on failures
- Handle specific exceptions (FileNotFoundError, JSONDecodeError, subprocess errors) with helpful messages

### .claude/quality_history.json(NEW)

Create an initial quality history JSON file with an empty history array. This file will be populated by the quality-ratchet.py script when checkpoints are recorded.

Structure:
- Root object with a "history" key containing an empty array
- This matches the data format specified in the GitHub issue

Note: The script will create this file automatically if it doesn't exist, but having an initial file with the correct structure serves as documentation and prevents confusion.

### CLAUDE.md(MODIFY)

Add a new "Quality Ratchet Checkpoints" section immediately after the "Commands" section (after line 43).

**New Section Content:**
- Section header: "## Quality Ratchet Checkpoints"
- Brief explanation: "Track code quality metrics across AI conversation sessions to detect context window degradation."
- Command reference: `python tools/quality-ratchet.py record --response-count N` (run after test suite)
- When to run checkpoints:
  - After completing a feature or bug fix
  - After running the full test suite with coverage
  - Before requesting code review
  - When conversation reaches 10+ responses
- What gets tracked: test pass rate, coverage percentage, response count
- Degradation alerts: System alerts when coverage or pass rate drops >10% from historical peak
- Recommendation: Reset AI context when degradation is detected
- Cross-reference to TESTING.md for quality gates and thresholds

**Additional Update:**
- In the "Commands" section, add a line for the quality ratchet tool:
  - `python tools/quality-ratchet.py check  # Check quality metrics`

This integration point makes the quality ratchet system discoverable to both human developers and AI agents working with the codebase, following the pattern established in the existing CLAUDE.md structure.

### TESTING.md(MODIFY)

Add a new section "Quality Ratchet Integration" after the "Automated Test Suite Verification" section (after line 310).

**New Section Content:**
- Section header: "### Test 11: Quality Ratchet System"
- Subsection "11.1 Record Quality Checkpoint":
  - Run: `python tools/quality-ratchet.py record --response-count 5`
  - Verify: Tests execute automatically
  - Verify: Coverage is collected
  - Verify: Checkpoint saved to `.claude/quality_history.json`
  - Verify: Success message displayed with metrics
- Subsection "11.2 Check Quality Status":
  - Run: `python tools/quality-ratchet.py check`
  - Verify: Current quality status displayed
  - Verify: No degradation alerts (on first run)
- Subsection "11.3 View Statistics":
  - Run: `python tools/quality-ratchet.py stats`
  - Verify: Table displays with coverage and pass rate metrics
  - Verify: Shows current, peak, and average values
- Subsection "11.4 Degradation Detection":
  - Simulate degradation by temporarily commenting out tests
  - Run record command again
  - Verify: Degradation alert appears
  - Verify: Recommendations include context reset suggestion
  - Restore commented tests
- Subsection "11.5 Reset History":
  - Run: `python tools/quality-ratchet.py reset --yes`
  - Verify: History file cleared
  - Verify: Success message displayed

**Update to Definition of Done:**
- Add checkbox: "✅ Quality ratchet system tracks metrics across sessions"
- Add checkbox: "✅ Degradation detection alerts when quality drops >10%"

This provides comprehensive testing documentation for the quality ratchet system, following the established pattern in TESTING.md with detailed verification steps.

### .github/workflows/quality-check.yml(NEW)

References: 

- .github/workflows/claude-code-review.yml
- pytest.ini
- .coveragerc

Create a new GitHub Actions workflow for automated quality tracking and trending in CI/CD.

**Workflow Configuration:**
- name: "Quality Check"
- Triggers:
  - on.pull_request: types [opened, synchronize]
  - on.push: branches [main, develop]
  - on.schedule: cron for daily runs (e.g., "0 0 * * *" for midnight UTC)
  - on.workflow_dispatch: for manual triggers

**Job: quality-check**
- runs-on: ubuntu-latest
- permissions: contents: read, pull-requests: write (for posting comments)

**Steps:**
1. Checkout repository (actions/checkout@v4 with fetch-depth: 0 for history)

2. Set up Python (actions/setup-python@v4 with python-version: "3.11")

3. Install dependencies:
   - Run: pip install -e ".[dev]"
   - This installs pytest, pytest-cov, pytest-json-report, and all dev dependencies from `pyproject.toml`

4. Run tests with coverage:
   - Run: pytest --cov=codeframe --cov-report=json --cov-report=term --json-report --json-report-file=.report.json -v
   - Set ANTHROPIC_API_KEY: "test-key" in env (following pattern from `.claude/settings.local.json`)
   - This generates both coverage.json and .report.json files

5. Record quality checkpoint:
   - Run: python tools/quality-ratchet.py record --response-count 1
   - Note: Use response-count=1 for automated CI runs (not a conversation)
   - Capture output for later use

6. Check for degradation:
   - Run: python tools/quality-ratchet.py check
   - Continue on error (don't fail the build, just warn)
   - Capture output

7. Generate statistics:
   - Run: python tools/quality-ratchet.py stats
   - Capture output for PR comment

8. Post PR comment (if pull_request event):
   - Use actions/github-script@v7 or gh CLI to post comment
   - Include quality metrics summary
   - Include degradation warnings if any
   - Include link to full coverage report
   - Conditional: only run on pull_request events

9. Upload coverage artifacts:
   - Use actions/upload-artifact@v3
   - Upload htmlcov/ directory
   - Upload coverage.json
   - Upload .claude/quality_history.json for trending

10. Fail on critical degradation (optional):
    - Check if degradation detected
    - Exit with failure code if critical thresholds breached
    - Make this configurable via workflow input

**Environment Variables:**
- ANTHROPIC_API_KEY: Set to "test-key" for test runs (following existing pattern)

**Artifacts:**
- Coverage HTML report (htmlcov/)
- Coverage JSON (coverage.json)
- Quality history (.claude/quality_history.json)
- Test report (.report.json)

This workflow provides automated quality tracking on every PR and push to main, with daily trending data collection. It follows GitHub Actions best practices and integrates with the existing test infrastructure.

### README.md(MODIFY)

Add a new section about the Quality Ratchet System in the README to make it discoverable to new contributors.

**Location:** Add after the testing/development section (typically near the end of the README)

**New Section Content:**
- Section header: "## Quality Ratchet System"
- Brief description: "CodeFRAME includes an automated quality tracking system that monitors code quality metrics across development sessions to detect context window degradation."
- Key features bullet list:
  - Tracks test pass rate, coverage percentage, and conversation length
  - Detects quality degradation (>10% drops from historical peak)
  - Recommends context resets when degradation occurs
  - Integrates with CI/CD for automated trending
- Usage subsection:
  - Record checkpoint: `python tools/quality-ratchet.py record --response-count N`
  - Check status: `python tools/quality-ratchet.py check`
  - View statistics: `python tools/quality-ratchet.py stats`
  - Reset history: `python tools/quality-ratchet.py reset --yes`
- When to use:
  - After completing features or bug fixes
  - Before requesting code review
  - When AI conversations exceed 10 responses
  - Automatically in CI/CD on every PR
- Reference: "See TESTING.md for detailed testing procedures and CLAUDE.md for AI workflow integration."

This makes the quality ratchet system visible in the main project documentation, helping new contributors understand and use the tool effectively.

### .gitignore(MODIFY)

Add entries to .gitignore to exclude temporary test artifacts and quality ratchet working files that shouldn't be committed.

**Add the following entries:**
- `.report.json` - pytest-json-report output file (temporary test artifact)
- `.pytest_cache/` - pytest cache directory (if not already present)
- `htmlcov/` - coverage HTML report directory (if not already present)
- `coverage.xml` - coverage XML report (if not already present)

**Note:** Do NOT add `.claude/quality_history.json` to .gitignore - this file should be committed to track quality trends over time in the repository.

Verify that existing entries already cover:
- `__pycache__/` directories
- `*.pyc` files
- `.coverage` file
- `venv/` and `env/` directories

This ensures that temporary test artifacts are not accidentally committed while preserving the quality history for team visibility and trending analysis.

---

Developer Humor

🎯 Quality's the game, ratchets the name,\n> We track every metric, no two are the same!\n> When coverage drops and tests start to fail,\n> Our ratchet alerts you—no need to derail.\n> \n> 📊 From peak to recent, we measure with care,\n> Context degradation? We'll catch it right there!\n> So code with confidence, let metrics be your guide,\n> The Quality Ratchet's got you—right by your side! 🚀

---

Execution Information

Branch: main
Commit: d0e2480

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[frankbria]

Resolved in PR #20 - Quality Ratchet System implemented with scripts/quality-ratchet.py. Tracks coverage %, pass rate, response count with degradation detection (>10% drop). Includes auto-suggestion for context resets. 14/14 tests passing.