Integrate QualityTracker with workflow system by frankbria · Pull Request #151 · frankbria/codeframe

frankbria · 2025-12-26T22:01:34Z

Add QualityTracker initialization to WorkerAgent with lazy loading
Record quality metrics after quality gates run in complete_task()
Check for quality degradation before allowing task completion
Create blockers when quality degrades >10% from peak
Add should_recommend_context_reset() method for proactive monitoring
Integrate quality stats into checkpoint metadata
Add 14 comprehensive integration tests

This enables continuous quality tracking across AI sessions, detecting degradation patterns and recommending context resets when quality drops significantly.

Closing Issue #110

Summary by CodeRabbit

New Features
- Quality metrics are now tracked and recorded automatically when tasks complete.
- Quality degradation is detected and prevents task completion if it exceeds thresholds.
- Context reset recommendations provided based on conversation length and quality trends.
- Checkpoint metadata now includes quality statistics and trend information.
Tests
- Comprehensive integration tests added for quality tracking and degradation detection workflows.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

- Add QualityTracker initialization to WorkerAgent with lazy loading - Record quality metrics after quality gates run in complete_task() - Check for quality degradation before allowing task completion - Create blockers when quality degrades >10% from peak - Add should_recommend_context_reset() method for proactive monitoring - Integrate quality stats into checkpoint metadata - Add 14 comprehensive integration tests This enables continuous quality tracking across AI sessions, detecting degradation patterns and recommending context resets when quality drops significantly.

coderabbitai · 2025-12-26T22:01:44Z

Walkthrough

This PR integrates quality tracking into WorkerAgent by adding response counting, quality metrics recording, and degradation detection. Task completion now triggers quality gate evaluation and blocker creation when degradation exceeds thresholds. CheckpointMetadata is extended with quality statistics and trends, and a comprehensive integration test suite validates the end-to-end workflow.

Changes

Cohort / File(s)	Summary
WorkerAgent Quality Tracking `codeframe/agents/worker_agent.py`	Added `response_count` and `quality_tracker` attributes; implemented `_ensure_quality_tracker()` for lazy initialization from project database; enhanced `execute_task()` and `complete_task()` to increment response count and record metrics; added `_record_quality_metrics()` to extract test pass rate and coverage; added `_check_quality_degradation()` to query tracker and detect threshold violations; added `_create_degradation_blocker()` to format and persist blocker records; added `should_recommend_context_reset()` to advise on context reset based on response count and degradation status.
CheckpointMetadata Extensions `codeframe/core/models.py`	Added `quality_stats` (Optional[Dict[str, Any]]) and `quality_trend` (Optional[str]) fields to `CheckpointMetadata` to store quality metrics and trend direction.
Checkpoint Manager Integration `codeframe/lib/checkpoint_manager.py`	Modified `_generate_metadata()` to fetch quality statistics from QualityTracker; populates `quality_stats` with current, peak, average, and total checkpoint counts; populates `quality_trend` with trend status or defaults to "insufficient_data"; includes error handling to proceed without quality data if retrieval fails.
Integration Test Suite `tests/integration/test_quality_tracker_integration.py`	Added 5 test classes with 11 test methods covering: quality metrics recording after gates pass; degradation detection triggering blocker creation; context reset recommendations under various conditions (high response count, degradation, healthy state); checkpoint metadata inclusion of quality stats and trends; `_ensure_quality_tracker()` caching and initialization; real QualityTracker history file creation and degradation threshold logic.

Sequence Diagram

sequenceDiagram
    autonumber
    actor User
    participant WorkerAgent
    participant Database
    participant QualityTracker
    participant BlockerService
    
    User->>WorkerAgent: complete_task(task)
    activate WorkerAgent
    
    Note over WorkerAgent: Execute LLM interactions<br/>(increment response_count)
    WorkerAgent->>WorkerAgent: response_count += 1
    
    Note over WorkerAgent: Run quality gates
    WorkerAgent->>WorkerAgent: quality_result = run_quality_gates()
    
    WorkerAgent->>WorkerAgent: _ensure_quality_tracker()
    WorkerAgent->>Database: Query project_id, workspace_path
    activate Database
    Database-->>WorkerAgent: workspace_path
    deactivate Database
    WorkerAgent->>QualityTracker: Initialize with workspace_path
    activate QualityTracker
    
    WorkerAgent->>WorkerAgent: _record_quality_metrics(quality_result)
    WorkerAgent->>QualityTracker: Record test_pass_rate, coverage
    deactivate QualityTracker
    
    WorkerAgent->>WorkerAgent: degradation = _check_quality_degradation()
    activate QualityTracker
    QualityTracker-->>WorkerAgent: Compare against threshold
    deactivate QualityTracker
    
    alt Degradation Detected
        WorkerAgent->>WorkerAgent: _create_degradation_blocker(task, degradation)
        WorkerAgent->>BlockerService: Create SYNC blocker
        activate BlockerService
        BlockerService->>Database: Persist blocker record
        Database-->>BlockerService: blocker_id
        deactivate BlockerService
        BlockerService-->>WorkerAgent: blocker_id
        WorkerAgent-->>User: Return blocked result
    else No Degradation
        WorkerAgent-->>User: Return success result
    end
    
    deactivate WorkerAgent

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

[P2] Integrate QualityTracker into WorkerAgent workflow #110: Directly addresses integration of QualityTracker into WorkerAgent with metric recording, degradation detection, blocker creation, and context-reset recommendations—this PR implements the exact changes described in the issue.

Possibly related PRs

fix: Reorganize test suite and fix API test failures #22: Modifies WorkerAgent __init__ and adds project_id precondition checks; touches the same class being extended with quality tracking attributes and initialization logic.
Add token tracking to WorkerAgent execute_task method #126: Changes execute_task in WorkerAgent; overlaps with this PR's modifications to execute_task for response count tracking.
Implement WorkerAgent execute_task method #125: Modifies WorkerAgent state and execute_task implementation; related through shared usage tracking and agent-state management patterns.

Poem

🐰 Hopping through the code with glee,
Quality tracked for all to see,
Metrics dance and trackers grow,
Degradation? Blockers steal the show!
Context resets, checkpoints shine—
A rabbit's gift: code divine! 🌟

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Integrate QualityTracker with workflow system' clearly and accurately describes the main change—adding quality tracking integration to the worker agent and checkpoint system.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch claude/integrate-quality-tracker-4m3fn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

claude · 2025-12-26T22:03:41Z

PR Review: Integrate QualityTracker with workflow system

This is a well-structured integration that adds continuous quality tracking to the WorkerAgent. The implementation demonstrates good software engineering practices with comprehensive testing and thoughtful design. Here's my detailed review:

✅ Strengths

1. Architecture & Design

Lazy initialization pattern (_ensure_quality_tracker()) is excellent - only initializes when project context is available
Clean separation of concerns - quality tracking is isolated in separate methods
Graceful degradation - failures in quality tracking don't crash task completion
Comprehensive integration - touches all the right points: task execution, completion, checkpoints

2. Code Quality

Excellent documentation - clear docstrings with examples throughout
Type hints - proper use of Optional[] and Dict[str, Any]
Logging - appropriate debug/info/warning levels
Error handling - try/except blocks with proper fallbacks

3. Testing

14 comprehensive integration tests covering all major workflows
Good test organization - grouped into logical test classes
Mocking strategy - appropriate use of mocks vs real objects
Edge cases covered - no database, no task, initialization failures

⚠️ Issues & Concerns

HIGH PRIORITY

1. Regex Import Inside Loop (worker_agent.py:1124)

# BEFORE (inside loop)
for failure in quality_result.failures:
    ...
    import re  # ❌ Imported on every iteration
    failed_match = re.search(...)

Issue: The import re statement is inside the loop, causing unnecessary overhead.

Fix: Move to top of file with other imports:

# At top of file (line 3-8 area)
import re
from datetime import datetime, timedelta, timezone

2. Hardcoded Assumption When No Failures (worker_agent.py:1144-1148)

# If no failures, assume perfect scores
if not quality_result.failures:
    test_pass_rate = 100.0
    coverage_percentage = 100.0  # ❌ Assumption may be incorrect

Issue: Assuming 100% coverage when gates pass is not accurate. Gates might pass with 85% coverage (the threshold), but this would record 100%.

Impact: Skews historical data, making trend analysis less useful.

Recommendation:

Try to extract actual metrics from quality gate results even when passed
OR set to a sentinel value (like None or -1) to indicate "not measured"
OR query the actual test runner output for real values

3. Race Condition in Quality Tracker Initialization (worker_agent.py:129-130)

if self.quality_tracker is not None:
    return self.quality_tracker

Issue: In async context, two concurrent calls could both pass the None check and create multiple trackers.

Impact: Low probability in single-threaded execution, but could cause issues in concurrent scenarios.

Fix: Use a lock or make initialization atomic:

async def _ensure_quality_tracker(self) -> Optional[QualityTracker]:
    if self.quality_tracker is not None:
        return self.quality_tracker
    
    async with self._quality_tracker_lock:  # Add lock
        if self.quality_tracker is not None:  # Double-check
            return self.quality_tracker
        # ... initialization

MEDIUM PRIORITY

4. SQL Injection Risk (worker_agent.py:140)

cursor.execute("SELECT workspace_path FROM projects WHERE id = ?", (project_id,))

Status: ✅ Actually safe - parameterized query is used correctly.

Note: This is fine, just flagging for security review. Consider using the repository pattern consistently though:

# More consistent with codebase architecture
project = self.db.projects_repo.get_project(project_id)
workspace_path = project.workspace_path

5. Inconsistent Error Handling (worker_agent.py:1101-1104)

tracker = self._ensure_quality_tracker()
if not tracker:
    logger.debug("Quality tracker not available, skipping metrics recording")
    return

Issue: Silent failure - no indication to user that quality tracking isn't working.

Recommendation: Log at INFO or WARNING level on first failure, DEBUG on subsequent calls:

if not tracker:
    if not hasattr(self, '_quality_tracking_warning_shown'):
        logger.warning("Quality tracker not available - ensure task has project context")
        self._quality_tracking_warning_shown = True
    return

6. Magic Numbers (worker_agent.py:942, complete_task method)

degradation_result = self._check_quality_degradation()
# Uses default threshold_percent=10.0 from method signature

Issue: Hardcoded 10% threshold should be configurable.

Recommendation:

# Add to __init__
self.quality_degradation_threshold = float(
    os.getenv("QUALITY_DEGRADATION_THRESHOLD", "10.0")
)

# Use in complete_task
degradation_result = self._check_quality_degradation(
    threshold_percent=self.quality_degradation_threshold
)

7. Incomplete Test Coverage (tests/integration/test_quality_tracker_integration.py:354)

The test file appears truncated at line 354 in the diff. The last test test_ensure_quality_tracker_in... is incomplete.

Recommendation: Ensure all tests are complete and passing before merge.

LOW PRIORITY

8. Type Annotation Inconsistency

project_root: Optional[Any] = None  # ❌ Should be Path or str

Recommendation: Use proper type:

from pathlib import Path
project_root: Optional[Path] = None

9. Duplicate Language Detection Import
The LanguageDetector import is inside _record_quality_metrics method (line 1099). Consider moving to top-level imports if used frequently.

🔒 Security Review

✅ No security concerns identified

Parameterized SQL queries used correctly
No user input directly used in commands
File paths validated through existing project constraints
No credential handling

🎯 Performance Considerations

Positive:

Lazy initialization prevents overhead when not needed
Quality tracker check short-circuits if already initialized
Graceful failure paths prevent blocking task completion

Concerns:

Regex parsing in _record_quality_metrics could be slow with many failures
Consider caching language detection results per project

📊 Test Coverage Assessment

Coverage: Excellent (14 integration tests)

Areas tested:

✅ Metrics recording after task completion
✅ Blocker creation on degradation
✅ Context reset recommendations
✅ Checkpoint metadata integration
✅ Quality tracker initialization
✅ Real tracker with file system
✅ Edge cases (no DB, no task, etc.)

Missing coverage:

⚠️ Concurrent access scenarios
⚠️ Large-scale regression testing with real quality gate outputs
⚠️ Response count edge cases (overflow, negative values)

📝 Best Practices & Style

Adherence to CLAUDE.md:
✅ Matches Python 3.11+ standards
✅ AsyncIO properly used
✅ Follows existing database patterns
✅ Documentation follows project style
✅ Test structure matches existing patterns

Minor style notes:

Consider using from __future__ import annotations for cleaner type hints
Some log messages could be more consistent in format

🚀 Recommendations for Merge

Required before merge:

✅ Fix regex import location (move to top of file)
⚠️ Address hardcoded 100% coverage assumption
⚠️ Complete truncated test at line 354

Recommended before merge:
4. Consider adding async lock for quality tracker initialization
5. Improve error logging visibility
6. Make degradation threshold configurable

Can defer to follow-up:
7. Type annotation cleanup
8. Performance optimizations for regex parsing
9. Additional concurrent access tests

🎉 Overall Assessment

Rating: APPROVE with minor changes ⭐⭐⭐⭐

This is a solid integration that adds valuable quality tracking capabilities. The architecture is sound, testing is comprehensive, and the code quality is high. The issues identified are mostly minor and easily addressable.

The lazy initialization pattern and graceful error handling show thoughtful design. The comprehensive test suite demonstrates good engineering discipline.

Primary concerns:

Regex import location (trivial fix)
Coverage assumption when gates pass (data accuracy issue)
Incomplete test in file (likely just diff truncation)

Once the regex import is fixed and the coverage assumption is addressed, this is ready to merge.

Great work on this integration! 🎊

greptile-apps · 2025-12-26T22:04:51Z

Greptile Summary

This PR successfully integrates QualityTracker into the WorkerAgent workflow to enable continuous quality monitoring and degradation detection across AI sessions.

Key Changes:

Added lazy initialization of QualityTracker in WorkerAgent via _ensure_quality_tracker() method that initializes when database and project context are available
Implemented response count tracking (self.response_count++) in execute_task() to monitor AI conversation length
Integrated quality metric recording in complete_task() workflow after quality gates run, extracting test pass rates and coverage from gate results
Added degradation detection logic that blocks task completion when quality drops >10% from peak, creating SYNC blockers with detailed recommendations
Implemented should_recommend_context_reset() method for proactive quality monitoring based on response count and degradation patterns
Extended CheckpointMetadata model with optional quality_stats and quality_trend fields for backward compatibility
Enhanced checkpoint creation to include quality statistics from QualityTracker.get_stats()
Added 14 comprehensive integration tests covering initialization, metric recording, degradation detection, context reset recommendations, and checkpoint integration

Implementation Quality:

Lazy initialization pattern prevents failures when project context unavailable
Graceful error handling throughout - quality tracking failures don't block core operations
Backward compatible model changes using Optional fields
Test coverage includes both mocked and real QualityTracker scenarios
Follows existing patterns for blocker creation and status transitions

Minor Issues Found:

import re statement placed inside function body instead of at module level (style issue)
Assumes 100% pass rate/coverage when gates pass, which may not reflect actual thresholds

Confidence Score: 4/5

This PR is safe to merge with minor style improvements recommended
Score reflects solid implementation with comprehensive test coverage (14 integration tests), proper error handling, and backward compatibility. Deducted one point for the re import placement (style issue) and the simplistic assumption of 100% metrics when gates pass. The lazy initialization pattern is well-designed, and the integration follows existing codebase patterns closely.
No files require special attention - all changes are well-structured and tested

Important Files Changed

Filename	Overview
codeframe/agents/worker_agent.py	Added QualityTracker integration with lazy initialization, response count tracking, metric recording after quality gates, and degradation detection. Includes one issue with `re` module import placement.
codeframe/core/models.py	Added optional quality tracking fields (`quality_stats` and `quality_trend`) to `CheckpointMetadata` for backward compatibility
codeframe/lib/checkpoint_manager.py	Integrated QualityTracker stats into checkpoint metadata with proper error handling
tests/integration/test_quality_tracker_integration.py	Comprehensive 14 integration tests covering quality metrics recording, degradation detection, context reset recommendations, and checkpoint integration

Sequence Diagram

sequenceDiagram
    participant Agent as WorkerAgent
    participant Gates as QualityGates
    participant Tracker as QualityTracker
    participant DB as Database
    participant FS as FileSystem

    Note over Agent: Task Execution
    Agent->>Agent: execute_task()
    Agent->>Agent: response_count++
    Agent->>Agent: Call LLM
    
    Note over Agent: Task Completion Flow
    Agent->>Agent: complete_task(task)
    Agent->>Agent: _ensure_quality_tracker()
    alt Tracker not initialized
        Agent->>DB: Get workspace_path
        DB-->>Agent: workspace_path
        Agent->>Tracker: new QualityTracker(project_path)
        Tracker->>FS: Setup .codeframe/quality_history.json
    end
    
    Agent->>Gates: run_all_gates(task)
    Gates-->>Agent: QualityGateResult
    
    Note over Agent: Record Quality Metrics
    Agent->>Agent: _record_quality_metrics()
    Agent->>Agent: Parse test/coverage from failures
    Agent->>Agent: Detect language with LanguageDetector
    Agent->>Tracker: record(QualityMetrics)
    Tracker->>FS: Append to quality_history.json
    
    Note over Agent: Check Degradation
    Agent->>Agent: _check_quality_degradation()
    Agent->>Tracker: check_degradation(threshold=10%)
    Tracker->>Tracker: Compare current vs peak metrics
    Tracker-->>Agent: degradation result
    
    alt Quality degraded >10%
        Agent->>Agent: _create_degradation_blocker()
        Agent->>DB: create_blocker(SYNC)
        DB-->>Agent: blocker_id
        Agent-->>Agent: Return blocked status
    else Quality OK and gates passed
        Agent->>DB: UPDATE task status=completed
        Agent-->>Agent: Return success
    else Quality OK but gates failed
        Agent->>DB: create_blocker() via gates
        Agent-->>Agent: Return blocked status
    end
    
    Note over Agent: Checkpoint Creation
    Agent->>Agent: CheckpointManager.create_checkpoint()
    Agent->>Tracker: get_stats()
    Tracker->>FS: Load quality_history.json
    Tracker-->>Agent: quality_stats + trend
    Agent->>DB: Store checkpoint with quality metadata

greptile-apps

_{4 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2025-12-26T22:04:50Z

+                        # Extract test counts from failure
+                        reason = getattr(failure, "reason", "")
+                        # Parse patterns like "3 tests failed" or "Pytest failed: 5 failed"
+                        import re


style: import re inside function body

Suggested change

import re

import re

Move to top-level imports (around line 3-7) per Python style conventions

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review. Path: codeframe/agents/worker_agent.py Line: 1124:1124 Comment: **style:** `import re` inside function body ```suggestion import re ``` Move to top-level imports (around line 3-7) per Python style conventions <sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub> How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-26T22:04:51Z

+            # If no failures, assume perfect scores
+            if not quality_result.failures:
+                test_pass_rate = 100.0
+                # Try to get coverage from other sources if available
+                coverage_percentage = 100.0  # Assume passed coverage check


style: Assuming 100% pass rate and coverage when gates pass may be inaccurate - quality gates might have different thresholds (e.g. 80% coverage requirement), so passing gates doesn't guarantee 100% metrics. Consider tracking actual metrics from quality gate results or setting these to None when actual values unavailable

Prompt To Fix With AI

This is a comment left during a code review. Path: codeframe/agents/worker_agent.py Line: 1144:1148 Comment: **style:** Assuming 100% pass rate and coverage when gates pass may be inaccurate - quality gates might have different thresholds (e.g. 80% coverage requirement), so passing gates doesn't guarantee 100% metrics. Consider tracking actual metrics from quality gate results or setting these to `None` when actual values unavailable How can I resolve this? If you propose a fix, please make it concise.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (2)

codeframe/lib/checkpoint_manager.py (1)
425-444: Minor inconsistency in quality_trend default handling.

When stats.get("has_data") is falsy, quality_trend remains None. However, when has_data is truthy but trend is missing, it defaults to "insufficient_data". This means the same semantic state ("no trend available") can be represented as either None or "insufficient_data".
🔎 Optional refactor for consistency
         try:
             from codeframe.enforcement.quality_tracker import QualityTracker
 
             tracker = QualityTracker(project_path=str(self.project_root))
             stats = tracker.get_stats()
 
             if stats.get("has_data"):
                 quality_stats = {
                     "current": stats.get("current"),
                     "peak": stats.get("peak"),
                     "average": stats.get("average"),
                     "total_checkpoints": stats.get("total_checkpoints"),
                 }
                 quality_trend = stats.get("trend", "insufficient_data")
+            else:
+                # Explicitly set trend when no data available for consistency
+                quality_trend = "insufficient_data"
         except Exception as e:
             logger.debug(f"Failed to get quality stats for checkpoint: {e}")
codeframe/agents/worker_agent.py (1)
1082-1183: Brittle parsing of quality gate failure messages.

Lines 1122-1142 use regex to extract test counts and coverage from failure reason strings (e.g., "(\d+)\s*failed"). This approach is fragile because:

Format dependency: Changes to failure message formatting in quality gates will break metric extraction

Silent failures: If regex doesn't match, metrics default to zero without warning

Assumption at line 1148: When no failures exist, coverage is assumed to be 100%, which may not reflect actual coverage if tests passed with low coverage

Consider having QualityGateResult expose structured metrics (test counts, coverage percentage) as first-class fields instead of embedding them in failure messages. This would eliminate parsing fragility.

Alternatively, add validation to warn when metrics cannot be extracted:
if total_tests == 0 and quality_result.failures:
    logger.warning("Could not extract test metrics from quality gate failures")

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3fa0e9c and 22dfc2b.

📒 Files selected for processing (4)

codeframe/agents/worker_agent.py
codeframe/core/models.py
codeframe/lib/checkpoint_manager.py
tests/integration/test_quality_tracker_integration.py

🧰 Additional context used

📓 Path-based instructions (5)

**/*.py