[Phase 4] Replace Fixed Self-Correction Cap with Adaptive Failure Handling

[frankbria]

Summary

The current 3-attempt self-correction cap for failing tests is arbitrary and doesn't distinguish between fixable bugs and architectural issues requiring plan revision.

Problem Analysis

Current behavior: When tests fail, agents retry up to 3 times before... (what? failing the task? escalating?).

This is problematic because:

1. Not all failures are equal: A typo causing a test failure needs 1 retry. An architectural mismatch needs 0 retries and plan revision instead.

2. Retrying the same approach rarely works: If the approach is fundamentally wrong, 3 attempts of the same strategy waste tokens and time.

3. No escalation path: After 3 failures, what happens? If the task just fails, we've learned nothing. If it continues anyway, we ship broken code.

4. Missing: "stop and think" mode: The Deep Agents pattern emphasizes that agents should recognize when they're stuck and revise their plan, not just retry.

State of the Art Comparison

From Philipp Schmid's Agents 2.0:

"No Recovery mechanism: If it goes down a rabbit hole, it rarely has the foresight to stop, backtrack, and try a new approach."

The fix isn't more retries—it's plan revision when retries aren't working.

12-Factor Agents:

"Embrace Errors: Keep error messages in the context."

Errors should inform strategy changes, not just trigger identical retries.

Proposed Improvement

Replace fixed retry count with adaptive failure handling:

Failure Type Detection → Strategy Selection → Escalation Path

Failure Classification

Failure Type	Signal	Action
Trivial	Single assertion, clear error message	Retry (max 2)
Moderate	Multiple failures, related to recent change	Analyze, then retry with adjustment
Architectural	Systemic failures, interface mismatch	Create blocker, request plan revision
External	Network, API, dependency issues	Wait and retry with backoff
Unknown	Unclear failure mode	Create blocker for human review

Implementation Approach

1. Failure analyzer: Before retrying, classify the failure type

   • Parse test output for patterns
   • Compare to previous attempt (same error = not fixable by retry)
   • Check if failure is in code agent just modified vs. elsewhere

2. Strategy selector:

   if failure_type == "trivial" and attempt < 2:
       return RetryStrategy(same_approach=True)
   elif failure_type == "moderate":
       return RetryStrategy(analyze_first=True, max_attempts=2)
   elif failure_type == "architectural":
       return EscalateStrategy(create_blocker=True, reason="architectural_mismatch")
   elif consecutive_same_errors >= 2:
       return EscalateStrategy(create_blocker=True, reason="stuck_in_loop")

3. Escalation to plan revision:

   • Create async blocker: "Test failures suggest approach X won't work. Options: A, B, C. Human guidance requested."
   • Or: Escalate to Lead Agent for task re-decomposition

4. Learning from failures:

   • When human resolves blocker, record the pattern
   • Build procedural memory: "When you see error type X in this codebase, approach Y works"

Success Criteria

• Failure classification implemented
• "Same error twice = don't retry" logic
• Architectural failures create blockers instead of retrying
• Reduced wasted tokens on futile retries
• Higher task completion rate (failures become blockers that get resolved, not terminal states)

Metrics to Track

• Retry efficiency: % of retries that succeed vs. fail again with same error
• Escalation rate: % of failures that become blockers
• Blocker resolution time: How long until human input resolves architectural issues
• Token waste: Tokens spent on retries that didn't help

References

• Agents 2.0: Deep Agents - Recovery mechanisms
• 12-Factor Agents - Error handling
• CodeFRAME blocker system - leverage existing async escalation infrastructure

[traycerai]

Plan

Observations

The current self-correction system uses a fixed 3-attempt cap hardcoded in both BackendWorkerAgent and TestWorkerAgent. The _self_correction_loop() method (file:codeframe/agents/backend_worker_agent.py lines 806-977) blindly retries the same approach without analyzing failure patterns. After exhaustion, a SYNC blocker is created, but there's no mechanism to detect repeated errors, classify failure types, or escalate to the Lead Agent for plan revision. The blocker system and database schema for tracking correction attempts are well-established, providing a solid foundation for adaptive failure handling.

Approach

The solution implements a failure analyzer that classifies errors before retry decisions, replacing the fixed cap with adaptive strategies. A new FailureAnalyzer class will parse test output and compare consecutive attempts to detect patterns (trivial vs. architectural failures). The _self_correction_loop() will be refactored to consult the analyzer and select strategies: retry with adjustments for fixable issues, or escalate to Lead Agent for plan revision when stuck. This leverages existing blocker infrastructure while adding intelligence to prevent futile retries, aligning with the Deep Agents pattern of "stop and think" when stuck.

Implementation Timeline

This enhancement is designated as a Phase 4 item with Future priority. The current 3-attempt self-correction cap is sufficient for v1/v2 releases. Adaptive failure handling should be deferred until after core Git/PR workflow integration is complete, as the missing Git workflow is a higher-priority blocker for production readiness.

Phase 4: Adaptive Failure Handling (Deferred)

When prioritized post-v2, the implementation will follow this sequence:

1. Failure Analysis Infrastructure - Create FailureAnalyzer class to parse test output and classify errors into: TRIVIAL, MODERATE, ARCHITECTURAL, EXTERNAL, UNKNOWN

2. Error Pattern Detection - Implement repeated error detection by comparing error signatures across consecutive correction attempts

3. Adaptive Strategy Selection - Build StrategySelector to choose between: retry with same approach, retry with analysis, escalate to blocker, or escalate to Lead Agent

4. Self-Correction Loop Refactoring - Integrate failure analysis into BackendWorkerAgent._self_correction_loop() and TestWorkerAgent._execute_and_correct_tests()

5. Lead Agent Escalation - Implement architectural failure escalation path that triggers task re-decomposition instead of futile retries

6. Database Schema Extension - Add columns to track failure_type, error_signature, and strategy_used for each correction attempt

7. Metrics and Monitoring - Implement tracking for retry efficiency, escalation rates, and token waste reduction

Rationale for Phase 4 Deferral

• Current sufficiency: The fixed 3-attempt cap handles typical test failures adequately for v1/v2
• Blocking dependencies: Git/PR workflow integration is a prerequisite for production readiness
• Complexity vs. impact: Adaptive handling adds significant complexity without addressing the core missing feature
• Prerequisite work: Lead Agent task re-decomposition logic should be stabilized before implementing escalation paths
• Measurement baseline: Defer until metrics infrastructure is in place to validate effectiveness

Future Considerations

When this enhancement is prioritized in Phase 4:

• Ensure Git/PR workflow is fully integrated and stable
• Establish baseline metrics for retry efficiency and token waste
• Implement feature flag for gradual rollout (10% → 50% → 100%)
• Coordinate with Lead Agent improvements for escalation handling
• Consider procedural memory system for learning error patterns across projects
• Validate failure classification accuracy against real-world test failures

Architecture Diagram

sequenceDiagram
    participant WA as Worker Agent
    participant FA as Failure Analyzer
    participant SS as Strategy Selector
    participant DB as Database
    participant LA as Lead Agent
    participant BL as Blocker System

    WA->>WA: Execute task, tests fail
    WA->>DB: Get previous correction attempts
    WA->>FA: analyze_test_failure(output, attempts)
    FA->>FA: Extract error signature
    FA->>FA: Detect if repeated error
    FA->>FA: Classify failure type
    FA-->>WA: FailureAnalysis
    
    WA->>SS: select_strategy(analysis, attempt, max)
    SS->>SS: Evaluate failure type & attempts
    SS-->>WA: CorrectionStrategy
    
    alt Strategy: RETRY_SAME or RETRY_WITH_ANALYSIS
        WA->>WA: Generate fix with LLM
        WA->>WA: Apply code changes
        WA->>DB: Store attempt with failure metadata
        WA->>WA: Re-run tests
    else Strategy: ESCALATE_TO_LEAD
        WA->>BL: Create SYNC blocker (architectural)
        WA->>LA: Notify for plan revision
        LA->>LA: Re-decompose task
        LA->>BL: Resolve blocker with new approach
    else Strategy: ESCALATE_TO_BLOCKER
        WA->>BL: Create SYNC/ASYNC blocker
        WA->>WA: Wait for human resolution
    end

Loading

Success Metrics

Metric	Current Baseline	Target	Measurement
Retry Efficiency	Unknown	>60%	% of retries that resolve the failure
Token Waste Reduction	Baseline	-40%	Tokens spent on futile retries (same error)
Architectural Escalation Rate	0%	15-25%	% of failures escalated to Lead Agent
Time to Resolution	Baseline	-30%	Average time from failure to resolution
False Escalation Rate	N/A	<10%	% of escalations that were unnecessary

Import In IDE

🤖 Prompt for AI Agents

## Observations

The current self-correction system uses a fixed 3-attempt cap hardcoded in both `BackendWorkerAgent` and `TestWorkerAgent`. The `_self_correction_loop()` method (file:codeframe/agents/backend_worker_agent.py lines 806-977) blindly retries the same approach without analyzing failure patterns. After exhaustion, a SYNC blocker is created, but there's no mechanism to detect repeated errors, classify failure types, or escalate to the Lead Agent for plan revision. The blocker system and database schema for tracking correction attempts are well-established, providing a solid foundation for adaptive failure handling.

## Approach

The solution implements a **failure analyzer** that classifies errors before retry decisions, replacing the fixed cap with adaptive strategies. A new `FailureAnalyzer` class will parse test output and compare consecutive attempts to detect patterns (trivial vs. architectural failures). The `_self_correction_loop()` will be refactored to consult the analyzer and select strategies: retry with adjustments for fixable issues, or escalate to Lead Agent for plan revision when stuck. This leverages existing blocker infrastructure while adding intelligence to prevent futile retries, aligning with the Deep Agents pattern of "stop and think" when stuck.

## Implementation Timeline

This enhancement is designated as a **Phase 4** item with **Future** priority. The current 3-attempt self-correction cap is sufficient for v1/v2 releases. Adaptive failure handling should be deferred until after core Git/PR workflow integration is complete, as the missing Git workflow is a higher-priority blocker for production readiness.

### Phase 4: Adaptive Failure Handling (Deferred)

When prioritized post-v2, the implementation will follow this sequence:

1. **Failure Analysis Infrastructure** - Create `FailureAnalyzer` class to parse test output and classify errors into: `TRIVIAL`, `MODERATE`, `ARCHITECTURAL`, `EXTERNAL`, `UNKNOWN`

2. **Error Pattern Detection** - Implement repeated error detection by comparing error signatures across consecutive correction attempts

3. **Adaptive Strategy Selection** - Build `StrategySelector` to choose between: retry with same approach, retry with analysis, escalate to blocker, or escalate to Lead Agent

4. **Self-Correction Loop Refactoring** - Integrate failure analysis into `BackendWorkerAgent._self_correction_loop()` and `TestWorkerAgent._execute_and_correct_tests()`

5. **Lead Agent Escalation** - Implement architectural failure escalation path that triggers task re-decomposition instead of futile retries

6. **Database Schema Extension** - Add columns to track `failure_type`, `error_signature`, and `strategy_used` for each correction attempt

7. **Metrics and Monitoring** - Implement tracking for retry efficiency, escalation rates, and token waste reduction

### Rationale for Phase 4 Deferral

- **Current sufficiency**: The fixed 3-attempt cap handles typical test failures adequately for v1/v2
- **Blocking dependencies**: Git/PR workflow integration is a prerequisite for production readiness
- **Complexity vs. impact**: Adaptive handling adds significant complexity without addressing the core missing feature
- **Prerequisite work**: Lead Agent task re-decomposition logic should be stabilized before implementing escalation paths
- **Measurement baseline**: Defer until metrics infrastructure is in place to validate effectiveness

### Future Considerations

When this enhancement is prioritized in Phase 4:
- Ensure Git/PR workflow is fully integrated and stable
- Establish baseline metrics for retry efficiency and token waste
- Implement feature flag for gradual rollout (10% → 50% → 100%)
- Coordinate with Lead Agent improvements for escalation handling
- Consider procedural memory system for learning error patterns across projects
- Validate failure classification accuracy against real-world test failures

## Architecture Diagram

```mermaid
sequenceDiagram
    participant WA as Worker Agent
    participant FA as Failure Analyzer
    participant SS as Strategy Selector
    participant DB as Database
    participant LA as Lead Agent
    participant BL as Blocker System

    WA->>WA: Execute task, tests fail
    WA->>DB: Get previous correction attempts
    WA->>FA: analyze_test_failure(output, attempts)
    FA->>FA: Extract error signature
    FA->>FA: Detect if repeated error
    FA->>FA: Classify failure type
    FA-->>WA: FailureAnalysis
    
    WA->>SS: select_strategy(analysis, attempt, max)
    SS->>SS: Evaluate failure type & attempts
    SS-->>WA: CorrectionStrategy
    
    alt Strategy: RETRY_SAME or RETRY_WITH_ANALYSIS
        WA->>WA: Generate fix with LLM
        WA->>WA: Apply code changes
        WA->>DB: Store attempt with failure metadata
        WA->>WA: Re-run tests
    else Strategy: ESCALATE_TO_LEAD
        WA->>BL: Create SYNC blocker (architectural)
        WA->>LA: Notify for plan revision
        LA->>LA: Re-decompose task
        LA->>BL: Resolve blocker with new approach
    else Strategy: ESCALATE_TO_BLOCKER
        WA->>BL: Create SYNC/ASYNC blocker
        WA->>WA: Wait for human resolution
    end
```

## Success Metrics

| Metric | Current Baseline | Target | Measurement |
|--------|------------------|--------|-------------|
| Retry Efficiency | Unknown | >60% | % of retries that resolve the failure |
| Token Waste Reduction | Baseline | -40% | Tokens spent on futile retries (same error) |
| Architectural Escalation Rate | 0% | 15-25% | % of failures escalated to Lead Agent |
| Time to Resolution | Baseline | -30% | Average time from failure to resolution |
| False Escalation Rate | N/A | <10% | % of escalations that were unnecessary |

---

Execution Information

Branch: main
Commit: 4d894a6

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[frankbria]

@traycerai Update plan

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

New Plan

Observations

The current self-correction system uses a fixed 3-attempt cap hardcoded in both BackendWorkerAgent and TestWorkerAgent. The _self_correction_loop() method (file:codeframe/agents/backend_worker_agent.py lines 806-977) blindly retries the same approach without analyzing failure patterns. After exhaustion, a SYNC blocker is created, but there's no mechanism to detect repeated errors, classify failure types, or escalate to the Lead Agent for plan revision. The blocker system and database schema for tracking correction attempts are well-established, providing a solid foundation for adaptive failure handling.

Approach

The solution implements a failure analyzer that classifies errors before retry decisions, replacing the fixed cap with adaptive strategies. A new FailureAnalyzer class will parse test output and compare consecutive attempts to detect patterns (trivial vs. architectural failures). The _self_correction_loop() will be refactored to consult the analyzer and select strategies: retry with adjustments for fixable issues, or escalate to Lead Agent for plan revision when stuck. This leverages existing blocker infrastructure while adding intelligence to prevent futile retries, aligning with the Deep Agents pattern of "stop and think" when stuck.

Implementation Steps

1. Create Failure Analysis Infrastructure

Create file:codeframe/testing/failure_analyzer.py:

• Implement FailureType enum with values: TRIVIAL, MODERATE, ARCHITECTURAL, EXTERNAL, UNKNOWN
• Implement FailureAnalysis dataclass containing: failure_type, error_signature, is_repeated, confidence_score, reasoning
• Implement FailureAnalyzer class with methods:
  • analyze_test_failure(test_output: str, previous_attempts: List[Dict]) -> FailureAnalysis
  • _extract_error_signature(test_output: str) -> str - Parse pytest output for error patterns
  • _detect_repeated_error(current_signature: str, previous_attempts: List[Dict]) -> bool - Compare error signatures
  • _classify_failure_type(test_output: str, error_signature: str, is_repeated: bool) -> FailureType - Pattern matching for classification
  • _parse_pytest_output(output: str) -> Dict - Extract assertion errors, stack traces, failure counts

Classification heuristics:

• TRIVIAL: Single assertion failure, clear error message, not repeated
• MODERATE: Multiple related failures, first occurrence
• ARCHITECTURAL: Import errors, interface mismatches, systemic failures across multiple files
• EXTERNAL: Network timeouts, API errors, dependency issues
• UNKNOWN: Unclear patterns, requires human review
• Any error repeated 2+ times automatically escalates

2. Extend Database Schema for Failure Tracking

Modify file:codeframe/persistence/database.py:

• In _create_schema() method, add columns to correction_attempts table:
  • failure_type TEXT - Store FailureType enum value
  • error_signature TEXT - Store normalized error pattern
  • strategy_used TEXT - Store retry strategy applied
• Update create_correction_attempt() method signature to accept new parameters
• Update get_correction_attempts_by_task() to return new fields
• Add method get_error_signatures_by_task(task_id: int) -> List[str] for pattern detection

3. Implement Adaptive Strategy Selection

Create file:codeframe/agents/correction_strategies.py:

• Implement CorrectionStrategy enum: RETRY_SAME, RETRY_WITH_ANALYSIS, ESCALATE_TO_BLOCKER, ESCALATE_TO_LEAD
• Implement StrategySelector class with method:
  • select_strategy(failure_analysis: FailureAnalysis, attempt_number: int, max_attempts: int) -> CorrectionStrategy
• Strategy selection logic:
  • TRIVIAL + attempt < 2 → RETRY_SAME
  • MODERATE + attempt < 3 → RETRY_WITH_ANALYSIS
  • ARCHITECTURAL → ESCALATE_TO_LEAD
  • is_repeated=True (2+ same errors) → ESCALATE_TO_LEAD
  • EXTERNAL → ESCALATE_TO_BLOCKER with retry suggestion
  • UNKNOWN → ESCALATE_TO_BLOCKER for human review
  • Max attempts reached → ESCALATE_TO_BLOCKER

4. Refactor Backend Worker Agent Self-Correction Loop

Modify file:codeframe/agents/backend_worker_agent.py:

• In __init__(), instantiate FailureAnalyzer and StrategySelector
• Refactor _self_correction_loop() method (lines 806-977):
  • Before each retry, call failure_analyzer.analyze_test_failure() with test output and previous attempts from database
  • Call strategy_selector.select_strategy() to determine action
  • If strategy is ESCALATE_TO_LEAD, call new method _escalate_to_lead_agent()
  • If strategy is ESCALATE_TO_BLOCKER, create blocker with failure analysis details
  • If strategy is RETRY_WITH_ANALYSIS, pass failure analysis to _attempt_self_correction() for context-aware fix generation
  • Store failure_type, error_signature, and strategy_used when calling db.create_correction_attempt()
• Update _attempt_self_correction() method signature to accept failure_analysis: Optional[FailureAnalysis]
• Enhance LLM prompt in _attempt_self_correction() to include failure classification and reasoning when available

5. Implement Lead Agent Escalation

Modify file:codeframe/agents/backend_worker_agent.py:

• Add method _escalate_to_lead_agent(task: Dict, failure_analysis: FailureAnalysis, attempts: List[Dict]) -> None:
  • Create SYNC blocker with type "ARCHITECTURAL_FAILURE" or "STUCK_IN_LOOP"
  • Include in blocker description: failure type, error signature, attempted fixes, suggestion for plan revision
  • Format blocker message with markdown table showing all attempts and their outcomes
  • Set blocker metadata indicating Lead Agent should review task decomposition

Modify file:codeframe/agents/lead_agent.py:

• In _execute_coordination_loop() method (around lines 1031-1521), add logic to detect architectural failure blockers
• When such blocker is detected, call existing decompose_prd() or task decomposition logic to re-analyze the failing task
• Provide options in blocker: "A) Revise task approach, B) Split into subtasks, C) Mark as requires human implementation"

6. Refactor Test Worker Agent

Modify file:codeframe/agents/test_worker_agent.py:

• In __init__(), instantiate FailureAnalyzer and StrategySelector
• Refactor _execute_and_correct_tests() method (lines 569-634):
  • Apply same failure analysis and strategy selection logic as Backend Worker Agent
  • Before retry, analyze failure and select strategy
  • Implement escalation paths for architectural test failures
  • Store failure metadata in database via create_correction_attempt()

7. Update Documentation

Modify file:docs/self_correction_workflow.md:

• Update workflow diagram to show failure analysis step before retry decision
• Add section "Failure Classification" with table of failure types and actions
• Update "Future Enhancements" section to mark adaptive failure handling as implemented
• Add examples of each failure type and how system responds
• Document new database schema fields

Create file:docs/adaptive_failure_handling.md:

• Document failure classification algorithm
• Provide examples of error signatures for each failure type
• Explain strategy selection logic with decision tree
• Document escalation paths and Lead Agent integration
• Include metrics for measuring effectiveness

8. Add Metrics and Monitoring

Modify file:codeframe/persistence/database.py:

• Add method get_correction_metrics(project_id: int) -> Dict:
  • Calculate retry efficiency: % of retries that succeed vs. fail with same error
  • Calculate escalation rate: % of failures that become blockers
  • Calculate token waste: tokens spent on futile retries (same error repeated)
  • Group by failure type for analysis
• Add method get_failure_type_distribution(project_id: int) -> Dict[FailureType, int]

Modify file:codeframe/ui/routers/blockers.py or create new metrics endpoint:

• Add API endpoint /api/projects/{project_id}/correction-metrics to expose metrics
• Return JSON with retry efficiency, escalation rates, failure type distribution

9. Testing Strategy

Create file:tests/testing/test_failure_analyzer.py:

• Test error signature extraction from pytest output
• Test repeated error detection across multiple attempts
• Test failure type classification for each category
• Test edge cases: empty output, malformed output, mixed failure types

Create file:tests/agents/test_correction_strategies.py:

• Test strategy selection for each failure type
• Test max attempts enforcement
• Test repeated error escalation
• Test strategy transitions (e.g., MODERATE → ESCALATE after 2 attempts)

Modify file:tests/agents/test_backend_worker_agent.py:

• Update existing self-correction tests to verify failure analysis integration
• Add test for architectural failure escalation to Lead Agent
• Add test for repeated error detection and escalation
• Add test for token waste reduction (fewer futile retries)

Create file:tests/integration/test_adaptive_correction_workflow.py:

• End-to-end test: trivial failure → 1 retry → success
• End-to-end test: architectural failure → immediate escalation
• End-to-end test: repeated error → escalation after 2 attempts
• End-to-end test: moderate failure → analysis → adjusted retry → success

10. Configuration and Rollout

Modify file:codeframe/core/config.py (if exists) or agent initialization:

• Add configuration option adaptive_failure_handling_enabled: bool = True for gradual rollout
• Add configuration for max attempts per failure type (e.g., trivial_max_attempts: int = 2)
• Add configuration for error signature similarity threshold

Migration path:

• Phase 1: Deploy with feature flag disabled, collect baseline metrics
• Phase 2: Enable for 10% of tasks, monitor retry efficiency
• Phase 3: Full rollout after validation
• Maintain backward compatibility: if analyzer fails, fall back to fixed 3-attempt behavior

Architecture Diagram

sequenceDiagram
    participant WA as Worker Agent
    participant FA as Failure Analyzer
    participant SS as Strategy Selector
    participant DB as Database
    participant LA as Lead Agent
    participant BL as Blocker System

    WA->>WA: Execute task, tests fail
    WA->>DB: Get previous correction attempts
    WA->>FA: analyze_test_failure(output, attempts)
    FA->>FA: Extract error signature
    FA->>FA: Detect if repeated error
    FA->>FA: Classify failure type
    FA-->>WA: FailureAnalysis
    
    WA->>SS: select_strategy(analysis, attempt, max)
    SS->>SS: Evaluate failure type & attempts
    SS-->>WA: CorrectionStrategy
    
    alt Strategy: RETRY_SAME or RETRY_WITH_ANALYSIS
        WA->>WA: Generate fix with LLM
        WA->>WA: Apply code changes
        WA->>DB: Store attempt with failure metadata
        WA->>WA: Re-run tests
    else Strategy: ESCALATE_TO_LEAD
        WA->>BL: Create SYNC blocker (architectural)
        WA->>LA: Notify for plan revision
        LA->>LA: Re-decompose task
        LA->>BL: Resolve blocker with new approach
    else Strategy: ESCALATE_TO_BLOCKER
        WA->>BL: Create SYNC/ASYNC blocker
        WA->>WA: Wait for human resolution
    end

Loading

Success Metrics

Metric	Current Baseline	Target	Measurement
Retry Efficiency	Unknown	>60%	% of retries that resolve the failure
Token Waste Reduction	Baseline	-40%	Tokens spent on futile retries (same error)
Architectural Escalation Rate	0%	15-25%	% of failures escalated to Lead Agent
Time to Resolution	Baseline	-30%	Average time from failure to resolution
False Escalation Rate	N/A	<10%	% of escalations that were unnecessary

Import In IDE

🤖 Prompt for AI Agents

## Observations

The current self-correction system uses a fixed 3-attempt cap hardcoded in both `BackendWorkerAgent` and `TestWorkerAgent`. The `_self_correction_loop()` method (file:codeframe/agents/backend_worker_agent.py lines 806-977) blindly retries the same approach without analyzing failure patterns. After exhaustion, a SYNC blocker is created, but there's no mechanism to detect repeated errors, classify failure types, or escalate to the Lead Agent for plan revision. The blocker system and database schema for tracking correction attempts are well-established, providing a solid foundation for adaptive failure handling.

## Approach

The solution implements a **failure analyzer** that classifies errors before retry decisions, replacing the fixed cap with adaptive strategies. A new `FailureAnalyzer` class will parse test output and compare consecutive attempts to detect patterns (trivial vs. architectural failures). The `_self_correction_loop()` will be refactored to consult the analyzer and select strategies: retry with adjustments for fixable issues, or escalate to Lead Agent for plan revision when stuck. This leverages existing blocker infrastructure while adding intelligence to prevent futile retries, aligning with the Deep Agents pattern of "stop and think" when stuck.

## Implementation Steps

### 1. Create Failure Analysis Infrastructure

**Create** `file:codeframe/testing/failure_analyzer.py`:
- Implement `FailureType` enum with values: `TRIVIAL`, `MODERATE`, `ARCHITECTURAL`, `EXTERNAL`, `UNKNOWN`
- Implement `FailureAnalysis` dataclass containing: `failure_type`, `error_signature`, `is_repeated`, `confidence_score`, `reasoning`
- Implement `FailureAnalyzer` class with methods:
  - `analyze_test_failure(test_output: str, previous_attempts: List[Dict]) -> FailureAnalysis`
  - `_extract_error_signature(test_output: str) -> str` - Parse pytest output for error patterns
  - `_detect_repeated_error(current_signature: str, previous_attempts: List[Dict]) -> bool` - Compare error signatures
  - `_classify_failure_type(test_output: str, error_signature: str, is_repeated: bool) -> FailureType` - Pattern matching for classification
  - `_parse_pytest_output(output: str) -> Dict` - Extract assertion errors, stack traces, failure counts

**Classification heuristics**:
- `TRIVIAL`: Single assertion failure, clear error message, not repeated
- `MODERATE`: Multiple related failures, first occurrence
- `ARCHITECTURAL`: Import errors, interface mismatches, systemic failures across multiple files
- `EXTERNAL`: Network timeouts, API errors, dependency issues
- `UNKNOWN`: Unclear patterns, requires human review
- Any error repeated 2+ times automatically escalates

### 2. Extend Database Schema for Failure Tracking

**Modify** `file:codeframe/persistence/database.py`:
- In `_create_schema()` method, add columns to `correction_attempts` table:
  - `failure_type TEXT` - Store FailureType enum value
  - `error_signature TEXT` - Store normalized error pattern
  - `strategy_used TEXT` - Store retry strategy applied
- Update `create_correction_attempt()` method signature to accept new parameters
- Update `get_correction_attempts_by_task()` to return new fields
- Add method `get_error_signatures_by_task(task_id: int) -> List[str]` for pattern detection

### 3. Implement Adaptive Strategy Selection

**Create** `file:codeframe/agents/correction_strategies.py`:
- Implement `CorrectionStrategy` enum: `RETRY_SAME`, `RETRY_WITH_ANALYSIS`, `ESCALATE_TO_BLOCKER`, `ESCALATE_TO_LEAD`
- Implement `StrategySelector` class with method:
  - `select_strategy(failure_analysis: FailureAnalysis, attempt_number: int, max_attempts: int) -> CorrectionStrategy`
- Strategy selection logic:
  - `TRIVIAL` + attempt < 2 → `RETRY_SAME`
  - `MODERATE` + attempt < 3 → `RETRY_WITH_ANALYSIS`
  - `ARCHITECTURAL` → `ESCALATE_TO_LEAD`
  - `is_repeated=True` (2+ same errors) → `ESCALATE_TO_LEAD`
  - `EXTERNAL` → `ESCALATE_TO_BLOCKER` with retry suggestion
  - `UNKNOWN` → `ESCALATE_TO_BLOCKER` for human review
  - Max attempts reached → `ESCALATE_TO_BLOCKER`

### 4. Refactor Backend Worker Agent Self-Correction Loop

**Modify** `file:codeframe/agents/backend_worker_agent.py`:
- In `__init__()`, instantiate `FailureAnalyzer` and `StrategySelector`
- Refactor `_self_correction_loop()` method (lines 806-977):
  - Before each retry, call `failure_analyzer.analyze_test_failure()` with test output and previous attempts from database
  - Call `strategy_selector.select_strategy()` to determine action
  - If strategy is `ESCALATE_TO_LEAD`, call new method `_escalate_to_lead_agent()`
  - If strategy is `ESCALATE_TO_BLOCKER`, create blocker with failure analysis details
  - If strategy is `RETRY_WITH_ANALYSIS`, pass failure analysis to `_attempt_self_correction()` for context-aware fix generation
  - Store `failure_type`, `error_signature`, and `strategy_used` when calling `db.create_correction_attempt()`
- Update `_attempt_self_correction()` method signature to accept `failure_analysis: Optional[FailureAnalysis]`
- Enhance LLM prompt in `_attempt_self_correction()` to include failure classification and reasoning when available

### 5. Implement Lead Agent Escalation

**Modify** `file:codeframe/agents/backend_worker_agent.py`:
- Add method `_escalate_to_lead_agent(task: Dict, failure_analysis: FailureAnalysis, attempts: List[Dict]) -> None`:
  - Create SYNC blocker with type "ARCHITECTURAL_FAILURE" or "STUCK_IN_LOOP"
  - Include in blocker description: failure type, error signature, attempted fixes, suggestion for plan revision
  - Format blocker message with markdown table showing all attempts and their outcomes
  - Set blocker metadata indicating Lead Agent should review task decomposition

**Modify** `file:codeframe/agents/lead_agent.py`:
- In `_execute_coordination_loop()` method (around lines 1031-1521), add logic to detect architectural failure blockers
- When such blocker is detected, call existing `decompose_prd()` or task decomposition logic to re-analyze the failing task
- Provide options in blocker: "A) Revise task approach, B) Split into subtasks, C) Mark as requires human implementation"

### 6. Refactor Test Worker Agent

**Modify** `file:codeframe/agents/test_worker_agent.py`:
- In `__init__()`, instantiate `FailureAnalyzer` and `StrategySelector`
- Refactor `_execute_and_correct_tests()` method (lines 569-634):
  - Apply same failure analysis and strategy selection logic as Backend Worker Agent
  - Before retry, analyze failure and select strategy
  - Implement escalation paths for architectural test failures
  - Store failure metadata in database via `create_correction_attempt()`

### 7. Update Documentation

**Modify** `file:docs/self_correction_workflow.md`:
- Update workflow diagram to show failure analysis step before retry decision
- Add section "Failure Classification" with table of failure types and actions
- Update "Future Enhancements" section to mark adaptive failure handling as implemented
- Add examples of each failure type and how system responds
- Document new database schema fields

**Create** `file:docs/adaptive_failure_handling.md`:
- Document failure classification algorithm
- Provide examples of error signatures for each failure type
- Explain strategy selection logic with decision tree
- Document escalation paths and Lead Agent integration
- Include metrics for measuring effectiveness

### 8. Add Metrics and Monitoring

**Modify** `file:codeframe/persistence/database.py`:
- Add method `get_correction_metrics(project_id: int) -> Dict`:
  - Calculate retry efficiency: % of retries that succeed vs. fail with same error
  - Calculate escalation rate: % of failures that become blockers
  - Calculate token waste: tokens spent on futile retries (same error repeated)
  - Group by failure type for analysis
- Add method `get_failure_type_distribution(project_id: int) -> Dict[FailureType, int]`

**Modify** `file:codeframe/ui/routers/blockers.py` or create new metrics endpoint:
- Add API endpoint `/api/projects/{project_id}/correction-metrics` to expose metrics
- Return JSON with retry efficiency, escalation rates, failure type distribution

### 9. Testing Strategy

**Create** `file:tests/testing/test_failure_analyzer.py`:
- Test error signature extraction from pytest output
- Test repeated error detection across multiple attempts
- Test failure type classification for each category
- Test edge cases: empty output, malformed output, mixed failure types

**Create** `file:tests/agents/test_correction_strategies.py`:
- Test strategy selection for each failure type
- Test max attempts enforcement
- Test repeated error escalation
- Test strategy transitions (e.g., MODERATE → ESCALATE after 2 attempts)

**Modify** `file:tests/agents/test_backend_worker_agent.py`:
- Update existing self-correction tests to verify failure analysis integration
- Add test for architectural failure escalation to Lead Agent
- Add test for repeated error detection and escalation
- Add test for token waste reduction (fewer futile retries)

**Create** `file:tests/integration/test_adaptive_correction_workflow.py`:
- End-to-end test: trivial failure → 1 retry → success
- End-to-end test: architectural failure → immediate escalation
- End-to-end test: repeated error → escalation after 2 attempts
- End-to-end test: moderate failure → analysis → adjusted retry → success

### 10. Configuration and Rollout

**Modify** `file:codeframe/core/config.py` (if exists) or agent initialization:
- Add configuration option `adaptive_failure_handling_enabled: bool = True` for gradual rollout
- Add configuration for max attempts per failure type (e.g., `trivial_max_attempts: int = 2`)
- Add configuration for error signature similarity threshold

**Migration path**:
- Phase 1: Deploy with feature flag disabled, collect baseline metrics
- Phase 2: Enable for 10% of tasks, monitor retry efficiency
- Phase 3: Full rollout after validation
- Maintain backward compatibility: if analyzer fails, fall back to fixed 3-attempt behavior

## Architecture Diagram

```mermaid
sequenceDiagram
    participant WA as Worker Agent
    participant FA as Failure Analyzer
    participant SS as Strategy Selector
    participant DB as Database
    participant LA as Lead Agent
    participant BL as Blocker System

    WA->>WA: Execute task, tests fail
    WA->>DB: Get previous correction attempts
    WA->>FA: analyze_test_failure(output, attempts)
    FA->>FA: Extract error signature
    FA->>FA: Detect if repeated error
    FA->>FA: Classify failure type
    FA-->>WA: FailureAnalysis
    
    WA->>SS: select_strategy(analysis, attempt, max)
    SS->>SS: Evaluate failure type & attempts
    SS-->>WA: CorrectionStrategy
    
    alt Strategy: RETRY_SAME or RETRY_WITH_ANALYSIS
        WA->>WA: Generate fix with LLM
        WA->>WA: Apply code changes
        WA->>DB: Store attempt with failure metadata
        WA->>WA: Re-run tests
    else Strategy: ESCALATE_TO_LEAD
        WA->>BL: Create SYNC blocker (architectural)
        WA->>LA: Notify for plan revision
        LA->>LA: Re-decompose task
        LA->>BL: Resolve blocker with new approach
    else Strategy: ESCALATE_TO_BLOCKER
        WA->>BL: Create SYNC/ASYNC blocker
        WA->>WA: Wait for human resolution
    end
```

## Success Metrics

| Metric | Current Baseline | Target | Measurement |
|--------|------------------|--------|-------------|
| Retry Efficiency | Unknown | >60% | % of retries that resolve the failure |
| Token Waste Reduction | Baseline | -40% | Tokens spent on futile retries (same error) |
| Architectural Escalation Rate | 0% | 15-25% | % of failures escalated to Lead Agent |
| Time to Resolution | Baseline | -30% | Average time from failure to resolution |
| False Escalation Rate | N/A | <10% | % of escalations that were unnecessary |

---

Execution Information

Branch: main
Commit: 360e111