[Phase 4] Verification Gate Wrapper for External Agents

[frankbria]

Summary

Ensure CodeFrame's verification gates (ruff, pytest, etc.) and self-correction loop work uniformly across all execution engines, not just the built-in ReactAgent.

Parent issue: #408

Motivation

Currently verification gates and self-correction are deeply embedded in react_agent.py::_run_final_verification(). When using external engines (Claude Code, Aider, Codex), CodeFrame still needs to:

1. Run gates after the agent finishes
2. If gates fail, re-invoke the agent with error context
3. Repeat up to N times

This is CodeFrame's core value-add over raw agent usage. A developer using Claude Code directly doesn't get automatic "fix the lint errors" retries. CodeFrame does.

Scope

New module: core/verification_loop.py

1. VerificationLoop class:

class VerificationLoop:
    def __init__(self, workspace, gates, max_retries=5):
        self.workspace = workspace
        self.gates = gates
        self.max_retries = max_retries
    
    def run_with_verification(
        self, 
        adapter: AgentAdapter,
        context: AgentContext,
        workspace_path: Path,
    ) -> AgentResult:
        """Execute agent, verify, retry on failure.
        
        Loop:
        1. adapter.execute(prompt, workspace)
        2. Run gates (ruff, pytest, etc.)
        3. If gates pass → return success
        4. If gates fail → append errors to context, re-invoke adapter
        5. Repeat up to max_retries
        """
        ...

2. Gate error → context enrichment:

   • When gates fail, append error output to context.previous_errors
   • Increment context.attempt
   • Re-invoke the adapter with enriched context
   • The prompt to the agent includes: "The following verification checks failed: [errors]. Please fix these issues."

3. Extract verification logic from react_agent.py:

   • Move _run_final_verification() logic into VerificationLoop
   • ReactAgent's internal verification becomes a call to VerificationLoop
   • External engines use the same VerificationLoop

Changes to core/runtime.py

4. execute_agent() uses VerificationLoop for all engines:

   adapter = registry.get(engine)
   context = packager.build_agent_context(task_id)
   loop = VerificationLoop(workspace, gates, max_retries=5)
   result = loop.run_with_verification(adapter, context, workspace_path)

Quick fix integration

5. Before re-invoking the agent, attempt quick fixes (core/quick_fixes.py):
   • If the error is a known pattern (missing import, lint error), apply quick fix first
   • Only re-invoke the agent for errors that quick fixes can't handle
   • This preserves CodeFrame's existing optimization

Acceptance Criteria

• VerificationLoop runs gates after any engine completes
• Gate failures trigger re-invocation with error context
• Quick fixes applied before agent re-invocation
• Max retry limit respected
• Blocker created if max retries exceeded
• Works with all registered engines (built-in, claude-code, aider, codex)
• ReactAgent refactored to use shared VerificationLoop (no behavior change)
• Integration tests: engine succeeds → gates fail → retry → gates pass

Dependencies

• Depends on [Phase 4] Agent Adapter Protocol Definition #409 (AgentAdapter protocol, AgentResult)
• Uses existing core/gates.py, core/quick_fixes.py, core/fix_tracker.py

[traycerai]

Plan

Observations

The verification/retry logic in _run_final_verification() at file:codeframe/core/react_agent.py (lines 425–534) already contains the full pattern: run gates → try quick fix → enrich context → re-invoke LLM → repeat up to max_verification_retries. The codebase uses dataclasses (not Pydantic) for pure-Python domain objects. Issue #409's AgentAdapter/AgentResult/AgentContext do not exist yet and must be defined first. The gates, quick_fixes, fix_tracker, and blockers modules in file:codeframe/core/ are all headless and ready for reuse.

Approach

Define the AgentAdapter protocol and supporting types in a new core/engine.py file, then extract the verification loop into core/verification_loop.py as a standalone class that any adapter can call. Finally, wire it into execute_agent() in core/runtime.py and refactor ReactAgent to delegate to it. This keeps the existing behavior identical while enabling all future engines to share the same gate-enforcement path.

---

Implementation Steps

Step 1 — Define the AgentAdapter Protocol and supporting types (core/engine.py)

Create file:codeframe/core/engine.py as a new headless module. Model its style after the existing dataclass-based modules in file:codeframe/core/.

Define three @dataclass classes and one Protocol:

• AgentContext — carries everything needed to invoke any engine for a task:

  • task_id: str
  • task_context: TaskContext (reuse the existing TaskContext from file:codeframe/core/context.py)
  • previous_errors: list[str] (accumulated gate failure messages across retries)
  • attempt: int (0-based retry index)

• AgentResult — wraps the outcome of a single adapter execution:

  • status: AgentStatus (reuse from file:codeframe/core/agent.py)
  • blocker_id: Optional[str]
  • error_summary: Optional[str]

• AgentAdapter (a typing.Protocol) — the interface every engine must satisfy:

  • def execute(self, context: AgentContext, workspace: Workspace) -> AgentResult: ...

• ReactAgentAdapter — a thin adapter that wraps the existing ReactAgent and delegates execute() to its run() method, mapping AgentStatus to AgentResult. It must accept the same constructor args as ReactAgent (workspace, llm_provider, dry_run, etc.), storing them and forwarding them on execute().

---

Step 2 — Create VerificationLoop in core/verification_loop.py

Create file:codeframe/core/verification_loop.py. The class mirrors the structure of _run_final_verification() but is decoupled from ReactAgent. Follow the same import style used in file:codeframe/core/react_agent.py.

VerificationLoop.__init__(self, workspace, max_retries=5)

• Accept workspace: Workspace and max_retries: int
• Instantiate a FixAttemptTracker internally (same as ReactAgent.__init__ does)
• Store a logger

VerificationLoop.run_with_verification(self, adapter, context, workspace_path) -> AgentResult

The loop logic:

for attempt in range(max_retries + 1):
    result = adapter.execute(enriched_context, workspace)
    if result.status is BLOCKED:
        return result immediately
    gate_result = gates.run(workspace)
    if gate_result.passed:
        return AgentResult(COMPLETED)
    if attempt >= max_retries:
        create escalation blocker via blockers.create()
        return AgentResult(BLOCKED, blocker_id=...)
    error_summary = gate_result.get_error_summary() or gate_result.summary
    # Try quick fix (no agent re-invoke needed)
    quick_fix_applied = _try_quick_fix(error_summary, workspace)
    if quick_fix_applied:
        continue  # re-run gates in next iteration without re-invoking adapter
    # Enrich context for next adapter invocation
    context.previous_errors.append(error_summary)
    context.attempt = attempt + 1
    record failure in fix_tracker
    check fix_tracker.should_escalate() → if True, create blocker and return BLOCKED

Private _try_quick_fix(self, error_summary, workspace) -> bool

• Call find_quick_fix(error_summary, repo_path=workspace.repo_path)
• If a fix is found, call apply_quick_fix(fix, workspace.repo_path)
• Return success bool — mirrors ReactAgent._try_quick_fix() in file:codeframe/core/react_agent.py (lines 915–976)

_build_error_prompt(self, context: AgentContext) -> str

• Construct the "The following verification checks failed: [errors]. Please fix these issues." addendum
• This becomes part of the enriched prompt when AgentContext.previous_errors is non-empty; the adapter is responsible for incorporating it into its LLM prompt

---

Step 3 — Refactor ReactAgent to delegate to VerificationLoop

In file:codeframe/core/react_agent.py:

• Import VerificationLoop from codeframe.core.verification_loop
• In ReactAgent.__init__, instantiate self._verification_loop = VerificationLoop(workspace, max_retries=max_verification_retries) instead of managing the tracker inline
• Modify ReactAgent.run() to replace the direct call to self._run_final_verification(system_prompt) with a call through _verification_loop. Because ReactAgent still runs its own main loop first, the simplest approach is: after the _react_loop completes, use VerificationLoop with a pass-through adapter that already has the main loop done — i.e., keep the LLM mini-loop variant. See the note below.

Compatibility note: _run_final_verification uses the ReactAgent's own LLM provider for its mini-loop, so the refactor must preserve that. Extract the gate-check + quick-fix + escalation logic into VerificationLoop, but keep the "mini ReAct fix loop" invocation inside ReactAgent._run_final_verification as a thin wrapper that now delegates gate-running, quick-fix, and tracker logic to VerificationLoop._check_and_fix_once(attempt, error_summary). This keeps ReactAgent's existing behavior exactly the same.

Specifically:

• Move _try_quick_fix, _create_escalation_blocker, and the fix_tracker usage out of ReactAgent into VerificationLoop
• ReactAgent._run_final_verification shrinks to: call gates.run(), ask VerificationLoop to run quick-fix + tracker check, if not escalating run the mini LLM loop, repeat

---

Step 4 — Update execute_agent() in core/runtime.py for external engines

In file:codeframe/core/runtime.py, extend the execute_agent() function:

• Add new engine name constants: "claude-code", "aider", "codex" alongside the existing "react" and "plan" validation set.

• After the existing if engine == "react": ... else: ... branching, add a third branch for external engines that:

  1. Builds an AgentContext from ContextLoader(workspace).load(run.task_id) and wraps it
  2. Instantiates the correct adapter from the engine registry (a simple dict or if/elif mapping engine name → adapter class, housed in core/engine.py)
  3. Creates a VerificationLoop(workspace, max_retries=5)
  4. Calls loop.run_with_verification(adapter, context, workspace.repo_path)
  5. Maps the returned AgentResult.status to AgentState for downstream compatibility

The flow for external engines:

sequenceDiagram
    participant R as runtime.execute_agent()
    participant VL as VerificationLoop
    participant A as ExternalAdapter
    participant G as gates.run()
    participant QF as quick_fixes

    R->>VL: run_with_verification(adapter, context, path)
    loop up to max_retries
        VL->>A: adapter.execute(context, workspace)
        A-->>VL: AgentResult
        VL->>G: gates.run(workspace)
        G-->>VL: GateResult
        alt gates passed
            VL-->>R: AgentResult(COMPLETED)
        else quick fix applies
            VL->>QF: find_quick_fix + apply_quick_fix
            QF-->>VL: applied
        else re-invoke agent
            VL->>VL: enrich context.previous_errors
            VL->>A: adapter.execute(enriched_context, workspace)
        else max retries exceeded
            VL->>VL: create blocker
            VL-->>R: AgentResult(BLOCKED)
        end
    end

---

Step 5 — Add external engine adapter stubs in core/engine.py

In the same file:codeframe/core/engine.py, add minimal adapter classes for each declared external engine. Each must satisfy the AgentAdapter protocol:

• ClaudeCodeAdapter — execute() invokes claude CLI via subprocess.run in the workspace directory, with the task prompt built from AgentContext
• AiderAdapter — same pattern for aider CLI
• CodexAdapter — same pattern for codex CLI

Each adapter constructs the prompt including the previous_errors from AgentContext when context.attempt > 0, so that errors from prior gate failures are communicated to the external tool.

Add a registry dict ENGINE_REGISTRY: dict[str, type[AgentAdapter]] mapping engine names to adapter classes, and a helper get_adapter(engine: str, workspace: Workspace, **kwargs) -> AgentAdapter.

---

Summary of files affected

File	Action
codeframe/core/engine.py	Create — AgentContext, AgentResult, AgentAdapter protocol, ReactAgentAdapter, external adapter stubs, ENGINE_REGISTRY
codeframe/core/verification_loop.py	Create — VerificationLoop class with gate-check, quick-fix, tracker, escalation, and run_with_verification
codeframe/core/react_agent.py	Modify — delegate _try_quick_fix, fix_tracker, and escalation to VerificationLoop; shrink _run_final_verification
codeframe/core/runtime.py	Modify — expand valid_engines, add third branch for external engines using VerificationLoop

Import In IDE

🤖 Prompt for AI Agents

## Observations

The verification/retry logic in `_run_final_verification()` at `file:codeframe/core/react_agent.py` (lines 425–534) already contains the full pattern: run gates → try quick fix → enrich context → re-invoke LLM → repeat up to `max_verification_retries`. The codebase uses `dataclasses` (not Pydantic) for pure-Python domain objects. Issue #409's `AgentAdapter`/`AgentResult`/`AgentContext` do not exist yet and must be defined first. The `gates`, `quick_fixes`, `fix_tracker`, and `blockers` modules in `file:codeframe/core/` are all headless and ready for reuse.

## Approach

Define the `AgentAdapter` protocol and supporting types in a new `core/engine.py` file, then extract the verification loop into `core/verification_loop.py` as a standalone class that any adapter can call. Finally, wire it into `execute_agent()` in `core/runtime.py` and refactor `ReactAgent` to delegate to it. This keeps the existing behavior identical while enabling all future engines to share the same gate-enforcement path.

---

## Implementation Steps

### Step 1 — Define the `AgentAdapter` Protocol and supporting types (`core/engine.py`)

Create `file:codeframe/core/engine.py` as a new headless module. Model its style after the existing `dataclass`-based modules in `file:codeframe/core/`.

Define three `@dataclass` classes and one `Protocol`:

- **`AgentContext`** — carries everything needed to invoke any engine for a task:
  - `task_id: str`
  - `task_context: TaskContext` (reuse the existing `TaskContext` from `file:codeframe/core/context.py`)
  - `previous_errors: list[str]` (accumulated gate failure messages across retries)
  - `attempt: int` (0-based retry index)

- **`AgentResult`** — wraps the outcome of a single adapter execution:
  - `status: AgentStatus` (reuse from `file:codeframe/core/agent.py`)
  - `blocker_id: Optional[str]`
  - `error_summary: Optional[str]`

- **`AgentAdapter`** (a `typing.Protocol`) — the interface every engine must satisfy:
  - `def execute(self, context: AgentContext, workspace: Workspace) -> AgentResult: ...`

- **`ReactAgentAdapter`** — a thin adapter that wraps the existing `ReactAgent` and delegates `execute()` to its `run()` method, mapping `AgentStatus` to `AgentResult`. It must accept the same constructor args as `ReactAgent` (`workspace`, `llm_provider`, `dry_run`, etc.), storing them and forwarding them on `execute()`.

---

### Step 2 — Create `VerificationLoop` in `core/verification_loop.py`

Create `file:codeframe/core/verification_loop.py`. The class mirrors the structure of `_run_final_verification()` but is decoupled from `ReactAgent`. Follow the same import style used in `file:codeframe/core/react_agent.py`.

**`VerificationLoop.__init__(self, workspace, max_retries=5)`**
- Accept `workspace: Workspace` and `max_retries: int`
- Instantiate a `FixAttemptTracker` internally (same as `ReactAgent.__init__` does)
- Store a logger

**`VerificationLoop.run_with_verification(self, adapter, context, workspace_path) -> AgentResult`**

The loop logic:

```
for attempt in range(max_retries + 1):
    result = adapter.execute(enriched_context, workspace)
    if result.status is BLOCKED:
        return result immediately
    gate_result = gates.run(workspace)
    if gate_result.passed:
        return AgentResult(COMPLETED)
    if attempt >= max_retries:
        create escalation blocker via blockers.create()
        return AgentResult(BLOCKED, blocker_id=...)
    error_summary = gate_result.get_error_summary() or gate_result.summary
    # Try quick fix (no agent re-invoke needed)
    quick_fix_applied = _try_quick_fix(error_summary, workspace)
    if quick_fix_applied:
        continue  # re-run gates in next iteration without re-invoking adapter
    # Enrich context for next adapter invocation
    context.previous_errors.append(error_summary)
    context.attempt = attempt + 1
    record failure in fix_tracker
    check fix_tracker.should_escalate() → if True, create blocker and return BLOCKED
```

**Private `_try_quick_fix(self, error_summary, workspace) -> bool`**
- Call `find_quick_fix(error_summary, repo_path=workspace.repo_path)`
- If a fix is found, call `apply_quick_fix(fix, workspace.repo_path)`
- Return success bool — mirrors `ReactAgent._try_quick_fix()` in `file:codeframe/core/react_agent.py` (lines 915–976)

**`_build_error_prompt(self, context: AgentContext) -> str`**
- Construct the "The following verification checks failed: [errors]. Please fix these issues." addendum
- This becomes part of the enriched prompt when `AgentContext.previous_errors` is non-empty; the adapter is responsible for incorporating it into its LLM prompt

---

### Step 3 — Refactor `ReactAgent` to delegate to `VerificationLoop`

In `file:codeframe/core/react_agent.py`:

- Import `VerificationLoop` from `codeframe.core.verification_loop`
- In `ReactAgent.__init__`, instantiate `self._verification_loop = VerificationLoop(workspace, max_retries=max_verification_retries)` instead of managing the tracker inline
- Modify `ReactAgent.run()` to replace the direct call to `self._run_final_verification(system_prompt)` with a call through `_verification_loop`. Because `ReactAgent` still runs its own main loop first, the simplest approach is: after the `_react_loop` completes, use `VerificationLoop` with a pass-through adapter that already has the main loop done — i.e., keep the LLM mini-loop variant. See the note below.

> **Compatibility note**: `_run_final_verification` uses the `ReactAgent`'s own LLM provider for its mini-loop, so the refactor must preserve that. Extract the gate-check + quick-fix + escalation logic into `VerificationLoop`, but keep the "mini ReAct fix loop" invocation inside `ReactAgent._run_final_verification` as a thin wrapper that now delegates gate-running, quick-fix, and tracker logic to `VerificationLoop._check_and_fix_once(attempt, error_summary)`. This keeps `ReactAgent`'s existing behavior exactly the same.

Specifically:
- Move `_try_quick_fix`, `_create_escalation_blocker`, and the `fix_tracker` usage out of `ReactAgent` into `VerificationLoop`
- `ReactAgent._run_final_verification` shrinks to: call `gates.run()`, ask `VerificationLoop` to run quick-fix + tracker check, if not escalating run the mini LLM loop, repeat

---

### Step 4 — Update `execute_agent()` in `core/runtime.py` for external engines

In `file:codeframe/core/runtime.py`, extend the `execute_agent()` function:

- Add new engine name constants: `"claude-code"`, `"aider"`, `"codex"` alongside the existing `"react"` and `"plan"` validation set.

- After the existing `if engine == "react": ... else: ...` branching, add a third branch for external engines that:
  1. Builds an `AgentContext` from `ContextLoader(workspace).load(run.task_id)` and wraps it
  2. Instantiates the correct adapter from the engine registry (a simple `dict` or `if/elif` mapping engine name → adapter class, housed in `core/engine.py`)
  3. Creates a `VerificationLoop(workspace, max_retries=5)`
  4. Calls `loop.run_with_verification(adapter, context, workspace.repo_path)`
  5. Maps the returned `AgentResult.status` to `AgentState` for downstream compatibility

The flow for external engines:
```
sequenceDiagram
    participant R as runtime.execute_agent()
    participant VL as VerificationLoop
    participant A as ExternalAdapter
    participant G as gates.run()
    participant QF as quick_fixes

    R->>VL: run_with_verification(adapter, context, path)
    loop up to max_retries
        VL->>A: adapter.execute(context, workspace)
        A-->>VL: AgentResult
        VL->>G: gates.run(workspace)
        G-->>VL: GateResult
        alt gates passed
            VL-->>R: AgentResult(COMPLETED)
        else quick fix applies
            VL->>QF: find_quick_fix + apply_quick_fix
            QF-->>VL: applied
        else re-invoke agent
            VL->>VL: enrich context.previous_errors
            VL->>A: adapter.execute(enriched_context, workspace)
        else max retries exceeded
            VL->>VL: create blocker
            VL-->>R: AgentResult(BLOCKED)
        end
    end
```

---

### Step 5 — Add external engine adapter stubs in `core/engine.py`

In the same `file:codeframe/core/engine.py`, add minimal adapter classes for each declared external engine. Each must satisfy the `AgentAdapter` protocol:

- **`ClaudeCodeAdapter`** — `execute()` invokes `claude` CLI via `subprocess.run` in the workspace directory, with the task prompt built from `AgentContext`
- **`AiderAdapter`** — same pattern for `aider` CLI
- **`CodexAdapter`** — same pattern for `codex` CLI

Each adapter constructs the prompt including the `previous_errors` from `AgentContext` when `context.attempt > 0`, so that errors from prior gate failures are communicated to the external tool.

Add a registry dict `ENGINE_REGISTRY: dict[str, type[AgentAdapter]]` mapping engine names to adapter classes, and a helper `get_adapter(engine: str, workspace: Workspace, **kwargs) -> AgentAdapter`.

---

### Summary of files affected

| File | Action |
|---|---|
| `codeframe/core/engine.py` | **Create** — `AgentContext`, `AgentResult`, `AgentAdapter` protocol, `ReactAgentAdapter`, external adapter stubs, `ENGINE_REGISTRY` |
| `codeframe/core/verification_loop.py` | **Create** — `VerificationLoop` class with gate-check, quick-fix, tracker, escalation, and `run_with_verification` |
| `codeframe/core/react_agent.py` | **Modify** — delegate `_try_quick_fix`, `fix_tracker`, and escalation to `VerificationLoop`; shrink `_run_final_verification` |
| `codeframe/core/runtime.py` | **Modify** — expand `valid_engines`, add third branch for external engines using `VerificationLoop` |

---

Execution Information

Branch: main
Commit: 705b160

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

🔗 Related PRs

#356 - fix: agent execution resilience + Phase 2.5 architecture research [merged]
#363 - feat: wire ReactAgent into runtime with --engine react CLI flag [merged]
#369 - test: add CLI integration tests for ReactAgent runtime parameters [merged]
#384 - feat: adaptive iteration budget, autofix, and loop detection for ReactAgent [merged]
#386 - [Phase 2.5-F] Harden verification gates and add batch-level QA [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!