[Phase 2.5-C] Integration tests for ReactAgent runtime parameters

[frankbria]

Context

PR #367 (issue #362) added full runtime parameter support to ReactAgent (dry_run, verbose, on_event, debug, output_logger, fix_coordinator). Unit test coverage is solid (60 tests), but end-to-end CLI integration tests are missing.

These tests would exercise the full path from CLI → runtime → ReactAgent, catching wiring issues that unit tests with mocked dependencies can't.

Integration Tests to Add

1. Verbose mode with ReactAgent engine

cf work start <id> --execute --verbose --engine react

• Verify stdout contains [ReactAgent] prefixed progress messages
• Verify output logger captures the same messages (for cf work follow)
• Compare behavior parity with plan-based agent verbose mode

2. Dry-run mode with ReactAgent engine

cf work start <id> --execute --dry-run --engine react

• Verify no files are created or modified in the workspace
• Verify [DRY RUN] stub results appear in output
• Verify read tools (read_file, list_files, search_codebase, run_tests) still execute
• Verify unknown tools are blocked (fail-safe default)

3. Streaming with cf work follow using ReactAgent

cf work start <id> --execute --engine react &
cf work follow <id>

• Verify SSE events are emitted during ReactAgent execution
• Verify output.log is written in real-time via RunOutputLogger
• Verify stream closes cleanly on task completion

Acceptance Criteria

• All 3 integration test scenarios pass
• Tests use real CLI invocation (subprocess or CliRunner) not just unit mocks
• Tests work with MockProvider (no real LLM calls needed)
• Tests marked with @pytest.mark.v2

Related

• Closes testing gap identified in PR feat: extend ReactAgent interface to support full runtime parameters (#362) #367 review
• Depends on: [Phase 2.5-C] Extend ReactAgent interface to support full runtime parameters #362 (ReactAgent runtime params — merged)
• Phase: 2.5-C (ReactAgent feature parity)

[traycerai]

Plan

Observations

The codebase has solid ReactAgent runtime parameter support (PR #367, issue #362) with 60 unit tests covering event_publisher wiring. However, end-to-end CLI integration tests are missing. The existing integration test suite in file:tests/cli/test_v2_cli_integration.py uses CliRunner with MockProvider and is marked with @pytest.mark.v2. ReactAgent currently only receives event_publisher from runtime (file:codeframe/core/runtime.py lines 686-690), and dry-run mode explicitly raises ValueError for react engine (lines 635-639). The file:codeframe/core/streaming.py module provides both file-based streaming (RunOutputLogger) and event-based streaming (EventPublisher) infrastructure.

Approach

Add three integration tests to file:tests/cli/test_v2_cli_integration.py that exercise the full CLI → runtime → ReactAgent path. Tests will use CliRunner for real CLI invocation and MockProvider to avoid LLM calls. The verbose test will verify ProgressEvent emissions via EventPublisher, the dry-run test will verify the current fail-fast behavior (ValueError), and the streaming test will verify SSE event flow with cf work follow. All tests will be marked with @pytest.mark.v2 for consistency with existing integration tests.

Implementation Steps

1. Add Verbose Mode Integration Test

Add test method test_react_verbose_mode to the TestAIIntegration class in file:tests/cli/test_v2_cli_integration.py (after line 907):

• Invoke CLI command: cf work start <id> --execute --verbose --engine react -w <workspace>
• Use MockProvider with a simple text response to complete quickly
• Verify exit code is 0 (success)
• Verify stdout contains verbose output indicators (task completion message)
• Note: Full ProgressEvent verification requires mocking EventPublisher, which is created internally by runtime. For integration testing, verify the command completes successfully with verbose flag - detailed event emission is already covered by unit tests in file:tests/core/test_react_agent.py (TestEventEmissions, TestPhaseEmission classes)

2. Add Dry-Run Mode Integration Test

Add test method test_react_dry_run_fails_fast to the TestAIIntegration class in file:tests/cli/test_v2_cli_integration.py:

• Invoke CLI command: cf work start <id> --execute --dry-run --engine react -w <workspace>
• Verify exit code is non-zero (failure)
• Verify error message contains "dry_run" or "not supported" (matches ValueError from file:codeframe/core/runtime.py lines 635-639)
• This test documents current behavior - if dry-run support is added later, update test to verify no files are modified

3. Add Streaming Integration Test with cf work follow

Add test method test_react_streaming_with_follow to a new test class TestReactAgentStreaming in file:tests/cli/test_v2_cli_integration.py:

Test structure:

• Create workspace with ready task
• Start execution in background thread: cf work start <id> --execute --engine react -w <workspace>
• Use MockProvider with text response for quick completion
• In main thread, invoke: cf work follow <id> -w <workspace> --timeout 5
• Verify follow command captures output (check for task completion message or run status)
• Verify output.log file exists at .codeframe/runs/<run_id>/output.log

Implementation pattern (reference file:tests/cli/test_work_follow.py line 126-162):

• Use threading.Thread to run work start in background
• Add small sleep before invoking follow to ensure run is created
• Use --timeout flag on follow command to prevent hanging
• Join thread with timeout to ensure cleanup

4. Test Fixtures and Helpers

Reuse existing fixtures from file:tests/cli/test_v2_cli_integration.py:

• workspace_with_ready_tasks (line 66) - provides workspace with READY tasks
• mock_llm (line 808) - provides MockProvider with canned responses
• runner (CliRunner instance, line 31)

Add helper constant for ReactAgent mock response:

MOCK_REACT_COMPLETION = "Task completed successfully."

5. Test Markers and Documentation

• Mark all three tests with @pytest.mark.v2 decorator
• Add docstrings explaining what each test verifies
• Reference issue [Phase 2.5-C] Integration tests for ReactAgent runtime parameters #368 in test docstrings for traceability

Visual Test Flow

sequenceDiagram
    participant Test as Integration Test
    participant CLI as cf work start
    participant Runtime as runtime.execute_agent()
    participant ReactAgent as ReactAgent
    participant Publisher as EventPublisher
    
    Note over Test,Publisher: Test 1: Verbose Mode
    Test->>CLI: cf work start --execute --verbose --engine react
    CLI->>Runtime: execute_agent(verbose=True, engine="react")
    Runtime->>ReactAgent: ReactAgent(event_publisher=...)
    ReactAgent->>Publisher: publish_sync(ProgressEvent)
    ReactAgent-->>Runtime: AgentStatus.COMPLETED
    Runtime-->>CLI: AgentState
    CLI-->>Test: Exit 0, stdout with completion message
    
    Note over Test,Publisher: Test 2: Dry-Run (Current Behavior)
    Test->>CLI: cf work start --execute --dry-run --engine react
    CLI->>Runtime: execute_agent(dry_run=True, engine="react")
    Runtime-->>CLI: ValueError("dry_run not supported")
    CLI-->>Test: Exit 1, error message
    
    Note over Test,Publisher: Test 3: Streaming with Follow
    Test->>CLI: cf work start --execute --engine react (background)
    CLI->>Runtime: execute_agent(engine="react")
    Runtime->>ReactAgent: ReactAgent(event_publisher=...)
    Test->>CLI: cf work follow <id> (main thread)
    CLI->>Publisher: subscribe(task_id)
    ReactAgent->>Publisher: publish_sync(ProgressEvent)
    Publisher-->>CLI: Stream events
    CLI-->>Test: Output with task progress

Loading

File Locations

• Test file: file:tests/cli/test_v2_cli_integration.py
• Insert location: After line 907 (end of TestAIIntegration class) for tests 1-2, new class after line 958 for test 3
• Reference implementations:
  • Existing dry-run test: lines 858-880
  • Streaming test pattern: file:tests/cli/test_work_follow.py lines 126-162
  • MockProvider usage: lines 808-907

Acceptance Criteria Verification

Criterion	Implementation
All 3 integration test scenarios pass	Three test methods added with distinct scenarios
Tests use real CLI invocation	CliRunner.invoke() with full command strings
Tests work with MockProvider	Use mock_llm fixture, no real LLM calls
Tests marked with @pytest.mark.v2	Decorator applied to all three tests

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has solid ReactAgent runtime parameter support (PR #367, issue #362) with 60 unit tests covering `event_publisher` wiring. However, end-to-end CLI integration tests are missing. The existing integration test suite in `file:tests/cli/test_v2_cli_integration.py` uses CliRunner with MockProvider and is marked with `@pytest.mark.v2`. ReactAgent currently only receives `event_publisher` from runtime (file:codeframe/core/runtime.py lines 686-690), and dry-run mode explicitly raises ValueError for react engine (lines 635-639). The `file:codeframe/core/streaming.py` module provides both file-based streaming (RunOutputLogger) and event-based streaming (EventPublisher) infrastructure.

## Approach

Add three integration tests to `file:tests/cli/test_v2_cli_integration.py` that exercise the full CLI → runtime → ReactAgent path. Tests will use CliRunner for real CLI invocation and MockProvider to avoid LLM calls. The verbose test will verify ProgressEvent emissions via EventPublisher, the dry-run test will verify the current fail-fast behavior (ValueError), and the streaming test will verify SSE event flow with `cf work follow`. All tests will be marked with `@pytest.mark.v2` for consistency with existing integration tests.

## Implementation Steps

### 1. Add Verbose Mode Integration Test

Add test method `test_react_verbose_mode` to the `TestAIIntegration` class in `file:tests/cli/test_v2_cli_integration.py` (after line 907):

- Invoke CLI command: `cf work start <id> --execute --verbose --engine react -w <workspace>`
- Use MockProvider with a simple text response to complete quickly
- Verify exit code is 0 (success)
- Verify stdout contains verbose output indicators (task completion message)
- **Note**: Full ProgressEvent verification requires mocking EventPublisher, which is created internally by runtime. For integration testing, verify the command completes successfully with verbose flag - detailed event emission is already covered by unit tests in `file:tests/core/test_react_agent.py` (TestEventEmissions, TestPhaseEmission classes)

### 2. Add Dry-Run Mode Integration Test

Add test method `test_react_dry_run_fails_fast` to the `TestAIIntegration` class in `file:tests/cli/test_v2_cli_integration.py`:

- Invoke CLI command: `cf work start <id> --execute --dry-run --engine react -w <workspace>`
- Verify exit code is non-zero (failure)
- Verify error message contains "dry_run" or "not supported" (matches ValueError from `file:codeframe/core/runtime.py` lines 635-639)
- This test documents current behavior - if dry-run support is added later, update test to verify no files are modified

### 3. Add Streaming Integration Test with `cf work follow`

Add test method `test_react_streaming_with_follow` to a new test class `TestReactAgentStreaming` in `file:tests/cli/test_v2_cli_integration.py`:

**Test structure:**
- Create workspace with ready task
- Start execution in background thread: `cf work start <id> --execute --engine react -w <workspace>`
- Use MockProvider with text response for quick completion
- In main thread, invoke: `cf work follow <id> -w <workspace> --timeout 5`
- Verify follow command captures output (check for task completion message or run status)
- Verify output.log file exists at `.codeframe/runs/<run_id>/output.log`

**Implementation pattern** (reference `file:tests/cli/test_work_follow.py` line 126-162):
- Use `threading.Thread` to run work start in background
- Add small sleep before invoking follow to ensure run is created
- Use `--timeout` flag on follow command to prevent hanging
- Join thread with timeout to ensure cleanup

### 4. Test Fixtures and Helpers

Reuse existing fixtures from `file:tests/cli/test_v2_cli_integration.py`:
- `workspace_with_ready_tasks` (line 66) - provides workspace with READY tasks
- `mock_llm` (line 808) - provides MockProvider with canned responses
- `runner` (CliRunner instance, line 31)

Add helper constant for ReactAgent mock response:
```python
MOCK_REACT_COMPLETION = "Task completed successfully."
```

### 5. Test Markers and Documentation

- Mark all three tests with `@pytest.mark.v2` decorator
- Add docstrings explaining what each test verifies
- Reference issue #368 in test docstrings for traceability

## Visual Test Flow

```mermaid
sequenceDiagram
    participant Test as Integration Test
    participant CLI as cf work start
    participant Runtime as runtime.execute_agent()
    participant ReactAgent as ReactAgent
    participant Publisher as EventPublisher
    
    Note over Test,Publisher: Test 1: Verbose Mode
    Test->>CLI: cf work start --execute --verbose --engine react
    CLI->>Runtime: execute_agent(verbose=True, engine="react")
    Runtime->>ReactAgent: ReactAgent(event_publisher=...)
    ReactAgent->>Publisher: publish_sync(ProgressEvent)
    ReactAgent-->>Runtime: AgentStatus.COMPLETED
    Runtime-->>CLI: AgentState
    CLI-->>Test: Exit 0, stdout with completion message
    
    Note over Test,Publisher: Test 2: Dry-Run (Current Behavior)
    Test->>CLI: cf work start --execute --dry-run --engine react
    CLI->>Runtime: execute_agent(dry_run=True, engine="react")
    Runtime-->>CLI: ValueError("dry_run not supported")
    CLI-->>Test: Exit 1, error message
    
    Note over Test,Publisher: Test 3: Streaming with Follow
    Test->>CLI: cf work start --execute --engine react (background)
    CLI->>Runtime: execute_agent(engine="react")
    Runtime->>ReactAgent: ReactAgent(event_publisher=...)
    Test->>CLI: cf work follow <id> (main thread)
    CLI->>Publisher: subscribe(task_id)
    ReactAgent->>Publisher: publish_sync(ProgressEvent)
    Publisher-->>CLI: Stream events
    CLI-->>Test: Output with task progress
```

## File Locations

- **Test file**: `file:tests/cli/test_v2_cli_integration.py`
- **Insert location**: After line 907 (end of TestAIIntegration class) for tests 1-2, new class after line 958 for test 3
- **Reference implementations**: 
  - Existing dry-run test: lines 858-880
  - Streaming test pattern: `file:tests/cli/test_work_follow.py` lines 126-162
  - MockProvider usage: lines 808-907

## Acceptance Criteria Verification

| Criterion | Implementation |
|-----------|----------------|
| All 3 integration test scenarios pass | Three test methods added with distinct scenarios |
| Tests use real CLI invocation | CliRunner.invoke() with full command strings |
| Tests work with MockProvider | Use `mock_llm` fixture, no real LLM calls |
| Tests marked with `@pytest.mark.v2` | Decorator applied to all three tests |

---

Execution Information

Branch: main
Commit: 957defa

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

🔗 Related PRs

#304 - test(cli): add comprehensive v2 CLI integration test suite [merged]
#319 - feat(cli): add cf work follow for live execution streaming [merged]
#356 - fix: agent execution resilience + Phase 2.5 architecture research [merged]
#363 - feat: wire ReactAgent into runtime with --engine react CLI flag [merged]
#364 - feat: phase-based event emission for ReactAgent progress reporting [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!