[P1] E2E Test cases with existing state issues

[frankbria]

The full E2E test suite has two failures in Chromium related to "returning user" cases:

The 2 Chromium failures are in state reconciliation tests:

• should show in-progress tasks when tasks already in progress (line 332)
• should show completed state when project already finished (line 514)

[traycerai]

Plan

Observations

The E2E test suite has two Chromium-specific failures related to "returning user" state reconciliation scenarios. The existing file:tests/e2e/test_late_joining_user.spec.ts file tests late-joining users but doesn't cover all edge cases. The issue references tests at lines 332 and 514 that need to handle: (1) users joining when tasks are already in progress, and (2) users joining when a project is already completed. The state management system uses a reducer pattern with FULL_RESYNC capability, but the UI components may not properly initialize state from API data when WebSocket events were missed.

Approach

The failures stem from insufficient state reconciliation when users navigate to a project after missing WebSocket events. The solution involves adding comprehensive E2E tests that verify the UI correctly displays state fetched from API endpoints (not WebSocket events) for returning users. These tests will validate that file:web-ui/src/lib/agentStateSync.ts properly syncs state and that dashboard components like file:web-ui/src/components/Dashboard.tsx render correctly based on API-fetched data. The tests will use seeded database state to simulate projects with in-progress and completed tasks, then verify the UI displays the correct state without relying on WebSocket history.

Implementation Steps

1. Add State Reconciliation Test Cases

File: file:tests/e2e/test_late_joining_user.spec.ts

Add two new test cases to the existing test suite:

Test 1: "should show in-progress tasks when tasks already in progress"

• Navigate to a project with tasks in in_progress status (use seeded project data)
• Verify the dashboard loads without WebSocket events
• Assert that [data-testid="in-progress-tasks"] displays the correct count
• Assert that task cards show "In Progress" status badges
• Verify file:web-ui/src/hooks/useAgentState.ts activeTasks derived state is populated
• Confirm agents assigned to tasks show "working" status
• Validate that the Tasks tab displays the task list with correct status indicators

Test 2: "should show completed state when project already finished"

• Navigate to a project in "completed" phase with all tasks finished
• Verify project status badge shows "COMPLETED"
• Assert that [data-testid="completed-tasks"] shows 100% completion
• Verify progress bar shows 100% (check file:web-ui/src/components/PhaseProgress.tsx)
• Confirm no active agents are displayed
• Validate that completion timestamp is visible
• Check that "View Results" or similar completion UI is accessible

2. Enhance Seed Data for Test Scenarios

File: file:tests/e2e/seed-test-data.py

Add two new test projects to the seeding script:

Project 3: "In-Progress State Project"

• Phase: development
• Create 5 tasks: 2 with status in_progress, 2 completed, 1 pending
• Assign 2 agents with status working to the in-progress tasks
• Set realistic timestamps for task start times
• Add activity feed entries for task assignments

Project 4: "Completed State Project"

• Phase: completed
• Create 3 tasks all with status completed
• Set completion timestamps for all tasks
• Mark all agents as idle
• Set project completion timestamp
• Add final activity entries showing project completion

3. Update Global Setup to Use New Seed Data

File: file:tests/e2e/global-setup.ts

• Ensure the seed script creates Projects 3 and 4
• Add environment variables E2E_TEST_PROJECT_INPROGRESS_ID=3 and E2E_TEST_PROJECT_COMPLETED_ID=4
• Verify database initialization includes these projects before tests run

4. Add API Response Validation

File: file:tests/e2e/test_late_joining_user.spec.ts

For both new tests, add explicit API response validation:

• Wait for /api/projects/{id}/status response and verify status code 200
• Wait for /api/projects/{id}/tasks response and validate task data structure
• Wait for /api/projects/{id}/agents response and verify agent assignments
• Assert that API responses contain expected state before checking UI
• Use waitForAPIResponse helper from file:tests/e2e/test-utils.ts

5. Verify State Sync Logic

File: file:web-ui/src/lib/agentStateSync.ts

Review and document (no code changes needed) that fullStateResync function:

• Fetches agents, tasks, and activity in parallel
• Handles missing data gracefully with empty arrays
• Returns complete state payload with timestamp
• Is called by file:web-ui/src/contexts/AgentStateContext.tsx on mount

6. Add Test Utilities for State Verification

File: file:tests/e2e/test-utils.ts

Add helper functions:

verifyTaskState(page, expectedState)

• Accepts expected task counts by status
• Queries DOM for task status indicators
• Returns assertion results for test validation

verifyProjectCompletion(page, projectId)

• Checks completion badge visibility
• Validates progress bar at 100%
• Confirms no active work indicators
• Returns boolean for completion state

7. Update Playwright Configuration

File: file:tests/e2e/playwright.config.ts

• Ensure test timeout is sufficient for state loading (currently 30s/60s is adequate)
• Verify revalidateOnFocus and revalidateOnReconnect SWR options are enabled
• Confirm adaptive polling intervals don't interfere with test execution

8. Add Test Documentation

File: file:tests/e2e/README.md

Update the "Known Issues and Limitations" section:

• Document the state reconciliation test scenarios
• Explain the difference between "late-joining user" and "returning user" tests
• Add troubleshooting steps for state sync failures
• Reference the FULL_RESYNC action in file:web-ui/src/reducers/agentReducer.ts

9. Verify Dashboard Component State Initialization

File: file:web-ui/src/components/Dashboard.tsx

Review (no changes needed) that the component:

• Uses SWR to fetch project data on mount (lines 197-237)
• Has adaptive polling based on isActiveWork detection
• Properly handles projectProgress with defensive defaults (line 334)
• Displays task counts from useAgentState hook derived state

10. Run and Validate Tests

Execute the new tests in Chromium specifically:

cd tests/e2e
npx playwright test test_late_joining_user.spec.ts --project=chromium

Verify both new tests pass:

• "should show in-progress tasks when tasks already in progress"
• "should show completed state when project already finished"

Check test report for any timing issues or flakiness, adjust timeouts if needed.

Test Execution Flow

sequenceDiagram
    participant Test as E2E Test
    participant DB as Test Database
    participant API as Backend API
    participant UI as Dashboard UI
    participant State as AgentState Hook

    Test->>DB: Seed project with in-progress tasks
    Test->>UI: Navigate to /projects/3
    UI->>API: GET /projects/3/status
    API->>UI: Return project data
    UI->>API: GET /projects/3/tasks
    API->>UI: Return tasks (2 in_progress)
    UI->>API: GET /projects/3/agents
    API->>UI: Return agents (2 working)
    UI->>State: Initialize state from API data
    State->>UI: Provide activeTasks, activeAgents
    UI->>UI: Render dashboard with correct state
    Test->>UI: Assert in-progress count = 2
    Test->>UI: Assert agent status = working
    Test->>Test: ✅ Test passes

Loading

Success Criteria

• Both new test cases pass consistently in Chromium browser
• Tests verify UI state matches API-fetched data without WebSocket dependency
• No flakiness or timing issues in test execution
• Test coverage includes both in-progress and completed project states
• Documentation clearly explains state reconciliation testing approach

Import In IDE

🤖 Prompt for AI Agents

## Observations

The E2E test suite has two Chromium-specific failures related to "returning user" state reconciliation scenarios. The existing `file:tests/e2e/test_late_joining_user.spec.ts` file tests late-joining users but doesn't cover all edge cases. The issue references tests at lines 332 and 514 that need to handle: (1) users joining when tasks are already in progress, and (2) users joining when a project is already completed. The state management system uses a reducer pattern with `FULL_RESYNC` capability, but the UI components may not properly initialize state from API data when WebSocket events were missed.

## Approach

The failures stem from insufficient state reconciliation when users navigate to a project after missing WebSocket events. The solution involves adding comprehensive E2E tests that verify the UI correctly displays state fetched from API endpoints (not WebSocket events) for returning users. These tests will validate that `file:web-ui/src/lib/agentStateSync.ts` properly syncs state and that dashboard components like `file:web-ui/src/components/Dashboard.tsx` render correctly based on API-fetched data. The tests will use seeded database state to simulate projects with in-progress and completed tasks, then verify the UI displays the correct state without relying on WebSocket history.

## Implementation Steps

### 1. Add State Reconciliation Test Cases

**File**: `file:tests/e2e/test_late_joining_user.spec.ts`

Add two new test cases to the existing test suite:

**Test 1: "should show in-progress tasks when tasks already in progress"**
- Navigate to a project with tasks in `in_progress` status (use seeded project data)
- Verify the dashboard loads without WebSocket events
- Assert that `[data-testid="in-progress-tasks"]` displays the correct count
- Assert that task cards show "In Progress" status badges
- Verify `file:web-ui/src/hooks/useAgentState.ts` `activeTasks` derived state is populated
- Confirm agents assigned to tasks show "working" status
- Validate that the Tasks tab displays the task list with correct status indicators

**Test 2: "should show completed state when project already finished"**
- Navigate to a project in "completed" phase with all tasks finished
- Verify project status badge shows "COMPLETED"
- Assert that `[data-testid="completed-tasks"]` shows 100% completion
- Verify progress bar shows 100% (check `file:web-ui/src/components/PhaseProgress.tsx`)
- Confirm no active agents are displayed
- Validate that completion timestamp is visible
- Check that "View Results" or similar completion UI is accessible

### 2. Enhance Seed Data for Test Scenarios

**File**: `file:tests/e2e/seed-test-data.py`

Add two new test projects to the seeding script:

**Project 3: "In-Progress State Project"**
- Phase: `development`
- Create 5 tasks: 2 with status `in_progress`, 2 `completed`, 1 `pending`
- Assign 2 agents with status `working` to the in-progress tasks
- Set realistic timestamps for task start times
- Add activity feed entries for task assignments

**Project 4: "Completed State Project"**
- Phase: `completed`
- Create 3 tasks all with status `completed`
- Set completion timestamps for all tasks
- Mark all agents as `idle`
- Set project completion timestamp
- Add final activity entries showing project completion

### 3. Update Global Setup to Use New Seed Data

**File**: `file:tests/e2e/global-setup.ts`

- Ensure the seed script creates Projects 3 and 4
- Add environment variables `E2E_TEST_PROJECT_INPROGRESS_ID=3` and `E2E_TEST_PROJECT_COMPLETED_ID=4`
- Verify database initialization includes these projects before tests run

### 4. Add API Response Validation

**File**: `file:tests/e2e/test_late_joining_user.spec.ts`

For both new tests, add explicit API response validation:

- Wait for `/api/projects/{id}/status` response and verify status code 200
- Wait for `/api/projects/{id}/tasks` response and validate task data structure
- Wait for `/api/projects/{id}/agents` response and verify agent assignments
- Assert that API responses contain expected state before checking UI
- Use `waitForAPIResponse` helper from `file:tests/e2e/test-utils.ts`

### 5. Verify State Sync Logic

**File**: `file:web-ui/src/lib/agentStateSync.ts`

Review and document (no code changes needed) that `fullStateResync` function:

- Fetches agents, tasks, and activity in parallel
- Handles missing data gracefully with empty arrays
- Returns complete state payload with timestamp
- Is called by `file:web-ui/src/contexts/AgentStateContext.tsx` on mount

### 6. Add Test Utilities for State Verification

**File**: `file:tests/e2e/test-utils.ts`

Add helper functions:

**`verifyTaskState(page, expectedState)`**
- Accepts expected task counts by status
- Queries DOM for task status indicators
- Returns assertion results for test validation

**`verifyProjectCompletion(page, projectId)`**
- Checks completion badge visibility
- Validates progress bar at 100%
- Confirms no active work indicators
- Returns boolean for completion state

### 7. Update Playwright Configuration

**File**: `file:tests/e2e/playwright.config.ts`

- Ensure test timeout is sufficient for state loading (currently 30s/60s is adequate)
- Verify `revalidateOnFocus` and `revalidateOnReconnect` SWR options are enabled
- Confirm adaptive polling intervals don't interfere with test execution

### 8. Add Test Documentation

**File**: `file:tests/e2e/README.md`

Update the "Known Issues and Limitations" section:

- Document the state reconciliation test scenarios
- Explain the difference between "late-joining user" and "returning user" tests
- Add troubleshooting steps for state sync failures
- Reference the `FULL_RESYNC` action in `file:web-ui/src/reducers/agentReducer.ts`

### 9. Verify Dashboard Component State Initialization

**File**: `file:web-ui/src/components/Dashboard.tsx`

Review (no changes needed) that the component:

- Uses SWR to fetch project data on mount (lines 197-237)
- Has adaptive polling based on `isActiveWork` detection
- Properly handles `projectProgress` with defensive defaults (line 334)
- Displays task counts from `useAgentState` hook derived state

### 10. Run and Validate Tests

Execute the new tests in Chromium specifically:

```bash
cd tests/e2e
npx playwright test test_late_joining_user.spec.ts --project=chromium
```

Verify both new tests pass:
- "should show in-progress tasks when tasks already in progress"
- "should show completed state when project already finished"

Check test report for any timing issues or flakiness, adjust timeouts if needed.

## Test Execution Flow

```mermaid
sequenceDiagram
    participant Test as E2E Test
    participant DB as Test Database
    participant API as Backend API
    participant UI as Dashboard UI
    participant State as AgentState Hook

    Test->>DB: Seed project with in-progress tasks
    Test->>UI: Navigate to /projects/3
    UI->>API: GET /projects/3/status
    API->>UI: Return project data
    UI->>API: GET /projects/3/tasks
    API->>UI: Return tasks (2 in_progress)
    UI->>API: GET /projects/3/agents
    API->>UI: Return agents (2 working)
    UI->>State: Initialize state from API data
    State->>UI: Provide activeTasks, activeAgents
    UI->>UI: Render dashboard with correct state
    Test->>UI: Assert in-progress count = 2
    Test->>UI: Assert agent status = working
    Test->>Test: ✅ Test passes
```

## Success Criteria

- Both new test cases pass consistently in Chromium browser
- Tests verify UI state matches API-fetched data without WebSocket dependency
- No flakiness or timing issues in test execution
- Test coverage includes both in-progress and completed project states
- Documentation clearly explains state reconciliation testing approach

---

Execution Information

Branch: main
Commit: e1539b8

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

⚠️ Possible Duplicate Issue

• [P0] Add State Reconciliation E2E Tests for All Features #227

🔗 Related PRs

#188 - feat(e2e): Add comprehensive error monitoring and strict verification [merged]
#191 - feat(e2e): Enable conditional tests and fix discovery startup [merged]
#221 - feat(api): add manual task generation endpoint [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[frankbria]

Related: New bug found that E2E tests should have caught

Issue #242 - Tasks not displayed and agents not assigned after approval

What happened:

• Task approval succeeded (Bug 016-2: Backend Automation - Planning Trigger + Task Approval #199 is FIXED ✅)
• Phase transitioned to Development
• But Tasks tab shows 0/0 when Progress shows 0/16 tasks
• Agents never got assigned

Why E2E didn't catch it:

The AI test agent appears to be treating "element not found" as a pass condition instead of a failure. When the tasks didn't render, the test likely said "no tasks to verify → pass" instead of "expected 16 tasks, found 0 → fail".

Suggested fixes for E2E tests:

1. Positive assertions required:

   // BAD: Passes if element missing
   if (tasksExist) { expect(taskCount).toBe(16) }
   
   // GOOD: Fails if element missing
   expect(tasksExist).toBe(true)
   expect(taskCount).toBe(16)
   

2. API response validation:

   // Assert endpoint exists and returns expected data
   expect(response.status).toBe(200)
   expect(response.data.tasks.length).toBeGreaterThan(0)
   

3. Explicit failure on missing elements:

   • If looking for N tasks, FAIL if N=0 (unless explicitly testing empty state)
   • If endpoint 404s or returns empty, FAIL (unless testing error handling)

The pattern to avoid:

"Endpoint doesn't exist → No data returned → Nothing on screen → Nothing to assert → ✅ Pass"

This masks real bugs like #242.