[V2] Add build validation gate to catch pyproject.toml/package.json configuration errors early

[frankbria]

Problem

When an agent creates project scaffolding (pyproject.toml, package.json), configuration errors aren't detected until later gates (pytest, npm-test) fail. By then:

1. The agent has written more code based on the broken foundation
2. The error message is misleading ("pytest failed" when really build failed)
3. The agent doesn't understand it's a config issue, not a code issue

Example from v2 CLI Testing

Agent created pyproject.toml with:

# Wrong TOML section name:
[tool.hatchling.build.targets.wheel]
packages = ["src/task_tracker"]

Should have been:

# Correct:
[tool.hatch.build.targets.wheel]
packages = ["src/task_tracker"]

This caused all subsequent tasks to fail with "hatchling.build.build_editable failed" but the agent couldn't diagnose the root cause.

Proposed Solutions

Option 1: Add a "build" gate

Add a new gate that explicitly validates the project can be built:

def _run_build(repo_path: Path, verbose: bool = False) -> GateCheck:
    """Verify project can be built/installed."""
    # For Python: uv sync --no-install-project or pip install -e . --dry-run
    # For Node: npm install --ignore-scripts

Run this gate immediately after pyproject.toml or package.json is created/modified.

Option 2: Better error parsing in existing gates

When uv run pytest fails due to build error, parse the output to detect:

• "Failed to build" → Report as BUILD_FAILED, not TEST_FAILED
• Provide actionable error message pointing to pyproject.toml

Option 3: Tech stack templates

When initializing with --detect or --tech-stack, provide the agent with proven pyproject.toml/package.json templates for that stack, reducing reliance on agent memory.

Acceptance Criteria

• Build configuration errors are caught before code-writing continues
• Error messages clearly indicate it's a configuration issue, not test failure
• Agent receives actionable guidance on how to fix the configuration

Related

• The orchestration escalation issue ([V2] Batch resume: Handle file conflicts from previous runs via orchestration escalation #297) would also help here - agent could escalate "build failed" to orchestrator which has more context about project setup.

Labels

enhancement, quality-gates, agent-system

[traycerai]

Plan

Observations

The codebase has a well-structured quality gates system with 6 gate types (TESTS, TYPE_CHECK, COVERAGE, CODE_REVIEW, LINTING, SKIP_DETECTION) defined in file:codeframe/core/models.py. Gates are executed in file:codeframe/lib/quality_gates.py via the run_all_gates() orchestrator, which is triggered by file:codeframe/agents/worker_agent.py when completing tasks. The system uses task classification (file:codeframe/lib/task_classifier.py) to determine which gates apply to each task category. Currently, build configuration errors (pyproject.toml/package.json) aren't detected until later gates fail, causing misleading error messages.

Approach

Implement Option 1 from the issue: Add a new BUILD gate that validates project configuration files can be built/installed before running other gates. This gate will run immediately after LINTING and TYPE_CHECK (fast gates) but before TESTS (slower gate). The BUILD gate will detect Python projects via pyproject.toml and Node.js projects via package.json, then run appropriate validation commands (uv sync --no-install-project for Python, npm install --dry-run for Node.js). This approach catches configuration errors early with clear, actionable error messages pointing to the config file issue.

Implementation Steps

1. Add BUILD Gate Type to Models

File: file:codeframe/core/models.py

Add BUILD = "build" to the QualityGateType enum after line 167, maintaining alphabetical order with other gate types.

2. Update Gate Rules Configuration

File: file:codeframe/lib/quality_gate_rules.py

Add QualityGateType.BUILD to the gate rules mapping:

• Add to TaskCategory.CODE_IMPLEMENTATION list (line 31-38)
• Add to TaskCategory.CONFIGURATION list (line 45-52) - critical for config tasks
• Add to TaskCategory.REFACTORING list (line 54-67)
• Add to TaskCategory.MIXED list (line 62-70)

Add skip reasons to _SKIP_REASONS dictionary for categories that don't need build validation:

• TaskCategory.DESIGN: "Design tasks do not produce buildable artifacts"
• TaskCategory.DOCUMENTATION: "Documentation tasks do not require build validation"
• TaskCategory.TESTING: "Testing tasks typically don't modify build configuration"

3. Implement BUILD Gate Method

File: file:codeframe/lib/quality_gates.py

Add async def run_build_gate(self, task: Task, task_category: Optional[str] = None) -> QualityGateResult after the run_linting_gate method (around line 510).

Implementation details:

• Detect project type using _task_has_python_files() and _task_has_javascript_files() helpers
• For Python projects: Call _run_python_build() helper method
• For Node.js projects: Call _run_node_build() helper method
• Parse output to detect configuration errors (invalid TOML syntax, missing dependencies, incorrect section names)
• Create QualityGateFailure with gate=QualityGateType.BUILD and clear error messages
• Include specific guidance: "Check pyproject.toml configuration" or "Check package.json configuration"
• Update database with gate status
• Create blocker if failed using _create_quality_blocker()

4. Add Build Command Helpers

File: file:codeframe/lib/quality_gates.py

Add helper methods in the "Helper Methods - Command Execution" section (after line 837):

_run_python_build(self) -> Dict[str, Any]:

• Check if pyproject.toml exists in self.project_root
• Run uv sync --no-install-project with 60-second timeout
• Fallback to pip install -e . --dry-run if uv not available
• Parse stderr/stdout for build errors (hatchling errors, TOML syntax errors, missing dependencies)
• Extract summary using _extract_build_error_summary() helper
• Return dict with returncode, output, summary, config_file keys

_run_node_build(self) -> Dict[str, Any]:

• Check if package.json exists in self.project_root
• Run npm install --dry-run --ignore-scripts with 60-second timeout
• Parse output for configuration errors (invalid JSON, missing dependencies, version conflicts)
• Extract summary using _extract_build_error_summary() helper
• Return dict with returncode, output, summary, config_file keys

5. Add Build Error Parser

File: file:codeframe/lib/quality_gates.py

Add _extract_build_error_summary(self, output: str, config_file: str) -> str in the "Output Parsers" section (after line 1099).

Parsing logic:

• Detect common patterns: "Failed to build", "Invalid TOML", "SyntaxError", "Cannot find module"
• Extract specific error location (line numbers, section names)
• For hatchling errors: Parse "tool.hatchling" vs "tool.hatch" confusion
• For npm errors: Parse dependency resolution failures
• Return concise summary: "Build failed: Invalid section [tool.hatchling.build] in pyproject.toml (should be [tool.hatch.build])"

6. Integrate BUILD Gate into Orchestrator

File: file:codeframe/lib/quality_gates.py

In run_all_gates() method (line 623), add BUILD gate execution after TYPE_CHECK gate (around line 732) and before SKIP_DETECTION gate:

# 2.5. Build gate (fast, validates configuration)
if QualityGateType.BUILD in applicable_gates:
    build_result = await self.run_build_gate(task, category.value)
    all_failures.extend(build_result.failures)
else:
    reason = rules.get_skip_reason(category, QualityGateType.BUILD)
    logger.info(f"Skipping build gate for task {task.id}: {reason}")
    skipped_gates.append(QualityGateType.BUILD.value)

Execution order rationale:

1. LINTING (fastest)
2. TYPE_CHECK (fast)
3. BUILD (fast, new) ← catches config errors early
4. SKIP_DETECTION (fast)
5. TESTS (slower)
6. COVERAGE (slower)
7. CODE_REVIEW (slowest)

7. Update Quality Gate Tool

File: file:codeframe/lib/quality_gate_tool.py

Add "build" to VALID_CHECKS list (line 34).

Add mapping in CHECK_NAME_TO_GATE dictionary (line 37-43):

"build": QualityGateType.BUILD,

Add "build" case to _run_specific_gates() method's gate_method dictionary (line 232-238).

8. Update Configuration

File: file:codeframe/config/quality_gates_config.py

Update docstring (line 11) to include build in the list of valid gate names for QUALITY_GATES_CUSTOM_RULES.

9. Add Tests for BUILD Gate

File: file:tests/lib/test_quality_gates.py

Add test class TestBuildGate with test methods:

• test_build_gate_passes_valid_python_project() - Valid pyproject.toml passes
• test_build_gate_fails_invalid_python_config() - Invalid TOML syntax fails
• test_build_gate_fails_wrong_hatchling_section() - Detects tool.hatchling vs tool.hatch error
• test_build_gate_passes_valid_node_project() - Valid package.json passes
• test_build_gate_fails_invalid_node_config() - Invalid JSON syntax fails
• test_build_gate_skipped_for_design_tasks() - BUILD gate skipped for DESIGN category
• test_build_gate_runs_for_configuration_tasks() - BUILD gate runs for CONFIGURATION category

File: file:tests/lib/test_quality_gate_rules.py

Add test methods to verify BUILD gate applicability:

• test_configuration_gets_build_gate() - Verify BUILD in CONFIGURATION gates
• test_design_does_not_get_build_gate() - Verify BUILD not in DESIGN gates

10. Update CLI and API

File: file:codeframe/cli/quality_gates_commands.py

No changes needed - CLI already supports running specific gates via --gate flag, which will automatically include the new BUILD gate.

File: file:codeframe/ui/routers/quality_gates.py

No changes needed - API already supports gate filtering via gate_types parameter, which will automatically include the new BUILD gate.

11. Documentation Updates

File: file:codeframe/lib/quality_gates.py

Update module docstring (lines 1-33) to include BUILD gate in the architecture description and execution order.

Update run_all_gates() docstring (lines 624-664) to include BUILD gate in the task classification matrix and execution order list.

Sequence Diagram

sequenceDiagram
    participant Agent as Worker Agent
    participant QG as Quality Gates
    participant Build as BUILD Gate
    participant FS as File System
    participant UV as uv/npm

    Agent->>QG: run_all_gates(task)
    QG->>QG: classify_task()
    QG->>QG: get_applicable_gates()
    
    Note over QG: Fast gates first
    QG->>QG: run_linting_gate()
    QG->>QG: run_type_check_gate()
    
    Note over QG,Build: NEW: Build validation
    QG->>Build: run_build_gate()
    Build->>FS: Check pyproject.toml exists?
    FS-->>Build: Yes
    Build->>UV: uv sync --no-install-project
    UV-->>Build: Error: Invalid section [tool.hatchling.build]
    Build->>Build: _extract_build_error_summary()
    Build-->>QG: QualityGateFailure(BUILD, "Invalid config...")
    
    QG->>QG: Create blocker with clear error
    QG-->>Agent: QualityGateResult(failed, failures=[...])
    
    Note over Agent: Agent receives actionable error:<br/>"Build failed: Invalid section in pyproject.toml"

Loading

Error Message Examples

Before (misleading):

❌ TESTS gate failed: pytest execution failed
   hatchling.build.build_editable failed

After (clear and actionable):

❌ BUILD gate failed: Configuration error in pyproject.toml
   Invalid section [tool.hatchling.build.targets.wheel]
   Should be: [tool.hatch.build.targets.wheel]
   
   Fix the configuration file and try again.

Benefits

1. Early Detection: Configuration errors caught before code execution
2. Clear Errors: Specific guidance pointing to config file issues
3. Fast Feedback: Build validation is fast (< 5 seconds)
4. Prevents Cascading Failures: Stops agent from writing more code on broken foundation
5. Task-Aware: Only runs for relevant task categories (CODE_IMPLEMENTATION, CONFIGURATION, REFACTORING, MIXED)
6. Backward Compatible: Existing gates continue to work unchanged

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has a well-structured quality gates system with 6 gate types (TESTS, TYPE_CHECK, COVERAGE, CODE_REVIEW, LINTING, SKIP_DETECTION) defined in `file:codeframe/core/models.py`. Gates are executed in `file:codeframe/lib/quality_gates.py` via the `run_all_gates()` orchestrator, which is triggered by `file:codeframe/agents/worker_agent.py` when completing tasks. The system uses task classification (`file:codeframe/lib/task_classifier.py`) to determine which gates apply to each task category. Currently, build configuration errors (pyproject.toml/package.json) aren't detected until later gates fail, causing misleading error messages.

## Approach

Implement Option 1 from the issue: Add a new BUILD gate that validates project configuration files can be built/installed before running other gates. This gate will run immediately after LINTING and TYPE_CHECK (fast gates) but before TESTS (slower gate). The BUILD gate will detect Python projects via pyproject.toml and Node.js projects via package.json, then run appropriate validation commands (`uv sync --no-install-project` for Python, `npm install --dry-run` for Node.js). This approach catches configuration errors early with clear, actionable error messages pointing to the config file issue.

## Implementation Steps

### 1. Add BUILD Gate Type to Models

**File:** `file:codeframe/core/models.py`

Add `BUILD = "build"` to the `QualityGateType` enum after line 167, maintaining alphabetical order with other gate types.

### 2. Update Gate Rules Configuration

**File:** `file:codeframe/lib/quality_gate_rules.py`

Add `QualityGateType.BUILD` to the gate rules mapping:
- Add to `TaskCategory.CODE_IMPLEMENTATION` list (line 31-38)
- Add to `TaskCategory.CONFIGURATION` list (line 45-52) - critical for config tasks
- Add to `TaskCategory.REFACTORING` list (line 54-67)
- Add to `TaskCategory.MIXED` list (line 62-70)

Add skip reasons to `_SKIP_REASONS` dictionary for categories that don't need build validation:
- `TaskCategory.DESIGN`: "Design tasks do not produce buildable artifacts"
- `TaskCategory.DOCUMENTATION`: "Documentation tasks do not require build validation"
- `TaskCategory.TESTING`: "Testing tasks typically don't modify build configuration"

### 3. Implement BUILD Gate Method

**File:** `file:codeframe/lib/quality_gates.py`

Add `async def run_build_gate(self, task: Task, task_category: Optional[str] = None) -> QualityGateResult` after the `run_linting_gate` method (around line 510).

**Implementation details:**
- Detect project type using `_task_has_python_files()` and `_task_has_javascript_files()` helpers
- For Python projects: Call `_run_python_build()` helper method
- For Node.js projects: Call `_run_node_build()` helper method
- Parse output to detect configuration errors (invalid TOML syntax, missing dependencies, incorrect section names)
- Create `QualityGateFailure` with `gate=QualityGateType.BUILD` and clear error messages
- Include specific guidance: "Check pyproject.toml configuration" or "Check package.json configuration"
- Update database with gate status
- Create blocker if failed using `_create_quality_blocker()`

### 4. Add Build Command Helpers

**File:** `file:codeframe/lib/quality_gates.py`

Add helper methods in the "Helper Methods - Command Execution" section (after line 837):

**`_run_python_build(self) -> Dict[str, Any]`:**
- Check if `pyproject.toml` exists in `self.project_root`
- Run `uv sync --no-install-project` with 60-second timeout
- Fallback to `pip install -e . --dry-run` if uv not available
- Parse stderr/stdout for build errors (hatchling errors, TOML syntax errors, missing dependencies)
- Extract summary using `_extract_build_error_summary()` helper
- Return dict with `returncode`, `output`, `summary`, `config_file` keys

**`_run_node_build(self) -> Dict[str, Any]`:**
- Check if `package.json` exists in `self.project_root`
- Run `npm install --dry-run --ignore-scripts` with 60-second timeout
- Parse output for configuration errors (invalid JSON, missing dependencies, version conflicts)
- Extract summary using `_extract_build_error_summary()` helper
- Return dict with `returncode`, `output`, `summary`, `config_file` keys

### 5. Add Build Error Parser

**File:** `file:codeframe/lib/quality_gates.py`

Add `_extract_build_error_summary(self, output: str, config_file: str) -> str` in the "Output Parsers" section (after line 1099).

**Parsing logic:**
- Detect common patterns: "Failed to build", "Invalid TOML", "SyntaxError", "Cannot find module"
- Extract specific error location (line numbers, section names)
- For hatchling errors: Parse "tool.hatchling" vs "tool.hatch" confusion
- For npm errors: Parse dependency resolution failures
- Return concise summary: "Build failed: Invalid section [tool.hatchling.build] in pyproject.toml (should be [tool.hatch.build])"

### 6. Integrate BUILD Gate into Orchestrator

**File:** `file:codeframe/lib/quality_gates.py`

In `run_all_gates()` method (line 623), add BUILD gate execution after TYPE_CHECK gate (around line 732) and before SKIP_DETECTION gate:

```
# 2.5. Build gate (fast, validates configuration)
if QualityGateType.BUILD in applicable_gates:
    build_result = await self.run_build_gate(task, category.value)
    all_failures.extend(build_result.failures)
else:
    reason = rules.get_skip_reason(category, QualityGateType.BUILD)
    logger.info(f"Skipping build gate for task {task.id}: {reason}")
    skipped_gates.append(QualityGateType.BUILD.value)
```

**Execution order rationale:**
1. LINTING (fastest)
2. TYPE_CHECK (fast)
3. **BUILD (fast, new)** ← catches config errors early
4. SKIP_DETECTION (fast)
5. TESTS (slower)
6. COVERAGE (slower)
7. CODE_REVIEW (slowest)

### 7. Update Quality Gate Tool

**File:** `file:codeframe/lib/quality_gate_tool.py`

Add `"build"` to `VALID_CHECKS` list (line 34).

Add mapping in `CHECK_NAME_TO_GATE` dictionary (line 37-43):
```python
"build": QualityGateType.BUILD,
```

Add `"build"` case to `_run_specific_gates()` method's gate_method dictionary (line 232-238).

### 8. Update Configuration

**File:** `file:codeframe/config/quality_gates_config.py`

Update docstring (line 11) to include `build` in the list of valid gate names for `QUALITY_GATES_CUSTOM_RULES`.

### 9. Add Tests for BUILD Gate

**File:** `file:tests/lib/test_quality_gates.py`

Add test class `TestBuildGate` with test methods:
- `test_build_gate_passes_valid_python_project()` - Valid pyproject.toml passes
- `test_build_gate_fails_invalid_python_config()` - Invalid TOML syntax fails
- `test_build_gate_fails_wrong_hatchling_section()` - Detects tool.hatchling vs tool.hatch error
- `test_build_gate_passes_valid_node_project()` - Valid package.json passes
- `test_build_gate_fails_invalid_node_config()` - Invalid JSON syntax fails
- `test_build_gate_skipped_for_design_tasks()` - BUILD gate skipped for DESIGN category
- `test_build_gate_runs_for_configuration_tasks()` - BUILD gate runs for CONFIGURATION category

**File:** `file:tests/lib/test_quality_gate_rules.py`

Add test methods to verify BUILD gate applicability:
- `test_configuration_gets_build_gate()` - Verify BUILD in CONFIGURATION gates
- `test_design_does_not_get_build_gate()` - Verify BUILD not in DESIGN gates

### 10. Update CLI and API

**File:** `file:codeframe/cli/quality_gates_commands.py`

No changes needed - CLI already supports running specific gates via `--gate` flag, which will automatically include the new BUILD gate.

**File:** `file:codeframe/ui/routers/quality_gates.py`

No changes needed - API already supports gate filtering via `gate_types` parameter, which will automatically include the new BUILD gate.

### 11. Documentation Updates

**File:** `file:codeframe/lib/quality_gates.py`

Update module docstring (lines 1-33) to include BUILD gate in the architecture description and execution order.

Update `run_all_gates()` docstring (lines 624-664) to include BUILD gate in the task classification matrix and execution order list.

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Agent as Worker Agent
    participant QG as Quality Gates
    participant Build as BUILD Gate
    participant FS as File System
    participant UV as uv/npm

    Agent->>QG: run_all_gates(task)
    QG->>QG: classify_task()
    QG->>QG: get_applicable_gates()
    
    Note over QG: Fast gates first
    QG->>QG: run_linting_gate()
    QG->>QG: run_type_check_gate()
    
    Note over QG,Build: NEW: Build validation
    QG->>Build: run_build_gate()
    Build->>FS: Check pyproject.toml exists?
    FS-->>Build: Yes
    Build->>UV: uv sync --no-install-project
    UV-->>Build: Error: Invalid section [tool.hatchling.build]
    Build->>Build: _extract_build_error_summary()
    Build-->>QG: QualityGateFailure(BUILD, "Invalid config...")
    
    QG->>QG: Create blocker with clear error
    QG-->>Agent: QualityGateResult(failed, failures=[...])
    
    Note over Agent: Agent receives actionable error:<br/>"Build failed: Invalid section in pyproject.toml"
```

## Error Message Examples

**Before (misleading):**
```
❌ TESTS gate failed: pytest execution failed
   hatchling.build.build_editable failed
```

**After (clear and actionable):**
```
❌ BUILD gate failed: Configuration error in pyproject.toml
   Invalid section [tool.hatchling.build.targets.wheel]
   Should be: [tool.hatch.build.targets.wheel]
   
   Fix the configuration file and try again.
```

## Benefits

1. **Early Detection**: Configuration errors caught before code execution
2. **Clear Errors**: Specific guidance pointing to config file issues
3. **Fast Feedback**: Build validation is fast (< 5 seconds)
4. **Prevents Cascading Failures**: Stops agent from writing more code on broken foundation
5. **Task-Aware**: Only runs for relevant task categories (CODE_IMPLEMENTATION, CONFIGURATION, REFACTORING, MIXED)
6. **Backward Compatible**: Existing gates continue to work unchanged

---

Execution Information

Branch: main
Commit: 589ae17

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.