Enhanced Verification and Reporting

[frankbria]

Summary

Create comprehensive verification scripts that validate AI claims with detailed reporting.

Tasks

1. Enhanced Verification Script

• Expand tools/verify-ai-claims.sh
• Add multi-step verification process
• Generate detailed reports
• Save artifacts for review

2. Verification Steps

• Run full test suite with verbose output
• Check coverage against threshold
• Detect skip decorator abuse
• Run code quality checks (black, mypy, isort)
• Verify no test modifications without approval

3. Reporting

• Create verification summary
• Save test output to file
• Generate coverage HTML report
• List any quality issues found
• Provide clear pass/fail status

4. Git Integration

• Create .gitmessage template
• Require test output in commits
• Require coverage report in commits
• Add checklist for AI commits

5. Documentation

• Add verification workflow to README
• Document what each check does
• Explain how to interpret results
• Add troubleshooting guide

Verification Flow

./tools/verify-ai-claims.sh
  → Run tests
  → Check coverage
  → Detect skip abuse
  → Check code quality
  → Generate report
  → Return exit code

Report Format

🔍 Comprehensive AI Verification
=================================

📋 Step 1: Running test suite...
✅ All tests passed (23 passed, 0 failed)

📊 Step 2: Checking coverage...
✅ Coverage: 87% (target: 80%)

🔍 Step 3: Checking for @skip abuse...
✅ No skip decorators found

🎨 Step 4: Code quality checks...
✅ Formatting: OK
✅ Type checking: OK

=================================
✅ ALL VERIFICATIONS PASSED
=================================

Success Criteria

• Single script validates all requirements
• Clear, actionable error messages
• Detailed reports saved for review
• Integrates with git workflow
• Fast enough for development iteration (<30s)

Performance Considerations

• Cache test results when possible
• Run checks in parallel where safe
• Fail fast on first critical error
• Provide progress indicators for slow steps

Dependencies

• Issue Add Claude Code GitHub Workflow #1 (enforcement foundation)
• Issue Brainstorming: Integrate remaining general concepts into specification #2 (skip detection)
• Optional: black, mypy, isort for quality checks

Estimated Effort: 3-4 hours

[traycerai]

Plan

Observations

Current State:

• No tools/ directory exists; needs creation
• Existing verify.sh is deployment-specific (cf-46), not for AI claims verification
• Test infrastructure: pytest with JSON report plugin, coverage via pytest-cov (85% threshold)
• Code quality tools configured: black (line-length=100), ruff (line-length=100), mypy (disort_untyped_defs=true)
• No isort configuration found
• No git commit templates or pre-commit hooks
• TestRunner class exists and parses pytest JSON output into TestResult model
• No skip decorators currently in codebase (clean baseline)
• Coverage outputs: terminal, HTML (htmlcov/), XML (coverage.xml)
• No CI workflows for running tests/linters (only Claude Code review workflows)

Approach

Create a comprehensive verification system with:

1. New tools/verify-ai-claims.sh script orchestrating multi-step verification
2. Git commit template (.gitmessage) with AI verification checklist
3. Documentation updates (README.md, TESTING.md) explaining workflow
4. Artifact storage structure for reports and test outputs

The script will run tests, check coverage, detect skip abuse, run code quality checks, generate detailed reports, and provide clear pass/fail status with <30s performance target.

Reasoning

Listed repository structure, read configuration files (pytest.ini, .coveragerc, pyproject.toml), analyzed test infrastructure (conftest.py, test_runner.py), reviewed documentation (README.md, TESTING.md, CONTRIBUTING.md), searched for existing verification scripts and git workflows, and deployed agents to research enforcement mechanisms, test infrastructure, code quality tools, and git integration patterns.

Proposed File Changes

📁 tools ^(NEW) 🔗

Create new directory to house verification and tooling scripts.

📄 tools/verify-ai-claims.sh ^(NEW) 🔗

References

• 📄 pytest.ini
• 📄 .coveragerc
• 📄 pyproject.toml ^(MODIFY)
• 📄 codeframe/testing/test_runner.py

Create comprehensive verification script that orchestrates all verification steps:

Header Section:

• Shebang and script description
• Color codes for output formatting (green ✅, red ❌, yellow ⚠️)
• Exit code constants (0=success, 1=failure)

Configuration Variables:

• COVERAGE_THRESHOLD=85 (from .coveragerc)
• ARTIFACTS_DIR="artifacts/verify/$(date +%Y%m%d_%H%M%S)"
• TEST_OUTPUT_FILE, COVERAGE_REPORT_FILE, QUALITY_REPORT_FILE paths

Step 1: Run Test Suite

• Execute pytest with verbose output and JSON report: pytest -v --json-report --json-report-file="$ARTIFACTS_DIR/test-report.json" --tb=short
• Capture exit code and output to $TEST_OUTPUT_FILE
• Parse JSON report to extract passed/failed/errors/skipped counts
• Display summary with emoji indicators
• Fail fast if tests fail (unless --no-fail-fast flag provided)

Step 2: Check Coverage

• Run pytest with coverage: pytest --cov=codeframe --cov-report=term --cov-report=html --cov-report=json --cov-config=.coveragerc
• Parse coverage.json to extract total coverage percentage
• Compare against COVERAGE_THRESHOLD (85%)
• Save HTML report to $ARTIFACTS_DIR/htmlcov/
• Display coverage percentage with pass/fail indicator

Step 3: Detect Skip Decorator Abuse

• Use grep to search for skip decorators: grep -r "@pytest.mark.skip\|@skip\|skipif" tests/ --include="*.py"
• Count occurrences and list files
• Warn if any skips found (not a hard failure, but flagged for review)
• Save results to $ARTIFACTS_DIR/skip-detection.txt

Step 4: Code Quality Checks

• Run black in check mode: black --check codeframe tests
• Run ruff: ruff check codeframe tests
• Run mypy: mypy codeframe
• Optionally run isort if configured: isort --check-only codeframe tests
• Capture all outputs to $QUALITY_REPORT_FILE
• Display summary of each tool's results

Step 5: Generate Verification Report

• Create comprehensive summary report at $ARTIFACTS_DIR/verification-report.md
• Include: timestamp, all step results, pass/fail status, artifact locations
• Format using the report template from the issue (with emoji indicators)
• Display report to console

Step 6: Exit with Status

• Return 0 if all critical checks passed
• Return 1 if any critical check failed
• Print final status banner

Command-line Options:

• --no-fail-fast: Continue all checks even if one fails
• --skip-tests: Skip test execution (for quick quality-only checks)
• --skip-coverage: Skip coverage check
• --skip-quality: Skip code quality checks
• --help: Display usage information

Performance Optimizations:

• Run tests and coverage in single pytest invocation when possible
• Use parallel pytest execution if available (-n auto with pytest-xdist)
• Cache test results between runs (pytest cache)
• Provide progress indicators for long-running steps

Refer to pytest.ini, .coveragerc, and pyproject.toml for configuration values. Use the TestRunner pattern from codeframe/testing/test_runner.py as reference for JSON parsing logic.

📄 .gitmessage ^(NEW) 🔗

Create git commit message template for AI-generated commits:

Template Structure:

# <type>(<scope>): <subject>
#
# <body>
#
# AI Verification Checklist:
# [ ] Ran ./tools/verify-ai-claims.sh
# [ ] All tests passing (X/Y passed)
# [ ] Coverage >= 85% (actual: X%)
# [ ] No skip decorator abuse
# [ ] Code quality checks passed (black, ruff, mypy)
# [ ] Verification report: artifacts/verify/YYYYMMDD_HHMMSS/
#
# Test Output Summary:
# <paste key test results here>
#
# Coverage Summary:
# <paste coverage percentage here>
#
# Quality Issues (if any):
# <paste any warnings or errors here>
#
# Related Issue: #<issue-number>

Type Options:

• feat: New feature
• fix: Bug fix
• test: Test additions/modifications
• refactor: Code refactoring
• docs: Documentation changes
• chore: Maintenance tasks

Instructions Section:
Include commented instructions explaining:

• How to use the template
• Requirement to run verification script before committing
• Where to find verification artifacts
• How to interpret verification results
• Link to documentation for troubleshooting

Template follows Conventional Commits format with AI-specific verification requirements.

📄 README.md ^(MODIFY) 🔗

References

• 📄 TESTING.md ^(MODIFY)

Add new section "AI Verification Workflow" after the "Testing" section (around line 596):

Content to Add:

AI Verification Workflow

Before committing AI-generated code, run the comprehensive verification script:

# Run full verification
./tools/verify-ai-claims.sh

# Quick quality check (skip tests)
./tools/verify-ai-claims.sh --skip-tests

# Continue on failures (see all issues)
./tools/verify-ai-claims.sh --no-fail-fast

What It Checks:

• ✅ Full test suite execution with verbose output
• ✅ Code coverage against 85% threshold
• ✅ Skip decorator abuse detection
• ✅ Code formatting (black)
• ✅ Linting (ruff)
• ✅ Type checking (mypy)

Verification Report:

The script generates detailed reports in artifacts/verify/YYYYMMDD_HHMMSS/:

• test-report.json - Full pytest JSON output
• test-output.txt - Human-readable test results
• coverage-report.json - Coverage data
• htmlcov/ - HTML coverage report
• quality-report.txt - Code quality check results
• verification-report.md - Comprehensive summary

Git Integration:

Configure git to use the AI commit template:

git config commit.template .gitmessage

The template includes a checklist for AI verification requirements.

Performance:

Typical execution time: 15-25 seconds for full verification.

See TESTING.md for detailed verification workflow documentation.

Update the table of contents if present to include the new section. Reference the verification script location and the .gitmessage template.

📄 TESTING.md ^(MODIFY) 🔗

References

• 📄 tools/verify-ai-claims.sh ^(NEW)
• 📄 .gitmessage ^(NEW)
• 📄 README.md ^(MODIFY)
• 📄 .coveragerc
• 📄 pytest.ini

Add new section "AI Verification Workflow" after the existing test sections (around line 408, before "Next Steps"):

Content to Add:

AI Verification Workflow

Overview

The AI verification system ensures all AI-generated code meets quality standards before commits. The tools/verify-ai-claims.sh script provides comprehensive validation.

Running Verification

# Full verification (recommended before commits)
./tools/verify-ai-claims.sh

# With options
./tools/verify-ai-claims.sh --no-fail-fast  # See all issues
./tools/verify-ai-claims.sh --skip-tests    # Quick quality check
./tools/verify-ai-claims.sh --help          # Show all options

Verification Steps

Step 1: Test Suite Execution

• Runs pytest with verbose output and JSON reporting
• Captures passed/failed/errors/skipped counts
• Saves output to artifacts/verify/TIMESTAMP/test-output.txt
• Pass Criteria: All tests must pass (0 failures, 0 errors)

Step 2: Coverage Check

• Measures code coverage using pytest-cov
• Generates terminal, HTML, and JSON reports
• Pass Criteria: Coverage >= 85% (configured in .coveragerc)
• HTML report available at artifacts/verify/TIMESTAMP/htmlcov/index.html

Step 3: Skip Decorator Detection

• Searches for @pytest.mark.skip, @skip, skipif in test files
• Flags any usage for manual review
• Pass Criteria: Warning only (not a hard failure)
• Prevents test suite degradation through excessive skipping

Step 4: Code Quality Checks

• Black: Verifies code formatting (line-length=100)
• Ruff: Runs linting checks (target=py311)
• Mypy: Type checking with strict settings
• Pass Criteria: All tools must report no errors

Step 5: Report Generation

• Creates comprehensive verification report
• Includes all step results and artifact locations
• Saved as artifacts/verify/TIMESTAMP/verification-report.md

Interpreting Results

Success Output:

🔍 Comprehensive AI Verification
=================================

📋 Step 1: Running test suite...
✅ All tests passed (93 passed, 0 failed)

📊 Step 2: Checking coverage...
✅ Coverage: 87% (target: 85%)

🔍 Step 3: Checking for @skip abuse...
✅ No skip decorators found

🎨 Step 4: Code quality checks...
✅ Formatting: OK
✅ Linting: OK  
✅ Type checking: OK

=================================
✅ ALL VERIFICATIONS PASSED
=================================

Failure Output:

📋 Step 1: Running test suite...
❌ Tests failed (89 passed, 4 failed)

Failed tests:
- tests/test_example.py::test_feature_x
- tests/test_example.py::test_feature_y

See: artifacts/verify/20251110_143022/test-output.txt

Troubleshooting

Issue: Tests Failing

• Review test output in artifacts/verify/TIMESTAMP/test-output.txt
• Run specific test: pytest tests/test_file.py::test_name -v
• Check test logs for error details

Issue: Coverage Below Threshold

• Open HTML report: artifacts/verify/TIMESTAMP/htmlcov/index.html
• Identify uncovered lines (highlighted in red)
• Add tests for uncovered code paths
• Review .coveragerc exclude patterns if needed

Issue: Skip Decorators Found

• Review artifacts/verify/TIMESTAMP/skip-detection.txt
• Evaluate if skips are justified (platform-specific, requires external service)
• Remove unnecessary skips or document reasoning
• Consider using pytest.mark.skipif with clear conditions

Issue: Code Quality Failures

• Black: Run black codeframe tests to auto-format
• Ruff: Review errors in quality report, fix manually or run ruff check --fix
• Mypy: Add type hints or adjust mypy configuration in pyproject.toml

Issue: Verification Too Slow

• Use --skip-coverage for quick checks during development
• Install pytest-xdist for parallel test execution: pip install pytest-xdist
• Script will automatically use -n auto if available

Git Integration

Setup Commit Template:

git config commit.template .gitmessage

Commit Workflow:

1. Make code changes
2. Run ./tools/verify-ai-claims.sh
3. Review verification report
4. Fix any issues
5. Commit with template (includes verification checklist)
6. Paste verification results into commit message

Template Checklist:

• Ran verification script
• All tests passing
• Coverage >= 85%
• No skip abuse
• Quality checks passed
• Verification report path included

Artifact Management

Location: artifacts/verify/YYYYMMDD_HHMMSS/

Files Generated:

• test-report.json - Pytest JSON output (machine-readable)
• test-output.txt - Human-readable test results
• coverage-report.json - Coverage data (JSON)
• htmlcov/ - Interactive HTML coverage report
• skip-detection.txt - List of skip decorators found
• quality-report.txt - Black/ruff/mypy output
• verification-report.md - Comprehensive summary

Cleanup:
Artifacts are timestamped and accumulate over time. Periodically clean old artifacts:

# Keep last 10 verification runs
ls -t artifacts/verify/ | tail -n +11 | xargs -I {} rm -rf artifacts/verify/{}

Performance Targets

• Full verification: < 30 seconds
• Test execution: < 20 seconds
• Coverage analysis: < 5 seconds
• Quality checks: < 5 seconds

Optimization Tips:

• Tests use pytest cache (.pytest_cache/)
• Coverage data cached between runs
• Parallel test execution with pytest-xdist
• Skip slow tests during development: pytest -m "not slow"

CI Integration (Future)

The verification script is designed for CI integration:

• Predictable exit codes (0=pass, 1=fail)
• Machine-readable JSON outputs
• Artifact paths for CI artifact storage
• Environment variable support for configuration

Example GitHub Actions usage:

- name: Run AI Verification
  run: ./tools/verify-ai-claims.sh
- name: Upload Artifacts
  uses: actions/upload-artifact@v3
  with:
    name: verification-report
    path: artifacts/verify/

Reference the verification script at tools/verify-ai-claims.sh and the git template at .gitmessage. Link to README.md AI Verification Workflow section for quick reference.

📄 CONTRIBUTING.md ^(MODIFY) 🔗

References

• 📄 tools/verify-ai-claims.sh ^(NEW)
• 📄 TESTING.md ^(MODIFY)

Update the "Testing" section (around line 34) to include verification workflow:

Replace existing Testing section with:

Testing

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=codeframe --cov-report=html

# Run specific test file
pytest tests/test_example.py -v

AI Verification

Before submitting PRs with AI-generated code, run the verification script:

./tools/verify-ai-claims.sh

This ensures:

• All tests pass
• Coverage >= 85%
• No skip decorator abuse
• Code quality standards met (black, ruff, mypy)

See TESTING.md for detailed verification workflow documentation.

Test Requirements

• Write unit tests for new features
• Maintain >85% code coverage (enforced by verification script)
• Follow existing test patterns in tests/
• Use fixtures from tests/conftest.py
• Mark slow tests with @pytest.mark.slow
• Mark integration tests with @pytest.mark.integration

Update the "Pull Request Process" section (around line 40):

Add step between "Add tests for new functionality" and "Update documentation":

4. Run verification script: `./tools/verify-ai-claims.sh`
5. Include verification report path in commit message

Reference the verification script at tools/verify-ai-claims.sh and link to TESTING.md for comprehensive documentation.

📄 pyproject.toml ^(MODIFY) 🔗

Add isort to dev dependencies and configure it:

In the [project.optional-dependencies] section (around line 46-55), add:

dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.23.0",
    "pytest-cov>=4.1.0",
    "pytest-json-report>=1.5.0",
    "black>=24.1.0",
    "ruff>=0.2.0",
    "mypy>=1.8.0",
    "isort>=5.12.0",  # ADD THIS LINE
]

Add new [tool.isort] section after [tool.mypy] (around line 81):

[tool.isort]
profile = "black"
line_length = 100
known_first_party = ["codeframe"]
sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]

This configures isort to:

• Be compatible with black formatting
• Use 100 character line length (matching black and ruff)
• Recognize codeframe as first-party package
• Follow standard import ordering (stdlib → third-party → local)

The verification script will use isort to check import ordering consistency.

📄 .gitignore ^(MODIFY) 🔗

Add verification artifacts directory to gitignore:

Add to the file (if .gitignore exists, otherwise create it):

# Verification artifacts
artifacts/

# Coverage reports (if not already present)
htmlcov/
.coverage
coverage.xml
coverage.json

# Pytest cache (if not already present)
.pytest_cache/

This prevents verification artifacts from being committed to the repository while keeping them available locally for review.

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

**Current State:**
- No `tools/` directory exists; needs creation
- Existing `verify.sh` is deployment-specific (cf-46), not for AI claims verification
- Test infrastructure: pytest with JSON report plugin, coverage via pytest-cov (85% threshold)
- Code quality tools configured: black (line-length=100), ruff (line-length=100), mypy (disort_untyped_defs=true)
- No isort configuration found
- No git commit templates or pre-commit hooks
- TestRunner class exists and parses pytest JSON output into TestResult model
- No skip decorators currently in codebase (clean baseline)
- Coverage outputs: terminal, HTML (htmlcov/), XML (coverage.xml)
- No CI workflows for running tests/linters (only Claude Code review workflows)


### Approach

Create a comprehensive verification system with:
1. New `tools/verify-ai-claims.sh` script orchestrating multi-step verification
2. Git commit template (`.gitmessage`) with AI verification checklist
3. Documentation updates (README.md, TESTING.md) explaining workflow
4. Artifact storage structure for reports and test outputs

The script will run tests, check coverage, detect skip abuse, run code quality checks, generate detailed reports, and provide clear pass/fail status with <30s performance target.

### Reasoning

Listed repository structure, read configuration files (pytest.ini, .coveragerc, pyproject.toml), analyzed test infrastructure (conftest.py, test_runner.py), reviewed documentation (README.md, TESTING.md, CONTRIBUTING.md), searched for existing verification scripts and git workflows, and deployed agents to research enforcement mechanisms, test infrastructure, code quality tools, and git integration patterns.

## Proposed File Changes

### tools(NEW)

Create new directory to house verification and tooling scripts.

### tools/verify-ai-claims.sh(NEW)

References: 

- pytest.ini
- .coveragerc
- pyproject.toml(MODIFY)
- codeframe/testing/test_runner.py

Create comprehensive verification script that orchestrates all verification steps:

**Header Section:**
- Shebang and script description
- Color codes for output formatting (green ✅, red ❌, yellow ⚠️)
- Exit code constants (0=success, 1=failure)

**Configuration Variables:**
- COVERAGE_THRESHOLD=85 (from `.coveragerc`)
- ARTIFACTS_DIR="artifacts/verify/$(date +%Y%m%d_%H%M%S)"
- TEST_OUTPUT_FILE, COVERAGE_REPORT_FILE, QUALITY_REPORT_FILE paths

**Step 1: Run Test Suite**
- Execute pytest with verbose output and JSON report: `pytest -v --json-report --json-report-file="$ARTIFACTS_DIR/test-report.json" --tb=short`
- Capture exit code and output to `$TEST_OUTPUT_FILE`
- Parse JSON report to extract passed/failed/errors/skipped counts
- Display summary with emoji indicators
- Fail fast if tests fail (unless --no-fail-fast flag provided)

**Step 2: Check Coverage**
- Run pytest with coverage: `pytest --cov=codeframe --cov-report=term --cov-report=html --cov-report=json --cov-config=.coveragerc`
- Parse coverage.json to extract total coverage percentage
- Compare against COVERAGE_THRESHOLD (85%)
- Save HTML report to `$ARTIFACTS_DIR/htmlcov/`
- Display coverage percentage with pass/fail indicator

**Step 3: Detect Skip Decorator Abuse**
- Use grep to search for skip decorators: `grep -r "@pytest.mark.skip\|@skip\|skipif" tests/ --include="*.py"`
- Count occurrences and list files
- Warn if any skips found (not a hard failure, but flagged for review)
- Save results to `$ARTIFACTS_DIR/skip-detection.txt`

**Step 4: Code Quality Checks**
- Run black in check mode: `black --check codeframe tests`
- Run ruff: `ruff check codeframe tests`
- Run mypy: `mypy codeframe`
- Optionally run isort if configured: `isort --check-only codeframe tests`
- Capture all outputs to `$QUALITY_REPORT_FILE`
- Display summary of each tool's results

**Step 5: Generate Verification Report**
- Create comprehensive summary report at `$ARTIFACTS_DIR/verification-report.md`
- Include: timestamp, all step results, pass/fail status, artifact locations
- Format using the report template from the issue (with emoji indicators)
- Display report to console

**Step 6: Exit with Status**
- Return 0 if all critical checks passed
- Return 1 if any critical check failed
- Print final status banner

**Command-line Options:**
- `--no-fail-fast`: Continue all checks even if one fails
- `--skip-tests`: Skip test execution (for quick quality-only checks)
- `--skip-coverage`: Skip coverage check
- `--skip-quality`: Skip code quality checks
- `--help`: Display usage information

**Performance Optimizations:**
- Run tests and coverage in single pytest invocation when possible
- Use parallel pytest execution if available (`-n auto` with pytest-xdist)
- Cache test results between runs (pytest cache)
- Provide progress indicators for long-running steps

Refer to `pytest.ini`, `.coveragerc`, and `pyproject.toml` for configuration values. Use the TestRunner pattern from `codeframe/testing/test_runner.py` as reference for JSON parsing logic.

### .gitmessage(NEW)

Create git commit message template for AI-generated commits:

**Template Structure:**
```
# <type>(<scope>): <subject>
#
# <body>
#
# AI Verification Checklist:
# [ ] Ran ./tools/verify-ai-claims.sh
# [ ] All tests passing (X/Y passed)
# [ ] Coverage >= 85% (actual: X%)
# [ ] No skip decorator abuse
# [ ] Code quality checks passed (black, ruff, mypy)
# [ ] Verification report: artifacts/verify/YYYYMMDD_HHMMSS/
#
# Test Output Summary:
# <paste key test results here>
#
# Coverage Summary:
# <paste coverage percentage here>
#
# Quality Issues (if any):
# <paste any warnings or errors here>
#
# Related Issue: #<issue-number>
```

**Type Options:**
- feat: New feature
- fix: Bug fix
- test: Test additions/modifications
- refactor: Code refactoring
- docs: Documentation changes
- chore: Maintenance tasks

**Instructions Section:**
Include commented instructions explaining:
- How to use the template
- Requirement to run verification script before committing
- Where to find verification artifacts
- How to interpret verification results
- Link to documentation for troubleshooting

Template follows Conventional Commits format with AI-specific verification requirements.

### README.md(MODIFY)

References: 

- TESTING.md(MODIFY)

Add new section **"AI Verification Workflow"** after the **"Testing"** section (around line 596):

**Content to Add:**

### AI Verification Workflow

Before committing AI-generated code, run the comprehensive verification script:

```bash
# Run full verification
./tools/verify-ai-claims.sh

# Quick quality check (skip tests)
./tools/verify-ai-claims.sh --skip-tests

# Continue on failures (see all issues)
./tools/verify-ai-claims.sh --no-fail-fast
```

**What It Checks:**
- ✅ Full test suite execution with verbose output
- ✅ Code coverage against 85% threshold
- ✅ Skip decorator abuse detection
- ✅ Code formatting (black)
- ✅ Linting (ruff)
- ✅ Type checking (mypy)

**Verification Report:**

The script generates detailed reports in `artifacts/verify/YYYYMMDD_HHMMSS/`:
- `test-report.json` - Full pytest JSON output
- `test-output.txt` - Human-readable test results
- `coverage-report.json` - Coverage data
- `htmlcov/` - HTML coverage report
- `quality-report.txt` - Code quality check results
- `verification-report.md` - Comprehensive summary

**Git Integration:**

Configure git to use the AI commit template:

```bash
git config commit.template .gitmessage
```

The template includes a checklist for AI verification requirements.

**Performance:**

Typical execution time: 15-25 seconds for full verification.

See `TESTING.md` for detailed verification workflow documentation.

Update the table of contents if present to include the new section. Reference the verification script location and the `.gitmessage` template.

### TESTING.md(MODIFY)

References: 

- tools/verify-ai-claims.sh(NEW)
- .gitmessage(NEW)
- README.md(MODIFY)
- .coveragerc
- pytest.ini

Add new section **"AI Verification Workflow"** after the existing test sections (around line 408, before "Next Steps"):

**Content to Add:**

## AI Verification Workflow

### Overview

The AI verification system ensures all AI-generated code meets quality standards before commits. The `tools/verify-ai-claims.sh` script provides comprehensive validation.

### Running Verification

```bash
# Full verification (recommended before commits)
./tools/verify-ai-claims.sh

# With options
./tools/verify-ai-claims.sh --no-fail-fast  # See all issues
./tools/verify-ai-claims.sh --skip-tests    # Quick quality check
./tools/verify-ai-claims.sh --help          # Show all options
```

### Verification Steps

#### Step 1: Test Suite Execution
- Runs pytest with verbose output and JSON reporting
- Captures passed/failed/errors/skipped counts
- Saves output to `artifacts/verify/TIMESTAMP/test-output.txt`
- **Pass Criteria:** All tests must pass (0 failures, 0 errors)

#### Step 2: Coverage Check
- Measures code coverage using pytest-cov
- Generates terminal, HTML, and JSON reports
- **Pass Criteria:** Coverage >= 85% (configured in `.coveragerc`)
- HTML report available at `artifacts/verify/TIMESTAMP/htmlcov/index.html`

#### Step 3: Skip Decorator Detection
- Searches for `@pytest.mark.skip`, `@skip`, `skipif` in test files
- Flags any usage for manual review
- **Pass Criteria:** Warning only (not a hard failure)
- Prevents test suite degradation through excessive skipping

#### Step 4: Code Quality Checks
- **Black:** Verifies code formatting (line-length=100)
- **Ruff:** Runs linting checks (target=py311)
- **Mypy:** Type checking with strict settings
- **Pass Criteria:** All tools must report no errors

#### Step 5: Report Generation
- Creates comprehensive verification report
- Includes all step results and artifact locations
- Saved as `artifacts/verify/TIMESTAMP/verification-report.md`

### Interpreting Results

**Success Output:**
```
🔍 Comprehensive AI Verification
=================================

📋 Step 1: Running test suite...
✅ All tests passed (93 passed, 0 failed)

📊 Step 2: Checking coverage...
✅ Coverage: 87% (target: 85%)

🔍 Step 3: Checking for @skip abuse...
✅ No skip decorators found

🎨 Step 4: Code quality checks...
✅ Formatting: OK
✅ Linting: OK  
✅ Type checking: OK

=================================
✅ ALL VERIFICATIONS PASSED
=================================
```

**Failure Output:**
```
📋 Step 1: Running test suite...
❌ Tests failed (89 passed, 4 failed)

Failed tests:
- tests/test_example.py::test_feature_x
- tests/test_example.py::test_feature_y

See: artifacts/verify/20251110_143022/test-output.txt
```

### Troubleshooting

**Issue: Tests Failing**
- Review test output in `artifacts/verify/TIMESTAMP/test-output.txt`
- Run specific test: `pytest tests/test_file.py::test_name -v`
- Check test logs for error details

**Issue: Coverage Below Threshold**
- Open HTML report: `artifacts/verify/TIMESTAMP/htmlcov/index.html`
- Identify uncovered lines (highlighted in red)
- Add tests for uncovered code paths
- Review `.coveragerc` exclude patterns if needed

**Issue: Skip Decorators Found**
- Review `artifacts/verify/TIMESTAMP/skip-detection.txt`
- Evaluate if skips are justified (platform-specific, requires external service)
- Remove unnecessary skips or document reasoning
- Consider using `pytest.mark.skipif` with clear conditions

**Issue: Code Quality Failures**
- **Black:** Run `black codeframe tests` to auto-format
- **Ruff:** Review errors in quality report, fix manually or run `ruff check --fix`
- **Mypy:** Add type hints or adjust mypy configuration in `pyproject.toml`

**Issue: Verification Too Slow**
- Use `--skip-coverage` for quick checks during development
- Install pytest-xdist for parallel test execution: `pip install pytest-xdist`
- Script will automatically use `-n auto` if available

### Git Integration

**Setup Commit Template:**
```bash
git config commit.template .gitmessage
```

**Commit Workflow:**
1. Make code changes
2. Run `./tools/verify-ai-claims.sh`
3. Review verification report
4. Fix any issues
5. Commit with template (includes verification checklist)
6. Paste verification results into commit message

**Template Checklist:**
- [ ] Ran verification script
- [ ] All tests passing
- [ ] Coverage >= 85%
- [ ] No skip abuse
- [ ] Quality checks passed
- [ ] Verification report path included

### Artifact Management

**Location:** `artifacts/verify/YYYYMMDD_HHMMSS/`

**Files Generated:**
- `test-report.json` - Pytest JSON output (machine-readable)
- `test-output.txt` - Human-readable test results
- `coverage-report.json` - Coverage data (JSON)
- `htmlcov/` - Interactive HTML coverage report
- `skip-detection.txt` - List of skip decorators found
- `quality-report.txt` - Black/ruff/mypy output
- `verification-report.md` - Comprehensive summary

**Cleanup:**
Artifacts are timestamped and accumulate over time. Periodically clean old artifacts:
```bash
# Keep last 10 verification runs
ls -t artifacts/verify/ | tail -n +11 | xargs -I {} rm -rf artifacts/verify/{}
```

### Performance Targets

- **Full verification:** < 30 seconds
- **Test execution:** < 20 seconds
- **Coverage analysis:** < 5 seconds
- **Quality checks:** < 5 seconds

**Optimization Tips:**
- Tests use pytest cache (`.pytest_cache/`)
- Coverage data cached between runs
- Parallel test execution with pytest-xdist
- Skip slow tests during development: `pytest -m "not slow"`

### CI Integration (Future)

The verification script is designed for CI integration:
- Predictable exit codes (0=pass, 1=fail)
- Machine-readable JSON outputs
- Artifact paths for CI artifact storage
- Environment variable support for configuration

Example GitHub Actions usage:
```yaml
- name: Run AI Verification
  run: ./tools/verify-ai-claims.sh
- name: Upload Artifacts
  uses: actions/upload-artifact@v3
  with:
    name: verification-report
    path: artifacts/verify/
```

Reference the verification script at `tools/verify-ai-claims.sh` and the git template at `.gitmessage`. Link to `README.md` AI Verification Workflow section for quick reference.

### CONTRIBUTING.md(MODIFY)

References: 

- tools/verify-ai-claims.sh(NEW)
- TESTING.md(MODIFY)

Update the **"Testing"** section (around line 34) to include verification workflow:

**Replace existing Testing section with:**

## Testing

### Running Tests

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=codeframe --cov-report=html

# Run specific test file
pytest tests/test_example.py -v
```

### AI Verification

Before submitting PRs with AI-generated code, run the verification script:

```bash
./tools/verify-ai-claims.sh
```

This ensures:
- All tests pass
- Coverage >= 85%
- No skip decorator abuse
- Code quality standards met (black, ruff, mypy)

See `TESTING.md` for detailed verification workflow documentation.

### Test Requirements

- Write unit tests for new features
- Maintain >85% code coverage (enforced by verification script)
- Follow existing test patterns in `tests/`
- Use fixtures from `tests/conftest.py`
- Mark slow tests with `@pytest.mark.slow`
- Mark integration tests with `@pytest.mark.integration`

**Update the "Pull Request Process" section (around line 40):**

Add step between "Add tests for new functionality" and "Update documentation":

```
4. Run verification script: `./tools/verify-ai-claims.sh`
5. Include verification report path in commit message
```

Reference the verification script at `tools/verify-ai-claims.sh` and link to `TESTING.md` for comprehensive documentation.

### pyproject.toml(MODIFY)

Add isort to dev dependencies and configure it:

**In the `[project.optional-dependencies]` section (around line 46-55), add:**

```toml
dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.23.0",
    "pytest-cov>=4.1.0",
    "pytest-json-report>=1.5.0",
    "black>=24.1.0",
    "ruff>=0.2.0",
    "mypy>=1.8.0",
    "isort>=5.12.0",  # ADD THIS LINE
]
```

**Add new `[tool.isort]` section after `[tool.mypy]` (around line 81):**

```toml
[tool.isort]
profile = "black"
line_length = 100
known_first_party = ["codeframe"]
sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]
```

This configures isort to:
- Be compatible with black formatting
- Use 100 character line length (matching black and ruff)
- Recognize codeframe as first-party package
- Follow standard import ordering (stdlib → third-party → local)

The verification script will use isort to check import ordering consistency.

### .gitignore(MODIFY)

Add verification artifacts directory to gitignore:

**Add to the file (if .gitignore exists, otherwise create it):**

```
# Verification artifacts
artifacts/

# Coverage reports (if not already present)
htmlcov/
.coverage
coverage.xml
coverage.json

# Pytest cache (if not already present)
.pytest_cache/
```

This prevents verification artifacts from being committed to the repository while keeping them available locally for review.

---

Developer Humor

🤖 An AI walks into a code review...
The reviewer says, \Did you run the tests?
AI replies, \I generated a script that generates a report
that verifies I generated the tests that verify the code!
Reviewer: ...but did they PASS?
AI: \Let me check the artifacts... 📊\

Verification: It's turtles all the way down! 🐢

---

Execution Information

Branch: main
Commit: d0e2480

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[frankbria]

Resolved in PR #20 - Enhanced Verification and Reporting complete. scripts/verify-ai-claims.sh provides 5-step verification (tests, coverage, skip detection, quality checks, report generation). Includes timestamped artifacts directory, HTML reports, CLI options, and .gitmessage template.