Create Comprehensive Test Template

[frankbria]

Summary

Provide a reference test template that demonstrates best practices for AI agents to follow when writing tests.

Goal

AI agents need concrete examples of:

• Traditional unit tests
• Property-based tests with Hypothesis
• Parametrized tests
• Integration tests
• Proper fixture usage

Tasks

1. Create Template File

• Create tests/test_template.py
• Add comprehensive docstrings
• Include multiple testing patterns
• Show Hypothesis integration

2. Test Pattern Examples

• Specific known cases
• Error handling tests
• Fixture usage
• Parametrized tests
• Property-based tests (Hypothesis)
• Integration tests

3. Hypothesis Patterns

• Idempotent operations
• Commutative properties
• Type stability
• Length preservation
• Never-crash properties

4. Documentation

• Explain when to use each pattern
• Add "why" comments throughout
• Link to pytest/Hypothesis docs
• Update .claude/rules.md to reference template

Example Patterns

# Traditional
def test_specific_case():
    assert function(input) == output

# Property-based
@given(st.integers())
def test_idempotent(x):
    assert f(f(x)) == f(x)

# Parametrized
@pytest.mark.parametrize("input,expected", [
    (1, 1), (2, 4), (3, 9)
])
def test_cases(input, expected):
    assert function(input) == expected

Pattern Coverage Matrix

Pattern	Use Case	Example
Unit test	Known input/output	login with valid credentials
Property test	Mathematical properties	reversing twice = original
Parametrized	Multiple similar cases	boundary values
Fixture	Shared setup	database connection
Integration	End-to-end workflow	complete user journey

Success Criteria

• Template covers all common test patterns
• AI agents can reference it successfully
• Reduces test quality issues
• Serves as team reference

Dependencies

• Requires hypothesis package
• Should reference Issue Add Claude Code GitHub Workflow #1 rules

Estimated Effort: 2-3 hours

[traycerai]

Plan

Observations

The codebase has a well-established testing infrastructure with pytest, but lacks examples of advanced testing patterns. Current tests use:

• Class-based organization with descriptive names
• Pytest markers (@pytest.mark.unit, @pytest.mark.integration)
• Fixtures from conftest.py (temp_dir, temp_db_path, mock_env, etc.)
• Good docstrings and comments
• Mock-based unit tests and real-component integration tests

Missing patterns:

• Hypothesis property-based testing (not in dependencies)
• Parametrized tests (configured but not used)
• Comprehensive fixture usage examples

Documentation structure:

• No .claude/rules.md exists (mentioned in issue)
• AGENTS.md serves as the AI agent navigation guide
• CLAUDE.md contains coding standards
• TESTING.md focuses on manual/integration testing

The template should demonstrate all patterns while following existing conventions and serving as both an AI agent reference and team resource.

Approach

Create a comprehensive test template file that demonstrates all major testing patterns with extensive documentation. Add Hypothesis as a dev dependency. Update documentation to reference the new template for AI agents and developers.

Reasoning

Explored the repository structure, examined existing test files to understand patterns and conventions, reviewed pytest configuration and fixtures, checked dependencies in pyproject.toml, and analyzed documentation structure to determine where to add references to the new template.

Proposed File Changes

📄 tests/test_template.py ^(NEW) 🔗

References

• 📄 tests/conftest.py
• 📄 tests/test_database.py
• 📄 pytest.ini

Create a comprehensive test template file that demonstrates all major testing patterns with extensive documentation.

Module-level docstring:

• Explain the purpose: reference template for AI agents and developers
• List all patterns covered: traditional unit tests, property-based tests, parametrized tests, integration tests, fixture usage
• Include links to pytest and Hypothesis documentation
• Explain when to use each pattern (decision matrix)

Pattern 1: Traditional Unit Tests

• Create a TestTraditionalUnitTests class demonstrating specific known input/output tests
• Include examples of testing pure functions with known cases
• Show error handling tests using pytest.raises
• Demonstrate assertion patterns
• Add comments explaining when to use: testing specific business logic, edge cases, known bugs

Pattern 2: Parametrized Tests

• Create a TestParametrizedTests class using @pytest.mark.parametrize
• Show single-parameter parametrization
• Show multi-parameter parametrization with tuples
• Demonstrate parametrize with ids for readable test names
• Include examples of boundary value testing
• Add comments explaining when to use: multiple similar test cases, boundary testing, regression suites

Pattern 3: Property-Based Tests with Hypothesis

• Create a TestPropertyBasedTests class using Hypothesis strategies
• Demonstrate idempotent operations (f(f(x)) == f(x))
• Show commutative properties (f(a, b) == f(b, a))
• Demonstrate type stability (output type matches input type)
• Show length preservation properties
• Include never-crash properties (function should not raise for any valid input)
• Use strategies from hypothesis.strategies: integers, text, lists, composite
• Add comments explaining when to use: mathematical properties, invariants, fuzzing for edge cases

Pattern 4: Fixture Usage

• Create a TestFixtureUsage class demonstrating various fixture patterns
• Show usage of fixtures from conftest.py (temp_dir, temp_db_path, mock_env)
• Demonstrate fixture composition (fixtures using other fixtures)
• Show function-scoped vs class-scoped fixtures
• Include autouse fixtures
• Add comments explaining fixture benefits: setup/teardown, dependency injection, test isolation

Pattern 5: Integration Tests

• Create a TestIntegrationPatterns class marked with @pytest.mark.integration
• Demonstrate testing with real components (database, file I/O)
• Show multi-step workflows
• Include transaction testing
• Add comments explaining when to use: end-to-end workflows, component interaction, system behavior

Pattern 6: Async Tests

• Create a TestAsyncPatterns class using @pytest.mark.asyncio
• Show async function testing
• Demonstrate async fixture usage
• Include comments about asyncio_mode configuration

Helper Functions:

• Create simple example functions to test (e.g., reverse_string, add_numbers, normalize_data)
• Keep them simple and focused on demonstrating test patterns
• Add docstrings explaining their purpose in the template

Pattern Coverage Matrix:

• Include a comprehensive docstring section with a table showing:
  • Pattern name
  • Use case
  • Example scenario
  • When to use vs when not to use

Best Practices Section:

• Add comments throughout explaining:
  • Test naming conventions (test_)
  • Arrange-Act-Assert pattern
  • One assertion per test (when appropriate)
  • Test independence
  • Meaningful test descriptions

📄 pyproject.toml ^(MODIFY) 🔗

Add Hypothesis as a development dependency to enable property-based testing.

In the [project.optional-dependencies] section under dev:

• Add hypothesis>=6.0.0 to the list of dev dependencies
• Place it after the existing pytest-related packages (pytest, pytest-asyncio, pytest-cov, pytest-json-report) for logical grouping
• Use version constraint >=6.0.0 to ensure modern Hypothesis features are available

Rationale:

• Hypothesis is only needed for development/testing, not runtime
• Version 6.0.0+ provides stable API and good Python 3.11+ support
• Follows existing pattern of grouping test-related dependencies together

📄 AGENTS.md ^(MODIFY) 🔗

References

• 📄 tests/test_template.py ^(NEW)

Update the AI agent documentation guide to reference the new test template.

In the "Quick Navigation" table:

• Update the "Understand testing" row to mention the test template
• Change description from just "Test requirements" to "Test requirements & patterns"

Add a new section after "Quick Reference Examples" (around line 272):

• Create a new section titled "### Writing Tests"
• Add guidance for AI agents on how to use the test template:
  • Reference tests/test_template.py as the canonical example
  • Explain it demonstrates traditional unit tests, property-based tests, parametrized tests, integration tests, and fixture usage
  • Note that it includes decision guidance on when to use each pattern
  • Mention that all examples follow project conventions (class-based organization, markers, fixtures from conftest.py)
• Add a code example showing how to reference the template:

  # View comprehensive test patterns
  cat tests/test_template.py  # All testing patterns with examples
  
  # Run the template tests to see patterns in action
  pytest tests/test_template.py -v

Rationale:

• AGENTS.md is the primary navigation guide for AI agents
• Adding test template reference here ensures AI agents can easily find testing guidance
• Follows the existing structure and style of the document

📄 TESTING.md ^(MODIFY) 🔗

References

• 📄 tests/test_template.py ^(NEW)

Add a reference to the test template at the beginning of the testing guide.

After the main heading "# CodeFRAME Testing Guide" (line 1):

• Add a new section titled "## Test Pattern Reference"
• Include a prominent callout box or section explaining:
  • For comprehensive test pattern examples, see tests/test_template.py
  • The template demonstrates all major testing patterns: traditional unit tests, property-based tests with Hypothesis, parametrized tests, integration tests, and fixture usage
  • Each pattern includes detailed comments explaining when and how to use it
  • The template follows all project conventions and serves as the canonical reference for writing tests
• Add a note that the template is designed for both AI agents and human developers
• Include a quick command to view it: cat tests/test_template.py

In the "Automated Test Suite Verification" section (around line 292):

• Add a note about running the template tests: pytest tests/test_template.py -v
• Mention that the template itself is tested and demonstrates working patterns

Rationale:

• TESTING.md is the primary testing documentation
• Adding the reference early ensures developers see it immediately
• Complements the existing manual testing checklist with automated testing patterns

📄 CLAUDE.md ^(MODIFY) 🔗

References

• 📄 tests/test_template.py ^(NEW)
• 📄 TESTING.md ^(MODIFY)

Add a reference to the test template in the coding standards documentation.

In the "Commands" section (around line 38):

• Add a new command entry for viewing test patterns:

  pytest tests/test_template.py -v  # View all test patterns

After the "Code Style" section (around line 48):

• Add a new section titled "## Testing Standards"
• Include bullet points:
  • Follow patterns demonstrated in tests/test_template.py
  • Use traditional unit tests for specific known cases
  • Use property-based tests (Hypothesis) for mathematical properties and invariants
  • Use parametrized tests for multiple similar cases
  • Use integration tests for end-to-end workflows
  • All tests should use class-based organization with descriptive names
  • Apply appropriate markers: @pytest.mark.unit, @pytest.mark.integration, @pytest.mark.slow
  • Target >85% test coverage for new code
• Add a reference: "See TESTING.md for detailed testing procedures"

Rationale:

• CLAUDE.md is the project-specific coding standards document
• AI agents reference this file for coding conventions
• Adding testing standards here ensures consistency in test writing
• Keeps the reference brief and points to detailed documentation

📄 .claude/rules.md ^(NEW) 🔗

References

• 📄 tests/test_template.py ^(NEW)
• 📄 TESTING.md ^(MODIFY)
• 📄 AGENTS.md ^(MODIFY)
• 📄 CLAUDE.md ^(MODIFY)

Create a new rules file for AI agents with testing guidelines as mentioned in the GitHub issue.

File purpose:

• Provide concise rules and guidelines specifically for AI agents
• Focus on actionable directives rather than comprehensive documentation

Content structure:

# AI Agent Rules for CodeFRAME

## Testing Rules

• ALWAYS reference tests/test_template.py when writing tests
• MUST use class-based test organization (TestClassName)
• MUST add appropriate pytest markers (@pytest.mark.unit, @pytest.mark.integration)
• MUST write descriptive test names: test_
• MUST include docstrings explaining what each test verifies
• MUST use fixtures from tests/conftest.py for common setup
• SHOULD use property-based tests (Hypothesis) for mathematical properties and invariants
• SHOULD use parametrized tests for multiple similar cases
• SHOULD follow Arrange-Act-Assert pattern
• MUST NOT write tests without consulting the template first

## Test Pattern Selection

• Use traditional unit tests for: specific known cases, edge cases, error handling
• Use property-based tests for: mathematical properties, invariants, fuzzing
• Use parametrized tests for: boundary values, multiple similar cases, regression suites
• Use integration tests for: end-to-end workflows, component interaction
• Use fixtures for: shared setup, dependency injection, test isolation

## Code Quality

• Target >85% test coverage for new code
• Each test should be independent and isolated
• Tests should be fast (unit tests <100ms, integration tests <1s)
• Use mocks for external dependencies in unit tests
• Use real components in integration tests

## Documentation

• See TESTING.md for comprehensive testing guide
• See tests/test_template.py for pattern examples
• See AGENTS.md for navigation guide
• See CLAUDE.md for coding standards

Rationale:

• The GitHub issue specifically mentions updating .claude/rules.md
• This file provides quick, actionable rules for AI agents
• Complements the detailed documentation in other files
• Follows imperative style (MUST, SHOULD) for clarity

Import In IDE

🤖 Prompt for AI Agents

I have created the following plan after thorough exploration and analysis of the codebase. Follow the below plan verbatim. Trust the files and references. Do not re-verify what's written in the plan. Explore only when absolutely necessary. First implement all the proposed file changes and then I'll review all the changes together at the end.

### Observations

The codebase has a well-established testing infrastructure with pytest, but lacks examples of advanced testing patterns. Current tests use:
- Class-based organization with descriptive names
- Pytest markers (`@pytest.mark.unit`, `@pytest.mark.integration`)
- Fixtures from `conftest.py` (temp_dir, temp_db_path, mock_env, etc.)
- Good docstrings and comments
- Mock-based unit tests and real-component integration tests

**Missing patterns:**
- Hypothesis property-based testing (not in dependencies)
- Parametrized tests (configured but not used)
- Comprehensive fixture usage examples

**Documentation structure:**
- No `.claude/rules.md` exists (mentioned in issue)
- `AGENTS.md` serves as the AI agent navigation guide
- `CLAUDE.md` contains coding standards
- `TESTING.md` focuses on manual/integration testing

The template should demonstrate all patterns while following existing conventions and serving as both an AI agent reference and team resource.


### Approach

Create a comprehensive test template file that demonstrates all major testing patterns with extensive documentation. Add Hypothesis as a dev dependency. Update documentation to reference the new template for AI agents and developers.


### Reasoning

Explored the repository structure, examined existing test files to understand patterns and conventions, reviewed pytest configuration and fixtures, checked dependencies in pyproject.toml, and analyzed documentation structure to determine where to add references to the new template.


## Proposed File Changes

### tests/test_template.py(NEW)

References: 

- tests/conftest.py
- tests/test_database.py
- pytest.ini

Create a comprehensive test template file that demonstrates all major testing patterns with extensive documentation.

**Module-level docstring:**
- Explain the purpose: reference template for AI agents and developers
- List all patterns covered: traditional unit tests, property-based tests, parametrized tests, integration tests, fixture usage
- Include links to pytest and Hypothesis documentation
- Explain when to use each pattern (decision matrix)

**Pattern 1: Traditional Unit Tests**
- Create a `TestTraditionalUnitTests` class demonstrating specific known input/output tests
- Include examples of testing pure functions with known cases
- Show error handling tests using `pytest.raises`
- Demonstrate assertion patterns
- Add comments explaining when to use: testing specific business logic, edge cases, known bugs

**Pattern 2: Parametrized Tests**
- Create a `TestParametrizedTests` class using `@pytest.mark.parametrize`
- Show single-parameter parametrization
- Show multi-parameter parametrization with tuples
- Demonstrate parametrize with ids for readable test names
- Include examples of boundary value testing
- Add comments explaining when to use: multiple similar test cases, boundary testing, regression suites

**Pattern 3: Property-Based Tests with Hypothesis**
- Create a `TestPropertyBasedTests` class using Hypothesis strategies
- Demonstrate idempotent operations (f(f(x)) == f(x))
- Show commutative properties (f(a, b) == f(b, a))
- Demonstrate type stability (output type matches input type)
- Show length preservation properties
- Include never-crash properties (function should not raise for any valid input)
- Use strategies from `hypothesis.strategies`: integers, text, lists, composite
- Add comments explaining when to use: mathematical properties, invariants, fuzzing for edge cases

**Pattern 4: Fixture Usage**
- Create a `TestFixtureUsage` class demonstrating various fixture patterns
- Show usage of fixtures from `conftest.py` (temp_dir, temp_db_path, mock_env)
- Demonstrate fixture composition (fixtures using other fixtures)
- Show function-scoped vs class-scoped fixtures
- Include autouse fixtures
- Add comments explaining fixture benefits: setup/teardown, dependency injection, test isolation

**Pattern 5: Integration Tests**
- Create a `TestIntegrationPatterns` class marked with `@pytest.mark.integration`
- Demonstrate testing with real components (database, file I/O)
- Show multi-step workflows
- Include transaction testing
- Add comments explaining when to use: end-to-end workflows, component interaction, system behavior

**Pattern 6: Async Tests**
- Create a `TestAsyncPatterns` class using `@pytest.mark.asyncio`
- Show async function testing
- Demonstrate async fixture usage
- Include comments about asyncio_mode configuration

**Helper Functions:**
- Create simple example functions to test (e.g., `reverse_string`, `add_numbers`, `normalize_data`)
- Keep them simple and focused on demonstrating test patterns
- Add docstrings explaining their purpose in the template

**Pattern Coverage Matrix:**
- Include a comprehensive docstring section with a table showing:
  - Pattern name
  - Use case
  - Example scenario
  - When to use vs when not to use

**Best Practices Section:**
- Add comments throughout explaining:
  - Test naming conventions (test_<what>_<condition>_<expected>)
  - Arrange-Act-Assert pattern
  - One assertion per test (when appropriate)
  - Test independence
  - Meaningful test descriptions

### pyproject.toml(MODIFY)

Add Hypothesis as a development dependency to enable property-based testing.

**In the `[project.optional-dependencies]` section under `dev`:**
- Add `hypothesis>=6.0.0` to the list of dev dependencies
- Place it after the existing pytest-related packages (pytest, pytest-asyncio, pytest-cov, pytest-json-report) for logical grouping
- Use version constraint `>=6.0.0` to ensure modern Hypothesis features are available

**Rationale:**
- Hypothesis is only needed for development/testing, not runtime
- Version 6.0.0+ provides stable API and good Python 3.11+ support
- Follows existing pattern of grouping test-related dependencies together

### AGENTS.md(MODIFY)

References: 

- tests/test_template.py(NEW)

Update the AI agent documentation guide to reference the new test template.

**In the "Quick Navigation" table:**
- Update the "Understand testing" row to mention the test template
- Change description from just "Test requirements" to "Test requirements & patterns"

**Add a new section after "Quick Reference Examples" (around line 272):**
- Create a new section titled "### Writing Tests"
- Add guidance for AI agents on how to use the test template:
  - Reference `tests/test_template.py` as the canonical example
  - Explain it demonstrates traditional unit tests, property-based tests, parametrized tests, integration tests, and fixture usage
  - Note that it includes decision guidance on when to use each pattern
  - Mention that all examples follow project conventions (class-based organization, markers, fixtures from conftest.py)
- Add a code example showing how to reference the template:
  ```bash
  # View comprehensive test patterns
  cat tests/test_template.py  # All testing patterns with examples
  
  # Run the template tests to see patterns in action
  pytest tests/test_template.py -v
  ```

**Rationale:**
- AGENTS.md is the primary navigation guide for AI agents
- Adding test template reference here ensures AI agents can easily find testing guidance
- Follows the existing structure and style of the document

### TESTING.md(MODIFY)

References: 

- tests/test_template.py(NEW)

Add a reference to the test template at the beginning of the testing guide.

**After the main heading "# CodeFRAME Testing Guide" (line 1):**
- Add a new section titled "## Test Pattern Reference"
- Include a prominent callout box or section explaining:
  - For comprehensive test pattern examples, see `tests/test_template.py`
  - The template demonstrates all major testing patterns: traditional unit tests, property-based tests with Hypothesis, parametrized tests, integration tests, and fixture usage
  - Each pattern includes detailed comments explaining when and how to use it
  - The template follows all project conventions and serves as the canonical reference for writing tests
- Add a note that the template is designed for both AI agents and human developers
- Include a quick command to view it: `cat tests/test_template.py`

**In the "Automated Test Suite Verification" section (around line 292):**
- Add a note about running the template tests: `pytest tests/test_template.py -v`
- Mention that the template itself is tested and demonstrates working patterns

**Rationale:**
- TESTING.md is the primary testing documentation
- Adding the reference early ensures developers see it immediately
- Complements the existing manual testing checklist with automated testing patterns

### CLAUDE.md(MODIFY)

References: 

- tests/test_template.py(NEW)
- TESTING.md(MODIFY)

Add a reference to the test template in the coding standards documentation.

**In the "Commands" section (around line 38):**
- Add a new command entry for viewing test patterns:
  ```bash
  pytest tests/test_template.py -v  # View all test patterns
  ```

**After the "Code Style" section (around line 48):**
- Add a new section titled "## Testing Standards"
- Include bullet points:
  - Follow patterns demonstrated in `tests/test_template.py`
  - Use traditional unit tests for specific known cases
  - Use property-based tests (Hypothesis) for mathematical properties and invariants
  - Use parametrized tests for multiple similar cases
  - Use integration tests for end-to-end workflows
  - All tests should use class-based organization with descriptive names
  - Apply appropriate markers: `@pytest.mark.unit`, `@pytest.mark.integration`, `@pytest.mark.slow`
  - Target >85% test coverage for new code
- Add a reference: "See `TESTING.md` for detailed testing procedures"

**Rationale:**
- CLAUDE.md is the project-specific coding standards document
- AI agents reference this file for coding conventions
- Adding testing standards here ensures consistency in test writing
- Keeps the reference brief and points to detailed documentation

### .claude/rules.md(NEW)

References: 

- tests/test_template.py(NEW)
- TESTING.md(MODIFY)
- AGENTS.md(MODIFY)
- CLAUDE.md(MODIFY)

Create a new rules file for AI agents with testing guidelines as mentioned in the GitHub issue.

**File purpose:**
- Provide concise rules and guidelines specifically for AI agents
- Focus on actionable directives rather than comprehensive documentation

**Content structure:**

**# AI Agent Rules for CodeFRAME**

**## Testing Rules**
- ALWAYS reference `tests/test_template.py` when writing tests
- MUST use class-based test organization (TestClassName)
- MUST add appropriate pytest markers (@pytest.mark.unit, @pytest.mark.integration)
- MUST write descriptive test names: test_<what>_<condition>_<expected>
- MUST include docstrings explaining what each test verifies
- MUST use fixtures from `tests/conftest.py` for common setup
- SHOULD use property-based tests (Hypothesis) for mathematical properties and invariants
- SHOULD use parametrized tests for multiple similar cases
- SHOULD follow Arrange-Act-Assert pattern
- MUST NOT write tests without consulting the template first

**## Test Pattern Selection**
- Use traditional unit tests for: specific known cases, edge cases, error handling
- Use property-based tests for: mathematical properties, invariants, fuzzing
- Use parametrized tests for: boundary values, multiple similar cases, regression suites
- Use integration tests for: end-to-end workflows, component interaction
- Use fixtures for: shared setup, dependency injection, test isolation

**## Code Quality**
- Target >85% test coverage for new code
- Each test should be independent and isolated
- Tests should be fast (unit tests <100ms, integration tests <1s)
- Use mocks for external dependencies in unit tests
- Use real components in integration tests

**## Documentation**
- See `TESTING.md` for comprehensive testing guide
- See `tests/test_template.py` for pattern examples
- See `AGENTS.md` for navigation guide
- See `CLAUDE.md` for coding standards

**Rationale:**
- The GitHub issue specifically mentions updating .claude/rules.md
- This file provides quick, actionable rules for AI agents
- Complements the detailed documentation in other files
- Follows imperative style (MUST, SHOULD) for clarity

---

Developer Humor

🧪 A test template walks into a bar...\n> The bartender says, \What'll it be?\n> Template replies, \I'll have:\n> - One traditional unit (neat)\n> - A parametrized flight (3-5 samples)\n> - Property-based on tap (infinite variety)\n> - And an integration chaser (the whole workflow)\n> \n> Bartender: \Coming right up! That'll be 100% coverage.\n> Template: \Perfect! I always assert my expectations.\ 🎯\n> \n> 🔬 May your tests be green and your coverage be high! 🚀

---

Execution Information

Branch: main
Commit: d0e2480

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[frankbria]

Resolved in PR #20 - Comprehensive Test Template created at tests/test_template.py with 36 test examples. Covers traditional, parametrized, property-based, fixture, integration, and async testing patterns. 36/36 tests passing.