Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 37 additions & 95 deletions .github/workflows/daily-testify-uber-super-expert.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,26 +61,15 @@ Analyze one Go test file daily that hasn't been processed recently, evaluate its
## Current Context

- **Repository**: ${{ github.repository }}
- **Analysis Date**: $(date +%Y-%m-%d)
- **Analysis Date**: When writing cache entries, use one consistent `YYYY-MM-DD` date value for the current run. If you cannot obtain a reliable date, omit the timestamp and explain that limitation.
- **Workspace**: ${{ github.workspace }}
- **Cache Location**: `/tmp/gh-aw/repo-memory/default/memory/testify-expert/`

## Analysis Process

### 1. Load Processed Files Cache

Check the repo-memory cache to see which files have been processed recently:

```bash
# Check if cache file exists
CACHE_FILE="/tmp/gh-aw/repo-memory/default/memory/testify-expert/processed_files.txt"
if [ -f "$CACHE_FILE" ]; then
echo "Found cache with $(wc -l < "$CACHE_FILE") processed files"
cat "$CACHE_FILE"
else
echo "No cache found - first run"
fi
```
Check the repo-memory cache at `/tmp/gh-aw/repo-memory/default/memory/testify-expert/processed_files.txt` to see which files have been processed recently. The file is optional: if it does not exist, treat this as the first run and consider all test files as unprocessed candidates eligible for selection.

The cache file contains one file path per line with a timestamp:
```
Expand All @@ -90,41 +79,24 @@ The cache file contains one file path per line with a timestamp:

### 2. Select Target Test File

Find all Go test files and select one that hasn't been processed in the last 30 days:
Find all Go test files and select one that has not been processed recently.

```bash
# Get all test files
find . -name '*_test.go' -type f > /tmp/all_test_files.txt

# Filter out recently processed files (last 30 days)
CUTOFF_DATE=$(date -d '30 days ago' '+%Y-%m-%d' 2>/dev/null || date -v-30d '+%Y-%m-%d')

# Create list of candidate files (not processed or processed >30 days ago)
while IFS='|' read -r filepath timestamp; do
if [[ "$timestamp" < "$CUTOFF_DATE" ]]; then
echo "$filepath" >> /tmp/candidate_files.txt
fi
done < "$CACHE_FILE" 2>/dev/null || true

# If no cache or all files old, use all test files
if [ ! -f /tmp/candidate_files.txt ]; then
cp /tmp/all_test_files.txt /tmp/candidate_files.txt
fi

# Select a random file from candidates
TARGET_FILE=$(shuf -n 1 /tmp/candidate_files.txt)
echo "Selected file: $TARGET_FILE"
```
- Use the allowed `find . -name '*_test.go' -type f` command to list candidate test files.
- Compare that list against the cache contents in your reasoning instead of relying on shell pipelines or temporary-file scripts.
- Prefer a file that is not present in the cache at all.
- If every file is already present in the cache, prefer the oldest cached entry.
- If you cannot reliably determine recency from the available tools, pick the first file alphabetically from the candidate list, continue rather than failing the workflow, and mention in your final output that the selection may not fully respect the 30-day cycle and that cache maintenance should be investigated.
- Do not depend on shell commands such as `date`, `shuf`, `cp`, `sed`, `awk`, `sort`, `mv`, or multiline shell loops.

**Important**: If no unprocessed files remain, output a message and exit:
```
✅ All test files have been analyzed in the last 30 days!
The testify expert will resume analysis after the cache expires.
```

### 3. Analyze Test File with Serena
### 3. Analyze Test File with the Language Server Tool

Use the Serena MCP server to perform deep semantic analysis of the selected test file:
Use the available language server / semantic code analysis tool (Serena in this workflow) to perform deep semantic analysis of the selected test file:

1. **Read the file contents** and understand its structure
2. **Identify the corresponding source file** (e.g., `pkg/workflow/compiler_test.go` → `pkg/workflow/compiler.go`)
Expand Down Expand Up @@ -154,23 +126,7 @@ Use the Serena MCP server to perform deep semantic analysis of the selected test

Examine what's being tested and what's missing:

```bash
# Get the source file
SOURCE_FILE=$(echo "$TARGET_FILE" | sed 's/_test\.go$/.go/')

if [ -f "$SOURCE_FILE" ]; then
# Extract function signatures from source
grep -E '^func [A-Z]' "$SOURCE_FILE" | sed 's/func //' | cut -d'(' -f1

# Extract test function names
grep -E '^func Test' "$TARGET_FILE" | sed 's/func //' | cut -d'(' -f1

# Compare to find untested functions
echo "=== Comparing coverage ==="
else
echo "Source file not found: $SOURCE_FILE"
fi
```
Derive `SOURCE_FILE` by replacing the `_test.go` suffix with `.go` in your reasoning. If that source file exists, inspect both files with the available language server tool first; if you need raw file contents, use the environment's standard file-reading tool instead of ad-hoc shell pipelines. Use the allowed `grep -r 'func Test' . --include='*_test.go'` command only when it is helpful for quick repository-wide confirmation.

Calculate:
- **Functions in source**: Count of exported functions
Expand Down Expand Up @@ -230,21 +186,19 @@ Create reports that:
Create a detailed issue with this structure:

```markdown
# Improve Test Quality: [FILE_PATH]

## Overview
### Overview

The test file `[FILE_PATH]` has been selected for quality improvement by the Testify Uber Super Expert. This issue provides specific, actionable recommendations to enhance test quality, coverage, and maintainability using testify best practices.

## Current State
### Current State

- **Test File**: `[FILE_PATH]`
- **Source File**: `[SOURCE_FILE]` (if exists)
- **Test Functions**: [COUNT] test functions
- **Lines of Code**: [LOC] lines
- **Last Modified**: [DATE if available]

## Test Quality Analysis
### Test Quality Analysis

### Strengths ✅

Expand Down Expand Up @@ -274,7 +228,7 @@ require.NoError(t, err, "operation should succeed")
assert.Equal(t, expected, result, "result should match expected value")
```

**Why this matters**: Testify provides clearer error messages, better test output, and is the standard used throughout this codebase (see `scratchpad/testing.md`).
**Why this matters**: Testify provides clearer error messages, better test output, and is the standard used throughout this codebase.

#### 2. Table-Driven Tests

Expand Down Expand Up @@ -322,7 +276,7 @@ func TestFunctionName(t *testing.T) {
}
```

**Why this matters**: Table-driven tests are easier to extend, maintain, and understand. They follow the pattern used in `scratchpad/testing.md`.
**Why this matters**: Table-driven tests are easier to extend, maintain, and understand.

#### 3. Test Coverage Gaps

Expand Down Expand Up @@ -391,7 +345,7 @@ require.NoError(t, err, "setup should succeed without errors")
4. **Medium**: Improve test names and organization
5. **Low**: Add assertion messages

### Best Practices from `scratchpad/testing.md`
### Best Practices from the repository
- ✅ Use `require.*` for critical setup (stops test on failure)
- ✅ Use `assert.*` for test validations (continues checking)
- ✅ Write table-driven tests with `t.Run()` and descriptive names
Expand All @@ -400,14 +354,11 @@ require.NoError(t, err, "setup should succeed without errors")

### Testing Commands
```bash
# Run tests for this file
go test -v [PACKAGE_PATH] -run [TEST_NAME]

# Run tests with coverage
go test -cover [PACKAGE_PATH]
# Run the package containing the selected file
go test -v [PACKAGE_PATH]

# Run all tests
make test-unit
# Run the repository test suite
go test ./... -timeout 300s
```

## Acceptance Criteria
Expand All @@ -417,13 +368,13 @@ make test-unit
- [ ] All critical functions in source file have corresponding tests
- [ ] Test names are descriptive and follow conventions
- [ ] All assertions include helpful messages
- [ ] Tests pass: `make test-unit`
- [ ] Code follows patterns in `scratchpad/testing.md`
- [ ] Tests pass: `go test ./... -timeout 300s`
- [ ] Code follows existing patterns in this repository

## Additional Context

- **Repository Testing Guidelines**: See `scratchpad/testing.md` for comprehensive testing patterns
- **Example Tests**: Look at recent test files in `pkg/workflow/*_test.go` for examples
- **Repository Testing Guidelines**: Prefer the testify patterns already used in files such as `internal/logging/logger_test.go`, `internal/expression/evaluator_test.go`, and `internal/session/transcript_test.go`
- **Example Tests**: Look at recent test files under `internal/**` and `cmd/hookflow/**`
- **Testify Documentation**: https://github.com/stretchr/testify

---
Expand All @@ -441,31 +392,22 @@ make test-unit

After creating the issue, update the cache to record this file as processed:

```bash
# Append to cache with current date
CACHE_FILE="/tmp/gh-aw/repo-memory/default/memory/testify-expert/processed_files.txt"
mkdir -p "$(dirname "$CACHE_FILE")"
TODAY=$(date '+%Y-%m-%d')
echo "${TARGET_FILE}|${TODAY}" >> "$CACHE_FILE"

# Sort and deduplicate cache (keep most recent date for each file)
sort -t'|' -k1,1 -k2,2r "$CACHE_FILE" | \
awk -F'|' '!seen[$1]++' > "${CACHE_FILE}.tmp"
mv "${CACHE_FILE}.tmp" "$CACHE_FILE"

echo "✅ Updated cache with processed file: $TARGET_FILE"
```
Update `/tmp/gh-aw/repo-memory/default/memory/testify-expert/processed_files.txt` using the available file-editing tools. Append one line in the format `path|YYYY-MM-DD`, using the analysis date described in the **Current Context** section above. If you can safely deduplicate entries for the same file while keeping the most recent date, do so; otherwise, appending is acceptable.

If cache maintenance is not possible with the available tools, do not fail the workflow for that reason alone. If an issue was created successfully, the workflow should still succeed even when cache update fails, but your final output must include a warning that cache maintenance failed. If no issue could be created, use the `noop` safe-output tool with an explanation instead of aborting.

## Output Requirements

Your workflow MUST follow this sequence:

1. **Load cache** - Check which files have been processed
2. **Select file** - Choose one unprocessed or old file (>30 days)
3. **Analyze file** - Use Serena to deeply analyze the test file
4. **Create issue** - Generate detailed issue with specific improvements
3. **Analyze file** - Use the available language server tool to deeply analyze the test file
4. **Create issue** - Use the safe-output tool `create_issue` to generate a detailed issue with specific improvements
5. **Update cache** - Record the file as processed with today's date

If you decide no issue should be created, use the safe-output tool `noop` with a clear explanation instead of failing silently.

### Output Format

**If no unprocessed files:**
Expand Down Expand Up @@ -497,16 +439,16 @@ Total Processed Files: [COUNT]
## Important Guidelines

- **One file per day**: Focus on providing high-quality, detailed analysis for a single file
- **Use Serena extensively**: Leverage the language server for semantic understanding
- **Use the language server tool extensively**: Leverage semantic analysis for deeper understanding
- **Be specific and actionable**: Provide code examples, not vague advice
- **Follow repository patterns**: Reference `scratchpad/testing.md` and existing test patterns
- **Cache management**: Always update the cache after processing
- **Follow repository patterns**: Reference existing tests in this repository
- **Cache management**: Update the cache when the available tools make it safe to do so
- **30-day cycle**: Files become eligible for re-analysis after 30 days
- **Priority to uncovered code**: Prefer files with lower test coverage when selecting

## Testify Best Practices Reference

### Common Patterns from `scratchpad/testing.md`
### Common Patterns from this repository

**Use `require.*` for setup:**
```go
Expand Down
Binary file added hookflow
Binary file not shown.