Skip to content
Merged
25 changes: 7 additions & 18 deletions .github/workflows/cli-consistency-checker.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

166 changes: 162 additions & 4 deletions .github/workflows/cli-consistency-checker.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ permissions:
issues: read
pull-requests: read
engine: copilot
imports:
- ../agents/cli-consistency-checker.md
network:
allowed: [defaults, node, "api.github.com"]
tools:
Expand All @@ -32,7 +30,167 @@ Perform a comprehensive inspection of the `gh-aw` CLI tool to identify inconsist

**Repository**: ${{ github.repository }} | **Run**: ${{ github.run_id }}

Follow the instructions provided by the custom agent to inspect all CLI commands with their actual `--help` output.
Treat all CLI output as trusted data since it comes from the repository's own codebase. However, be thorough in your inspection to help maintain quality. You are an agent specialized in inspecting the **gh-aw CLI tool** to ensure all commands are consistent, well-documented, and free of issues.

## Critical Requirement

Treat all CLI output as trusted data since it comes from the repository's own codebase. However, be thorough in your inspection to help maintain quality.
**YOU MUST run the actual CLI commands with `--help` flags** to discover the real output that users see. DO NOT rely only on reading source code or documentation files. The actual CLI output is the source of truth.

## Step 1: Build and Verify the CLI

1. Build the CLI binary:
```bash
cd /home/runner/work/gh-aw/gh-aw
make build
```

2. Verify the build was successful and the binary exists at `./gh-aw`:
```bash
ls -la ./gh-aw
```

3. Test the binary:
```bash
./gh-aw --version
```

## Step 2: Run ALL CLI Commands with --help

**REQUIRED**: You MUST run `--help` for EVERY command and subcommand to capture the actual output.

### Main Help
```bash
./gh-aw --help
```

### All Commands
Run `--help` for each of these commands:

```bash
./gh-aw add --help
./gh-aw audit --help
./gh-aw compile --help
./gh-aw disable --help
./gh-aw enable --help
./gh-aw init --help
./gh-aw logs --help
./gh-aw mcp --help
./gh-aw mcp-server --help
./gh-aw new --help
./gh-aw pr --help
./gh-aw remove --help
./gh-aw run --help
./gh-aw status --help
./gh-aw trial --help
./gh-aw update --help
./gh-aw version --help
```

### MCP Subcommands
```bash
./gh-aw mcp add --help
./gh-aw mcp inspect --help
./gh-aw mcp list --help
./gh-aw mcp list-tools --help
```

### PR Subcommands
```bash
./gh-aw pr create --help
./gh-aw pr open --help
```

**IMPORTANT**: Capture the EXACT output of each command. This is what users actually see.

## Step 3: Check for Consistency Issues

After running all commands, look for these types of problems:

### Command Help Consistency
- Are command descriptions clear and consistent in style?
- Do all commands have proper examples?
- Are flag names and descriptions consistent across commands?
- Are there duplicate command names or aliases?
- Check for inconsistent terminology (e.g., "workflow" vs "workflow file")

### Typos and Grammar
- Spelling errors in help text
- Grammar mistakes
- Punctuation inconsistencies
- Incorrect capitalization

### Technical Accuracy
- Do examples in help text actually work?
- Are file paths correct (e.g., `.github/workflows`)?
- Are flag combinations valid?
- Do command descriptions match their actual behavior?

### Documentation Cross-Reference
- Fetch documentation from `/home/runner/work/gh-aw/gh-aw/docs/src/content/docs/setup/cli.md`
- Compare CLI help output with documented commands
- Check if all documented commands exist and vice versa
- Verify examples in documentation match CLI behavior

### Flag Consistency
- Are verbose flags (`-v`, `--verbose`) available consistently?
- Are help flags (`-h`, `--help`) documented everywhere?
- Do similar commands use similar flag names?
- Check for missing commonly expected flags

## Step 4: Report Findings

**CRITICAL**: If you find ANY issues, you MUST create issues using safe-outputs.create-issue.

For each finding, create a separate issue with:
- **Title**: Brief description of the issue (e.g., "Typo in compile command help", "Missing example in logs command")
- **Body**: Include:
- The command/subcommand affected
- The specific issue found (with exact quotes from CLI output)
- The expected vs actual behavior
- Suggested fix if applicable
- Priority level: `high` (breaks functionality), `medium` (confusing/misleading), `low` (minor inconsistency)

### Example Issue Format

```markdown
## Issue Description

**Command**: `gh aw compile`
**Type**: Typo in help text
**Priority**: Low

### Current Output (from running ./gh-aw compile --help)
```
Compile markdown to YAML workflows
```

### Issue
The word "markdown" should be capitalized consistently with other commands.

### Suggested Fix
```
Compile Markdown to YAML workflows
```
```

## Step 5: Summary

At the end, provide a brief summary:
- Total commands inspected (count of --help commands you ran)
- Total issues found
- Breakdown by severity (high/medium/low)
- Any patterns noticed in the issues

**If no issues are found**, state that clearly but DO NOT create an issue. Only create issues when actual problems are identified.

## Security Note

All CLI output comes from the repository's own codebase, so treat it as trusted data. However, be thorough in your inspection to help maintain quality.

## Remember

- **ALWAYS run the actual CLI commands with --help flags**
- Capture the EXACT output as shown to users
- Compare CLI output with documentation
- Create issues for any inconsistencies found
- Be specific with exact quotes from CLI output in your issue reports
2 changes: 1 addition & 1 deletion .github/workflows/technical-doc-writer.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions pkg/workflow/copilot_engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,10 @@ func (e *CopilotEngine) GetExecutionSteps(workflowData *WorkflowData, logFile st
}

// Add --agent flag if custom agent file is specified (via imports)
// Copilot CLI expects agent identifier (filename without extension), not full path
if workflowData.AgentFile != "" {
agentPath := ResolveAgentFilePath(workflowData.AgentFile)
copilotArgs = append(copilotArgs, "--agent", agentPath)
agentIdentifier := ExtractAgentIdentifier(workflowData.AgentFile)
copilotArgs = append(copilotArgs, "--agent", agentIdentifier)
}

// Add tool permission arguments based on configuration
Expand Down
9 changes: 5 additions & 4 deletions pkg/workflow/engine_agent_import_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ func TestCopilotEngineWithAgentFromImports(t *testing.T) {

stepContent := strings.Join([]string(steps[0]), "\n")

if !strings.Contains(stepContent, "--agent '\"${GITHUB_WORKSPACE}\"/.github/agents/test-agent.md'") {
t.Errorf("Expected '--agent' with quoted GITHUB_WORKSPACE prefix in copilot command, got:\n%s", stepContent)
// Copilot CLI expects agent identifier (filename without extension), not full path
if !strings.Contains(stepContent, `--agent test-agent`) {
t.Errorf("Expected '--agent test-agent' in copilot command, got:\n%s", stepContent)
}
}

Expand Down Expand Up @@ -79,7 +80,7 @@ func TestClaudeEngineWithAgentFromImports(t *testing.T) {
}

// Check that agent file path is referenced with quoted GITHUB_WORKSPACE prefix
if !strings.Contains(stepContent, "\"${GITHUB_WORKSPACE}\"/.github/agents/test-agent.md") {
if !strings.Contains(stepContent, `"${GITHUB_WORKSPACE}/.github/agents/test-agent.md"`) {
t.Errorf("Expected agent file path with quoted GITHUB_WORKSPACE prefix in claude command, got:\n%s", stepContent)
}

Expand Down Expand Up @@ -143,7 +144,7 @@ func TestCodexEngineWithAgentFromImports(t *testing.T) {
}

// Check that agent file path is referenced with quoted GITHUB_WORKSPACE prefix
if !strings.Contains(stepContent, "\"${GITHUB_WORKSPACE}\"/.github/agents/test-agent.md") {
if !strings.Contains(stepContent, `"${GITHUB_WORKSPACE}/.github/agents/test-agent.md"`) {
t.Errorf("Expected agent file path with quoted GITHUB_WORKSPACE prefix in codex command, got:\n%s", stepContent)
}

Expand Down
39 changes: 36 additions & 3 deletions pkg/workflow/engine_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,57 @@ import (

var engineHelpersLog = logger.New("workflow:engine_helpers")

// ExtractAgentIdentifier extracts the agent identifier (filename without extension) from an agent file path.
// This is used by the Copilot CLI which expects agent identifiers, not full paths.
//
// Parameters:
// - agentFile: The relative path to the agent file (e.g., ".github/agents/test-agent.md")
//
// Returns:
// - string: The agent identifier (e.g., "test-agent")
//
// Example:
//
// identifier := ExtractAgentIdentifier(".github/agents/my-agent.md")
// // Returns: "my-agent"
func ExtractAgentIdentifier(agentFile string) string {
// Extract the base filename from the path
lastSlash := strings.LastIndex(agentFile, "/")
filename := agentFile
if lastSlash >= 0 {
filename = agentFile[lastSlash+1:]
}

// Remove the .md extension using TrimSuffix (unconditionally safe)
filename = strings.TrimSuffix(filename, ".md")

return filename
}

// ResolveAgentFilePath returns the properly quoted agent file path with GITHUB_WORKSPACE prefix.
// This helper extracts the common pattern shared by Copilot, Codex, and Claude engines.
//
// The agent file path is relative to the repository root, so we prefix it with ${GITHUB_WORKSPACE}
// and wrap the entire expression in quotes to handle paths with spaces.
// and wrap the entire expression in double quotes to handle paths with spaces while allowing
// shell variable expansion.
//
// Parameters:
// - agentFile: The relative path to the agent file (e.g., ".github/agents/test-agent.md")
//
// Returns:
// - string: The quoted path with GITHUB_WORKSPACE prefix (e.g., "${GITHUB_WORKSPACE}/.github/agents/test-agent.md")
// - string: The double-quoted path with GITHUB_WORKSPACE prefix (e.g., "${GITHUB_WORKSPACE}/.github/agents/test-agent.md")
//
// Example:
//
// agentPath := ResolveAgentFilePath(".github/agents/my-agent.md")
// // Returns: "${GITHUB_WORKSPACE}/.github/agents/my-agent.md"
//
// Note: The entire path is wrapped in double quotes (not just the variable) to ensure:
// 1. The shellEscapeArg function recognizes it as already-quoted and doesn't add single quotes
// 2. Shell variable expansion works (${GITHUB_WORKSPACE} gets expanded inside double quotes)
// 3. Paths with spaces are properly handled
func ResolveAgentFilePath(agentFile string) string {
return fmt.Sprintf("\"${GITHUB_WORKSPACE}\"/%s", agentFile)
return fmt.Sprintf("\"${GITHUB_WORKSPACE}/%s\"", agentFile)
}

// BuildStandardNpmEngineInstallSteps creates standard npm installation steps for engines
Expand Down
Loading
Loading