Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/aw/create-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow
- Do not create or edit workflow files.
- Return a compact recommendation covering trigger, any scoped `paths:` filters for file-event triggers, read tools, safe outputs, permissions, and explicit `noop` criteria.
- For recurring reports or digests, always include the report window, grouping dimensions, and deduplication key. See [triggers.md](triggers.md) for key-format examples.
- If custom-agent invocation is unavailable in the environment, use fallback evaluation mode: provide the recommendation directly from local gh-aw guidance, record scoring/response quality as unavailable, and continue with partial-results reporting.
- Exit ad hoc evaluation mode only when the user explicitly asks to create, implement, or write the workflow file.
- End by offering to turn the recommendation into `.github/workflows/<workflow-id>.md` if the user wants to proceed.

Expand Down
1 change: 1 addition & 0 deletions .github/aw/github-agentic-workflows.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .github/workflows/agent-persona-explorer.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 5 additions & 3 deletions .github/workflows/agent-persona-explorer.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ Store all scenarios in cache memory.
- Prompt clarity
- Completeness
- Notable patterns or issues (be concise)
- If invocation fails: mark the scenario as `invocation_unavailable`, set quality scoring to `N/A`, and continue.

**Assessment questions:** Does the suggestion include appropriate triggers (`on:`)? Correct tools (github, web-fetch, playwright, etc.)? Proper safe-outputs? Security best practices (minimal permissions, network restrictions)? A clear, actionable prompt?

Expand All @@ -161,7 +162,7 @@ For each selected scenario, invoke the "agentic-workflows" custom agent tool, pr
- You are ONLY testing the agent's responses, NOT creating actual workflows
- **Keep responses focused and concise** - summarize findings instead of verbose descriptions
- Aim for quality over quantity - fewer well-analyzed scenarios are better than many shallow ones
- **If any tool call fails, record the error briefly and move on to the next scenario** - do NOT retry or get stuck
- **If any tool call fails, record the error briefly, mark scoring as unavailable for that scenario, and move on to the next scenario** - do NOT retry or get stuck

## Phase 4: Analyze Results (4 minutes)

Expand All @@ -175,6 +176,7 @@ Review all captured responses and identify:
### Quality Insights (summarize briefly)
- Which scenarios received the best responses (average score > 4)?
- Which scenarios received weak responses (average score < 3)?
- If scenario invocation failed, note that scoring is unavailable for affected scenarios and exclude them from the numeric average.

### Potential Issues (only list critical issues)
- Does the agent ever suggest insecure configurations?
Expand All @@ -189,7 +191,7 @@ Review all captured responses and identify:

**MANDATORY OUTPUT**: Regardless of how many phases completed successfully, you MUST call either the `create issue` or the `noop` safe-output tool before finishing. Failing to call a safe-output tool is the most common cause of workflow failures.

Create a GitHub issue with a **concise** summary report. Use the `create issue` safe-output to publish your findings. Even if only 1-2 scenarios were tested, create the issue with partial results.
Create a GitHub issue with a **concise** summary report. Use the `create issue` safe-output to publish your findings. Even if only 1-2 scenarios were tested, create the issue with partial results. Treat invocation failures as a standard partial-results outcome and explicitly mark scoring as unavailable where applicable.

**Issue title**: "Agent Persona Exploration - [DATE]" (e.g., "Agent Persona Exploration - 2024-01-16")

Expand Down Expand Up @@ -220,7 +222,7 @@ Example:
- **Agent**: [name]
- **Personas This Run**: [3 persona names]
- **Scenarios Tested**: [count - should be 3-4, selected from the 6 generated in Phase 2 (2 per persona × 3 personas)]
- **Average Quality Score**: [X.X/5.0]
- **Average Quality Score**: [X.X/5.0 or N/A when invocation/scoring is unavailable]

### Key Findings (3-5 bullet points max)
[High-level insights - keep concise]
Expand Down