From 8043fba218bd4f19bc47bfd6baa111338211edf2 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Feb 2026 05:30:51 +0000 Subject: [PATCH 1/4] Initial plan From 2bb96842e5cff7ad43ddd6f702a73aaf78dad721 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Feb 2026 05:36:34 +0000 Subject: [PATCH 2/4] feat: add test-coverage prompt and link from agentic-workflows agent Adds .github/aw/test-coverage.md with guidance that distinguishes: - Preferred: fetch pre-computed coverage artifacts via gh run download or actions toolset - Fallback: run tests when no prior artifacts exist Links the new prompt from .github/agents/agentic-workflows.agent.md. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/agents/agentic-workflows.agent.md | 11 ++ .github/aw/test-coverage.md | 206 ++++++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 .github/aw/test-coverage.md diff --git a/.github/agents/agentic-workflows.agent.md b/.github/agents/agentic-workflows.agent.md index e34a4da6123..c0f21877e1b 100644 --- a/.github/agents/agentic-workflows.agent.md +++ b/.github/agents/agentic-workflows.agent.md @@ -18,6 +18,7 @@ This is a **dispatcher agent** that routes your request to the appropriate speci - **Creating report-generating workflows**: Routes to `report` prompt — consult this whenever the workflow posts status updates, audits, analyses, or any structured output as issues, discussions, or comments - **Creating shared components**: Routes to `create-shared-agentic-workflow` prompt - **Fixing Dependabot PRs**: Routes to `dependabot` prompt — use this when Dependabot opens PRs that modify generated manifest files (`.github/workflows/package.json`, `.github/workflows/requirements.txt`, `.github/workflows/go.mod`). Never merge those PRs directly; instead update the source `.md` files and rerun `gh aw compile --dependabot` to bundle all fixes +- **Analyzing test coverage**: Routes to `test-coverage` prompt — consult this whenever the workflow reads, analyzes, or reports on test coverage data from PRs or CI runs Workflows may optionally include: @@ -118,6 +119,16 @@ When you interact with this agent, it will: - "Bundle and close the Dependabot PRs for workflow dependencies" - "Update @playwright/test to fix the Dependabot PR" +### Analyze Test Coverage +**Load when**: The workflow reads, analyzes, or reports test coverage — whether triggered by a PR, a schedule, or a slash command. Always consult this prompt before designing the coverage data strategy. + +**Prompt file**: https://github.com/github/gh-aw/blob/main/.github/aw/test-coverage.md + +**Use cases**: +- "Create a workflow that comments coverage on PRs" +- "Analyze coverage trends over time" +- "Add a coverage gate that blocks PRs below a threshold" + ## Instructions When a user interacts with you: diff --git a/.github/aw/test-coverage.md b/.github/aw/test-coverage.md new file mode 100644 index 00000000000..39d8217d95a --- /dev/null +++ b/.github/aw/test-coverage.md @@ -0,0 +1,206 @@ +--- +description: Guidance for creating agentic workflows that analyze test coverage — prefer reading pre-computed CI artifacts over re-running tests. +--- + +# Test Coverage Workflow Guidance + +Consult this file when creating or updating an agentic workflow that analyzes test coverage (e.g., coverage PR analyzers, coverage trend reporters, coverage gate enforcers). + +## Core Principle: Read Artifacts First + +**Always prefer reading pre-computed coverage artifacts from a previous CI run over re-running the full test suite inside the workflow.** + +Re-running tests is slow, resource-intensive, and duplicates work that CI has already done. Coverage data is almost always available as a CI artifact from the same commit or PR. + +## Two Patterns: Preferred vs Fallback + +### ✅ Preferred: Fetch Pre-Computed Coverage Artifacts + +Use this when a CI workflow already runs tests and uploads coverage reports as artifacts. + +**Using `gh run download`** (bash tool): + +```bash +# Find the latest successful CI run for this commit/branch +gh run list --branch "$BRANCH" --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId' + +# Download the coverage artifact +gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage + +# Parse coverage data (format varies: lcov, JSON, XML, plain text) +cat /tmp/coverage/coverage-summary.json +``` + +**Using the `actions` toolset** (MCP tool): + +```yaml +tools: + github: + toolsets: [actions] +``` + +Then in the prompt: +``` +Use the `list_workflow_run_artifacts` tool to find the coverage artifact from the latest +CI run on this PR's head commit. Download and parse it using `download_workflow_run_artifact`. +``` + +### ⚠️ Fallback: Run Tests to Compute Fresh Coverage + +Use this **only when**: +- No prior CI run exists for this commit (e.g., first push on a brand-new branch) +- The existing CI does not upload coverage artifacts +- The user explicitly requests fresh coverage data + +```bash +# Example: run tests with coverage (language-specific) +# Node.js +npx jest --coverage --coverageReporters=json-summary + +# Python +python -m pytest --cov=src --cov-report=json + +# Go +go test ./... -coverprofile=/tmp/coverage.out +go tool cover -func=/tmp/coverage.out +``` + +When using the fallback, always inform the user that no prior coverage artifact was found and tests are being re-run. + +## Detecting Coverage Artifact Availability + +Before falling back to running tests, the agent should: + +1. **Check for prior CI runs** on the same commit or branch head: + ```bash + gh run list --commit "$HEAD_SHA" --status success --json databaseId,workflowName + ``` + +2. **Check for coverage artifacts** on found runs: + ```bash + gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage 2>/dev/null \ + && echo "artifact found" || echo "no artifact" + ``` + +3. **Fall back gracefully** if no artifact is found. + +## Recommended Decision Logic for the Agent Prompt + +Include this decision logic in the workflow prompt body: + +``` +## Coverage Data Strategy + +1. First, look for a pre-computed coverage artifact from the latest successful CI run + on this PR's head commit using `gh run download --name `. +2. If an artifact is found, parse and analyze it directly — do NOT re-run tests. +3. If no artifact is found, run the test suite with coverage enabled and note in + your report that coverage was computed fresh (not from CI artifacts). +``` + +## Frontmatter Configuration + +Coverage analysis workflows typically need: + +```yaml +engine: copilot +triggers: + pull_request: + types: [opened, synchronize] +permissions: + pull-requests: write # to post coverage comment + actions: read # to download artifacts +network: + defaults: true + # Add language ecosystem if running tests as fallback: + # egosystems: [node] # or python, go, etc. +tools: + github: + toolsets: [default, actions] +safe-outputs: + add-comment: + hide-older-comments: true # replace previous coverage comment +``` + +## Common Coverage Report Formats + +| Tool | Artifact format | Key file | +|---|---|---| +| Jest (JS) | `coverage-report` | `coverage-summary.json` | +| Istanbul/nyc | `coverage-report` | `coverage-summary.json` | +| pytest-cov | `coverage-report` | `coverage.json` | +| Go cover | `coverage-report` | `coverage.out` | +| Cobertura (XML) | `coverage-report` | `coverage.xml` | +| lcov | `coverage-report` | `lcov.info` | +| simplecov (Ruby) | `coverage-report` | `.last_run.json` | + +## Example: Coverage PR Analyzer + +Below is a minimal example workflow that reads coverage artifacts and posts a summary comment: + +```markdown +--- +engine: copilot +triggers: + pull_request: + types: [opened, synchronize] +permissions: + pull-requests: write + actions: read +network: + defaults: true +tools: + github: + toolsets: [default, actions] +safe-outputs: + add-comment: + hide-older-comments: true +--- + +Analyze test coverage for this pull request. + +## Coverage Data Strategy + +1. Find the latest successful CI run for the PR's head commit: + `gh run list --commit "${{ github.event.pull_request.head.sha }}" --status success --limit 5 --json databaseId,workflowName` + +2. Download the coverage artifact (try common names: `coverage-report`, `coverage`, `test-results`): + `gh run download --name coverage-report --dir /tmp/coverage` + +3. If a coverage artifact is found, parse it and report coverage metrics. + **Do NOT re-run tests** — use the existing CI data. + +4. If no artifact is found, run the test suite with coverage enabled and note + that coverage was computed fresh. + +## Report Format + +Post a comment on the PR with: +- Overall coverage percentage (and delta vs base branch if determinable) +- Files with decreased coverage (⚠️) +- Files with increased coverage (✅) +- Uncovered lines for changed files +``` + +## Anti-Patterns to Avoid + +❌ **Never default to re-running tests** without first checking for artifacts: +``` +# BAD: Always re-runs tests even when CI already computed coverage +npm test -- --coverage +``` + +❌ **Never ignore the `actions` toolset** when coverage data is available in GitHub: +```yaml +# BAD: Missing actions toolset +tools: + github: + toolsets: [default] +``` + +✅ **Always check for artifacts first**, then fall back: +``` +# GOOD: Check artifact → fallback to test run only if needed +gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage \ + && analyze_artifact || run_tests_with_coverage +``` From 7271a4b2493288d4ac4f8c9da93fff46fccef4d0 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Feb 2026 05:45:36 +0000 Subject: [PATCH 3/4] =?UTF-8?q?refactor:=20trim=20test-coverage.md=20for?= =?UTF-8?q?=20token=20efficiency=20(206=E2=86=9254=20lines)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/agents/agentic-workflows.agent.md | 11 -- .github/aw/test-coverage.md | 194 +++------------------- 2 files changed, 21 insertions(+), 184 deletions(-) diff --git a/.github/agents/agentic-workflows.agent.md b/.github/agents/agentic-workflows.agent.md index c0f21877e1b..e34a4da6123 100644 --- a/.github/agents/agentic-workflows.agent.md +++ b/.github/agents/agentic-workflows.agent.md @@ -18,7 +18,6 @@ This is a **dispatcher agent** that routes your request to the appropriate speci - **Creating report-generating workflows**: Routes to `report` prompt — consult this whenever the workflow posts status updates, audits, analyses, or any structured output as issues, discussions, or comments - **Creating shared components**: Routes to `create-shared-agentic-workflow` prompt - **Fixing Dependabot PRs**: Routes to `dependabot` prompt — use this when Dependabot opens PRs that modify generated manifest files (`.github/workflows/package.json`, `.github/workflows/requirements.txt`, `.github/workflows/go.mod`). Never merge those PRs directly; instead update the source `.md` files and rerun `gh aw compile --dependabot` to bundle all fixes -- **Analyzing test coverage**: Routes to `test-coverage` prompt — consult this whenever the workflow reads, analyzes, or reports on test coverage data from PRs or CI runs Workflows may optionally include: @@ -119,16 +118,6 @@ When you interact with this agent, it will: - "Bundle and close the Dependabot PRs for workflow dependencies" - "Update @playwright/test to fix the Dependabot PR" -### Analyze Test Coverage -**Load when**: The workflow reads, analyzes, or reports test coverage — whether triggered by a PR, a schedule, or a slash command. Always consult this prompt before designing the coverage data strategy. - -**Prompt file**: https://github.com/github/gh-aw/blob/main/.github/aw/test-coverage.md - -**Use cases**: -- "Create a workflow that comments coverage on PRs" -- "Analyze coverage trends over time" -- "Add a coverage gate that blocks PRs below a threshold" - ## Instructions When a user interacts with you: diff --git a/.github/aw/test-coverage.md b/.github/aw/test-coverage.md index 39d8217d95a..4c04310a19c 100644 --- a/.github/aw/test-coverage.md +++ b/.github/aw/test-coverage.md @@ -4,103 +4,26 @@ description: Guidance for creating agentic workflows that analyze test coverage # Test Coverage Workflow Guidance -Consult this file when creating or updating an agentic workflow that analyzes test coverage (e.g., coverage PR analyzers, coverage trend reporters, coverage gate enforcers). +Consult this file when creating or updating a workflow that analyzes test coverage. ## Core Principle: Read Artifacts First -**Always prefer reading pre-computed coverage artifacts from a previous CI run over re-running the full test suite inside the workflow.** +**Always prefer fetching pre-computed coverage artifacts from CI over re-running the test suite.** Re-running tests is slow and duplicates work CI has already done. -Re-running tests is slow, resource-intensive, and duplicates work that CI has already done. Coverage data is almost always available as a CI artifact from the same commit or PR. - -## Two Patterns: Preferred vs Fallback - -### ✅ Preferred: Fetch Pre-Computed Coverage Artifacts - -Use this when a CI workflow already runs tests and uploads coverage reports as artifacts. - -**Using `gh run download`** (bash tool): - -```bash -# Find the latest successful CI run for this commit/branch -gh run list --branch "$BRANCH" --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId' - -# Download the coverage artifact -gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage - -# Parse coverage data (format varies: lcov, JSON, XML, plain text) -cat /tmp/coverage/coverage-summary.json -``` - -**Using the `actions` toolset** (MCP tool): - -```yaml -tools: - github: - toolsets: [actions] -``` - -Then in the prompt: -``` -Use the `list_workflow_run_artifacts` tool to find the coverage artifact from the latest -CI run on this PR's head commit. Download and parse it using `download_workflow_run_artifact`. -``` - -### ⚠️ Fallback: Run Tests to Compute Fresh Coverage - -Use this **only when**: -- No prior CI run exists for this commit (e.g., first push on a brand-new branch) -- The existing CI does not upload coverage artifacts -- The user explicitly requests fresh coverage data - -```bash -# Example: run tests with coverage (language-specific) -# Node.js -npx jest --coverage --coverageReporters=json-summary - -# Python -python -m pytest --cov=src --cov-report=json - -# Go -go test ./... -coverprofile=/tmp/coverage.out -go tool cover -func=/tmp/coverage.out -``` - -When using the fallback, always inform the user that no prior coverage artifact was found and tests are being re-run. - -## Detecting Coverage Artifact Availability - -Before falling back to running tests, the agent should: - -1. **Check for prior CI runs** on the same commit or branch head: - ```bash - gh run list --commit "$HEAD_SHA" --status success --json databaseId,workflowName - ``` - -2. **Check for coverage artifacts** on found runs: - ```bash - gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage 2>/dev/null \ - && echo "artifact found" || echo "no artifact" - ``` - -3. **Fall back gracefully** if no artifact is found. - -## Recommended Decision Logic for the Agent Prompt +## Coverage Data Strategy -Include this decision logic in the workflow prompt body: +Include this decision block in every coverage workflow prompt: ``` -## Coverage Data Strategy - -1. First, look for a pre-computed coverage artifact from the latest successful CI run - on this PR's head commit using `gh run download --name `. -2. If an artifact is found, parse and analyze it directly — do NOT re-run tests. -3. If no artifact is found, run the test suite with coverage enabled and note in - your report that coverage was computed fresh (not from CI artifacts). +1. Find the latest successful CI run for this commit: + `gh run list --commit "$HEAD_SHA" --status success --limit 5 --json databaseId,workflowName` +2. Download the coverage artifact (try names: coverage-report, coverage, test-results): + `gh run download --name coverage-report --dir /tmp/coverage` +3. If found, parse and analyze it — do NOT re-run tests. +4. If not found, run tests with coverage and note in the report that data was computed fresh. ``` -## Frontmatter Configuration - -Coverage analysis workflows typically need: +## Frontmatter Template ```yaml engine: copilot @@ -108,99 +31,24 @@ triggers: pull_request: types: [opened, synchronize] permissions: - pull-requests: write # to post coverage comment - actions: read # to download artifacts + pull-requests: write # post coverage comment + actions: read # download artifacts network: defaults: true - # Add language ecosystem if running tests as fallback: - # egosystems: [node] # or python, go, etc. tools: github: - toolsets: [default, actions] -safe-outputs: - add-comment: - hide-older-comments: true # replace previous coverage comment -``` - -## Common Coverage Report Formats - -| Tool | Artifact format | Key file | -|---|---|---| -| Jest (JS) | `coverage-report` | `coverage-summary.json` | -| Istanbul/nyc | `coverage-report` | `coverage-summary.json` | -| pytest-cov | `coverage-report` | `coverage.json` | -| Go cover | `coverage-report` | `coverage.out` | -| Cobertura (XML) | `coverage-report` | `coverage.xml` | -| lcov | `coverage-report` | `lcov.info` | -| simplecov (Ruby) | `coverage-report` | `.last_run.json` | - -## Example: Coverage PR Analyzer - -Below is a minimal example workflow that reads coverage artifacts and posts a summary comment: - -```markdown ---- -engine: copilot -triggers: - pull_request: - types: [opened, synchronize] -permissions: - pull-requests: write - actions: read -network: - defaults: true -tools: - github: - toolsets: [default, actions] + toolsets: [default, actions] # actions toolset enables artifact download safe-outputs: add-comment: hide-older-comments: true ---- - -Analyze test coverage for this pull request. - -## Coverage Data Strategy - -1. Find the latest successful CI run for the PR's head commit: - `gh run list --commit "${{ github.event.pull_request.head.sha }}" --status success --limit 5 --json databaseId,workflowName` - -2. Download the coverage artifact (try common names: `coverage-report`, `coverage`, `test-results`): - `gh run download --name coverage-report --dir /tmp/coverage` - -3. If a coverage artifact is found, parse it and report coverage metrics. - **Do NOT re-run tests** — use the existing CI data. - -4. If no artifact is found, run the test suite with coverage enabled and note - that coverage was computed fresh. - -## Report Format - -Post a comment on the PR with: -- Overall coverage percentage (and delta vs base branch if determinable) -- Files with decreased coverage (⚠️) -- Files with increased coverage (✅) -- Uncovered lines for changed files ``` -## Anti-Patterns to Avoid +## Fallback: Run Tests -❌ **Never default to re-running tests** without first checking for artifacts: -``` -# BAD: Always re-runs tests even when CI already computed coverage -npm test -- --coverage -``` +Use **only when** no prior CI artifact exists or CI doesn't upload coverage. Supported commands: -❌ **Never ignore the `actions` toolset** when coverage data is available in GitHub: -```yaml -# BAD: Missing actions toolset -tools: - github: - toolsets: [default] -``` - -✅ **Always check for artifacts first**, then fall back: -``` -# GOOD: Check artifact → fallback to test run only if needed -gh run download "$RUN_ID" --name coverage-report --dir /tmp/coverage \ - && analyze_artifact || run_tests_with_coverage -``` +| Language | Command | +|---|---| +| Node.js | `npx jest --coverage --coverageReporters=json-summary` | +| Python | `python -m pytest --cov=src --cov-report=json` | +| Go | `go test ./... -coverprofile=/tmp/coverage.out` | From da6ed5c6b4cc59e6f402be52f806bbaf80e9fb7c Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Feb 2026 06:00:06 +0000 Subject: [PATCH 4/4] fix: restore test-coverage routing in agent file, fix on/network schema Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/agents/agentic-workflows.agent.md | 11 +++++++++++ .github/aw/test-coverage.md | 5 ++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/.github/agents/agentic-workflows.agent.md b/.github/agents/agentic-workflows.agent.md index e34a4da6123..c0f21877e1b 100644 --- a/.github/agents/agentic-workflows.agent.md +++ b/.github/agents/agentic-workflows.agent.md @@ -18,6 +18,7 @@ This is a **dispatcher agent** that routes your request to the appropriate speci - **Creating report-generating workflows**: Routes to `report` prompt — consult this whenever the workflow posts status updates, audits, analyses, or any structured output as issues, discussions, or comments - **Creating shared components**: Routes to `create-shared-agentic-workflow` prompt - **Fixing Dependabot PRs**: Routes to `dependabot` prompt — use this when Dependabot opens PRs that modify generated manifest files (`.github/workflows/package.json`, `.github/workflows/requirements.txt`, `.github/workflows/go.mod`). Never merge those PRs directly; instead update the source `.md` files and rerun `gh aw compile --dependabot` to bundle all fixes +- **Analyzing test coverage**: Routes to `test-coverage` prompt — consult this whenever the workflow reads, analyzes, or reports on test coverage data from PRs or CI runs Workflows may optionally include: @@ -118,6 +119,16 @@ When you interact with this agent, it will: - "Bundle and close the Dependabot PRs for workflow dependencies" - "Update @playwright/test to fix the Dependabot PR" +### Analyze Test Coverage +**Load when**: The workflow reads, analyzes, or reports test coverage — whether triggered by a PR, a schedule, or a slash command. Always consult this prompt before designing the coverage data strategy. + +**Prompt file**: https://github.com/github/gh-aw/blob/main/.github/aw/test-coverage.md + +**Use cases**: +- "Create a workflow that comments coverage on PRs" +- "Analyze coverage trends over time" +- "Add a coverage gate that blocks PRs below a threshold" + ## Instructions When a user interacts with you: diff --git a/.github/aw/test-coverage.md b/.github/aw/test-coverage.md index 4c04310a19c..19e476897c1 100644 --- a/.github/aw/test-coverage.md +++ b/.github/aw/test-coverage.md @@ -27,14 +27,13 @@ Include this decision block in every coverage workflow prompt: ```yaml engine: copilot -triggers: +on: pull_request: types: [opened, synchronize] permissions: pull-requests: write # post coverage comment actions: read # download artifacts -network: - defaults: true +network: defaults tools: github: toolsets: [default, actions] # actions toolset enables artifact download