[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-02 (1,000 PRs, 80.8% merged)

[github-actions[bot]]

Daily NLP clustering of Copilot coding-agent PRs in github/gh-aw. TF-IDF + K-means over task prompts extracted from PR titles/bodies, enriched with outcome and iteration metrics.

Summary

• Analysis period: last 30 days (2026-06-09 → 2026-07-02)
• Tasks analyzed: 1,000 Copilot PRs — 808 merged, 182 closed, 10 open
• Overall merge (success) rate: 80.8% (▼ 1.6 pp vs 82.4% on 2026-07-01)
• Clusters identified: 11 (silhouette 0.029 — expected-low for short technical prompts)
• Avg commits / PR (iteration proxy): 3.73

Key findings

1. Merge rate held above the 80% line but slipped from yesterday's local peak — 80.8% vs 82.4%. Still within the ~79–82% band the repo has run at all month; no structural regression, just reversion toward the mean.
2. Small, well-scoped changes merge best. The two highest-merging clusters — dashboard / ts (87.2%) and pkg / linters / string (87.1%) — are also the smallest by diff size (6–16 files avg). Tight, single-purpose PRs are the reliable path.
3. Config/docs/workflow-authoring is the soft spot. The md / workflow / workflows cluster (182 PRs, 18% of all work) merges at only 74.7% — the lowest and second-largest cluster. Prompts touching .md, lock.yml, and workflow definitions are the biggest drag on the overall rate.
4. Big migrations cost iterations. The claude / smoke-test cluster averages 5.88 commits and 181 files / 2,272 additions per PR — large threat-detect/usage-artifact migrations that need the most back-and-forth (though they still land at 86%).
5. "WIP"-titled PRs are a strong failure signal. 31 PRs carried a WIP marker and merged at only 41.9% — roughly half the baseline. WIP titles reliably flag work that ends up abandoned or superseded.

Cluster analysis

#	Theme (top terms)	PRs	%	Merge %	Avg commits	Avg files
2	gh / gh aw / aw · env · step	234	23.4%	81.2%	3.97	29.7
6	md / workflow / workflows · lock · yml	182	18.2%	74.7%	2.60	18.9
5	safe / safe outputs / cjs · handler	142	14.2%	78.9%	3.66	11.1
0	pkg / linters / string · analyzer · cli	101	10.1%	87.1%	3.74	16.2
3	chef / sous-chef · refactor	72	7.2%	86.1%	5.58	12.0
4	claude / smoke-claude · migration	59	5.9%	86.4%	5.88	181.2
7	tool / sdk / denial · copilot-sdk	55	5.5%	78.2%	3.02	12.1
8	awf / sandbox / sudo · firewall	52	5.2%	75.0%	3.19	70.4
9	dashboard / ts / catch · extension	47	4.7%	87.2%	3.21	6.0
10	permissions / requests · auth	42	4.2%	81.0%	3.79	10.9
1	github-actions / job / failing	14	1.4%	85.7%	2.86	7.5

Representative PRs per cluster

[2] gh / gh aw / aw — engine & CLI plumbing

• Improve assign-to-agent diagnostics for Copilot assignee alias lookup #41047 Improve assign-to-agent diagnostics for Copilot assignee alias lookup
• copilot engine: launch from ${GITHUB_WORKSPACE} to enable APM skill discovery #41459 copilot engine: launch from ${GITHUB_WORKSPACE} to enable APM skill discovery
• Propagate resolved AI credits into failure footer context #38412 Propagate resolved AI credits into failure footer context

[6] md / workflow / workflows — docs & workflow authoring (lowest merge rate)

• Impact Efficiency Report: progressive disclosure + fix artificial/missing data + recommendations #39830 Impact Efficiency Report: progressive disclosure + fix artificial/missing data
• docs: expand CentralRepoOps with agentic factory pattern #38462 docs: expand CentralRepoOps with agentic factory pattern
• Remove imports.if and migrate experiment-driven import usage to runtime imports #38247 Remove imports.if and migrate experiment-driven import usage to runtime imports

[5] safe / safe outputs / cjs — safe-output handlers

• Migrate set_issue_type safe output from GraphQL to single REST issues.update call #41241 Migrate set_issue_type safe output from GraphQL to single REST issues.update
• Support templatable safe-outputs.staged values and GitHub expressions #41296 Support templatable safe-outputs.staged values and GitHub expressions

[0] pkg / linters / string — package/linter fixes (top merge rate)

• Fix /help routing fallthrough, error handling, reaction, and mention sanitization #40476 Fix /help routing fallthrough, error handling, reaction, and mention sanitization
• Cache compiled regexp in applySanitizePattern #41762 Cache compiled regexp in applySanitizePattern
• fix(windows): remove compat import to prevent ConPTY startup crash #41235 fix(windows): remove compat import to prevent ConPTY startup crash

[3] chef / sous-chef — refactors

• Reduce YAMLGeneration time by collapsing duplicate run-script scans #41333 Reduce YAMLGeneration time by collapsing duplicate run-script scans
• refactor(copilot-sdk): multi-provider only — remove single-provider legacy #42740 refactor(copilot-sdk): multi-provider only — remove single-provider legacy

[4] claude / smoke-claude — large migrations (most iterations, largest diffs)

• Migrate threat detection to external threat-detect binary behind feature flag #40166 Migrate threat detection to external threat-detect binary behind feature flag
• Precompute usage-artifact activity aggregates and include GitHub API rate limit data for usage-only reporting #40504 Precompute usage-artifact activity aggregates + GitHub API rate-limit data

[8] awf / sandbox / sudo — sandbox/firewall

• feat: omit sudo from generated lock.yml when network-isolation is enabled; bump firewall to v0.27.10 and mcpg to v0.3.30 #41269 omit sudo from generated lock.yml when network-isolation is enabled
• feat: detect AWF firewall startup failures and surface them in the agent failure issue #41472 detect AWF firewall startup failures and surface them in the agent failure issue

[9] dashboard / ts — dashboard extension (top merge rate)

• Add a Copilot Canvas extension for agentic workflow operations #42137 Add a Copilot Canvas extension for agentic workflow operations
• Detect missing gh aw in dashboard and surface install guidance #42270 Detect missing gh aw in dashboard and surface install guidance

[1] github-actions / failing — CI fixes (many WIP)

• [WIP] Fix failing GitHub Actions job 'js-typecheck' #38397 [WIP] Fix failing GitHub Actions job 'js-typecheck'
• [WIP] Fix failing GitHub Actions job Integration: Workflow Features #40007 [WIP] Fix failing GitHub Actions job Integration: Workflow Features

Recommendations

1. Target the md / workflow cluster — it is 18% of all Copilot work and the lowest merger (74.7%). Docs/workflow-authoring prompts likely need tighter acceptance criteria and concrete file targets up front rather than open-ended "expand/improve" asks.
2. Split large migrations — the claude / smoke-claude cluster (5.9 commits, 181-file diffs) burns the most iterations. Decomposing multi-hundred-file migrations into staged PRs should cut turn counts and review load.
3. Treat "WIP" as a triage flag, not a task — WIP-titled PRs merge at 42%. Consider prompting the agent to resolve WIP work to a mergeable state or close it, rather than leaving long-lived WIP PRs open.
4. Keep favoring small-diff scoping — the 6–16-file clusters merge at 87%. Prompt patterns that constrain scope correlate with the best outcomes.

Methodology & limitations

• Prompt source: PR title + body (firewall-warning blocks, code fences, and HTML <details> stripped). Only 5 PRs carried the explicit START COPILOT ORIGINAL PROMPT marker, so titles/bodies are the practical prompt proxy.
• Iteration metric: commit count is used as an iteration proxy — per-run turn counts from aw_info.json are not available for Copilot coding-agent PRs (these are not gh aw engine runs), so gh-aw logs enrichment was skipped. Documented edge case.
• State freshness: merge/close state taken from today's PR search snapshot (full-data cache can be stale for recently-updated PRs).
• Clustering: TF-IDF (1–3 grams, min_df=3) → K-means; k chosen by silhouette over k∈[8,12]. Low silhouette (0.029) is normal for short, jargon-heavy prompts — clusters are thematically coherent on inspection.

References: §28583469944

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 232.7 AIC · ⌖ 18.2 AIC · ⊞ 13.2K · ◷

• expires on Jul 3, 2026, 2:59 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #43174.