[prompt-clustering] Copilot Agent Prompt Clustering — Daily Report (2026-07-01)

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-06-09 → 2026-07-01)
Total PRs Analyzed: 1,000 Copilot-authored PRs in github/gh-aw
Clusters Identified: 11 (K-means on TF-IDF of title+body, silhouette 0.028)
Overall Merge Success Rate: 82.4% (705 merged / 856 decided) — up from ~80.7% a week ago

State split: 705 merged · 151 closed · 144 open. Merge rate is computed over decided PRs (merged + closed); the 144 still-open PRs are excluded from rates. Iteration proxy = commits per PR (avg 3.75), since per-run turn logs were not available offline.

Full Analysis Report

General Insights

• Most common task type: step / command / job — CI/workflow-step plumbing (190 PRs, 19% of volume), merging at a middling 80%.
• Highest success: domains / firewall dependency & firewall bumps (98%) and sous / chef refactor/consolidation work (92%) — mechanical, well-scoped changes.
• Lowest success: mcp / tool / server MCP-tooling changes (69%, 13 pts below overall) — the clearest quality gap this period.
• Most iteration-heavy: domains / firewall (6.6 commits) and sous / chef (5.5 commits) — high commit counts but high merge rates, i.e. iteration converges here.
• WIP drag: 37 titles marked [WIP] merged at only 37.9% vs 82.4% overall — WIP PRs remain the single largest predictor of non-merge.

Trend vs. prior runs

Overall merge rate has been remarkably stable in the 79–82% band across June, ticking up to 82.4% today. The mcp/tool and workflows/report clusters are the recurring laggards; domains/firewall and sous chef refactors are consistently the top performers.

Success Rate by Cluster

Cluster (top keywords)	Tasks	Success	Avg Commits	Open	Example PRs
domains / firewall / blocked	43	98%	6.63	2	#41129, #39772, #41430
sous / chef / sous chef	59	92%	5.53	10	#41809, #42552, #39386
docs / org / github	147	87%	3.85	15	#41714, #38722, #40064
package / analyzer / linter	87	87%	3.59	19	#39054, #39773, #40248
actions job / failing github	14	85%	2.86	1	#38265, #40239, #40007
safe / schema / safe outputs	150	83%	3.65	23	#38114, #38738, #40061
model / sub / failure	84	81%	3.54	17	#41184, #38258, #42040
step / command / job	190	80%	3.63	31	#39064, #40166, #38184
skill / skills / install	33	77%	3.64	3	#42138, #42043, #42365
workflows / report / guidance	123	77%	2.83	12	#41728, #42127, #40127
mcp / tool / server	70	69%	3.20	11	#41770, #38544, #41041

Cluster notes

• step / command / job (190, 80%) — largest bucket: workflow-step wiring, env expressions, version/awf bumps. High volume + average success → the biggest absolute source of closed PRs. Worth splitting further next iteration.
• safe / schema / safe outputs (150, 83%) — safe-output schema/validator changes; solid, at-overall success.
• docs / org / github (147, 87%) — docs, permission scopes, org config. Reliable, low-risk.
• workflows / report / guidance (123, 77%) — agentic-workflow prompt/report tuning; lowest commit count (2.83) but below-average merge, suggesting these get abandoned rather than iterated.
• mcp / tool / server (70, 69%) — MCP server/tool-denial fixes; the quality gap. Likely under-specified or blocked on external tool behavior.
• domains / firewall (43, 98%) — dependency/firewall version bumps; near-perfect, just commit-heavy (retry loops on smoke tests).

Key Findings

1. MCP-tooling tasks are the weak spot — the mcp/tool/server cluster merges at 69%, 13 points below overall and the lowest for the 4th+ consecutive run. These tasks (tool-denial limits, MCP child-process guardrails) appear under-specified.
2. WIP PRs merge at less than half the baseline (37.9% vs 82.4%). 37 open/closed WIP PRs are dragging the pool; they behave like a distinct low-yield class.
3. Refactor/bump work is the safe zone — domains/firewall (98%) and sous chef (92%) are mechanical and converge reliably despite high commit counts.
4. Low-iteration + low-merge = abandonment signal — workflows/report has the fewest commits (2.83) yet below-average merge, indicating early abandonment rather than iteration.

Recommendations

1. Tighten MCP-tool task prompts: require reproduction steps and explicit expected tool-permission behavior before dispatch — target the 69% cluster.
2. Triage WIP PRs: WIP items rarely convert (38%). Auto-flag [WIP] PRs older than N days for close-or-promote to reduce open-PR backlog (currently 144 open).
3. Split the step/command/job mega-cluster: at 190 PRs it hides sub-patterns; a finer breakdown next run would localize the 20% failure rate.
4. Replicate refactor scoping: the tight, single-concern framing that makes sous chef/firewall tasks succeed at 90%+ should be templated for higher-risk clusters.

---

Generated by Prompt Clustering Analysis (Run: §28512163732)

References: §28512163732

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 167.2 AIC · ⌖ 17.9 AIC · ⊞ 13.2K · ◷

• expires on Jul 2, 2026, 3:15 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #42938.