[copilot-cli-research] Copilot CLI Deep Research - 2026-07-27

[github-actions]

Analysis Date: 2026-07-27
Repository: github/gh-aw
Scope: 262 total workflows, 183 effectively using Copilot engine (70%), 37 explicit + 146 default

---

📊 Executive Summary

Research Topic: Copilot CLI Optimization Opportunities
Key Findings:

1. max-tool-denials remains at 0/67 Copilot SDK workflows — a critical persistent gap across 18+ consecutive research runs
2. 73% of explicit Copilot workflows (27/37) have no evals: block for quality assurance
3. engine.token-weights, startup-timeout, and network.blocked are near-zero despite clear availability
4. 12 Copilot workflows have no network configuration; 18 have no sandbox config
5. Three custom agent files (create-safe-output-type, custom-engine-implementation, w3c-specification-writer) remain completely unused

Primary Recommendation: Add max-tool-denials: 3 to all 67 copilot-sdk workflows — this prevents infinite tool-denial loops and has been flagged as the most critical gap for 18+ research cycles with zero remediation progress.

The repository has grown to 262 workflows (70% effective Copilot usage) with MCP-scripts adoption surging from 3 → 13 workflows. However, three core protective features remain systematically absent: tool-denial circuit breakers, LLM cost guards, and MCP startup safeguards. The evals gap (73% of copilot workflows have none) also represents a significant quality assurance blind spot.

---

Critical Findings

🔴 High Priority Issues

1. max-tool-denials = 0 in ALL 67 Copilot SDK workflows (18+ consecutive runs without fix)
The default is 5 denials before aborting. None of the 67 SDK workflows override this, meaning they may enter infinite tool-permission loops burning tokens. This has been flagged every research run since May 2026 with zero remediation.

2. Evals gap: 27/37 explicit Copilot workflows (73%) have no quality assurance
Workflows like release.md, repo-tree-map.md, repository-quality-improver.md, slide-deck-maintainer.md, ubuntu-image-analyzer.md, video-analyzer.md, visual-regression-checker.md, workflow-health-manager.md, weekly-editors-health-check.md, weekly-safe-outputs-spec-review.md, update-astro.md, and super-linter.md run without any correctness validation.

3. Security gaps: 12 workflows without network config, 18 without sandbox config
Among production Copilot workflows (excluding smoke tests), these are missing both:

• stale-pr-cleanup.md, terminal-stylist.md, uk-ai-operational-resilience.md, video-analyzer.md, workflow-health-manager.md

🟡 Medium Priority Opportunities

4. engine.token-weights = 0 — No AI credit budgeting across any workflow despite feature availability. High-value for expensive workflows like release.md (20 min), repository-quality-improver.md, spec-librarian.md.

5. startup-timeout = 1 workflow — Only ruflo-backed-task.md protects against slow MCP server startups. All other MCP-using workflows risk hanging silently.

6. max-continuations missing on 19 high-timeout Copilot workflows — Workflows with 20+ min timeouts (release.md, weekly-issue-summary.md, stale-pr-cleanup.md, etc.) don't limit autopilot continuation cycles.

---

View Full Analysis

1️⃣ Current State Analysis

View Copilot CLI Capabilities Inventory

Copilot CLI Capabilities Inventory

Core Features Available (from pkg/workflow/copilot_engine*.go):

• engine.id: copilot — explicit engine selection
• engine.version — version pinning (e.g., "0.0.422")
• engine.model — model override (e.g., gpt-5, copilot/gpt-5.4)
• engine.args — custom CLI arguments (e.g., ["--add-dir", "/workspace"])
• engine.agent — custom agent file from .github/agents/
• engine.command — custom executable path
• engine.env — environment variable injection
• engine.api-target — custom API endpoint for GHEC/GHES
• engine.bare — disable context loading (--no-custom-instructions)
• engine.harness — custom harness script for retry logic
• max-continuations — Copilot-exclusive autopilot continuation limit
• --allow-all-tools / --allow-tool — granular tool permissions

Sandbox Options:

• AWF (sandbox.agent.id: awf) — network firewalling
• SRT (sandbox.agent.id: srt) — process isolation

MCP Integration:

• JSON MCP config at $HOME/.copilot/mcp-config.json
• Localhost → docker.internal rewriting when firewall enabled
• Guard policies for write-sink tools

LSP Support:

• Language server configuration via lsp: frontmatter

Available (but unused) CLI flags:

• --add-dir — add additional directories to context
• --share — conversation tracking/sharing
• --disable-builtin-mcps — disable built-in MCP servers

View Usage Statistics

Usage Statistics

Metric	Count	Notes
Total Workflows	262	Jul-27-2026
Copilot (explicit)	37	14%
Copilot (default/blank)	146	56%
Copilot Effective Total	183	70%
Copilot SDK workflows	67	Use SDK-style driver
Claude	43	16%
Codex	9	3%
With strict: true (copilot)	27/37	73% — good adoption
With evals: (copilot)	10/37	27% — major gap
With sandbox: (copilot)	19/37	51%
With network: (copilot)	25/37	68%
With max-continuations	11	All engines
With copilot-requests: write	69	Good adoption

---

2️⃣ Feature Usage Matrix

Feature Category	Available	Used	Not Used	Usage Rate
max-tool-denials	All SDK workflows	0	67	0% 🔴
engine.token-weights	All workflows	0	262	0% 🔴
startup-timeout	All MCP workflows	1	~50+	<2% 🔴
evals:	Copilot workflows	10/37	27/37	27% 🟡
sandbox:	Copilot workflows	19/37	18/37	51% 🟡
network:	Copilot workflows	25/37	12/37	68% 🟢
strict: true	Copilot workflows	27/37	10/37	73% 🟢
engine.agent	Copilot workflows	3/37	34/37	8% 🟡
max-continuations	Copilot only	11	~26	30% 🟡
engine.api-target	All workflows	0	262	0% (by design?)
network.blocked	All workflows	0*	262	0% 🟡
engine.bare	All workflows	~5	~257	~2%
mcp-scripts	All workflows	~13	~249	5%

---

3️⃣ Missed Opportunities

View High Priority Opportunities

🔴 High Priority

Opportunity 1: max-tool-denials for All Copilot SDK Workflows

• What: 67 SDK-based workflows have no protection against infinite tool-permission loops
• Why It Matters: Default max-tool-denials is 5 in the SDK. Without explicit configuration, Copilot can cycle through tool denials consuming tokens/credits indefinitely
• Where: All 67 files with copilot-sdk: true
• How to Implement: Add to frontmatter:

  engine:
    copilot-sdk: true
    max-tool-denials: 3

• Impact: Prevents cost overruns and silent infinite loops; should be standard on every SDK workflow

Opportunity 2: evals: for Production Copilot Workflows

• What: 27/37 explicit Copilot workflows have no quality assurance gates
• Why It Matters: Without evals, there's no way to detect when a workflow silently fails to achieve its goal
• Where: release.md, repo-tree-map.md, repository-quality-improver.md, slide-deck-maintainer.md, ubuntu-image-analyzer.md, video-analyzer.md, visual-regression-checker.md, workflow-health-manager.md, weekly-editors-health-check.md, update-astro.md, super-linter.md
• How to Implement:

  evals:
    - id: task-completed
      question: Did the agent successfully complete the primary task without errors?
    - id: output-quality
      question: Is the output relevant, accurate, and actionable?

Opportunity 3: Security Gap — 12 Copilot Workflows Without Network Config

• What: Production workflows without network: config use open network access
• Where: stale-pr-cleanup.md, terminal-stylist.md, uk-ai-operational-resilience.md, video-analyzer.md, workflow-health-manager.md, agent-persona-explorer.md, craft.md, daily-safe-output-integrator.md, refiner.md, repo-tree-map.md, repository-quality-improver.md, super-linter.md
• How to Implement: Add minimal network allowlist:

  network:
    allowed:
      - defaults

View Medium Priority Opportunities

🟡 Medium Priority

Opportunity 4: engine.token-weights for Cost Budgeting

• What: No workflow budgets AI token costs per tool, allowing expensive operations to run unchecked
• Why It Matters: Long-running workflows like release.md (20 min), spec-librarian.md could be costing significantly more than necessary
• Where: High-value production workflows: release.md, repository-quality-improver.md, spec-librarian.md, weekly-issue-summary.md
• How to Implement:

  engine:
    id: copilot
    token-weights:
      search: 2
      edit: 5
      bash: 3

Opportunity 5: startup-timeout for MCP-Using Workflows

• What: Only ruflo-backed-task.md protects against slow MCP server startups
• Why It Matters: If an MCP server hangs on startup, the workflow silently consumes timeout-minutes worth of GitHub Actions runner time
• Where: All workflows using tools.github, tools.bash, imports: shared/mcp/
• How to Implement:

  tools:
    startup-timeout: 60
    github:
      mode: gh-proxy

Opportunity 6: max-continuations on High-Timeout Copilot Workflows

• What: 19 Copilot workflows have 20+ minute timeouts without continuation limits
• Where: release.md (20 min), weekly-issue-summary.md (20 min), stale-pr-cleanup.md (30 min), spec-librarian.md, repository-quality-improver.md, refactoring-cadence.md, uk-ai-operational-resilience.md
• How to Implement:

  max-continuations: 3  # limit autopilot restarts

Opportunity 7: network.blocked for Sensitive Workflows

• What: No workflow uses network.blocked to proactively block categories of domains
• Where: Workflows with access to sensitive data: release.md, stale-pr-cleanup.md, sub-issue-closer.md
• How to Implement:

  network:
    allowed:
      - defaults
    blocked:
      - "*.analytics.com"
      - "*.telemetry.io"

Opportunity 8: Unused Custom Agent Files

• What: Three agent files in .github/agents/ are never referenced by any workflow
• Why It Matters: These represent investment in custom agent behavior that isn't being leveraged
• Unused files: create-safe-output-type.agent.md, custom-engine-implementation.agent.md, w3c-specification-writer.agent.md
• Where to apply:
  • create-safe-output-type: Any workflow creating new safe-output patterns
  • w3c-specification-writer: spec-librarian.md, spec-enforcer.md, spec-extractor.md
  • custom-engine-implementation: workflow-generator.md, developer workflows

View Low Priority Opportunities

🟢 Low Priority

Opportunity 9: engine.agent in More Copilot Workflows

• What: Only 3/37 explicit Copilot workflows (agent-persona-explorer.md, craft.md, daily-safe-output-integrator.md) use custom agent files
• Available agents: adr-writer, ci-cleaner, contribution-checker, technical-doc-writer, grumpy-reviewer
• Suggestions: release.md → technical-doc-writer; stale-pr-cleanup.md → ci-cleaner

Opportunity 10: engine.version Pinning for Stability

• What: No explicit Copilot workflows pin the engine version for reproducibility
• Where: release.md, spec-librarian.md — workflows where consistency matters most
• Example:

  engine:
    id: copilot
    version: "0.0.422"

Opportunity 11: engine.bare for Simple Read-Only Workflows

• What: Workflows that just read data and report could disable context loading to reduce token overhead
• Where: daily-credit-limit-test.md, weekly-editors-health-check.md, repo-tree-map.md
• How: Add bare: true under engine: extended config

---

4️⃣ Specific Workflow Recommendations

View Workflow-Specific Recommendations

release.md

• Current State: 20-min timeout, no evals, no sandbox config, no max-continuations
• Recommended Changes: Add evals, max-continuations: 3, sandbox config with sudo: false
• Expected Benefits: Catch release failures automatically, prevent runaway continuation cycles

stale-pr-cleanup.md

• Current State: Has good tools config, sandbox missing, no network config, no evals
• Recommended Changes: Add sandbox:, network:, 2-3 evals for closure accuracy
• Expected Benefits: Security hardening + quality validation

weekly-issue-summary.md

• Current State: Has sandbox + network + strict, missing startup-timeout, max-continuations
• Recommended Changes: Add startup-timeout: 60, max-continuations: 2
• Expected Benefits: Prevent MCP hangs on weekly scheduled runs

spec-librarian.md / spec-enforcer.md

• Current State: Long-running, no evals, no startup-timeout
• Recommended Changes: Add evals:, consider engine.agent: w3c-specification-writer
• Expected Benefits: Leverage specialized agent knowledge for spec work

All 67 copilot-sdk: true workflows

• Current State: No max-tool-denials protection
• Recommended Changes: Add max-tool-denials: 3 to engine config block
• Expected Benefits: Eliminate infinite permission-loop risk uniformly

---

5️⃣ Trends & Insights

View Historical Trends

Changes Since Last Analysis (Jul-26 → Jul-27)

Metric	Jul-26	Jul-27	Trend
Total Workflows	262	262	→ Stable
Copilot Explicit	37	37	→ Stable
Copilot SDK	67	67	→ Stable
max-tool-denials	0	0	🔴 Persistent
engine.token-weights	0	0	🔴 Persistent
startup-timeout	1	1	🔴 Persistent
mcp-scripts	13	~13	→ Stable
copilot-requests: write	69	69	→ Stable

Long-Term Trends (May-21 → Jul-27)

• Growth: 233 → 262 workflows (+13%)
• MCP adoption: 3 → 13 mcp-scripts (+333%) — major improvement
• SDK workflows: 63 → 67 (+6%)
• max-tool-denials: 0 for 18+ consecutive research cycles — most persistent unaddressed gap in the history of this analysis

Persistent Pattern

The three most critical protective features (max-tool-denials, engine.token-weights, startup-timeout) have seen zero adoption despite being flagged in every analysis since May 2026. This suggests either: (a) lack of awareness, (b) documentation gap, or (c) the features may require framework changes to apply easily at scale.

---

6️⃣ Best Practice Guidelines

Based on this research, here are recommended best practices:

1. Every copilot-sdk: true workflow needs max-tool-denials: 3 — This is the single most impactful safety net missing from the codebase. Default of 5 may be too high; 3 is the recommended sweet spot.

2. Add evals: to all production Copilot workflows — Even 1-2 basic questions dramatically improve observability. Make it a workflow authoring checklist item.

3. startup-timeout: 60 should be the default for any workflow using MCP tools — Silent MCP hangs waste the most runner time without any signal.

4. Use network: allowed: [defaults] as the minimum baseline — Even without custom rules, explicit network config makes intent clear and enables security auditing.

5. Leverage the custom agent file library — .github/agents/ contains specialized agents for ADR writing, CI cleanup, contribution checking, and documentation. Match them to appropriate workflows rather than relying on the default system prompt.

6. Set max-continuations on any workflow with timeout-minutes: > 15 — Long-running workflows with autopilot enabled need an explicit upper bound on self-continuation cycles.

---

---

7️⃣ Action Items

Immediate Actions (this week):

• Add max-tool-denials: 3 to all 67 copilot-sdk: true workflows — this is the rejig docs #1 critical gap across 18+ research runs
• Add evals: blocks to the top 5 highest-impact Copilot workflows: release.md, stale-pr-cleanup.md, weekly-issue-summary.md, workflow-health-manager.md, repository-quality-improver.md

Short-term (this month):

• Add startup-timeout: 60 to all workflows using MCP tools (currently only 1 of ~50 has it)
• Add network: configuration to the 12 Copilot workflows missing it
• Reference w3c-specification-writer.agent.md from spec-librarian.md and spec-enforcer.md
• Add max-continuations: 3 to release.md, stale-pr-cleanup.md, weekly-issue-summary.md

Long-term (this quarter):

• Implement engine.token-weights for cost budgeting in high-value workflows
• Create a workflow authoring checklist that includes evals:, max-tool-denials, and startup-timeout
• Audit and activate the 3 unused custom agent files with appropriate workflow pairings
• Consider automating max-tool-denials as a compile-time default for all SDK workflows

---

View Supporting Evidence & Methodology

📚 References

• Copilot Engine Source: pkg/workflow/copilot_engine.go, copilot_engine_execution.go, copilot_engine_tools.go, copilot_mcp.go
• Documentation: docs/src/content/docs/reference/engines.md
• Custom Agent Files: .github/agents/*.agent.md
• Previous Analysis: memory/copilot-cli-research branch (18+ research runs since May 2026)

Research Methodology

1. File inventory: find pkg -name 'copilot*.go' to enumerate all engine files
2. Feature extraction: Reviewed copilot_engine_execution.go (38.9KB), copilot_engine_tools.go, copilot_mcp.go for available features
3. Usage survey: grep -rn "engine: copilot" + pattern matching across all 262 .github/workflows/*.md files
4. Gap analysis: Cross-referenced documented features vs actual frontmatter usage
5. Historical comparison: Loaded memory/copilot-cli-research/latest.json for trend tracking
6. Targeted counts: Per-feature adoption rates using grep -l + wc -l pipelines

---

Generated by Copilot CLI Deep Research (Run: §30238230557)

References:

• §30238230557 — This research run

Generated by 🔬 Copilot CLI Deep Research Agent · sonnet46 · 99.4 AIC · ⌖ 9.28 AIC · ⊞ 7.5K · ◷

• expires on Jul 27, 2026, 9:00 PM UTC-08:00