Reduce Copilot token analyzer prompt verbosity to avoid max-ai-credits trips by Copilot · Pull Request #4944 · github/gh-aw-firewall

Copilot · 2026-06-14T16:57:36Z

Daily Copilot Token Usage Analyzer intermittently hit the configured max-ai-credits guardrail due to overly verbose report generation behavior. This change tightens prompt constraints so the agent emits a bounded, compact report instead of spending extra turns on repeated output reshaping.

Prompt constraints (workflow source)
- Updated copilot-token-usage-analyzer.md guidance to explicitly cap verbose detail:
  - “Top Per-Workflow Details” section naming (signals intentional truncation)
  - Limit detailed bullets to top 10 workflows by token usage
  - Prefer trimming detail over repeated full-report rewrites when output is large
Regression coverage
- Added scripts/ci/copilot-token-usage-analyzer-workflow.test.ts to assert these compact-output guardrails remain present in the workflow prompt.

- **Keep output compact**: include detailed bullets for at most the top 10 workflows by token usage; keep remaining workflows in the summary table only.
- **Avoid verbose rewrites**: if output size is high, trim detail instead of repeatedly reformatting the full report.

github-actions · 2026-06-14T17:20:10Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	96.60%	96.88%	📈 +0.28%
Statements	96.47%	96.75%	📈 +0.28%
Functions	98.80%	98.80%	➡️ +0.00%
Branches	91.18%	91.28%	📈 +0.10%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	92.6% → 94.4% (+1.85%)	92.6% → 94.4% (+1.85%)
`src/artifact-preservation.ts`	85.4% → 100.0% (+14.64%)	85.4% → 100.0% (+14.64%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR reduces the verbosity of the Copilot Token Usage Analyzer’s generated issue content by adding explicit compact-output constraints to the workflow prompt, helping avoid intermittently tripping the max-ai-credits guardrail. It also adds a regression test to ensure the prompt retains these compact-output guardrails over time.

Changes:

Renames the report section header to “Top Per-Workflow Details” to signal intentional truncation.
Adds explicit prompt guidance to cap detailed bullets to the top 10 workflows by token usage and to trim detail instead of repeatedly reformatting large output.
Adds a Jest test asserting the presence of these compact-output guardrails in the workflow prompt.

Show a summary per file

File	Description
`scripts/ci/copilot-token-usage-analyzer-workflow.test.ts`	Adds a regression test asserting compact-output guardrails exist in the workflow prompt.
`.github/workflows/copilot-token-usage-analyzer.md`	Updates the prompt/report template and guidelines to encourage bounded, compact reporting.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 2/2 changed files
Comments generated: 1

+    expect(source).toContain('<summary><b>Top Per-Workflow Details</b></summary>');
+    expect(source).toContain('include detailed bullets for at most the top 10 workflows by token usage');
+    expect(source).toContain('if output size is high, trim detail instead of repeatedly reformatting the full report');
+  });


github-actions · 2026-06-14T17:29:41Z

@Copilot @lpcox

GitHub MCP connectivity: ✅
GitHub.com HTTP: ✅
File write/read: ✅
BYOK inference: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

Ensures the old '## Per-Workflow Details' header cannot be accidentally reintroduced alongside the new compact '<summary>' version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-14T19:18:23Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #4944 · ◷

github-actions · 2026-06-14T19:23:03Z

Smoke Test: Copilot PAT Auth — PASS ✅

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅
File write/read	✅

Overall: PASS | Auth mode: PAT (COPILOT_GITHUB_TOKEN)
PR: "Reduce Copilot token analyzer prompt verbosity to avoid max-ai-credits trips" | Author: @Copilot | Assignees: @lpcox, @Copilot

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-14T19:23:33Z

✅ Smoke Test: Copilot BYOK (Direct) Mode — PASS

Test	Result
GitHub MCP Connectivity	✅
GitHub.com Connectivity (HTTP 200)	✅
File Write/Read	✅
BYOK Inference (api-proxy → api.githubcopilot.com)	✅

Status: Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar to api.githubcopilot.com

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-06-14T19:23:37Z

🤖 Smoke Test Results — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅ 200
File write/read	✅

PR: Reduce Copilot token analyzer prompt verbosity to avoid max-ai-credits trips
Author: @Copilot | Assignees: @lpcox @Copilot

Overall: PASS ✅

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-14T19:23:58Z

Merged PRs:

Refactor OpenAI BYOK base URL parsing to reuse shared proxy URL normalization
refactor(api-proxy): split proxy-request.js into http-client.js and body-handler.js
Reduce Copilot token analyzer prompt verbosity to avoid max-ai-credits trips
✅ GitHub PR review
✅ safeinputs-gh CLI query
✅ Playwright GitHub title check
✅ File write/read smoke test
✅ Discussion lookup/comment
✅ npm ci && npm run build
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-14T19:26:35Z

Smoke Test: Services Connectivity

Check	Result
Redis PING	❌ timeout
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ timeout

Overall: FAIL

host.docker.internal resolves to 172.17.0.1 but ports 6379 and 5432 are unreachable — service containers appear not running.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-06-14T19:29:27Z

@Copilot @lpcox Tests:

MCP PR fetch: ✅
GitHub.com HTTP: ✅
File write/read: ✅
BYOK inference: ✅
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-06-14T19:31:29Z

Smoke Test Results

GitHub MCP Testing: ✅
GitHub.com Connectivity: ❌ (SSL Error 35 / Squid 400)
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall Status: FAIL

Last 2 Merged PRs:

refactor(api-proxy): split proxy-request.js into http-client.js and body-handler.js (refactor(api-proxy): split proxy-request.js into http-client.js and body-handler.js #4939)
Refactor OpenAI BYOK base URL parsing to reuse shared proxy URL normalization (Refactor OpenAI BYOK base URL parsing to reuse shared proxy URL normalization #4949)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

Initial plan

b7707ee

Copilot AI assigned Copilot and lpcox Jun 14, 2026

Copilot started work on behalf of lpcox June 14, 2026 17:10 View session

Reduce copilot token analyzer report verbosity to avoid AI credit cap

4468960

Copilot AI changed the title ~~[WIP] Update max-ai-credits in Daily Copilot Token Usage Analyzer~~ Reduce Copilot token analyzer prompt verbosity to avoid max-ai-credits trips Jun 14, 2026

Copilot finished work on behalf of lpcox June 14, 2026 17:16

Copilot AI requested a review from lpcox June 14, 2026 17:16

lpcox marked this pull request as ready for review June 14, 2026 17:18

Copilot AI review requested due to automatic review settings June 14, 2026 17:18

Copilot started reviewing on behalf of lpcox June 14, 2026 17:19 View session

Copilot AI temporarily deployed to aoai-model June 14, 2026 17:20 Inactive

Copilot AI reviewed Jun 14, 2026

View reviewed changes