fix(test): sync doc-maintainer test with max-turns 15 + prompt rewrite by lpcox · Pull Request #5587 · github/gh-aw-firewall

lpcox · 2026-06-26T20:02:00Z

Problem

npm test is currently red on main: scripts/ci/doc-maintainer-workflow.test.ts has 2 failing assertions.

PR #5564 (fix(doc-maintainer): prevent maxRuns 403 from wasted shell turns) raised the Documentation Maintainer max-turns from 8 → 15 and rewrote the shell-restriction prompt line in .github/workflows/doc-maintainer.md (and recompiled the lock), but did not update the test guarding those values.

Failing assertions

Expected substring: "max-turns: 8"          (source)  -> now max-turns: 15
Expected substring: "GH_AW_MAX_TURNS: 8"     (lock)    -> now GH_AW_MAX_TURNS: 15
Expected substring: "**Do not run any `git` commands**"  -> replaced by new wording

Fix

Update the three stale assertions to match the current source workflow and lock:

max-turns: 8 → max-turns: 15
GH_AW_MAX_TURNS: 8 → GH_AW_MAX_TURNS: 15
**Do not run any \git` commands**→Do not use the `shell` tool (and the `bash` tool is disabled). Do not attempt to run `git`, `npm test`, `ls`, or any other shell command`

No production code changes — test-only sync.

Verification

Test Suites: 196 passed, 196 total
Tests:       3220 passed, 3220 total

npm test is fully green.

PR #5564 raised the Documentation Maintainer `max-turns` from 8 to 15 (to stop the agent exhausting its maxRuns budget on denied shell turns and surfacing a misleading 403) and rewrote the shell-restriction prompt line, but did not update scripts/ci/doc-maintainer-workflow.test.ts. This left `npm test` red with two stale assertions: - `max-turns: 8` / `GH_AW_MAX_TURNS: 8` -> now 15 - `**Do not run any \`git\` commands**` -> replaced by the new `**Do not use the \`shell\` tool** ...` wording Update the assertions to match the current source workflow and lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Updates the doc-maintainer workflow guard test to match the current workflow configuration after #5564 increased the agent turn budget and rewrote the shell-restriction prompt text, restoring npm test on main.

Changes:

Sync source-workflow assertion from max-turns: 8 to max-turns: 15.
Sync lock-workflow assertion from GH_AW_MAX_TURNS: 8 to GH_AW_MAX_TURNS: 15.
Update the prompt-string assertion to match the new “do not use the shell tool” wording.

Show a summary per file

File	Description
scripts/ci/doc-maintainer-workflow.test.ts	Updates stale assertions so the test matches the current doc-maintainer workflow source + lock.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 1/1 changed files
Comments generated: 0
Review effort level: Low

github-actions · 2026-06-26T20:03:51Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.24%	98.28%	📈 +0.04%
Statements	98.17%	98.21%	📈 +0.04%
Functions	99.53%	99.53%	➡️ +0.00%
Branches	94.00%	94.00%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	92.7% → 94.5% (+1.82%)	92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-06-26T21:09:19Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-06-26T21:09:20Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-06-26T21:09:22Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-06-26T21:09:22Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

github-actions · 2026-06-26T21:09:23Z

✅ Contribution Check completed successfully!

Contribution guidelines review complete for PR #5587: no important missing items found; no comment needed.

github-actions · 2026-06-26T21:09:24Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-06-26T21:09:26Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-06-26T21:09:27Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-06-26T21:09:29Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-06-26T21:09:29Z

✅ Smoke Claude passed

github-actions · 2026-06-26T21:09:30Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-06-26T21:09:40Z

✅ Build Test Suite completed successfully!

github-actions · 2026-06-26T21:09:48Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-06-26T21:09:52Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-06-26T21:12:26Z

Smoke Test: Claude Engine Validation

API check: ✅ PASS
gh CLI check: ✅ PASS
File status check: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for issue #5587 · 61.5 AIC · ⊞ 3.3K · ◷

github-actions · 2026-06-26T21:12:57Z

🔬 Smoke Test: Copilot PAT Auth — FAIL

Test	Result
GitHub MCP connectivity	✅ Listed PRs from `github/gh-aw-firewall`
GitHub.com HTTP	❌ Pre-step output not resolved (template variable unsubstituted)
File write/read	❌ Pre-step output not resolved (template variable unsubstituted)

Overall: FAIL — Pre-step outputs (SMOKE_HTTP_CODE, SMOKE_FILE_PATH, SMOKE_FILE_CONTENT) were not substituted; workflow step likely failed silently.

Auth mode: PAT (COPILOT_GITHUB_TOKEN) | PR author: @lpcox

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-26T21:13:03Z

Smoke Test: BYOK Direct Mode ✅

Test Results:

✅ GitHub MCP: PRs retrieved (perf(contribution-check): cut token/tool overhead per #5558 #5576, Split squid config tests by concern #5575)
✅ GitHub.com: HTTP 200
✅ Tests: doc-maintainer suite passing (2/2)
✅ BYOK Inference: Running via api-proxy → api.githubcopilot.com

Status: PASS — direct BYOK mode (COPILOT_PROVIDER_API_KEY) operational

cc @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-06-26T21:13:03Z

Smoke Test: Gemini Engine Validation

GitHub MCP Testing: ❌ (Could not fetch PR titles)
GitHub.com Connectivity: ❌ (HTTP 000 / SSL Error)
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-06-26T21:13:35Z

@lpcox

✅ GitHub MCP Testing
✅ GitHub.com Connectivity
✅ File Write/Read Test
✅ BYOK Inference Test

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-06-26T21:14:02Z

🔬 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Detail
1. Module Loading	✅ Pass	`otel.js` loads, `isEnabled: true`, exports 14 functions (`startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, ...)
2. Test Suite	✅ Pass	39 tests passed, 0 failed (`otel.test.js`)
3. Env Var Forwarding	⚠️ Expected	`api-proxy-service.ts` doesn't directly forward OTEL vars — forwarding lives in `api-proxy-env-config.ts` instead; workflow step reports "expected during development"
4. Token Tracker Integration	✅ Pass	`onUsage` callback present in `token-tracker-http.js` (OTEL hook point confirmed)
5. OTEL Diagnostics	✅ Pass	`/tmp/gh-aw/otel.jsonl` found — 1 span exported during this run

Overall: ✅ All scenarios pass (Scenario 3 ⚠️ is expected per workflow definition).

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-26T21:14:15Z

Smoke test

perf(contribution-check): cut token/tool overhead per #5558
Split squid config tests by concern
GitHub PR query: ✅
Discussion comment: ✅
Playwright: ✅
File write/read: ✅
Build: ✅
Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-26T21:14:28Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌ NO
Node.js	v24.17.0	v22.23.0	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-06-26T21:14:54Z

🔬 Smoke Test Results

Test	Result
GitHub MCP connectivity	✅ PASS
GitHub.com HTTP connectivity	❌ FAIL (pre-step data not resolved)
File write/read	❌ FAIL (pre-step data not resolved)

PR: fix(test): sync doc-maintainer test with max-turns 15 + prompt rewrite
Author: @lpcox

Overall: FAIL — pre-step template variables were not substituted; tests 2 & 3 could not be verified.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-26T21:16:03Z

Smoke Test Results

Check	Result
Redis PING	❌ timeout (no PONG)
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ skipped (host unreachable)

Overall: FAIL — host.docker.internal is not reachable from this environment. Service containers are not accessible.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-06-26T21:16:07Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #5587 · 72.7 AIC · ⊞ 7.8K · ◷

Copilot AI review requested due to automatic review settings June 26, 2026 20:02

Copilot started reviewing on behalf of lpcox June 26, 2026 20:02 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

lpcox added the ready-for-aw label Jun 26, 2026

lpcox temporarily deployed to aoai-model June 26, 2026 21:09 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Jun 26, 2026

[aw] No-Op Runs #5368

Open

github-actions Bot added the smoke-claude label Jun 26, 2026

github-actions Bot added the smoke-copilot-byok label Jun 26, 2026

lpcox temporarily deployed to aoai-model June 26, 2026 21:13 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jun 26, 2026

lpcox temporarily deployed to aoai-model June 26, 2026 21:13 — with GitHub Actions Inactive

github-actions Bot added the smoke-codex label Jun 26, 2026

github-actions Bot added the build-test label Jun 26, 2026

github-actions Bot mentioned this pull request Jun 26, 2026

[aw] Smoke Copilot BYOK AOAI (api-key) is missing required tool #5588

Closed

lpcox merged commit 948e007 into main Jun 26, 2026
87 of 89 checks passed

lpcox deleted the fix/doc-maintainer-test-max-turns-15 branch June 26, 2026 21:46

github-actions Bot mentioned this pull request Jun 27, 2026

Extract OIDC/auth helpers from proxy-utils into oidc-adapter-utils #5604

Merged

Uh oh!

Conversation

lpcox commented Jun 26, 2026

Problem

Failing assertions

Fix

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Jun 26, 2026

🔬 Smoke Test: Copilot PAT Auth — FAIL

Uh oh!

github-actions Bot commented Jun 26, 2026

Smoke Test: BYOK Direct Mode ✅

Uh oh!

github-actions Bot commented Jun 26, 2026

Smoke Test: Gemini Engine Validation

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

🔬 Smoke Test: API Proxy OpenTelemetry Tracing

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Chroot Version Comparison Results

Uh oh!

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading