fix: move smoke-gemini tests into agent container by lpcox · Pull Request #2401 · github/gh-aw-firewall

lpcox · 2026-05-02T18:05:10Z

Summary

The smoke-gemini workflow was running all tests (curl connectivity, file write/read, gh pr list) in a host pre-step, then having the Gemini agent merely verify pre-computed results. This meant the tests validated host connectivity, not AWF sandbox connectivity.

Changes

Removed the Pre-compute smoke test data host pre-step that ran curl, file write, and gh commands on the host
Moved test requirements into the agent prompt so the Gemini agent performs all tests inside the AWF sandbox container (matching the pattern used by smoke-claude.md)
Agent now runs curl connectivity, file write/read, and MCP calls itself inside the sandbox
Post-step safe-output validation unchanged

Why

Smoke tests should exercise the firewall's domain allowlist, bash tool, and MCP connectivity from within the container. Running tests on the host bypasses the sandbox entirely.

Pattern comparison

Workflow	Before	After
smoke-claude	Agent does all tests ✅	(unchanged)
smoke-copilot	Host pre-computes, agent verifies	(unchanged, separate fix)
smoke-gemini	Host pre-computes, agent verifies ❌	Agent does all tests ✅

Previously, smoke-gemini ran all tests (curl connectivity, file write/read, gh pr list) in a host pre-step, then had the agent merely verify pre-computed results. This meant the tests validated host connectivity, not AWF sandbox connectivity. Now the agent performs all tests inside the sandbox (like smoke-claude), properly exercising the firewall's domain allowlist, bash tool, and MCP connectivity from within the container. Changes: - Remove Pre-compute smoke test data host pre-step - Move test requirements into agent prompt - Agent now runs curl, file write/read, and MCP calls itself - Keep post-step safe-output validation unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-02T18:06:13Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	85.76%	85.84%	📈 +0.08%
Statements	85.64%	85.72%	📈 +0.08%
Functions	88.11%	88.11%	➡️ +0.00%
Branches	78.65%	78.69%	📈 +0.04%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR updates the smoke-gemini workflow so that connectivity and filesystem checks are executed inside the AWF agent sandbox container (instead of being pre-computed on the host), aligning the Gemini smoke test pattern with the existing smoke-claude workflow approach.

Changes:

Removed the host “Pre-compute smoke test data” step (curl / file I/O / gh pr list) so results are no longer produced outside the sandbox.
Updated the agent prompt to directly perform GitHub MCP PR listing, curl connectivity, and file write/read via bash inside the container.
Regenerated the compiled .lock.yml workflow accordingly (removing smoke-data outputs and wiring github.run_id through prompt substitutions).

Show a summary per file

File	Description
.github/workflows/smoke-gemini.md	Removes host-side precomputation and instructs the Gemini agent to run all smoke checks inside the sandbox.
.github/workflows/smoke-gemini.lock.yml	Regenerated compiled workflow removing the pre-step and its derived env/expressions; keeps prompt interpolation functional.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 2/2 changed files
Comments generated: 0

github-actions · 2026-05-02T18:08:40Z

Smoke Test Results:

✅ GitHub MCP: gh pr list — last 2 PRs: feat: unify schema versioning — use repo release tag for all schemas, publish JSONL schemas as release assets #2398, fix: load schema via require() for pkg/esbuild compat #2396
✅ Playwright: github.com title contains "GitHub"
✅ File writing: /tmp/gh-aw/agent/smoke-test-claude-25258365342.txt created
✅ Bash: File verified

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-05-02T18:09:13Z

🔥 Smoke Test Results

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP connectivity (200)	✅
File write/read	✅

Overall: PASS

PR by @lpcox. Reviewer: @Mossaka.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-05-02T18:09:52Z

Smoke Test: Copilot BYOK Offline Mode — PR #2401 by @lpcox, reviewer @Mossaka

Test	Result
GitHub MCP connectivity	✅ Listed merged PR #2398
GitHub.com HTTP connectivity	⚠️ Pre-step template vars not expanded (`$\{\{ steps.smoke-data.outputs.SMOKE_HTTP_CODE }}`)
File write/read	⚠️ Pre-step template vars not expanded
BYOK inference (agent → api-proxy → api.githubcopilot.com)	✅ Responding via BYOK path

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com.

Overall: PARTIAL — BYOK inference and MCP work; pre-step smoke data was not injected into prompt (unexpanded $\{\{ }} variables).

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-05-02T18:10:04Z

Smoke Test

Merged PRs: "feat: unify schema versioning — use repo release tag for all schemas, publish JSONL schemas as release assets"; "fix: load schema via require() for pkg/esbuild compat"
Queried PRs: "fix: move smoke-gemini tests into agent container"; "[docs] docs: add awf-config.v1.schema.json references to config spec"
GitHub merged PR review ✅
GH CLI query ❌ (safeinputs-gh unavailable; fallback gh used)
Playwright ✅ | Tavily ❌
File+cat ✅ | Discussion comment ✅ | Build ✅
Overall: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-05-02T18:10:50Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #2401 · ● 483.6K · ◷

github-actions · 2026-05-02T18:11:15Z

Smoke Test: GitHub Actions Services Connectivity

Check	Result
Redis PING	❌ Timeout (no response)
PostgreSQL pg_isready	❌ No response
PostgreSQL SELECT 1	❌ Timeout

Overall: FAIL

host.docker.internal is unreachable from this environment — all three service connectivity checks failed.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-05-02T18:27:14Z

PR Titles:

feat: unify schema versioning — use repo release tag for all schemas, publish JSONL schemas as release assets
fix: load schema via require() for pkg/esbuild compat

Test Results:

GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall status: PASS

💎 Faceted by Smoke Gemini

lpcox requested a review from Mossaka as a code owner May 2, 2026 18:05

Copilot AI review requested due to automatic review settings May 2, 2026 18:05

Copilot started reviewing on behalf of lpcox May 2, 2026 18:05 View session

github-actions Bot mentioned this pull request May 2, 2026

[aw] Smoke OpenCode failed #2402

Closed

Copilot AI reviewed May 2, 2026

View reviewed changes

github-actions Bot added the smoke-claude label May 2, 2026

github-actions Bot added the smoke-copilot label May 2, 2026

github-actions Bot added the build-test label May 2, 2026

github-actions Bot added the smoke-gemini label May 2, 2026

lpcox merged commit bd3eb05 into main May 2, 2026
67 of 69 checks passed

lpcox deleted the fix/smoke-gemini-agent-testing branch May 2, 2026 18:31

lpcox mentioned this pull request May 2, 2026

Gemini CLI: NumericalClassifierStrategy routing errors flood agent logs #2403

Closed

This was referenced May 2, 2026

fix: wire MCP gateway into Gemini CLI settings (localhost) #2405

Closed

refactor(api-proxy): extract ProviderAdapter abstraction, make core provider-agnostic #2409

Merged

feat: add 4 agentic code quality workflows #2417

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: move smoke-gemini tests into agent container#2401

fix: move smoke-gemini tests into agent container#2401
lpcox merged 1 commit into
mainfrom
fix/smoke-gemini-agent-testing

lpcox commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lpcox commented May 2, 2026

Summary

Changes

Why

Pattern comparison

Uh oh!

github-actions Bot commented May 2, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

🔥 Smoke Test Results

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Smoke Test

Uh oh!

github-actions Bot commented May 2, 2026

🏗️ Build Test Suite Results

Uh oh!

github-actions Bot commented May 2, 2026

Smoke Test: GitHub Actions Services Connectivity

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants