ARC/DinD implementation gaps identified from community feedback (gh-aw#40107)

[lpcox]

Context

While improving the ARC/DinD setup guide in response to community issue github/gh-aw#40107 and comment, we identified several AWF gaps that documentation alone cannot address.

Related PRs reviewed: #5696, #5890, #5963, #6101, #6195, #6315.

Update (2026-07-17): Rescoped after confirming the gh-aw compiler already enables --network-isolation by default (and enforces it under strict mode). See the "Revised scope" note below. Gaps 3 and 4 are addressed in #6328.

Revised scope

The gh-aw compiler defaults NetworkIsolation=true for every compiled workflow (frontmatter_extraction_security.go), emitting AWF --network-isolation. Disabling it (sudo: true) is a compile-time error in strict mode, and strict mode is the default. The DinD runtime path prefix and chroot config are also already emitted automatically (ADRs github/gh-aw#31614, github/gh-aw#38911).

This changes two of the original gaps:

• Gap 1 (topology auto-defaults) is largely moot for gh-aw users — the effective config the runner.topology: arc-dind selector was meant to auto-set is already driven by the gh-aw compiler/runtime. Rescoped below to standalone AWF usage only.
• Gap 2 (network-isolation CI coverage) is now the primary follow-up: network isolation is the default, strict-mode-enforced path for all gh-aw copilot workflows, yet it has near-zero smoke-test coverage.

Gaps

✅ Gap 3 — DinD-spawned containers cannot resolve Kubernetes DNS names (addressed in #6328)

When the DIFC proxy or other services use Kubernetes service names as hostnames (e.g., awmg-cli-proxy), Docker containers spawned by the DinD daemon get getaddrinfo EAI_AGAIN because the DinD Docker network does not forward DNS to the Kubernetes cluster resolver. The awf-cli-proxy could not connect to the external DIFC proxy error hid this root cause.

Fixed in #6328: detectDnsResolutionFailure() scans cli-proxy logs for EAI_AGAIN/ENOTFOUND and augments the startup error with a DNS-isolation explanation and fix guidance (address by IP, or configure dockerd --dns).

✅ Gap 4 — Post-job chmod fails on rootless/non-privileged containers (addressed in #6328)

The best-effort artifact permission repair emitted a scary [WARN] Rootless artifact permission repair failed when denied CHOWN/chmod (Operation not permitted) on restricted runners, even though the run had already succeeded.

Fixed in #6328: benign permission errors are downgraded to debug; genuine failures still warn.

🔴 Gap 2 (PRIMARY) — Network-isolation mode has near-zero CI coverage

Per issue #6193, 19/21 smoke tests use sudo + host iptables; 0/21 use --network-isolation. PR #6195 (migrate gVisor smoke tests to strict/no-sudo mode) was closed without merging.

Because the gh-aw compiler makes --network-isolation the default (and strict-mode-enforced) path, the mode every real gh-aw copilot workflow runs is the least tested. Regressions in this path (like the EAI_AGAIN DNS failure in github/gh-aw#40107) are not exercised by CI.

Proposed fix:

1. Add one baseline smoke test running --network-isolation (no sudo, no host iptables) covering egress allow, egress deny, and agent completion — using the standard agent runtime (not gVisor) to avoid the JIT/DNS complications that stalled feat: migrate gVisor smoke tests to strict security mode (no sudo, network-isolation) #6195.
2. Wire it into the smoke-test workflow matrix (requires a reachable Docker daemon).
3. Follow-up: incrementally migrate additional smoke tests off the legacy sudo/iptables path.

⚠️ Gap 1 (RESCOPED) — runner.topology: arc-dind only partially auto-configures AWF

Issue #5591 proposed that runner.topology: arc-dind auto-enable 6 lower-level fields (network.isolation, container.dockerHostPathPrefix, container.runnerToolCachePath, chroot.*, dind.*). PR #5696 implemented only sysroot staging + tool-cache warning.

For gh-aw users this is moot — the gh-aw compiler/runtime already drives network.isolation (default on) and the DinD path prefix / chroot config (via ADRs github/gh-aw#31614, github/gh-aw#38911). The remaining fields are effectively set through the gh-aw path.

Remaining question (standalone AWF only): should runner.topology: arc-dind imply network.isolation/dind.preStageDirs defaults for users invoking awf directly without the gh-aw compiler? If standalone ARC/DinD invocation is a supported use case, apply those defaults (only when unset) and emit a startup log line listing them. Otherwise, close this as won't-fix and document that topology: arc-dind requires --network-isolation alongside it.

Priority

Gap	Status	Severity
3. DinD DNS diagnosis	✅ #6328	—
4. chmod warning noise	✅ #6328	—
2. network-isolation CI coverage	🔴 open (primary)	High
1. topology auto-defaults	⚠️ open (rescoped; likely won't-fix for gh-aw)	Low

[lpcox]

Opened #6328 addressing gaps 3 (DinD DNS isolation diagnosis) and 4 (rootless chmod noise). Gaps 1 (topology auto-defaults) and 2 (network-isolation smoke-test coverage) remain open here as larger follow-ups.

[lpcox]

Rescoped the issue after confirming the gh-aw compiler already enables --network-isolation by default and enforces it under strict mode (frontmatter_extraction_security.go; strict is the default per compiler_yaml_policy.go).

Net effect:

• Gap 1 (topology auto-defaults) is largely moot for gh-aw users — network isolation and the DinD path-prefix/chroot config are already driven by the compiler/runtime (ADRs Auto-detect ARC/DinD and emit AWF --docker-host-path-prefix in generated workflows gh-aw#31614, [ARC/DinD] Emit chroot.binariesSourcePath and chroot.identity in AWF stdin-config gh-aw#38911). Narrowed to standalone awf invocations only; likely won't-fix.
• Gap 2 (network-isolation smoke-test coverage) is now the primary follow-up: it's the default, strict-enforced path for every gh-aw copilot workflow, yet 0/21 smoke tests exercise it.

Gaps 3 and 4 remain addressed by #6328.

[lpcox]

Gap 2 (network-isolation CI coverage) is addressed by #6330, which adds a CI-integrated smoke-copilot-network-isolation smoke test. It runs the Copilot engine under isolation: true and asserts egress allow (api.github.com reachable) + egress deny (example.com blocked) from inside the sandbox, plus agent-completion via the add_comment safe-output. It's wired into the ready-for-aw label matrix so it runs automatically on PRs.