docs: update runner doctor catalog — D1 gVisor resolved, add B11 silent repair exit-1 - #6118
Conversation
- Update D1: gVisor resolved in #6093; Kata Containers still open - Add B11: silent repair failure causes exit 1, fixed in #6072 - Add B11 error-string lookup entry in all three catalog files - Update Known unresolved items: narrow D1 to Kata only - Update symptom mapping and known-gaps step in both doctor files - Update CI test assertions for D1 and B11 Closes #6103
There was a problem hiding this comment.
Pull request overview
Updates the runner-doctor catalog with gVisor support status and a new rootless cleanup diagnosis.
Changes:
- Marks gVisor support resolved while retaining Kata as unresolved.
- Adds B11 cleanup-failure guidance across mirrored catalogs.
- Adds synchronization assertions.
Show a summary per file
| File | Description |
|---|---|
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Tests catalog updates. |
.github/workflows/shared/self-hosted-failure-modes.md |
Updates canonical failure catalog. |
.github/workflows/self-hosted-runner-doctor.md |
Updates diagnostic playbook. |
.github/agents/self-hosted-runner-doctor.md |
Updates portable mirrored catalog. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 6
- Review effort level: Medium
| | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | ||
| | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | | ||
| | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 | | ||
| | B11 | Agent task succeeds but AWF process exits with code 1; logs show `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` and `[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES`; no actionable detail visible — repair container stderr was silently discarded | `fixArtifactPermissionsForRootless()` ran a `docker run` for chown/chmod but did not capture or log stderr, so the root cause was opaque. `removeWorkDirectories()` then escalated the subsequent `EACCES` from `fs.rmSync` on remapped-UID files to a fatal exit, overriding the successful agent exit code. | **Fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured and included in the `[WARN]` message. Chroot-home removal failure is downgraded from `warn` to `debug` so it no longer propagates as a non-zero exit code — the post-step cleanup via `sudo` handles it. Upgrade to AWF including github/gh-aw-firewall#6072. | `[WARN] Rootless artifact permission repair failed` with no additional detail (no stderr), followed by `Command completed with exit code: 1` even though the agent log shows success | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | |
|
|
||
| B10 / github/gh-aw-firewall#6025 — `fixArtifactPermissionsForRootless()` compound `tag@digest` ref timeout is **fixed** in AWF version including github/gh-aw-firewall#6025. `resolvePermFixerImageRef()` now returns tag-only refs, eliminating registry I/O during `--pull never` repair. | ||
|
|
||
| B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and chroot-home removal failure escalated to exit 1 is **fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**. Stderr is now included in the `[WARN]` message; chroot-home removal failure is downgraded to `debug`. |
| | B8 | `EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'` (or any `/tmp/gh-aw/...` path) — failure occurs **before any container starts**, at `writeConfigs` time, on a **persistent self-hosted runner** | A previous AWF run or the Docker daemon left `/tmp/gh-aw/sandbox/firewall/` (or a parent) owned by **root**. With `--network-isolation` now the default, AWF runs without `sudo`, so `mkdirSync` on the root-owned parent fails with EACCES. | **Fixed in AWF (PR github/gh-aw-firewall#5983)**: added `preflight-reclaim.ts` — on non-root invocation, walks upward from the target path to find the first non-writable ancestor and removes it via `sudo rm -rf` with `fs.rmSync` fallback; protected paths (`/`, `/tmp`, `/home/runner`) are never touched. Workaround (older AWF): `sudo rm -rf /tmp/gh-aw/sandbox` before re-running. | `ls -la /tmp/gh-aw/sandbox/firewall/` — dirs owned by root (uid 0) confirm the mode; `docker info | grep -i rootless` | github/gh-aw-firewall#5983 | | ||
| | B9 | `No CA certificates were loaded from the system` — Copilot CLI or other HTTPS tools fail inside AWF chroot on RHEL, Fedora, or Amazon Linux runners; all HTTPS traffic returns TLS verification errors | AWF chroot mounts only Debian/Ubuntu CA paths (`/etc/ssl:ro`, `/etc/ca-certificates:ro`). On RHEL/Amazon Linux the system CA bundle lives under `/etc/pki/ca-trust/` which is not mounted. | **Fixed in AWF (PR github/gh-aw-firewall#5783)**: `copy_system_ca_bundle()` in agent entrypoint detects the CA bundle from 5 candidate paths (Debian, RHEL, Fedora, macOS, Alpine), copies it to `/tmp/awf-lib/system-ca-certificates.crt` if not directly accessible, and sets `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `GIT_SSL_CAINFO`. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set those env vars. | `ls /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` — present on RHEL/Amazon Linux confirms the mode | github/gh-aw-firewall#5733, github/gh-aw-firewall#5783 | | ||
| | B10 | `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` × 3 directories, each taking exactly ~30 s; total ~90 s wasted; cascading `EACCES: permission denied` / `Failed to remove chroot home directory after permission repair` | `fixArtifactPermissionsForRootless()` builds compound `tag@digest` image refs (e.g. `agent:0.27.22@sha256:55f065...`) for its `docker run --pull never` repair container. Despite `--pull never`, Docker still attempts registry manifest verification for compound refs; when GHCR credentials are unavailable (cleaned in an earlier workflow step or expired) the verification TCP-connects and times out (~30 s per directory). | **Fixed in AWF (PR github/gh-aw-firewall#6025)**: `resolvePermFixerImageRef()` now strips the `@sha256:...` digest and uses a tag-only ref. `--pull never` with a tag-only ref skips all registry I/O. Upgrade to AWF version that includes github/gh-aw-firewall#6025. Workaround (older AWF): ensure GHCR credentials are available until after the AWF cleanup step. | `[WARN] Rootless artifact permission repair failed` messages each followed by exactly ~30 s delay in the workflow log; `docker pull --dry-run agent:...@sha256:...` times out while `docker inspect agent:...` succeeds | github/gh-aw-firewall#6025 | | ||
| | B11 | Agent task succeeds but AWF process exits with code 1; logs show `[WARN] Rootless artifact permission repair failed for .../sandbox/firewall/logs (exit 1)` and `[WARN] Failed to remove chroot home directory after permission repair: Error: EACCES`; no actionable detail visible — repair container stderr was silently discarded | `fixArtifactPermissionsForRootless()` ran a `docker run` for chown/chmod but did not capture or log stderr, so the root cause was opaque. `removeWorkDirectories()` then escalated the subsequent `EACCES` from `fs.rmSync` on remapped-UID files to a fatal exit, overriding the successful agent exit code. | **Fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**: stderr from the repair container is now captured and included in the `[WARN]` message. Chroot-home removal failure is downgraded from `warn` to `debug` so it no longer propagates as a non-zero exit code — the post-step cleanup via `sudo` handles it. Upgrade to AWF including github/gh-aw-firewall#6072. | `[WARN] Rootless artifact permission repair failed` with no additional detail (no stderr), followed by `Command completed with exit code: 1` even though the agent log shows success | github/gh-aw-firewall#6070, github/gh-aw-firewall#6072 | |
| // B11 new failure mode | ||
| expect(content).toContain('| B11 |'); | ||
| expect(content).toContain('github/gh-aw-firewall#6070, github/gh-aw-firewall#6072'); | ||
| expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#6072, merged 2026-07-10)**'); | ||
| expect(content).toContain('| `[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success | B11 |'); |
| | ID | Signal | Root cause | Status | Probe | Citations | | ||
| |---|---|---|---|---|---| | ||
| | D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model | Known unresolved research area | `docker info | grep -i runtime`; inspect NAT rules in `awf-iptables-init` | #3264 | | ||
| | D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model in their default configuration | **gVisor (`runsc`) resolved** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor`; AWF detects `runsc` and applies the appropriate seccomp/capability profile. **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; use `--container-runtime gvisor` with AWF including github/gh-aw-firewall#6093 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093 | |
| | ID | Signal | Root cause | Status | Probe | Citations | | ||
| |---|---|---|---|---|---| | ||
| | D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model | Known unresolved research area | `docker info | grep -i runtime`; inspect NAT rules in `awf-iptables-init` | #3264 | | ||
| | D1 | AWF does not start or isolation is ineffective under gVisor / Kata | These runtimes do not support AWF's expected netns / capability model in their default configuration | **gVisor (`runsc`) resolved** in PR github/gh-aw-firewall#6093 (merged 2026-07-10): use `--container-runtime gvisor`; AWF detects `runsc` and applies the appropriate seccomp/capability profile. **Kata Containers** remain an unresolved research area. | `docker info | grep -i runtime`; use `--container-runtime gvisor` with AWF including github/gh-aw-firewall#6093 | github/gh-aw-firewall#3264, github/gh-aw-firewall#6093 | |
|
@copilot address review feedback |
Addressed in 53083c3. I updated all mirrored runner-doctor catalogs and the sync test to align with review feedback:
|
|
✅ Copilot review passed with no inline comments. @copilot Add the |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
✅ Smoke Gemini completed. All facets verified. 💎 Checking connectivity and file creation. |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
✅ Build Test Suite completed successfully! |
|
🔌 Smoke Services — All services reachable! ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🚀 Security Guard has started processing this pull request |
|
📰 VERDICT: Smoke Docker Sbx has concluded. All systems operational. This is a developing story. 🎤 |
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
Smoke Test: Claude Engine Validation
Overall Result: PASS ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test Results
Overall: PARTIAL — workflow template variables were not substituted (pre-step issue, not engine failure) PR author: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Copilot BYOK (Direct) Mode ✅✅ GitHub MCP connectivity Status: PASS Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY via api-proxy sidecar) Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🔍 Smoke Test Results✅ GitHub MCP — connected Overall: PASS | Author: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test: Copilot PAT Auth
Overall: PASS Author: Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test Results
Overall: FAIL — Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
PR titles:
✅ GitHub PR query Overall status: PASS Warning Firewall blocked 2 domainsThe following domains were blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"
- "registry.npmjs.org"See Network Configuration for more information.
|
Smoke Test: API Proxy OpenTelemetry Tracing
All scenarios pass. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Chroot Version Comparison
Not all runtimes matched — Go matched but Python and Node.js versions differ between host and chroot. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Smoke Test Results: Gemini Engine
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Smoke Test Results
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra Overall: PASS
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
|
✅ GitHub MCP connectivity (pre-fetched) Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Overall PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
Two new lessons from the 2026-06-26–07-10 scan window applied to all three mirrored catalog locations (
shared/self-hosted-failure-modes.md,self-hosted-runner-doctor.md,.github/agents/self-hosted-runner-doctor.md).D1 — gVisor partially resolved
runsc) is supported as of PR feat: gVisor runtime support #6093 via--container-runtime gvisor; AWF detectsrunscand applies the appropriate seccomp/capability profile. Kata Containers remain the only open research item.B11 — Silent repair failure overrides successful agent exit code
New failure mode: agent task completes successfully but
awfexits with code 1. Root cause was two-fold:fixArtifactPermissionsForRootless()discarded repair-container stderr, making the[WARN]message actionless.removeWorkDirectories()escalated the subsequentEACCESfromfs.rmSyncon remapped-UID files to a fatal exit, overriding the agent's zero exit code.Fixed in PR #6072: stderr is now captured and surfaced in the warning; chroot-home removal failure is downgraded from
warntodebug.Distinguishing signal vs B10 (which has ~30 s delays per attempt): B11 shows
[WARN] Rootless artifact permission repair failedwith no stderr detail followed immediately byCommand completed with exit code: 1despite apparent agent success.CI test
Updated
scripts/ci/self-hosted-runner-doctor-workflow.test.tsto assert the new B11 row, D1 content, updated Known-unresolved entry, and absence of the old D1 wording across all three catalog files.