fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak - #85
Draft
CarlesUIPath wants to merge 1 commit into
Draft
fix(docker): harness isolation via uid-drop barrier — close the criteria/grader leak#85CarlesUIPath wants to merge 1 commit into
CarlesUIPath wants to merge 1 commit into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if errors is not None: | ||
| errors.append((path, exc)) | ||
| else: | ||
| logger.debug("container_perms: could not chown/chmod %s: %s", path, exc) |
CarlesUIPath
changed the title
CarlesUIPath
force-pushed
the
fix/docker-harness-isolation
branch
from
August 5, 2026 14:57
83aad3c to
61a157f
Compare
…eria/grader leak Under --driver docker, run the agent-under-test's CLI subprocess as an unprivileged `agent` uid (2000) while the container stays root, so grading material (success_criteria, task_full.json, skills-repo graders check_*.py, RESOLUTION.md, reference agents, the per-task-dir mount, reference solutions) is root-owned mode-0700 and returns EACCES to the agent, while grading runs as root. Closes the leak where agents read their own answer key (measured ~2.4% of nightly replicates; ~100% under an adversarial "check /work" prompt). Covers claude-code / codex / antigravity: a setpriv drop shim, claude via ClaudeAgentOptions.user, codex via launch_args_override, antigravity via the localharness PATH-shadow. Criteria stripped from the agent-readable task.yaml (agent_safe_dump) and restored root-only from task_full.json before grading (raw-dict merge). Hardening found via multi-model review + real containerized runs: - forward raw plugin/skills-repo host mounts via context.json and lock them - mount locked staging dirs read-write so the in-container 0700 chmod applies (a :ro mount EROFS-no-op'd the lock silently); lock now fails LOUD - relocate HOME/CODEX_HOME to an agent-owned /home/agent under the drop - lock the reference-solution mount (reference.file / reference.directory) - reject agent_run_uid authored from YAML/CLI/variant (framework-set only) - CE033 lint guards the container_perms choke point, forbids a container --user, and flags a raw task.model_dump into the staged yaml Test coverage: `make test-docker-isolation` runs a root-in-container six-surface EACCES-as-uid-2000 proof + a real dropped-CLI acceptance inside the built image (new `docker-isolation` CI job). The barrier was independently confirmed with an out-of-tree reachability harness (reproduce vs verify-fixed) showing the jailbreak reproduces on the vulnerable image and is EACCES under the barrier. Barrier is Linux-authoritative (native overlayfs); macOS Docker Desktop's bind-mount uid-remap makes local bind-mount checks unreliable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CarlesUIPath
force-pushed
the
fix/docker-harness-isolation
branch
from
August 5, 2026 15:05
61a157f to
e67c70b
Compare
| # the barrier sets, not mkdtemp's default. | ||
| anc = tmp_path | ||
| while anc != anc.parent and str(anc).startswith("/tmp"): | ||
| os.chmod(anc, 0o711) # traverse-only (o+x); NOT world-readable |
| os.chmod(anc, 0o711) # traverse-only (o+x); NOT world-readable | ||
| anc = anc.parent | ||
| for extra in (output_dir, output_dir / "artifacts"): | ||
| os.chmod(extra, 0o711) # traverse-only (o+x); NOT world-readable |
| # exercise the LEAF (agent-owned ws) perms, not mkdtemp's default. | ||
| anc = tmp_path | ||
| while anc != anc.parent and str(anc).startswith("/tmp"): | ||
| os.chmod(anc, 0o711) # traverse-only (o+x); NOT world-readable |
| # Make the temp ancestor chain traversable (production /work mounts are). | ||
| anc = tmp_path | ||
| while anc != anc.parent and str(anc).startswith("/tmp"): | ||
| os.chmod(anc, 0o711) # traverse-only (o+x); NOT world-readable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under
--driver docker, the agent-under-test could read its own grading material —success_criteria, the skills-repo graders (check_*.py),RESOLUTION.md, referenceagents, the staged
task.yaml, and the per-task-dir mount. A colleague's nightly auditmeasured ~2.4% of replicates reading the suite instead of solving the task (claude-code
highest); an adversarial "check /work" prompt drives it to ~100% across frontier and
open-weight models. Prior denylist mitigations (zip/base64/compile/delete) did not hold.
This installs an allowlist permission barrier: the container stays root, but the agent's
CLI subprocess is dropped to an unprivileged
agentuid (2000). All grading material isroot-owned
chmod 0700→ EACCES to the agent, while grading (SuccessChecker/run_command/ judges) runs as root. The agent can read only its skill-docs and write itsown workspace.
How it works
setprivshim; claude viaClaudeAgentOptions.user, codex vialaunch_args_override, antigravity via thelocalharnessPATH-shadow. The orchestrator/grading process never leaves root.
task.yamlis criteria-stripped(
agent_safe_dump); the real criteria/reference ride root-only intask_full.jsonand aremerged back (raw-dict, before parsing) for grading.
container_perms.lock_harness_root_0700/grant_agent_ownership,guarded by lint rule CE033.
/work/input,task_full.json)check_*.py,RESOLUTION.md, reference agents)reference.file/directory)TASK_DIR,SKILLS_REPO_PATH)📖 Setup & design reference: the agent-uid ↔ root-grader arrangement, how tasks are
staged/authored under it, the barrier surfaces, the Docker-Desktop caveat, and the nightly
rollout are documented in
docs/DOCKER_ISOLATION.md(extended in this PR).Verification
New Docker test coverage (permanent, CI-gated). Adds a root-in-container acceptance
suite (
tests/test_docker_user_separation.py, run by the newdocker-isolationCI job viamake test-docker-isolation): it forks a child dropped to the agent uid and asserts EACCESon all six surfaces + a real dropped-CLI check (
id -u == 2000, HOME writable, grader/task_full.jsonunreadable) inside the freshly built image on native Linux — theauthoritative environment (see caveat).
Independently confirmed the jailbreak reproduces → is closed. Beyond the committed
suite, an out-of-tree reachability harness was run in two modes on the same image:
reproduce(barrier omitted) shows the six surfaces ARE readable to the dropped uid —reproducing the jailbreak — and
verify-fixed(barrier applied via the real primitives)shows they become EACCES. The before/after pair proves the fix actually closes the leak
(and that the check isn't vacuous).
Local: gates green; the Docker acceptance suite passes; full
make test= 4009 passedwith only pre-existing/environmental failures (live-API judge tests needing creds, local
TASK_DIR/task-tag pollution, a stale model-name pin, and report-command output stringsthat live on
main) — none touch this diff.Out of scope / follow-ups
tempdirdriver + the Windows nightly slice (no container/uid boundary) — separate.