Skip to content

fix(docker): stale bind-mount inode to Docker Desktop on up/delete/up#519

Merged
skevetter merged 6 commits into
mainfrom
fix/docker-desktop-bind-mount-stale-inode
Jun 23, 2026
Merged

fix(docker): stale bind-mount inode to Docker Desktop on up/delete/up#519
skevetter merged 6 commits into
mainfrom
fix/docker-desktop-bind-mount-stale-inode

Conversation

@skevetter

@skevetter skevetter commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Problem

Workspace up fails intermittently on Docker Desktop after an up → delete → up cycle. The daemon rejects the workspace bind mount with:

docker: Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /host_mnt/.../workspace-1/agent/content-...

...even though os.Lstat confirms the host path exists. Symptom: the workspace cannot be brought up twice consecutively (up → delete → up again fails).

Root cause

Docker Desktop's file-sharing layer (VirtioFS/gRPC-FUSE) caches a stale inode for the workspace's recreated parent directory.

  1. workspace delete runs os.RemoveAll(.../workspaces/<id>).
  2. The next up recreates the same .../agent/ path with a new host inode.
  3. The VM still maps the old inode and cannot resolve children created under it, so the bind mount fails. Retrying the same mount in place never succeeds (verified: a 24s retry loop never recovered, while a totally fresh path mounts in ~0.2s).

This was reproduced deterministically:

Setup Result
Recreated parent + unique leaf (current content-<uid>) cycle 1 OK, cycles 2–5 POISONED
Stable parent + unique leaf 5/5 OK
bind-create-src=true across recreated-parent cycles 6/6 OK, real content

PR #512's content-<uid> leaf didn't help because the poisoned directory is the parent agent/, which the lookup must traverse regardless of the leaf name.

Fix

Add bind-create-src=true to the workspace bind mount on runtimes that parse it (Docker only; Podman/nerdctl reject unknown mount options). This forces the daemon to re-resolve the source through the file share — busting the stale inode — and because the directory really exists, it binds the real content, not an empty placeholder.

The option is gated on an os.Stat existence check: a genuinely missing source is left untouched so docker still fails loudly (exit 125 bind source path does not exist) instead of bind-create-src silently materializing an empty directory and starting the container against an empty workspace.

Also drops the now-pointless content-<uid> suffix back to a plain content path — it never delivered the protection it was added for, and bind-create-src is the real fix.

Summary by CodeRabbit

  • New Features

    • Added Docker Desktop 29+ support for automatic bind-mount source re-resolution to handle potentially stale-cached mount paths.
  • Bug Fixes

    • Improved workspace content directory cleanup during deletion to better preserve host inode references across recreate operations.
    • Simplified workspace content directory naming structure for more reliable mount handling.
  • Refactor

    • Streamlined workspace content directory path management.

Workspace `up` failed intermittently on Docker Desktop after an
up -> delete -> up cycle: the daemon rejected the workspace bind mount
with "bind source path does not exist: /host_mnt/..." even though the
host path existed.

Root cause is Docker Desktop's file-sharing layer caching a stale inode
for the workspace's recreated parent directory. `workspace delete` does
os.RemoveAll on .../workspaces/<id>, and the next `up` recreates the
same .../agent/ path with a new host inode; the VM still maps the old
one and cannot resolve children created under it. Retrying the same
mount in place never succeeds.

PR #512's content-<uid> leaf did not help because the poisoned directory
is the parent agent/, which the lookup must traverse regardless of the
leaf name.

Fix: add bind-create-src=true to the workspace bind mount on runtimes
that parse it (Docker only). This forces the daemon to re-resolve the
source through the file share, busting the stale inode; because the
directory really exists it binds the real content. The option is gated
on an os.Stat existence check so a genuinely missing source still fails
loudly (exit 125) instead of being silently created as an empty dir.

Also drop the now-pointless content-<uid> suffix back to a plain
"content" path, since it never delivered the protection it was added
for and bind-create-src is the real fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@netlify

netlify Bot commented Jun 22, 2026

Copy link
Copy Markdown

Deploy Preview for devsydev canceled.

Name Link
🔨 Latest commit db0e986
🔍 Latest deploy log https://app.netlify.com/projects/devsydev/deploys/6a3abbc95838e80009bbad50

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 18412d7a-0b85-4cf0-9e4e-865a1bce8af4

📥 Commits

Reviewing files that changed from the base of the PR and between 51c028f and db0e986.

📒 Files selected for processing (10)
  • cmd/internal/agentworkspace/up.go
  • pkg/agent/agent.go
  • pkg/agent/workspace.go
  • pkg/client/clientimplementation/workspace_client.go
  • pkg/config/pathmanager.go
  • pkg/config/pathmanager_linux_test.go
  • pkg/docker/helper.go
  • pkg/driver/docker/docker.go
  • pkg/driver/docker/docker_test.go
  • pkg/provider/dir.go
💤 Files with no reviewable changes (1)
  • cmd/internal/agentworkspace/up.go

📝 Walkthrough

Walkthrough

The PR removes the UID suffix from the agent workspace content directory (replacing content-<uid> with a fixed content path), adds WorkspaceContentsDir/WorkspaceContentDir to PathManager and provider helpers, reworks resolveContentFolder to support host-invocation routing, cleans up content folder leaves on workspace deletion, and adds bind-create-src=true injection for Docker Desktop bind mounts on non-Linux hosts with Docker CLI ≥ 29.

Changes

Workspace content directory path simplification

Layer / File(s) Summary
PathManager interface and provider helpers
pkg/config/pathmanager.go, pkg/provider/dir.go, pkg/config/pathmanager_linux_test.go
Adds WorkspaceContentsDir and WorkspaceContentDir to the PathManager interface and basePathManager, and exposes GetWorkspaceContentsDir/GetWorkspaceContentDir as provider-level helpers. Tests validate both new paths resolve under the contents subdirectory.
GetAgentWorkspaceContentDir and resolveContentFolder
pkg/agent/workspace.go, pkg/agent/agent.go, cmd/internal/agentworkspace/up.go
Removes the uid parameter from GetAgentWorkspaceContentDir, changes the returned path from content-<uid> to content, reworks resolveContentFolder to early-return if ContentFolder is already set, use provider2.GetWorkspaceContentDir for host invocations, and fall back otherwise. Updates the prepareWorkspace call site to match.
DeleteWorkspaceFolder content leaf cleanup
pkg/client/clientimplementation/workspace_client.go
Adds best-effort removal of the workspace content directory leaf inside DeleteWorkspaceFolder while preserving the parent contents directory to keep the host inode stable across workspace recreates.

Docker bind-cache busting

Layer / File(s) Summary
bind-create-src helpers and mount integration
pkg/docker/helper.go, pkg/driver/docker/docker.go, pkg/driver/docker/docker_test.go
Adds DockerHelper.ClientVersion, shouldBustBindCache, dockerMajorAtLeast, and withBindCreateSrc; wires them into addWorkspaceMountArgs to conditionally append bind-create-src=true for Docker Desktop bind mounts on non-Linux hosts with CLI ≥ 29. Tests cover withBindCreateSrc idempotency and dockerMajorAtLeast edge cases.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • devsy-org/devsy#74: Introduced the PathManager interface that this PR extends with WorkspaceContentsDir and WorkspaceContentDir.
  • devsy-org/devsy#443: Introduced IsHostAgentInvocation and host-vs-container routing semantics that the reworked resolveContentFolder now branches on.

Suggested labels

size/m

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main fix: addressing stale bind-mount inode issues on Docker Desktop during the up/delete/up cycle.
Docstring Coverage ✅ Passed Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@netlify

netlify Bot commented Jun 22, 2026

Copy link
Copy Markdown

Deploy Preview for images-devsy-sh canceled.

Name Link
🔨 Latest commit db0e986
🔍 Latest deploy log https://app.netlify.com/projects/images-devsy-sh/deploys/6a3abbc995e52f00094906e6

The bind-create-src mount option is only parsed by the docker CLI v29+,
and only relevant on Docker Desktop. Emitting it unconditionally broke
every `up` on Linux CI (docker 28.0.4), which rejects the unknown mount
key at parse time:

    invalid argument ... unexpected key 'bind-create-src'

Gate it three ways instead of the previous runtime-name-only capability:
  - Docker runtime (Podman/nerdctl reject the option)
  - Docker Desktop only (GOOS != linux); native Linux binds the host
    path directly and never hits the file-share inode cache
  - docker CLI >= v29, detected via `docker version`; older clients
    error out, which would turn the intermittent bug into a hard failure

Adds DockerHelper.ClientVersion and replaces SupportsBindCreateSrc() with
shouldBustBindCache()/dockerMajorAtLeast() in the driver.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The up -> delete -> up bind-mount failure on Docker Desktop is caused by
the bind-mount source's PARENT directory being delete+recreated at the
same host path. Docker Desktop's file share caches the parent inode;
once it is recreated, the VM can no longer resolve bind sources beneath
it. The content folder lived at workspaces/<id>/agent/content, entirely
inside the workspaces/<id> tree that `delete` removes with RemoveAll, so
every delete churned the parent inode.

Move the host-side content folder to a per-context contents/<id> dir.
The contents/ parent is created once and never removed on delete (only
the <id> leaf is), so its host inode stays stable across recreate and
the file share keeps resolving the bind source.

Verified: up/delete/up cycles pass 5/5 with bind-create-src disabled
(i.e. this fixes Docker Desktop versions older than v29 that lack the
flag), and inode tracking confirms contents/ is never recreated while
the leaf is.

bind-create-src is retained as a fast path / extra safety on v29+.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added size/l and removed size/m labels Jun 23, 2026
skevetter and others added 2 commits June 23, 2026 11:17
Condense the explanatory comments added for the bind-mount fix to concise
doc comments and one-liners.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the content-folder cleanup if-init statement so the line stays
under the 100-char golines limit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@skevetter skevetter changed the title fix(docker): bust Docker Desktop stale bind-mount inode on up/delete/up fix(docker): stale bind-mount inode to Docker Desktop on up/delete/up Jun 23, 2026
@skevetter
skevetter marked this pull request as ready for review June 23, 2026 19:11
@skevetter
skevetter merged commit 65a0c08 into main Jun 23, 2026
100 of 104 checks passed
@skevetter
skevetter deleted the fix/docker-desktop-bind-mount-stale-inode branch June 23, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant