[BUG] Unable to bind mount when mount source is within the daemon root

[bohde]

Description

After upgrading Compose v2.29.6 from v2.29.2, the following service began failing:

services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro

Running docker compose up -d gives the following error message:

Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "rprivate"

Steps To Reproduce

No response

Compose Version

Docker Compose version v2.29.6

Docker Environment

Client: Docker Engine - Community
 Version:    27.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.6
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 26.1.3
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.16-linuxkit
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.657GiB
 Name: d4d8c60ee1fe
 ID: 186ebd80-8c59-49b8-92ae-007c1c620318
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[zhaoqizqwang]

same thing happened to us with v2.29.6 today. Running docker compose -f docker-compose.yaml up --build -d and got Error response from daemon, asking to switch to using Volume Bind Mounts

[thaJeztah]

Looks related to;

• set propagation default #12133

The error is related to;

• Use rslave propagation for mounts from daemon root moby/moby#36055

cc @ndeloof @glours

[thaJeztah]

I guess as a workaround, it would work to explicitly set the bind propagation for that mount

[mariovitale1979]

had similar issue. Fixed by changing definition of volumes in cadvisor. I redeclared the / and the /var/lib/docker volumes in :

volumes:
- type: bind
source: /
target: /rootfs
read_only: true
bind:
propagation: rslave
- type: bind
source: /var/lib/docker/
target: /var/lib/docker
read_only: true
bind:
propagation: rslave
- /var/run:/var/run:rw
- /sys:/sys:ro

[ridoo]

We observed a similar issue (Note: we did not mount /var/lib/docker/).

We just had deployed our service under /var/lib/docker-compose/ (for whatever reason). It seems the daemon just checks via some startsWith and complains the above exception. Moving to a different directory solved the issue for us.

[ndeloof]

@thaJeztah here are the identified corner cases:

• we have to set propagation for WSL2 to be happy
• windows container don't support propagation to be set (must be left empty)
• daemon root require propagation to be rslave (while default is rprivate), but client can't guess what "daemon root" is.

based on those, I hardly can find a way for client to rely on mount API as a replacement for bind, as this require significant knowledge of the engine setup.

[glours]

Thanks @ndeloof for the summary, I'm totally aligned with your final remark

[thaJeztah]

daemon root require propagation to be rslave (while default is rprivate), but client can't guess what "daemon root" is.

based on those, I hardly can find a way for client to rely on mount API as a replacement for bind, as this require significant knowledge of the engine setup.

So those are the parts I'm confused about, because they should be handled automatically by the daemon, unless a specific option is set. I started to verify that, using the steps below;

Bind-mounting current directory

I omitted the "create" destination part (for --mount) in these tests.

Mount current directory with --mount

docker run -d --name foo --mount type=bind,src=.,dst=/somewhere nginx:alpine
3342e1206f3a78789e158aa6a0f5f5cd187bf8b481b0e3b6f33ada46fa1b225d

checking the request;

DEBU[2024-09-20T12:48:25.845152423Z] Calling POST /v1.47/containers/create?name=foo
DEBU[2024-09-20T12:48:25.845291798Z] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":null,"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":[],"BlkioDeviceReadIOps":[],"BlkioDeviceWriteBps":[],"BlkioDeviceWriteIOps":[],"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","CgroupnsMode":"","ConsoleSize":[32,139],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"DeviceRequests":null,"Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"Mounts":[{"Source":"/root/foo","Target":"/somewhere","Type":"bind"}],"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":[],"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"nginx:alpine","Labels":{},"NetworkingConfig":{"EndpointsConfig":{"default":{"Aliases":null,"DNSNames":null,"DriverOpts":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAMConfig":null,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","Links":null,"MacAddress":"","NetworkID":""}}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}

relevant part (HostConfig.Mounts):

[
  {
    "Source": "/root/foo",
    "Target": "/somewhere",
    "Type": "bind"
  }
]

Which matches container inspect;

docker container inspect --format '{{ json .HostConfig.Mounts}}' foo
[{"Type":"bind","Source":"/root/foo","Target":"/somewhere"}]

The .Mounts set for the container do get rprivate set;

docker container inspect --format '{{ json .Mounts}}' foo
[{"Type":"bind","Source":"/root/foo","Destination":"/somewhere","Mode":"","RW":true,"Propagation":"rprivate"}]

Checking the actual OCI spec, it includes rbind and rprivate;

ctr -n moby c info 3342e1206f3a78789e158aa6a0f5f5cd187bf8b481b0e3b6f33ada46fa1b225d | jq -c '[ .Spec.mounts[] |select( .destination | contains("/somewhere")) ]'
[{"destination":"/somewhere","type":"bind","source":"/root/foo","options":["rbind","rprivate"]}]

Mount current directory with -v

Trying the same with the -v option;

docker rm -fv foo
docker run -d --name foo -v .:/somewhere nginx:alpine
e05f380f942b4107299465ce96d7ef32a345476302528a845f9f26b4731bbceb

Checking the request:

DEBU[2024-09-20T12:53:36.116380427Z] Calling POST /v1.47/containers/create?name=foo
DEBU[2024-09-20T12:53:36.116530052Z] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":null,"HostConfig":{"AutoRemove":false,"Binds":["/root/foo:/somewhere"],"BlkioDeviceReadBps":[],"BlkioDeviceReadIOps":[],"BlkioDeviceWriteBps":[],"BlkioDeviceWriteIOps":[],"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","CgroupnsMode":"","ConsoleSize":[32,139],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"DeviceRequests":null,"Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":[],"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"nginx:alpine","Labels":{},"NetworkingConfig":{"EndpointsConfig":{"default":{"Aliases":null,"DNSNames":null,"DriverOpts":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAMConfig":null,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","Links":null,"MacAddress":"","NetworkID":""}}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}

Relevant part is .HostConfig.Binds;

[
  "/root/foo:/somewhere"
]

Which matches container inspect;

docker container inspect --format '{{ json .HostConfig.Binds}}' foo
["/root/foo:/somewhere"]

The .Mounts set for the container do get rprivate set;

docker container inspect --format '{{ json .Mounts}}' foo
[{"Type":"bind","Source":"/root/foo","Destination":"/somewhere","Mode":"","RW":true,"Propagation":"rprivate"}]

Checking the actual OCI spec, it includes rbind and rprivate;

ctr -n moby c info e05f380f942b4107299465ce96d7ef32a345476302528a845f9f26b4731bbceb | jq -c '[ .Spec.mounts[] |select( .destination | contains("/somewhere")) ]'
[{"destination":"/somewhere","type":"bind","source":"/root/foo","options":["rbind","rprivate"]}]

So far; the net result for the OCI spec and the .Mounts in the container config is identical.

Mounting /var/lib/docker

Mounting /var/lib/docker with --mount option

Trying the same with the --mount option;

docker rm -fv foo
docker run -d --name foo --mount type=bind,src=/var/lib/docker,dst=/somewhere nginx:alpine
1b6e9bb3d39156254da57c6017d14ca0f4e6361d84bbe487ae30fe16776b6ed9

Checking the request:

DEBU[2024-09-20T13:09:38.722705429Z] Calling HEAD /_ping
DEBU[2024-09-20T13:09:38.723268429Z] Calling POST /v1.47/containers/create?name=foo
DEBU[2024-09-20T13:09:38.723383095Z] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":null,"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":[],"BlkioDeviceReadIOps":[],"BlkioDeviceWriteBps":[],"BlkioDeviceWriteIOps":[],"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","CgroupnsMode":"","ConsoleSize":[32,139],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"DeviceRequests":null,"Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"Mounts":[{"Source":"/var/lib/docker","Target":"/somewhere","Type":"bind"}],"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":[],"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"nginx:alpine","Labels":{},"NetworkingConfig":{"EndpointsConfig":{"default":{"Aliases":null,"DNSNames":null,"DriverOpts":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAMConfig":null,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","Links":null,"MacAddress":"","NetworkID":""}}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}

Relevant part is .HostConfig.Mounts;

[
  {
    "Source": "/var/lib/docker",
    "Target": "/somewhere",
    "Type": "bind"
  }
]

Which matches container inspect;

docker container inspect --format '{{ json .HostConfig.Mounts}}' foo
[{"Type":"bind","Source":"/var/lib/docker","Target":"/somewhere"}]

The .Mounts set for the container does get rslave set (compoared to rprivate for other locations);

docker container inspect --format '{{ json .Mounts}}' foo
[{"Type":"bind","Source":"/var/lib/docker","Destination":"/somewhere","Mode":"","RW":true,"Propagation":"rslave"}]

Checking the actual OCI spec, it includes rbind and rslave (compoared to rprivate for other locations);

ctr -n moby c info 1b6e9bb3d39156254da57c6017d14ca0f4e6361d84bbe487ae30fe16776b6ed9 | jq -c '[ .Spec.mounts[] |select( .destination | contains("/somewhere")) ]'
[{"destination":"/somewhere","type":"bind","source":"/var/lib/docker","options":["rbind","rslave"]}]

Mounting /var/lib/docker with -v option

Trying the same with the -v option;

docker rm -fv foo
docker run -d --name foo -v /var/lib/docker:/somewhere nginx:alpine
a1c7f5cdf3c801664c9da6380d953c59c9bd65debe5e5e840a0b61c3d7cf03b8

Checking the request:

DEBU[2024-09-20T13:01:20.652415795Z] Calling HEAD /_ping
DEBU[2024-09-20T13:01:20.652997087Z] Calling POST /v1.47/containers/create?name=foo
DEBU[2024-09-20T13:01:20.653169170Z] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":null,"HostConfig":{"AutoRemove":false,"Binds":["/var/lib/docker:/somewhere"],"BlkioDeviceReadBps":[],"BlkioDeviceReadIOps":[],"BlkioDeviceWriteBps":[],"BlkioDeviceWriteIOps":[],"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","CgroupnsMode":"","ConsoleSize":[32,139],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"DeviceRequests":null,"Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":[],"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"nginx:alpine","Labels":{},"NetworkingConfig":{"EndpointsConfig":{"default":{"Aliases":null,"DNSNames":null,"DriverOpts":null,"EndpointID":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAMConfig":null,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","Links":null,"MacAddress":"","NetworkID":""}}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}

Relevant part is .HostConfig.Binds;

[
  "/var/lib/docker:/somewhere"
]

Which matches container inspect;

docker container inspect --format '{{ json .HostConfig.Binds}}' foo
["/var/lib/docker:/somewhere"]

Checking the actual OCI spec, it includes rbind and rslave (compoared to rprivate for other locations);

ctr -n moby c info a1c7f5cdf3c801664c9da6380d953c59c9bd65debe5e5e840a0b61c3d7cf03b8 | jq -c '[ .Spec.mounts[] |select( .destination | contains("/somewhere")) ]'
[{"destination":"/somewhere","type":"bind","source":"/var/lib/docker","options":["rbind","rslave"]}]

[thaJeztah]

Basically, in all of the above, the client does not set an explicit bind-propagation, and (AFAICS), the daemon automatically sets the correct one (either rprivate or rslave), so I'm curious why that failed on Docker Desktop (at least)

[ndeloof]

@thaJeztah the issue is that Docker Desktop with WSL2 require propagation to be set. Maybe this should better be fixed there