Compose volume `subpath:` option fails with `invalid mode`

[rpatterson]

Description

When a compose project service uses a volume with the subpath: option as documented, the service fails with a confusing invalid mode error.

Reproduce

1. Create the following compose project:

   # mkdir -pv /tmp/foo/qux
   mkdir: created directory '/tmp/foo'
   mkdir: created directory '/tmp/foo/qux'
   # touch /tmp/foo/qux/bah.txt
   # cat /tmp/foo/docker-compose.yml
   volumes:
   
     foo_volume:
       driver: "local"
       driver_opts:
         type: "none"
         o: "bind"
         device: "/tmp/foo"
   
   services:
   
     bar-service:
       image: "ubuntu"
       command: "ls -alR /tmp/foo/"
       volumes:
         - type: "volume"
           source: "foo_volume"
           target: "/tmp/foo"
           volume:
             subpath: "qux"
   

2. Note the resulting error when run:

   # docker compose down --remove-orphans -v && docker compose up
   [+] Running 3/3
    ✔ Container foo-bar_service-1  Removed                                                                                                                                                  0.1s
    ✔ Volume foo_foo_volume        Removed                                                                                                                                                  0.0s
    ✔ Network foo_default          Removed                                                                                                                                                  0.9s
   [+] Running 2/3
    ✔ Network foo_default          Created                                                                                                                                                  0.3s
    ✔ Volume "foo_foo_volume"      Created                                                                                                                                                  0.1s
    ⠼ Container foo-bar-service-1  Creating                                                                                                                                                 0.3s
   Error response from daemon: invalid mode: rw,qux
   

Expected behavior

The subpath from the volume should be mounted inside the serive's container.

docker version

Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:29 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:29 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

# docker info >./docker-info.log
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
# cat ./docker-info.log
Client: Docker Engine - Community
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 34
  Running: 33
  Paused: 0
  Stopped: 1
 Images: 38
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Pop!_OS 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.31GiB
 Name: library.moodysalonnet
 ID: 930e1cc3-4d06-44a2-828b-b203b25ad7a9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

N/A

Additional Info

No *diagnose* tool is available from this installation:

# aptitude search -F '%p' ~i~ndocker
docker-buildx-plugin
docker-ce
docker-ce-cli
docker-ce-rootless-extras
docker-compose-plugin
nvidia-docker2
# aptitude search -F '%p' '~i~ndocker' | xargs -d '\n' -- dpkg -L | grep 'diagnose'

[stivyw]

I have the same issue.

[willsunnn]

Faced this issue with Docker client and engine versions 27.0.3, and docker compose version v2.28.1.
Tried upgrading to 27.1.1 and docker dompose version v2.29.1, but still no luck.

[stivyw]

This is a Docker issue.

[rpatterson]

This is a Docker issue.

Got a link, @stivyw?

[stivyw]

They changed the docker compose documentation to create sub directories after create the volume.
So I think that's not a real problem, is only because the docker does not create the sub dirs when not exists into created volume.
Maybe one way to solve this, is when putting the subdir for an volume, create that if not exists and not return any error.
This is the changed documentation in last week build.
https://github.com/docker/docs/pull/20577/files

Reference for some issues (Have many questions in slack too)
docker/cli#5826
moby/moby#47842
moby/moby#45687

[rpatterson]

They changed the docker compose documentation to create sub directories after create the volume. So I think that's not a real problem, is only because the docker does not create the sub dirs when not exists into created volume.

But in the example in the description, you can see I do create the sub-directory before bringing up the service, so this doesn't sound like what you describe or what's added to the docs in that diff, @stivyw.

[stivyw]

Yeap, I got it now. :)
Here when I create the dir, haven't problem.
I will sleep but Tomorrow I will compare my versions.

[stivyw]

@rpatterson sorry the confusing, I was commented the use of local driver, so I changed to normal volume.

volumes:
  balance:
    name: sws-balance
    # driver: local
    # driver_opts:
    #   o: bind
    #   type: none
    #   device: /sw/etc/balance
...

With local driver and opts, I continue with the same issue.

Error response from daemon: invalid mode: rw,nocopy,www

BUT When volume is external, it works:

mkdir -p $PWD/volume/etc $PWD/volume/www
touch $PWD/volume/etc/ETC $PWD/volume/www/WWW
docker volume create -d local -o type=none -o o=bind -o device=$PWD/volume testvol

docker-compose.yaml

volumes:
  myvol:
    name: testvol
    external: true
    # driver: local
    # driver_opts:
    #   o: bind
    #   type: none
    #   device: $PWD/volume
services:
  test:
    image: alpine
    command: sh -l -c 'ls /myvol && ls /var/myvol/* && ls /etc/myvol/*'
    volumes:
      - myvol:/myvol
      - type: volume
        source: myvol
        target: /var/myvol
        volume:
          # nocopy: true
          subpath: www
      - type: volume
        source: myvol
        target: /etc/myvol
        volume:
          nocopy: true
          subpath: etc

Result

sws:/sw/apps/test$ docker compose up
[+] Running 2/1
 ✔ Network test_default   Created                                                                                                                                                             0.2s 
 ✔ Container test-test-1  Created                                                                                                                                                             0.1s 
Attaching to test-1
test-1  | etc
test-1  | www
test-1  | /var/myvol/WWW
test-1  | /etc/myvol/ETC
test-1 exited with code 0
sws:/sw/apps/test$ 

So, I think maybe this is a docker-compose issue.
docker/compose#12075 (I opened this now)

[rpatterson]

BUT When volume is external, it works:

Ah, thanks for the workaround!

So, I think maybe this is a docker-compose issue. docker/compose#12075 (I opened this now)

Thanks for that too, the docs were unclear about the right place to open an issue so I ended up opening this one here because it was the only one the docs pointed to.

[stivyw]

Nice. I think the solution is in progress.
docker/compose#12078