[BUG] Renew Anon Volumes doesn't work as expected compared to v1

[ryanneufeld]

Description

-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving data from the previous containers.

This functions differently than in V1

Steps To Reproduce

With docker-compose v1 it was possible to run docker-compose up -dV and have all the containers restarted with fresh anonymous volumes.

Currently this no longer works expected and up simply updates any containers with new versions fetched via pull

version: "3"
services:
  mysql:
    restart: unless-stopped
    image: mysql:5.7
    ports:
      - "3306:3306"
    expose:
      - "3306"
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"

Compose Version

Docker Compose version v2.18.1

Docker Environment

MacOS Ventura 13.4.1 (22F82)

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/rneufeld/.docker/cli-plugins/docker-scout

Server:
 Containers: 19
  Running: 1
  Paused: 0
  Stopped: 18
 Images: 17
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 5.805GiB
 Name: docker-desktop
 ID: a21ae72f-a439-4ca4-b8f2-4a3afa43de1e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

The example use-case I have is a mysql container that comes up and populates the database with pre-installed SQL. With docker-compose v1, I was able to run docker-compose up -dV and the mysql container would be restarted with a fresh anon volume, and the DB would be imported freshly. With V2 I must down the pool to achieve the same outcome.

[dolzenko]

I'm seeing the similar issue

Client: Docker Engine - Community
Version: 24.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.2
Path: /home/ev/.docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan

Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 285
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.2.0-27-generic
Operating System: Ubuntu 23.04
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 15.32GiB
Name: katana
ID: eb11b5d1-b704-4e65-a495-26c4ac1d77f5
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Docker Compose version v2.20.2

[ndeloof]

That's indeed not a strict backward compatibility here, but with Docker Compose v2 in the wild for some time I'd prefer we don't change this behavior now

As a simple workaround, you can use --force-recreate to get the desired behavior:

$ docker compose up -d
[+] Running 2/2
 ✔ Network chose_default    Created                                        0.0s 
 ✔ Container chose-mysql-1  Started                                        0.5s 

$ docker inspect chose-mysql-1 --format '{{.Mounts}}'
[{volume 624433dd888107e5259f01ebb29c9ce74166c96604690713ebfd3f203e3369a9 /var/lib/docker/volumes/624433dd888107e5259f01ebb29c9ce74166c96604690713ebfd3f203e3369a9/_data /var/lib/mysql local  true }]

$ docker compose up --force-recreate -d
[+] Running 1/1
 ✔ Container chose-mysql-1  Started                                        0.9s 

 $ docker inspect chose-mysql-1 --format '{{.Mounts}}'
[{volume 624433dd888107e5259f01ebb29c9ce74166c96604690713ebfd3f203e3369a9 /var/lib/docker/volumes/624433dd888107e5259f01ebb29c9ce74166c96604690713ebfd3f203e3369a9/_data /var/lib/mysql local z true }]
# Same volume has been (re)used with recreated container

$  docker compose up --force-recreate -dV
[+] Running 1/1
 ✔ Container chose-mysql-1  Started                                        1.6s 

$ docker inspect chose-mysql-1 --format '{{.Mounts}}'
[{volume 0e49be586b09dbfde54782822bdd529af9b988d012eecbf3944820e7b9291d8b /var/lib/docker/volumes/0e49be586b09dbfde54782822bdd529af9b988d012eecbf3944820e7b9291d8b/_data /var/lib/mysql local  true }]
 # A fresh new volume has been created while re-creating container

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[rneufeldcisco]

I strongly feel that this should be fixed. Since -V doesn't do what it says on the tin, that is a bug.

 -V, --renew-anon-volumes        Recreate anonymous volumes instead of retrieving data from the previous containers.
 

The fact the software has been out for some time only makes this bug more annoying. This flag is part of our entire organizations workflow, and it's resulting in us spending more time than necessary on internal support.

[ndeloof]

@rneufeldcisco considering --force-recreate --renew-anon-volumes offers a simple workaround and makes this behavior more explicit, in addition to risk for other users to consider any change here as a regression, I'm closing this issue as "not planned"

[ryanneufeld]

This is disappointing. Can you explain the risk?

…

On Wed., Sep. 11, 2024, 00:25 Nicolas De loof, ***@***.***> wrote: @rneufeldcisco <https://github.com/rneufeldcisco> considering --force-recreate --renew-anon-volumes offers a simple workaround and makes this behavior more explicit, in addition to risk for other users to consider any change here as a regression, I'm closing this issue as "not planned" — Reply to this email directly, view it on GitHub <#10774 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEO5BWG5NRBRX4MXHJMFXDZV7V7LAVCNFSM6AAAAABN7Z2R72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBSHA3DGNRXGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ndeloof]

Looking at compose v1 codebase and how this used to be implemented I notice it was far for being clever, and just forced all containers to be recreated: https://github.com/docker/compose/blob/v1/compose/cli/main.py#L1291-L1292
i.e --renew-anon-volumes involves --force-recreate. We can do the same with compose v2