Description
Separating out an issue described on #9600
[BUG] Compose v2 is needlessly recreating containers when NO changes have been made. Towards the end of the troubleshooting, you will see I make a change to an image tag and run "docker compose up -d" multiple times, and it recreates all the containers every time.
I have spent some more time troubleshooting this.
Using this docker-compose.yml. Note that no services rely on image3:
services:
image1:
image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image1:3.24-pdns-815"
restart: always
env_file:
- .env_image1
network_mode: host
image2:
image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image2:4.5.10-pdns-817"
restart: always
env_file:
- .env_image2
network_mode: host
depends_on:
- image1
- image4
image3:
image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image3:1.7.0-pdns-818"
restart: always
env_file:
- .env_image3
network_mode: host
depends_on:
- image2
- image4
image4:
image: "XXX.dkr.ecr.ap-southeast-2.amazonaws.com/image4:1.1.0-15-pdns-816"
restart: always
env_file:
- .env_image4
network_mode: host
Performed the following:
[root@dockerhost ]# docker compose up -d
[root@dockerhost ]# docker compose config --hash "*"
image3 ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
image1 26f81a297463e0ffc24ba98a820c26a3543ae6c4bdaab3341b245442d4a0dbc6
image4 6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8
image2 ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57
All the hashes are correct:
[root@dockerhost ~]# docker inspect -image3-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
[root@dockerhost ~]# docker inspect -image1-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
26f81a297463e0ffc24ba98a820c26a3543ae6c4bdaab3341b245442d4a0dbc6
[root@dockerhost ~]# docker inspect -image4-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8
[root@dockerhost ~]# docker inspect -image2-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57
Then, I changed the image tag of image1:
[root@dockerhost ~]# docker compose up -d
[root@dockerhost ]# docker compose config --hash "*"
image2 ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57
image3 ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
image1 5615a16902d9d1af30efcf431c17a683205d8b4d5c6d39a17677cf90bce5dce5 < -------------------- This changed as expected
image4 6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8
[root@dockerhost ]#
[root@dockerhost ]# docker inspect image3-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ce9447046b6b4356a16296de91f39f4b4ff927f263ade3d26f2e9396b41203f6
[root@dockerhost ]# docker inspect image1-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
5615a16902d9d1af30efcf431c17a683205d8b4d5c6d39a17677cf90bce5dce5
[root@dockerhost ]# docker inspect image4-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
6a66dd02ef2dc447934770824b5eb71b48a5e036a0d24542d22fb603466019a8 < -------------------- This changed as expected
[root@dockerhost ]# docker inspect image2-1 -f '{{index .Config.Labels "com.docker.compose.config-hash"}}'
ed9a8168cee4947db92b3b0261dec13d9a4cf39b379378745ccfa83f84bfba57
Only image1-1 hash changed (which is expected), however, all other containers are recreated (except image3, again noting none of the other services depend on it).
I have tested this on compose 2.16.0, 2.15.1 and even 2.0.1 and all have the same behviour.
I retested using the latest version of compose v1 i.e 1.29.2 and the issue is NOT present, so it clearly it something to do with how v2 handles dependant images:
[root@dockerhost ]# docker-compose up -d
Creating image3 ... done
Creating image1 ... done
Creating image2 ... done
Creating image4 ... done
# change image tag of image1-1
[root@dockerhost ]# docker-compose up -d
image3 is up-to-date
Recreating image1 ... done <------------------------------------ only this container got changed (as expected)
image2 is up-to-date
image4 is up-to-date
Then, I went back to 2.16.0 to do some more testing:
Using compose 2.16.0:
[root@dockerhost ]# docker compose up -d
[+] Running 4/0
⠿ Container image3-1 Running 0.0s
⠿ Container image4-1 Running 0.0s
⠿ Container image2-1 Running 0.0s
⠿ Container image1-1 Running 0.0s
[root@dockerhost ]# docker compose up -d
[+] Running 4/0
⠿ Container image3-1 Running 0.0s
⠿ Container image4-1 Running 0.0s
⠿ Container image2-1 Running 0.0s
⠿ Container image1-1 Running 0.0s
[root@dockerhost ]# docker compose up -d
[+] Running 4/0
⠿ Container image4-1 Running 0.0s
⠿ Container image3-1 Running 0.0s
⠿ Container image2-1 Running 0.0s
⠿ Container image1-1 Running
Made no changes, and looks good, no action taken. And no changes after running the command 3 times in a row.
Again, modified the image tag of 1 of the images:
[root@dockerhost ]# docker compose up -d
[+] Running 4/4
⠿ Container image4-1 Started 21.0s < -------------- got recreated
⠿ Container image3-1 Running 0.0s
⠿ Container image2-1 Started 20.9s < -------------- got recreated
⠿ Container image1-1 Started 10.8s < -------------- got recreated
********<I did NOT make any changes here, I simply ran compose up again, and again it recreated the containers! And yes, they took EXACTLY the same amount of time to create>***********
[root@dockerhost ]# docker compose up -d
[+] Running 4/4
⠿ Container image4-1 Started 21.0s < -------------- got recreated
⠿ Container image3-1 Running 0.0s
⠿ Container image2-1 Started 20.9s < -------------- got recreated
⠿ Container image1-1 Started 10.8s < -------------- got recreated
Tried it a few more times, and it still recreates the containers:
[root@dockerhost ]# docker compose up -d
[+] Running 4/4
⠿ Container image3-1 Running 0.0s
⠿ Container image4-1 Started 21.2s
⠿ Container image2-1 Started 21.1s
⠿ Container image1-1 Started 10.7s
[root@dockerhost ]# docker compose up -d
[+] Running 2/3
⠿ Container image4-1 Recreated 0.2s
⠿ Container image3-1 Running 0.0s
⠧ Container image2-1 Recreate 2.7s
[root@dockerhost ]# docker compose config --hash "*"
XXX 3aa471602b2be184bbc2f78ba579c08e841e92d488f6dc6c499717c512bf0636
XXX 20b943ef0b80c65fcbba74cfdcc8c9ca8758cd7533a4f9383ca062ba7812a969
XXX 00765ce04ab7fd8166d8de909a69e366edd51f7bbf67390e7e69d998b5674b6b
XXX f364475068952084b4ff86358831ffd95e020ed84cd5e99b5d43871c61f5ba66
[root@dockerhost ]# docker compose up -d
[+] Running 4/4
⠿ Container image4-1 Started 21.3s
⠿ Container image3-1 Running 0.0s
⠿ Container image2-1 Started 21.2s
⠿ Container image1-1 Started 10.7s
XXX 00765ce04ab7fd8166d8de909a69e366edd51f7bbf67390e7e69d998b5674b6b
XXX f364475068952084b4ff86358831ffd95e020ed84cd5e99b5d43871c61f5ba66
XXX 3aa471602b2be184bbc2f78ba579c08e841e92d488f6dc6c499717c512bf0636
XXX 20b943ef0b80c65fcbba74cfdcc8c9ca8758cd7533a4f9383ca062ba7812a969
[root@dockerhost ]#
Steps To Reproduce
As per description.
Compose Version
Docker Compose version v2.16.0
Docker Environment
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.16.0)
scan: Docker Scan (Docker Inc., v0.21.0)
Server:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 6
Server Version: 20.10.21
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1160.80.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.06GiB
Name: XXX
ID: GR4X:Q35F:5E4H:TGFY:O34K:JEYN:WYBW:3QY5:4G7N:TXP6:4GLE:QOG2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Anything else?
Refer to #9600 for alot of history.
Description
Separating out an issue described on #9600
[BUG] Compose v2 is needlessly recreating containers when NO changes have been made. Towards the end of the troubleshooting, you will see I make a change to an image tag and run "docker compose up -d" multiple times, and it recreates all the containers every time.
I have spent some more time troubleshooting this.
Using this docker-compose.yml. Note that no services rely on image3:
Performed the following:
All the hashes are correct:
Then, I changed the image tag of image1:
Only image1-1 hash changed (which is expected), however, all other containers are recreated (except image3, again noting none of the other services depend on it).
I have tested this on compose 2.16.0, 2.15.1 and even 2.0.1 and all have the same behviour.
I retested using the latest version of compose v1 i.e 1.29.2 and the issue is NOT present, so it clearly it something to do with how v2 handles dependant images:
Then, I went back to 2.16.0 to do some more testing:
Made no changes, and looks good, no action taken. And no changes after running the command 3 times in a row.
Again, modified the image tag of 1 of the images:
Tried it a few more times, and it still recreates the containers:
Steps To Reproduce
As per description.
Compose Version
Docker Environment
Anything else?
Refer to #9600 for alot of history.