Please provide a way to limit parallelism in Compose v2

[pagelypete]

Description

This issue is a report/feature request for Compose v2 but is essentially the same as this problem in v1 - #8226

Having no limit to parallelism makes compose effectively unusable when the compose file contains more than a few dozen services, as it consumes all resources on the host system when doing operations like bringing containers up.

Currently we have no way to migrate to compose v2 because of this and indeed we are stuck on v1 with a third party patch.

Please provide a way to limit parallelism.

Steps to reproduce the issue:

1. Have a compose file with some container definitions that do something for a few seconds. Have at least 50 in your compose file.
2. Run docker compose up -d --remove-orphans (or equivalent)
3. Watch system slow to a crawl and resource usage spike hugely, depending on the resources the host has

Describe the results you received:

As above.

Describe the results you expected:

As above, since there is no way to limit parallelism.

Additional information you deem important (e.g. issue happens only occasionally):

N/A

Output of docker compose version:

Docker Compose version v2.0.0-rc.3

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-rc.3)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 271
 Server Version: 20.10.10
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-38-generic
 Operating System: Ubuntu 21.04
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.98GiB
 Name: pagely
 ID: B6KI:5FVQ:KIOY:RIHT:PP55:IKJ4:QWV4:X6LH:RYYD:NYLN:QFWL:K6K4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

[hroemer]

Confer buildkit #1032 → new setting MaxParallelism (only on daemon start). I guess unless buildkit #593 or buildkit #1131 are addressed, it seems to be a "won't fix".

[pagelypete]

Confer buildkit #1032 → new setting MaxParallelism (only on daemon start). I guess unless buildkit #593 or buildkit #1131 are addressed, it seems to be a "won't fix".

Well, for the building part of it sure, but compose can independently provide a parallelism limit for container operations which is mostly what this issue is about as opposed to building.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pagelypete]

Issue is not stale.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[oocx]

We could use this as well. We have a docker compose file with 11 services. If all containers need to build first, 11 builds will try to do a "nuget restore" at the same time. This takes so much bandwidth at the same time that many of them run into timeout problems and fail.

The only and really annoying workaround I found is to manually comment out some of the services in my docker compose and build that first, so that not all of them are built at the same time.

[mmguero]

An easier workaround that doesn't involve modifying your compose file is to use yq and xargs:

$ yq '.services | keys | .[]' docker-compose.yml | xargs -r -P 1 -I XXX docker compose build "XXX"

If you want to, replace the 1 with however many concurrent builds you want.

[Holzhaus]

This issue is probably also related: #9837