Fix COMPOSE_PARALLEL_LIMIT

[pagelypete]

Description of the issue

There have already been multiple issues opened about this bug:
#7486
#5864

A PR already seems to exist which fixes the issue which was posted with reference to #5864 - could this be checked, tested, and merged so this bug can finally be fixed in master?

The bug causes a fairly critical issue, it completely deadlocks compose, and in the worst case scenario, it also seems to be able to cause interactions with the Docker daemon to hang due to the amount of open connections it is holding.

The bug is worse when your compose file is dynamically generated, and can contain varying numbers of services, which means that you have to set a global system wide environment variable to be an arbitrarily high number - completely defeating the point of the variable in the first place (to reduce CPU usage for container operations).

Anecdotally, I have tested that PR on 1.28.5, and up/down/restart/etc. operations all seem to work. In a service file with 150 services, this deadlocks on master:

COMPOSE_PARALLEL_LIMIT=2 docker-compose up -d --remove-orphans

With the linked PR, it does not, and actually functions as you would expect (essentially sequentially bringing up the containers)

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.28.5, build unknown
docker-py version: 4.4.4
CPython version: 3.8.6
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Steps to reproduce the issue

1. Create a docker-compose.yml file with a lot of services (ideally 100+)
2. Set COMPOSE_PARALLEL_LIMIT to 2 and run an up operation

Observed result

Compose will deadlock.

Expected result

Compose should bring up containers using the parallel limit.

Stacktrace / full error message

N/A

Additional information

Not relevant

[devZer0]

hello, we also have massive problems with docker-compose. it does not work reliable to start containers on a host . we use docker-compose part of a ci/cd environment and jenkins job fail multiple times a day because docker-compose will not reliably return "success" to bring a bunch of containers online on a remote host via ssh.

[devZer0]

now, months later this is still open and not further commented. is docker-composed orphaned/abandoned software nobody cares for ?

[ndeloof]

Have you tried Docker Compose v2? With a new golang codebase, I guess concurency is better addressed.

[pagelypete]

Have you tried Docker Compose v2? With a new golang codebase, I guess concurency is better addressed.

I personally have not tried v2 however the issue here was the load caused by doing so many Docker operations at once, not any issues with the performance of compose itself. Is there a way to limit parallel Docker operations in v2 similar to COMPOSE_PARALLEL_LIMIT (except working properly)?

Either way, it would be really great to get this changed merged, we have been using it in production for a long time now and have to build our own docker-compose just to use it, and it really does seem to fix the parallel problem and make COMPOSE_PARALLEL_LIMIT work properly. Servers that would otherwise be hugely overloaded can instead work normally.

[pagelypete]

@ndeloof I gave v2 a try and I don't see any way to limit parallelism, and while I can see that performance of compose itself has been hugely improved, I'm concerned about not seeing a way to limit what it sends to the Docker API in v2 at all (I grepped the source a little).

For example if you have a compose file with 200 containers, that can otherwise function fine on a system, running docker-compose up results in an absolutely massive load spike, whereas with a parallel limit in place it can be done more slowly but with no system load issues at all.

[devZer0]

we also cannot use v2 as there is no mechanism to limit parallelism. our servers are getting totally overloaded if there are too many services getting started in parallel, i.e. services fail to start correctly if there is too much load, so please either fix v1 or add COMPOSE_PARALLEL_LIMIT to v2

[devZer0]

we are also using the patch in production now, with good results so far

[devZer0]

i'm very sad that nobody cares and this is being ignored

[ondravondra]

We also ran into the issue of massive spike in memory and cpu usage when building. A way to limit parallelism is a must have for us.