[BUG] Inconsistent `compose publish` behavior with variables

[apollo13]

Description

It is not exactly clear from the docs how compose publish is supposed to handle variables. There is an --with-env flag which somehow seems to suggest that env variables are included in the OCI manifest. When using it docker compose asks whether sensitive variables are included, but the final manifest doesn't seem to include them anyways?

Steps To Reproduce

Using this compose file:

services:
  test1:
    image: nginx:latest
    command: ["${SECRET}"]
    environment:
      TEST: "${SECRET}"

and publishing it via:

SECRET=test docker compose publish localhost:5000/test:latest --insecure-registry

results in an error:

service "test1" has environment variable(s) declared.
To avoid leaking sensitive data, you must either explicitly allow the sending of environment variables by using the --with-env flag,
or remove sensitive data from your Compose configuration

So far so good, but publishing with --with-env doesn't really seem to result in leakage either:

SECRET=test docker compose publish localhost:5000/test:latest --insecure-registry --with-env                                                                                                                                                                                                                                                              1 ✘  14:40:46  
? you are about to publish environment variables within your OCI artifact.
please double check that you are not leaking sensitive data
Service/Config  test1
TEST=test
Are you ok to publish these environment variables? Yes
[+] push 1/1
 ✔ test1 Skipped                                                                                                                                                                                                                                                                                                                                                                       0.0s 
[+]  2/2t:5000/test:latest publishing 

When checking the published data:

regctl --host reg=localhost:5000,tls=disabled blob get localhost:5000/test:latest sha256:5868bab8f786242ba43d6604ea68ac5f4509457d6e8bfb3222ee7e85b4b9575f

it does not include the env variables:

services:
  test1:
    image: nginx:latest
    command: ["${SECRET}"]
    environment:
      TEST: "${SECRET}"

The result seems plausible to me, one usually does not want to publish with environment variables interpolated to allow for customization, but what is --with-env warning about then?

Compose Version

Docker Compose version v5.0.0-rc.2

[ndeloof]

The result indeed is the expected one, i.e the original compose file is published to support full customization by consumer.

This check was introduced with use of local .env file in mind, as some user rely on this to pass security tokens - which they obviously don't want to be made public - so the review step to confirm variables in use. Compose is not clever enough (or designed) to track where a specific variable was resolved for, and as such can't guess you just set SECRET on the command line and this has no effect.

[apollo13]

The result indeed is the expected one, i.e the original compose file is published to support full customization by consumer.

Perfect, that is what I thought. So how can I imagine this working? Does compose basically publish the output of docker compose config --no-interpolate (well that is what I would expect, but it does not, see below about required variables)?

This check was introduced with use of local .env file in mind, as some user rely on this to pass security tokens - which they obviously don't want to be made public - so the review step to confirm variables in use.

But how do those make it into the published compose ever (ie how to get it to actually leak anything)? Even if I create a .env file like:

SECRET=MY_SECRET_VERY_SECRET

and confirm that it interpolates correctly with docker compose config it will not end up in the published data…

Also if the compose file uses a variable that is required (ie "${REQUIRED?ERROR}") it will complain that it cannot publish the file unless I set the variable. But why, publishing happens without the variable anyways, so why require it to be set.

[ndeloof]

Does compose basically publish the output of docker compose config --no-interpolate

publish use your plain, unchanged, compose.yaml files to be published.

But how do those make it into the published compose ever

indeed, this is inconsistent. I wonder we should publish .env then (which would change command behavior and maybe put some users at risks) - or just drop all those useless checks (cc @glours)

Also if the compose file uses a variable that is required (ie "${REQUIRED?ERROR}") it will complain that it cannot publish the file

publish actually load the compose file as other docker commands, while this should not be strictly necessary. AFAICT it does because the loaded project is used to process those security checks.

[apollo13]

But how do those make it into the published compose ever

indeed, this is inconsistent. I wonder we should publish .env then (which would change command behavior and maybe put some users at risks) - or just drop all those useless checks (cc @glours)

Maybe only publish it if --with-env is selected and don't do the security checks otherwise?