allow setting env vars only at build time

[juchem]

Description
This is a feature request to allow setting environment variables at build time, which don't linger around at container runtime.

This would be equivalent to adding the --env cli argument to docker build.

The best use case for this is proxy servers:
When building images in a CI/CD or development environment, it would be highly beneficial to have caching proxies for HTTP/HTTPS and package managers (e.g.: apt-cacher-ng). This could greatly speed up the development, build and test cycles; as well as relieve pressure on the outbound internet connection and local uncached artifact servers and whatnot.

The straightforward way one would expect to do this is through --build-arg and ARG. It is common, though, to have the default value for an environment variable be configurable at build time:

ARG http_proxy="..."
ENV http_proxy="${http_proxy}"

Overriding the value of ARG http_proxy at build time, in that case, would end up also overriding the value of ENV http_proxy at runtime.

One may argue that, for proxies, specifically, it's a bad idea to default a value in the image and it should be left empty, therefore the proper way to do this would be:

ARG http_proxy="..."  # use it at build time
ENV http_proxy=""  # leave it blank at runtime

That's generally true, but use-cases vary and it does make sense for very specific use-cases.
Also, environment variables aren't only used for proxy configurations. They're rather general tools, so a general solution is much in demand. It just so happens that the point is more easily expressed with proxy servers.

Other alternatives could look something like:

ARG build_time_http_proxy=""  # overridable at build time
ARG http_proxy=""  # default value for runtime
ENV http_proxy="${http_proxy}"

RUN [ -z "${build_time_http_proxy}" ] || export http_proxy="${build_time_http_proxy}"

That leads to a less-elegant solution, though, and doesn't work when the proxy is set in a base image. It requires mass editing all Dockerfiles and adding the above workaround.

The proposed solution would enable the intended effect to be obtained by simply adding --env https_proxy=... to docker build in the associated build system target, thus propagating to all images for free.

[tonistiigi]

Just set --build-arg http_proxy=foo. No Dockerfile changes required.

[juchem]

@tonistiigi I believe I've covered in the description why that doesn't work, unless I'm missing something

[tonistiigi]

 # cat Dockerfile
from alpine
run env && stop
 # docker build --build-arg http_proxy=foo .
[+] Building 0.4s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 34B                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                      0.0s
 => CACHED [1/2] FROM docker.io/library/alpine                                                                                                        0.0s
 => ERROR [2/2] RUN env && stop                                                                                                                       0.3s
------
 > [2/2] RUN env && stop:
#4 0.240 SHLVL=1
#4 0.240 HOME=/root
#4 0.240 http_proxy=foo
#4 0.240 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#4 0.240 PWD=/
#4 0.240 HTTP_PROXY=foo
#4 0.241 /bin/sh: stop: not found
------

[juchem]

@tonistiigi this is a good workaround in the case one is not building the base image along using the same build system

Here's a different minimalist scenario:

Base image:

FROM alpine
ARG http_proxy=""
ENV http_proxy="${http_proxy}"

# do stuff that can benefit from a proxy server at build time

Dependent image:

FROM our-base-image

# do stuff that can benefit from a proxy server at build time

Note that there are legitimate cases where one may want to embed a default proxy server in the image for runtime. E.g.: distributing a standardized build environment - proxy should be set for internal use, but unset for external distribution of the build image.

Also, in the case of internal use, one my want to force a local proxy server at build time, but point to a load balancer for internal runtime use.

To illustrate the ideas above:

• build internal use images with build time proxy = "local.proxy", default runtime proxy = "lb.proxy"
• build exernal use images with build time proxy = "local.proxy", default runtime proxy = ""

Btw, don't get me wrong. If you can help me find a workaround, I'd rather go with that and solve my problem right away.

Last but not least, thanks for following up on this.

[tonistiigi]

Dockerfile RUN on top of FROM image and docker run image have the same behavior (without any extra arguments provided to either). It would be very messy if that was not the case. Use a different image or provide extra flag for either side for different behavior.

In my example no proxy settings are left to the resulting image.

[thaJeztah]

proxy should be set for internal use, but unset for external distribution of the build image.

This is where the (available by default) XX_PROXY build-args can be useful, as they will be used during build (internal use), but don't persist in the distributed image.

You can configure proxies to use in the CLI configuration file (~/.docker/config.json); see #93 and https://docs.docker.com/network/proxy/#configure-the-docker-client. Those proxies will be automatically set as build-args, and at runtime set as environment variable for containers.

I would recommend against setting a ENV HTTP_PROXY=... in your Dockerfile, as that would bake the proxy configuration in your image, which means that the image is not portable (as in; it would require the person running the image to have the specified proxy available); specifying a proxy should be set at runtime, so that the correct proxy (if any) can be specified for the environment in which the container needs to run. That said, if the container at "runtime" is started using the docker CLI, of course the same configuration can be used in the ~/.docker/config.json. Alternatively (if HTTP_PROXY is set in the docker CLI environment), you can start containers with --env HTTP_PROXY (without passing a value), which should copy the value from HTTP_PROXY in the current environment).

[juchem]

@tonistiigi: I can understand how the suggestion can solve the problem in theory, but not without duplication of effort either in the form of extra build targets hierarchy or workarounds in the images themselves. The fault lies on me, due to the fact that I haven't clearly presented the specifics of the problem, which would require sharing more information than I can afford to.

@thaJeztah part of the problem is a communication issue. I've used proxies as an illustrative example only, but now the discussion has been framed around it, which wasn't my intention.

My takeaway at this point is that ARG and ENV aren't intended to be used in conjunction (e.g.: the former being used to parameterize the latter). At least not for my intended use case. What I'm left to do is devise a way to employ ARG for build time settings only, and ENV for runtime settings only.

I can see how to do it, but not in a rather elegant way.

It may just as well be that my use case is legitimate, but specific enough that it doesn't warrant a feature request.

Should there be wider interest from the community for the proposed feature, I'd be glad to see it implemented.

For the time being, I'll just look for a different approach that solves my problem, taking the suggestions above into account.

I deeply appreciate the follow ups and suggestions.

[thaJeztah]

Yes, both act in the same namespace (are available as env-vars during build), so order can be important;

FROM alpine
ARG VAR1=var1-default-arg
ENV VAR1=${VAR1:-var1-default-env}
ENV VAR2=var2-default-env
ARG VAR2=var2-default-arg
RUN echo $VAR1 $VAR2

Without passing --build-arg (using defaults):

docker build --no-cache --progress=plain -t foo .
...
#5 0.213 var1-default-arg var2-default-arg

docker inspect --format='{{json .Config.Env}}' foo
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VAR1=var1-default-arg","VAR2=var2-default-env"]

With passing --build-arg:

docker build --no-cache --progress=plain -t foo --build-arg VAR1=var1-cli --build-arg VAR2=var2-cli .
...
#5 0.226 var1-cli var2-cli
...
docker inspect --format='{{json .Config.Env}}' foo
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VAR1=var1-cli","VAR2=var2-default-env"]

If you need a combination of "bake some variable into the image, but use a different value at build-time", something like this could work:

• ENV VAR variable has a default value, but can be overridden using the VAR_RUNTIME build-arg (if set)
• ARG VAR also has a default value (could be omitted), and can be overridden using the VAR build-arg

FROM alpine
ARG VAR_RUNTIME
ENV VAR=${VAR_RUNTIME:-var-default-env}
ARG VAR=var-default-arg
RUN echo $VAR

Building without --build-arg set, will use teh default buildtime value during build:

docker build --no-cache --progress=plain -t foo .
...
#5 0.216 var-default-arg

And persist the default runtime value in the image:

docker run --rm foo sh -c 'echo $VAR'
var-default-env

docker inspect --format='{{json .Config.Env}}' foo
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VAR=var-default-env"]

And both can be set individually during build; this uses VAR=var-cli-buildtime during build:

docker build --no-cache --progress=plain -t foo --build-arg VAR=var-cli-buildtime --build-arg VAR_RUNTIME=var-cli-runtime .
...
#5 0.245 var-cli-buildtime
...

And VAR=var-cli-runtime in the image (so, as default at runtime)

docker run --rm foo sh -c 'echo $VAR'
var-cli-runtime

docker inspect --format='{{json .Config.Env}}' foo
["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","VAR=var-cli-runtime"]

[juchem]

@thaJeztah this is exactly what I needed. I did not realize that order matters. It was also unclear to me how ARG affected the environment variables of a container.

I'll close the issue since there's an elegant solution for it.

I hope this discussion serves as documentation for anyone looking for this feature.

Thank you very much for your time and knowledge.