`.env` with `export` Lines No Longer Works

[berwyn]

Description of the issue

I recently upgraded Compose to 1.24.0-rc1 (build 0f3d4dda) and now Compose's automatic parsing of my .env file fails. I keep export statements in my .env file so I can easily source it in addition to using it as a standard .env. In previous versions of Compose, this worked fine and didn't give me any issues, however with this new update I instead get an error about spaces inside a value.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.0-rc1, build 0f3d4dda

Output of docker version

Docker version 18.09.1, build 4c52b90

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  db:
    image: mdillon/postgis:10-alpine
    ports:
    - 5432:5432/tcp
    volumes:
    - pgdata:/var/lib/postgresql/data:rw
  elasticsearch:
    environment:
      discovery.type: single-node
    image: docker.elastic.co/elasticsearch/elasticsearch:6.1.3
  memcached:
    image: memcached:1.5.10-alpine
  redis:
    image: redis:4.0.6-alpine
  web:
    build:
      context: /Users/berwyn/dev/<repo>
    depends_on:
    - db
    - elasticsearch
    - memcached
    - redis
    environment:
      DB_HOST: db
      DB_USERNAME: postgres
      FOUNDELASTICSEARCH_URL: elasticsearch
      REDIS_CACHE_HOST: redis
      # Several more removed variables here
    ports:
    - 3000:3000/tcp
    stdin_open: true
    tty: true
    volumes:
    - /Users/berwyn/dev/<repo>:/app:rw
version: '3.0'
volumes:
  pgdata: {}

Steps to reproduce the issue

1. Create a .env file with something like export FOO=1
2. Attempt to run a one-off container docker-compose run --rm service bash
3. Notice the error

Observed result

ERROR: In file ./.env: environment variable name `export FOO` may not contains whitespace.

Expected result

The container runs and FOO is correctly set in it.

Stacktrace / full error message

ERROR: In file ./.env: environment variable name `export FOO` may not contains whitespace.

Additional information

macOS 10.14.3
Docker & Friends installed using Cask (cask install docker-edge)

[esvm]

I'm getting the same error in a similar environment (MAC OS). Some news about this without rollback version?

[rene-s]

My guess is that this is a backwards-incompatible bugfix that has not been announced in advance. Not a new bug.

See https://github.com/docker/compose/blob/master/CHANGELOG.md#bugfixes

The best approach would be to avoid exporting variables in .env files.

edit:typo

[thaJeztah]

Looks related to #6403 @shin- @hirochachacha

[shin-]

I don't think that we ever supported this, but moreover, it was certainly never the intent: see our docs which clearly state

Compose expects each line in an env file to be in VAR=VAL format.

[hirochachacha]

Presumably, docker-compose's .env format has not been supporting export syntax since day one.

https://docs.docker.com/compose/env-file/

I also checked the following test case with old docker-compose version:

$ cat .env
VAR2=2
export VAR3=3
$ cat docker-compose.yaml
version: "3"
services:
  env_test:
    image: alpine
    environment:
      - VAR1=1
      - VAR2=${VAR2}
      - VAR3=${VAR3}
    command: env
$ docker-compose up && docker-compose down
WARNING: The VAR3 variable is not set. Defaulting to a blank string.
Creating network "d_default" with the default driver
Creating d_env_test_1 ... done
Attaching to d_env_test_1
env_test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
env_test_1  | HOSTNAME=f137dd513f8f
env_test_1  | VAR1=1
env_test_1  | VAR2=2
env_test_1  | VAR3
env_test_1  | HOME=/root
d_env_test_1 exited with code 0
WARNING: The VAR3 variable is not set. Defaulting to a blank string.
Removing d_env_test_1 ... done
Removing network d_default

As you can see, export VAR3=3 line in .env wasn't recognized by docker-compose.

I also tested source .env scenario, but I cannot see nice output:

$ cat .env
VAR2=2
export VAR3=3
$ cat docker-compose.yaml
version: "3"
services:
  env_test:
    image: alpine
    environment:
      - source .env
    command: env
$ docker-compose up && docker-compose down
Creating network "d_default" with the default driver
Creating d_env_test_1 ... done
Attaching to d_env_test_1
env_test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
env_test_1  | HOSTNAME=468523d0ef4b
env_test_1  | HOME=/root
d_env_test_1 exited with code 0
Removing d_env_test_1 ... done
Removing network d_default

So, if I understand correctly, this is not a bug.

[morganchristiansson]

Previously this was a warning and these lines were ignored. Now docker-compose fails with an error.

My setup was relying on this behaviour. For now I've downgraded to 1.23.2 which has old behaviour.

[hirochachacha]

I really cannot understand the use-case of the old behavior...
If someone find the misconfiguration, they should fix the configuration.
If you don't want to evaluate some lines, you could comment out these lines.
That makes your intentions clearer.

[morganchristiansson]

My usecase is currently to use default .env file to set COMPOSE_PROJECT_NAME and IMAGE_TAG

The same file is also used by a python application that has a very long JSON array that is split across many lines using single quotes. .env is read from inside the container by python-dotenv which permits much wider syntax.

I don't expect docker-compose to parse these values. The previous behaviour of ignoring them worked fine for me. With new behaviour I'll have to tell docker-compose not to use default .env file.

[hirochachacha]

Thank you for the response.

But I don't think copying .env into the docker image is a best practice.
In that approach, you need to recreate images per environments.
If you cannot apply .env settings on the fly, that's not a kind of envvars.
Perhaps, you can save it as settings.py or something.

You maybe know, a work-around for multi-line support is using base64 encoding.

[berwyn]

I have a much different use-case for this, actually. As per the original comment, I like to source my .env for use with orchestration scripts. Using dotenv libraries from Ruby, Node, .NET, etc are perfectly happy to ignore the export lines and parse the rest, and previously docker-compose was also happy to do this.

Very handy to support both docker-compose up and source .env && scripts/do-some-ci-thing.sh

[morganchristiansson]

@hirochachacha we use .env for application default values. Then actual environment variables take precedence.

Can we get old behaviour of warning of ignored lines back?

[hirochachacha]

I see. In the context of the original change, I'm fine with warning instead of error.
While I still feel those tricks are sort of implementation specific hack, but I have no strong opinion about that.

[jonmchan]

Please note that ruby's dotenv recommends or allows you to store the .env file with the caveat that you do not put any sensitive information in there. Since ruby's dotenv supports combining multiple .env files such as .env.local (which would be .gitignore'd and never committed), using .env file for general default environment variables is totally legitimate for that library.

Reference: https://github.com/bkeepers/dotenv#what-other-env-files-can-i-use