Improve Compose extends and make it less verbose and more powerful.

[Vad1mo]

Currently compose extends is very verbose.

We use extends in the way that we have a base file and then for each environment we replace the significant parts that we want to override or have different.

In the current state of extends if you want to achieve the same you need almost rewrite everything.
Look at out actual example and how verbose it is. There are only two lines that matter.

common.yaml

elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"

child.yaml

elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

With the new extends feature one would add a global_extends/import like this:

extended_child.yaml

global_extends: 
 - common.yaml
 - extra.yaml 

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx

Thats it.

Advantages

1. A less verbose code is easy to maintain and understand
2. You can add remove service in the base.yaml without adding/removing entries in the children. (Missing a container in Prod because someone forgot to ad it as into the prod.yaml.
3. You can aggregate services into one.
4. The configuration can be easily created by CI or or maintained by an application.
5. You have a nice overview what gets set and you can check in this in a repo.
6. This will make Interpolate environment variables in docker-compose.yml #1377 and Pass variables inside fig.yml on runtime #495 obsolete and 75 +1er happy :)

Main featues

• append nodes that are declared in parent yaml
• modify/overwrite nodes declared in parent
• add new nodes
• the only thing you can't do is to removed nodes
  see examples in Improve Compose extends and make it less verbose and more powerful. #1380 (comment)

[dnephin]

I think this is very similar to what I proposed in #318 , and it's a feature I'm still very interested in. There's a bunch of complexity with something like this, so it's taking a while to get there, but I think we're getting closer.

[aanm]

Also #1361, but I agree, the optimal would be docker-compose up --extends ../common.yml and that's it, all the merge would be automagical since the services' names would be the same in ../common.yml and docker-compose.yml

[Vad1mo]

indeed #318 and #1361 has the same intention. @dnephin I am wondering why it is such a complex task? Does this go beyond merging two yaml files into one tree.

[aanm]

@Vad1mo Don't forget that extends merge both configuration depending on each option. For example:

In the case of build and image, using one in the local service causes Compose to discard the other, if it was defined in the original service.

(@Vad1mo you have a malformed link on the previous comment)

[andrerom]

"global_extends" could just be called "import", here is format used in Symfony framework, addopted/simplified a bit:

# Any config blocks imported here can be selectively be changed below
# wouldn't normally have one file per service/container, this is simplified example
imports:
  - ../parameters.yml
  - app/rails.yml
  - db/mysql.yml

# override some stuff
rails:
  environment:
    SESSION_SECRET: SomeSecret

parameters.yml

# Additional concept: parameters, don't need to change on each machine where the app is deployed
# http://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
# Would typically be the place to use env variables and defaults when #1377 is in, to keep it in 1 place
parameters:
  app_env: prod

rails.yml

rails:
# (...)
  links:
    - db
  environment:
    RACK_ENV: "%app_env%"

mysql.yml

db:
# (...)

[Vad1mo]

@andrerom
There is a conceptual difference between extends and import. Import is for composing and orchestrating services together to create something new/big out of small peaces.
Example: Frontend + backend + db merged together into one service file to start everything together.

On the other side extends means you want to extend or modify/override the behavior of the services you already have in place. You extend something for a specific situation.
Example: add/change port or set env var. of prod.

[andrerom]

I agree, but then what you imply is that your global_extends does not really import any of the services, so your example would need to be extended so you define all services again (with a extends.service part, just saving repeating the extends.file part). So your example to me looks more like an import, where you redefine part of the doorman service, exactly like imports does in my example.

[erichaus]

@Vad1mo @andrerom I am new to docker/compose etc and working on a new microservice based project... I'm running into issue where I wanted to have a top level repo with just a docker-compose.yml that orchestrates all the pieces together e.g.

Example: Frontend + web gateway service + content service + user auth service + async service etc together into one service file to start everything together.

It seems that the current extends implementation doesnt support links from child compose files so I'm wondering if what you guys are talking about applies to this case (a master compose.yml that ties all the services together and individual compose.yml files within each service that is decoupled mostly and can have links etc)

Just curious thx!

[Vad1mo]

@erichonkanen what you describe was initially not in my intention but the proposal would also allow this kind of behavior.

[aanand]

@erichonkanen What you describe is something we've talked about before - see #758 - and which we should continue to explore.

[aanand]

I'd like to think more about "global extends". It would be a much less verbose solution to the problem of configuring an app for multiple environments, which is one of the primary things users are asking for better ways to do.

Let's imagine the simplest possible implementation: you can import all services from a single Compose file, and can then add new services and reconfigure imported ones.

base.yml

web:
  build: .
  links:
    - db

db:
  image: postgres

docker-compose.yml

EXTENDS: base.yml

web:
  ports:
    - "8000:8000"
  volumes:
    - .:/code

production.yml

EXTENDS: base.yml

frontend:
  image: registry.mycompany.com/reverse-proxy
  ports:
    - "80:8000"
  links:
    - web

web:
  image: registry.mycompany.com/webapp

I'm a bit scared of this feature because of the amount of implicit-ness, but it seems to be the only sensible way to avoid the huge amount of boilerplate currently needed to make small changes to a large file.

As for extending multiple files (as in @Vad1mo's example), the potential for confusion there seems much greater. What use cases would it serve?

[erichaus]

@aanand I am wondering if the use case of a microservice based architecture is a use case? Currently Im working on a new project using both docker and microservices for the first time... compose is a neat and convenient tool but the one thing I wish it had were the ability to decouple extended services/repos more cleanly, aka not have to put the links in the top level docker-compose.yml.

Something like this would be sweet:

orchestrate.yml

service-content:
  import:
    file: ../service-content/docker-compose.yml

service-auth:
  import:
    file: ../service-auth/docker-compose.yml

service-accounting:
  import:
    file: ../service-accounting/docker-compose.yml

service-content/docker-compose.yml

web-db:
  image: postgres:latest

web:
  build: .
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - .:/code
  ports:
    - "8000:8000"
  links:
    - web-db

Which would make containers prefixed by the top-level import name e.g. service-content_web-db_1

What are your thoughts on something like this? Right now we are definining everything in the orchestrate.yml file which works but it couples everything to that file/repo whereas it'd be ideal to allow each service to define it's own config/build/link instructions etc...