[BUG] Compose Develop does not sync changes to services without build section

[sfc-gh-kbregula]

Description

Hello,

My docker-compose file consists of one image, but it is used by multiple services. So I defined three services:

• The first service contains the build definitions and image name. This service builds image.
• The second and third service uses this image and depends on first service.

I found this trick on Stackoverflow:
https://stackoverflow.com/questions/40899236/how-to-prevent-docker-compose-building-the-same-image-multiple-times

Next, I wanted to add integration with docker compose watch, so I added develop sections like below:

• The first service rebuilds the image if dependencies are changed.
• Second and third service sync application code

Unfortunately, if my services with the application do not have a build section, the files are not synchronized without any error message, if I add the build section, everything starts working, but this causes docker compose build to build the same image multiple times after executing docker compose build.

From what I understand from the source code this is not the expected behavior. The sync action should work without a build section. The rebuild action requires a build section.

compose/pkg/compose/watch.go

Lines 337 to 339 in 11c7a25

	if trigger.Action == types.WatchActionRebuild && service.Build == nil {
	return nil, fmt.Errorf("service %s doesn't have a build section, can't apply 'rebuild' on watch", service.Name)
	}

Steps To Reproduce

I have a simple application as below:

#./src/main.py
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()

# requirements.txt
flask

# Dockerfile
# syntax = docker/dockerfile:1.9
# check=error=true

FROM python:3.10-slim-bullseye

WORKDIR /app

RUN apt-get update; apt-get install -y dumb-init curl

COPY requirements.txt /app/

RUN pip install -r requirements.txt

ENV PYTHONPATH=/app/src
ENV FLASK_APP=/app/src/main.py

COPY ./src /app/src

RUN curl "http://host.docker.internal:9500/$(hostname)" || true

ENTRYPOINT ["/usr/bin/dumb-init", "--"]

# docker-compose.yaml
services:
  my-app-image:
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    command: ['echo', 'The docker image (${COMPOSE_PROJECT_NAME}-docparser) is ready to be used.']
    develop:
      watch:
        - path: ./requirements.txt
          action: rebuild

  my-app-instance-1:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false
    environment:
      INSTANCE_NAME: instance1
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    ports:
      - "3001:3001"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3001' ]
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src

  my-app-instance-2:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false

    image: ${COMPOSE_PROJECT_NAME}-my-app
    environment:
      INSTANCE_NAME: instance2
    ports:
      - "3002:3002"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3002' ]
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src

Now I run my environment:

docker compose build --no-cache
docker compose up --wait

And then I run Compose Watch

docker --debug --log-level=debug compose watch

I have the following output:

DEBU[0000] Enabled Docker Desktop integration (experimental) @ unix:///Users/my-user/Library/Containers/com.docker.docker/Data/docker-cli.sock
DEBU[0000] Watch configuration for service "my-app-instance-1":
  - Action sync for path "/Users/my-user/compose_develop_seperate_image/src"
DEBU[0000] Watch configuration for service "my-app-image":
  - Action rebuild for path "/Users/kbregula//compose_develop_seperate_image/requirements.txt"
Watch enabled
DEBU[0005] otel error                                    error="<nil>"

As you can see, one service is not tracked.

In a separate terminal, I check if synchronization is working.

$ curl localhost:3001
INSTANCE_NAME: instance1
REQUIREMENTS:
flask

MAIN_APP:
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()

$ curl localhost:3002
INSTANCE_NAME: instance2
REQUIREMENTS:
flask

MAIN_APP:
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()

Then I make a change in the source code:

echo "# TEST $(date)" >> src/main.py

Next I checked again:

$ curl -s localhost:3001 | grep TEST
# TEST Sat Aug 17 17:42:27 CEST 2024

$ curl -s localhost:3002 | grep TEST

Here's the full log.

$ docker --debug --log-level=debug compose watch --no-up
DEBU[0000] Enabled Docker Desktop integration (experimental) @ unix:///Users/kbregula/Library/Containers/com.docker.docker/Data/docker-cli.sock
DEBU[0000] Watch configuration for service "my-app-image":
  - Action rebuild for path "/Users/my-user/compose_develop_seperate_image/requirements.txt"
DEBU[0000] Watch configuration for service "my-app-instance-1":
  - Action sync for path "/Users/my-user/compose_develop_seperate_image/src"
Watch enabled
DEBU[0005] otel error                                    error="<nil>"
DEBU[0090] change for /Users/my-user/compose_develop_seperate_image/src/main.py - comparing with /Users/my-user/compose_develop_seperate_image/src
DEBU[0090] batch start: service[my-app-instance-1] count[1]
Syncing "my-app-instance-1" after changes were detected
DEBU[0090] batch complete: service[my-app-instance-1] duration[29.016667ms] count[1]
DEBU[0095] otel error                                    error="<nil>"

I will add that if a change is made in dependencies (requirements.txt), the image is correctly updated.

$ echo "### TEST2 $(date)" >> requirements.txt
curl -s localhost:3001 | grep TEST2
### TEST2 Sat Aug 17 17:45:45 CEST 2024
$ curl -s localhost:3002 | grep TEST2
### TEST2 Sat Aug 17 17:45:45 CEST 2024

Compose Version

Docker Compose version v2.29.1-desktop.1

Docker Environment

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/my-user/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/my-user/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/my-user/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/my-user/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/my-user/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/my-user/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/my-user/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 2
  Paused: 0
  Stopped: 8
 Images: 18
 Server Version: 27.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 16.07GiB
 Name: docker-desktop
 ID: 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/my-user/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

No response

[idsulik]

compose/pkg/compose/watch.go

Line 131 in 11c7a25

continue

here is the line where it skips the service, because next line requires the build context.
watch.LoadDockerIgnore(service.Build.Context)

compose/pkg/compose/watch.go

Line 138 in 11c7a25

dockerIgnores, err := watch.LoadDockerIgnore(service.Build.Context)

[ndeloof]

each of your service should be fully defined, even those share a common image. To avoid copy-pasting the build definition you can use extends or rely on yaml anchors to get the same build section applied to other services