docker --context remotehost compose up -d shows WARN[0006] commandConn.CloseWrite

[StefanScherer]

Description

I tried docker compose up with an SSH remote context. A warning appear on each up command when using such a remote context.
Other commands like docker --context remotehost compose ps or docker --context remotehost compose down work fine without this warning.

Steps to reproduce the issue:

1. docker context create remotehost --docker "host=ssh://user@remotehost"
2. docker --context remotehost compose up -d

Describe the results you received:

$ docker --context remotehost compose up -d
WARN[0006] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
...

Describe the results you expected:

The warning shouldn't appear.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

$ docker version
Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:57 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:13 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:
You can also run docker context inspect context-name to give us more details but don't forget to remove sensitive content.

default

Output of docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.942GiB
 Name: docker-desktop
 ID: P7BD:XDIY:BKCQ:7BID:S55U:Y7Q6:5AYS:W2WZ:KL2R:HULO:I5S6:HIBD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS ECS, Azure ACI, local, etc.):

Target machine is an AWS t4g ARM64 machine, but it also happened connection to a local Rasperry Pi ARM64 ubuntu box.

[ndeloof]

Is this still relevant now we have switched to a CLI plugin architecture?
@StefanScherer seems you have an easy way to reproduce this with your Raspberry Pi, can you please confirm?

[StefanScherer]

@ndeloof It seems to work better now.
But sporadically I see this warning

$ docker --context remotehost compose up
[+] Running 1/0
 ⠿ Container nginx-golang_frontend_1  Recreated                                                                    0.0s
Attaching to frontend_1
frontend_1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
frontend_1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
frontend_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
WARN[0001] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
frontend_1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
frontend_1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
frontend_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
frontend_1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
frontend_1  | /docker-entrypoint.sh: Configuration complete; ready for start up
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: using the "epoll" event method
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: nginx/1.21.0
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6) 
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: OS: Linux 5.11.0-11-generic
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: start worker processes
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: start worker process 32
frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: start worker process 33

Or when I press Ctrl+C

frontend_1  | 2021/06/23 12:32:39 [notice] 1#1: start worker process 33
^CGracefully stopping... (press Ctrl+C again to force)
WARN[0060] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 
[+] Running 1/1
 ⠿ Container nginx-golang_frontend_1  Stopped                                                                      0.2s
canceled

But when nobody else complains I'm fine when we close this issue. 😺

[ndeloof]

ok thanks for confirmation.
I don't want this issue to be closed, as still reproducible - I prefer we find the root cause before this becomes an actual issue for more users :P

[ducan-ne]

I got this issue too, when "upping" compose file from mac to ssh.
My mac docker version: 20.10.6
Ssh version: 20.10.5

[champloo]

I am seeing similar issues with these warning message. Additionally sometimes the connection terminates

error during connect: Post "http://docker/v1.41/containers/b2bf9d6f62744423025e27a4cb570bc1c0aca1b54a9db4ced036ff1cf9f8ac69/start": command [ssh -- test-deployment docker system dial-stdio] has exited with signal: killed, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

The warning mesasge is readily reproducible with using a dind image with ssh in it, say https://github.com/EugenMayer/docker-image-dind-sshd. And then setting up either DOCKER_HOST or the context to point to the host.

The error message I have been getting in a Jenkins CI/CD environment, but have not been able to reproduce locally. It only started happening once we switched to docker compose V2. We are using the rc1 release.

[plashenkov]

I see these warnings too when using DOCKER_HOST=ssh://user@address with docker-compose up -d command.

[emersonf]

It only started happening once we switched to docker compose V2. We are using the rc1 release.

Thanks @champloo, that just saved me a lot of grief. When using a Docker context with endpoint ssh://foo, I get frequent errors (most runs) with Compose V2 (on Docker Desktop 4.0.1 on macOS), and none with docker-compose (and the Use Docker Compose V2 feature disabled). That SSH host is an EC2 instance running the latest Docker version.

Here are the typical docker compose errors (stuff in <angle brackets> is redacted):

% docker compose ps
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: killed
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: killed
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: killed
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[0002] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
error during connect: Get "http://docker/v1.41/containers/<hash>/json": command [ssh -- <context name> docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: read: Connection reset by peer

[ndeloof]

for the records: https://github.com/docker/cli/blob/master/cli/connhelper/commandconn/commandconn.go#L203-L214

[donnykurnia]

So, still no solution? At least, a way to silence the warning? Or increase the timeout?

[nlfiedler]

Seems like docker via ssh is a second class citizen, I'm just glad it works at all. Would be even better not seeing disconcerting errors.

[donnykurnia]

I'm now using tcp connection instead of ssh. I copied the way docker-machine works, and now I had a very stable connection to the docker daemon inside vagrant vm.