Installing docker-ce through systemd "fails" due to docker.service not finding socket on first try

[Nacknime]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

get-docker.sh is invoked in a systemd service.

Expected behavior

The systemd service successfully installs Docker using the get-docker.sh script and apt-get.

Actual behavior

Once get-docker.sh gets to apt-get installing docker-ce, it fails once it gets to the docker-ce postinstall step.

journalctl -xu docker-install.service:

Apr 27 16:27:09 home init.sh[293]: Setting up docker-ce (5:19.03.8~3-0~raspbian-buster) ...
Apr 27 16:27:10 home init.sh[293]: Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Apr 27 16:27:11 home init.sh[293]: Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Apr 27 16:27:13 home init.sh[293]: Job for docker.service failed because the control process exited with error code.
Apr 27 16:27:13 home init.sh[293]: dpkg: error processing package docker-ce (--configure):
Apr 27 16:27:13 home init.sh[293]:  installed docker-ce package post-installation script subprocess returned error exit status 1

This causes the whole get-docker.sh script to be considered failed and the service failed as well.

More information about why docker.service fails can be found in journalctl -xu docker.service:

Apr 27 16:27:13 home systemd[1]: Starting Docker Application Container Engine...
Apr 27 16:27:13 home dockerd[1084]: time="2020-04-27T16:27:13.852785635+01:00" level=info msg="Starting up"
Apr 27 16:27:13 home dockerd[1084]: failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Apr 27 16:27:13 home systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 27 16:27:13 home systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 27 16:27:13 home systemd[1]: Failed to start Docker Application Container Engine.

When the docker.service is automatically retried 2 seconds later, it starts with no problems.

Afterwards, Docker seems to be installed properly, even though apt-get/dpkg report a failure. dpkg --status docker-ce:

Package: docker-ce
Status: install ok half-configured

Steps to reproduce the behavior

• Download the get-docker.sh script.
• Create a systemd service (e.g. docker-install.service) that invokes get-docker.sh and enable it.
• Reboot

Simply invoking the installation manually from shell does not lead to the same issue.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:35:24 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:22 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 5
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.97-v7+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 926.1MiB
 Name: home
 ID: JXJO:MVBV:XODO:WSKR:UCMS:T6P7:5GY5:AJEH:FSCD:HF56:EFNE:OAXZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Additional environment details (AWS, VirtualBox, physical, etc.)

Raspberry Pi 4, Raspbian Buster, uname -a:

Linux home 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

[thaJeztah]

No idea what the difference would be; possibly other services that are expected to be "up" are not yet running when your service starts?

The convenience install script was designed to be ran manually (at least your approach is not tested); I'm closing this issue because this is not what the script was designed for, but if you happen to find a way to make this work, feel free to open a pull request in the repository that contains the script; https://github.com/docker/docker-install

[Nacknime]

@thaJeztah thank you for your answer! It does not seem to be an issue with other services being up or not, as there are no other services launching anymore at the point that the script is run. It must somehow be related to being called from systemd.

I get that the script is not meant to be run that way, do you have another suggestion on how to create images such that Docker is available after first startup? Is it possible to install Docker in a chrooted environment on an image-creating host?

[thaJeztah]

For raspberry Pi's, Hypriot could be an option; https://blog.hypriot.com

[Nacknime]

Thank you, I'm actually trying to build something similar (an image creator for Raspbian + Docker + many other services), but I will check how they achieve the Docker install.

[rafaelhrtd]

I am having the same problem during installation on Ubuntu 18.04.
sudo apt-get --allow-downgrades -yq install docker-ce=18.06.3*
leads to:

Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  docker-ce
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 0 B/40.2 MB of archives.
After this operation, 198 MB of additional disk space will be used.
Selecting previously unselected package docker-ce.
(Reading database ... 349591 files and directories currently installed.)
Preparing to unpack .../docker-ce_18.06.3~ce~3-0~ubuntu_amd64.deb ...
Unpacking docker-ce (18.06.3~ce~3-0~ubuntu) ...
Setting up docker-ce (18.06.3~ce~3-0~ubuntu) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-09-02 11:49:04 EDT; 3ms ago
     Docs: https://docs.docker.com
  Process: 6682 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 6682 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (237-3ubuntu10.51) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

journalctl -xe shows this at the time of failure:

Sep 02 11:50:53 office64-001 dockerd[9306]: Failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Sep 02 11:50:53 office64-001 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 11:50:53 office64-001 systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 02 11:50:53 office64-001 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

Immediately after the installation failure, docker starts without issues.

[thaJeztah]

Not sure what's the cause of that error, but Docker 18.06 reached EOL three years ago, so I would definitely not recommend running that version.

[rafaelhrtd]

Thanks for your response. Purging and installing without a specified version leads to the same result:

sudo apt-get --allow-downgrades -yq install docker-ce
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  docker-ce-cli docker-ce-rootless-extras docker-scan-plugin
Recommended packages:
  slirp4netns
The following NEW packages will be installed:
  docker-ce docker-ce-cli docker-ce-rootless-extras docker-scan-plugin
0 upgraded, 4 newly installed, 0 to remove and 2 not upgraded.
Need to get 0 B/71.8 MB of archives.
After this operation, 293 MB of additional disk space will be used.
Selecting previously unselected package docker-ce-cli.
(Reading database ... 349591 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a20.10.8~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.8~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_5%3a20.10.8~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:20.10.8~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../docker-ce-rootless-extras_5%3a20.10.8~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:20.10.8~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../docker-scan-plugin_0.8.0~ubuntu-bionic_amd64.deb ...
Unpacking docker-scan-plugin (0.8.0~ubuntu-bionic) ...
Setting up docker-ce-rootless-extras (5:20.10.8~3-0~ubuntu-bionic) ...
Setting up docker-scan-plugin (0.8.0~ubuntu-bionic) ...
Setting up docker-ce-cli (5:20.10.8~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:20.10.8~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-09-02 11:59:11 EDT; 3ms ago
     Docs: https://docs.docker.com
  Process: 12628 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 12628 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (237-3ubuntu10.51) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

From journalctl -xe:

Sep 02 11:58:41 office64-001 systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has begun starting up.
Sep 02 11:58:41 office64-001 dockerd[11252]: Failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Sep 02 11:58:41 office64-001 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 11:58:41 office64-001 systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 02 11:58:41 office64-001 systemd[1]: Failed to start Docker Application Container Engine.

[thaJeztah]

what version of containerd.io is installed? Are there custom changes made in the systemd unit files, or a drop-in / override file? (systemctl cat <service name> should give you all unit files that are in effect for a service, e.g. systemctl cat docker.service, and systemctl cat docker.socket)

[rafaelhrtd]

containerd.io/bionic,now 1.4.9-1 amd64 [installed]
systemctl cat docker.service returns:

# /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500

[Install]
WantedBy=multi-user.target

and systemctl cat docker.socket returns:

# /lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

[thaJeztah]

nope, no direct clue. Perhaps the logs provide more details? (journalctl -n 50 -xu docker.service to view the last 50 lines from the "docker" service logs; perhaps the containerd.service and/or docker.socket as well 🤔

[rafaelhrtd]

These are from journalctl -n 50 -xu docker.service. As I mentioned, docker seems to run properly after the initial failure:

-- Logs begin at Tue 2021-08-17 16:29:50 EDT, end at Thu 2021-09-02 12:16:15 EDT. --
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287678970-04:00" level=debug msg="Registering POST, /swarm/leave"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287700513-04:00" level=debug msg="Registering GET, /swarm"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287716970-04:00" level=debug msg="Registering GET, /swarm/unlockkey"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287756158-04:00" level=debug msg="Registering POST, /swarm/update"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287778614-04:00" level=debug msg="Registering POST, /swarm/unlock"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287796645-04:00" level=debug msg="Registering GET, /services"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287818179-04:00" level=debug msg="Registering GET, /services/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287847057-04:00" level=debug msg="Registering POST, /services/create"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287866053-04:00" level=debug msg="Registering POST, /services/{id}/update"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287893243-04:00" level=debug msg="Registering DELETE, /services/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287918099-04:00" level=debug msg="Registering GET, /services/{id}/logs"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287949881-04:00" level=debug msg="Registering GET, /nodes"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287968460-04:00" level=debug msg="Registering GET, /nodes/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.287996352-04:00" level=debug msg="Registering DELETE, /nodes/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288021011-04:00" level=debug msg="Registering POST, /nodes/{id}/update"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288049033-04:00" level=debug msg="Registering GET, /tasks"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288065282-04:00" level=debug msg="Registering GET, /tasks/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288088346-04:00" level=debug msg="Registering GET, /tasks/{id}/logs"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288120729-04:00" level=debug msg="Registering GET, /secrets"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288160542-04:00" level=debug msg="Registering POST, /secrets/create"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288180917-04:00" level=debug msg="Registering DELETE, /secrets/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288220182-04:00" level=debug msg="Registering GET, /secrets/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288252679-04:00" level=debug msg="Registering POST, /secrets/{id}/update"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288287093-04:00" level=debug msg="Registering GET, /configs"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288306521-04:00" level=debug msg="Registering POST, /configs/create"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288325142-04:00" level=debug msg="Registering DELETE, /configs/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288348876-04:00" level=debug msg="Registering GET, /configs/{id}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288375465-04:00" level=debug msg="Registering POST, /configs/{id}/update"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288412836-04:00" level=debug msg="Registering GET, /plugins"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288431625-04:00" level=debug msg="Registering GET, /plugins/{name:.*}/json"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288458696-04:00" level=debug msg="Registering GET, /plugins/privileges"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288480145-04:00" level=debug msg="Registering DELETE, /plugins/{name:.*}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288507924-04:00" level=debug msg="Registering POST, /plugins/{name:.*}/enable"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288535981-04:00" level=debug msg="Registering POST, /plugins/{name:.*}/disable"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288568056-04:00" level=debug msg="Registering POST, /plugins/pull"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288588371-04:00" level=debug msg="Registering POST, /plugins/{name:.*}/push"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288616369-04:00" level=debug msg="Registering POST, /plugins/{name:.*}/upgrade"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288642065-04:00" level=debug msg="Registering POST, /plugins/{name:.*}/set"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288666940-04:00" level=debug msg="Registering POST, /plugins/create"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288687990-04:00" level=debug msg="Registering GET, /distribution/{name:.*}/json"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288716456-04:00" level=debug msg="Registering POST, /grpc"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288733657-04:00" level=debug msg="Registering GET, /networks"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288750767-04:00" level=debug msg="Registering GET, /networks/"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288771030-04:00" level=debug msg="Registering GET, /networks/{id:.+}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288798635-04:00" level=debug msg="Registering POST, /networks/create"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288819610-04:00" level=debug msg="Registering POST, /networks/{id:.*}/connect"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288847777-04:00" level=debug msg="Registering POST, /networks/{id:.*}/disconnect"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288880092-04:00" level=debug msg="Registering POST, /networks/prune"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288900683-04:00" level=debug msg="Registering DELETE, /networks/{id:.*}"
Sep 02 12:16:14 office64-001 dockerd[7076]: time="2021-09-02T12:16:14.288992858-04:00" level=info msg="API listen on /var/run/docker.sock"

Here is some more context before the failure:

Sep 02 12:15:23 office64-001 systemd[1]: Started Docker Application Container Engine.
-- Subject: Unit docker.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 02 12:15:36 office64-001 systemd[1]: Stopping Docker Application Container Engine...
-- Subject: Unit docker.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has begun shutting down.
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.815825499-04:00" level=info msg="Processing signal 'terminated'"
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.815882712-04:00" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.815896296-04:00" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816186617-04:00" level=debug msg="Unix socket /run/docker/libnetwork/fb90ee6c6293eb12b8bc26cf05e8c0ee9ab20105facc751
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816204549-04:00" level=debug msg="Cleaning up old mountid : start."
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816352432-04:00" level=debug msg="Cleaning up old mountid : done."
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816438705-04:00" level=debug msg="Clean shutdown succeeded"
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816461250-04:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816474263-04:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816506816-04:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36-04:00" level=debug msg="received signal" signal=terminated
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816939725-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420247180, TRANSIENT_FAILURE" modul
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816952503-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420247180, CONNECTING" module=grpc
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.816909707-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420382170, TRANSIENT_FAILURE" modul
Sep 02 12:15:36 office64-001 dockerd[3605]: time="2021-09-02T12:15:36.817007915-04:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420382170, CONNECTING" module=grpc
Sep 02 12:15:37 office64-001 systemd[1]: Stopped Docker Application Container Engine.
-- Subject: Unit docker.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has finished shutting down.
Sep 02 12:16:11 office64-001 systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has begun starting up.
Sep 02 12:16:11 office64-001 dockerd[6507]: time="2021-09-02T12:16:11.780567786-04:00" level=info msg="Starting up"
Sep 02 12:16:11 office64-001 dockerd[6507]: failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Sep 02 12:16:11 office64-001 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 12:16:11 office64-001 systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 02 12:16:11 office64-001 systemd[1]: Failed to start Docker Application Container Engine.

From containerd.service:

Sep 02 11:42:40 office64-001 systemd[1]: Started containerd container runtime.
-- Subject: Unit containerd.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit containerd.service has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 02 12:13:39 office64-001 systemd[1]: Stopping containerd container runtime...
-- Subject: Unit containerd.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit containerd.service has begun shutting down.
Sep 02 12:13:39 office64-001 systemd[1]: Stopped containerd container runtime.
-- Subject: Unit containerd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit containerd.service has finished shutting down.
-- Reboot --
Sep 02 12:16:11 office64-001 systemd[1]: Starting containerd container runtime...
-- Subject: Unit containerd.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit containerd.service has begun starting up.
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.055198735-04:00" level=info msg="starting containerd" revision=e25210fe30a0a703442421b0f60afac609f950a3 version=1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.069814030-04:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.conte
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.069980887-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snap
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071098516-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.sna
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071372840-04:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071414155-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071425958-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmappe
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071448843-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.sn
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071603196-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071745712-04:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snaps
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071863486-04:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/li
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071874126-04:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadat
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071885036-04:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not 
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.071890812-04:00" level=info msg="metadata content store policy set" policy=shared
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072538562-04:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072571317-04:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072590761-04:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.co
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072621610-04:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.conta
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072654183-04:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containe
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072663157-04:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072670737-04:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.container
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072678230-04:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.container
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072685614-04:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.conta
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072693550-04:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.contai
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072700223-04:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072775030-04:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.072803895-04:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monit
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073043940-04:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073076001-04:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.inte
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073100846-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073109013-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073117713-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073124071-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073130469-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073137169-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073143503-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073150061-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073156374-04:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073177797-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v
Sep 02 12:16:11 office64-001 containerd[6377]: time="2021-09-02T12:16:11.073210514-04:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1

And from docker.socket:

Sep 02 12:13:34 office64-001 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 02 12:13:39 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:39 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:39 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:39 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:40 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:40 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:40 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:41 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:41 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:41 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:41 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:41 office64-001 systemd[1]: docker.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not fun
Sep 02 12:13:54 office64-001 systemd[1]: Closed docker.socket.
-- Subject: Unit docker.socket has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has finished shutting down.
-- Reboot --
Sep 02 12:15:23 office64-001 systemd[1]: Starting Docker Socket for the API.
-- Subject: Unit docker.socket has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has begun starting up.
Sep 02 12:15:23 office64-001 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 02 12:15:37 office64-001 systemd[1]: Closed Docker Socket for the API.
-- Subject: Unit docker.socket has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has finished shutting down.
Sep 02 12:16:13 office64-001 systemd[1]: Starting Docker Socket for the API.
-- Subject: Unit docker.socket has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has begun starting up.
Sep 02 12:16:13 office64-001 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is RESULT.

[rafaelhrtd]

At the time of failure, it looks like the Docker socket is started after the Docker service:

-- Unit docker.service has begun starting up.
Sep 02 12:16:11 office64-001 dockerd[6507]: time="2021-09-02T12:16:11.780567786-04:00" level=info msg="Starting up"

vs.

-- Unit docker.socket has finished shutting down.
Sep 02 12:16:13 office64-001 systemd[1]: Starting Docker Socket for the API.
-- Subject: Unit docker.socket has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

[thaJeztah]

At the time of failure, it looks like the Docker socket is started after the Docker service:

Oh! I see the issue; it looks like the systemd unit in the 20.10 packages has a Requires=docker.socket but is missing After=docker.socket 🤔
https://github.com/docker/docker-ce-packaging/blob/3eb20305c00b8110a26be8e13b25d39deaddcf45/systemd/docker.service#L4

Requires only defines the dependency, but doesn't influence startup order, so the docker service won't wait for the docker.socket to start.

I see a fix for that was added in moby/moby@fe68df3, but at some point the packaging switched to using a different systemd unit (I changed that back to using the ones from the upstream (this) repository in docker/docker-ce-packaging#547, but that's not yet in use for the 20.10 release).