Dask client connects to SLURM workers, then rapidly loses them

[bw4sz]

Summary

When adding workers to a SLURM dask client, workers are added as resources are provisioned by the scheduler, but then they quickly disappear. Presumably they are killed by the client because a lack of connection (--death_timeout flag). Its not clear whether this is intended behavior. My goal is to add workers to a dask client, connect to that client from my local laptop using jupyter lab. By the time I ssh tunnel in from my laptop, all the workers are killed.

(pangeo) [b.weinstein@c30b-s1 ~]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> from datetime import datetime
>>> from time import sleep
>>> 
>>> cluster = SLURMCluster(project='ewhite',death_timeout=200)
>>> cluster.start_workers(5)
[3, 5, 7, 9, 11]
>>> 
>>> from dask.distributed import Client
>>> client = Client(cluster)
>>> 
>>> while True:
...     print(datetime.now().strftime("%a, %d %B %Y %I:%M:%S"))
...     print(client)
...     sleep(30)
... 
Wed, 21 March 2018 10:57:19
<Client: scheduler='tcp://172.16.194.66:35459' processes=0 cores=0>
Wed, 21 March 2018 10:57:49
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 10:58:19
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 10:58:49
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 10:59:19
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 10:59:49
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 11:00:20
<Client: scheduler='tcp://172.16.194.66:35459' processes=6 cores=24>
Wed, 21 March 2018 11:00:50
<Client: scheduler='tcp://172.16.194.66:35459' processes=5 cores=20>
Wed, 21 March 2018 11:01:20
<Client: scheduler='tcp://172.16.194.66:35459' processes=5 cores=20>
Wed, 21 March 2018 11:01:50
<Client: scheduler='tcp://172.16.194.66:35459' processes=0 cores=0>
Wed, 21 March 2018 11:02:20
<Client: scheduler='tcp://172.16.194.66:35459' processes=0 cores=0>
Wed, 21 March 2018 11:02:50
<Client: scheduler='tcp://172.16.194.66:35459' processes=0 cores=0>

Expected Behavior

Following this helpful screencast, I thought that once workers were added, they would remain available for computation. Either the client is very aggressive about pruning unused workers, or something else is wrong.

Comments

• May be relevant, but unknown. Once I quit the above python session, I am greeted with several error messages saying the TCP stream is closed. Example

Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 200, in read
    convert_stream_closed_error(self, e)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 128, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed

I can confirm that the workers that were once there, are now gone.

(pangeo) [b.weinstein@c30b-s1 ~]$ squeue -u b.weinstein
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          18361726 hpg2-comp     bash b.weinst  R      22:14      1 c30b-s1

presumably killed by the client.

Edited desk.err file produced, with many hundreds of duplicate lines removed.

(pangeo) [b.weinstein@c30b-s1 ~]$ cat dask.err 
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:36916'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:38675'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:32914'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:44959'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:44970'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:36426'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:35157'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.194.178:40539'
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-r9gleghg', purging
distributed.worker - INFO -       Start worker at: tcp://172.16.194.178:39384
distributed.worker - INFO -          Listening to: tcp://172.16.194.178:39384
distributed.worker - INFO -              nanny at:       172.16.194.178:44959
distributed.worker - INFO -              bokeh at:        172.16.194.178:8789
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-l7jp7he1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Failed to start worker process.  Restarting
...
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.worker - INFO -       Start worker at: tcp://172.16.194.178:46551
distributed.worker - INFO -          Listening to: tcp://172.16.194.178:46551
distributed.worker - INFO -              nanny at:       172.16.194.178:38675
distributed.worker - INFO -              bokeh at:       172.16.194.178:41314
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-yjnhwx2e
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
...
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.192.163:44012'
...
:35125
distributed.worker - INFO -          Listening to: tcp://172.16.194.178:35125
distributed.worker - INFO -              nanny at:       172.16.194.178:32914
distributed.worker - INFO -              bokeh at:       172.16.194.178:45869
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-w9w338nb
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.178:35125
distributed.worker - INFO -       Start worker at: tcp://172.16.194.184:46207
distributed.worker - INFO -          Listening to: tcp://172.16.194.184:46207
distributed.worker - INFO -              nanny at:       172.16.194.184:45711
distributed.worker - INFO -              bokeh at:        172.16.194.184:8789
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-g0jjm93z
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.184:46207
orker - INFO -              nanny at:       172.16.194.178:38675
distributed.worker - INFO -              bokeh at:       172.16.194.178:45033
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-u2_1pu5v
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.178:35568
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 155, in _start
    response = yield self.instantiate()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 223, in instantiate
    self.process.start()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 363, in start
    self._wait_until_started())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.194.178:37068', 'dir': '/home/b.weinstein/dask-worker-space/worker-5ipv17oi'}
distributed.dask_worker - INFO - End worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
/envs/pangeo/bin/dask-worker", line 6, in <module>
    sys.exit(distributed.cli.dask_worker.go())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 252, in go
    main()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 243, in main
    loop.run_sync(run)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/ioloop.py", line 582, in run_sync
    return future_cell[0].result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 236, in run
    yield [n._start(addr) for n in nannies]
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 155, in _start
    response = yield self.instantiate()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 223, in instantiate
    self.process.start()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 363, in start
    self._wait_until_started())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.194.178:37661', 'dir': '/home/b.weinstein/dask-worker-space/worker-_b3fhcyu'}
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-25, started daemon)>
...
distributed.worker - INFO -          Listening to: tcp://172.16.194.184:43288
distributed.worker - INFO -              nanny at:       172.16.194.184:43216
distributed.worker - INFO -              bokeh at:        172.16.194.184:8789
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-mo6fnfvo
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.184:43288
distributed.nanny - INFO - Failed to start worker process.  Restarting
...
:36157
distributed.worker - INFO -          Listening to: tcp://172.16.194.184:36157
distributed.worker - INFO -              nanny at:       172.16.194.184:42948
distributed.worker - INFO -              bokeh at:        172.16.194.184:8789
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-h0jj6gi3
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.184:36157
distributed.nanny - INFO - Failed to start worker process.  Restarting
...
distributed.worker - INFO -          Listening to: tcp://172.16.194.184:46197
distributed.worker - INFO -              nanny at:       172.16.194.184:38837
distributed.worker - INFO -              bokeh at:        172.16.194.184:8789
distributed.worker - INFO - Waiting to connect to:  tcp://172.16.194.66:35459
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-lft5bgwk
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.194.184:46197
...
distributed.nanny - WARNING - Worker process 27618 was killed by unknown signal
...
distributed.dask_worker - INFO - End worker
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-392, 
...
distributed.nanny - WARNING - Worker process still alive after 159 seconds, killing
distributed.nanny - WARNING - Worker process 13695 was killed by unknown signal
distributed.nanny - WARNING - Worker process still alive after 159 seconds, killing
...
distributed.nanny - WARNING - Worker process 13712 was killed by unknown signal
...
distributed.nanny - WARNING - Worker process still alive after 159 seconds, killing
distributed.dask_worker - INFO - End worker
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-303, started daemon)>
(pangeo) [b.weinstein@c30b-s1 ~]$ 

[guillaumeeb]

Given your logs, it looks like the worker are failing to start for some reason:

distributed.nanny - INFO - Failed to start worker process.  Restarting

Not clear to me why. Could you provide the SLURMCluster generated job_script:

cluster.job_script()

For the sake of log clarity, could you try to launch only one worker and give us the log output?

Those lines in the stack trace look weird to me:

  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.194.178:37068', 'dir': '/home/b.weinstein/dask-worker-space/worker-5ipv17oi'}

[mrocklin]

Clients don't kill workers themselves. Nothing will kill them by default.

What is the output of the following:

import distributed
print(distributed.__version__)

[mrocklin]

Yeah, I agree that these lines are concerning:

  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.194.178:37068', 'dir': '/home/b.weinstein/dask-worker-space/worker-5ipv17oi'}

We've seen this very intermittently in our CI tests and I haven't yet been able to figure out how they occur. As a test you might try setting the following in your ~/.dask/config.yaml file

# multiprocessing-method: forkserver
multiprocessing-method: spawn

I would be quite interested to learn if that fixes the problem

[mrocklin]

If you had time to test out dask/distributed#1848 that would also be welcome.

It's hard for me to reproduce this error locally, which is one reason why it has lingered for a while.

[bw4sz]

Okay, i'm up and will be working on this today. To summarize the above plans of action. I will add to this thread as I have answers to the following questions.

1. Give version info and job script

import distributed
print(distributed.__version__)

cluster.job_script()

2. Run on a single worker (cluster.start_workers(1)) and report entire log file.
3. Fork dask/distributed and try PR #1848
4. Edit dask config file

~/.dask/config.yaml file

# multiprocessing-method: forkserver
multiprocessing-method: spawn

[bw4sz]

1. Version and job scripts.

>>> import distributed
>>> print(distributed.__version__)
1.21.3

Job script, with formatted line breaks.

>>> cluster.job_script()

#!/bin/bash
#SBATCH -J dask
#SBATCH -n 8
#SBATCH -A ewhite
#SBATCH -t 00:30:00
#SBATCH -e dask.err
#SBATCH -o dask.out

export LANG="en_US.utf8"
export LANGUAGE="en_US.utf8"
export LC_ALL="en_US.utf8"

"/home/b.weinstein/miniconda3/envs/pangeo/bin/dask-worker" tcp://172.16.194.158:37289     --nthreads 4     --nprocs 8     --memory-limit 7GB     --name dask-4     --death-timeout 200

[mrocklin]

And to be clear, running that script breaks?

[bw4sz]

Ya, posting now. Not sure best way to separate the thread. It breaks from within cluster.start_workers. I have not tried separately to push that slurm script.

This is the dask.err file for that one worker. Note that client has not been called.

(pangeo) [b.weinstein@c31b-s21 dask-jobqueue]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> cluster = SLURMCluster(project='ewhite',death_timeout=30)
>>> cluster.start_workers(1)
[3]
>>> exit()
(pangeo) [b.weinstein@c31b-s21 dask-jobqueue]$ squeue -u b.weinstein
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          18382054 hpg2-comp     bash b.weinst  R      18:04      1 c31b-s21
(pangeo) [b.weinstein@c31b-s21 dask-jobqueue]$ cat dask.err
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:37297'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:35788'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:35218'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:44645'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:41522'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:44451'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:37657'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.51:41545'
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 31388 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31391 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31395 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31397 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 31435 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:37297'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:35788'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:35218'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:44645'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:41522'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:44451'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:37657'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:41545'
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 31537 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31535 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31540 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31543 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31547 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31553 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31556 was killed by unknown signal
distributed.nanny - WARNING - Worker process 31549 was killed by unknown signal
distributed.dask_worker - INFO - End worker
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-53, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-55, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-49, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-56, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-50, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-54, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-51, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-52, started daemon)>

[mrocklin]

Another possibility here is that it's just taking your system a long time to start a new process. Currently we restart the worker if we don't hear an initial hello from the forked worker process within five seconds. On a normal system this should happen in well under a second. I would not be very surprised though if systems with fancy file systems and OS's take far longer.

[jakirkham]

Currently we restart the worker if we don't hear an initial hello from the forked worker process within five seconds.

Is that timeout customizable? If not, could we make customizable?

[mrocklin]

No and Yes. Ideally it also just wouldn't be necessary if we can find the root of the problem.

[bw4sz]

One thing i'm noticing is that when you do call for one worker, I never seem to get connected to the client. But if I try two, I do get a connection, and then the dask.err file is different. I'm trying to separate this behavior from the death_timeout flag. If the client isn't responsible for killing the workers, what is the function of this arg?

        death_timeout : float
            Seconds to wait for a scheduler before closing workers

Example here of requesting two workers gets synced up, albeit briefly.

>>> cluster = SLURMCluster(project='ewhite',death_timeout=200)
>>> cluster.start_workers(2)
[3, 5]
>>> 
>>> from dask.distributed import Client
>>> client = Client(cluster)
>>> 
>>> while True:
...     print(datetime.now().strftime("%a, %d %B %Y %I:%M:%S"))
...     print(client)
...     sleep(30)
... 
Thu, 22 March 2018 01:02:01
<Client: scheduler='tcp://172.16.194.158:33354' processes=0 cores=0>
Thu, 22 March 2018 01:02:31
<Client: scheduler='tcp://172.16.194.158:33354' processes=2 cores=8>
Thu, 22 March 2018 01:03:01
<Client: scheduler='tcp://172.16.194.158:33354' processes=2 cores=8>
Thu, 22 March 2018 01:03:31
<Client: scheduler='tcp://172.16.194.158:33354' processes=2 cores=8>
Thu, 22 March 2018 01:04:01
<Client: scheduler='tcp://172.16.194.158:33354' processes=5 cores=20>
Thu, 22 March 2018 01:04:31
<Client: scheduler='tcp://172.16.194.158:33354' processes=5 cores=20>
Thu, 22 March 2018 01:05:01
<Client: scheduler='tcp://172.16.194.158:33354' processes=7 cores=28>
Thu, 22 March 2018 01:05:31
<Client: scheduler='tcp://172.16.194.158:33354' processes=6 cores=24>
Thu, 22 March 2018 01:06:01
<Client: scheduler='tcp://172.16.194.158:33354' processes=1 cores=4>
Thu, 22 March 2018 01:06:31
<Client: scheduler='tcp://172.16.194.158:33354' processes=1 cores=4>
>>> 
[1]+  Stopped                 python

Atleast one worker still exists when I killed it.

(pangeo) [b.weinstein@c31b-s21 ~]$ squeue -u b.weinstein
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          18383642 hpg2-comp     dask b.weinst  R       4:50      1 c25a-s38
          18382054 hpg2-comp     bash b.weinst  R      32:27      1 c31b-s21

Error File

(pangeo) [b.weinstein@c31b-s21 ~]$ cat dask.err
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:45179'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:37653'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:44663'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:34624'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:34534'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:36418'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:46417'
distributed.nanny - INFO -         Start Nanny at: 'tcp://172.16.193.37:44895'
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-l7jp7he1', purging
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-yjnhwx2e', purging
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-5ipv17oi', purging
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-oqjzke10', purging
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-bm2z_ut8', purging
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-_b3fhcyu', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:36876
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:36876
distributed.worker - INFO -              nanny at:        172.16.193.37:34534
distributed.worker - INFO -              bokeh at:         172.16.193.37:8789
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-6el6x5a8
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 11616 was killed by unknown signal
distributed.nanny - WARNING - Worker process 11619 was killed by unknown signal
distributed.nanny - WARNING - Worker process 11621 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 11789 was killed by unknown signal
distributed.nanny - WARNING - Worker process 11790 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-enltse1k
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 6713 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.core - WARNING - Event loop was unresponsive in Nanny for 3.61s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:42822
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:42822
distributed.worker - INFO -              nanny at:        172.16.193.37:36418
distributed.worker - INFO -              bokeh at:        172.16.193.37:33011
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-yzma4wfe
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:42822', 'dir': '/home/b.weinstein/dask-worker-space/worker-yzma4wfe'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:42822', 'dir': '/home/b.weinstein/dask-worker-space/worker-yzma4wfe'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 12298 was killed by unknown signal
distributed.nanny - WARNING - Worker process 12291 was killed by unknown signal
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 12301 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:35731
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:35731
distributed.worker - INFO -              nanny at:        172.16.193.37:44663
distributed.worker - INFO -              bokeh at:        172.16.193.37:41203
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-btuqdh1h
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:35731', 'dir': '/home/b.weinstein/dask-worker-space/worker-btuqdh1h'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:35731', 'dir': '/home/b.weinstein/dask-worker-space/worker-btuqdh1h'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-btuqdh1h', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:37075
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:37075
distributed.worker - INFO -              nanny at:        172.16.193.37:34624
distributed.worker - INFO -              bokeh at:        172.16.193.37:44202
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-y1r323fs
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:37075', 'dir': '/home/b.weinstein/dask-worker-space/worker-y1r323fs'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:37075', 'dir': '/home/b.weinstein/dask-worker-space/worker-y1r323fs'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-y1r323fs', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:39963
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:39963
distributed.worker - INFO -              nanny at:        172.16.193.37:46417
distributed.worker - INFO -              bokeh at:        172.16.193.37:42135
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-ujdmow3a
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:33235
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:33235
distributed.worker - INFO -              nanny at:        172.16.193.37:44895
distributed.worker - INFO -              bokeh at:        172.16.193.37:39107
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-wh6nho8w
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:33235', 'dir': '/home/b.weinstein/dask-worker-space/worker-wh6nho8w'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:33235', 'dir': '/home/b.weinstein/dask-worker-space/worker-wh6nho8w'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-wh6nho8w', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:38405
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:38405
distributed.worker - INFO -              nanny at:        172.16.193.37:44895
distributed.worker - INFO -              bokeh at:        172.16.193.37:39540
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-_s2movv7
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:38405', 'dir': '/home/b.weinstein/dask-worker-space/worker-_s2movv7'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:38405', 'dir': '/home/b.weinstein/dask-worker-space/worker-_s2movv7'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-_s2movv7', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:36385
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:36385
distributed.worker - INFO -              nanny at:        172.16.193.37:37653
distributed.worker - INFO -              bokeh at:        172.16.193.37:34652
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-wqfzo61b
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:36385', 'dir': '/home/b.weinstein/dask-worker-space/worker-wqfzo61b'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:36385', 'dir': '/home/b.weinstein/dask-worker-space/worker-wqfzo61b'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-wqfzo61b', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:37970
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:37970
distributed.worker - INFO -              nanny at:        172.16.193.37:37653
distributed.worker - INFO -              bokeh at:        172.16.193.37:41849
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-8hr6m2l4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:37970', 'dir': '/home/b.weinstein/dask-worker-space/worker-8hr6m2l4'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:37970', 'dir': '/home/b.weinstein/dask-worker-space/worker-8hr6m2l4'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-8hr6m2l4', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:42689
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:42689
distributed.worker - INFO -              nanny at:        172.16.193.37:45179
distributed.worker - INFO -              bokeh at:        172.16.193.37:38875
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-prqyjzpk
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:42689', 'dir': '/home/b.weinstein/dask-worker-space/worker-prqyjzpk'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:42689', 'dir': '/home/b.weinstein/dask-worker-space/worker-prqyjzpk'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.core - WARNING - Event loop was unresponsive in Nanny for 3.57s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 7779 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7776 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7773 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7782 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7785 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7788 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 7824 was killed by unknown signal
distributed.nanny - WARNING - Worker process 7827 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 7913 was killed by unknown signal
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 12710 was killed by unknown signal
distributed.nanny - WARNING - Worker process 12712 was killed by unknown signal
distributed.nanny - WARNING - Worker process 12707 was killed by unknown signal
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:33064
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:33064
distributed.worker - INFO -              nanny at:        172.16.193.37:34624
distributed.worker - INFO -              bokeh at:        172.16.193.37:43279
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-dshkgia7
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:38705
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:38705
distributed.worker - INFO -              nanny at:        172.16.193.37:45179
distributed.worker - INFO -              bokeh at:        172.16.193.37:43251
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-walv8bwc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:38705', 'dir': '/home/b.weinstein/dask-worker-space/worker-walv8bwc'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:38705', 'dir': '/home/b.weinstein/dask-worker-space/worker-walv8bwc'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-walv8bwc', purging
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:35667
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:35667
distributed.worker - INFO -              nanny at:        172.16.193.37:45179
distributed.worker - INFO -              bokeh at:        172.16.193.37:36515
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-59y3zgvs
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
tornado.application - ERROR - Exception in Future <Future finished exception=AssertionError({'address': 'tcp://172.16.193.37:35667', 'dir': '/home/b.weinstein/dask-worker-space/worker-59y3zgvs'},)> after timeout
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 936, in error_callback
    future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:35667', 'dir': '/home/b.weinstein/dask-worker-space/worker-59y3zgvs'}
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Worker process 8197 was killed by unknown signal
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - WARNING - Restarting worker
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.diskutils - WARNING - Found stale lock file and directory '/home/b.weinstein/dask-worker-space/worker-59y3zgvs', purging
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Failed to start worker process.  Restarting
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:39452'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:39527'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:42840'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:38489'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:35320'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:39109'
distributed.nanny - INFO - Closing Nanny at 'tcp://172.16.193.51:33045'
distributed.worker - INFO - Stopping worker at tcp://172.16.193.51:41034
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:41538
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:41538
distributed.worker - INFO -              nanny at:        172.16.193.37:44663
distributed.worker - INFO -              bokeh at:        172.16.193.37:39385
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-502r_ai_
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.193.37:41538
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:44847
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.37:37823
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:44847
distributed.worker - INFO -          Listening to:  tcp://172.16.193.37:37823
distributed.worker - INFO -              nanny at:        172.16.193.37:44895
distributed.worker - INFO -              nanny at:        172.16.193.37:45179
distributed.worker - INFO -              bokeh at:        172.16.193.37:46176
distributed.worker - INFO -              bokeh at:        172.16.193.37:38671
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-djymwodh
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-g1h_6kpl
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.193.37:44847
distributed.worker - INFO - Stopping worker at tcp://172.16.193.37:37823
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 155, in _start
    response = yield self.instantiate()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 223, in instantiate
    self.process.start()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 363, in start
    self._wait_until_started())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:33064', 'dir': '/home/b.weinstein/dask-worker-space/worker-dshkgia7'}
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 155, in _start
    response = yield self.instantiate()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 223, in instantiate
    self.process.start()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 363, in start
    self._wait_until_started())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:39963', 'dir': '/home/b.weinstein/dask-worker-space/worker-ujdmow3a'}
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
  File "/home/b.weinstein/miniconda3/envs/pangeo/bin/dask-worker", line 6, in <module>
    sys.exit(distributed.cli.dask_worker.go())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 252, in go
    main()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 243, in main
    loop.run_sync(run)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/ioloop.py", line 582, in run_sync
    return future_cell[0].result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 236, in run
    yield [n._start(addr) for n in nannies]
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 849, in callback
    result_list.append(f.result())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 155, in _start
    response = yield self.instantiate()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 223, in instantiate
    self.process.start()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1107, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 363, in start
    self._wait_until_started())
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1099, in run
    value = future.result()
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1113, in run
    yielded = self.gen.send(value)
  File "/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/distributed/nanny.py", line 471, in _wait_until_started
    assert msg == 'started', msg
AssertionError: {'address': 'tcp://172.16.193.37:40506', 'dir': '/home/b.weinstein/dask-worker-space/worker-oom6rvvo'}
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-195, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-185, started daemon)>
distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-5, started daemon)>
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.51:45811
distributed.worker - INFO -          Listening to:  tcp://172.16.193.51:45811
distributed.worker - INFO -              nanny at:        172.16.193.51:39527
distributed.worker - INFO -              bokeh at:         172.16.193.51:8789
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-aw1j2b4o
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.193.51:45811
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.51:41891
distributed.worker - INFO -          Listening to:  tcp://172.16.193.51:41891
distributed.worker - INFO -              nanny at:        172.16.193.51:42840
distributed.worker - INFO -              bokeh at:         172.16.193.51:8789
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-tbm72j6l
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.193.51:41891
distributed.worker - INFO -       Start worker at:  tcp://172.16.193.51:33100
distributed.worker - INFO -          Listening to:  tcp://172.16.193.51:33100
distributed.worker - INFO -              nanny at:        172.16.193.51:39452
distributed.worker - INFO -              bokeh at:         172.16.193.51:8789
distributed.worker - INFO - Waiting to connect to: tcp://172.16.194.158:33354
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          4
distributed.worker - INFO -                Memory:                    7.00 GB
distributed.worker - INFO -       Local Directory: /home/b.weinstein/dask-worker-space/worker-s0t0xf97
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Stopping worker at tcp://172.16.193.51:33100

[bw4sz]

Maybe its something about how resources are allocated, but just driving home the point here. Exact same call as above, except just one worker, and I never get synced up to client.

(pangeo) [b.weinstein@c31b-s21 ~]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> from datetime import datetime
>>> from time import sleep
>>> 
>>> cluster = SLURMCluster(project='ewhite',death_timeout=200)
>>> cluster.start_workers(1)
[3]
>>> 
>>> from dask.distributed import Client
>>> client = Client(cluster)
>>> 
>>> while True:
...     print(datetime.now().strftime("%a, %d %B %Y %I:%M:%S"))
...     print(client)
...     sleep(30)
... 
Thu, 22 March 2018 01:11:00
<Client: scheduler='tcp://172.16.194.158:40049' processes=0 cores=0>
Thu, 22 March 2018 01:11:30
<Client: scheduler='tcp://172.16.194.158:40049' processes=0 cores=0>
Thu, 22 March 2018 01:12:00
<Client: scheduler='tcp://172.16.194.158:40049' processes=0 cores=0>
Thu, 22 March 2018 01:12:30
<Client: scheduler='tcp://172.16.194.158:40049' processes=0 cores=0>
Thu, 22 March 2018 01:13:00
<Client: scheduler='tcp://172.16.194.158:40049' processes=0 cores=0>

[mrocklin]

I see no reason why any worker would leave after it has connected. I'm as surprised as you are.

Death timeout is used by the worker itself when it can't connect to the scheduler in time

[bw4sz]

okay, next thing is to try that PR. What's the best way to do that? Fork the PR, clone on the cluster and replace

'/home/b.weinstein/miniconda3/envs/pangeo/lib/python3.6/site-packages/dask/distributed.py'

will that be sufficient?