Subsecond cluster startup time

[mrocklin]

We used to be able to start up clusters in tens of milliseconds. Now it takes a second or two.

In [1]: from dask.distributed import LocalCluster

In [2]: %%time
   ...: with LocalCluster(processes=False, n_workers=0):
   ...:     pass
   ...: 
CPU times: user 751 ms, sys: 68.5 ms, total: 819 ms
Wall time: 2.79 s

In [3]: %%time
   ...: with LocalCluster(processes=False, n_workers=0):
   ...:     pass
   ...: 
CPU times: user 73.4 ms, sys: 28 µs, total: 73.4 ms
Wall time: 1.08 s

This would make testing nicer.

[mrocklin]

Looking at this, it looks like SpecCluster calls self.scheduler_comm.terminate(), the Scheduler calls Server.close, which waits a second for all comms to close, but there is still a comm asking us to terminate which is still active.