Investigate use of preemptible GCP instances for GWAS

[tomwhite]

In #390 (and processing in general), using preemptible instances on GCP would bring a cost saving of ~5x.

[tomwhite]

I ran a few experiments to simulate preemption by stopping a worker VM midway through a job.

Here is a normal run on a cluster on 16 instances with no preemption. It took 102s.

Here is a run where I stopped one worker. The job took longer (124s), but completed fine:

Stopping two workers extends the runtime even more (162s), but the job still completes:

When I tried combining persisting the input dataset (#449) with preemption the results were more mixed. Stopping one worker had the effect of causing disk spilling:

[tomwhite]

Note that all of these experiments were done just by stopping the worker abruptly. There is an unmerged Dask issue to make workers handle shutdown gracefully. The idea is that the worker shutting down would copy its memory state to other workers in the cluster, so the work doesn't need to be recomputed.

This might be a challenge for the GCP preemption limits, however. An instance being preempted on GCP is given 30 seconds notice before being forcibly terminated. A n1-standard-8 instance has 30GB of memory, and a maximum egress bandwidth of 16Gbps. So that means it would take half the notice period (15s) to copy the data to another machine (at 2GB/s), not counting serialization cost etc.

The maximum worker-worker bandwidth I've seen on a cluster has been <0.5GB/s, so there's quite a gap there. (It still might be useful to give the worker notice of an impending shutdown so it doesn't accept new tasks though.)

[tomwhite]

Related: https://www.coiled.io/blog/save-money-with-spot