Refactor SLURMCluster : use memory and thread kwargs, and dynamicaly create the job_script

[guillaumeeb]

First, there is something weird with the SLURMCluster template : only processes are used as a limit, thread number are not taken into account. Moreover, there is no memory limit given.

Secondly, SLURMCluster will need to be updated to match the PBSCluster improvements from #10 (once it is merge): Mainly a more dynamic construct of the submit script. This should prevent some problems as noted in #7, #19 and maybe #20.

[guillaumeeb]

From the conversation in #20, for example in #20 (comment), I have some concern about the SLURM script used for submission.

Googling around, I believe we should at least use the following options:

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<dask_processes * dask_threads>
#SBATCH --mem=<dask_memory * dask_processes>

Another solution maybe to play along with Slurm ntasks=, and tweak the dask-worker command line to only use nprocs=1.

Finaly I have some concern on using sbatch vs srun.

Do we have some Slurm users that could answer that?

cc @bw4sz, @leej3, @bocklund. @luizirber.

[jhamman]

Also ping @davidedelvento, @kmpaul, @jedwards4b as potentially interested user/developers.

[bw4sz]

On our system at UF, Sbatch is preferred. Srun is really on interactive nodes for short periods of time. I believe after 24 hours all jobs are wiped away. Beyond that, we have not experienced a difference in environment.

[guillaumeeb]

For the resources allocations, here are some useful links:

• https://slurm.schedmd.com/cpu_management.html#Example4 shows that with only using ntasks, reservation may span accross several nodes, as pointed out by @bw4sz.
• https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html#shared-memory-example-openmp shows an example for OpenMP, which is quite close to what we want to achieve here, even if we have several processes.
• see also https://slurm.schedmd.com/cpu_management.html#Example2 for how allocations are documented to work.

[jakirkham]

RE: Interactive vs. Batch Jobs

Would think that you would want to have an option to choose either one. Reason being interactive jobs can be good for a Distributed Adaptive Cluster (i.e. dynamic scaling based on demand) whereas batch jobs would be good when not using a Distributed Adaptive Cluster.