No workers info available after start_workers() with SLURMCluster

[DAM00x12]

Start workers are "not working" (partially) for me on a SLURMCluster but at the SLURM status I can still see the jobs running, but no workers info available in client.scheduler_info() !! so, I can not stop, manage or run graphs code using Dask API.

In [1]: from dask_jobqueue import SLURMCluster

In [2]: cluster =  SLURMCluster(job_cpu=2,memory='50MB',cores=1,interface='enp0s8')

In [3]: cluster                                                                                                                                   
Out [3]: SLURMCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)

In [4]: from dask.distributed import Client

In [6]: client  = Client(cluster)

In [7]: client
Out [7]: <Client: scheduler='tcp://10.0.1.8:35619' processes=0 cores=0>

In [7]: workers = cluster.start_workers(2)

In [8]: workers
Out[9]:  # NO OUTPUT OF WORKERS instance

In [9]: client.scheduler_info()                                                                                                                  
Out[9]: 
{'type': 'Scheduler',
 'id': 'Scheduler-b04cb227-caec-44fe-9e0d-9180908c1b6e',
 'address': 'tcp://10.0.1.8:35619',
 'services': {'bokeh': 8787},
 'workers': {}}

• dask.version is 1.1.2 and tornado version is 5.1.1

from SLURM perspective, dask workers are running correctly on the HPC:

[admin@master ~]$ qstat 
Job id              Name             Username        Time Use S Queue          
------------------- ---------------- --------------- -------- - --------------- 
166                 dask-worker      root            00:00:00 R VM-CPU-Node    
167                 dask-worker      root            00:00:00 R VM-CPU-Node 

[admin@master ~]$ squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               167 VM-CPU-No dask-wor     root R       0:01      1 worker3
               166 VM-CPU-No dask-wor     root  R       0:01      1 worker2

Well, i'm not sure if I'm causing this problem or samething goes wrong with Dask.

[guillaumeeb]

Hi @DAM00x12, thanks for using dask-jobqueue!

First thing I would do in you case is look at the stdout and stderr of the Slurm jobs, and see if you can find something there. By experience it is often due to network problem (wrong interface or things like that).

For more information on how to debug, it's there: http://jobqueue.dask.org/en/latest/debug.html.

Also, the standard way of starting worker is using the scale() function. I believe that start_worker should work as you intended, but if I'm not mistaken, it doesn't output anything anymore. You can print your cluster object to see if workers are seen as pending or running, or directly use cluster.pending_jobs or cluster.running_jobs.

[DAM00x12]

I think if the problem comes from the network config or interface, I can't figure out the jobs running in each NODELIST of SLURM (workers).

Debugging dask_jobqueue :

In [1]: import logging 
   ...: logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)                                                              

In [2]: from dask_jobqueue import SLURMCluster                                                                                                    
DEBUG:Using selector: EpollSelector

In [3]: from dask.distributed import Client                                                                                                       

In [4]: cluster =  SLURMCluster(job_cpu=1,memory='50MB',cores=1,interface='enp0s8')                                                               
DEBUG:Using selector: EpollSelector
DEBUG:Using selector: EpollSelector
DEBUG:Job script: 
 #!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=48M
#SBATCH -t 00:30:00
JOB_ID=${SLURM_JOB_ID%;*}


/software/miniconda/bin/python -m distributed.cli.dask_worker tcp://10.0.1.8:39704 --nthreads 1 --memory-limit 50.00MB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface enp0s8


In [5]: cluster.scale(2)                                                                                                                          

In [6]: DEBUG:Found 0 active/pending workers.                                                                                                     
DEBUG:Scaling up to 2 workers.
DEBUG:Found 0 active/pending workers.
DEBUG:starting 2 workers
DEBUG:writing job script: 
#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=48M
#SBATCH -t 00:30:00
JOB_ID=${SLURM_JOB_ID%;*}



/software/miniconda/bin/python -m distributed.cli.dask_worker tcp://10.0.1.8:39704 --nthreads 1 --memory-limit 50.00MB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface enp0s8

DEBUG:Executing the following command to command line
sbatch --parsable /tmp/tmph2vdjsx8.sh
DEBUG:started job: 176
DEBUG:writing job script: 
#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=48M
#SBATCH -t 00:30:00
JOB_ID=${SLURM_JOB_ID%;*}



/software/miniconda/bin/python -m distributed.cli.dask_worker tcp://10.0.1.8:39704 --nthreads 1 --memory-limit 50.00MB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface enp0s8

DEBUG:Executing the following command to command line
sbatch --parsable /tmp/tmpjep_pkqv.sh
DEBUG:started job: 177
DEBUG:adding worker tcp://10.0.1.8:44354
DEBUG:job id for new worker: 176
DEBUG:176 is a new job or restarting worker
DEBUG:176 is a new job, adding to running_jobs
DEBUG:removing worker tcp://10.0.1.8:44354
DEBUG:removing worker name (dask-worker--176--) and job_id (176)
DEBUG:that was the last worker for job 176
In [6]: client  = Client(cluster)                                                                                                                 
DEBUG:Using selector: EpollSelector
DEBUG:Using selector: EpollSelector

In [7]: client.scheduler_info()                                                                                                                   
Out[7]: 
{'type': 'Scheduler',
 'id': 'Scheduler-a4600b08-b7d7-408f-a638-12f7936055c6',
 'address': 'tcp://10.0.1.8:39704',
 'services': {'bokeh': 8787},
 'workers': {}}

In [8]: cluster.running_jobs                                                                                                                      
Out[8]: OrderedDict()

In [9]: cluster.pending_jobs                                                                                                                     
Out[9]: OrderedDict([('177', {})])

In [10]: def inc(x): 
    ...:     return x + 1                                                                                                                                        

In [11]: a = client.submit(inc, 10)                                                                                                               

In [12]: a                                                                                                                                        
Out[12]: <Future: status: pending, key: inc-e9ab0fd06a201a9b82686bb9e31e845c>

In [13]: a.result()       ### NO OUTPUT .... ###

SLURM perspective:

[admin@master ~]$ squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               176 VM-CPU-No dask-wor     root  R       0:11      1 worker3
               177 VM-CPU-No dask-wor     root  R       0:11      1 worker1

NOTE: Slurm work properly ( srun, salloc, sbatch, ...) without the Dask API.

[guillaumeeb]

It seems like one of the worker is connecting, but it looks it's running on the same node as your scheduler:

DEBUG:job id for new worker: 176
DEBUG:176 is a new job or restarting worker
DEBUG:176 is a new job, adding to running_jobs
DEBUG:removing worker tcp://10.0.1.8:44354
DEBUG:removing worker name (dask-worker--176--) and job_id (176)
DEBUG:that was the last worker for job 176

You should look into the output files of your jobs, so here 176 et 177, that would give more information.

For the network, try to enter ifconfig on both the node where you start the cluster, and a compute node.

[DAM00x12]

``

[guillaumeeb]

Did you find something @DAM00x12 ?