No workers info available after scale() with SLURMCluster

[AChatzigoulas]

I have a very similar issue with #246. After I create the cluster and scale it the nodes are running but no CPUs are available.

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
from dask_jobqueue import SLURMCluster
cluster = SLURMCluster()

DEBUG:Using selector: EpollSelector
DEBUG:Using selector: EpollSelector

cluster.scale(jobs = 2)

DEBUG:Starting worker: 1
DEBUG:writing job script: 
#!/usr/bin/env bash

#SBATCH -J dask-worker
#SBATCH -p compute
#SBATCH -A pr008033
#SBATCH -n 1
#SBATCH --cpus-per-task=20
#SBATCH --mem=45G
#SBATCH -t 00:30:00
#SBATCH -o myjob.%j.out
#SBATCH -e myjob.%j.err

JOB_ID=${SLURM_JOB_ID%;*}

/users/pr008/...
tcp://195.251.23.79:33491 --nthreads 20 --memory-limit 48.00GB --name 1 --nanny --death-timeout 60 --interface ib0

DEBUG:Executing the following command to command line
sbatch /tmp/tmpoals04ok.sh
DEBUG:Starting job: 804606
DEBUG:Starting worker: 0
DEBUG:writing job script: 
#!/usr/bin/env bash

#SBATCH -J dask-worker
#SBATCH -p compute
#SBATCH -A pr008033
#SBATCH -n 1
#SBATCH --cpus-per-task=20
#SBATCH --mem=45G
#SBATCH -t 00:30:00
#SBATCH -o myjob.%j.out
#SBATCH -e myjob.%j.err

JOB_ID=${SLURM_JOB_ID%;*}

/users/pr008/...
tcp://195.251.23.79:33491 --nthreads 20 --memory-limit 48.00GB --name 0 --nanny --death-timeout 60 --interface ib0

DEBUG:Executing the following command to command line
sbatch /tmp/tmpsuq3ysbm.sh
DEBUG:Starting job: 804607

from dask.distributed import Client
client = Client(cluster)
client

Client Scheduler: tcp://195.251.23.79:33491 Dashboard: http://195.251.23.79:8787/status	Cluster Workers: 0 Cores: 0 Memory: 0 B

client.scheduler_info()

{'type': 'Scheduler',
 'id': 'Scheduler-5198da07-3700-42aa-a068-036146073ef6',
 'address': 'tcp://195.251.23.79:33491',
 'services': {'dashboard': 8787},
 'workers': {}}

When I run squeue

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
804606   compute dask-wor  ...  R       6:04      1 node287
804607   compute dask-wor  ...  R       6:04      1 node003

The HPC has infiniband and each node has 20 CPUs. If you need any other info please tell me.

Thank you

[guillaumeeb]

Hi @Alexinho3,

First a naive question: do you wait for some times before printing your Client object?

Second, can you look at the stderr and stdout files for your worker jobs?

[AChatzigoulas]

Thank you @guillaumeeb for your answer. Yes I wait until the nodes are active. The stderr and stdout files are empty. I work in a miniconda environment with Python version 3.6.9 and

# Name                    Version                   Build  Channel
dask                      2.9.0                      py_0  
dask-core                 2.9.0                      py_0  
dask-jobqueue             0.7.0                      py_0    conda-forge
distributed               2.9.0                      py_0  

Additionally, when I type cluster, it returns:

Workers | 0 / 2
Cores | 0
Memory | 0 B

[lesteve]

Just a small comment: readability counts, a lot! Please use triple back-quotes aka fenced code blocks to format error messages code snippets. Bonus points if you use syntax highlighting with py for python snippets and pytb for tracebacks.

I have edited your earlier messages accordingly but it would be great if you could try to do it next time!

[lesteve]

It is weird that you don't get any stdout/stderr in your SLURM logs ...

Anyway what this looks like is that your jobs start correctly but that the Dask workers can not connect to the Dask scheduler. One general way of fixing this is to specify interface in SLURMCluster.

Have a look at https://blog.dask.org/2019/08/28/dask-on-summit#workers-dont-connect-to-the-scheduler for more details and let us know if that helps!

[AChatzigoulas]

Thank you for your answer and the readability corrections @lesteve . I 'll keep that in mind for the future.

I have already specified the interface to be ib0. If I understand correctly, when I type
client.scheduler_info()
the address shown should be that of the ib0? If that 's true then in my case is one of the Ethernet IPs.

[lesteve]

Thank you for your answer and the readability corrections @lesteve . I 'll keep that in mind for the future.

Great to hear! IMO it does make a big difference if you want to have the best chance of getting good feed-back.

I have already specified the interface to be ib0. If I understand correctly, when I type
client.scheduler_info()
the address shown should be that of the ib0? If that 's true then in my case is one of the Ethernet IPs.

Since you are using SLURMCluster() without argument in your snippet file, it does look like you are using a jobqueue.yaml configuration file and you set the inteface in jobqueue.yaml. Can you confirm this is the case?

If this is the case you may well be hit by #358 which is fixed in master (#366).

• as a work-around you could try to specify the interface in the SLURMCluster arguments:

cluster = SLURMCluster(interface='ib0')

• or alternatively, keep interface in your jobqueue.yaml and install the developement version of dask-jobqueue which has a fix for this:

pip install git+https://github.com/dask/dask-jobqueue

In any case let us know whether that works for you!

[AChatzigoulas]

The work-around and the alternative way both worked and the IP is now the Infiniband's. I also found out that I was using a wrong shebang and I fixed that too, but sadly the problem still persists. Can you please guide me into what else might be wrong?

The os Linux version is Red Hat Enterprise Linux Server release 6.10 (Santiago), I don't know if it matters.

[lesteve]

Hmmm hard to tell, can you see your SLURM logs now?

You may have seen it already but maybe this can help:
https://jobqueue.dask.org/en/latest/debug.html

Something else I would try is to do it by hand (i.e. launching dask-scheduler and dask-worker yourself) outside of dask-jobqueue following:
https://docs.dask.org/en/latest/setup/hpc.html#using-a-shared-network-file-system-and-a-job-scheduler

[AChatzigoulas]

Yeap, still empty.

I tried the manual way
qsub -b y ~/miniconda3/envs/alekos/bin/dask-scheduler --scheduler-file /users/pr008/user/scheduler.json --interface ib0

qsub -b y ~/miniconda3/envs/alekos/bin/dask-worker --scheduler-file /users/pr008/user/scheduler.json

squeue -u user

805813   compute dask-wor  user  R       3:31      1 node144
805809   compute dask-sch  user  R       6:54      1 node144

then I try to create the client

from dask.distributed import Client
client = Client(scheduler_file='/users/pr008/user/scheduler.json')

and this hangs on forever

[lesteve]

Hmmm, can you post the content of scheduler.json?

[AChatzigoulas]

I just saw that it does not create the scheduler.json file.

[lesteve]

Other questions:

• can you see your SLURM logs with the manual way?
• you use qsub in your message above but I assume you meant sbatch right?

[AChatzigoulas]

Yes these logs are again empty. I tried both qsub and sbatch and both of them start the scheduler and the worker but still no connections with the client. Also, the scheduler.json file is not created. When I try on the interactive node (without the --interface ib0) the file is written just fine.

[lesteve]

This is very weird, and probably due to some quirks specific to your cluster setup. Can you post a webpage for the cluster doc if it exists, mostly out of curiosity?

When I try on the interactive node (without the --interface ib0) the file is written just fine.

• by interactive node, do you mean the login node, i.e. the nodes that you ssh to when you want to run commands like squeue, sbatch, etc ...
• why without --interface ib0, because ib0 is not an interface on the login node?

[AChatzigoulas]

The webpage is http://doc.aris.grnet.gr/

Yes I mean the login node and indeed the ib0 is an interface on the login node. The sysadmin told me that although he doesn't know about dask, if the client starts from the login node it won't work. So, I tried it on the "compute" partition which also has a different ib0 address than the login node, but still no luck.

I will try to set up everything from the beginning in the "compute" partition to see if it works.