Dask client connects to PBS workers, then rapidly loses them

[apatlpo]

The title is intentionally analogous to #20 as I have the feeling the explanation for the observed behavior is similar.

I'm on a PBS cluster whose nodes are made of 2 cpus with 14 cores each.

I was initially calling:

cluster = PBSCluster(queue='mpi_1', local_directory=local_dir, interface='ib0', walltime='24:00:00',
                     threads=4, processes=7, memory='10GB', resource_spec='select=1:ncpus=28:mem=100g', 
                     death_timeout=100)

This led to the creation of workers but they died after creation.

The following choice seems to fix the issue:

threads=14, processes=2, memory='50GB', 

Here is a link that describes dask workers:
http://distributed.readthedocs.io/en/latest/worker.html
this may be useful to readers having similar issues

Note that the link between cluster architecture and options that can be passed to PBSCluster is still not entirely clear to me.

So my issue seems to be fixed, but I wanted to put this experience visible to people that may encounter similar issues.

[guillaumeeb]

The only difference I see between the two choices is that the second one will only spawn 2 linux processes where the first one tries to spawn 7. Could you ask your cluster admin if there is some limit on the number of spawned processes on the queue or the servers belonging to the queue you are using?

Another thing will be to try to launch worker manually and see what happens...

[apatlpo]

thanks guillaume.

I am not aware of such limit and will ask admins.

Here is qstat outputs if anybody can see useful information about this:

iwave_sst/datarmor% qstat -Qf mpi_1
Queue: mpi_1
    queue_type = Execution
    total_jobs = 3
    state_count = Transit:0 Queued:0 Held:1 Waiting:0 Running:2 Exiting:0 Begun
	:0
    resources_max.ncpus = 28
    resources_max.nodect = 1
    resources_max.walltime = 48:00:00
    resources_min.ncpus = 28
    resources_default.mem = 60gb
    resources_default.mpiprocs = 28
    resources_default.ncpus = 28
    resources_default.place = group=switch
    resources_default.select = 1:ncpus=28:mpiprocs=28:mem=60gb
    default_chunk.cluster = icexa
    default_chunk.vntype = no_hyperthreads
    resources_assigned.mem = 184320mb
    resources_assigned.mpiprocs = 36
    resources_assigned.ncpus = 56
    resources_assigned.nodect = 2
    enabled = True
    started = True

I'm travelling right now and accessing the cluster is slow (at least in Orly) but will also
give a shot to interactive launch.

[apatlpo]

Here is the reply that I got from admins:

hi aurelien,

spawn means with mpi spawning?

tina

Le 16/04/2018 à 15:55, Aurelien Ponte a écrit :
> Tina,
> Here is there some limit to the number of spawned processes on mpi_1 ?
> thanks
> aurélien 

How can I described what type of processes jobqueue is spawning?

[guillaumeeb]

I'm not really confident that this could be the issue, but here is what append:

• When starting with 2 processes, 14 threads, there will be 2 linux processes launched in your job, 2 processes running python, using up to 14 cpus with threads.
• With 7 processes: 7 independent linux processes, so things you could see using top or ps. There may potentially be a limit here.

The other thing is that this module does not use MPI at all. The queue you are using seems dedicated to MPI jobs, I don't know if there may be some other things preventing not MPI processes to run well?

[jhamman]

@apatlpo - if you have some time, it may be worth giving jobqueue another go. We've had a number of new features land in recent weeks and you may find things easier to work.

[apatlpo]

yes, I have been keeping up with these new developments and have been smoothly using dask-jobqueue 0.2.0 on a routine basis since its availability.
I am looking forward to try 0.3.0 soon.

I did not realise it was still open and am thus going close this issue, sorry about that

[jhamman]

No worries @apatlpo. Glad to hear you got things working for you.