dask dashboard not showing workers after cluster.stop_workers(cluster.jobs)

[apatlpo]

On PBS with dask-jobqueue #25
If I start a cluster, kill workers with cluster.stop_workers(cluster.jobs) and try to restart a new cluster, then workers do not show up in the dashboard.
Is it expected behavior?
Could we do something about this?

[apatlpo]

I should also mention that the new set of worker seem to be working correctly

[guillaumeeb]

@apatlpo can you describe precisely the commands you are launching? Are you using cluster.restart()?

If so, I imagine it is failing because we do not update worker launching command to point to the new scheduler address (scheduler port is probably changing on restart).

But you could achieve something similar by only launching cluster.start_worker(n) or cluster.scale_up.

[mrocklin]

Unrelated note, but the scheduler port should not be affected during a restart. In general the cluster managers (like dask-jobqueue) shouldn't have to care much about restart events, that should be handled well within dask itself (hypothetically at least).

…

On Wed, Apr 4, 2018 at 3:30 PM, Guillaume EB ***@***.***> wrote: @apatlpo <https://github.com/apatlpo> can you describe precisely the commands you are launching? Are you using cluster.restart()? If so, I imagine it is failing because we do not update worker launching command to point to the new scheduler address (scheduler port is probably changing on restart). But you could achieve something similar by only launching cluster.start_worker(n) or cluster.scale_up. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFRlW28Y2hYrJPAfOxnk7SEKeV2Nks5tlR9AgaJpZM4TG8KJ> .

[guillaumeeb]

Obviously you are right @mrocklin. We'll wait for @apatlpo to clarify.

[apatlpo]

I believe this was just noise on my part, deeply sorry.

It works fine today with:

cluster.stop_workers(cluster.jobs)

and then:

cluster.start_workers(4)

Even client.restart() and client.upload_file() are working.

I have other issues (workers dying) but these will motivate other posts.

Sorry again