Allow global access to dashboard by default

[mrocklin]

When using interfaces like infiniband it's often the case that one can't get the dashboard to show up. This is because the bokeh server is expecting to be called by its IP address on the infiniband network, not the normal network.

One solution to this would be to provide services={('bokeh', ('', 8787)): BokehScheduler} to LocalCluster.

[lesteve]

Interesting, I do seem to remember to have bumped into this problem ...

[lesteve]

Hmmm I think I got something like you suggested working, but the _repr_html_ is not showing the right port for some reason ...

[lesteve]

Just to put a bit more context, here is my diff (I hard-code the BokehScheduler port to 12345):

diff --git a/dask_jobqueue/core.py b/dask_jobqueue/core.py
index ab4b602..9bc467e 100644
--- a/dask_jobqueue/core.py
+++ b/dask_jobqueue/core.py
@@ -9,6 +9,7 @@ import warnings
 import dask
 import docrep
 from distributed import LocalCluster
+from distributed.bokeh.scheduler import BokehScheduler
 from distributed.deploy import Cluster
 from distributed.utils import (get_ip_interface, ignoring, parse_bytes, tmpfile,
                                format_bytes)
@@ -153,7 +154,8 @@ class JobQueueCluster(Cluster):
         else:
             host = socket.gethostname()
 
-        self.local_cluster = LocalCluster(n_workers=0, ip=host, **kwargs)
+        self.local_cluster = LocalCluster(n_workers=0, ip=host,
+                                          services={('bokeh', ('', 12345)): BokehScheduler}, *kwargs)
 
         # Keep information on process, cores, and memory, for use in subclasses
         self.worker_memory = parse_bytes(memory)

I run this ssh proxy command (redirecting the 12345 port to 9898 locally):

ssh -N lesteve@sequoia -L 9898:sequoia:12345 # dashboard

and then I can access the dashboard at http://localhost:9898/status.

The widget for client.cluster shows the dashboard at 8787 though:

Dashboard: http://10.149.255.254:8787/status

I made sure I had the latest release of dask (0.18.1) and distributed (1.22).

[mrocklin]

Indeed. It looks like this is broken in dask/distributed. We just do a hard overwrite of the bokeh key:

https://github.com/dask/distributed/blob/fd8ca5ebc3a781d5f6625d3daa4d09ce806938a3/distributed/deploy/local.py#L119-L127

[guillaumeeb]

Don't you think this modification should be done upstream in distributed? Always bind bokeh UI to all network address by default?

[mrocklin]

That would make things simpler, yes, but I think that we don't want to do it in the true LocalCluster case, where we expect to be on the localhost and would be confused if someone else connected to our machine and wanted to see our dashboard. The fact that our cluster objects use LocalCluster is a bit of a historical artifact. Ideally they would create a scheduler manually with all of their own options.

…

On Sun, Jul 15, 2018 at 6:32 AM, Guillaume EB ***@***.***> wrote: Don't you think this modification should be done upstream in distributed? Always bind bokeh UI to all network address by default? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#90 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszEds3YlOsQf6q5kGiJzDYpij1qmfks5uGyg2gaJpZM4VLb_x> .

[guillaumeeb]

and would be confused if someone else connected to our machine and wanted to see our dashboard.

Would we? Would that be happening a lot?

The fact that our cluster objects use LocalCluster is a bit of a historical artifact. Ideally they would create a scheduler manually with all of their own options.

Should we do that? Or maybe implement some more generic superclass to LocalCluster upstream, that we can reuse here?

[mrocklin]

Perhaps. I haven't investigated this enough to know what the right approach is. If someone wants to make progress on this they might look at all of the modifications that LocalCluster does when starting up a scheduler and write those up so that it can help inform discussion. I suspect that we might be able to put some of that logic into the Scheduler constructor itself, or perhaps some other class as you suggest. I suspect that there will be some balance between making Scheduler construction fully featured and keeping it lightweight.

…

On Wed, Aug 1, 2018 at 7:27 AM, Guillaume EB ***@***.***> wrote: and would be confused if someone else connected to our machine and wanted to see our dashboard. Would we? Would that be happening a lot? The fact that our cluster objects use LocalCluster is a bit of a historical artifact. Ideally they would create a scheduler manually with all of their own options. Should we do that? Or maybe implement some more generic superclass to LocalCluster upstream, that we can reuse here? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#90 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszOdcw5okFAmShvm1bcS_tCNiurggks5uMZCkgaJpZM4VLb_x> .

[guillaumeeb]

This feature is important for being able to use Dask easily with infiniband, and to make sure it works with nbserverproxy.

I observed the same problem as @lesteve trying its diff, the other way around though (specifing 8787, having some random port displayed, but dashboard up on 8787), but I assume it is the same issue.

Current solution is not really elegant though, it is more of a hack. Ideally, I would prefer to have at least a kwargs in LocalCluster, so we can call something like:

self.local_cluster = LocalCluster(n_workers=0, ip=host, dashboard_bind_ip='0.0.0.0', **kwargs)

What do you think about that?

[guillaumeeb]

Another solution, that does not need modification to LocalCluster, is to just use ip='' parameter when launching LocalCluster. From what I understant looking at dask distributed sources, it will bind all services (scheduler too), to listen to all interfaces.

Do someone believe that this is a problem ?

With #107, we can easily do this:

cluster = PBSCluster( ... , interface='ib0', ip='')

But we could also make it by default directly in core.py:

        # Find the IP address and (if requested) set interface for workers
        localhost_kwargs = {}
        if interface:
            localhost_kwargs['ip'] = get_ip_interface(interface)
            extra += ' --interface  %s ' % interface
        else:
            localhost_kwargs['ip'] = ''

Or just drop #107, because binding on '' by default should solve it...

Any thoughts?

[mrocklin]

Binding on all interfaces seems appropriate to me for HPC situations

…

On Tue, Aug 7, 2018 at 5:38 AM, Guillaume EB ***@***.***> wrote: Another solution, that does not need modification to LocalCluster, is to just use ip='' parameter when launching LocalCluster. From what I understant looking at dask distributed sources, it will bind all services (scheduler too), to listen to all interfaces. Do someone believe that this is a problem ? With #107 <#107>, we can easily do this: cluster = PBSCluster( ... , interface='ib0', ip='') But we could also make it by default directly in core.py: # Find the IP address and (if requested) set interface for workers localhost_kwargs = {} if interface: localhost_kwargs['ip'] = get_ip_interface(interface) extra += ' --interface %s ' % interface else: localhost_kwargs['ip'] = '' Or just drop #107 <#107>, because binding on '' by default should solve it... Any thoughts? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#90 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszPSXjv-irNeS0dXXFhuVSh4tXaXFks5uOWAKgaJpZM4VLb_x> .

[guillaumeeb]

So @mrocklin, once #107 is in (I believe we should keep it, to give the choice to the user to not bind on all interfaces if he wants), is this okay for you if I propose another PR wich sets LocalCluster ip = '' by default ?

This will probably remove some lines from the one added in #107, as this default will get set no matter if interface is provided or not. I prefer to do this in another PR to keep things clear.

[mrocklin]

The common case use of LocalCluster is for local machines. It should continue to respond only to localhost or 127.0.0.1 by default for these cases. Otherwise we open up all dask users to having their machines stolen. Recall that Dask opens up a connection that other people can connect to to run arbitrary Python code. Having the required address for the TCP server remain localhost is a very useful way to provide some level of security. This is the same as with other applications, like Jupyter.

…

On Tue, Aug 7, 2018 at 8:37 AM, Guillaume EB ***@***.***> wrote: So @mrocklin <https://github.com/mrocklin>, once #107 <#107> is in (I believe we should keep it, to give the choice to the user to not bind on all interfaces if he wants), is this okay for you if I propose another PR wich sets LocalCluster ip = '' by default ? This will probably remove some lines from the one added in #107 <#107>, as this default will get set no matter if interface is provided or not. I prefer to do this in another PR to keep things clear. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#90 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszGgjnWh5xWhD_8gqvYELsKWHFbhpks5uOYoLgaJpZM4VLb_x> .