Make it possible to configure different interfaces for scheduler and workers

[ogrisel]

I am using creating SLURMCluster(cores=8, memory="24 GB", interface="ib0") on the cluster front node that has an infiniband with IPv4 address on ib0. However I cannot create workers with scale as the ib0 interface on the worker nodes have no IPv4 addresses:

Traceback (most recent call last):
  File "/scratch/ogrisel/miniconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 259, in <module>
    go()
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 255, in go
    main()
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/distributed/cli/dask_worker.py", line 206, in main
    host = get_ip_interface(interface)
  File "/scratch/ogrisel/miniconda3/lib/python3.7/site-packages/distributed/utils.py", line 169, in get_ip_interface
    raise ValueError("interface %r doesn't have an IPv4 address" % (ifname,))
ValueError: interface 'ib0' doesn't have an IPv4 address

However those worker nodes have a working IPv4 address on interface ib1.

Would it be possible to configure different interfaces for the workers and the scheduler?

[guillaumeeb]

I think you can specify Scheduler interface by using ip kwarg, and then interface will only apply to worker. So something like

cluster = SlurmCluster(..., ip=ib0schedulerIP, interface='ib1')

But Scheduler needs to listen on the good interface.

Another option may be to make Scheduler listen on every interface using ip='' IIRC.

[guillaumeeb]

@ogrisel I'm closing this as I believe you should be able to make it work with my first answer. Feel free to reopen if it does not work.

[jernsting]

This seems to be still a problem for me. The usage of the ip= parameter is not working in my setup. The ip address in the SLURMCluster() address is still the same.

[lesteve]

Which version of dask-jobqueue are you using? If you are using dask-jobqueue 0.7, you may want to try using host instead of ip:

cluster = SLURMCluster(..., host=ib0schedulerIP, interface='ib1')

[MaximilianHoffmann]

I have a related problem?

cluster = SLURMCluster(n_workers=100,cores=24,
                       processes=1,
                       memory="64GB", host="172.16.59.141"   ,walltime="04:00:00", interface='em1')

results in

ValueError: interface 'em1' doesn't have an IPv4 address

[lesteve]

If you install the development version of Dask-Jobqueue you can specify a different interface for the Dask workers and the Dask scheduler using something like #382 (comment).

If you still have a problem with IPv4, please open a separate issue with the full stack-trace.