LSFCluster may be overly specific?

[mrocklin]

I was trying out dask-jobqueue on the Summit supercomputer at Oak Ridge National Labs. I ran into a number of problems with our current configuration that seem to be special cases. I propose un-special-casing these, but I would like to get feedback from others who use LSF.

bsub command

We currently do some odd things with bsub. This made things fail for me on a login node (although they did work on a compute node). Removing this special-cased behavior made things work well for me in both places.

diff --git a/dask_jobqueue/lsf.py b/dask_jobqueue/lsf.py
index 95042a8..12126ee 100644
--- a/dask_jobqueue/lsf.py
+++ b/dask_jobqueue/lsf.py
@@ -53,7 +53,7 @@ class LSFJob(Job):
     """,
         4,
     )
-    submit_command = "bsub <"
+    submit_command = "bsub"
     cancel_command = "bkill"

     def __init__(
@@ -134,10 +134,6 @@ class LSFJob(Job):

         logger.debug("Job script: \n %s" % self.job_script())

-    def _submit_job(self, script_filename):
-        piped_cmd = [self.submit_command + " " + script_filename + " 2> /dev/null"]
-        return self._call(piped_cmd, shell=True)

cc'ing @raybellwaves and @guillaumeeb , who show up under git blame for this code

-R "span[hosts=1]"

My particular deployment didn't like these lines. I don't know if these are very useful generally though, and I can work around them.

            if ncpus > 1:
                # span[hosts=1] _might_ affect queue waiting
                # time, and is not required if ncpus==1
                header_lines.append('#BSUB -R "span[hosts=1]"')

cc @louisabraham (also pointed to by git blame)

[raybellwaves]

Not at UM anymore. Pinging a colleague who may use this or know someone who can give feedback @milancurcic

[louisabraham]

Hi,

-    submit_command = "bsub <"
+    submit_command = "bsub"

I know bsub can read the command from the args (not sure about stdin). Is this what this is about?

-R "span[hosts=1]"

I can confirm that this is necessary to allocate all processes on the same node. I have run into situations (with manually scheduled jobs on 32 cores) where not giving this argument resulted in the cores getting divided between multiple machines (like 16+16). For smaller numbers of cores, it might not be a problem though as LSF probably won't kill your job aggressively, but for the principle you should declare what you are actually using.

[lesteve]

We currently do some odd things with bsub. This made things fail for me on a login node (although they did work on a compute node). Removing this special-cased behavior made things work well for me in both places.

This is weird and from what I remember from stdin was needed in @raybellwaves case (#78). Edit: I found this: #78 (comment)

[guillaumeeb]

@mrocklin, what is you LSF version?

Its pretty certain you've got to use bsub < jobscript with LSF before. You can find it here or there

But I can find it in newer IBM docs.

For -R "span[hosts=1]", this is still documented.

[mrocklin]

10.1.0.0, Jul 30 2019

[guillaumeeb]

So this is the last one, maybe they changed the job script behaviour with this one, do you have access to IBM support through Summit staff?

The non working -R is weird though.

[louisabraham]

I'm using IBM Spectrum LSF 10.1.0.0 build 492718, May 25 2018 and -R "span[hosts=1]" works flawlessly

[guillaumeeb]

@louisabraham do you use bsub < job_script or bsub job_script?

[louisabraham]

@guillaumeeb bsub job_script

[lesteve]

@louisabraham I am a bit confused:

• are you using dask_jobqueue.LSFCluster on your cluster right now? It seems to me that dask-jobqueue.LSFCLuster is using bsub < job_script.
• does both bsub < job_script and bsub job_script work on your cluster?

In an ideal world we would figure out a way to make LSFCluster work both for older version of LSF and the newer versions of LSF.

[louisabraham]

@lesteve Sorry for not being clear, I'm not currently using dask_jobqueue.LSFCluster but I use bsub job_script daily. Do you want me to test dask_jobqueue.LSFCluster?

[lesteve]

Do you want me to test dask_jobqueue.LSFCluster?

Can you first try to post the error you get if you do: bsub < job_script.

If you can give dask_jobqueue.LSFCluster a quick go, it would be useful too. Something like this should be good enough:

import time

from dask.distributed import Client

from dask_jobqueue import LSFCluster

# you may need additional arguments like queue and possibly others in `LSFCluster`
# look at cluster.job_script(), this is the script that is actually used with bsub
# and LSFCluster docstring
cluster = LSFCluster(cores=1, memory='1GB')
cluster.scale(1)

client = Client(cluster)

while len(client.scheduler_info()['workers']) < 1:
    print('waiting for workers')
    time.sleep(5)

fut = client.submit(lambda x: x, 1)
result = fut.result()
print('Got result:', result)

I am expecting is that you'll get an error at the cluster.scale line saying that the submit_command returned with a non-zero exit code.

If you have any feed-back about why you stopped using dask-jobqueue on your cluster, I would be curious to hear more about it!

[SimonBoothroyd]

Its pretty certain you've got to use bsub < jobscript with LSF before. You can find it here or there

On the cluster we use jobqueue on we seem to have to use bsub < jobscript - using just bsub jobscript results in the jobscript being ignored.

[stuarteberg]

I'm also having trouble with this change:

-    submit_command = "bsub <"
+    submit_command = "bsub"

LSF outputs the following error:

/groups/flyem/home/bergs/.lsbatch/1571082744.66049265: line 8: /tmp/tmpggve1epx.sh: No such file or directory

So, /tmp/tmpggve1epx.sh is the submission script generated by dask_jobqueue/lsf.py, but once it gets sent to an LSF host (worker node), that node can't find the script. Of course that's no surprise -- it was written to a location (/tmp) that isn't on the shared file system.

If dask-jobqueue uses the old submit command (bsub <), then the entire script is "spooled", so there's no need for the target machine to have access to the file.

Thankfully, I notice that there is a new configuration option available, introduced in #347:

jobqueue:
  lsf:
    use-stdin: true

Using that setting fixes the issue for me.

BTW, I notice that use-stdin defaults to True IFF you're using (LSF < 10). I didn't track down the reasoning behind that, but in my case I'm using LSF 10.1.0.8 and use-stdin is still necessary for me. I wonder if that configuration option should default to true?

FWIW, here's my LSF version:

$ lsid | head -n1
IBM Spectrum LSF Standard 10.1.0.8, May 10 2019