From what environment should I call SLURMCluster? Parsing Issue in sbatch?

[bw4sz]

I am running on a university cluster with a SLURM submission scheduler. I installed conda and sourced the pangeo environment without error.

I git cloned this repo, opened python and tried

(pangeo) [b.weinstein@gator4 dask-jobqueue]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> 
>>> cluster = SLURMCluster(project="ewhite")
>>> cluster.start_workers(10)
sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

[2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

Where ewhite is the group tag (see below).

In previous submission scripts, I would typically start with something like

#!/bin/bash
#SBATCH --job-name=NEON_LIDAR   # Job name
#SBATCH --mail-type=END               
#SBATCH --mail-user=#######@gmail.com
#SBATCH --account=ewhite
#SBATCH --qos=ewhite-b

#SBATCH --ntasks=1                 # Number of MPI ranks
#SBATCH --cpus-per-task=1            # Number of cores per MPI rank
#SBATCH --mem=4000
#SBATCH --time=00:59:00       #Time limit hrs:min:sec
#SBATCH --output=NEON_LIDAR.out   # Standard output and error log
#SBATCH --error=NEON_LIDAR.err

which works without a problem.

Attempts

I changed

#SBATCH -A %(project)s

to

#SBATCH --account %(project)s

to match previous submission scripts. No change.

So I tried removing the line, which yielded

>>> cluster.start_workers(10)
sbatch: error: Invalid argument: 00:30:00

sbatch: error: Invalid argument: 00:30:00

sbatch: error: Invalid argument: 00:30:00

...

This is an error in the walltime argument. This makes me believe that it doesn't have to do with the argument value, but how all the arguments are parsed and passed to bash. The most likely explanation is that i'm calling this script from an incorrect location, and the variable expansion or quotes are not being parsed in the way I anticipate.

[bw4sz]

If its useful, similar error was reported here, sbatch would not accept spaces. I'll trying to look around and see how/if this could be connected.

[mrocklin]

Maybe we should be enclosing the executable in quotes since we can't control the base_path ?

…

On Tue, Mar 20, 2018 at 5:26 PM, Ben Weinstein ***@***.***> wrote: If its useful, similar error here <nextflow-io/nextflow#218> was reported here, sbatch would not accept spaces. I'll trying to look around and see how/if this could be connected. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszC-wYXPNmutzo7hf4Cz2T9aLSL_aks5tgXQAgaJpZM4SyplE> .

[bw4sz]

I'm still playing with it. Its not clear to me exactly where it is failing. I added a logger to see from sys out,

logger = logging.getLogger(__name__)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)

but I never see the command called.

It yields

(pangeo) [b.weinstein@gator4 dask-jobqueue]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> 
>>> cluster = SLURMCluster(project='ewhite')
>>> cluster.start_workers(3)
2018-03-20 18:07:43,667 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite

2018-03-20 18:07:43,687 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite

2018-03-20 18:07:43,707 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite

but its clear that core should first log the cmd, and that will give us a sense for the parsing.

Line 79

        logger.debug("Submitting the following calls to command line")
        for cmd in cmds:
            logger.debug(' '.join(cmd))

I've tried manually removing the spacing, but without see the cmd parse, i'm guessing a bit. All thoughts welcome.

[mrocklin]

Can you try the solution in #18 ?

…

On Tue, Mar 20, 2018 at 6:12 PM, Ben Weinstein ***@***.***> wrote: I'm still playing with it. Its not clear to me exactly where it is failing. I added a logger to see from sys out, logger = logging.getLogger(__name__) ch = logging.StreamHandler(sys.stdout) ch.setLevel(logging.DEBUG) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') ch.setFormatter(formatter) logger.addHandler(ch) but I never see the command called. It yields (pangeo) ***@***.*** dask-jobqueue]$ python Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from dask_jobqueue import SLURMCluster >>> >>> cluster = SLURMCluster(project='ewhite') >>> cluster.start_workers(3) 2018-03-20 18:07:43,667 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite 2018-03-20 18:07:43,687 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite 2018-03-20 18:07:43,707 - dask_jobqueue.core - ERROR - sbatch: error: Invalid argument: ewhite but its clear that core should first log the cmd, and that will give us a sense for the parsing. Line 79 <https://github.com/dask/dask-jobqueue/blob/9dbdf4b60e074ce3d4cd7de75bc003063343320c/dask_jobqueue/core.py#L77> logger.debug("Submitting the following calls to command line") for cmd in cmds: logger.debug(' '.join(cmd)) I've tried manually removing the spacing, but without see the cmd parse, i'm guessing a bit. All thoughts welcome. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBpd0yIyg0MD3t-kLQZN5_qyc5mRks5tgX7ZgaJpZM4SyplE> .

[bw4sz]

Same behavior.

(pangeo) [b.weinstein@gator3 dask-jobqueue]$ python
Python 3.6.4 | packaged by conda-forge | (default, Dec 23 2017, 16:31:06) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_jobqueue import SLURMCluster
>>> 
>>> cluster = SLURMCluster(project='ewhite')
>>> cluster.start_workers(3)
sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

sbatch: error: Invalid argument: ewhite

[2, 3, 4]
>>> quit()
(pangeo) [b.weinstein@gator3 dask-jobqueue]$ cat dask_jobqueue/slurm.py 
import logging
import os
import socket
import sys

from distributed import LocalCluster
from distributed.utils import get_ip_interface

from .core import JobQueueCluster

logger = logging.getLogger(__name__)

dirname = os.path.dirname(sys.executable)


class SLURMCluster(JobQueueCluster):
    """ Launch Dask on a SLURM cluster

    Examples
    --------
    >>> from pangeo import SLURMCluster
    >>> cluster = SLURMCluster(project='...')
    >>> cluster.start_workers(10)  # this may take a few seconds to launch

    >>> from dask.distributed import Client
    >>> client = Client(cluster)

    This also works with adaptive clusters.  This automatically launches and
    kill workers based on load.

    >>> cluster.adapt()
    """
    def __init__(self,
                 name='dask',
                 queue='',
                 project=None,
                 threads_per_worker=4,
                 processes=8,
                 memory='7GB',
                 walltime='00:30:00',
                 interface=None,
                 death_timeout=60,
                 extra='',
                 **kwargs):
        """ Initialize a SLURM Cluster

        Parameters
        ----------
        name : str
            Name of worker jobs. Passed to `#SBATCH -J` option.
        queue : str
            Destination queue for each worker job.
            Passed to `#SBATCH -p` option.
        project : str
            Accounting string associated with each worker job. Passed to
            `#SBATCH -A` option.
        threads_per_worker : int
            Number of threads per process.
        processes : int
            Number of processes per node.
        memory : str
            Bytes of memory that the worker can use. This should be a string
            like "7GB" that can be interpretted both by PBS and Dask.
        walltime : str
            Walltime for each worker job.
        interface : str
            Network interface like 'eth0' or 'ib0'.
        death_timeout : float
            Seconds to wait for a scheduler before closing workers
        extra : str
            Additional arguments to pass to `dask-worker`
        kwargs : dict
            Additional keyword arguments to pass to `LocalCluster`
        """
        self._template = """
#!/bin/bash

#SBATCH -J %(name)s
#SBATCH -n %(processes)d
#SBATCH -p %(queue)s
#SBATCH -A %(project)s
#SBATCH -t %(walltime)s
#SBATCH -e %(name)s.err
#SBATCH -o %(name)s.out

export LANG="en_US.utf8"
export LANGUAGE="en_US.utf8"
export LC_ALL="en_US.utf8"

"%(base_path)s/dask-worker" %(scheduler)s \
    --nthreads %(threads_per_worker)d \
    --nprocs %(processes)s \
    --memory-limit %(memory)s \
    --name %(name)s-%(n)d \
    --death-timeout %(death_timeout)s \
     %(extra)s
""".lstrip()

        if interface:
            host = get_ip_interface(interface)
            extra += ' --interface  %s ' % interface
        else:
            host = socket.gethostname()

        project = project or os.environ.get('SLURM_ACCOUNT')
        if not project:
            raise ValueError("Must specify a project like `project='UCLB1234' "
                             "or set SLURM_ACCOUNT environment variable")
        self.cluster = LocalCluster(n_workers=0, ip=host, **kwargs)
        memory = memory.replace(' ', '')
        self.config = {'name': name,
                       'queue': queue,
                       'project': project,
                       'threads_per_worker': threads_per_worker,
                       'processes': processes,
                       'scheduler': self.scheduler.address,
                       'walltime': walltime,
                       'base_path': dirname,
                       'memory': memory,
                       'death_timeout': death_timeout,
                       'extra': extra}
        self.jobs = dict()
        self.n = 0
        self._adaptive = None
        self._submitcmd = 'sbatch'
        self._cancelcmd = 'scancel'

        logger.debug("Job script: \n %s" % self.job_script())

Am I correct in assessing the problem as it adding a space, " ewhite"? I'll be back tomorrow morning.

[mrocklin]

Am I correct in assessing the problem as it adding a space, " ewhite"? I'll be back tomorrow morning.

I don't know. I don't personally have much experience here. You might try creating a SLURM cluster, seeing the template that it produces, and then trying to use that template manually, making sufficient tweaks as necessary. This might allow you to diagnose the problem with the dask-jobqueue project out of the pitcture..

[bw4sz]

Understood, will report back today. I'm certain it's something minor. Hopefully a PR soon.

…

On Wed, Mar 21, 2018, 05:24 Matthew Rocklin ***@***.***> wrote: Am I correct in assessing the problem as it adding a space, " ewhite"? I'll be back tomorrow morning. I don't know. I don't personally have much experience here. You might try creating a SLURM cluster, seeing the template that it produces, and then trying to use that template manually, making sufficient tweaks as necessary. This might allow you to diagnose the problem with the dask-jobqueue project out of the pitcture.. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJwrDA6qKriBh8D9HnkBlshhbBY5HYtks5tgkZ9gaJpZM4SyplE> .