Set -N=1/--nodes=1 for SLURM workers

[lpsinger]

The SLURM worker hardcodes --ntasks=1/-n=1, but on the SLURM cluster that I have experience with (SDSC Expanse), you also have to set -N=1/--nodes=1.

[guillaumeeb]

These configurations are always tricky and specific, and I'm not sure this particular option is really needed in most Slurm cluster? The real question actually is: does this option is available and mandatory in most Slurm cluster configuration?

You can always set it with dask-jobqueue kwargs in your context.

[lpsinger]

These configurations are always tricky and specific, and I'm not sure this particular option is really needed in most Slurm cluster? The real question actually is: does this option is available and mandatory in most Slurm cluster configuration?

I think it is quite likely that it is available in every Slurm cluster configuration because this option is documented by the sbatch man page. I don't know whether it is mandatory on every cluster, but I also don't see that there would be any harm even if it isn't required. Slurm won't let you set --nodes to a value greater than --ntasks. Here's what happens if you try to do that:

$ srun --pty -A project -p debug --nodes=2 --ntasks=1 --cpus-per-task=1 -t 00:00:05 sleep 1
srun: warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: job 38602701 queued and waiting for resources
srun: job 38602701 has been allocated resources