Unparsable sbatch output

[nateybear]

Software Versions

Snakemake version: 9.13.7

Plugin version (via pip freeze):
snakemake-executor-plugin-slurm==1.9.2
snakemake-executor-plugin-slurm-jobstep==0.3.0

Slurm version: 23.11.11

Describe the bug
On my cluster, snakemake fails when trying to parse the name of the log file from sbatch output. It looks like the parsable flag in this case does not produce any output with semicolons, only the job id. Am I doing something wrong with this invocation? I can't see anything else in the sbatch call you're constructing that would lead to a parsable output.

Logs
The snakemake command I am running is:
snakemake --executor slurm --default-resources tasks=1 cpus_per_task=48 nodes=1 runtime=120 slurm_partition=skx mem_mb_per_cpu=150000 --config platform=hpc threads=auto --jobs 20

The error output is:

Traceback (most recent call last):

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/cli.py", line 2193, in args_to_api
    dag_api.execute_workflow(

  File <HOME>/.local/lib/python3.12/site-packages/snakemake/api.py", line 634, in execute_workflow
    workflow.execute(

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/workflow.py", line 1445, in execute
    raise e

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/workflow.py", line 1441, in execute
    success = self.scheduler.schedule()
              ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/scheduling/job_scheduler.py", line 389, in schedule
    raise e

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/scheduling/job_scheduler.py", line 367, in schedule
    self.run(runjobs)

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake/scheduling/job_scheduler.py", line 496, in run
    executor.run_jobs(jobs)

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake_interface_executor_plugins/executors/base.py", line 73, in run_jobs
    self.run_job(job)

  File "<HOME>/.local/lib/python3.12/site-packages/snakemake_executor_plugin_slurm/__init__.py", line 387, in run_job
    slurm_logfile = slurm_logfile.with_name(
                    ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/apps/python/3.12.11/lib/python3.12/pathlib.py", line 634, in with_name
    raise ValueError("Invalid name %r" % (name))

ValueError: Invalid name '-----------------------------------------------------------------\n          Welcome to the Stampede3 Supercomputer                 \n-----------------------------------------------------------------\n\nNo reservation for this job\n--> Verifying valid submit host (login4)...OK\n--> Verifying valid jobname...OK\n--> Verifying valid ssh keys...OK\n--> Verifying access to desired queue (skx)...OK\n--> Checking available allocation <ALLOCATION>...OK\n--> Quotas are not currently enabled for filesystem <HOME>...OK\n--> Verifying that quota for filesystem <WORK> is at 10.62% allocated...OK\n2545659.log'

[nateybear]

Update: I see this issue referenced also in #146, but with no fix issued. Would you suggest I just edit my downloaded version to change how slurm_jobid is parsed? I understand the block comment above that about --parsable output now, looks like my admins prefer verbose output...

[nateybear]

Update2: And it worked for me by changing the parsing: out.strip().splitlines()[-1]

Would it be reasonable to be more modular and let the user control parsing the job id? Even if it's just parsing a string as code from a command line option, it's not the most elegant thing but it's also easy and there's not an attack surface (typical concern for parsing user input as code)

snakemake --executor slurm --default-resources [...] jobid_parser="lambda out: out.strip().splitlines()[-1]"

[cmeesters]

Hi,

thanks for reporting this issue!

You suffer from your administrator eagerness to inform you. In fact, the referenced issue has been solved - I am lagging behind for various reasons.

Well, I do not think that we can require all users to supply lambda functions on the command line, whenever the admins decide to be fancy. I was imaging a regex might help. However: which? Simply parsing an integer at the end? There is no guarantee for it. Any integer? There is no guarantee, that will be the only one. SLURM be default writes: Submitted batch job <id> [on cluster <cluster name>] (the last bit is optional). With --parsable this ought to be `[;]. Your cluster violates this rule.

I am now thinking of a validator: If there is no suitable integer, try to get one from the output. Have multiple attempts, validate again. It is getting cumbersome, but this is the wonderful HPC world: no standardisation between clusters. Not even modest bits.

Enough of my lament: edit: the PR is #375. You may check it out and try it. Do not expect it to be merged until a few days into next week.