Skip to content

souporcell_pipeline.py: ensure that logs subdirectory exists#278

Open
carandraug wants to merge 1 commit into
wheaton5:masterfrom
carandraug:mkdir-logs
Open

souporcell_pipeline.py: ensure that logs subdirectory exists#278
carandraug wants to merge 1 commit into
wheaton5:masterfrom
carandraug:mkdir-logs

Conversation

@carandraug
Copy link
Copy Markdown

@carandraug carandraug commented May 11, 2026

souporcell looks at whether out_dir already exists to identify if it is restarting a pipeline. This is then used to know if it needs to create the logs subdirectory.

Why is this an issue for me? I'm using mktemp -d to create separate directories for parallel souporcell_pipeline.py processes. So I'm passing an existing, but empy, out_dir to souporcell. Because souporcell uses whether out_dir exists to identify if it is resuming a previous run, it then skips the creation of the logs directory, leading to the following error later on:

Traceback (most recent call last):
  File "/opt/souporcell/souporcell_pipeline.py", line 631, in <module>
    minimap_tmp_files = remap(args, region_fastqs, all_fastqs)
  File "/opt/souporcell/souporcell_pipeline.py", line 246, in remap
    with open(args.out_dir + "/logs/minimap.err",'w') as minierr:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tmp-souporcell-kMN0mH/logs/minimap.err'

Actually, the whole condition is unnecessary since os.makedirs(... exist_ok=True) will already do the check and only create the directories if needed.

If out_dir already exists it does not mean that it is restarting a
pipeline.  Even if it does, it does not mean that a logs subdirectory
was successfully created on the previous run.

This commit ensures that whether the logs directory is created is
based on whether that directory exists and not whether its parent
directory exists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant