Skip to content

08. FAQ

Jump to bottom
Ian Brennan edited this page Sep 3, 2024 · 1 revision

Frequently Asked Questions

Running into trouble? Here's some solutions to common difficulties.

1. What's the best way to install Nextflow?

Nextflow installation should be relatively painless, but it does rely on a recent version of Java (11+). You can check which Java you're running with $ java -version. If you need a newer version of Java, you can start by installing SDK with the command $ curl -s "https://get.sdkman.io" | bash. Once that installation has completed and you've refreshed your terminal, you should be able to install a newer Java with SDK, e.g. $ sdk install java 17.0.6-amzn. Finally, you can download Nextflow with $ wget -qO- https://get.nextflow.io | bash, then make the binary executable with $ chmod +x nextflow. Feel free to move the nextflow file to the directory of your choice, and add it to your $PATH variable so you can call it with $ nextflow instead of specifying the full path name.

2. How do I customize memory/CPU requirements?

Memory and CPU allotment are easy to customize on a per-process basis. In your pipesnake directory, open conf/base.config. Adjust the relevant parameters and save this new file (e.g. new_params.config). Once you are ready to run a new analysis, pass this new config file along to the pipeline with the -c flag, e.g. -c new_params.config. Make sure to store your new_params.config file somewhere safe, as it will be removed if you pull a new version of the pipeline.

3. How do I control what output files to keep?

By default, pipesnake will keep the output files of the following processes:

  • MERGE_TREES (in [outdir]/merge/)
  • MAKE_PRG (in [outdir]/make/)
  • GBLOCKS (in [outdir]/gblocks/)
  • IQTREE or RAXML (in [outdir]/iqtree or [outdir]/raxml)
  • ASTER (in [outdir]/aster)
  • MAFFT (in [outdir]/mafft)

If you would like to keep the files produced by other processes, you can do that in one of two ways.

(1) By specifying the process you'd like to keep with the appropriate flag set to true, e.g. --bbmap_dedupe_keep_output true.

(2) You could do the above for all the processes, but to simplify the pipesnake command you can instead adjust the relevant parameters in the conf/base.config file (the same as above for memory/CPU usage). Open the file and scroll down to the the keep_output section (lines 200-229), change whichever parameters you'd like, then pass the new config file along to the pipeline with the -c flag, e.g. -c new_params.config.

4. How do I avoid my run getting disconnected?

Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).

5. Something is wrong with my current pipesnake

Weird things happen with computers. One of the first things you can try is removing the cached version of pipesnake stored on your computer and replace it with a fresh pull from github. Start by locating and removing the cached version, it's likely in the directory ~/.nextflow/assets/ausarg/pipesnake. Let's remove this whole directory for a start.

rm -rf ~/.nextflow/assets/ausarg/pipesnake

Now that we've removed the cached version, let's pull down the most recent version available.

nextflow pull ausarg/pipesnake

Hopefully that resolves your problem.

6. Pipeline completed with errors

Workflows can always run into errors, so resolving them is a great skill to learn. Here are some common issues and how they can be resolved.

ERROR ~ Error executing process > 'AUSARG_PIPESNAKE:PIPESNAKE:[*INSERT_PROGRAM*] (ToyData_Sample3)'

Caused by:
  Process requirement exceeds available memory -- req: 24 GB; avail: 16 GB

The above error message is a result of a disconnect between requested (24 GB) and available (16 GB) memory. This can be easily adjusted by (1) adapting the existing conf/base.config file to suit your machine's resources, or (2) making a new conf/[name].config file with appropriate resources indicated.

7. uncommitted changes Error

Sometimes we get stuck where we can't run the pipeline and we can't pull down a new version. This is generally accompanied by the error message:

Checking ausarg/pipsnake ...
ausarg/pipesnake contains uncommitted changes -- cannot pull from repository

The easiest solution is to follow the instructions for removing and re-pulling pipesnake:

rm -rf ~/.nextflow/assets/ausarg/pipesnake

Now that we've removed the cached version, let's pull down the most recent version available.

nextflow pull ausarg/pipesnake

8. Can I specify my working directory?

If you're running pipesnake on an HPC you may have limited storage on a shared drive. pipesnake generates a lot of temporary files that help to provide a cache if you need to -resume a run. Unfortunately these can take up a lot of space. If you have room on another available drive, you can specify the working directory using the Nextflow command -work-dir. We suggest testing this using the test data to make sure you've identified the correct path:

nextflow run ausarg/pipesnake -profile test,docker --outdir <PATH/TO/WDIR/RUN1> -work-dir <PATH/TO/WDIR>

9. Docker compute architecture issues

Users of Apple computers with M1 and M2 silicon chips have run into issues with platform issues. If you get this error message:

The requested image’s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

You will need to specify the pipesnake image build so it is consistent with your compute architecture. Resolving this is relatively straightforward and follows this solution found on StackOverflow. Start by removing your current pipesnake image.

rm -rf ~/.nextflow/assets/ausarg/pipesnake

You can then tell Docker which platform to use when downloading pipesnake. Open your .bashrc or .zprofile and add the following line:

export DOCKER_DEFAULT_PLATFORM=linux/amd64

Make sure to close your terminal and open another, or source your bashrc/zprofile before running again.

Clone this wiki locally