Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ch_transcripts and ch_pubids for AWS #30

Merged
merged 15 commits into from
Jan 3, 2025
Merged

Conversation

avani-bhojwani
Copy link
Collaborator

@avani-bhojwani avani-bhojwani commented Dec 27, 2024

The pipeline was running locally and on HPC, but failing tests on AWS even when run without fusion.

  • Checking the errors indicated that it was probably be due to how ch_transcripts and ch_pubids were defined. They were using glob patterns. The pipeline now uses a different approach to create these channels. This has been tested by launching the AWS test action manually (for test profile).
  • Upon fixing the issue above, the Transrate module was still not running on AWS, as it required a temporary directory. Creation of this directory is now included in the module.
  • When running the full_test on AWS, it needs to be run without fusion or with less logs. If we don't do this, the fusion log files become too large during the Trinity module and the step is reattempted again and again with higher resources. I've added resource limits for full_test, so that we don't end up requesting >1TB of memory with reattempts.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/denovotranscript branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@nf-core-bot
Copy link
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.1.0.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@avani-bhojwani avani-bhojwani marked this pull request as ready for review January 3, 2025 07:53
@avani-bhojwani avani-bhojwani merged commit 9063826 into dev Jan 3, 2025
6 checks passed
@avani-bhojwani avani-bhojwani deleted the aws-filepath branch January 3, 2025 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants