Refactor pipeline to nf-core DSL2 modular architecture

.github/workflows/ci.yml

@@ -0,0 +1,64 @@
+name: nf-core CI
+# This workflow runs the pipeline with the minimal test dataset to check that it completes without any errors
+on:
+  push:
+    branches:
+      - main
+      - master
+  pull_request:
+  release:
+    types: [published]
+
+# Cancel previous runs if a new one is triggered
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: Run pipeline with test data
+    # Only run on push if this is the main branch (merged PRs)
+    if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'seandavi/curatedMetagenomicsNextflow') }}"
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        NXF_VER:
+          - "23.04.0"
+          - "latest-everything"
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v4
+
+      - name: Install Nextflow
+        uses: nf-core/setup-nextflow@v2
+        with:
+          version: "${{ matrix.NXF_VER }}"
+
+      - name: Disk space cleanup
+        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+
+      - name: Run pipeline with test data (stub)
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} -profile test,docker -stub-run --outdir ./results
+
+  profile:
+    name: Run profile tests
+    if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'seandavi/curatedMetagenomicsNextflow') }}"
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        profile:
+          - "test"
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v4
+
+      - name: Install Nextflow
+        uses: nf-core/setup-nextflow@v2
+
+      - name: Disk space cleanup
+        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+
+      - name: Run pipeline with test data
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.profile }},docker -stub-run --outdir ./results

.github/workflows/linting.yml

@@ -0,0 +1,85 @@
+name: nf-core linting
+# This workflow is triggered on pushes and PRs to the repository.
+on:
+  push:
+    branches:
+      - main
+      - master
+  pull_request:
+  release:
+    types: [published]
+
+# Cancel previous runs if a new one is triggered
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - uses: pre-commit/action@v3.0.1
+        # FIXME Flip this off once we get to a good state
+        continue-on-error: true
+
+  prettier:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Install NodeJS
+        uses: actions/setup-node@v4
+
+      - name: Install Prettier
+        run: npm install -g prettier
+
+      - name: Run Prettier --check
+        run: prettier --check .
+
+  editorconfig:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: editorconfig-checker/action-editorconfig-checker@main
+
+      - run: editorconfig-checker -exclude README.md $(git ls-files | grep -v test)
+
+  nf-core-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v4
+
+      - name: Install Nextflow
+        uses: nf-core/setup-nextflow@v2
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          architecture: "x64"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install nf-core
+
+      - name: Run nf-core lint
+        run: nf-core lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
+        continue-on-error: true
+
+      - name: Upload linting log file artifact
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: linting-logs
+          path: |
+            lint_results.md

.nf-core.yml

@@ -0,0 +1,16 @@
+repository_type: pipeline
+nf_core_version: "2.14.1"
+org_path: seandavi
+lint:
+  files_exist:
+    - .github/workflows/ci.yml
+    - .github/workflows/linting.yml
+    - CHANGELOG.md
+    - CODE_OF_CONDUCT.md
+    - CITATIONS.md
+  files_unchanged: []
+  nextflow_config:
+    - manifest.name
+    - manifest.version
+    - manifest.description
+    - manifest.author

.prettierignore

@@ -0,0 +1,13 @@
+# Ignore Nextflow and Groovy files (no parser available)
+*.nf
+*.config
+
+# Ignore existing files from before refactoring
+docker/cloudbuild.yaml
+nextflow_schema.json
+unitn_setup.md
+
+# Ignore build artifacts and dependencies
+work/
+.nextflow*
+results/

CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.3.0] - 2025-10-30
+
+### Added
+
+- Complete nf-core refactoring of pipeline structure
+- Modularized all processes following nf-core DSL2 conventions
+- Added `workflows/` directory with main workflow
+- Added `modules/local/` directory with individual process modules
+- Added `conf/` directory with configuration files:
+  - `base.config` for base process configuration
+  - `modules.config` for module-specific settings
+  - `test.config` for test profile
+- Added `.nf-core.yml` configuration file
+- Added GitHub Actions CI/CD workflows for linting and testing
+- Added module metadata files (`meta.yml`) for documentation
+- Updated README with comprehensive nf-core-style documentation
+- Standardized parameter naming (`--input`, `--outdir`)
+- Added `check_max()` function for resource management
+- Added support for multiple container engines (Docker, Singularity, Podman)
+
+### Changed
+
+- Refactored monolithic `main.nf` into modular structure
+- Updated `nextflow.config` to follow nf-core conventions
+- Improved parameter handling with backwards compatibility
+- Enhanced error handling and validation
+- Updated manifest information
+- Improved output organization with `pipeline_info` directory
+
+### Improved
+
+- Better separation of concerns with modular architecture
+- Easier maintenance and updates
+- More portable across different compute environments
+- Better documentation and help messages
+- Improved resource allocation with labels
+- Enhanced container management
+
+## [1.2.0] and earlier
+
+Previous versions before nf-core refactoring. See git history for details.

CITATIONS.md

@@ -0,0 +1,51 @@
+# Citations
+
+## Pipeline Tools
+
+### MetaPhlAn 4
+
+> Blanco-Míguez A, Beghini F, Cumbo F, McIver LJ, Thompson KN, Zolfo M, Manghi P, Dubois L, Huang KD, Thomas AM, Nickols WA, Piccinno G, Piperni E, Punčochář M, Valles-Colomer M, Tett A, Giordano F, Davies R, Wolf J, Berry SE, Spector TD, Franzosa EA, Pasolli E, Asnicar F, Huttenhower C, Segata N. Extending and improving metagenomic taxonomic profiling with uncharacterized species using MetaPhlAn 4. Nat Biotechnol. 2023 Nov;41(11):1633-1644. doi: 10.1038/s41587-023-01688-w. Epub 2023 Sep 25. PMID: 37709786; PMCID: PMC10579592.
+
+### HUMAnN 3 / bioBakery 3
+
+> Beghini F, McIver LJ, Blanco-Míguez A, Dubois L, Asnicar F, Maharjan S, Mailyan A, Manghi P, Scholz M, Thomas AM, Valles-Colomer M, Weingart G, Zhang Y, Zolfo M, Huttenhower C, Franzosa EA, Segata N. Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3. eLife. 2021 May 4;10:e65088. doi: 10.7554/eLife.65088. PMID: 33944776; PMCID: PMC8096432.
+
+### KneadData
+
+> McIver LJ, Abu-Ali G, Franzosa EA, Schwager R, Morgan XC, Waldron L, Segata N, Huttenhower C. bioBakery: a meta'omic analysis environment. Bioinformatics. 2018 Apr 1;34(7):1235-1237. doi: 10.1093/bioinformatics/btx754. PMID: 29194469; PMCID: PMC6030888.
+
+### SRA Toolkit (fasterq-dump)
+
+> Leinonen R, Sugawara H, Shumway M; International Nucleotide Sequence Database Collaboration. The sequence read archive. Nucleic Acids Res. 2011 Jan;39(Database issue):D19-21. doi: 10.1093/nar/gkq1019. Epub 2010 Nov 9. PMID: 21062823; PMCID: PMC3013647.
+
+### FastQC
+
+> Andrews S. FastQC: A Quality Control Tool for High Throughput Sequence Data [Online]. Available online at: http://www.bioinformatics.babraham.ac.uk/projects/fastqc/
+
+### Nextflow
+
+> Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PMID: 28398311.
+
+### Docker
+
+> Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2.
+
+### Singularity
+
+> Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. PMID: 28494014; PMCID: PMC5426675.
+
+## Software packaging/containerisation tools
+
+### Conda
+
+> Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web.
+
+### Docker
+
+> Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2.
+
+## Data
+
+If you use data from curatedMetagenomicData, please cite:
+
+> Pasolli E, Schiffer L, Manghi P, Renson A, Obenchain V, Truong DT, Beghini F, Malik F, Ramos M, Dowd JB, Huttenhower C, Morgan M, Segata N, Waldron L. Accessible, curated metagenomic data through ExperimentHub. Nat Methods. 2017 Nov;14(11):1023-1024. doi: 10.1038/nmeth.4468. Epub 2017 Oct 31. PMID: 29088129; PMCID: PMC5685312.