Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3,001 changes: 3,001 additions & 0 deletions modules/nf-core/colabfold/.conda-lock/linux_amd64-bd-50a22b7e73876edb_1.txt

Large diffs are not rendered by default.

14 changes: 14 additions & 0 deletions modules/nf-core/colabfold/environment.gpu.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/environment-schema.json
channels:
- conda-forge
- bioconda
dependencies:
- "bioconda::colabfold=1.5.5"
# CUDA 12.0 is the lowest CUDA whose jaxlib build ships ptxas (via cuda-nvcc)
- "conda-forge::cuda-version=12.0"
# colabfold 1.5.5 needs jax <0.4.26; newer dm-haiku releases need a newer jax
- "conda-forge::dm-haiku=0.0.12"
- "conda-forge::flax=0.8.2"
- "conda-forge::jax=0.4.25"
- "conda-forge::jaxlib=0.4.25=*cuda*"
10 changes: 10 additions & 0 deletions modules/nf-core/colabfold/environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/modules/environment-schema.json
channels:
- conda-forge
- bioconda
dependencies:
- "bioconda::colabfold=1.5.5"
# colabfold 1.5.5 needs jax <0.4.26; newer dm-haiku releases need a newer jax
- "conda-forge::dm-haiku=0.0.12"
- "conda-forge::flax=0.8.2"
57 changes: 57 additions & 0 deletions modules/nf-core/colabfold/main.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
process COLABFOLD {
tag "${meta.id}"
label 'process_high'
label 'process_gpu'

conda "${task.accelerator ? "${moduleDir}/environment.gpu.yml" : "${moduleDir}/environment.yml"}"
container "${workflow.containerEngine in ['singularity', 'apptainer'] && !task.ext.singularity_pull_docker_container
? (task.accelerator ? 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/ee/eefb564f01b59ec8f7d85de34df208ba245d830ed9fecf1eb5bf8625cfbd68ef/data' : 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/d2/d2de6276b0c23cc1aaab4af2aa1336fa943b0c45d43ce4ed483911fb99a3db33/data')
: (task.accelerator ? 'community.wave.seqera.io/library/colabfold_cuda-version_dm-haiku_flax_pruned:ccfbc77af1b9acd9' : 'community.wave.seqera.io/library/colabfold_dm-haiku_flax:50a22b7e73876edb')}"

input:
tuple val(meta), path(fasta)
tuple val(meta2), path(weights)

output:
tuple val(meta), path("${prefix}/*_rank_*.pdb") , emit: pdb , optional: true
tuple val(meta), path("${prefix}/*_scores_rank_*.json"), emit: json, optional: true
tuple val(meta), path("${prefix}/*.a3m") , emit: a3m , optional: true
tuple val(meta), path("${prefix}/*.png") , emit: png , optional: true
tuple val(meta), path("${prefix}/log.txt") , emit: log , optional: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several files that colabfold_batch writes are not emitted: config.json, cite.bibtex, *_predicted_aligned_error_v1.json, the *.pickle files (--msa-only, --save-all), the *_repr_*.npy files (--save-single-representations, --save-pair-representations), *.result.zip (--zip) and *_template_domain_names.json (--templates). Please emit them as optional outputs, and update meta.yml and the stub to match.

Suggested change
tuple val(meta), path("${prefix}/*_rank_*.pdb") , emit: pdb , optional: true
tuple val(meta), path("${prefix}/*_scores_rank_*.json"), emit: json, optional: true
tuple val(meta), path("${prefix}/*.a3m") , emit: a3m , optional: true
tuple val(meta), path("${prefix}/*.png") , emit: png , optional: true
tuple val(meta), path("${prefix}/log.txt") , emit: log , optional: true
tuple val(meta), path("${prefix}/*_rank_*.pdb") , emit: pdb , optional: true
tuple val(meta), path("${prefix}/*_scores_rank_*.json") , emit: json , optional: true
tuple val(meta), path("${prefix}/*_predicted_aligned_error_v1.json"), emit: pae , optional: true
tuple val(meta), path("${prefix}/*.a3m") , emit: a3m , optional: true
tuple val(meta), path("${prefix}/*.png") , emit: png , optional: true
tuple val(meta), path("${prefix}/*.pickle") , emit: pickle , optional: true
tuple val(meta), path("${prefix}/*_repr_*.npy") , emit: npy , optional: true
tuple val(meta), path("${prefix}/*.result.zip") , emit: zip , optional: true
tuple val(meta), path("${prefix}/*_template_domain_names.json") , emit: templates, optional: true
tuple val(meta), path("${prefix}/config.json") , emit: config , optional: true
tuple val(meta), path("${prefix}/cite.bibtex") , emit: bibtex , optional: true
tuple val(meta), path("${prefix}/log.txt") , emit: log , optional: true

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a5aaff6, with your suggested outputs (pae, pickle, npy, zip, templates, config, bibtex), plus meta.yml entries and EDAM terms where they exist (ZIP format_3987, NumPy format_4003, pickle format_4002; [] for BibTeX). I checked the file names against colabfold 1.5.5's source: representations are *_single_repr_*.npy / *_pair_repr_*.npy and the pickles are *_all_*.pickle (--save-all) or <query>.pickle (--msa-only), so the globs match.

tuple val("${task.process}"), val('colabfold'), eval("python -c 'import importlib.metadata as m; print(m.version(\"colabfold\"))'"), topic: versions, emit: versions_colabfold
tuple val("${task.process}"), val('cuda'), eval('python -c "import glob, json, sys; f = glob.glob(sys.prefix + \'/conda-meta/cuda-version-*.json\'); print(json.load(open(f[0]))[\'version\'] if f else \'no CUDA available\')"'), topic: versions, emit: versions_cuda

when:
task.ext.when == null || task.ext.when

script:
def args = task.ext.args ?: ''
prefix = task.ext.prefix ?: "${meta.id}"
def data = weights ? "--data ${weights}" : ''
"""
# Under Docker the container runs as the host UID, which has no home
# directory; colabfold_batch needs a writable HOME for its cache.
export HOME="\$PWD"

if [ ! -s ${fasta} ]; then
echo "WARN: ${fasta} is empty, no structure predicted" >&2
else
colabfold_batch \\
${fasta} \\
${prefix} \\
${data} \\
${args}
fi
"""

stub:
prefix = task.ext.prefix ?: "${meta.id}"
"""
mkdir ${prefix}
touch ${prefix}/${prefix}_unrelaxed_rank_001_alphafold2_ptm_model_1_seed_000.pdb
touch ${prefix}/${prefix}_scores_rank_001_alphafold2_ptm_model_1_seed_000.json
touch ${prefix}/${prefix}.a3m
touch ${prefix}/${prefix}_plddt.png
touch ${prefix}/log.txt
"""
}
166 changes: 166 additions & 0 deletions modules/nf-core/colabfold/meta.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
name: colabfold
description: Predicts protein structure with AlphaFold2, taking the multiple sequence alignment from an MMseqs2 server rather than local databases
keywords:
- protein structure
- structure prediction
- AlphaFold2
- MSA
- folding
tools:
- colabfold:
description: |
ColabFold runs AlphaFold2 (and other folding models) while sourcing its
multiple sequence alignment from a remote MMseqs2 server, which removes
the ~2.6 TB of reference databases a local AlphaFold2 install needs.
The public server (api.colabfold.com) is used by default; pass
`--host-url` through `ext.args` to use your own instance. A GPU speeds
inference up but is not required; Amber relaxation is enabled with
`--amber` (and `--use-gpu-relax` on a GPU).
homepage: https://github.com/sokrypton/ColabFold
documentation: https://github.com/sokrypton/ColabFold
doi: 10.1038/s41592-022-01488-1
licence: ["MIT"]
identifier: biotools:colabfold
args_id: "$args"

input:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- fasta:
type: file
description: Protein sequences to fold, one prediction per record (an empty file produces no output)
pattern: "*.{fa,faa,fasta}"
ontologies:
- edam: http://edamontology.org/format_1929 # FASTA
- - meta2:
type: map
description: |
Groovy Map containing weights information
e.g. [ id:'alphafold2_params' ]
- weights:
type: directory
description: Optional ColabFold data directory with the AlphaFold2 parameters in `params/`; pass `[]` to download them

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

colabfold_batch checks for an empty file, params/download_finished.txt, which colabfold's own downloader creates when it finishes. It checks even when --data is given. If the file is missing (e.g. weights extracted by hand, or reused from proteinfold), colabfold downloads the 3.5 GB again into the staged weights folder. That writes through the symlink into the upstream folder, or fails on read-only storage. Please say this in the description, so users who bring their own weights know to add the file.

Suggested change
description: Optional ColabFold data directory with the AlphaFold2 parameters in `params/`; pass `[]` to download them
description: ColabFold data directory with the AlphaFold2 parameters in `params/`. It must contain the empty `params/download_finished.txt` that colabfold's downloader creates (e.g. via `colabfold/downloadmodels`); without it colabfold downloads the weights again

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied as suggested.

ontologies: []
output:
pdb:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- ${prefix}/*_rank_*.pdb:
type: file
description: Predicted structures (rank_001 is the best model; per-residue pLDDT in the B-factor column)
pattern: "*_rank_*.pdb"
ontologies:
- edam: http://edamontology.org/format_1476 # PDB
json:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- ${prefix}/*_scores_rank_*.json:
type: file
description: Per-model scores (per-residue pLDDT, PAE, pTM)
pattern: "*_scores_rank_*.json"
ontologies:
- edam: http://edamontology.org/format_3464 # JSON
a3m:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- ${prefix}/*.a3m:
type: file
description: Multiple sequence alignment used for each query
pattern: "*.a3m"
ontologies: []
png:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- ${prefix}/*.png:
type: file
description: pLDDT, PAE and MSA coverage plots
pattern: "*.png"
ontologies:
- edam: http://edamontology.org/format_3603 # PNG
log:
- - meta:
type: map
description: |
Groovy Map containing sample information
e.g. [ id:'sample1' ]
- ${prefix}/log.txt:
type: file
description: colabfold_batch log
pattern: "log.txt"
ontologies:
- edam: http://edamontology.org/format_2330 # Textual format
versions_colabfold:
- - ${task.process}:
type: string
description: The name of the process
- colabfold:
type: string
description: The name of the tool
- python -c 'import importlib.metadata as m; print(m.version("colabfold"))':
type: eval
description: The expression to obtain the version of the tool
versions_cuda:
- - ${task.process}:
type: string
description: The name of the process
- cuda:
type: string
description: Name of the runtime
- python -c "import glob, json, sys; f = glob.glob(sys.prefix + '/conda-meta/cuda-version-*.json'); print(json.load(open(f[0]))['version'] if f else 'no CUDA available')":
type: eval
description: CUDA version of the task's environment, or `no CUDA available` on the CPU path

topics:
versions:
- - ${task.process}:
type: string
description: The name of the process
- colabfold:
type: string
description: The name of the tool
- python -c 'import importlib.metadata as m; print(m.version("colabfold"))':
type: eval
description: The expression to obtain the version of the tool
- - ${task.process}:
type: string
description: The name of the process
- cuda:
type: string
description: Name of the runtime
- python -c "import glob, json, sys; f = glob.glob(sys.prefix + '/conda-meta/cuda-version-*.json'); print(json.load(open(f[0]))['version'] if f else 'no CUDA available')":
type: eval
description: CUDA version of the task's environment, or `no CUDA available` on the CPU path

authors:
- "@Bboy010"
maintainers:
- "@Bboy010"
containers:
docker:
linux/amd64:
name: community.wave.seqera.io/library/colabfold_dm-haiku_flax:50a22b7e73876edb
build_id: bd-50a22b7e73876edb_1
scan_id: sc-7d6a4f09ec9a8b34_1
singularity:
linux/amd64:
name: oras://community.wave.seqera.io/library/colabfold_dm-haiku_flax:797eb5b15587ba43
build_id: bd-797eb5b15587ba43_1
https: https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/d2/d2de6276b0c23cc1aaab4af2aa1336fa943b0c45d43ce4ed483911fb99a3db33/data
conda:
linux/amd64:
lock_file: modules/nf-core/colabfold/.conda-lock/linux_amd64-bd-50a22b7e73876edb_1.txt
65 changes: 65 additions & 0 deletions modules/nf-core/colabfold/tests/main.gpu.nf.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
nextflow_process {

name "Test Process COLABFOLD (GPU)"
script "../main.nf"
process "COLABFOLD"

tag "gpu"
tag "modules"
tag "modules_nfcore"
tag "colabfold"

// Weights are passed as [] so colabfold_batch downloads them (~3.5 GB);
// they cannot be shipped in nf-core/test-datasets. nextflow.gpu.config
// runs a single-sequence prediction, so no MSA server is queried.
test("proteomics - faa - single sequence - gpu") {

config "./nextflow.gpu.config"

when {
process {
"""
input[0] = [
[ id:'test' ],
file(params.modules_testdata_base_path + 'proteomics/interproscan/l_arginase.faa', checkIfExists: true)
]
input[1] = [ [:], [] ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test passes weights, templates or pdb_hits, so the --data, --custom-template-path and --pdb-hit-file / --local-pdb-path branches never run in CI. Once the weights come from a setup block (see the review body), the --data path is tested. For templates, the cheapest test is a second GPU test with the 20-residue Trp-cage sequence and the 1L2Y template you already tried locally (both small enough for test-datasets), with --templates in module_args.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the weights come from the setup block, and two new GPU tests cover templates and pdb_hits / local_pdb on Trp-cage (details in the general comment). Test data: nf-core/test-datasets#2354.

"""
}
}

then {
assert process.success
assertAll(
{ assert snapshot(sanitizeOutput(process.out, unstableKeys: ['pdb', 'json', 'png', 'log'])).match() },
{ assert path(process.out.pdb[0][1]).readLines().any { it.startsWith('ATOM') } },
{ assert path(process.out.log[0][1]).text.contains('Done') }
)
Comment on lines +33 to +36

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please put the stable checks for the unstable channels inside the snapshot, so a change shows up as a snapshot diff, and add one for json. This also replaces the implicit it with a named closure parameter. The atom count of the unrelaxed model depends only on the sequence, so it stays stable across runs.

Suggested change
assert process.success
assertAll(
{ assert snapshot(sanitizeOutput(process.out, unstableKeys: ['pdb', 'json', 'png', 'log'])).match() },
{ assert path(process.out.pdb[0][1]).readLines().any { it.startsWith('ATOM') } },
{ assert path(process.out.log[0][1]).text.contains('Done') }
)
assert process.success
assertAll(
{ assert snapshot(
sanitizeOutput(process.out, unstableKeys: ['pdb', 'json', 'png', 'log']),
path(process.out.pdb[0][1]).readLines().count { line -> line.startsWith('ATOM') },
path(process.out.json[0][1]).text.contains('plddt'),
path(process.out.log[0][1]).text.contains('Done')
).match() }
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a5aaff6: the ATOM count, the plddt check on the scores JSON and the Done check on the log are now in the snapshot, with a named closure parameter. pae, config and bibtex are added to unstableKeys. Two local GPU runs gave the same snapshot.

}
}

test("proteomics - faa - stub - gpu") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stub test names should end with - stub.

Suggested change
test("proteomics - faa - stub - gpu") {
test("proteomics - faa - gpu - stub") {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to proteomics - faa - gpu - stub in a5aaff6.


config "./nextflow.gpu.config"
options "-stub"

when {
process {
"""
input[0] = [
[ id:'test' ],
file(params.modules_testdata_base_path + 'proteomics/interproscan/l_arginase.faa', checkIfExists: true)
]
input[1] = [ [:], [] ]
"""
}
}

then {
assert process.success
assertAll(
{ assert snapshot(sanitizeOutput(process.out)).match() }
)
}
}
}
Loading
Loading