Skip to content

AGENTS.md

Latest commit

 

History

History
213 lines (146 loc) · 12 KB

AGENTS.md

File metadata and controls

213 lines (146 loc) · 12 KB
Raw

nf-core/modules: agents

This is the AI agent context file for the nf-core modules repository. All AI agents and coding assistants MUST read and strictly follow the rules contained in this document.

Natural language

All comments and documentation MUST be written in English with British spelling.

Key terms

  • Module: a single process that achieves a single, well defined task (e.g. aligning reads to a genome)
  • Subworkflow: a sequence of chained modules that achieve a specific objective (e.g. FASTQ cleanup and quality check)
  • Component: a module or subworkflow

Repository structure

You are working in the nf-core modules repository. Repository structure:

.
├── modules
│   ├── environment-schema.json // schema for module's environment.yml
│   ├── meta-schema.json        // schema for module's meta.yml
│   └── nf-core                 // directory containing nf-core modules, described below
├── nf-test.config // configuration for nf-test
├── subworkflows
│   ├── nf-core          // directory containing nf-core subworkflows, described below
│   └── yaml-schema.json // schema for subworkflow meta.yaml
└── tests // assets for running tests
    └── config // Nextflow configurations, do not edit unless directly prompted

Certain rarely edited files were omitted in the treemap. You MUST NOT edit them unless the user explicitly asks for it.

modules/nf-core structure

Module directories are under modules/nf-core, either directly or with one nesting level. The following rules apply:

  • If a tool has only one command, it is included directly (e.g. modules/nf-core/iqtree)
  • If a tool has subcommands, they are contained as subdirectories of the tool directory (e.g. modules/nf-core/bedops/convert2bed); if the tool without subcommands has a separate functionality, it is included as a subcommand (e.g. modules/nf-core/kraken2/kraken2)
  • If a tool has multiple levels of subcommands, the subcommands are concatenated (e.g. aws s3 ls would be modules/nf-core/aws/s3ls)

Module directory structure

Required structure:

modules/nf-core/{module}/
├── environment.yml       // list of Conda packages with pinned versions
├── main.nf               // Nextflow code of the module
├── meta.yml              // contains information about inputs, outputs and tools in the module
└── tests
    ├── main.nf.test      // nf-test unit tests for the module
    ├── main.nf.test.snap // snapshots for the tests (see note below)
    └── nextflow.config   // Nextflow configuration used for testing ONLY

You MAY manually change version strings in test snapshots after a version bump. Any other change MUST ONLY be introduced through nf-test or nf-core commands.

Subworkflow directory structure

Each subworkflow has a single top-level directory, without nesting. Required structure:

subworkflows/nf-core/{subworkflow}/
├── main.nf               // Nextflow code of the subworkflow
├── meta.yml              // contains information about inputs, outputs and tools in the subworkflow
└── tests
    ├── main.nf.test      // nf-test unit tests for the subworkflow
    ├── main.nf.test.snap // snapshots for the tests (see note above)
    └── nextflow.config   // Nextflow configuration used for testing ONLY; multiple config files may exist in some cases

Structure of a module

Each module contains several files. Each file has a well-defined structure that you MUST follow.

main.nf

meta.yml

meta.yml MUST follow the specification at https://nf-co.re/docs/specifications/components/modules/documentation

environment.yml

This file lists Conda channels and packages necessary to run the module with Conda and to build Seqera containers.

  • You SHOULD NOT add channels unless strictly necessary.
  • You MUST NOT add "defaults" to channels.
  • Each dependency MUST specify the channel and version, but NOT the build number.
  • If pip dependencies are used, you MUST also pin the version of pip.

tests/main.nf.test

The test file MUST follow the specificaton at https://nf-co.re/docs/specifications/components/modules/testing.

tests/nextflow.config

tests/nextflow.config has no effect on pipeline runtime, it is only applied in unit tests. You MUST NOT create additional configs in modules. To create multiple configurations, use module_args as described at https://nf-co.re/docs/specifications/components/modules/testing#configuration-of-extargs-in-tests.

Structure of a subworkflow

Subworkflows are structured similarly to modules. Subworkflows have no environment.yml, since Conda information is inherited from the included modules. The differences in specific files are described below.

main.nf

  • The file starts with include statements for modules and other subworkflows.
  • The subworkflow is defined in a single named workflow block, with take (input), main (script), and emit (output) sections.
  • The subworkflow MAY take any inputs that are required for the tools and produce any output that may reasonably be useful.
  • The subworkflow SHOULD contain multiple modules (at least 3, or 2 with complex logic) and any Nextflow logic required to connect them.

meta.yml

The following sections differ:

  • a components section MUST be present, containing a list of called modules, instead of tools,
  • input and output sections MUST contain all variables and files, but these sections NEED NOT exactly match the channel structure of the Nextflow file,
  • a topics section MUST NOT be present.

Meta map

The meta map is a Nextflow map passed along with each file that contains sample-specific information.

Modules:

  • nf-core modules SHOULD accept a meta map for at least the main input, unless the purpose of the module makes it clearly pointless.
  • If a meta map is accepted, it MUST also be propagated in at least 1 output.
  • If there are multiple file input channels, the module MAY define additional meta inputs called meta2, meta3, and so on.
  • Modules MAY assume that the meta map contains meta.id (always) and meta.single_end (where relevant).
  • Modules MUST NOT assume the existence of any other keys in the meta map.

Subworkflows:

  • Subworkflows NEED NOT specifically accept meta inputs.
  • Any channel operations in subworkflows MUST preserve the meta where present.
  • The presence of metas in input and output channels MUST be documented through comments (example: // channel: [ val(meta), [ bam ] ]) and meta.yml.

ext options

The following ext options are allowed in nf-core:

  • ext.args: arguments for the underlying tool, passed directly if present (for multi-tool modules, args2 through args99 are allowed as well)
  • ext.prefix: prefix used to name output files
  • ext.prefix2: a second prefix used in case of multiple outputs; you MUST NOT use it for other purposes, e.g. file formats
  • ext.use_gpu: a flag to determine whether a module with optional acceleration should use GPUs

Modules MUST NOT assume the presence of other ext options.

GPU-capable modules

Modules that use GPUs MUST follow the documentation at https://nf-co.re/docs/developing/components/gpu-modules.

nf-core tools

nf-core provides a CLI toolkit for working with the nf-core template. The core command is nf-core.

  • You SHOULD use the tools instead of creating files manually.
  • You SHOULD use nf-core modules --help to discover commands relevant for modules.
  • You SHOULD use nf-core subworkflows --help to discover commands relevant for subworkflows.
  • You MUST write subtool module names in commands with a slash, for example samtools/sort.

nf-test and testing

  • Run tests with nf-test test {modules|subworkflows}/{path}/tests --profile=+{docker|singularity|conda} --stop-on-first-failure
  • If you expect the output to change (e.g. after a tool update), you SHOULD update the snapshot with nf-test test {modules|subworkflows}/{path}/tests --profile=+{docker|singularity|conda} --update-snapshot. You MUST regenerate snapshots on the same CPU architecture as CI.

git and branch policy

  • You MUST create a new branch with a meaningful name for each feature, whether you are working directly in the nf-core repository or on a fork.
  • If you work on multiple features in parallel, you SHOULD use a separate worktree for each task to prevent clobber.

Push routine

  • You SHOULD only push after implementing some meaningful changes and if the code is working. You MUST obtain permission to push from the user.
  • Before pushing, you MUST run nf-core modules/subworkflows lint {name}. You MUST resolve all errors and all possible warnings. You MUST repeat this routine until there are no solvable outstanding issues.
  • You MUST run nf-test test {modules|subworkflows}/{path}/tests --profile=+{docker|singularity|conda}. If the run fails, you MUST resolve the underlying issues. If the test fails due to mismatching snapshots, update them with nf-test test {modules|subworkflows}/{path}/tests --profile=+{docker|singularity|conda} --update-snapshot only if permitted (see "nf-test" above). Otherwise, you MUST fix the issue that caused the unexpected change.
  • If the module you have updated is used in subworkflows or in tests of other modules, update their test snapshots as well.
  • If you know the code will cause issues or you intend to push more changes, you SHOULD add [skip ci] at the end of the commit title. Omit this tag if the changes for final, review-ready commits.

PR procedure

  • All changes MUST be submitted to nf-core master through GitHub pull request.
  • You MUST NOT create "ready for review" pull requests. You MAY create draft pull requests if authorised by the user.
  • A PR SHOULD contain changes in a single component.
  • A PR MAY instead introduce an identical, small change in multiple components. The number of files changed MUST stay low enough for proper human review.
  • The PR MUST use the nf-core PR template, including the checklist. The PR message SHOULD start with a brief explanation of the changes made and the motivation.
  • Each PR requires 1 approving review. A second review is recommended for exceptionally complex PRs. PR review can be requested on Slack by a human.
  • All CI checks MUST pass before the PR can be merged.

Agent self-disclosure

  • If you generated a majority of the code in a commit, you MUST add "Generated by {your name}" at the end of the commit message body.
  • If you open a PR autonomously, you MUST add "Generated by {your name}" at the end of the PR message (above the checklist).
  • If you post a comment in a PR on behalf of the user, you MUST add "Comment added by {your name}" at the end of the comment.

References