Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ jobs:
- "gis"
- "googlebatch"
- "hasta"
- "hazel"
- "hki_genie"
- "hypatia"
- "humantechnopole"
Expand Down
55 changes: 55 additions & 0 deletions conf/hazel.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
// NC State University - Hazel HPC Cluster
// nf-core institutional config
// Scheduler: SLURM | Containers: Singularity/Apptainer
//
// To activate: nextflow run <pipeline> -profile hazel
//
// Submit an issue or PR to add this to the nf-core/configs repo:
// https://github.com/nf-core/configs

params {
config_profile_name = 'NC State University - Hazel HPC'
config_profile_description = 'nf-core config for the Hazel HPC cluster at NC State University'
config_profile_contact = 'Seth Weaver (@sweavs111)'
config_profile_url = 'https://brc.ncsu.edu/'

max_memory = 128.GB
max_cpus = 24
max_time = 120.h

// Default GPU allocation. Override at runtime with --hazel_gpu 'gpu:<type>:<num>'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a validation{} scope ignoreParams list to ignore this to avoid ugly warning for users
(Review from phone, so no real example but check other configs)

hazel_gpu = 'gpu:h100:1'
}

// ─── Singularity ────────────────────────────────────────────────────────────
singularity {
enabled = true
autoMounts = true
cacheDir = System.getenv('APPTAINER_CACHEDIR') ?: "/share/${System.getenv('GROUP')}/${System.getenv('USER')}/tmp"
}

// ─── Executor tuning ────────────────────────────────────────────────────────
executor {
queueSize = 16
pollInterval = '1 min'
queueStatInterval = '5 min'
submitRateLimit = '10 sec'
}

// ─── Process defaults ────────────────────────────────────────────────────────
process {
executor = 'slurm'
queue = 'compute'

resourceLimits = [
memory: 128.GB,
cpus: 24,
time: 120.h

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not wrong, but suggested to have in only one place (e.g. reuse the params to support old pipelines) to avoid drift with updates.

]

// GPU jobs use gpu partition
withLabel: 'process_gpu' {
queue = 'gpu'
clusterOptions = { "--gres=${params.hazel_gpu}" }
}
}
40 changes: 40 additions & 0 deletions docs/hazel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# nf-core/configs: NC State Hazel HPC Configuration

nf-core pipelines have been configured for use on the Hazel HPC at NC State University.

To use, run the pipeline with `-profile hazel`.

Example:

`nextflow run nf-core/<pipeline> -profile hazel`

Running this command will download the `hazel` config from the nf-core repository, and will submit nextflow proesses as jobs to the `slurm` job scheduler. Therefore, the `nextflow run` command must be run from the `login node`. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline. These images are cached in `/share/$GROUP/$USER/tmp` so they don't have to be re-downloaded on subsequent runs.

## Loading required environment modules

Before running the pipeline you will need to load Nextflow and Singularity using the environment module system on hazel. Some of these modules are managed by the BRC, and are not in the default module path. See this resource for information about using these modules: https://hurwitzlab.github.io/COS_Compute_Handbook/chapters/14_loading_brc_modules.html

```bash
## Load Nextflow and Singularity environment modules
module purge # Optional, unload any currently loaded modules.
module load nextflow/26.04.3
module load singularity

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is required to make singularity/apptainer available, I suggest adding a beforeScript to load the module to the process as well.

(Slurm environment forwarding works great except when it doesn't.)

module load nf-core # Optional, manages and downloads nf-core pipelines for common use. Requires singularity.
```

## Job submission partitions

By default, using this profile will submit jobs to the `compute` partition. To specify a different partition for submission, add the `--partition <PARTITION NAME>` argument to the `nextflow run` command. Nextflow processes with the `process_gpu` label will be submitted to the `gpu` partition, and by default will run with the `--gres=gpu:h100:1` request. To request a different GPU type, or multiple GPUs, use `--hazel_gpu 'gpu:<gpu-type>:<num-gpus>'` at runtime.


## Overriding Hazel config option

Options passed to your `nextflow run` command will override options contained in the Hazel institutional profile.
Comment thread
sweavs111 marked this conversation as resolved.
Outdated

:::note
You will need an account to use the Hazel HPC in order to run the pipeline. If in doubt contact IT or your PI to get access.
:::

:::note
Nextflow will need to submit the jobs via the job scheduler to the HPC cluster and as such the commands above will have to be executed on one of the login nodes. If in doubt contact IT.
:::
3 changes: 3 additions & 0 deletions nfcore_custom.config
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,9 @@ profiles {
hasta {
includeConfig "${params.custom_config_base}/conf/hasta.config"
}
hazel {
includeConfig "${params.custom_config_base}/conf/hazel.config"
}
hki_genie {
includeConfig "${params.custom_config_base}/conf/hki_genie.config"
}
Expand Down
Loading