-
Notifications
You must be signed in to change notification settings - Fork 365
Add North Carolina State institutional Config #1127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+116
−0
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
e17f1af
Add NC State Hazel HPC institutional config
sweavs111 0327977
minor changes
sweavs111 9b0928e
Merge branch 'nf-core:master' into master
sweavs111 4c2e716
Update docs/hazel.md
sweavs111 c7a260e
responding to comments, pushing for testing on HPC
sweavs111 599069d
Revise Nextflow module loading instructions
sweavs111 f0d8f80
final commit, fix override partition option
sweavs111 cfa4a25
final commit, fixing --partition option
sweavs111 758bbe8
Merge branch 'master' of https://github.com/sweavs111/configs
sweavs111 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -121,6 +121,7 @@ jobs: | |
| - "gis" | ||
| - "googlebatch" | ||
| - "hasta" | ||
| - "hazel" | ||
| - "hki_genie" | ||
| - "hypatia" | ||
| - "humantechnopole" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| // NC State University - Hazel HPC Cluster | ||
| // nf-core institutional config | ||
| // Scheduler: SLURM | Containers: Singularity/Apptainer | ||
| // | ||
| // To activate: nextflow run <pipeline> -profile hazel | ||
| // | ||
| // Submit an issue or PR to add this to the nf-core/configs repo: | ||
| // https://github.com/nf-core/configs | ||
|
|
||
| params { | ||
| config_profile_name = 'NC State University - Hazel HPC' | ||
| config_profile_description = 'nf-core config for the Hazel HPC cluster at NC State University' | ||
| config_profile_contact = 'Seth Weaver (@sweavs111)' | ||
| config_profile_url = 'https://brc.ncsu.edu/' | ||
|
|
||
| max_memory = 128.GB | ||
| max_cpus = 24 | ||
| max_time = 120.h | ||
|
|
||
| // Default GPU allocation. Override at runtime with --hazel_gpu 'gpu:<type>:<num>' | ||
| hazel_gpu = 'gpu:h100:1' | ||
|
|
||
| // Default partitions. Override at runtime with --partition or --gpu_partition | ||
| partition = 'compute' | ||
| gpu_partition = 'gpu' | ||
| } | ||
|
|
||
| validation { | ||
| ignoreParams = ['hazel_gpu', 'partition', 'gpu_partition'] | ||
| } | ||
|
|
||
| // ─── Singularity ──────────────────────────────────────────────────────────── | ||
| singularity { | ||
| enabled = true | ||
| autoMounts = true | ||
| cacheDir = System.getenv('APPTAINER_CACHEDIR') ?: "/share/${System.getenv('GROUP')}/${System.getenv('USER')}/tmp" | ||
| } | ||
|
|
||
| // ─── Executor tuning ──────────────────────────────────────────────────────── | ||
| executor { | ||
| queueSize = 16 | ||
| pollInterval = '1 min' | ||
| queueStatInterval = '5 min' | ||
| submitRateLimit = '10 sec' | ||
| } | ||
|
|
||
| // ─── Process defaults ──────────────────────────────────────────────────────── | ||
| process { | ||
| beforeScript = 'module load singularity' | ||
| executor = 'slurm' | ||
| queue = params.partition | ||
|
|
||
| resourceLimits = [ | ||
| memory: params.max_memory, | ||
| cpus: params.max_cpus, | ||
| time: params.max_time | ||
| ] | ||
|
|
||
| // GPU jobs use gpu partition | ||
| withLabel: 'process_gpu' { | ||
| queue = params.gpu_partition | ||
| clusterOptions = { "--gres=${params.hazel_gpu}" } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # nf-core/configs: NC State Hazel HPC Configuration | ||
|
|
||
| nf-core pipelines have been configured for use on the Hazel HPC at NC State University. | ||
|
|
||
| To use, run the pipeline with `-profile hazel`. | ||
|
|
||
| Example: | ||
|
|
||
| `nextflow run nf-core/<pipeline> -profile hazel` | ||
|
|
||
| Running this command will download the `hazel` config from the nf-core repository, and will submit nextflow processes as jobs to the `slurm` job scheduler. Therefore, the `nextflow run` command must be run from the `login node`. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline. These images are cached in `/share/$GROUP/$USER/tmp` so they don't have to be re-downloaded on subsequent runs. | ||
|
|
||
| ## Loading required environment modules | ||
|
|
||
| Before running the pipeline you will need to load Nextflow using the environment module system on hazel. Some of these modules are managed by the BRC, and are not in the default module path. See this resource for information about using these modules: https://hurwitzlab.github.io/COS_Compute_Handbook/chapters/14_loading_brc_modules.html | ||
|
|
||
| ```bash | ||
| ## Load Nextflow environment modules | ||
| module purge # Optional, unload any currently loaded modules. | ||
| module load nextflow/26.04.3 # Required | ||
| module load nf-core # Optional, manages and downloads nf-core pipelines for common use. Requires singularity. | ||
| ``` | ||
|
|
||
| ## Job submission partitions | ||
|
|
||
| By default, using this profile will submit jobs to the `compute` partition. To specify a different partition for submission, use `--partition <PARTITION NAME>` in the `nextflow run` command. | ||
|
|
||
| Nextflow processes with the `process_gpu` label will be submitted to the `gpu` partition by default; override with `--gpu_partition <PARTITION NAME>`. To request a different GPU type or multiple GPUs, use `--hazel_gpu 'gpu:<gpu-type>:<num-gpus>'` at runtime. | ||
|
|
||
|
|
||
| ## Overriding Hazel config options | ||
|
|
||
| Nextflow-level options passed to your `nextflow run` command will override options contained in the Hazel institutional profile: | ||
|
|
||
| ```bash | ||
| nextflow run nf-core/<pipeline> -profile hazel --max_memory 256.GB # raise memory ceiling | ||
| nextflow run nf-core/<pipeline> -profile hazel --hazel_gpu 'gpu:a100:2' # different GPU | ||
| ``` | ||
|
|
||
| For persistent overrides, use a local config file with `-c my_overrides.config`. | ||
|
|
||
| :::note | ||
| You will need an account to use the Hazel HPC in order to run the pipeline. If in doubt contact IT or your PI to get access. | ||
| ::: | ||
|
|
||
| :::note | ||
| Nextflow will need to submit the jobs via the job scheduler to the HPC cluster and as such the commands above will have to be executed on one of the login nodes. If in doubt contact IT. | ||
| ::: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a
validation{}scopeignoreParamslist to ignore this to avoid ugly warning for users(Review from phone, so no real example but check other configs)