From 67e3d67831af641c02782a5f5a1c74ddc883ac51 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 14:46:54 -0500 Subject: [PATCH 01/18] Add information to README for prospective users --- README.md | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 63 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index cb75209..bbc4825 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,38 @@ -CAPTURE -A framework and command line interface (CLI) for computational science. +# CAPTURE +> Build reproducible, FAIR computational workflows from the start. -Table of Contents -- [Installation](#installation) +Starting a computational science project means navigating countless decisions about structure, tooling, and reproducibility—and even experienced developers struggle to get it right. + +CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a framework and command line interface (CLI) that standardizes these decisions through strong conventions for project structure, execution, and validation, enabling teams to build scalable, reproducible, and FAIR workflows from the start. + +## Why CAPTURE? + +CAPTURE helps you build computational science projects that are consistent, reproducible, and scalable—without reinventing the wheel each time. + +- **Standardized project structure** + Organize data, code, and results using consistent, predictable conventions. + +- **Reproducible execution** + Run analyses in controlled, versioned environments across local, HPC, and cloud systems. + +- **Built-in validation and verification** + Ensure outputs are correct and reproducible with automated checks. + +- **Seamless HPC integration** + Scale workflows across SLURM clusters without rewriting pipelines. + +- **Integrated version control workflows** + Leverage Git and GitHub best practices for collaboration and traceability. + +- **Convention over configuration** + Reduce decision fatigue by adopting opinionated defaults that promote best practices. + +- **FAIR-ready by design** + Produce outputs that are Findable, Accessible, Interoperable, and Reusable. + +# Table of Contents +- [Quick Start](#quick-start) +- [Upgrade](#upgrade) - [CLI usage](#cli-usage) - [cap env](#env) - [cap help](#help) @@ -24,12 +54,39 @@ Table of Contents - [cap_data_link](#cap_data_link) - [Contributions](#contributions) -# Installation +# Quick Start +Get up and running with CAPTURE in minutes. + +## 1. Install CAPTURE ``` curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash source ~/.bash_profile ``` -# Update to the current version + +## 2. Initialize a new project +``` +cap new my-project +cd my-project +``` +This creates a standardized project structure for data, code, results, and configuration. + +## 3. Run an example workflow +``` +cap run src/example.sh +``` +CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. + +## 4. Verify results +``` +cap verify verifications/example.sh +git diff verfications/example.out +``` +Outputs are checked for consistency and reproducibility. The results reproduced if there is no difference in `verifications/example.out`. + +Congratulations!! You new have a fully structured, reproducible computational project. + +# Upgrade +Update to the current version ``` cap update ``` From 5dba18fa6ee00e80e3ae7db0a35ec36504d8d6b2 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 14:59:57 -0500 Subject: [PATCH 02/18] Improve README layout and fix typos --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index bbc4825..83751d6 100644 --- a/README.md +++ b/README.md @@ -9,25 +9,25 @@ CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Ef CAPTURE helps you build computational science projects that are consistent, reproducible, and scalable—without reinventing the wheel each time. -- **Standardized project structure** +- **Standardized project structure**: Organize data, code, and results using consistent, predictable conventions. -- **Reproducible execution** - Run analyses in controlled, versioned environments across local, HPC, and cloud systems. +- **Reproducible execution**: + Run analyses in controlled, versioned environments across HPC systems. -- **Built-in validation and verification** +- **Built-in validation and verification**: Ensure outputs are correct and reproducible with automated checks. -- **Seamless HPC integration** +- **Seamless HPC integration**: Scale workflows across SLURM clusters without rewriting pipelines. -- **Integrated version control workflows** +- **Integrated version control workflows**: Leverage Git and GitHub best practices for collaboration and traceability. -- **Convention over configuration** +- **Convention over configuration**: Reduce decision fatigue by adopting opinionated defaults that promote best practices. -- **FAIR-ready by design** +- **FAIR-ready by design**: Produce outputs that are Findable, Accessible, Interoperable, and Reusable. # Table of Contents @@ -81,9 +81,9 @@ CAPTURE will execute the workflow using its built-in conventions for job executi cap verify verifications/example.sh git diff verfications/example.out ``` -Outputs are checked for consistency and reproducibility. The results reproduced if there is no difference in `verifications/example.out`. +Outputs are checked for consistency and reproducibility. The example results reproduced if there is no difference in `verifications/example.out`. -Congratulations!! You new have a fully structured, reproducible computational project. +Congratulations!! You now have a fully structured, reproducible computational project. # Upgrade Update to the current version From 0d2dee99117134d89110c55825d7b37849346f44 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 15:03:06 -0500 Subject: [PATCH 03/18] Replace SLURM with HPC --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 83751d6..1a81f6b 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ CAPTURE helps you build computational science projects that are consistent, repr Ensure outputs are correct and reproducible with automated checks. - **Seamless HPC integration**: - Scale workflows across SLURM clusters without rewriting pipelines. + Scale workflows across HPC clusters without rewriting pipelines. - **Integrated version control workflows**: Leverage Git and GitHub best practices for collaboration and traceability. From 420428095d8bd2230099cb25020edd9da726dec0 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 15:26:31 -0500 Subject: [PATCH 04/18] Improve README --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 1a81f6b..1a996c3 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,8 @@ CAPTURE helps you build computational science projects that are consistent, repr # Quick Start Get up and running with CAPTURE in minutes. +Complete the following steps in an HPC terminal session. + ## 1. Install CAPTURE ``` curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash @@ -302,16 +304,16 @@ Options: ``` Example: ``` -$ cap new lasseignelab PKD_Research +$ cap new -o lasseignelab pkd-research -Create an empty repository for 'PKD_Research' on GitHub by using the +Create an empty repository for 'pkd-research' on GitHub by using the following link and settings: https://github.com/organizations/lasseignelab/repositories/new * No template * Owner: lasseignelab - * Repository name: PKD_Research + * Repository name: pkd-research * Private * No README file * No .gitignore @@ -320,7 +322,7 @@ following link and settings: Where you able to create a repository (y/N)? y -Cloning into 'PKD_Research'... +Cloning into 'pkd-research'... done. ... From 872cc0fa7d79ec1c8869ab37d84d3159534e2adf Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 15:42:19 -0500 Subject: [PATCH 05/18] Add a blank line to the end of README code snippets --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 1a996c3..e7f7674 100644 --- a/README.md +++ b/README.md @@ -63,18 +63,21 @@ Complete the following steps in an HPC terminal session. ``` curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash source ~/.bash_profile + ``` ## 2. Initialize a new project ``` cap new my-project cd my-project + ``` This creates a standardized project structure for data, code, results, and configuration. ## 3. Run an example workflow ``` cap run src/example.sh + ``` CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. @@ -82,6 +85,7 @@ CAPTURE will execute the workflow using its built-in conventions for job executi ``` cap verify verifications/example.sh git diff verfications/example.out + ``` Outputs are checked for consistency and reproducibility. The example results reproduced if there is no difference in `verifications/example.out`. From 3d09ac74b5458f87cdef5fcb95d229073f08b305 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 16:32:27 -0500 Subject: [PATCH 06/18] Separate README and DOCUMENTATION to make the README more compelling --- DOCUMENTATION.md | 791 +++++++++++++++++++++++++++++++++++++++++++++++ README.md | 722 +----------------------------------------- 2 files changed, 798 insertions(+), 715 deletions(-) create mode 100644 DOCUMENTATION.md diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md new file mode 100644 index 0000000..0360062 --- /dev/null +++ b/DOCUMENTATION.md @@ -0,0 +1,791 @@ +# CAPTURE +> Build reproducible, FAIR computational workflows from the start. + +Beginning a computational science project can feel daunting—there are countless decisions about structure, tooling, and reproducibility. Even experienced developers struggle to build projects that remain consistent, scalable, and FAIR over time. + +High-throughput biomedical technologies (e.g., sequencing, imaging, electronic health records) have driven an enormous increase in biological data volume, creating significant challenges for researchers. + +A lack of relevant training and experience can lengthen analysis timelines, impede research progress, and lead to inconsistencies. These issues worsen the current "reproducibility crisis" in biological science, hindering clinical adoption of research discoveries. As such, researchers need skills in data science, effective software development processes, and resource scaling across HPC clusters and cloud platforms. + +CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a software development framework and command line interface (CLI) that provides opinionated conventions and tooling, helping teams build reproducible, high-quality computational workflows from the start. CAPTURE includes conventions for project structure (e.g., directory structure), job execution, verified results, version control (e.g., Git and GitHub), and standardized environments. These conventions improve computational workflows, promote best practices, facilitate internal code review, address the reproducibility crisis, and help projects meet Findable, Accessible, Interoperable, and Reusable (FAIR) principles. + +# Table of Contents +- [Quick Start](#quick-start) +- [Upgrade](#upgrade) +- [CLI usage](#cli-usage) + - [cap env](#env) + - [cap help](#help) + - [cap md5](#md5) + - [cap new](#new) + - [cap run](#run) + - [Runtime environment](#runtime-environment) + - [cap update](#update) + - [cap verify](#verify) + - [cap version](#version) +- [Job helper functions](#job-helper-functions) + - [cap_array_value](#cap_array_value) + - [cap_data_download](#cap_data_download) + - [cap_container](#cap_container) +- [Verification helper functions](#verification-helper-functions) + - [cap_verify_append](#cap_verify_append) + - [cap_verify_md5](#cap_verify_md5) +- [Environment helper functions](#environment-helper-functions) + - [cap_data_link](#cap_data_link) +- [Contributions](#contributions) + +# Quick Start +Get up and running with CAPTURE in minutes. + +Complete the following steps in an HPC terminal session. + +## 1. Install CAPTURE +``` +curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash +source ~/.bash_profile + +``` + +## 2. Initialize a new project +``` +cap new my-project +cd my-project + +``` +This creates a standardized project structure for data, code, results, and configuration. + +## 3. Run an example workflow +``` +cap run src/example.sh +head data/* + +``` +CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. + +## 4. Verify results +``` +cap verify verifications/example.sh +git diff verfications/example.out + +``` +Outputs are checked for consistency and reproducibility. The example results reproduced if there is no difference in `verifications/example.out`. + +Congratulations!! You now have a fully structured, reproducible computational project. + +# Upgrade +Update to the current version +``` +cap update +``` +# CLI usage +The `cap` CLI provides commands to help with reproducible research. +``` +cap params... +``` +## env +Displays CAPTURE environment variables. This command must be executed from +the project root directory. + +Usage: +``` +cap env + +Options: + +-e,--environment + Specifies the environment to show variables for. +``` +Example: +``` +$ cap env + +CAP_ENV_PATH=/data/user/acrumley/3xtg-repurposing/bin/env +CAP_CONTAINER_PATH=/data/user/acrumley/3xtg-repurposing/bin/container +CAP_DATA_PATH=/data/user/acrumley/3xtg-repurposing/data +CAP_ENVIRONMENT=default +CAP_LOGS_PATH=/data/user/acrumley/3xtg-repurposing/logs +CAP_PROJECT_NAME=3xtg-repurposing +CAP_PROJECT_PATH=/data/user/acrumley/3xtg-repurposing +CAP_RANDOM_SEED=16600 +CAP_RESULTS_PATH=/data/user/acrumley/3xtg-repurposing/results +CAP_VERIFICATIONS_PATH=/data/user/acrumley/3xtg-repurposing/verifications +``` + +## help +Shows help for the cap command line tool. + +Usage: +``` +cap help [COMMAND] +``` +Example: +``` +$ cap help + + Usage: cap COMMAND ... + + Commands: + The following subcommands are available. + + COMMAND + env Displays CAPTURE environment variables. + help Shows help for the cap command line tool. + md5 Calculates a combined MD5 checksum for one or more files. + new Creates a new reproducible research project. + run Runs a CAPTURE framework job. + update Updates the CAPTURE framework to the latest version. + version Displays the currently installed version of CAPTURE. + +$ cap help md5 + + Calculates a combined MD5 checksum for one or more files. + + The "md5" command produces a combined MD5 checksum for all the files + specified. It will show a list of all files included to ensure that the + result is as expected. + + Usage: + cap md5 FILE... + + FILE... can be one or more file and/or directory specifications. + + Example: + $ cap md5 * + + Files included: + 43bd364a97a38fb1da7c57e6381886c1 capture/LICENSE + b794df25f796ac80680c0e4d27308bce capture/commands/md5.sh + 0d9281c3586c420130bcb5d25c8a151a capture/lab + 5e79c988140af1b7bd5735b0bf96306b capture/README.md + 783a44ffae97afbce3f1649c5ff517a5 capture/install.sh + + Combined MD5 checksum: + a225199964b84bdeef33bafe3df7c10b +``` + +## md5 +The `md5` command produces an MD5 checksum for each file specified and a +combined MD5 checksum for all the files. The purpose of this command is to +determine whether files downloaded or created are complete and accurate. If +the MD5 checksums from two sets of files match then the files are all the same. + +Usage: +``` +cap md5 [options] FILE... + +FILE... One or more file and/or directory names or patterns. For directories, + all files in the directory and its subdirectories will be included. + +Options: + +--append + Append to the output file if it already exists. + +-n,--dry-run + Lists the files that will have md5sums calculated in order to + verify the expected files are included. This is helpful when + the files are large and take a long time to process. + +--ignore=PATTERN + Exclude files matching the file PATTERN based on the full relative + path. If the option is specified multiple times, all files matching + any of the patterns will be EXCLUDED (logical OR). The selector will + generally have wildcards. Ensure patterns are quoted ("*pattern*") to + prevent unintended shell expansion. + +-o,--output=FILE + Specify an output file name to write the results to. See examples for + the output format. + +--output-files-only + Output only the file names with their md5sum. This facilitates + programmatic verification of files. + +--normalize + Normalizes the output file paths so that files in different root + directories can be easily compared. + +--select=PATTERN + Include only files matching the file PATTERN based on the full relative + path. If the option is specified multiple times, all files matching + any of the patterns will be INCLUDED (logical OR). The selector will + generally have wildcards. Ensure patterns are quoted ("*pattern*") to + prevent unintended shell expansion. + +-s,--slurm=[batch|run] + Runs the md5 command as a Slurm job. If the value is run then + srun is used and the output stays connected to the current + terminal session. If the value is batch then sbatch is used and + the output is written to cap-md5-.out unless the -o or --output + option is specified. +``` +Examples: + +Calculate md5 sums for all files in a directory and its subdirectories. +``` +cap md5 files/* + +Files included: +b3ac2b8b9998bf504ef708ec837a4cce files/one.bin +8d62064673ecb2a440b8802a2f752e8a files/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5ae files/outs/three.bin +009c79f013fe8d4d97c95bf5ceea68ed files/two.bin + +Combined MD5 checksum: +1060bcc0958e5cc774f84ccd24a3b010 +``` + +Calculate md5 sums for files in the subdirectory named "outs". +``` +cap md5 --select "*/outs/*" files/* + +Files included: +8d62064673ecb2a440b8802a2f752e8a files/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5ae files/outs/three.bin + +Combined MD5 checksum: +feaaf18494b99f6570ab6e4730f9e4af +``` + +Calculate md5 sums for files not in the subdirectory named "outs". +``` +cap md5 --ignore "*/outs/*" files/* + +Files included: +b3ac2b8b9998bf504ef708ec837a4cce files/one.bin +009c79f013fe8d4d97c95bf5ceea68ed files/two.bin + +Combined MD5 checksum: +c6f882353ed4c63582276bdd49974a86 +``` + +## new +The `cap new` command will create a new research project based on the +project-template submodule in the capture repository. The project +repository will be created with the origin remote pointed to a Github +repository owner specified by the Github account and project name parameters. + +Usage: +``` +cap new [options] PROJECT_NAME + +PROJECT_NAME Name of the project which will be used for the directory name. + It should also match the git host repo name if one is used. + +Options: + +--git-host= + Git host for the repository used for creating git remotes. The + default is "github.com". + +-o,--owner= + Git host owner the project repo will be created under. This may + be a personal or organization account. + +--skip-git + Skip making the project a git repository in order to allow + the use of other source control software. + +``` +Example: +``` +$ cap new -o lasseignelab pkd-research + +Create an empty repository for 'pkd-research' on GitHub by using the +following link and settings: + + https://github.com/organizations/lasseignelab/repositories/new + + * No template + * Owner: lasseignelab + * Repository name: pkd-research + * Private + * No README file + * No .gitignore + * No license + +Where you able to create a repository (y/N)? y + + +Cloning into 'pkd-research'... +done. + +... + +Happy researching!!! +``` +## run +The `cap run` command runs a CAPTURE framework job within the context of a +reproducible research project. It will configure the environment based +on configuration defined by the current user. By default, the job runs in +the current terminal session. This command must be executed from the project +root directory. + +Usage: +``` +cap run [options] FILE + +FILE File name of the job to run. + +Options: + +-e,--environment + Specifies the environment to run jobs in. Environments allow + different setups for a pipeline. For instance, a pipeline may + use internal copies of data during development but download that + data when the pipeline is ran in a different environment. +-n,--dry-run + Displays the contents of the job to run along with the context + it will run in. +-s,--slurm=[batch|run] + Runs the script as a Slurm job. If the value is run then + srun is used and the output stays connected to the current + terminal session. If the value is batch then sbatch is used and + the output is written to the log file in the logs directory. +``` +Example: +``` +$ cap run src/01_download.sh + +CAPTURE environment: default + +View job output with the following command: +cat logs/01_down_20241118_090854_tcrumley* + +Submitted batch job 29818073 +``` +### Runtime environment + +The runtime environment is configured with the following variables available +to Slurm scripts. +- **CAP_CONTAINER_PATH**: Path to where container files such as Docker will be +maintained. Defaults to `/bin/container`. +- **CAP_DATA_PATH**: Path to where data files will be written. Defaults to +`/data`. +- **CAP_ENVIRONMENT**: The name of the current execution environment. Defaults +to the value "default". A shell script in `config/environments` with a name +matching the environment name will be executed during the CAPTURE configuration +process, e.g. `config/environments/default.sh`. This variable will generally +be set in the `~/.caprc` file. It is possible to set it as a shell environment +variable somewhere like `~/.bash_profile`. Another option is to provide it +before a command, e.g. `CAP_ENVIRONMENT=mylab cap run foo.sh`. Finally, some +commands provide an option for environment such as +`cap run --environment=mylab foo.sh`. +- **CAP_ENV_PATH**: Path to where conda and other runtime environment files +will be maintained. Defaults to `/bin/env`. +- **CAP_LOGS_PATH**: Path to where log files will be written. Defaults to +`/logs`. +- **CAP_PROJECT_NAME**: The name of the project given with the `cap new` +command. +- **CAP_PROJECT_PATH**: Path to the root directory of the project. +- **CAP_RANDOM_SEED**: A randomly generated seed to facilitate reproducible +random number generation. +- **CAP_RESULTS_PATH**: Path to where analysis results will be written. +Defaults to `/results`. +- **CAP_VERIFICATIONS_PATH**: Path to where verification files and the +result files they produce are written. Defaults to +`/verifications`. + +Environment variables can be configured with the following configuration files. +``` +/ +|-- etc/ +` |-- caprc + +~/ +`-- .caprc + +/ +|-- .caprc +|-- config/ +| |-- pipeline.sh +| `-- environment/ +| |-- default.sh +` `-- .sh +``` +Configuration files are loaded in the following order: +- **\/config/pipeline.sh**: Configuration to bootstrap the +runtime environment. This file is configured by the `cap new` command with the +`CAP_PROJECT_NAME` variable set to the name given as a parameter. +- **defaults**: The defaults described in the environment variable section +are set at this point. +- **/etc/caprc**: Configuration set by an organization. +- **~/.caprc**: Configuration set for a specific user. This is a good place +to `source` in lab specific configuration. +- **\/.labrc**: Configuration specific to a project. +- **\/config/environments/.sh**: Configuration specific +to a project and the environment it is being executed in. The `default.sh` +configuration should only contain reproducible configuration that will work in +any Slurm environment. Other lab specific environment files can contain non- +reproducible configuration but the job must also work in the default environment +for reproducibility. An example of environment specific configuration would be +creating symlinks in the data directory for sharing large datasets internal to +a lab while also downloading the data when the symlink does not exist. See +[cap_data_link](#cap_data_link). + +## update +The `cap update` command will upgrade the CAPTURE framework to the latest +version. + +Usage: +``` +cap update +``` +Example: +``` +$ cap update + + +Switched to branch 'main' +Already up-to-date. + +CAPTURE updated to version v0.0.1. +``` + +## verify +The `verify` command runs CAPTURE verifications which are shell scripts that +determine whether outputs are reproducible. The output of verification scripts +will be written to the verifications folder with the same name as the script +with a ".out" extension. These files should be committed to source control so +that reviewers can compare their results. This command must be executed from +the project root directory. + +See also [verification helper functions](#verification-helper-functions). + +Environment variables (useful for custom verifcations): + +CAP_VERIFICATION_DRY_RUN: Boolean value ("true", "false") indicating whether +the current verification is a dry run. + +CAP_VERIFICATION_OUTPUT_FILE: File name to append verification output. + +Usage: +``` +cap verify [options] FILE + +FILE One file name. + +Options: + +-n,--dry-run + Lists the files that will have verifications performed in order to + verify the expected files are included. This is helpful when + the files are large and take a long time to process. +-s,--slurm=[batch|run] + Runs the verify command as a Slurm job with sbatch or srun. +``` + +Example: + +Perform verifications for a step in the pipeline which will produce an +output file named `verifications/01_download.out`. +``` +cap verify verifications/01_download.sh +``` + +## version +The `cap version` command will display the currently installed version +of CAPTURE. + +Usage: +``` +cap version +``` +Example: +``` +$ cap version + +v0.0.3 + +``` + +# Job helper functions +## cap_array_value +Retrieves a value from an array file based on a zero based index. +``` +cap_array_value FILE [INDEX] +``` +- `FILE` The file containing an array value on each line. +- `INDEX` The optional zero based index for the value of the array. + +If a value is not provided for `INDEX` then the SLURM_ARRAY_TASK_ID +environment variable will be used as the default. + +Example that retrieves array values based on the Slurm environment +variable default index. +``` +sample=$(cap_array_value "$CAP_DATA_PATH/sample_list.array") +``` + +Example with a `for` loop: +``` +for index in {1..10}; do + sample=$(cap_array_value "$CAP_DATA_PATH/sample_list.array" index) + # Do something with each sample value. +done +``` + +## cap_data_download +Downloads data into the data directory. +``` +cap_data_download [options] URL +``` +- `URL` The URL of the file to download. + +Options: +- `--source-file-name` The name of the file being downloaded. When the source +file URL does not end in a proper file name, this option allows a name to be +provided. Final downloaded file and/or directory names may be different if +the `--unzip` option is used. +- `--md5sum` The md5sum to check against the file being downloaded. +- `--unzip` Unzips and/or unarchives downloaded files. +- `--subdirectory` Specifies a subdirectory within the data directory where the +downloaded file will be stored. If the subdirectory does not exist, it will be created. + +The file will be downloaded to the file name specified by the URL or the +`--source-file-name` option. If the `--unzip` option is provided then it will +be unarchived into the data directory and possibly have a different final name. +The data directory is specified by `CAP_DATA_PATH` which defaults to +`CAP_PROJECT_PATH/data`. If the `--subdirectory` option is provided, the +downloaded file will be saved in `CAP_PROJECT_PATH/data/subdirectory`. + +If the file or directory already exists in the `data` directory (or subdirectory +if `--subdirectory` is provided) then it will not be downloaded again. This is +also true when the file or directory has been symlinked into the `data` directory +by [cap_data_link](#cap_data_link). + +The following example will download and unarchive a directory into +`CAP_DATA_PATH/refdata-gex-GRCm39-2024-A`. +``` +cap_data_download \ + --unzip \ + --md5sum="37c51137ccaeabd4d151f80dc86ce0b3" \ + "https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz" +``` + +The following example will download and unarchive a directory into +`CAP_DATA_PATH/reference/refdata-gex-GRCm39-2024-A`. +``` +cap_data_download \ + --unzip \ + --subdirectory "reference" \ + --md5sum="37c51137ccaeabd4d151f80dc86ce0b3" \ + "https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz" +``` + +## cap_container +Downloads the proper docker or singularity container. +``` +cap_container [options] REFERENCE +``` +- `REFERENCE` The Docker image reference found on DockerHub. The format of the reference +is /:[tag]. + +Options +- `-c singularity` If specified, cap_container will use `singularity pull` +instead of `docker pull`. If `CAP_CONTAINER_TYPE` is specified in a `caprc` file +then the -c option is not necessary. `CAP_CONTAINER_TYPE` is the preferred +method. + +`cap_container` first checks whether the Docker image or Singularity .sif file +already exists in `CAP_CONTAINER_PATH`. If the image is not found, it is downloaded +from DockerHub. By default, `cap_container` uses Docker, but specifying the +`-c singularity` option or `CAP_CONTAINER_TYPE=singularity` in a `caprc` +directs it to generate a Singularity .sif file in the `CAP_CONTAINER_PATH` +directory instead. + +The following example checks for the corresponding .sif file in `CAP_CONTAINER_PATH`. +If the file is not found, it downloads and converts the Docker image into the +Singularity .sif file - ollama_0.5.8.sif. +``` +cap_container \ + -c singularity \ + "ollama/ollama:0.5.8" +``` + +# Verification helper functions +Functions to facilitate verifying that pipeline results are reproducible. +Verification scripts are stored in the `verifications` directory in the +project root directory and should be committed to the code repository. + +The output file will be given the same name as the verification file with a +`.out` extension and will be stored in the same directory. The output file +should also be committed to the code repository. When reproducing results, use +the `git diff` command to confirm that results of a verification match the +original results. + +## cap_verify_append +The `cap_verify_append` function appends text to the verification's `.out` +file. The purpose of this command is facilitate custom verifications and to +add comments between groupings of verification output. + +``` +cap_verify_append TEXT +``` +- TEXT Text that will be appended directly to the end of the `.out` file. + +### Examples for a verification named `verifications/verify_example.sh`: + +1. Verify files with comments. + +``` +cap_verify_append "##### Mouse data #####" +cap_verify_md5 "data/mouse/*" +cap_verify_append "##### Human data #####" +cap_verify_md5 "data/human/*" +``` +Results in `verifications/verify_example.out`: +``` +##### Mouse data ##### +b3ac2b8b9998bf504ef708ec837a4cce data/mouse/one.bin +8d62064673ecb2a440b8802a2f752e8a data/mouse/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5ae data/mouse/outs/three.bin +009c79f013fe8d4d97c95bf5ceea68ed data/mouse/two.bin +##### Human data ##### +b3ac2b8b9998bf504ef708ec837a4cc1 data/human/one.bin +8d62064673ecb2a440b8802a2f752e82 data/human/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5a5 data/human/outs/three.bin +009c79f013fe8d4d97c95bf5ceea68e8 data/human/two.bin +``` + +2. Custom verification from a Python script. + +The environment variable CAP_VERIFICATION_DRY_RUN can be used to add dry run +functionality to custom verifcation scripts, which will be equal to "true" on a +dry run. + +``` +cap_verify_append "$(python3 $CAP_VERIFICATIONS_PATH/verify_example.py)" +``` +Results in `verifications/verify_example.out`: +``` +Cell Count: 1000 +Gene Count: 5000 +``` + +## cap_verify_md5 +The `cap_verify_md5` function produces an MD5 checksum for each file specified, +storing the results in an output file to be checked into the repository for +verifying future reproducibility. The purpose of this command is to determine +whether files downloaded or created are complete and accurate when reproduced. +If the MD5 checksums from two sets of files match then the files are all the +same. + +``` +cap_verify_md5 [options] FILE... +``` +- FILE... One or more file and/or directory names or patterns. For directories, + all files in the directory and its subdirectories will be included. + +Options +- `--ignore=PATTERN` Exclude files matching the file PATTERN based on the full +relative path. If the option is specified multiple times, all files matching +any of the patterns will be EXCLUDED (logical OR). The selector will generally +have wildcards. Ensure patterns are quoted ("*pattern*") to prevent unintended +shell expansion. + +- `--select=PATTERN` Include only files matching the file PATTERN based on the +full relative path. If the option is specified multiple times, all files +matching any of the patterns will be INCLUDED (logical OR). The selector will +generally have wildcards. Ensure patterns are quoted ("*pattern*") to prevent +unintended shell expansion. + +### Examples for a verification named `verifications/verify_example.sh`: + +1. Verify all files in a directory and its subdirectories. +``` +cap_verify_md5 "data/*" +``` +Results in `verifications/verify_example.out`: +``` +b3ac2b8b9998bf504ef708ec837a4cce data/one.bin +8d62064673ecb2a440b8802a2f752e8a data/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5ae data/outs/three.bin +009c79f013fe8d4d97c95bf5ceea68ed data/two.bin +``` +2. Verify all files in the subdirectory named "outs". +``` +cap_verify_md5 --select "*/outs/*" "data/*" +``` +Results in `verifications/verify_example.out`: +``` +8d62064673ecb2a440b8802a2f752e8a data/outs/four.bin +74a08ee2de381ec8e19da52ad36bb5ae data/outs/three.bin +``` +3. Verify all files not in the subdirectory named "outs". +``` +cap_verify_md5 --ignore "*/outs/*" "data/*" +``` +Results in `verifications/verify_example.out`: +``` +b3ac2b8b9998bf504ef708ec837a4cce data/one.bin +009c79f013fe8d4d97c95bf5ceea68ed data/two.bin +``` + +# Environment helper functions +Functions to facilitate setting up environments for CAPTURE to operate in. +Environments help create reproducible pipelines by allowing authors to +work in their unique development setup, which may only work for them, and +reviewers to run pipelines in a default environment that should work anywhere. +Environment files are stored in the `config/environments` directory. + +## cap_data_link +Creates a symbolic link in the data directory. A common use is to prevent +duplicate storage of large datasets in the author's compute environment. By +linking to a shared copy, multiple authors won't create multiple copies. This +function is often used in conjunction with +[cap_data_download](#cap_data_download), where cap_data_link prevents +cap_data_download from downloading a new version of previously downloaded data +while ensuring the data will be downloaded in other environments such as the +default environment. +``` +cap_data_link | +``` +- `|` The full path to a file or directory. + +The symbolic link will have the same name as the specified file or directory +and will be created in the directory specified by `CAP_DATA_PATH` which +defaults to `CAP_PROJECT_PATH/data`. + +The following example will create a symbolic link at `$CAP_DATA_PATH/mouse` +and should be included in an environment file in `config/environments`, e.g +`config/environments/my_lab.sh`. The `$MY_LAB` environment variable should +be created in a `.caprc` file (See [Runtime environment](#runtime-environment)). +``` +cap_data_link "$MY_LAB/genome/mouse" +``` +To use the `my_lab` environment when running a job, use the `cap run` command +with the -e/--environment option like in the following example. +``` +cap run -e my_lab src/01_download.sh +``` + +# Contributions +## Tests +All pull requests must include BATS tests covering the changes. + +The testing framework is installed by the following command. +``` +tests/install +``` +The entire test suite is executed by the following command. +``` +tests/run +``` +The tests can be filtered with the --filter option. This saves time by +allowing subsets of the test suite to be ran while coding. The following +examples of using --filter are based on this hypothetical BATS test. +``` +@test "cap md5: All files in a folder" { + ... +} +``` +### Examples of using --filter +How to run just the `cap md5` tests: +``` +tests/run --filter "cap md5" +``` + +How to run just the single hypothetical test: +``` +tests/run --filter "cap md5: All files in a folder" +``` diff --git a/README.md b/README.md index e7f7674..b39db3f 100644 --- a/README.md +++ b/README.md @@ -30,31 +30,7 @@ CAPTURE helps you build computational science projects that are consistent, repr - **FAIR-ready by design**: Produce outputs that are Findable, Accessible, Interoperable, and Reusable. -# Table of Contents -- [Quick Start](#quick-start) -- [Upgrade](#upgrade) -- [CLI usage](#cli-usage) - - [cap env](#env) - - [cap help](#help) - - [cap md5](#md5) - - [cap new](#new) - - [cap run](#run) - - [Runtime environment](#runtime-environment) - - [cap update](#update) - - [cap verify](#verify) - - [cap version](#version) -- [Job helper functions](#job-helper-functions) - - [cap_array_value](#cap_array_value) - - [cap_data_download](#cap_data_download) - - [cap_container](#cap_container) -- [Verification helper functions](#verification-helper-functions) - - [cap_verify_append](#cap_verify_append) - - [cap_verify_md5](#cap_verify_md5) -- [Environment helper functions](#environment-helper-functions) - - [cap_data_link](#cap_data_link) -- [Contributions](#contributions) - -# Quick Start +## Quick Start Get up and running with CAPTURE in minutes. Complete the following steps in an HPC terminal session. @@ -77,6 +53,7 @@ This creates a standardized project structure for data, code, results, and confi ## 3. Run an example workflow ``` cap run src/example.sh +head data/* ``` CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. @@ -91,696 +68,11 @@ Outputs are checked for consistency and reproducibility. The example results rep Congratulations!! You now have a fully structured, reproducible computational project. -# Upgrade -Update to the current version -``` -cap update -``` -# CLI usage -The `cap` CLI provides commands to help with reproducible research. -``` -cap params... -``` -## env -Displays CAPTURE environment variables. This command must be executed from -the project root directory. - -Usage: -``` -cap env - -Options: - --e,--environment - Specifies the environment to show variables for. -``` -Example: -``` -$ cap env - -CAP_ENV_PATH=/data/user/acrumley/3xtg-repurposing/bin/env -CAP_CONTAINER_PATH=/data/user/acrumley/3xtg-repurposing/bin/container -CAP_DATA_PATH=/data/user/acrumley/3xtg-repurposing/data -CAP_ENVIRONMENT=default -CAP_LOGS_PATH=/data/user/acrumley/3xtg-repurposing/logs -CAP_PROJECT_NAME=3xtg-repurposing -CAP_PROJECT_PATH=/data/user/acrumley/3xtg-repurposing -CAP_RANDOM_SEED=16600 -CAP_RESULTS_PATH=/data/user/acrumley/3xtg-repurposing/results -CAP_VERIFICATIONS_PATH=/data/user/acrumley/3xtg-repurposing/verifications -``` - -## help -Shows help for the cap command line tool. - -Usage: -``` -cap help [COMMAND] -``` -Example: -``` -$ cap help - - Usage: cap COMMAND ... - - Commands: - The following subcommands are available. - - COMMAND - env Displays CAPTURE environment variables. - help Shows help for the cap command line tool. - md5 Calculates a combined MD5 checksum for one or more files. - new Creates a new reproducible research project. - run Runs a CAPTURE framework job. - update Updates the CAPTURE framework to the latest version. - version Displays the currently installed version of CAPTURE. - -$ cap help md5 - - Calculates a combined MD5 checksum for one or more files. - - The "md5" command produces a combined MD5 checksum for all the files - specified. It will show a list of all files included to ensure that the - result is as expected. - - Usage: - cap md5 FILE... - - FILE... can be one or more file and/or directory specifications. - - Example: - $ cap md5 * - - Files included: - 43bd364a97a38fb1da7c57e6381886c1 capture/LICENSE - b794df25f796ac80680c0e4d27308bce capture/commands/md5.sh - 0d9281c3586c420130bcb5d25c8a151a capture/lab - 5e79c988140af1b7bd5735b0bf96306b capture/README.md - 783a44ffae97afbce3f1649c5ff517a5 capture/install.sh - - Combined MD5 checksum: - a225199964b84bdeef33bafe3df7c10b -``` - -## md5 -The `md5` command produces an MD5 checksum for each file specified and a -combined MD5 checksum for all the files. The purpose of this command is to -determine whether files downloaded or created are complete and accurate. If -the MD5 checksums from two sets of files match then the files are all the same. - -Usage: -``` -cap md5 [options] FILE... - -FILE... One or more file and/or directory names or patterns. For directories, - all files in the directory and its subdirectories will be included. - -Options: - ---append - Append to the output file if it already exists. - --n,--dry-run - Lists the files that will have md5sums calculated in order to - verify the expected files are included. This is helpful when - the files are large and take a long time to process. - ---ignore=PATTERN - Exclude files matching the file PATTERN based on the full relative - path. If the option is specified multiple times, all files matching - any of the patterns will be EXCLUDED (logical OR). The selector will - generally have wildcards. Ensure patterns are quoted ("*pattern*") to - prevent unintended shell expansion. - --o,--output=FILE - Specify an output file name to write the results to. See examples for - the output format. - ---output-files-only - Output only the file names with their md5sum. This facilitates - programmatic verification of files. - ---normalize - Normalizes the output file paths so that files in different root - directories can be easily compared. - ---select=PATTERN - Include only files matching the file PATTERN based on the full relative - path. If the option is specified multiple times, all files matching - any of the patterns will be INCLUDED (logical OR). The selector will - generally have wildcards. Ensure patterns are quoted ("*pattern*") to - prevent unintended shell expansion. - --s,--slurm=[batch|run] - Runs the md5 command as a Slurm job. If the value is run then - srun is used and the output stays connected to the current - terminal session. If the value is batch then sbatch is used and - the output is written to cap-md5-.out unless the -o or --output - option is specified. -``` -Examples: - -Calculate md5 sums for all files in a directory and its subdirectories. -``` -cap md5 files/* - -Files included: -b3ac2b8b9998bf504ef708ec837a4cce files/one.bin -8d62064673ecb2a440b8802a2f752e8a files/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5ae files/outs/three.bin -009c79f013fe8d4d97c95bf5ceea68ed files/two.bin - -Combined MD5 checksum: -1060bcc0958e5cc774f84ccd24a3b010 -``` - -Calculate md5 sums for files in the subdirectory named "outs". -``` -cap md5 --select "*/outs/*" files/* - -Files included: -8d62064673ecb2a440b8802a2f752e8a files/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5ae files/outs/three.bin - -Combined MD5 checksum: -feaaf18494b99f6570ab6e4730f9e4af -``` - -Calculate md5 sums for files not in the subdirectory named "outs". -``` -cap md5 --ignore "*/outs/*" files/* - -Files included: -b3ac2b8b9998bf504ef708ec837a4cce files/one.bin -009c79f013fe8d4d97c95bf5ceea68ed files/two.bin - -Combined MD5 checksum: -c6f882353ed4c63582276bdd49974a86 -``` - -## new -The `cap new` command will create a new research project based on the -project-template submodule in the capture repository. The project -repository will be created with the origin remote pointed to a Github -repository owner specified by the Github account and project name parameters. - -Usage: -``` -cap new [options] PROJECT_NAME - -PROJECT_NAME Name of the project which will be used for the directory name. - It should also match the git host repo name if one is used. - -Options: - ---git-host= - Git host for the repository used for creating git remotes. The - default is "github.com". - --o,--owner= - Git host owner the project repo will be created under. This may - be a personal or organization account. - ---skip-git - Skip making the project a git repository in order to allow - the use of other source control software. - -``` -Example: -``` -$ cap new -o lasseignelab pkd-research - -Create an empty repository for 'pkd-research' on GitHub by using the -following link and settings: - - https://github.com/organizations/lasseignelab/repositories/new - - * No template - * Owner: lasseignelab - * Repository name: pkd-research - * Private - * No README file - * No .gitignore - * No license - -Where you able to create a repository (y/N)? y - - -Cloning into 'pkd-research'... -done. - -... - -Happy researching!!! -``` -## run -The `cap run` command runs a CAPTURE framework job within the context of a -reproducible research project. It will configure the environment based -on configuration defined by the current user. By default, the job runs in -the current terminal session. This command must be executed from the project -root directory. - -Usage: -``` -cap run [options] FILE - -FILE File name of the job to run. - -Options: - --e,--environment - Specifies the environment to run jobs in. Environments allow - different setups for a pipeline. For instance, a pipeline may - use internal copies of data during development but download that - data when the pipeline is ran in a different environment. --n,--dry-run - Displays the contents of the job to run along with the context - it will run in. --s,--slurm=[batch|run] - Runs the script as a Slurm job. If the value is run then - srun is used and the output stays connected to the current - terminal session. If the value is batch then sbatch is used and - the output is written to the log file in the logs directory. -``` -Example: -``` -$ cap run src/01_download.sh - -CAPTURE environment: default - -View job output with the following command: -cat logs/01_down_20241118_090854_tcrumley* - -Submitted batch job 29818073 -``` -### Runtime environment - -The runtime environment is configured with the following variables available -to Slurm scripts. -- **CAP_CONTAINER_PATH**: Path to where container files such as Docker will be -maintained. Defaults to `/bin/container`. -- **CAP_DATA_PATH**: Path to where data files will be written. Defaults to -`/data`. -- **CAP_ENVIRONMENT**: The name of the current execution environment. Defaults -to the value "default". A shell script in `config/environments` with a name -matching the environment name will be executed during the CAPTURE configuration -process, e.g. `config/environments/default.sh`. This variable will generally -be set in the `~/.caprc` file. It is possible to set it as a shell environment -variable somewhere like `~/.bash_profile`. Another option is to provide it -before a command, e.g. `CAP_ENVIRONMENT=mylab cap run foo.sh`. Finally, some -commands provide an option for environment such as -`cap run --environment=mylab foo.sh`. -- **CAP_ENV_PATH**: Path to where conda and other runtime environment files -will be maintained. Defaults to `/bin/env`. -- **CAP_LOGS_PATH**: Path to where log files will be written. Defaults to -`/logs`. -- **CAP_PROJECT_NAME**: The name of the project given with the `cap new` -command. -- **CAP_PROJECT_PATH**: Path to the root directory of the project. -- **CAP_RANDOM_SEED**: A randomly generated seed to facilitate reproducible -random number generation. -- **CAP_RESULTS_PATH**: Path to where analysis results will be written. -Defaults to `/results`. -- **CAP_VERIFICATIONS_PATH**: Path to where verification files and the -result files they produce are written. Defaults to -`/verifications`. - -Environment variables can be configured with the following configuration files. -``` -/ -|-- etc/ -` |-- caprc - -~/ -`-- .caprc - -/ -|-- .caprc -|-- config/ -| |-- pipeline.sh -| `-- environment/ -| |-- default.sh -` `-- .sh -``` -Configuration files are loaded in the following order: -- **\/config/pipeline.sh**: Configuration to bootstrap the -runtime environment. This file is configured by the `cap new` command with the -`CAP_PROJECT_NAME` variable set to the name given as a parameter. -- **defaults**: The defaults described in the environment variable section -are set at this point. -- **/etc/caprc**: Configuration set by an organization. -- **~/.caprc**: Configuration set for a specific user. This is a good place -to `source` in lab specific configuration. -- **\/.labrc**: Configuration specific to a project. -- **\/config/environments/.sh**: Configuration specific -to a project and the environment it is being executed in. The `default.sh` -configuration should only contain reproducible configuration that will work in -any Slurm environment. Other lab specific environment files can contain non- -reproducible configuration but the job must also work in the default environment -for reproducibility. An example of environment specific configuration would be -creating symlinks in the data directory for sharing large datasets internal to -a lab while also downloading the data when the symlink does not exist. See -[cap_data_link](#cap_data_link). - -## update -The `cap update` command will upgrade the CAPTURE framework to the latest -version. - -Usage: -``` -cap update -``` -Example: -``` -$ cap update - - -Switched to branch 'main' -Already up-to-date. - -CAPTURE updated to version v0.0.1. -``` - -## verify -The `verify` command runs CAPTURE verifications which are shell scripts that -determine whether outputs are reproducible. The output of verification scripts -will be written to the verifications folder with the same name as the script -with a ".out" extension. These files should be committed to source control so -that reviewers can compare their results. This command must be executed from -the project root directory. - -See also [verification helper functions](#verification-helper-functions). - -Environment variables (useful for custom verifcations): - -CAP_VERIFICATION_DRY_RUN: Boolean value ("true", "false") indicating whether -the current verification is a dry run. - -CAP_VERIFICATION_OUTPUT_FILE: File name to append verification output. - -Usage: -``` -cap verify [options] FILE - -FILE One file name. - -Options: - --n,--dry-run - Lists the files that will have verifications performed in order to - verify the expected files are included. This is helpful when - the files are large and take a long time to process. --s,--slurm=[batch|run] - Runs the verify command as a Slurm job with sbatch or srun. -``` - -Example: - -Perform verifications for a step in the pipeline which will produce an -output file named `verifications/01_download.out`. -``` -cap verify verifications/01_download.sh -``` - -## version -The `cap version` command will display the currently installed version -of CAPTURE. - -Usage: -``` -cap version -``` -Example: -``` -$ cap version - -v0.0.3 - -``` - -# Job helper functions -## cap_array_value -Retrieves a value from an array file based on a zero based index. -``` -cap_array_value FILE [INDEX] -``` -- `FILE` The file containing an array value on each line. -- `INDEX` The optional zero based index for the value of the array. - -If a value is not provided for `INDEX` then the SLURM_ARRAY_TASK_ID -environment variable will be used as the default. - -Example that retrieves array values based on the Slurm environment -variable default index. -``` -sample=$(cap_array_value "$CAP_DATA_PATH/sample_list.array") -``` - -Example with a `for` loop: -``` -for index in {1..10}; do - sample=$(cap_array_value "$CAP_DATA_PATH/sample_list.array" index) - # Do something with each sample value. -done -``` - -## cap_data_download -Downloads data into the data directory. -``` -cap_data_download [options] URL -``` -- `URL` The URL of the file to download. - -Options: -- `--source-file-name` The name of the file being downloaded. When the source -file URL does not end in a proper file name, this option allows a name to be -provided. Final downloaded file and/or directory names may be different if -the `--unzip` option is used. -- `--md5sum` The md5sum to check against the file being downloaded. -- `--unzip` Unzips and/or unarchives downloaded files. -- `--subdirectory` Specifies a subdirectory within the data directory where the -downloaded file will be stored. If the subdirectory does not exist, it will be created. - -The file will be downloaded to the file name specified by the URL or the -`--source-file-name` option. If the `--unzip` option is provided then it will -be unarchived into the data directory and possibly have a different final name. -The data directory is specified by `CAP_DATA_PATH` which defaults to -`CAP_PROJECT_PATH/data`. If the `--subdirectory` option is provided, the -downloaded file will be saved in `CAP_PROJECT_PATH/data/subdirectory`. - -If the file or directory already exists in the `data` directory (or subdirectory -if `--subdirectory` is provided) then it will not be downloaded again. This is -also true when the file or directory has been symlinked into the `data` directory -by [cap_data_link](#cap_data_link). - -The following example will download and unarchive a directory into -`CAP_DATA_PATH/refdata-gex-GRCm39-2024-A`. -``` -cap_data_download \ - --unzip \ - --md5sum="37c51137ccaeabd4d151f80dc86ce0b3" \ - "https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz" -``` - -The following example will download and unarchive a directory into -`CAP_DATA_PATH/reference/refdata-gex-GRCm39-2024-A`. -``` -cap_data_download \ - --unzip \ - --subdirectory "reference" \ - --md5sum="37c51137ccaeabd4d151f80dc86ce0b3" \ - "https://cf.10xgenomics.com/supp/cell-exp/refdata-gex-GRCm39-2024-A.tar.gz" -``` - -## cap_container -Downloads the proper docker or singularity container. -``` -cap_container [options] REFERENCE -``` -- `REFERENCE` The Docker image reference found on DockerHub. The format of the reference -is /:[tag]. - -Options -- `-c singularity` If specified, cap_container will use `singularity pull` -instead of `docker pull`. If `CAP_CONTAINER_TYPE` is specified in a `caprc` file -then the -c option is not necessary. `CAP_CONTAINER_TYPE` is the preferred -method. - -`cap_container` first checks whether the Docker image or Singularity .sif file -already exists in `CAP_CONTAINER_PATH`. If the image is not found, it is downloaded -from DockerHub. By default, `cap_container` uses Docker, but specifying the -`-c singularity` option or `CAP_CONTAINER_TYPE=singularity` in a `caprc` -directs it to generate a Singularity .sif file in the `CAP_CONTAINER_PATH` -directory instead. - -The following example checks for the corresponding .sif file in `CAP_CONTAINER_PATH`. -If the file is not found, it downloads and converts the Docker image into the -Singularity .sif file - ollama_0.5.8.sif. -``` -cap_container \ - -c singularity \ - "ollama/ollama:0.5.8" -``` - -# Verification helper functions -Functions to facilitate verifying that pipeline results are reproducible. -Verification scripts are stored in the `verifications` directory in the -project root directory and should be committed to the code repository. - -The output file will be given the same name as the verification file with a -`.out` extension and will be stored in the same directory. The output file -should also be committed to the code repository. When reproducing results, use -the `git diff` command to confirm that results of a verification match the -original results. - -## cap_verify_append -The `cap_verify_append` function appends text to the verification's `.out` -file. The purpose of this command is facilitate custom verifications and to -add comments between groupings of verification output. - -``` -cap_verify_append TEXT -``` -- TEXT Text that will be appended directly to the end of the `.out` file. - -### Examples for a verification named `verifications/verify_example.sh`: - -1. Verify files with comments. - -``` -cap_verify_append "##### Mouse data #####" -cap_verify_md5 "data/mouse/*" -cap_verify_append "##### Human data #####" -cap_verify_md5 "data/human/*" -``` -Results in `verifications/verify_example.out`: -``` -##### Mouse data ##### -b3ac2b8b9998bf504ef708ec837a4cce data/mouse/one.bin -8d62064673ecb2a440b8802a2f752e8a data/mouse/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5ae data/mouse/outs/three.bin -009c79f013fe8d4d97c95bf5ceea68ed data/mouse/two.bin -##### Human data ##### -b3ac2b8b9998bf504ef708ec837a4cc1 data/human/one.bin -8d62064673ecb2a440b8802a2f752e82 data/human/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5a5 data/human/outs/three.bin -009c79f013fe8d4d97c95bf5ceea68e8 data/human/two.bin -``` - -2. Custom verification from a Python script. - -The environment variable CAP_VERIFICATION_DRY_RUN can be used to add dry run -functionality to custom verifcation scripts, which will be equal to "true" on a -dry run. - -``` -cap_verify_append "$(python3 $CAP_VERIFICATIONS_PATH/verify_example.py)" -``` -Results in `verifications/verify_example.out`: -``` -Cell Count: 1000 -Gene Count: 5000 -``` - -## cap_verify_md5 -The `cap_verify_md5` function produces an MD5 checksum for each file specified, -storing the results in an output file to be checked into the repository for -verifying future reproducibility. The purpose of this command is to determine -whether files downloaded or created are complete and accurate when reproduced. -If the MD5 checksums from two sets of files match then the files are all the -same. - -``` -cap_verify_md5 [options] FILE... -``` -- FILE... One or more file and/or directory names or patterns. For directories, - all files in the directory and its subdirectories will be included. - -Options -- `--ignore=PATTERN` Exclude files matching the file PATTERN based on the full -relative path. If the option is specified multiple times, all files matching -any of the patterns will be EXCLUDED (logical OR). The selector will generally -have wildcards. Ensure patterns are quoted ("*pattern*") to prevent unintended -shell expansion. - -- `--select=PATTERN` Include only files matching the file PATTERN based on the -full relative path. If the option is specified multiple times, all files -matching any of the patterns will be INCLUDED (logical OR). The selector will -generally have wildcards. Ensure patterns are quoted ("*pattern*") to prevent -unintended shell expansion. - -### Examples for a verification named `verifications/verify_example.sh`: - -1. Verify all files in a directory and its subdirectories. -``` -cap_verify_md5 "data/*" -``` -Results in `verifications/verify_example.out`: -``` -b3ac2b8b9998bf504ef708ec837a4cce data/one.bin -8d62064673ecb2a440b8802a2f752e8a data/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5ae data/outs/three.bin -009c79f013fe8d4d97c95bf5ceea68ed data/two.bin -``` -2. Verify all files in the subdirectory named "outs". -``` -cap_verify_md5 --select "*/outs/*" "data/*" -``` -Results in `verifications/verify_example.out`: -``` -8d62064673ecb2a440b8802a2f752e8a data/outs/four.bin -74a08ee2de381ec8e19da52ad36bb5ae data/outs/three.bin -``` -3. Verify all files not in the subdirectory named "outs". -``` -cap_verify_md5 --ignore "*/outs/*" "data/*" -``` -Results in `verifications/verify_example.out`: -``` -b3ac2b8b9998bf504ef708ec837a4cce data/one.bin -009c79f013fe8d4d97c95bf5ceea68ed data/two.bin -``` - -# Environment helper functions -Functions to facilitate setting up environments for CAPTURE to operate in. -Environments help create reproducible pipelines by allowing authors to -work in their unique development setup, which may only work for them, and -reviewers to run pipelines in a default environment that should work anywhere. -Environment files are stored in the `config/environments` directory. - -## cap_data_link -Creates a symbolic link in the data directory. A common use is to prevent -duplicate storage of large datasets in the author's compute environment. By -linking to a shared copy, multiple authors won't create multiple copies. This -function is often used in conjunction with -[cap_data_download](#cap_data_download), where cap_data_link prevents -cap_data_download from downloading a new version of previously downloaded data -while ensuring the data will be downloaded in other environments such as the -default environment. -``` -cap_data_link | -``` -- `|` The full path to a file or directory. - -The symbolic link will have the same name as the specified file or directory -and will be created in the directory specified by `CAP_DATA_PATH` which -defaults to `CAP_PROJECT_PATH/data`. - -The following example will create a symbolic link at `$CAP_DATA_PATH/mouse` -and should be included in an environment file in `config/environments`, e.g -`config/environments/my_lab.sh`. The `$MY_LAB` environment variable should -be created in a `.caprc` file (See [Runtime environment](#runtime-environment)). -``` -cap_data_link "$MY_LAB/genome/mouse" -``` -To use the `my_lab` environment when running a job, use the `cap run` command -with the -e/--environment option like in the following example. -``` -cap run -e my_lab src/01_download.sh -``` +## Documentation +Full CAPTURE documentaion can be found [here](DOCUMENTATION.md). -# Contributions -## Tests +## Contributions +### Tests All pull requests must include BATS tests covering the changes. The testing framework is installed by the following command. @@ -799,7 +91,7 @@ examples of using --filter are based on this hypothetical BATS test. ... } ``` -### Examples of using --filter +#### Examples of using --filter How to run just the `cap md5` tests: ``` tests/run --filter "cap md5" From 4ab682c77627bb8323779e2271bc987b753edf06 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 16:38:03 -0500 Subject: [PATCH 07/18] Improve quick start instructions --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index b39db3f..9e1d994 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,7 @@ CAPTURE helps you build computational science projects that are consistent, repr Produce outputs that are Findable, Accessible, Interoperable, and Reusable. ## Quick Start -Get up and running with CAPTURE in minutes. - -Complete the following steps in an HPC terminal session. +Get up and running with CAPTURE in minutes by completing the following steps in an HPC terminal session. ## 1. Install CAPTURE ``` From 3a7a1c4f5671b96ecf1b60197b795e579f650c12 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 16:45:45 -0500 Subject: [PATCH 08/18] Point documentation quick start to README quick start --- DOCUMENTATION.md | 37 +------------------------------------ 1 file changed, 1 insertion(+), 36 deletions(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 0360062..62fb079 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -34,42 +34,7 @@ CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Ef - [Contributions](#contributions) # Quick Start -Get up and running with CAPTURE in minutes. - -Complete the following steps in an HPC terminal session. - -## 1. Install CAPTURE -``` -curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash -source ~/.bash_profile - -``` - -## 2. Initialize a new project -``` -cap new my-project -cd my-project - -``` -This creates a standardized project structure for data, code, results, and configuration. - -## 3. Run an example workflow -``` -cap run src/example.sh -head data/* - -``` -CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. - -## 4. Verify results -``` -cap verify verifications/example.sh -git diff verfications/example.out - -``` -Outputs are checked for consistency and reproducibility. The example results reproduced if there is no difference in `verifications/example.out`. - -Congratulations!! You now have a fully structured, reproducible computational project. +Get up and running with CAPTURE in minutes by completing the README quick start guide found [here](README.md#quick-start). # Upgrade Update to the current version From ebdee5bcdf5a92c495677deef302e1cb377c9ca4 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 16:49:43 -0500 Subject: [PATCH 09/18] Fix README quick start subheaders --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9e1d994..eb0e8d4 100644 --- a/README.md +++ b/README.md @@ -33,14 +33,14 @@ CAPTURE helps you build computational science projects that are consistent, repr ## Quick Start Get up and running with CAPTURE in minutes by completing the following steps in an HPC terminal session. -## 1. Install CAPTURE +### 1. Install CAPTURE ``` curl -sSL https://raw.githubusercontent.com/lasseignelab/capture/refs/heads/main/install.sh | bash source ~/.bash_profile ``` -## 2. Initialize a new project +### 2. Initialize a new project ``` cap new my-project cd my-project @@ -48,7 +48,7 @@ cd my-project ``` This creates a standardized project structure for data, code, results, and configuration. -## 3. Run an example workflow +### 3. Run an example workflow ``` cap run src/example.sh head data/* @@ -56,7 +56,7 @@ head data/* ``` CAPTURE will execute the workflow using its built-in conventions for job execution, logging, and output organization. -## 4. Verify results +### 4. Verify results ``` cap verify verifications/example.sh git diff verfications/example.out From e5950999251fccb27a7eee09cee0e74fb904f33e Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:03:26 -0500 Subject: [PATCH 10/18] Improve contributing section of the README --- README.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index eb0e8d4..f37c695 100644 --- a/README.md +++ b/README.md @@ -69,17 +69,51 @@ Congratulations!! You now have a fully structured, reproducible computational pr ## Documentation Full CAPTURE documentaion can be found [here](DOCUMENTATION.md). -## Contributions +## Contributing + +We welcome contributions from both new and experienced developers. + +Whether you're fixing a bug, improving documentation, or proposing a new feature, CAPTURE is designed to support reproducible, high-quality computational workflows—and contributions should follow the same principles. + +### Getting started + +1. Fork the repository and create a new branch +2. Make your changes with clear, focused commits +3. Add or update tests and documentation as needed +4. Submit a pull request with a clear description of your changes + +### Development guidelines + +- Follow CAPTURE conventions for project structure and naming +- Write reproducible, testable code +- Prefer simple, transparent solutions over complex abstractions +- Ensure scripts and workflows run consistently across environments (local, HPC, cloud) + +### Reporting issues + +If you encounter a bug or have a feature request, please open an issue and include: + +- A clear description of the problem +- Steps to reproduce (if applicable) +- Relevant logs or error messages +- Your environment (OS, HPC, container, etc.) + +### Code of conduct + +Be respectful and constructive. We aim to foster an inclusive and collaborative community. + ### Tests All pull requests must include BATS tests covering the changes. The testing framework is installed by the following command. ``` tests/install + ``` The entire test suite is executed by the following command. ``` tests/run + ``` The tests can be filtered with the --filter option. This saves time by allowing subsets of the test suite to be ran while coding. The following @@ -88,14 +122,27 @@ examples of using --filter are based on this hypothetical BATS test. @test "cap md5: All files in a folder" { ... } + ``` #### Examples of using --filter How to run just the `cap md5` tests: ``` tests/run --filter "cap md5" + ``` How to run just the single hypothetical test: ``` tests/run --filter "cap md5: All files in a folder" + ``` + +--- + +## Contributors + +Thanks to everyone who has contributed to CAPTURE: + +- Tonie Crumley +- TC Howton +- Lasseigne Lab contributors From 34c0c112f5bc1e50164972a502769909cb83eebd Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:07:55 -0500 Subject: [PATCH 11/18] Encourage contributions to CAPTURE core tenets --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f37c695..c90ea60 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ Full CAPTURE documentaion can be found [here](DOCUMENTATION.md). We welcome contributions from both new and experienced developers. -Whether you're fixing a bug, improving documentation, or proposing a new feature, CAPTURE is designed to support reproducible, high-quality computational workflows—and contributions should follow the same principles. +Whether you're fixing a bug, improving documentation, or proposing a new feature, CAPTURE is designed to support reproducible, high-quality computational workflows—and contributions should follow the same principles. Contributions that improve reproducibility, validation, and portability are especially valuable. ### Getting started From 3328005946cfd851135fe985bd7960f22c428199 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:16:56 -0500 Subject: [PATCH 12/18] Improve README and DOCUMENTATION copy --- DOCUMENTATION.md | 4 ++-- README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 62fb079..c4826ce 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -3,9 +3,9 @@ Beginning a computational science project can feel daunting—there are countless decisions about structure, tooling, and reproducibility. Even experienced developers struggle to build projects that remain consistent, scalable, and FAIR over time. -High-throughput biomedical technologies (e.g., sequencing, imaging, electronic health records) have driven an enormous increase in biological data volume, creating significant challenges for researchers. +High-throughput biomedical technologies (e.g., sequencing, imaging, electronic health records) have driven an enormous increase in biological data volume, creating significant challenges for researchers. As such, researchers need skills in data science, effective software development processes, and resource scaling across HPC clusters and cloud platforms. -A lack of relevant training and experience can lengthen analysis timelines, impede research progress, and lead to inconsistencies. These issues worsen the current "reproducibility crisis" in biological science, hindering clinical adoption of research discoveries. As such, researchers need skills in data science, effective software development processes, and resource scaling across HPC clusters and cloud platforms. +A lack of relevant training and experience can lengthen analysis timelines, impede research progress, and lead to inconsistencies. These issues worsen the current "reproducibility crisis" in biological science, hindering clinical adoption of research discoveries. CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a software development framework and command line interface (CLI) that provides opinionated conventions and tooling, helping teams build reproducible, high-quality computational workflows from the start. CAPTURE includes conventions for project structure (e.g., directory structure), job execution, verified results, version control (e.g., Git and GitHub), and standardized environments. These conventions improve computational workflows, promote best practices, facilitate internal code review, address the reproducibility crisis, and help projects meet Findable, Accessible, Interoperable, and Reusable (FAIR) principles. diff --git a/README.md b/README.md index c90ea60..e4962ea 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ Starting a computational science project means navigating countless decisions about structure, tooling, and reproducibility—and even experienced developers struggle to get it right. -CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a framework and command line interface (CLI) that standardizes these decisions through strong conventions for project structure, execution, and validation, enabling teams to build scalable, reproducible, and FAIR workflows from the start. +CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a framework and command line interface (CLI) that standardizes these decisions through strong conventions for project structure, execution, and validation, enabling teams to build scalable, reproducible, and FAIR workflows from the start to finish. ## Why CAPTURE? From 6857065ac54159eff58473aadcb0b4f608d7dc2e Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:20:40 -0500 Subject: [PATCH 13/18] Improve README and DOCUMENTATION copy --- DOCUMENTATION.md | 4 ++-- README.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index c4826ce..ce801ae 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -1,5 +1,5 @@ # CAPTURE -> Build reproducible, FAIR computational workflows from the start. +> Build reproducible, FAIR computational workflows from start to finish. Beginning a computational science project can feel daunting—there are countless decisions about structure, tooling, and reproducibility. Even experienced developers struggle to build projects that remain consistent, scalable, and FAIR over time. @@ -7,7 +7,7 @@ High-throughput biomedical technologies (e.g., sequencing, imaging, electronic h A lack of relevant training and experience can lengthen analysis timelines, impede research progress, and lead to inconsistencies. These issues worsen the current "reproducibility crisis" in biological science, hindering clinical adoption of research discoveries. -CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a software development framework and command line interface (CLI) that provides opinionated conventions and tooling, helping teams build reproducible, high-quality computational workflows from the start. CAPTURE includes conventions for project structure (e.g., directory structure), job execution, verified results, version control (e.g., Git and GitHub), and standardized environments. These conventions improve computational workflows, promote best practices, facilitate internal code review, address the reproducibility crisis, and help projects meet Findable, Accessible, Interoperable, and Reusable (FAIR) principles. +CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a software development framework and command line interface (CLI) that provides opinionated conventions and tooling, helping teams build reproducible, high-quality computational workflows from start to finish. CAPTURE includes conventions for project structure (e.g., directory structure), job execution, verified results, version control (e.g., Git and GitHub), and standardized environments. These conventions improve computational workflows, promote best practices, facilitate internal code review, address the reproducibility crisis, and help projects meet Findable, Accessible, Interoperable, and Reusable (FAIR) principles. # Table of Contents - [Quick Start](#quick-start) diff --git a/README.md b/README.md index e4962ea..bce5fe2 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ # CAPTURE -> Build reproducible, FAIR computational workflows from the start. +> Build reproducible, FAIR computational workflows from start to finish. Starting a computational science project means navigating countless decisions about structure, tooling, and reproducibility—and even experienced developers struggle to get it right. -CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a framework and command line interface (CLI) that standardizes these decisions through strong conventions for project structure, execution, and validation, enabling teams to build scalable, reproducible, and FAIR workflows from the start to finish. +CAPTURE (Custom Analysis Pipelines Tailored for Universal Reproducibility and Efficiency) is a framework and command line interface (CLI) that standardizes these decisions through strong conventions for project structure, execution, and validation, enabling teams to build scalable, reproducible, and FAIR workflows from start to finish. ## Why CAPTURE? From 784a5ee64f674d65e6ca4377dd1c74ef7f2c245e Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:29:35 -0500 Subject: [PATCH 14/18] Improve README copy --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index bce5fe2..822c328 100644 --- a/README.md +++ b/README.md @@ -62,12 +62,12 @@ cap verify verifications/example.sh git diff verfications/example.out ``` -Outputs are checked for consistency and reproducibility. The example results reproduced if there is no difference in `verifications/example.out`. +Outputs are checked for consistency and reproducibility. If there is no difference in `verifications/example.out`, the example results were fully reproduced. Congratulations!! You now have a fully structured, reproducible computational project. ## Documentation -Full CAPTURE documentaion can be found [here](DOCUMENTATION.md). +Comprehensive CAPTURE documentation can be found [here](DOCUMENTATION.md). ## Contributing From 7566d2b66e97d29bd11248c8f1aa2f2f64223b34 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Fri, 17 Apr 2026 17:34:07 -0500 Subject: [PATCH 15/18] Remove a repeative word from the README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 822c328..8796dc4 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ git diff verfications/example.out ``` Outputs are checked for consistency and reproducibility. If there is no difference in `verifications/example.out`, the example results were fully reproduced. -Congratulations!! You now have a fully structured, reproducible computational project. +Congratulations!! You now have a well-structured, reproducible computational project. ## Documentation Comprehensive CAPTURE documentation can be found [here](DOCUMENTATION.md). From 613b8be83c37739266fbddacb5c68136ca0a1dc6 Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Tue, 21 Apr 2026 13:18:53 -0500 Subject: [PATCH 16/18] Fix README quick start typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8796dc4..88654d7 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ CAPTURE will execute the workflow using its built-in conventions for job executi ### 4. Verify results ``` cap verify verifications/example.sh -git diff verfications/example.out +git diff verifications/example.out ``` Outputs are checked for consistency and reproducibility. If there is no difference in `verifications/example.out`, the example results were fully reproduced. From 0b555450a5d036911c732d73e226b6706d33c47b Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Tue, 21 Apr 2026 14:06:20 -0500 Subject: [PATCH 17/18] Add new project-template subproject commit --- project-template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project-template b/project-template index 96685c2..782636f 160000 --- a/project-template +++ b/project-template @@ -1 +1 @@ -Subproject commit 96685c25f24d77917665ea281573fde4c2b0ec11 +Subproject commit 782636f9407881b1e75c5f8d1c4b0bf134cf30a9 From 1593a3921447d0707956e302757db116c959fa5b Mon Sep 17 00:00:00 2001 From: Tonie Crumley Date: Tue, 21 Apr 2026 15:11:32 -0500 Subject: [PATCH 18/18] Improve verification feedback in the quick start guide --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 88654d7..22b1d2c 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ CAPTURE will execute the workflow using its built-in conventions for job executi ### 4. Verify results ``` cap verify verifications/example.sh -git diff verifications/example.out +git diff --quiet verifications/example.out && echo "Verification succeeded" || git diff ``` Outputs are checked for consistency and reproducibility. If there is no difference in `verifications/example.out`, the example results were fully reproduced.