Updating tooling docs

Mitchob · Mitchob · commit b5f5c3ed455c · 2026-06-18T15:32:48.000+10:00
diff --git a/_data/sidebars/main.yml b/_data/sidebars/main.yml
@@ -38,14 +38,20 @@ subitems:
         subitems:
           - title: Verify CVMFS
             url: /tools#cvmfs
-          - title: Find and install with Bio-Shelley
+          - title: Find and install with Shelley-Bio
             url: /tools#bio-shelley
           - title: Load a tool
             url: /tools#load-tool
           - title: Reference datasets
             url: /tools#reference-data
-          - title: Advanced — manual SHPC
+          - title: Advanced — manual sHPC
             url: /tools#shpc-manual
+          - title: Older or unlisted versions
+            url: /tools#older-versions
+          - title: Troubleshooting
+            url: /tools#troubleshooting
+          - title: Further reading
+            url: /tools#further-reading
       - title: Interactive environments
         url: /interactive
         subitems:
diff --git a/pages/about.md b/pages/about.md
@@ -7,7 +7,7 @@ description: How to cite this guide, contributor acknowledgements, and reference
 
 If you use this guide in your research or training, please cite it as:
 
-> O'Brien M, Jaya F, Samaha G, Xue W, Al Bkhetan Z, Botting A, Gustafsson J. (2026).
+> O'Brien M, Jaya F, Xue W, Al Bkhetan Z, Botting A, Gustafsson J. Samaha G (2026).
 > **BioShell how-to guide**. Australian BioCommons.
 > [AUTHOR TO SUPPLY — Zenodo DOI once minted]
 
@@ -19,15 +19,15 @@ If you use this guide in your research or training, please cite it as:
 
 ## Contributors {#contributors}
 
-| Name | GitHub | ORCID |
-|------|--------|-------|
-| Mitchell O'Brien | [@Mitchob](https://github.com/Mitchob) | [0000-0003-0662-9101](https://orcid.org/0000-0003-0662-9101) |
-| Fred Jaya | [@fredjaya](https://github.com/fredjaya) | [0000-0002-4019-7026](https://orcid.org/0000-0002-4019-7026) |
-| Georgina Samaha | [@georgiesamaha](https://github.com/georgiesamaha) | [0000-0003-0419-1476](https://orcid.org/0000-0003-0419-1476) |
-| Wenjing Xue | [@eileen-xue](https://github.com/eileen-xue) | — |
-| Ziad Al Bkhetan | [@ziadbkh](https://github.com/ziadbkh) | [0000-0002-4032-5331](https://orcid.org/0000-0002-4032-5331) |
-| Andy Botting | [@andybotting](https://github.com/andybotting) | — |
-| Johan Gustafsson | [@supernord](https://github.com/supernord) | [0000-0002-2977-5032](https://orcid.org/0000-0002-2977-5032) |
+| Name             | GitHub                                             | ORCID                                                        |
+| ---------------- | -------------------------------------------------- | ------------------------------------------------------------ |
+| Mitchell O'Brien | [@Mitchob](https://github.com/Mitchob)             | [0000-0003-0662-9101](https://orcid.org/0000-0003-0662-9101) |
+| Fred Jaya        | [@fredjaya](https://github.com/fredjaya)           | [0000-0002-4019-7026](https://orcid.org/0000-0002-4019-7026) |
+| Georgina Samaha  | [@georgiesamaha](https://github.com/georgiesamaha) | [0000-0003-0419-1476](https://orcid.org/0000-0003-0419-1476) |
+| Wenjing Xue      | [@eileen-xue](https://github.com/eileen-xue)       | —                                                            |
+| Ziad Al Bkhetan  | [@ziadbkh](https://github.com/ziadbkh)             | [0000-0002-4032-5331](https://orcid.org/0000-0002-4032-5331) |
+| Andy Botting     | [@andybotting](https://github.com/andybotting)     | —                                                            |
+| Johan Gustafsson | [@supernord](https://github.com/supernord)         | [0000-0002-2977-5032](https://orcid.org/0000-0002-2977-5032) |
 
 ---
 
diff --git a/pages/tools.md b/pages/tools.md
@@ -1,25 +1,27 @@
 ---
 title: Tools and reference data
-description: How to find, install, and load bioinformatics tools and reference datasets in BioShell using CVMFS, Bio-Shelley, and SHPC.
+description: How to find, install, and load bioinformatics tools and reference datasets in BioShell using CVMFS, Shelley-Bio, and sHPC.
 ---
 
 BioShell gives you access to thousands of bioinformatics tools and reference datasets through
-[CernVM-FS (CVMFS)](https://cvmfs.readthedocs.io/en/stable/), a read-only shared file system
-that delivers containerised tools and reference data on demand. Rather than downloading or
-installing software yourself, you load tools as modules — they are pulled from CVMFS only when
-you need them.
+two underlying systems — **CVMFS** and **sHPC** — and a built-in assistant called
+**Shelley-Bio** that automates working with both.
 
-Through CVMFS, BioShell connects to three repositories:
+[CernVM-FS (CVMFS)](https://cvmfs.readthedocs.io/en/stable/) is a read-only, network-backed
+file system that delivers containerised tools and reference data on demand. Rather than
+downloading or installing software yourself, you access tools as modules — files are pulled
+from CVMFS only when you need them and cached locally.
+
+BioShell mounts three CVMFS repositories automatically:
 
 | Repository | Contents |
 |-----------|----------|
-| `singularity.galaxyproject.org` | Containerised tools from [BioContainers](https://biocontainers.pro/) |
+| `singularity.galaxyproject.org` | 100,000+ containerised tools from [BioContainers](https://biocontainers.pro/) |
 | `data.galaxyproject.org` | Reference genome builds and pre-built indexes from the Galaxy Project |
 | `data.biocommons.aarnet.edu.au` | [AUTHOR TO SUPPLY — confirm final name and description] |
 
-> **Note:** CVMFS does not download everything in advance. It fetches files only when you
-> access them, so the first time you use a tool or dataset it may take a moment to load.
-> Subsequent access is faster because the files are cached locally.
+> **Recommended approach:** Start with Shelley-Bio. Use the manual CVMFS/sHPC methods only
+> if you need finer control or are troubleshooting.
 
 ---
 
@@ -62,17 +64,21 @@ You should see `OK` for each repository. If a repository shows `Failed!`, wait a
 try again. Contact [Australian BioCommons support](https://www.biocommons.org.au/helpdesk)
 if the problem persists.
 
+> **Note:** CVMFS does not download everything in advance. It fetches files only when you
+> access them, so the first time you use a tool or dataset it may take a moment to load.
+> Subsequent access is faster because the files are cached locally.
+
 ---
 
-## Step 2 — Find and install tools with Bio-Shelley {#bio-shelley}
+## Step 2 — Find and install tools with Shelley-Bio {#bio-shelley}
 
-**Bio-Shelley** is BioShell's built-in assistant for finding and installing tools. It indexes
-over 700 tools and 118,000 container versions from the BioContainers catalogue so you do not
-need to know container paths or SHPC commands.
+**Shelley-Bio** is BioShell's built-in assistant for finding and installing tools. It indexes
+over 700 tools and 118,000 container versions from the BioContainers catalogue, so you do not
+need to know container paths or sHPC commands.
 
-You can use Bio-Shelley in two ways:
+You can use Shelley-Bio in two ways:
 
-**Directly from the command line:**
+**From the command line:**
 
 ```bash
 shelley-bio find <tool>
@@ -81,7 +87,7 @@ shelley-bio versions <tool>
 shelley-bio build <tool>
 ```
 
-**Or in interactive mode:**
+**In interactive mode:**
 
 ```bash
 shelley-bio interactive
@@ -93,11 +99,28 @@ shelley-bio interactive
 | `search "<function>"` | Search by keyword or function | `shelley-bio search "quality control"` |
 | `versions <tool>` | List all available versions | `shelley-bio versions samtools` |
 | `build <tool[/version]>` | Install the tool as a loadable module | `shelley-bio build samtools/1.21` |
-| `interactive` | Launch Bio-Shelley in interactive mode | `shelley-bio interactive` |
+| `interactive` | Launch Shelley-Bio in interactive mode | `shelley-bio interactive` |
 
 The `build` command handles the full installation automatically — it finds the correct
 container in CVMFS, generates the module file, and makes the tool ready to load.
 
+### Example: installing `samtools`
+
+```bash
+shelley-bio find samtools
+# samtools — Tools for manipulating next-generation sequencing data
+
+shelley-bio versions samtools
+# samtools/1.21
+# samtools/1.20
+# samtools/1.19.2
+# ...
+
+shelley-bio build samtools/1.21
+# Installing samtools/1.21 from CVMFS...
+# Module samtools/1.21 was created.
+```
+
 ![](images/bioshell/SCREENSHOT_NEEDED_bioshelley_build.png)
 
 *Fig 3. Using `shelley-bio build samtools/1.21` to install a tool module.*
@@ -109,7 +132,7 @@ container in CVMFS, generates the module file, and makes the tool ready to load.
 
 ## Step 3 — Load a tool {#load-tool}
 
-Once Bio-Shelley has built a module, load it with:
+Once Shelley-Bio has built a module, load it with:
 
 ```bash
 module use ~/shpc/modules
@@ -130,9 +153,9 @@ Verify the tool is ready:
 ## Reference datasets {#reference-data}
 
 CVMFS also provides access to reference genome builds and pre-built indexes from the Galaxy
-Project. You can use these directly in your analyses by referencing their absolute file path.
-
-List the available reference data:
+Project. You can browse the full repository at
+[datacache.galaxyproject.org](http://datacache.galaxyproject.org) before triggering any
+downloads on the VM. On BioShell, the same content is available at:
 
 ```bash
 ls /cvmfs/data.galaxyproject.org/byhand/
@@ -144,28 +167,35 @@ ls /cvmfs/data.galaxyproject.org/managed/
 | `/managed` | Datasets generated with Galaxy Data Manager tools. Organised by index type, then genome build. |
 | `/byhand` | Older, manually curated datasets. Organised by genome build, then index type. |
 
-To use a reference file in your analysis, pass its absolute path directly to your tool.
-For example, the human CHM13 T2T v2.0 FASTA file is at:
+To use a reference file in your analysis, pass its absolute path directly to your tool or
+pipeline config. For example, the human CHM13 T2T v2.0 FASTA file is at:
 
 ```
 /cvmfs/data.galaxyproject.org/byhand/CHM13_T2T_v2.0/seq/CHM13_T2T_v2.0.fa
 ```
 
-> **Note:** The reference datasets available through CVMFS are those hosted by the Galaxy
-> Project and may not be comprehensive. This is not a replacement for your institution's
-> primary data access methods.
+The T2T genome directory also includes pre-built indexes for common aligners:
+
+```bash
+ls /cvmfs/data.galaxyproject.org/byhand/CHM13_T2T_v2.0/
+# bowtie2_index/  bwa_mem_index/  bwameth_index/  hisat2_index/  len/  rnastar/  seq/
+```
+
+> **Note:** The reference datasets available through CVMFS are maintained by the Galaxy Project
+> and may not be comprehensive. This is not a replacement for your institution's primary data
+> access methods.
 
 ---
 
-## Advanced: manual SHPC commands {#shpc-manual}
+## Advanced: manual sHPC commands {#shpc-manual}
 
-If you need to install a tool version that Bio-Shelley cannot find, or you prefer to work
-directly with SHPC, use the steps below.
+If you need to install a tool version that Shelley-Bio cannot find, or you prefer to work
+directly with sHPC, use the steps below. This example uses `plink` throughout.
 
-> **Tip:** Bio-Shelley handles most cases, including older tool versions not in the default
+> **Tip:** Shelley-Bio handles most cases, including older tool versions not in the default
 > registry. Try `shelley-bio versions <tool>` before attempting manual installation.
 
-**Load SHPC:**
+**Load sHPC:**
 
 ```bash
 module load shpc
@@ -174,24 +204,141 @@ module load shpc
 **Search for a tool and view versions:**
 
 ```bash
-shpc show -f <toolname>
-shpc show quay.io/biocontainers/<tool>
+shpc show -f plink
+shpc show quay.io/biocontainers/plink
 ```
 
 **Install a module directly from CVMFS:**
 
 ```bash
-shpc install quay.io/biocontainers/<tool>:<tag> \
-  /cvmfs/singularity.galaxyproject.org/all/<tool>:<tag> \
+shpc install quay.io/biocontainers/plink:1.90b7.7--h18e278d_1 \
+  /cvmfs/singularity.galaxyproject.org/all/plink:1.90b7.7--h18e278d_1 \
   --keep-path
 ```
 
+The `--keep-path` flag tells sHPC to use the container already present in CVMFS rather than
+downloading a fresh copy.
+
 **Make the module available and load it:**
 
 ```bash
 module use ~/shpc/modules
-module load quay.io/biocontainers/<tool>/<tag>
+module load quay.io/biocontainers/plink/1.90b7.7--h18e278d_1
+plink --version
+```
+
+**Or run the container directly without a module:**
+
+```bash
+singularity exec \
+  /cvmfs/singularity.galaxyproject.org/all/plink:1.90b7.7--h18e278d_1 \
+  plink --version
 ```
 
 For further documentation see the
-[SHPC user guide](https://singularity-hpc.readthedocs.io/en/latest/getting_started/user-guide.html).
+[sHPC user guide](https://singularity-hpc.readthedocs.io/en/latest/getting_started/user-guide.html).
+
+---
+
+## Installing older or unlisted versions {#older-versions}
+
+Some older tool versions exist in CVMFS but are not listed in the default sHPC registry.
+Shelley-Bio handles this automatically via its `build` command, but if you need to do it
+manually, follow these steps.
+
+**1. Check CVMFS for the version:**
+
+```bash
+ls /cvmfs/singularity.galaxyproject.org/all/plink:1.90b4*
+```
+
+**2. Get its checksum:**
+
+```bash
+sha256sum /cvmfs/singularity.galaxyproject.org/all/plink:1.90b4--h0a6d026_2
+```
+
+**3. Create a local registry entry:**
+
+```bash
+sudo mkdir -p /apps/local/quay.io/biocontainers/plink
+
+# Fetch the existing remote recipe as a base
+curl -fsSL \
+  https://raw.githubusercontent.com/singularityhub/shpc-registry/main/quay.io/biocontainers/plink/container.yaml \
+  -o /apps/local/quay.io/biocontainers/plink/container.yaml
+
+# Edit the YAML to add the missing version tag and its checksum
+```
+
+**4. Register your local registry:**
+
+```bash
+sudo shpc config add registry /apps/local
+```
+
+Verify your local registry appears first — it must take precedence over the remote registry:
+
+```bash
+shpc config get registry
+# ['/apps/local', 'https://github.com/singularityhub/shpc-registry']
+```
+
+**5. Install from CVMFS:**
+
+```bash
+sudo shpc install quay.io/biocontainers/plink:1.90b4--h0a6d026_2 \
+  /cvmfs/singularity.galaxyproject.org/all/plink:1.90b4--h0a6d026_2 \
+  --keep-path
+```
+
+> **Tip:** If Shelley-Bio recognises the container path but the version is not in the
+> registry, `shelley-bio build` handles the local registry creation automatically.
+
+---
+
+## Troubleshooting {#troubleshooting}
+
+**CVMFS probe fails for a repository**
+
+The repository may be temporarily unavailable. Wait a moment and run `cvmfs_config probe`
+again. Contact [Australian BioCommons support](https://www.biocommons.org.au/helpdesk) if
+the problem persists.
+
+**`shpc show` returns no results**
+
+The default registry may not include every BioContainers tool. Try searching the remote
+registry explicitly:
+
+```bash
+shpc show quay.io/biocontainers/<tool> \
+  --registry https://github.com/singularityhub/shpc-registry
+```
+
+The official registry is updated nightly.
+
+**Module appears in `module avail` but will not load**
+
+Check that both `shpc` and `singularity` are loaded:
+
+```bash
+module list
+```
+
+sHPC requires Singularity to execute containers.
+
+**Shelley-Bio cannot find a tool**
+
+Try `search` with different keywords — for example, `shelley-bio search "alignment"` instead
+of a specific tool name. If the container exists in CVMFS but Shelley-Bio does not index it,
+fall back to the manual sHPC method above.
+
+---
+
+## Further reading {#further-reading}
+
+- [sHPC user guide](https://singularity-hpc.readthedocs.io/en/latest/getting_started/user-guide.html)
+- [BioContainers registry](https://biocontainers.pro/registry)
+- [Shelley-Bio on GitHub](https://github.com/Sydney-Informatics-Hub/shelley-bio)
+- [CVMFS documentation](https://cvmfs.readthedocs.io/en/stable/)
+- [Galaxy Project CVMFS repositories](https://galaxyproject.org/admin/cvmfs/)
