Use shpc to build modules

[fredjaya]

This PR makes shelley 🐢 build modules using shpc install under the hood, replacing creating a dummy modules.lua file.

Manual tests

blast:2.5.0--hc0b0e79_3

• Old version that does not exist in remote registry
• Requires multiple aliases to different binaries e.g. blastn, blastp

shelley-bio build blast:2.5.0--hc0b0e79_3
module load blast/2.5.0--hc0b0e79_3
blastn -version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container blastn: 2.5.0+
 Package: blast 2.5.0, build Sep 20 2018 01:34:18

blastp -version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
blastp: 2.5.0+
 Package: blast 2.5.0, build Sep 20 2018 01:34:18

Notes:

• creating local registry and adding new container works
• aliases are preserved this way
• /etc/resolv.conf warning

star/2.7.11a--h0033a41_0

• Latest version that exists in the registry
• Issues with bad $PATH exports due to manual module.lua build shelley-bio build: executable file not found in $PATH #9

shelley-bio build star
module load star/2.7.11b--h5ca1c30_8
STAR --version

2.7.11b # success

star-fusion:1.0.0

• Older version of star-fusion that is not in the registry
• Has many aliases in the rmeote registry

shelley-bio build star-fusion:1.0.0
module load star-fusion

STARlong --version

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
STAR_2.5.3a

salmon

WARNING: Skipping mount /opt/singularity/4.2.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
FATAL:   stat /usr/local/bin/salmon: no such file or directory

• Issue founds, aliases may not reflect older builds, or are broken. Inspect.

star-fusion-exec find /usr/local/bin -type f | grep salmon # no salmon
star-fusion-exec find /usr/local/bin -type f | grep STAR

/usr/local/bin/STAR
/usr/local/bin/STAR-Fusion
/usr/local/bin/STARlong

Confirmed: registry aliases does not reflect older versions.

Possible solution is to use an automated binary detection like shpc-guts that is compatible with singularity?

[fredjaya]

Spike test: Inspecting CVMFS images for aliases during build

Utilising shpc-registry helper scripts to create accurate registries

from add_container.py:

    # Generate guts
    cli = ManifestGenerator()
    manifests = cli.diff(args.image)

    # Assemble aliases
    aliases = {}
    for path in list(manifests.values())[0]["diff"]["unique_paths"]:
        # Don't include system bin
        if "sbin" in path or "/usr/bin" in path:
            print(f"Skipping system bin {path}")
            continue

        name = os.path.basename(path)
        if name in aliases:
            print(f"Warning, duplicate alias {name}")
        aliases[name] = path

    container["aliases"] = aliases
    print(f"Writing with aliases to {container_yaml}")
    shpc.utils.write_yaml(container, container_yaml)
    print(f"container_yaml={container_yaml} >> $GITHUB_OUTPUT")

container_guts.main.ManifestGenerator() looks likes the way the shpc workflow automatically updates the aliases, but uses Docker so this is a no go for the BioShell environment

---

Using find?

module load star-fusion
star-fusion-exec find /usr/local/bin -perm +111 -maxdepth 1 | wc -l

146 # MANY binaries...

Can we diff out the utility tools (e.g. non bioinformatics binaries) from /usr/bin?

star-fusion-exec find /usr/bin -perm +111 -maxdepth 1 | head

/usr/bin/[
/usr/bin/[[
/usr/bin/ar
/usr/bin/awk
/usr/bin/basename
/usr/bin/bunzip2
/usr/bin/bzcat
/usr/bin/chrt
/usr/bin/chvt

STAR-fusion v1.0.0 conda build script:

• https://github.com/bioconda/bioconda-recipes/tree/75505efe265e4c7143e57ff4fa03506f554715d0/recipes/star-fusion

Resource for biocontainer builds:

• https://cyverse-container-camp-workshop-2018.readthedocs-hosted.com/en/latest/biocontainer/biocontainers.html#introduction-to-biocontainers

[Mitchob]

Initial testing Summary for issue-9 PR

**Environment:**Nectar. bioimage_Shelley_with_Rstudio

blast 2.17.0 (latest, registry hit)

shelley-bio find blast correctly identified 48 available versions and the most recent. shelley-bio build blast:2.17.0 resolved to 2.17.0--h66d330f_0 and built successfully with the module symlinked to the SHPC-generated modulefile.

blast 2.16.0 (registry hit, multiple builds)

Build correctly prompted to select between available builds, user selected 2.16.0--hc155240_3. Module built successfully. All expected BLAST aliases present on PATH after module load (blastn, blastp, blastx, makeblastdb, etc.).

Issue found: blastn/blastp initially failed with singularity: command not found. Root cause: singularity module was not loaded. Once module load singularity was run, both blastn -version and blastp -version succeeded.

Note for PR: The module should either auto-load singularity as a dependency, or shelley-bio should warn the user at build time that singularity must be loaded before using the module.

blast 2.5.0--hc0b0e79_3 (old version, registry miss)

Build failed as expected as this version predates the SHPC registry. The error message correctly lists the valid registered tags. However the suggestion text ("Check that CVMFS is mounted") is misleading here since the real issue is a registry gap, not a CVMFS problem.

Note for PR: The error message should be updated to reflect the registry miss and point the user toward those options when they land.

---

A thought on potential solutions

Ignore me if I am increadibly wrong, otherwise we can discuss this week.

Is the brunt of container_guts.main.ManifestGenerator() detecting executables that it then includes as aliases? if yes, I appreciate it might not be this simple, maybe a quicker solution or potion could be something like this:

A potential

flowchart TD
    A([shelley-bio build\ntool:version]) --> B[Check SHPC registry\nIs tool:version known?]

    B -->|yes| L([Build success!])
    B -->|no| W[⚠️ Warning: Tool not in SHPC registry\nConsider a newer version or choose a build option]

    W --> F[Find a registered version\nshelley-bio find tool]
    W --> M[Specify binaries manually\n--bins name1,name2]
    W --> D[Auto-discover executables\n--detect-bins]

    F --> V[Highlight versions in registry]
    V --> S[singularity exec tool:version]
    S --> R([shelley-bio build tool:version])
    R -->|yes| L

    M --> P[Parse provided binary list]
    D --> I["Interactive checklist\nSelect executables you need:\n( )nblасtn, \n( )blastp, \n( )blastx, \n( )tblastn, \n( )etc."]

    P --> K[Write local registry entry\n.shelley/shpc-registry/container.yaml]
    I --> K

    K --> L

Loading

The branch point is whether the requested tool version exists in the SHPC registry. If yes, build proceeds automatically. If no, there are three pathways:

Pathway 1 — Use a registered version (easiest)
Redirect the user to pick a version that SHPC already knows about. Requires a small fix to shelley-bio find to explicitly flag which versions are SHPC-ready, so the user can make an informed choice without guessing.

Pathway 2 — Manual alias build (--bins)
User already knows the binary names they need and passes them directly. Requires knowing the alias names upfront, which may not always be obvious.

Pathway 3 — Auto-detect executables (--detect-bins)
Shelley inspects the container, finds all executables, and presents an interactive checklist for the user to select from. Most automated option — no prior knowledge of alias names needed.

[fredjaya]

Rationale

Container manifests on shpc are generated via a GitHub action which was enabled in 2025, so either very recent tool builds, or pre-2025 builds are not included.

When running shelley-bio build to generate a loadable module, there are two main routes:

1. The tool version exists in the shpc-registry and a module file is simply built using the container.yaml registry via shpc install
2. The tool version does not exist in the registry, and a new container.yaml needs to be built from scratch to correctly capture the binaries in the CVMFS image

This PR adds support for both of these cases by running shpc install under the hood, and using a development branch of singularityhub/guts to generate the manifest with correct aliasing for the provided tool version.

Changes

• Adds shpc_install() in cvmfs_builder.py to support building real module .lua files, either by (1) using the upstream shpc-registry container.yaml or (2) generating a local registry if a tool version does not exist
• Adds guts_integration.py to work with https://github.com/Sydney-Informatics-Hub/guts/tree/singularity. Used to diff a CVMFS image against base OS image and conda container manifests to extract the correct aliases for a module.
• shelley_bio/data/guts_db/ - additional manifests for conda images.
• All other tests and documentation
• Moved CVMFS cache and tool metadata into shelley_bio/data/

Key things to test

Building a tool version that's in the registry vs. not, checking that these are built, module loadable, and correct binaries usable

Known issues

• When building a tag from local registry, it shows up in shelley-bio find as Buildable after building. e.g. shelley-bio find fastqc:0.X.X not buildable -> shelley-bio build fastqc0.X.X -> shelley-bio find fastqc:0.X.X buildable
• Current setup instructions might not ideal for production, may need to incorporate adding the local guts version and uv package manager in BioShell playbooks?

Setup instructions for review

• Create a new test image
• Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

• Install repos and checkout to the correct branches

# 1. Clone both repos side by side
git clone https://github.com/Sydney-Informatics-Hub/shelley-bio
git clone https://github.com/Sydney-Informatics-Hub/guts.git

# Change to the SIH dev branch that supports singularity
cd guts && git checkout singularity && cd ..

# 2. Create and activate the virtual environment
cd shelley-bio
uv venv .venv
source .venv/bin/activate

# 3. Install shelley-bio in editable mode with dev extras
#    uv resolves the local ../guts path from pyproject.toml automatically
uv pip install -e ".[dev]"

# 4. Load required modules
module load singularity shpc

# 5. Verify
shelley-bio
pytest --collect-only # check pytest is installed correctly, but do not run

• Run shelley-bio commands to your hearts content

[wiz-8f2719713e]

Wiz Scan Summary

Scanner	Findings
Vulnerabilities	5  5  5
Sensitive Data	-
Secrets	-
IaC Misconfigurations	-
SAST Findings	2
Software Management Findings	-
Total	5  7  5

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension.

[Mitchob]

Testing Summary

Manual Testing: vcftools and bcftools because they're common bioinformatics usecases and have multiple binaries to alias.

bcftools/1.23.1--hb2cee57_0 (registry hit)

Build completed successfully via shpc registry. Module loaded and bcftools -h confirmed version 1.23.1. Alias set is clean and minimal — only genuine bcftools binaries exposed (bcftools, bgzip, tabix, htsfile, helper scripts). No issues.

shelley-bio build bcftools
module load bcftools/1.23.1--hb2cee57_0
bcftools -h  ✅

bcftools/1.6--1 (registry miss → guts fallback)

Build correctly identified the version as not in the shpc registry, triggered the guts-based local registry fallback, and completed successfully. bcftools -h confirmed version 1.6. The core tool and aliases seem to work.

shelley-bio build bcftools/1.6
module load bcftools/1.6--1
bcftools -h  ✅

However the generated alias set is overly permissive, the wrapper bin includes system and dependency-level binaries that should not be exposed as module aliases:

• Kerberos utilities (kadmin, kinit, klist, krb5*)
• Network tools (curl, telnet, tftp, openssl)
• Compression tools (bzip2, xz, lzma and variants)
• Conda internals (conda, activate, deactivate)

---

vcftools/0.1.17--pl5321h077b44d_0 (registry hit)

Build succeeded via shpc registry but the registry manifest is notably incomplete , only 6 aliases were exposed despite the container having 21 binaries in /usr/local/bin:

Exposed by registry: vcf-annotate, vcf-compare, vcf-concat, vcf-consensus, vcf-contrast, vcf-convert

Missing from registry but present in container: vcf-merge, vcf-stats, vcf-subset, vcf-isec, vcf-query, vcf-sort, vcf-shuffle-cols, vcf-phased-join, vcf-indel-stats, vcf-fix-ploidy, vcf-fix-newlines, vcf-tstv, vcf-validator, vcf-to-tab, tab-to-vcf

This might be a limitation of relying solely on the shpc registry commonly used tools like vcf-merge and vcf-stats are silently unavailable to users. We should include a note about this in the documentation (Maybe a troubleshooting section)

vcftools/0.1.12b--pl5262h2e03b76_2 (registry miss → guts fallback)

This is the standout result of this review. The guts fallback correctly captured the full alias set for this older version, and all tested aliases worked after module load:

vcf-merge --help   ✅
vcf-stats --help   ✅
vcf-subset --help  ✅
vcf-isec --help    ✅

Your version captures aliases that the shpc registry misses and handles older versions entirely absent from the registry.

---

Key Findings

✅ The registry miss → guts fallback pipeline works well. Both bcftools/1.6 and vcftools/0.1.12b built successfully, modules loaded correctly, and aliases were callable. This is the core feature of the PR and it delivers.

✅ Guts captures aliases that the shpc registry misses. For vcftools/0.1.17, the shpc registry exposed only 6 of 21 available binaries. The guts approach would have caught all of them. Superstar!!!

⚠️ Guts fallback is too permissive. For bcftools/1.6, the generated alias set includes system utilities and dependency binaries unrelated to bcftools. The diffing against base images (miniconda, busybox, alpine etc.) is the right strategy but needs refinement.

---

Suggested Enhancement: Interactive or Post-build alias filtering

Sorry for repeating myself, BUT I think the over-permissive alias issue could be addressed with an interactive or post-processing step that lets users curate the generated alias set. For example if interaactive:

shelley-bio build bcftools/1.6 --detect-bins

or post-script

shelley-bio build bcftools/1.6 --select-aliases

Which could present an interactive checklist similar to what I suggested earlier:

The following aliases were detected for bcftools/1.6.
Select the ones to keep:

[x] bcftools
[x] bgzip
[x] tabix
[ ] curl
[ ] kadmin
[ ] kinit
...

A non-interactive alternative could be a --exclude-aliases flag:

shelley-bio build bcftools/1.6 --exclude-aliases curl,kadmin,kinit

Or a post-build subcommand built into shelley:

shelley-bio aliases bcftools/1.6 --remove curl,kadmin,kinit
shelley-bio aliases bcftools/1.6 --interactive

This keeps the build pipeline clean while giving power users a way to tidy up the module after the fact.

---

[Mitchob]

Quick Test: Container Alias overlap

When multiple modules are loaded simultaneously, conflicting aliases silently resolve to whichever module was loaded last. For example, with both vcftools/0.1.12b and bcftools/1.6 loaded, system utilities captured by the guts fallback (freeramdisk, telnet) resolve to the vcftools wrapper (last loaded module):

$ which freeramdisk
/home/ubuntu/shpc/wrappers/quay.io/biocontainers/vcftools/0.1.12b--pl5262h2e03b76_2/bin/freeramdisk

$ which telnet
/home/ubuntu/shpc/wrappers/quay.io/biocontainers/vcftools/0.1.12b--pl5262h2e03b76_2/bin/telnet

Not sure this is a big deal but it means irrelevant binaries are being exposed, and they can silently shadow each other depending on module load order. Might influence usability and reproducibility in multi-tool workflows. There is no warning to the user that this has happened.

[fredjaya]

Thanks Mitchell!

I had a look at the same bcftools versions, and the star-fusion:1.0.0 i've been using to test, and agree that a shelley-bio build bcftools/1.6 --detect-bins approach will be suitable.

Two different cases for the two tools:

1. bcftools understandably bundles compression tools which don't need to be aliased, but cannot be diffed out because they are genuinely unique binaries in the CVMFS image
2. star-fusion may have used a perl base image? I imagine there are a lot of other images that use different bases that cannot all be accounted for.

Also conda binaries are in different paths between the base and target image, no diffing without additional parsing.