Skip to content

fix: validate Xenium bundle after UNTAR (drop test-profile skip-validation guard) #159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
heylf merged 5 commits into from
May 20, 2026
Merged

fix: validate Xenium bundle after UNTAR (drop test-profile skip-validation guard) #159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0400c4
fix: validate xenium bundle after UNTAR; drop test-profile skip-valid…
an-altosian May 18, 2026
55fa0b3
ci: use volume=80gb (mirrors fix in PR #158)
an-altosian May 18, 2026
8996511
ci: bump nf-test 0.9.3 → 0.9.5; rebase subworkflow on PR #158's version
an-altosian May 18, 2026
b7beb7f
Revert subworkflow file to PR #159's standalone version
an-altosian May 18, 2026
074bfe4
fix(untar): emit valid empty gzip in stub to fix non-deterministic sn…
an-altosian May 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/nf-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ concurrency:

env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NFT_VER: "0.9.3"
NFT_VER: "0.9.5"
NFT_WORKDIR: "~"
NXF_ANSI_LOG: false
NXF_SINGULARITY_CACHEDIR: ${{ github.workspace }}/.singularity
Expand Down Expand Up @@ -64,7 +64,7 @@ jobs:
runs-on: # use self-hosted runners
- runs-on=${{ github.run_id }}-nf-test
- runner=4cpu-linux-x64
- disk=large
- volume=80gb
strategy:
fail-fast: false
matrix:
Expand Down
27 changes: 20 additions & 7 deletions modules/nf-core/untar/main.nf
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

46 changes: 40 additions & 6 deletions modules/nf-core/untar/untar.diff
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

89 changes: 3 additions & 86 deletions subworkflows/local/utils_nfcore_spatialxe_pipeline/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,12 +139,9 @@ workflow PIPELINE_INITIALISATION {
}


//
// Check and validate xenium bundle
//
if (!workflow.profile.contains('test')) {
validateXeniumBundle(ch_samplesheet)
}
// Xenium bundle file-presence validation now runs in the main workflow
// (workflows/spatialxe.nf) AFTER UNTAR staging, so it works uniformly for
// both directory inputs and tarball inputs.

emit:
samplesheet = ch_samplesheet
Expand Down Expand Up @@ -276,86 +273,6 @@ def validateInputParameters(
}
}

//
// Check and validate xenium bundle
//
def validateXeniumBundle(ch_samplesheet) {

// define xenium bundle directory structure - required files
def bundle_required_files = [
"cell_boundaries.csv.gz",
"cell_boundaries.parquet",
"cell_feature_matrix.h5",
"cell_feature_matrix.zarr.zip",
"cells.csv.gz",
"cells.parquet",
"cells.zarr.zip",
"experiment.xenium",
"gene_panel.json",
"metrics_summary.csv",
"morphology.ome.tif",
"morphology_focus/",
"nucleus_boundaries.csv.gz",
"nucleus_boundaries.parquet",
"transcripts.parquet",
"transcripts.zarr.zip",
]

// bundle optional files
def bundle_optional_files = [
"analysis.tar.gz",
"analysis.zarr.zip",
"analysis_summary.html"
]

// get bundle path (keep raw string for remote-path detection)
def ch_bundle_info = ch_samplesheet.map { _meta, bundle, _image ->
def rawPath = bundle.toString().replaceFirst(/\/$/, '')
def bundle_path = file(rawPath)
return [rawPath, bundle_path]
}

// Skip file-level validation for remote paths (S3, GS, AZ) because
// file().exists() is unreliable on cloud storage during initialization
// (Fusion mounts s3://bucket as /bucket, breaking startsWith checks).
// Files will be validated at task staging time instead.
ch_bundle_info.map { rawPath, path ->
if (rawPath.startsWith('s3://') || rawPath.startsWith('gs://') || rawPath.startsWith('az://')) {
log.info("Skipping bundle file validation for remote path: ${rawPath}")
return
}

def missing_required_files = []
def missing_optional_files = []

def requiredExist = bundle_required_files.every { filename ->
def fullPath = file("${path}/${filename}")
if (!fullPath.exists()) {
missing_required_files.add(filename)
return false
}
return true
}
if (!requiredExist) {
error("❌ Missing file(s) at bundle path provided in the samplesheet: ${missing_required_files}")
}

def optionalExist = bundle_optional_files.every { filename ->
def fullPath = file("${path}/${filename}")
if (!fullPath.exists()) {
missing_optional_files.add(filename)
return false
}
return true
}
if (!optionalExist) {
log.warn("⚠️ Missing optional file(s) at bundle path provided in the samplesheet: ${missing_optional_files}")
}

log.info("✅ Xenium bundle validated.\n")
}
}

//
// Generate methods description for MultiQC
//
Expand Down
45 changes: 41 additions & 4 deletions tests/coordinate_mode.nf.test.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,29 @@
"coordinate/spatialdata/write/spatialdata/test_run/redefined_bundle/fake_file.txt",
"coordinate/untar",
"coordinate/untar/test_run",
"coordinate/untar/test_run/.end-of-run",
"coordinate/untar/test_run/analysis.tar.gz",
"coordinate/untar/test_run/analysis.zarr.zip",
"coordinate/untar/test_run/analysis_summary.html",
"coordinate/untar/test_run/aux_outputs.tar.gz",
"coordinate/untar/test_run/cell_boundaries.csv.gz",
"coordinate/untar/test_run/cell_boundaries.parquet",
"coordinate/untar/test_run/cell_feature_matrix.h5",
"coordinate/untar/test_run/cell_feature_matrix.tar.gz",
"coordinate/untar/test_run/cell_feature_matrix.zarr.zip",
"coordinate/untar/test_run/cells.csv.gz",
"coordinate/untar/test_run/cells.parquet",
"coordinate/untar/test_run/cells.zarr.zip",
"coordinate/untar/test_run/experiment.xenium",
"coordinate/untar/test_run/gene_panel.json",
"coordinate/untar/test_run/metrics_summary.csv",
"coordinate/untar/test_run/morphology.ome.tif",
"coordinate/untar/test_run/morphology_focus",
"coordinate/untar/test_run/morphology_focus/morphology_focus_0000.ome.tif",
"coordinate/untar/test_run/nucleus_boundaries.csv.gz",
"coordinate/untar/test_run/nucleus_boundaries.parquet",
"coordinate/untar/test_run/transcripts.parquet",
"coordinate/untar/test_run/transcripts.zarr.zip",
"coordinate/xeniumranger",
"coordinate/xeniumranger/import_segementation",
"coordinate/xeniumranger/import_segementation/test_run",
Expand All @@ -99,17 +118,35 @@
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
".end-of-run:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis_summary.html:md5,d41d8cd98f00b204e9800998ecf8427e",
"aux_outputs.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_boundaries.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_boundaries.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.h5:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"experiment.xenium:md5,d41d8cd98f00b204e9800998ecf8427e",
"gene_panel.json:md5,d41d8cd98f00b204e9800998ecf8427e",
"metrics_summary.csv:md5,d41d8cd98f00b204e9800998ecf8427e",
"morphology.ome.tif:md5,d41d8cd98f00b204e9800998ecf8427e",
"morphology_focus_0000.ome.tif:md5,d41d8cd98f00b204e9800998ecf8427e",
"nucleus_boundaries.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"nucleus_boundaries.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"transcripts.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"transcripts.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"experiment.xenium:md5,d41d8cd98f00b204e9800998ecf8427e"
]
],
"timestamp": "2026-05-18T21:58:08.945192717",
"meta": {
"nf-test": "0.9.3",
"nextflow": "25.10.2"
},
"timestamp": "2026-03-22T19:54:10.312439732"
"nf-test": "0.9.5",
"nextflow": "25.10.4"
}
}
}
45 changes: 41 additions & 4 deletions tests/default.nf.test.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,29 @@
"coordinate/spatialdata/write/spatialdata/test_run/redefined_bundle/fake_file.txt",
"coordinate/untar",
"coordinate/untar/test_run",
"coordinate/untar/test_run/.end-of-run",
"coordinate/untar/test_run/analysis.tar.gz",
"coordinate/untar/test_run/analysis.zarr.zip",
"coordinate/untar/test_run/analysis_summary.html",
"coordinate/untar/test_run/aux_outputs.tar.gz",
"coordinate/untar/test_run/cell_boundaries.csv.gz",
"coordinate/untar/test_run/cell_boundaries.parquet",
"coordinate/untar/test_run/cell_feature_matrix.h5",
"coordinate/untar/test_run/cell_feature_matrix.tar.gz",
"coordinate/untar/test_run/cell_feature_matrix.zarr.zip",
"coordinate/untar/test_run/cells.csv.gz",
"coordinate/untar/test_run/cells.parquet",
"coordinate/untar/test_run/cells.zarr.zip",
"coordinate/untar/test_run/experiment.xenium",
"coordinate/untar/test_run/gene_panel.json",
"coordinate/untar/test_run/metrics_summary.csv",
"coordinate/untar/test_run/morphology.ome.tif",
"coordinate/untar/test_run/morphology_focus",
"coordinate/untar/test_run/morphology_focus/morphology_focus_0000.ome.tif",
"coordinate/untar/test_run/nucleus_boundaries.csv.gz",
"coordinate/untar/test_run/nucleus_boundaries.parquet",
"coordinate/untar/test_run/transcripts.parquet",
"coordinate/untar/test_run/transcripts.zarr.zip",
"coordinate/xeniumranger",
"coordinate/xeniumranger/import_segementation",
"coordinate/xeniumranger/import_segementation/test_run",
Expand All @@ -99,17 +118,35 @@
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
"fake_file.txt:md5,d41d8cd98f00b204e9800998ecf8427e",
".end-of-run:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"analysis_summary.html:md5,d41d8cd98f00b204e9800998ecf8427e",
"aux_outputs.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_boundaries.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_boundaries.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.h5:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.tar.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cell_feature_matrix.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"cells.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"experiment.xenium:md5,d41d8cd98f00b204e9800998ecf8427e",
"gene_panel.json:md5,d41d8cd98f00b204e9800998ecf8427e",
"metrics_summary.csv:md5,d41d8cd98f00b204e9800998ecf8427e",
"morphology.ome.tif:md5,d41d8cd98f00b204e9800998ecf8427e",
"morphology_focus_0000.ome.tif:md5,d41d8cd98f00b204e9800998ecf8427e",
"nucleus_boundaries.csv.gz:md5,d41d8cd98f00b204e9800998ecf8427e",
"nucleus_boundaries.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"transcripts.parquet:md5,d41d8cd98f00b204e9800998ecf8427e",
"transcripts.zarr.zip:md5,d41d8cd98f00b204e9800998ecf8427e",
"experiment.xenium:md5,d41d8cd98f00b204e9800998ecf8427e"
]
],
"timestamp": "2026-05-18T21:55:37.544217665",
"meta": {
"nf-test": "0.9.3",
"nextflow": "25.10.2"
},
"timestamp": "2026-03-22T19:55:52.515294044"
"nf-test": "0.9.5",
"nextflow": "25.10.4"
}
}
}
Loading
Loading