Skip to content

Add MAGMA as --mode magma (pinned to v0.3.0; version-isolated; experimental)#18

Draft
abhi18av wants to merge 8 commits into
devfrom
integrate-magma
Draft

Add MAGMA as --mode magma (pinned to v0.3.0; version-isolated; experimental)#18
abhi18av wants to merge 8 commits into
devfrom
integrate-magma

Conversation

@abhi18av

Copy link
Copy Markdown
Member

Add MAGMA as the second analysis mode (--mode magma)

Integrates the TORCH-consortium/magma nf-core port (from mycobactopia-org/MAGMA, branch nf-core-tbanalyzer) into the tbanalyzer meta-pipeline as the second bundled pipeline, after MTBseq (--mode mtbseq, #17).

Warning

MAGMA mode is experimental / not yet biologically validated. The full Docker run (GATK/VQSR/phylogeny) is deferred to a follow-up. --mode mtbseq is unaffected. Draft until the MAGMA Docker test is green.

Design principle: version-isolated, orchestration-only

Each bundled pipeline may need different versions of the same tools, so tbanalyzer does not dedup/share modules and does not modify the pipelines' logic — it only facilitates execution. MAGMA's complete module set is vendored under a magma-scoped path with its own container pins:

  • modules/local/magma/ (17 custom modules) + modules/local/magma/nf-core/ (24 nf-core module copies at MAGMA's own versions — not sharing tbanalyzer's modules/nf-core/{gatk4,samtools,bcftools,fastqc,…})
  • subworkflows/local/magma/ (7), workflows/magmanf.nf (entry workflow MAGMA), bin/ (15 scripts), data/magma-references/ (~28 MB H37Rv + VQSR truth VCFs + GVCF panel + SnpEff DB)

Wiring

  • main.nf dispatches --mode magmaMAGMA; mode enum + magma_options schema group (61 params, all magma_-namespaced; tool *_path params dropped)
  • Mode-gated conf/magma_base.config + conf/magma_modules.config; test_magma profile + conf/magma_test.config (3 downsampled M. tuberculosis samples on Zenodo, doi 10.5281/zenodo.20671479)
  • assets/schema_input.json uses anyOf so the shared input schema accepts both the mtbseq (sample,fastq_1) and MAGMA-native (Sample,R1) samplesheet shapes; MAGMA self-validates its own CSV

Upstream fixes (made on mycobactopia-org/MAGMA, then re-vendored)

Completed gaps in the in-progress nf-core-module migration: GATK_MARK_DUPLICATES_DELLY call (.out.out.bam), and def prefix/def annotationPrefix in two stub blocks. Port nextflow lint is clean; the MAGMA DAG now compiles.

Validation

  • Config parses (strict v2) for both modes; samplesheet validates; full MAGMA DAG compiles & schedules.
  • -stub-run can't fully exercise MAGMA (its workflow does splitJson() on validator output, which stubs leave empty — a stub-mode limitation). Biological validation = the deferred Docker run.

dev → master is the intended release flow.

abhi18av added 4 commits June 23, 2026 17:44
Integrates the TORCH-consortium/magma nf-core port (mycobactopia-org/MAGMA
branch nf-core-tbanalyzer) as the second analysis mode, following the
orchestration-layer principle: each bundled pipeline keeps its OWN modules and
tool versions; tbanalyzer only facilitates execution.

- Vendored MAGMA's complete module set version-isolated under
  modules/local/magma/ (17 custom) + modules/local/magma/nf-core/ (24 nf-core
  module copies at MAGMA's pins) so they never collide with tbanalyzer's
  existing modules/nf-core/{gatk4,samtools,bcftools,fastqc,...}.
- subworkflows/local/magma/ (7), workflows/magmanf.nf (entry workflow MAGMA),
  bin/ (15 MAGMA python scripts), data/magma-references/ (~28MB bundled H37Rv +
  VQSR truth VCFs + GVCF panel + SnpEff DB).
- Include paths rewritten to the magma-scoped layout; pipeline-boilerplate util
  repointed to tbanalyzer's utils_nfcore_tbanalyzer_pipeline; nf-core utils
  subworkflow shared.
- conf/magma_base.config (61 magma_ param defaults, refs -> data/magma-references)
  and conf/magma_modules.config (per-process ext.args/when/publishDir), both
  mode-gated in nextflow.config. test_magma profile + conf/magma_test.config
  (3 downsampled Zenodo samples). main.nf dispatches --mode magma -> MAGMA.
- nextflow_schema.json: magma_options group (61 params) + mode enum adds 'magma'.

Config parses under strict v2 for both modes. NOTE: PIPELINE_INITIALISATION
still validates --input against the mtbseq-oriented schema_input.json; MAGMA's
native 9-column samplesheet needs the samplesheet-handling reconciled next.
Per the mode-permissive samplesheet decision: replace the single required
[sample, fastq_1] with an anyOf so the shared input schema validates BOTH the
mtbseq lowercase shape (sample+fastq_1) and MAGMA's native capitalized shape
(Sample+R1). MAGMA's extra columns stay 'additional' (not added to the parsed
tuple, so mtbseq's [meta,fastq_1,fastq_2] destructure is unaffected); MAGMA's
own samplesheet_validation.py does the real per-row validation from params.input.
…modules)

The mode-gated includeConfig in nextflow.config evaluates params.mode at parse
time (default 'mtbseq'), so -profile test_magma alone didn't trigger the magma
base/modules includes. conf/magma_test.config now includes them explicitly, so
the profile works without also passing --mode magma on the CLI.

Verified via -stub-run -profile test_magma: config loads, the MAGMA-native
samplesheet passes the anyOf schema_input.json validation, and the full MAGMA
DAG compiles/schedules. Execution then hits a PRE-EXISTING upstream port bug
(GATK_MARK_DUPLICATES_DELLY call/signature mismatch, magmanf.nf:410 — identical
in mycobactopia-org/MAGMA feat/plan1-nfcore-modules) from the incomplete nf-core
module migration. Per the don't-modify-pipeline-code principle, that's to be
fixed upstream and re-vendored, not patched here.
Re-syncs the 3 fixes made on mycobactopia-org/MAGMA@nf-core-tbanalyzer:
- magmanf.nf: GATK_MARK_DUPLICATES_DELLY uses SAMTOOLS_MERGE_DELLY.out.bam (was
  .out → expanded to multiple args → 'declares 3 inputs but called with N').
- tbprofiler/collate + utils/eliminate_annotation: def prefix/annotationPrefix
  added to stub blocks.

-stub-run -profile test_magma now compiles the full MAGMA DAG and clears the
GATK error; it then stops at a splitJson() on the SAMPLESHEET_VALIDATION stub's
empty output — a stub-mode limitation (the workflow parses real JSON emitted by
the validator at runtime), so biological validation is via the Docker run.
@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit 86aa9d2

+| ✅ 208 tests passed       |+
#| ❔   6 tests were ignored |#
!| ❗ 103 tests had warnings |!
-| ❌  97 tests failed       |-
Details

❌ Test failures:

  • nextflow_config - Default value from the Nextflow schema params.magma_input_samplesheet = samplesheet.magma.csv`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_vcf_name = joint`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_ref_fasta_basename = NC-000962-3-H37Rv`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_use_ref_gvcf = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_median_coverage = 10.0`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_breadth_of_coverage = 0.9`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_rel_abundance = 0.7`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_ntm_fraction = 0.2`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_site_representation = 0.95`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_cutoff_strand_bias = 0.0`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_only_validate_fastqs = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_merge_analysis = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_variant_recalibration = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_base_recalibration = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_minor_variants_gatk = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_phylogeny_and_clustering = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_complex_regions = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_ntmprofiler = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_tbprofiler_fastq = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_spotyping = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_rdanalyzer = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_skip_ismapper = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_iqtree_standard_bootstrap = True`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_iqtree_fast_ml_only = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_iqtree_fast_bootstrapped_phylogeny = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_iqtree_accurate_ml_only = False`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_snpeff_genome_id = Mycobacterium_tuberculosis_h37rv`` not found in nextflow.config.
  • nextflow_config - Default value from the Nextflow schema params.magma_should_publish_bam = False`` not found in nextflow.config.
  • schema_params - Default value for param magma_input_samplesheet invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_vcf_name invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_basename invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_dir invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_fai invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_dict invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_amb invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_ann invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_bwt invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_pac invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_sa invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_gb invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_fasta_rdanalyzer invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_use_ref_gvcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_gvcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_ref_gvcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_dbsnp_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_dbsnp_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_benavente2015_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_benavente2015_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_coll2014_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_coll2014_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_coll2018_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_coll2018_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_napier2020_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_napier2020_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_walker2015_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_walker2015_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_zeng2018_vcf invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_zeng2018_vcf_tbi invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_excluded_loci_list invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_rrna_list invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_drgenes_list invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_queries_multifasta invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_median_coverage invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_breadth_of_coverage invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_rel_abundance invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_ntm_fraction invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_site_representation invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_cutoff_strand_bias invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_only_validate_fastqs invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_merge_analysis invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_variant_recalibration invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_base_recalibration invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_minor_variants_gatk invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_phylogeny_and_clustering invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_complex_regions invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_ntmprofiler invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_tbprofiler_fastq invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_spotyping invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_rdanalyzer invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_skip_ismapper invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_iqtree_standard_bootstrap invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_iqtree_fast_ml_only invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_iqtree_fast_bootstrapped_phylogeny invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_iqtree_accurate_ml_only invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_snpeff_genome_id invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_snpeff_config_file invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_snpeff_db_dir invalid: Not in pipeline parameters. Check nextflow.config.
  • schema_params - Default value for param magma_should_publish_bam invalid: Not in pipeline parameters. Check nextflow.config.
  • container_configs - conf/containers_conda_lock_files_amd64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_conda_lock_files_arm64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_docker_amd64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_docker_arm64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_singularity_https_amd64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_singularity_https_arm64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_singularity_oras_amd64.config is out of date – please regenerate the container configuration files.
  • container_configs - conf/containers_singularity_oras_arm64.config is out of date – please regenerate the container configuration files.

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)
  • pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in CONTRIBUTING.md: Add any pipeline specific contribution guidelines here, such as coding styles, procedures, checklists etc.
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in gatk4.nf: If in doubt look at other nf-core/modules to see how we are doing things! :)
  • pipeline_todos - TODO string in gatk4.nf: A module file SHOULD only define input and output files as command-line parameters.
  • pipeline_todos - TODO string in gatk4.nf: Software that can be piped together SHOULD be added to separate module files
  • pipeline_todos - TODO string in gatk4.nf: Optional inputs are not currently supported by Nextflow. However, using an empty
  • pipeline_todos - TODO string in gatk4.nf: List required Conda package(s).
  • pipeline_todos - TODO string in gatk4.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in gatk4.nf: Where applicable all sample-specific information e.g. "id", "single_end", "read_group"
  • pipeline_todos - TODO string in gatk4.nf: Where applicable please provide/convert compressed files as input/output
  • pipeline_todos - TODO string in gatk4.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in gatk4.nf: List additional required output channels/values here
  • pipeline_todos - TODO string in gatk4.nf: Where possible, a command MUST be provided to obtain the version number of the software e.g. 1.10
  • pipeline_todos - TODO string in gatk4.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "task.ext.args" directive
  • pipeline_todos - TODO string in gatk4.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
  • pipeline_todos - TODO string in gatk4.nf: Please replace the example samtools command below with your module's command
  • pipeline_todos - TODO string in gatk4.nf: Please indent the command appropriately (4 spaces!!) to help with readability ;)
  • pipeline_todos - TODO string in gatk4.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • schema_params - Schema param magma_input_samplesheet not found from nextflow config
  • schema_params - Schema param magma_vcf_name not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_basename not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_dir not found from nextflow config
  • schema_params - Schema param magma_ref_fasta not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_fai not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_dict not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_amb not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_ann not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_bwt not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_pac not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_sa not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_gb not found from nextflow config
  • schema_params - Schema param magma_ref_fasta_rdanalyzer not found from nextflow config
  • schema_params - Schema param magma_use_ref_gvcf not found from nextflow config
  • schema_params - Schema param magma_ref_gvcf not found from nextflow config
  • schema_params - Schema param magma_ref_gvcf_tbi not found from nextflow config
  • schema_params - Schema param magma_dbsnp_vcf not found from nextflow config
  • schema_params - Schema param magma_dbsnp_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_benavente2015_vcf not found from nextflow config
  • schema_params - Schema param magma_benavente2015_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_coll2014_vcf not found from nextflow config
  • schema_params - Schema param magma_coll2014_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_coll2018_vcf not found from nextflow config
  • schema_params - Schema param magma_coll2018_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_napier2020_vcf not found from nextflow config
  • schema_params - Schema param magma_napier2020_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_walker2015_vcf not found from nextflow config
  • schema_params - Schema param magma_walker2015_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_zeng2018_vcf not found from nextflow config
  • schema_params - Schema param magma_zeng2018_vcf_tbi not found from nextflow config
  • schema_params - Schema param magma_excluded_loci_list not found from nextflow config
  • schema_params - Schema param magma_rrna_list not found from nextflow config
  • schema_params - Schema param magma_drgenes_list not found from nextflow config
  • schema_params - Schema param magma_queries_multifasta not found from nextflow config
  • schema_params - Schema param magma_cutoff_median_coverage not found from nextflow config
  • schema_params - Schema param magma_cutoff_breadth_of_coverage not found from nextflow config
  • schema_params - Schema param magma_cutoff_rel_abundance not found from nextflow config
  • schema_params - Schema param magma_cutoff_ntm_fraction not found from nextflow config
  • schema_params - Schema param magma_cutoff_site_representation not found from nextflow config
  • schema_params - Schema param magma_cutoff_strand_bias not found from nextflow config
  • schema_params - Schema param magma_only_validate_fastqs not found from nextflow config
  • schema_params - Schema param magma_skip_merge_analysis not found from nextflow config
  • schema_params - Schema param magma_skip_variant_recalibration not found from nextflow config
  • schema_params - Schema param magma_skip_base_recalibration not found from nextflow config
  • schema_params - Schema param magma_skip_minor_variants_gatk not found from nextflow config
  • schema_params - Schema param magma_skip_phylogeny_and_clustering not found from nextflow config
  • schema_params - Schema param magma_skip_complex_regions not found from nextflow config
  • schema_params - Schema param magma_skip_ntmprofiler not found from nextflow config
  • schema_params - Schema param magma_skip_tbprofiler_fastq not found from nextflow config
  • schema_params - Schema param magma_skip_spotyping not found from nextflow config
  • schema_params - Schema param magma_skip_rdanalyzer not found from nextflow config
  • schema_params - Schema param magma_skip_ismapper not found from nextflow config
  • schema_params - Schema param magma_iqtree_standard_bootstrap not found from nextflow config
  • schema_params - Schema param magma_iqtree_fast_ml_only not found from nextflow config
  • schema_params - Schema param magma_iqtree_fast_bootstrapped_phylogeny not found from nextflow config
  • schema_params - Schema param magma_iqtree_accurate_ml_only not found from nextflow config
  • schema_params - Schema param magma_snpeff_genome_id not found from nextflow config
  • schema_params - Schema param magma_snpeff_config_file not found from nextflow config
  • schema_params - Schema param magma_snpeff_db_dir not found from nextflow config
  • schema_params - Schema param magma_should_publish_bam not found from nextflow config
  • schema_description - No description provided in schema for parameter: mtbseq_user

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.2
  • Run at 2026-06-24 21:30:21

Replaces the earlier vendoring (from the divergent feat/plan1-nfcore-modules
tip) with the SciVer-validated v0.3.0 release tag, per the inclusion
recommendation. Source: mycobactopia-org/MAGMA branch tbanalyzer-v0.3.0
(v0.3.0 + magma_ namespacing + stub-def fixes).

- v0.3.0 splits the workflow into 16 subworkflows (adds call_wf/map_wf/merge_wf/
  quality_check_wf/reports_wf/structural_variants_analysis_wf/validate_fastqs_wf);
  re-vendored under subworkflows/local/magma with sibling ./ includes preserved.
- modules/local/magma (36) + modules/local/magma/nf-core (23 version-isolated
  copies); conf/magma_base.config + conf/magma_modules.config (96 process scopes)
  + magma_options schema regenerated from v0.3.0; data/magma-references re-synced.
- The GATK_MARK_DUPLICATES_DELLY bug I had to hand-fix on feat/plan1 is ABSENT
  in v0.3.0 (correctly structured there); deferred trio (variantrecalibrator,
  snpsites, select_variants PHYLOGENY) intentionally local — hermetic, no impact.

Verified: both modes parse (strict v2); full MAGMA DAG compiles & schedules
under -stub-run (stops only at splitJson on stub output — stub-mode limitation;
biological validation via Docker).
@abhi18av abhi18av changed the title Add MAGMA as --mode magma (version-isolated; experimental) Add MAGMA as --mode magma (pinned to v0.3.0; version-isolated; experimental) Jun 24, 2026
@abhi18av

Copy link
Copy Markdown
Member Author

Re-based the vendoring onto the pinned MAGMA v0.3.0 tag (f33a913, SciVer-validated 'Plan 1 complete') instead of the divergent feat/plan1-nfcore-modules tip.

  • Source: mycobactopia-org/MAGMA branch tbanalyzer-v0.3.0 (v0.3.0 + magma_ namespacing + stub-def fixes).
  • v0.3.0 splits into 16 subworkflows (adds call_wf/map_wf/merge_wf/quality_check_wf/reports_wf/structural_variants_analysis_wf/validate_fastqs_wf); re-vendored with sibling includes preserved.
  • The GATK_MARK_DUPLICATES_DELLY arg-count bug (hand-fixed on feat/plan1) is absent in v0.3.0. Deferred trio (variantrecalibrator, snpsites, select_variants PHYLOGENY) is intentionally local — hermetic, no impact here.
  • Next milestone to track: v0.4.0 (after Plan 3 Phase 4, MAGMA consumes the XBS subworkflow); SciVer surface expected to shift modestly (PLAN_3 §11.5).

Both modes parse (strict v2); full MAGMA DAG compiles under -stub-run. Biological Docker validation remains the deferred follow-up (draft until green).

abhi18av added 3 commits June 24, 2026 15:49
MAGMA's local modules (python utils + tool wrappers) carry no container
directive — they rely on per-process-name container assignments that lived in
MAGMA's conf/abc_cluster.config (not vendored). On Nomad/Docker this surfaced as
'[NOMAD] Missing container image for process ...:MAGMA:SAMPLESHEET_VALIDATION'.

Adds conf/magma_containers.config (mode-gated + loaded by test_magma) mirroring
MAGMA v0.3.0's validated container layout: magma-container-1 (python utils,
GATK/LOFREQ/FASTQC/MULTIQC/ISMAPPER), magma-container-2 (BWA/SAMTOOLS/IQTREE/
BCFTOOLS/BGZIP/SNPEFF/SNPSITES/CLUSTERPICKER/SNPDISTS), per-tool biocontainers
(SPOTYPING/RDANALYZER/TB-/NTM-PROFILER), and the functional DELLY/BCFTOOLS_VIEW
overrides. Dropped MAGMA's errorStrategy=ignore (so failures surface) and the
seedling-specific gcp-spot nomadOptions.
.gitignore excludes data/, so the 30 data/magma-references files (H37Rv genome
+ BWA indices, VQSR truth VCFs, GVCF panel, SnpEff DB, region lists) were never
committed — the cluster clone lacked them and BWA_MEM_DELLY failed with
'Can't stage file NC-000962-3-H37Rv.fa -- file does not exist'. Force-added
(git add -f), mirroring how data/mtbseq-references is tracked.
NTMPROFILER_PROFILE failed on the downsampled (~11x) test samples on the
cluster. It's an optional contamination QC profiler whose output only feeds
NTMPROFILER_COLLATE -> reports (not the variant/cohort/phylogeny path), so it's
safe to skip for the minimal test — consistent with the already-skipped
spotyping/rdanalyzer/ismapper/tbprofiler_fastq profilers. Full NTMprofiler
coverage remains in MAGMA's own SciVer validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant