Skip to content

Misc output cleanup#226

Merged
muffato merged 5 commits into
devfrom
mm49_output_cleanup
Jun 10, 2026
Merged

Misc output cleanup#226
muffato merged 5 commits into
devfrom
mm49_output_cleanup

Conversation

@muffato

@muffato muffato commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member
  • Removed the --write_to_portal and --genome_notes_api options as it is now certain the data will be pulled from UGN rather than pushed to UGN.
    • Some more cleanup to follow as we keep reorganising the pipelines.
  • Renamed the modules/local/createtable.nf output because GCA*.csv was far too generic
  • Compress the .only.bed output as it can get big (and following our convention)

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@muffato muffato requested a review from DLBPointon June 4, 2026 07:17
@muffato muffato self-assigned this Jun 4, 2026
@sanger-tolsoft

sanger-tolsoft commented Jun 4, 2026

Copy link
Copy Markdown

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.5.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@github-actions

github-actions Bot commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 6aad68a

+| ✅ 221 tests passed       |+
#| ❔  23 tests were ignored |#
!| ❗  23 tests had warnings |!
Details

❗ Test warnings:

  • files_exist - File not found: conf/igenomes_ignored.config
  • nextflow_config - Config manifest.version should end in dev: 2.2.0
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline 2. Present QC for raw reads (MultiQC)
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_if_empty_null - ifEmpty(null) found in /home/runner/work/genomenote/genomenote/subworkflows/local/combine_note_data/main.nf: _ versions = ch_versions.ifEmpty(null) // channel: [versions.yml]
    _
  • pipeline_if_empty_null - ifEmpty(null) found in /home/runner/work/genomenote/genomenote/subworkflows/local/genome_metadata/main.nf: _ versions = ch_versions.ifEmpty(null) // channel: [versions.yml]
    _
  • schema_lint - Schema 'description' should be 'Creating standardised genome assembly publications'
    Found: 'Creating standarised genome assembly publications'
  • local_component_structure - createtable.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - run_wget.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - generate_higlass_link.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - fetch_gbif_metadata.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - combine_statistics_and_metadata.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - params_check.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - upload_higlass_data.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - combine_metadata.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - extract_annotation_statistics_info.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - populate_template.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - parse_metadata.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - fetch_ensembl_metadata.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure

❔ Tests ignored:

  • files_exist - File is ignored: assets/nf-core-genomenote_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-genomenote_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-genomenote_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: conf/igenomes.config
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File does not exist: assets/nf-core-genomenote_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-genomenote_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-genomenote_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/genomenote/genomenote/.github/workflows/awstest.yml
  • template_strings - template_strings

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-06-10 13:31:12

@muffato

muffato commented Jun 4, 2026

Copy link
Copy Markdown
Member Author

assets/mapping_taxids-busco_dataset_name.eukaryota_odb10.2019-12-16.txt was somehow truncated. I've restored it.

DLBPointon
DLBPointon approved these changes Jun 8, 2026
View reviewed changes

@DLBPointon DLBPointon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@muffato muffato force-pushed the mm49_output_cleanup branch from 4a1c771 to c445ef6 Compare June 10, 2026 10:42
@muffato

muffato commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

Following our discussion on Slack I've reverted 660e4a6 where I had extended the BUSCO mappings to all clades. It's still eukaryotes only, named eukaryota, with matches across the git history.
I think there's no need to support prokaryotes here, especially as we're trying to offload running BUSCO.

@DLBPointon

DLBPointon commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

I've been noticing this alot recently


era_odb10.Merian_elements_full_table.bu era_odb10.Merian_elements_full_table.bu
  N E X T F L O W  ~  version 25.04.0
scopainter.png:md5,cdd73e334318cf915df7 scopainter.png:md5,cdd73e334318cf915df7
  Launching `/home/runner/work/genomenote/genomenote/tests/../main.nf` [confident_bohr] DSL2 - revision: 26aae0a5ce
cf5fa750892a",                          cf5fa750892a",                         
  Downloading plugin nf-schema@2.5.1
- 298         "GCA_963859965.1.blob.png                                        
  
:md5,147e7413247cbb8f52caae9c7b79400b",                                        
  ------------------------------------------------------
- 299         "GCA_963859965.1.blob_chr                                        
     _____                                _______    _
.png:md5,147e7413247cbb8f52caae9c7b7940                                        
    / ____|                              |__   __|  | |
   | (___   __ _ _ __   __ _  ___ _ __  ___ | | ___ | |
0b",                                                                           
    \___ \ / _` | '_ \ / _` |/ _ \ '__||___|| |/ _ \| |
- 300         "GCA_963859965.1.grid.png                                        
    ____) | (_| | | | | (_| |  __/ |        | | (_) | |____
:md5,1dd41fe93a5644b64817f8ceec461a5f",                                        
   |_____/ \__,_|_| |_|\__, |\___|_|        |_|\___/|______|
- 301         "GCA_963859965.1.grid_chr                                        
                        __/ |
.png:md5,1dd41fe93a5644b64817f8ceec461a                                        
                       |___/
5f",                                                                           
    sanger-tol/genomenote 2.2.0
  302         "GCA_963859965.1.lepidopt   289         "GCA_963859965.1.lepidopt
  ------------------------------------------------------
era_odb12.full_table.tsv:md5,6d8e1467c2 era_odb12.full_table.tsv:md5,6d8e1467c2
          
4c90a0b1a2c4de9a5a45d5",                4c90a0b1a2c4de9a5a45d5",               
  Input/output options

Multiple job outputs getting mixed up, i'm assuming it's a github thing we won't be able to fix. Just noting it.

@muffato

muffato commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

I don't understand why BLOBTK_PLOT doesn't run any more – I've even confirmed the behaviour on the farm !
Looking into it

@muffato

muffato commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

It's a BTK error, due to the module ignoring errors:

ERROR: File not found: https://blobtoolkit.genomehubs.org/api/v1/dataset/id/CALUEP01/meta.json

    > [39/0560ca] NOTE: Process `SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.blob_chr)` terminated with an error exit status (1) -- Error is ignored
    > [e4/9afbfa] Submitted process > SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.grid_chr)
    > [1c/a766f1] Submitted process > SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.grid)
    > [ac/4e7042] NOTE: Process `SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.blob)` terminated with an error exit status (1) -- Error is ignored
    > [e4/9afbfa] NOTE: Process `SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.grid_chr)` terminated with an error exit status (1) -- Error is ignored
    > [18/60c586] Submitted process > SANGERTOL_GENOMENOTE:GENOMENOTE:CONTACT_MAPS:GET_CHROMLIST (GCA_963859965.1)
    > [1c/a766f1] NOTE: Process `SANGERTOL_GENOMENOTE:GENOMENOTE:GET_BLOBTK_PLOTS:BLOBTK_PLOT (GCA_963859965.1.grid)` terminated with an error exit status (1) -- Error is ignored

Maybe caused by the web issues at the Sanger this morning

@muffato

muffato commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

The VM was down, indeed.

I've pushed the same module patch as in sanger-tol/blobtoolkit, to not set the error strategy to ignore.

The test passes locally.

@DLBPointon

DLBPointon commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Looks/sounds good to me! Still an approve

@muffato muffato merged commit a546b2a into dev Jun 10, 2026
7 checks passed
@muffato muffato deleted the mm49_output_cleanup branch June 10, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DLBPointon DLBPointon DLBPointon approved these changes

Assignees

@muffato muffato

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@muffato @sanger-tolsoft @DLBPointon