Implement wave container commands #3954

JulianFlesch · 2025-12-02T11:13:05Z

Closes #3952

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

…anFlesch/nf-core-tools into feature/3952-create-waveyml

…t instead of printing. simplify checks

…le path

…/nf-core-tools into feature/3952-create-waveyml

… add dry_run arg

nf_core/components/nfcore_component.py

…/nf-core-tools into feature/3952-create-waveyml

…if they exist.

nf_core/components/nfcore_component.py

nf_core/modules/containers.py

mashehu · 2025-12-05T11:04:04Z

nf_core/modules/containers.py

+        ]
+        return containers_flat
+
+    def _resolve_module_dir(self, module: str | Path) -> Path:


do we not have this funciton already somehwere else?

I might be overlooking something, but not really ... As far as I can tell, we have ModulesInfo, which was used at some point, but it never provides the environment.yml or meta.yml paths, or confirms their existance.

Maybe this should be in ModulesInfo?

mashehu · 2025-12-05T11:04:39Z

nf_core/modules/containers.py

+        return module_dir
+
+    @staticmethod
+    def _environment_path(module_dir: Path) -> Path:


because we lint for this, I think we should be able to reuse something there.

mashehu · 2025-12-05T11:06:50Z

nf_core/utils.py



+@contextmanager
+def set_wd_tempdir() -> Generator[None, None, None]:


this function is for sure created somewhere else already.

Yes, In pipeline downloads! But that version throws Download-specific errors ...
The plan was move it out of the utils.py over there into the general utils.
But now we don't use it at all anymore ...

Should we remove it again?

…f containers section missing

mashehu · 2025-12-05T14:38:43Z

nf_core/modules/containers.py

+    IMAGE_KEY = "name"
+    BUILD_ID_KEY = "buildId"
+    SCAN_ID_KEY = "scanId"


I don't think any of these will change in the future. clearer if we just use them as strings in the code I think.

If you don't feel strongly about this, I think we should keep them. They only exist in the class scope, close to where they are being used, but there we use them quite a lot as it helps us to not have redundant literals

mashehu · 2025-12-05T14:44:19Z

nf_core/modules/containers.py

+            for platform in CONTAINER_PLATFORMS:
+                containers[cs][platform] = self.request_container(cs, platform, env_path, await_, dry_run)
+
+        for platform in CONTAINER_PLATFORMS:


do we need to loop twice over container_platforms?

I thought it makes it more readable what's being done with being less nested and clearly separated. But for comparing, applied your suggestion in af97d90

nf_core/modules/containers.py

Fix typo Co-authored-by: Matthias Hörtenhuber <[email protected]>

JulianFlesch · 2025-12-09T13:27:54Z

Test still missing ( @ningyuxin1999 )
Linting still missing
Sorting the resulting meta.yml after running create() still missing
Refactoring ModuleInfo to be helpful here -> TODO: ISSUE
Decision required: Behavior if modules don't have meta.yml and/or environment.yml

WIP: Start working on wave commands: Add command dummies.

cb07e9d

ningyuxin1999 self-assigned this Dec 2, 2025

[automated] Update CHANGELOG.md

aa40d8b

JulianFlesch changed the title ~~switch pre-commit to prek for development~~ Implement wave container commands Dec 2, 2025

ningyuxin1999 and others added 20 commits December 2, 2025 13:30

added architecture

6b73c3c

Merge branch 'dev' into feature/3952-create-waveyml

5819387

wire commands with functions

e689754

Merge branch 'feature/3952-create-waveyml' of https://github.com/Juli…

c8873db

…anFlesch/nf-core-tools into feature/3952-create-waveyml

containers obj

8328f6f

Merge branch 'dev' into feature/3952-create-waveyml

6fba8ae

Update resolve module_dir. Change to using ValueError

78358ae

Add TODOs

4493cad

WIP: add container listing

1915c15

Move container constants to global utils.

109d0b8

Use constants from global utils. Change list_containers to return lis…

e723d5b

…t instead of printing. simplify checks

add dry-run to the containers create, fixed the uitls import and modu…

f44757b

…le path

Add contextmanager that extends set_wd but changes into a tempdir

7c369a4

Add extracting container string from a module main.nf

81ddaf8

Rename method to match field name

882ed67

Print container info as richt table

aa9fa01

Merge branch 'feature/3952-create-waveyml' of github.com:JulianFlesch…

3e119a4

…/nf-core-tools into feature/3952-create-waveyml

Move calling wave to ModuleContainers class. Finish create method and…

11228d4

… add dry_run arg

Fix output encoding

c08a6dd

Remove comment

c6e8360

mashehu reviewed Dec 4, 2025

View reviewed changes

nf_core/components/nfcore_component.py Show resolved Hide resolved

JulianFlesch and others added 5 commits December 4, 2025 18:14

Add TODOs

4317977

buildid and scanid saved to the dict

e1d7f7d

Merge branch 'feature/3952-create-waveyml' of github.com:JulianFlesch…

6cd3380

…/nf-core-tools into feature/3952-create-waveyml

meta yaml

180287e

Update calling wave command to get buildId/scanId. Only add to outpu …

d6e744f

…if they exist.

Simplify create method

ad5463c

mashehu reviewed Dec 5, 2025

View reviewed changes

JulianFlesch added 4 commits December 5, 2025 12:17

Refactor to reuse key constants.

e0edf97

Add conda-lock information to containers dict

141c2b8

Fix space in url

972cd5e

Implement conda lock functions. Do not fail in containers_from_meta i…

78bd273

…f containers section missing

mashehu reviewed Dec 5, 2025

View reviewed changes

mashehu approved these changes Dec 5, 2025

View reviewed changes

JulianFlesch and others added 17 commits December 5, 2025 16:10

Simplify class

ad640fa

Implement updating meta - WIP: sorting

9b821e7

Fix ModuleContainers initialisation

f117fca

Fix expected meta.yml path

d0f42cc

Update containers in meta.yml - WIP: sorting

3ccd190

update container from meta method

cf1dcfa

Review: Rename conda lock methods

76c64f2

Fix typo

6a69fb2

Update nf_core/modules/containers.py

ff23c7d

Fix typo Co-authored-by: Matthias Hörtenhuber <[email protected]>

Review command: Remove dry_run flag for module container creation#

b7e5102

remove unnecessary string casting

2fa3538

Review comment: Remove extra loop for conda-lock files

af97d90

Merge

77e263e

Make wave container requests run concurrently

8f3bfb1

fix gathering concurrent results

3221030

fix building conda section

6aca629

Remove threads option.

69b6424



		@contextmanager
		def set_wd_tempdir() -> Generator[None, None, None]:

Implement wave container commands #3954

Are you sure you want to change the base?

Implement wave container commands #3954

Uh oh!

Conversation

JulianFlesch commented Dec 2, 2025

PR checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JulianFlesch commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JulianFlesch commented Dec 9, 2025 •

edited

Loading