RO-crate content appear in the nf-core lint warning if there is a TODO by ningyuxin1999 · Pull Request #3493 · nf-core/tools

ningyuxin1999 · 2025-03-17T20:51:51Z

Changes:

The RO-crate-metadata.json is ignored in linting
Added a test script nf_core/pipelines/lint/rocrate_readme_sync.py for synchronizing the content of ROcrate metadata and the repo README.

The warning message mentioned here disappears when you create a new pipeline without deleting the TODO commands. But I still wonder where I should add the --fix flag so that the use can decide whether they want to synchronize the metadata with the readme file.

PR checklist

This comment contains a description of changes (with reason)
CHANGELOG.md is updated
If you've fixed a bug or added code that should be tested, add tests!
Documentation in docs is updated

…o dev-sync

mirpedrol

leaving a couple of comments, otherwise is looking good!

Co-authored-by: Júlia Mir Pedrol <[email protected]>

mashehu

~~There is some problem with the self-hosted runners atm, so that they don't pick up any jobs. fix is on the way~~
Fixed

codecov · 2025-04-15T15:32:40Z

Codecov Report

Attention: Patch coverage is 69.38776% with 15 lines in your changes missing coverage. Please review.

Project coverage is 76.47%. Comparing base (59f8aff) to head (d74230b).
Report is 1 commits behind head on dev.

Files with missing lines	Patch %	Lines
nf_core/pipelines/lint/rocrate_readme_sync.py	68.75%	15 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mashehu · 2025-04-15T15:40:41Z

can we also add a test for the fix functionality?

mirpedrol

some last suggestions 🙂

mirpedrol · 2025-04-15T15:30:58Z

    - name: nf-core pipelines lint
      shell: bash
-      run: nf-core --verbose --log-file log.txt --hide-progress pipelines lint --dir nf-core-testpipeline --fail-ignored --fail-warned
+      run: nf-core --verbose --log-file log.txt --hide-progress pipelines lint --fix rocrate_readme_sync --dir nf-core-testpipeline --fail-ignored --fail-warned


Was this failing with adding the --fix argument? 🤔 I think we shouldn't add it here, since we want the fresh template to be a valid pipeline

I changed the script so the rocreate_readme_sync function would directly copy the description from the README.md if there's no description key found, therefore there won't be an ignored case added in such situation.

mirpedrol · 2025-04-15T15:31:21Z

    - name: nf-core pipelines lint in release mode
      shell: bash
-      run: nf-core --log-file log.txt --hide-progress pipelines lint --dir nf-core-testpipeline --fail-ignored --fail-warned --release
+      run: nf-core --log-file log.txt --hide-progress pipelines lint --dir nf-core-testpipeline --fix rocrate_readme_sync --fail-ignored --fail-warned --release


same as I mentioned before

mirpedrol · 2025-04-15T15:31:44Z

can be removed

Co-authored-by: Júlia Mir Pedrol <[email protected]>

ningyuxin1999 · 2025-04-16T12:54:47Z

can we also add a test for the fix functionality?

Yes, I added a new test into the test script.

mirpedrol

This is what I would do with the description, it adds it when we use the flag --fix, and the test is ignored if we don't have a description. The CI test should pass, but showt if it doesn't and I will have another look.

mirpedrol · 2025-04-16T15:42:41Z

+        if "description" not in graph[0]:
+            graph[0]["description"] = readme_content
+            passed.append(
+                "No description found in `ro-crate-metadata.json`, add the same description from `README.md` to the RO-Crate metadata."
+            )


Suggested change

if "description" not in graph[0]:

graph[0]["description"] = readme_content

passed.append(

"No description found in `ro-crate-metadata.json`, add the same description from `README.md` to the RO-Crate metadata."

)

if "description" not in graph[0]:

ignored.append(

"No description found in `ro-crate-metadata.json`."

)

mirpedrol · 2025-04-16T15:45:12Z

+    if readme_content != rc_description_graph:
+        # If the --fix flag is set, you could overwrite the RO-Crate description with the README content:
+        if "rocrate_readme_sync" in self.fix:
+            metadata_dict.get("@graph")[0]["description"] = readme_content


Suggested change

metadata_dict.get("@graph")[0]["description"] = readme_content

if "description" not in graph[0]:

graph[0]["description"] = readme_content

fixed.append(

"Fixed: add the same description from `README.md` to the RO-Crate metadata."

)

return {"passed": passed, "failed": failed, "ignored": ignored, "fixed": fixed, "could_fix": could_fix}

metadata_dict.get("@graph")[0]["description"] = readme_content

mirpedrol · 2025-04-16T15:55:15Z

+        with open(json_path, "w") as f:
+            f.write('{ "@graph": [{"description": "This is a test script"}] }')


To parse the file as a JSON you can use the library json as you did in nf_core/pipelines/lint/rocrate_readme_sync.py. This code is untested so make sure that I didn't make any mistake before accepting the suggestion 😄

Suggested change

with open(json_path, "w") as f:

f.write('{ "@graph": [{"description": "This is a test script"}] }')

with open(json_path, "rw") as f:

try:

rocrate = json.load(fh)

except json.JSONDecodeError as e:

raise UserWarning(f"Unable to load JSON file '{json_path}' due to error {e}")

rocrate["@graph"][0]["description"] = "This is a test script"

json.dump(rocrate, fh, indent=4)

Thanks Julia!
The tests are still failing. it seems like in the git actions, the rocrate created a metadata file without the description key.In this case, when the --fix is not provided, the tests will anyway fail all the ignored tests.

Co-authored-by: Júlia Mir Pedrol <[email protected]>

mashehu · 2025-04-24T10:31:44Z

okay, I found why the CI is failing. When we remove the todo statements in https://github.com/nf-core/tools/blob/main/.github/actions/create-lint-wf/action.yml#L52 we remove the whole line. which also removed the description part. I changed it to only remove the specific words

concerning fefac22: the ro-crates are genereated during the nf-core pipelines create command, so there was no need to manually create them again

ningyuxin1999 added 2 commits March 17, 2025 21:34

rocrate readme sync

f9ab845

merge branch 'main' into dev-sync

e2e5285

ningyuxin1999 requested review from JulianFlesch, ewels, mashehu and mirpedrol March 17, 2025 20:51

ningyuxin1999 self-assigned this Mar 17, 2025

ningyuxin1999 and others added 15 commits March 24, 2025 11:22

final test done

a728a47

Merge branch 'dev' into dev-sync

7b23550

failing tests

4989b14

Merge remote-tracking branch 'upstream/dev' into dev-sync

51e183c

Merge branch 'dev-sync' of https://github.com/ningyuxin1999/tools int…

70cc699

…o dev-sync

fix the script

c76713d

api update

409f75c

prettier

e76ec56

Merge branch 'dev' into dev-sync

eb0dd5b

test

bcd16e1

format

36a75ea

testscript

0e04e7f

raise error when script fails

21c5c8d

Merge branch 'dev' into dev-sync

3f98c1d

json error

338f2c1

ningyuxin1999 added this to Hackathon March 2025 Mar 25, 2025

github-project-automation Bot moved this to To do in Hackathon March 2025 Mar 25, 2025

mirpedrol reviewed Mar 25, 2025

View reviewed changes

ningyuxin1999 and others added 5 commits March 25, 2025 15:40

Update nf_core/pipelines/lint/pipeline_todos.py

cd83b0d

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

1699e58

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

7949ccf

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

1f3d232

Co-authored-by: Júlia Mir Pedrol <[email protected]>

comments

8a7791c

mashehu reviewed Apr 15, 2025

View reviewed changes

Comment thread tests/pipelines/lint/test_rocrate_readme_sync.py Outdated

mashehu reviewed Apr 15, 2025

View reviewed changes

Merge branch 'dev' into dev-sync

7c29f9b

mirpedrol reviewed Apr 15, 2025

View reviewed changes

ningyuxin1999 and others added 6 commits April 16, 2025 14:14

Update nf_core/pipelines/lint/rocrate_readme_sync.py

43df8a3

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

644553a

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

43f030d

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

ec9cbc4

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Update nf_core/pipelines/lint/rocrate_readme_sync.py

5e6eb12

Co-authored-by: Júlia Mir Pedrol <[email protected]>

Merge branch 'dev' into dev-sync

108f659

ningyuxin1999 added 2 commits April 16, 2025 14:59

add test, remove --fix in git action

146e68b

add description directly when there is no description key found

7e26eec

mirpedrol reviewed Apr 16, 2025

View reviewed changes

ningyuxin1999 and others added 7 commits April 17, 2025 11:34

restructure

1ffc229

test

4df9af0

Merge branch 'dev' into dev-sync

4713d10

Update tests/pipelines/lint/test_rocrate_readme_sync.py

975f834

Co-authored-by: Júlia Mir Pedrol <[email protected]>

json load in test scripts

5d8045a

no need add the ro-crate manually

fefac22

fix sed command to handle TODO statements inside singe-line json entries

6f3bb3b

mashehu and others added 2 commits April 24, 2025 13:25

Merge branch 'dev' into dev-sync

5b2f18f

Merge branch 'dev' into dev-sync

d74230b

ningyuxin1999 merged commit 463dbc7 into nf-core:dev Apr 28, 2025
97 checks passed

github-project-automation Bot moved this from In progress to Done in Hackathon March 2025 Apr 28, 2025

ningyuxin1999 deleted the dev-sync branch July 9, 2025 07:42

edmundmiller mentioned this pull request Jul 14, 2025

Initialize Claude Code #3678

Closed

		with open(json_path, "w") as f:
		f.write('{ "@graph": [{"description": "This is a test script"}] }')

-        with open(json_path, "w") as f:
-            f.write('{ "@graph": [{"description": "This is a test script"}] }')
+        with open(json_path, "rw") as f:
+            try:
+                rocrate = json.load(fh)
+            except json.JSONDecodeError as e:
+                raise UserWarning(f"Unable to load JSON file '{json_path}' due to error {e}")
+            rocrate["@graph"][0]["description"] = "This is a test script"
+            json.dump(rocrate, fh, indent=4)

Conversation

ningyuxin1999 commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes:

PR checklist

Uh oh!

mirpedrol left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mashehu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mashehu commented Apr 15, 2025

Uh oh!

mirpedrol left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ningyuxin1999 commented Apr 16, 2025

Uh oh!

mirpedrol left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mashehu commented Apr 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ningyuxin1999 commented Mar 17, 2025 •

edited

Loading

mashehu left a comment •

edited

Loading

codecov Bot commented Apr 15, 2025 •

edited

Loading