Add WDL parser to Janis #69

illusional · 2021-03-15T22:06:56Z

Adds:

"foreach" which is similar to scatter. Perform some operation prior to scattering - this is the analogue to how WDL performs "scattering", and has greater flexibility for doing things like allowing you to iterate over a range of values.
- this is implemented for CWL and WDL, and contains a small test.
to_python for every operator. This isn't exactly useful for this PR (it will be for my future Hail Batch PR), but it was difficult to separate.

Known limitations:

Doesn't support structs, maps, or objects
Some methods don't have Janis equivalents (read_* and write_*, cross, dot)
Not sure how it handles expression placeholder options
Doesn't support nested scatters
Setting variables in the call-space in workflow execution is not well supported.

codecov-io · 2021-03-15T22:08:39Z

Codecov Report

Merging #69 (3d5bcea) into master (bcbc1da) will decrease coverage by 1.53%.
The diff coverage is 28.13%.

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
- Coverage   69.12%   67.58%   -1.54%     
==========================================
  Files          78       79       +1     
  Lines       10282    10697     +415     
==========================================
+ Hits         7107     7230     +123     
- Misses       3175     3467     +292

Impacted Files	Coverage Δ
janis_core/ingestion/fromwdl.py	`0.00% <0.00%> (ø)`
janis_core/operators/stringformatter.py	`83.82% <16.66%> (-3.10%)`	⬇️
janis_core/operators/logical.py	`65.65% <34.78%> (-1.39%)`	⬇️
janis_core/operators/standard.py	`55.69% <45.07%> (-3.14%)`	⬇️
janis_core/operators/selectors.py	`69.53% <52.17%> (-1.39%)`	⬇️
janis_core/operators/operator.py	`65.34% <53.33%> (-0.82%)`	⬇️
janis_core/translations/cwl.py	`77.26% <70.83%> (-0.20%)`	⬇️
janis_core/workflow/workflow.py	`47.86% <71.42%> (-0.08%)`	⬇️
janis_core/tests/test_operators.py	`98.80% <100.00%> (ø)`
janis_core/tests/test_translation_cwl.py	`99.53% <100.00%> (+<0.01%)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bcbc1da...3d5bcea. Read the comment docs.

junyk

@janis-bot

junyk

@janis-bot

illusional · 2021-05-18T22:45:15Z

Hey @junyk, is there anything outstanding on this PR? This isn't packaged in a complete catch-all script, nor is it "production-ready" (it probably doesn't gracefully handle errors).

I have a few ideas on how we could add some of the "read_*" operators into Janis, but it's a little bit of work.

junyk · 2021-05-25T02:14:20Z

@janis-bot

illusional · 2021-05-25T03:26:22Z

Hey @junyk, is it possible to get more information about the tests? I see some are failing, some are passing, keen to know if it's this PR which is causing something to go astray.

junyk · 2021-05-25T03:43:56Z

Hey @illusional , sorry I haven't responded cause I am still trying to figure out what caused the failures (it could be environment). I'll get back to you!

junyk

@janis-bot try again!

junyk

@janis-bot

junyk

@janis-bot

…anis-core into add-from-wdl

codecov-commenter · 2021-05-27T05:15:39Z

Codecov Report

Merging #69 (7d8b7b3) into master (e74e6fe) will decrease coverage by 1.75%.
The diff coverage is 28.07%.

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
- Coverage   69.11%   67.36%   -1.76%     
==========================================
  Files          78       79       +1     
  Lines       10320    10789     +469     
==========================================
+ Hits         7133     7268     +135     
- Misses       3187     3521     +334

Impacted Files	Coverage Δ
janis_core/ingestion/fromwdl.py	`0.00% <0.00%> (ø)`
janis_core/operators/stringformatter.py	`84.44% <20.00%> (-2.48%)`	⬇️
janis_core/operators/logical.py	`65.65% <34.78%> (-1.39%)`	⬇️
janis_core/operators/standard.py	`54.70% <46.00%> (-4.13%)`	⬇️
janis_core/operators/selectors.py	`69.18% <50.00%> (-1.75%)`	⬇️
janis_core/operators/operator.py	`65.34% <53.33%> (-0.82%)`	⬇️
janis_core/translations/cwl.py	`77.26% <70.83%> (-0.20%)`	⬇️
janis_core/workflow/workflow.py	`48.45% <71.42%> (-0.08%)`	⬇️
janis_core/tests/test_operators.py	`98.80% <100.00%> (ø)`
janis_core/tests/test_translation_cwl.py	`99.53% <100.00%> (+<0.01%)`	⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e74e6fe...7d8b7b3. Read the comment docs.

junyk

@janis-bot

illusional · 2021-05-27T09:19:42Z

Oooh, this is pretty! Just need to update the title of the page :p

janis-bot · 2021-05-27T19:23:14Z

Test Failed for group 60af2b627eeab92ee27259f1.

Bioinformatics Tools:

ParseFastqcAdaptors.None (Nectar - cromwell): Test Succeeded
ParseFastqcAdaptors.None (Spartan - cromwell): Test Succeeded
SamToolsFlagstat.None (Nectar - cromwell): Test Succeeded
SamToolsFlagstat.None (Spartan - cromwell): Test Succeeded
Gatk4HaplotypeCaller.None (Nectar - cromwell): Test Succeeded
Gatk4HaplotypeCaller.None (Spartan - cromwell): Test Succeeded
performanceSummary.None (Nectar - cromwell): Test Succeeded
performanceSummary.None (Spartan - cromwell): Test Succeeded

WGSGermlineGATK:

WGSGermlineGATK.brca1 (Spartan - cromwell): Test Failed 60af2b627eeab92ee27259fa

Small Bioinformatics Workflow:

bwaaligner.None (Nectar - cromwell): Test Succeeded
bwaaligner.None (Nectar - cwltool): Test Succeeded

illusional · 2021-05-31T00:29:14Z

Thanks for the report @junyk! I tracked it down to a double unwrap of the basename function introduced in 260a02c, and resolved in 0ee5a82. Great thing we had these tests!

junyk · 2021-05-31T00:43:36Z

@illusional thanks for looking into it, Michael! I will run the tests one more time :)

junyk

@janis-bot

janis-bot · 2021-05-31T03:30:41Z

Test Failed for group 60b4314f7423dc52134f1c56.

Bioinformatics Tools:

ParseFastqcAdaptors.None (Nectar - cromwell): Test Succeeded
ParseFastqcAdaptors.None (Spartan - cromwell): Test Failed 60b4314f7423dc52134f1c58
SamToolsFlagstat.None (Nectar - cromwell): Test Succeeded
SamToolsFlagstat.None (Spartan - cromwell): Test Succeeded
Gatk4HaplotypeCaller.None (Nectar - cromwell): Test Succeeded
Gatk4HaplotypeCaller.None (Spartan - cromwell): Test Succeeded
performanceSummary.None (Nectar - cromwell): Test Succeeded
performanceSummary.None (Spartan - cromwell): Test Succeeded

WGSGermlineGATK:

WGSGermlineGATK.brca1 (Spartan - cromwell): Test Succeeded

Small Bioinformatics Workflow:

bwaaligner.None (Nectar - cromwell): Test Succeeded
bwaaligner.None (Nectar - cwltool): Test Succeeded

junyk · 2021-06-03T05:00:49Z

I have manually re-tested ParseFastqcAdaptors , it works. Merging this now.

junyk · 2021-06-03T05:01:02Z

thanks @illusional !

mr-c · 2021-08-30T13:02:26Z

@illusional How do I test this? I installed the latest janis-core and janis repositories locally but janisdk fromwdl doesn't work for me

illusional · 2021-09-01T12:34:27Z

Hey @mr-c, I think you've figured it out already, but you have to run the script directly. I've opened a PR to add it to janisdk: PMCC-BioinformaticsCore/janis#58

illusional added 11 commits March 15, 2021 12:56

Add fromwdl ingestor

67fec4a

Add ForEachSelector

7a51594

Add _foreach to workflow (unimplemented) - required for WDL ingestion

ad9762b

Add WDL implementation for "foreach"

c61df3f

Add to_python for all operators

260a02c

Move node.returntype modifiers to nodes + foreach modifier

1e3a289

Fix minor issue with WDL foreach implementation

0579c58

Don't bind defaults to inputs.json if it's a selector

573177d

Format with Black

53d9246

Implement ForEach on CWL

54fe032

Add tests for ForEach

3d5bcea

illusional mentioned this pull request Mar 15, 2021

Toy implementation for WDL -> Janis parser #68

Closed

illusional requested a review from junyk March 15, 2021 22:07

illusional mentioned this pull request Mar 16, 2021

Add Hail batch + translator base refactor #70

Open

junyk mentioned this pull request Apr 23, 2021

documentation on mapping WDL concepts to CWL concepts PMCC-BioinformaticsCore/janis#46

Open

junyk reviewed Apr 26, 2021

View reviewed changes

mr-c mentioned this pull request Apr 26, 2021

compare our results with how Janis does WDL→CWL conversions common-workflow-lab/wdl-cwl-translator#6

Open

1 task

junyk reviewed Apr 29, 2021

View reviewed changes

junyk reviewed May 25, 2021

View reviewed changes

junyk reviewed May 26, 2021

View reviewed changes

junyk reviewed May 27, 2021

View reviewed changes

Merge branch 'master' of https://github.com/PMCC-BioinformaticsCore/j…

b506638

…anis-core into add-from-wdl

junyk reviewed May 27, 2021

View reviewed changes

illusional added 8 commits May 31, 2021 07:59

Remove inbuilt string formatting because it's too smart

1decd6b

Add replace operator

d7dbd70

Fix basename + add select_all operators to fromwdl

c801850

Improve bracket matching to not include ${*} placeholders

8033b01

Improve FilterNullOperator output type detection

911f89b

Add error boundary to WDL parser

fa12bd0

WDL parser bug fixes

e6772fe

Fix double unwrapped WDL implementation of basename operator

0ee5a82

Update fromwdl.py to read file path from sys.argv

7d8b7b3

illusional mentioned this pull request May 31, 2021

Question: Is there a way to use your tool to convert WDL to CWL format #76

Closed

junyk reviewed May 31, 2021

View reviewed changes

junyk merged commit ef1ba7b into master Jun 3, 2021

illusional mentioned this pull request Aug 18, 2021

Missing .gitmodules? PMCC-BioinformaticsCore/janis#54

Open

Add WDL parser to Janis #69

Add WDL parser to Janis #69

Uh oh!

Conversation

illusional commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Mar 15, 2021

Codecov Report

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

illusional commented May 18, 2021

Uh oh!

junyk commented May 25, 2021

Uh oh!

illusional commented May 25, 2021

Uh oh!

junyk commented May 25, 2021

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented May 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

illusional commented May 27, 2021

Uh oh!

janis-bot commented May 27, 2021

Uh oh!

illusional commented May 31, 2021

Uh oh!

junyk commented May 31, 2021

Uh oh!

junyk left a comment

Choose a reason for hiding this comment

Uh oh!

janis-bot commented May 31, 2021

Uh oh!

junyk commented Jun 3, 2021

Uh oh!

junyk commented Jun 3, 2021

Uh oh!

mr-c commented Aug 30, 2021

Uh oh!

illusional commented Sep 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

illusional commented Mar 15, 2021 •

edited

Loading

codecov-commenter commented May 27, 2021 •

edited

Loading