added WDL2CWL conversion examples#80
Closed
ambarishK wants to merge 83 commits intocommon-workflow-library:releasefrom
Closed
added WDL2CWL conversion examples#80ambarishK wants to merge 83 commits intocommon-workflow-library:releasefrom
ambarishK wants to merge 83 commits intocommon-workflow-library:releasefrom
Conversation
mr-c
requested changes
May 14, 2020
Member
mr-c
left a comment
mr-c
left a comment
There was a problem hiding this comment.
Thank you @ambarishK for starting this. The layout needs to match the repository layout, so please pull everything up from the WDL2CWL_example. Did you test the conversion from WDL?
| @@ -0,0 +1,58 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
| @@ -0,0 +1,46 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
Author
There was a problem hiding this comment.
ngsap2@ubuntu:~$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":"$PWD" -w="$PWD" commonworkflowlanguage/cwltool Ambarish_Kumar_SOP-GATK-SAR-CoV-2.cwl Ambarish_Kumar_SOP-GATK-SAR-CoV-2_inputs.yml
INFO /usr/local/bin/cwltool 2.0.20200107113851
INFO Resolved 'Ambarish_Kumar_SOP-GATK-SAR-CoV-2.cwl' to 'file:///home/ngsap2/Ambarish_Kumar_SOP-GATK-SAR-CoV-2.cwl'
tools/picard/picard_MarkDuplicates.cwl:24:3: object id `tools/picard/picard_MarkDuplicates.cwl#alignments` previously defined
ERROR Tool definition failed validation:
Ambarish_Kumar_SOP-GATK-SAR-CoV-2.cwl:83:12: Workflow step output 'bam_duprem' does not correspond
to
tools/picard/picard_MarkDuplicates.cwl:101:1: tool output (expected 'alignments', 'log',
'metrics')
Ambarish_Kumar_SOP-GATK-SAR-CoV-2.cwl:81:5: Step is missing required parameters 'alignments',
'alignments_are_sorted', 'remove_duplicates'
```
| @@ -0,0 +1,58 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
| @@ -0,0 +1,58 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
| @@ -0,0 +1,54 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
Author
There was a problem hiding this comment.
ngsap2@ubuntu:~$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":"$PWD" -w="$PWD" commonworkflowlanguage/cwltool /home/ngsap2/tools/bowtie2/bowtie2_build.cwl -c /home/ngsap2/testdata16march/ebola_ref.fasta --bt2_index_base eb
INFO /usr/local/bin/cwltool 2.0.20200107113851
INFO Resolved '/home/ngsap2/tools/bowtie2/bowtie2_build.cwl' to 'file:///home/ngsap2/tools/bowtie2/bowtie2_build.cwl'
ERROR testdata16march/ebola_ref.fasta:1:1: while scanning a block scalar
testdata16march/ebola_ref.fasta:1:2: expected chomping or indentation indicators, but found 'g'
| @@ -0,0 +1,42 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
Please use existing tool descriptions, such as https://github.com/common-workflow-library/bio-cwl-tools/blob/release/samtools/samtools_index.cwl
| "arguments": [ | ||
| { | ||
| "shellQuote": false, | ||
| "valueFrom": "samtools index $(inputs.inputBAM.path)samtools mpileup -B -f $(inputs.ReferenceGenome.path) $(inputs.inputBAM.path) > $(inputs.sampleName).mpileup" |
Member
There was a problem hiding this comment.
Did you test this? I think a | is missing.
wdl2cwl is a very old tool, I'm not sure if it ever worked.
| @@ -0,0 +1,42 @@ | |||
| #!/usr/bin/env cwl-runner | |||
Member
There was a problem hiding this comment.
Member
|
Okay, the directory is restructured, but there are still many tool descriptions for tools that have descriptions for. So you need to try out the existing descriptions and either (1) propose changes or (2) motivate why a new description is needed. Examples of (1) are : extra flag, additional output capture. Examples of (2) are: additional subcommand or fundamentally different mode of operation using an already described (sub)command. |
mr-c
reviewed
May 15, 2020
Comment on lines
+52
to
+54
| arguments: | ||
| valueFrom: bowtie2-build $(inputs.ReferenceGenome.path) $(inputs.index) bowtie2 -q -x $(inputs.index) -1 $(inputs.leftFastq.path) -2 $(inputs.rightFastq.path) -S $(inputs.sampleName).sam | ||
| shellQuote: false |
Member
There was a problem hiding this comment.
Suggested change
| arguments: | |
| valueFrom: bowtie2-build $(inputs.ReferenceGenome.path) $(inputs.index) bowtie2 -q -x $(inputs.index) -1 $(inputs.leftFastq.path) -2 $(inputs.rightFastq.path) -S $(inputs.sampleName).sam | |
| shellQuote: false | |
| arguments: | |
| valueFrom: bowtie2-build $(inputs.ReferenceGenome.path) $(inputs.index) | bowtie2 -q -x $(inputs.index) -1 $(inputs.leftFastq.path) -2 $(inputs.rightFastq.path) -S $(inputs.sampleName).sam | |
| shellQuote: false |
Author
There was a problem hiding this comment.
Piped bowtie2-build and bowtie2.
Author
There was a problem hiding this comment.
Please validate the change.
There is error.
(env) ngsap2@ubuntu:~$ cwl-runner alignmentDocker.cwl
INFO /home/ngsap2/.local/bin/cwl-runner 3.0.20200324120055
INFO Resolved 'alignmentDocker.cwl' to 'file:///home/ngsap2/alignmentDocker.cwl'
ERROR Tool definition failed validation:
alignmentDocker.cwl:14:24: mapping values are not allowed here
Author
There was a problem hiding this comment.
I have run cwl-format and cwltool --validate for AlignmentDocker.cwl. But cwltool run fails. Error is below.
ngsap2@ubuntu:~$ sudo docker run -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":"$PWD" -w="$PWD" commonworkflowlanguage/cwltool /home/ngsap2/samtoolsDocker/AlignmentDockerNew.cwl --leftFastq /home/ngsap2/Downloads/testdata16march/ebola_mut_reads1.fq --rightFastq /home/ngsap2/Downloads/testdata16march/ebola_mut_reads2.fq --ReferenceGenome /home/ngsap2/Downloads/testdata16march/ebola_ref.fasta --sampleName ebola --index eb --docker comics/bowtie2
INFO /usr/local/bin/cwltool 2.0.20200107113851
INFO Resolved '/home/ngsap2/samtoolsDocker/AlignmentDockerNew.cwl' to 'file:///home/ngsap2/samtoolsDocker/AlignmentDockerNew.cwl'
INFO ['docker', 'pull', 'docker']
Using default tag: latest
latest: Pulling from library/docker
cbdbe7a5bc2a: Pulling fs layer
6f4c731d6eb3: Pulling fs layer
5d79d4ac3cd0: Pulling fs layer
65de10866d6a: Pulling fs layer
ffd69ed7f17f: Pulling fs layer
07ba9e5a2fa6: Pulling fs layer
75478c283dd2: Pulling fs layer
ffd69ed7f17f: Waiting
07ba9e5a2fa6: Waiting
75478c283dd2: Waiting
65de10866d6a: Waiting
5d79d4ac3cd0: Verifying Checksum
5d79d4ac3cd0: Download complete
cbdbe7a5bc2a: Verifying Checksum
cbdbe7a5bc2a: Download complete
6f4c731d6eb3: Verifying Checksum
6f4c731d6eb3: Download complete
cbdbe7a5bc2a: Pull complete
ffd69ed7f17f: Download complete
07ba9e5a2fa6: Verifying Checksum
07ba9e5a2fa6: Download complete
6f4c731d6eb3: Pull complete
5d79d4ac3cd0: Pull complete
75478c283dd2: Verifying Checksum
75478c283dd2: Download complete
65de10866d6a: Verifying Checksum
65de10866d6a: Download complete
65de10866d6a: Pull complete
ffd69ed7f17f: Pull complete
07ba9e5a2fa6: Pull complete
75478c283dd2: Pull complete
Digest: sha256:8c119115801ae96e1f55b2ccb69cc84537c79eea057a00f363df0196f3d09fd4
Status: Downloaded newer image for docker:latest
INFO [job AlignmentDocker] /tmp/jx5vfv7x$ docker \
run \
-i \
--volume=/tmp/jx5vfv7x:/BScJye:rw \
--volume=/tmp/e3m0q78a:/tmp:rw \
--volume=/home/ngsap2/Downloads/testdata16march/ebola_mut_reads1.fq:/var/lib/cwl/stgfeed78c8-d5f0-4802-9b1d-d8e9ab026a04/ebola_mut_reads1.fq:ro \
--volume=/home/ngsap2/Downloads/testdata16march/ebola_mut_reads2.fq:/var/lib/cwl/stg425eea77-c9d2-44d6-b616-53e511a4b32d/ebola_mut_reads2.fq:ro \
--volume=/home/ngsap2/Downloads/testdata16march/ebola_ref.fasta:/var/lib/cwl/stg5c94f135-a5af-4a66-8fe8-fb0d1541a958/ebola_ref.fasta:ro \
--workdir=/BScJye \
--read-only=true \
--user=0:0 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/BScJye \
--cidfile=/tmp/b66i6_y0/20200517054753-879463.cid \
docker \
/bin/sh \
-c \
bowtie2-build /var/lib/cwl/stg5c94f135-a5af-4a66-8fe8-fb0d1541a958/ebola_ref.fasta eb | bowtie2 -q -x eb -1 /var/lib/cwl/stgfeed78c8-d5f0-4802-9b1d-d8e9ab026a04/ebola_mut_reads1.fq -2 /var/lib/cwl/stg425eea77-c9d2-44d6-b616-53e511a4b32d/ebola_mut_reads2.fq -S ebola.sam
/bin/sh: bowtie2-build: not found
/bin/sh: bowtie2: not found
INFO [job AlignmentDocker] Max memory used: 0MiB
ERROR [job AlignmentDocker] Job error:
("Error collecting output for parameter 'rawSAM':\nsamtoolsDocker/AlignmentDockerNew.cwl:31:5: Did not find output file with glob pattern: '['ebola.sam']'", {})
WARNING [job AlignmentDocker] completed permanentFail
{}
WARNING Final process status is permanentFail
mr-c
reviewed
May 15, 2020
Comment on lines
+13
to
+14
| -class: DockerRequirement | ||
| dockerPull: docker |
Member
There was a problem hiding this comment.
Suggested change
| -class: DockerRequirement | |
| dockerPull: docker | |
| - class: DockerRequirement | |
| dockerPull: docker |
Author
There was a problem hiding this comment.
Made the changes.
Now the error is into arguments section.
https://github.com/ambarishK/bio-cwl-tools/blob/release/bowtie2/AlignmentDocker.cwl
(env) ngsap2@ubuntu:~$ cwl-runner alignmentDocker.cwl
INFO /home/ngsap2/.local/bin/cwl-runner 3.0.20200324120055
INFO Resolved 'alignmentDocker.cwl' to 'file:///home/ngsap2/alignmentDocker.cwl'
ERROR Tool definition failed validation:
alignmentDocker.cwl:7:5: Object `alignmentDocker.cwl` is not valid because
tried `CommandLineTool` but
alignmentDocker.cwl:104:5: the `arguments` field is not valid because
value is a CommentedMap, expected null or array of <string or
Expression or CommandLineBinding>
Author
There was a problem hiding this comment.
cwl-format and cwltool --validate is complete but cwltool run fails. Error is posted in above comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is my pull request for WDL2CWL conversion example.
Will fix #79