Cromwell & WDL Bioinformatics workflows at any
Description: Cromwell WDL Bioinformatics workflows at any scale Jeff Gentry Data Sciences Platform The backdrop: data generation set to explode Story begins here Quarterly output (in TBases) of the Genomics Platform Plenty of workflow solutions to go
Related Topics
Download Presentation
"Cromwell & WDL Bioinformatics workflows at any" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Cromwell & WDL Bioinformatics workflows at any scale Jeff Gentry
Data Sciences Platform<br>
slide2. The backdrop: data generation set to explode Story begins here Quarterly output (in TBases) of the Genomics Platform<br>
slide3. Plenty of workflow solutions to go around Randall Munroe, XKCD
https://www.xkcd.com/927/ So of course we decided to create a new one.<br>
slide4. Workflow language that humans can read/write
Methods developers and biomedical scientists at large
https://software.broadinstitute.org/wdl/
Execution engine that can
Run on any platform (on-prem and on Cloud)
Scale elastically based on workflow needs
https://github.com/broadinstitute/cromwell Meet WDL + Cromwell<br>
slide5. Workflow Description Language https://software.broadinstitute.org/wdl/<br>
slide6. Basic WDL plumbing call stepA call stepB { input: in=stepA.out } call stepC { input: in=stepB.out } Linear CHAINING Multi-In/out call stepC { input :
in1=stepB.out1, in2=stepB.out2 } Array[File] inputFiles scatter(oneFile in inputFiles) { call stepA { input: in=oneFile } } call stepB { input: files=stepA.out } Scatter-Gather<br>
slide7. Cromwell execution engine Cromwell … HPC GA4GH Local Google Funnel Multiple backends for maximum flexibility Coming Soon: AWS, Azure, Alicloud<br>
slide8. One-off Simple self-contained command
Appropriate for independent analysts Server mode API endpoints
More scalable
Some devops needs
Appropriate for production environments
Call-caching! (aka “ka-ching”) Two main ways to run Cromwell java -jar cromwell.jar \
run hello.wdl \
hello_inputs.json<br>
slide9. Our production system: Genomes On The Cloud NFS Broad on-premises systems Zamboni workflow engine GS data buckets ad-hoc GCE cluster (created on the fly) PAPI Google Cloud Persistent Cromwell server<br>
slide10. Our development setup: on-prem + on-cloud GS data buckets ad-hoc GCE cluster (created on the fly) PAPI Google Cloud Persistent Cromwell server REST API Direct
CLI<br>
slide11. Example external implementation: Google wdl_runner GS data bucket ad-hoc GCE cluster (created on the fly) Creates GCE VM
Executes wdl_runner.py
Sets up Cromwell
Parses WDL workflow
Submits jobs to PAPI
Polls for completion
Copies metadata & outputs to output path
Destroys GCE VM https://cloud.google.com/genomics/v1alpha2/gatk Barebones implementation:<br>
slide12. Example external implementation: wdlRunR Submit workflows to Cromwell
Use R values as inputs
Monitor jobs for completion
Retrieve data back into R
Outputs
Logs
Job metadata https://github.com/seandavi/wdlRunR Direct integration with R:<br>
slide13. Dan Billings
Miguel Covarrubias
Thibault Jeandet
Chris Llanwarne
Ruchi Munshi
Khalid Shakir
Kate Voss The rest of the team<br>
slide14. Thanks! My Email:
jgentry@broadinstitute.org
User Forum:
https://gatkforums.broadinstitute.org/wdl/categories/ask-the-wdl-team
More Information:
https://software.broadinstitute.org/wdl
https://www.github.com/broadinstitute/wdl
https://www.github.com/broadinstitute/cromwell<br>
Data Sciences Platform<br>
slide2. The backdrop: data generation set to explode Story begins here Quarterly output (in TBases) of the Genomics Platform<br>
slide3. Plenty of workflow solutions to go around Randall Munroe, XKCD
https://www.xkcd.com/927/ So of course we decided to create a new one.<br>
slide4. Workflow language that humans can read/write
Methods developers and biomedical scientists at large
https://software.broadinstitute.org/wdl/
Execution engine that can
Run on any platform (on-prem and on Cloud)
Scale elastically based on workflow needs
https://github.com/broadinstitute/cromwell Meet WDL + Cromwell<br>
slide5. Workflow Description Language https://software.broadinstitute.org/wdl/<br>
slide6. Basic WDL plumbing call stepA call stepB { input: in=stepA.out } call stepC { input: in=stepB.out } Linear CHAINING Multi-In/out call stepC { input :
in1=stepB.out1, in2=stepB.out2 } Array[File] inputFiles scatter(oneFile in inputFiles) { call stepA { input: in=oneFile } } call stepB { input: files=stepA.out } Scatter-Gather<br>
slide7. Cromwell execution engine Cromwell … HPC GA4GH Local Google Funnel Multiple backends for maximum flexibility Coming Soon: AWS, Azure, Alicloud<br>
slide8. One-off Simple self-contained command
Appropriate for independent analysts Server mode API endpoints
More scalable
Some devops needs
Appropriate for production environments
Call-caching! (aka “ka-ching”) Two main ways to run Cromwell java -jar cromwell.jar \
run hello.wdl \
hello_inputs.json<br>
slide9. Our production system: Genomes On The Cloud NFS Broad on-premises systems Zamboni workflow engine GS data buckets ad-hoc GCE cluster (created on the fly) PAPI Google Cloud Persistent Cromwell server<br>
slide10. Our development setup: on-prem + on-cloud GS data buckets ad-hoc GCE cluster (created on the fly) PAPI Google Cloud Persistent Cromwell server REST API Direct
CLI<br>
slide11. Example external implementation: Google wdl_runner GS data bucket ad-hoc GCE cluster (created on the fly) Creates GCE VM
Executes wdl_runner.py
Sets up Cromwell
Parses WDL workflow
Submits jobs to PAPI
Polls for completion
Copies metadata & outputs to output path
Destroys GCE VM https://cloud.google.com/genomics/v1alpha2/gatk Barebones implementation:<br>
slide12. Example external implementation: wdlRunR Submit workflows to Cromwell
Use R values as inputs
Monitor jobs for completion
Retrieve data back into R
Outputs
Logs
Job metadata https://github.com/seandavi/wdlRunR Direct integration with R:<br>
slide13. Dan Billings
Miguel Covarrubias
Thibault Jeandet
Chris Llanwarne
Ruchi Munshi
Khalid Shakir
Kate Voss The rest of the team<br>
slide14. Thanks! My Email:
jgentry@broadinstitute.org
User Forum:
https://gatkforums.broadinstitute.org/wdl/categories/ask-the-wdl-team
More Information:
https://software.broadinstitute.org/wdl
https://www.github.com/broadinstitute/wdl
https://www.github.com/broadinstitute/cromwell<br>