CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language
Abstract
Background
Massive growth in the amount of research data and computational analysis has led to increased utilization of pipeline managers in biomedical computational research. However, each of more than 100 such managers uses its own way to describe pipelines, leading to difficulty porting workflows to different environments and therefore poor reproducibility of computational studies. For this reason, the Common Workflow Language (CWL) was recently introduced as a specification for platform-independent workflow description, and work began to transition existing pipelines and workflow managers to CWL.
Findings
Here, we present CWL-Airflow, an extension for the Apache Airflow pipeline manager supporting CWL. CWL-Airflow utilizes CWL v1.0 specification and can be used to run workflows on standalone MacOS/Linux servers, on clusters, or on a variety of cloud platforms. A sample CWL pipeline for processing of ChIP-Seq data is provided.
Conclusions
CWL-Airflow will provide users with the features of a fully-fledged pipeline manager and an ability to execute CWL workflows anywhere Airflow can run—from a laptop to cluster or cloud environment.
Availability
CWL-Airflow is available under Apache license v.2 and can be downloaded from <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="https://barski-lab.github.io/cwl-airflow">https://barski-lab.github.io/cwl-airflow</ext-link>, <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://doi.org/10.5281/zenodo.2669582">http://doi.org/10.5281/zenodo.2669582</ext-link>, RRID: SCR_017196.
Related articles
Related articles are currently not available for this article.