https://livesys.se/tags/airflow/ Recent content in Airflow on Living Systems_ Hugo en-us Wed, 10 Jun 2015 12:03:00 +0200 https://livesys.se/posts/workflows-dataflow-not-task-deps/ Wed, 10 Jun 2015 12:03:00 +0200 https://livesys.se/posts/workflows-dataflow-not-task-deps/ <h3 id="upsurge-in-workflow-tools">Upsurge in workflow tools</h3> <p><p class="image"> <img src="selection_201.png" alt="Workflow tool logos" class="align_right" /> </p> There seem to be a little upsurge in light-weight - often python-based - workflow tools for data pipelines in the last couple of years: Spotify&rsquo;s <a href="https://github.com/spotify/luigi" target="_blank" rel="noopener">Luigi</a> , OpenStack&rsquo;s <a href="https://wiki.openstack.org/wiki/Mistral" target="_blank" rel="noopener">Mistral</a> , Pinterest&rsquo;s <a href="https://github.com/pinterest/pinball" target="_blank" rel="noopener">Pinball</a> , and recently AirBnb&rsquo;s <a href="https://github.com/airbnb/airflow" target="_blank" rel="noopener">Airflow</a> , to name a few. These are all interesting tools, and it is an interesting trend for us at <a href="http://www.farmbio.uu.se/research/researchgroups/pb/?languageId=1" target="_blank" rel="noopener">pharmbio</a> , who try to see how we can use workflow tools to automate bio- and cheminformatics tasks on compute clusters.</p>