https://livesys.se/tags/data-engineering/ Recent content in Data Engineering on Living Systems_ Hugo en-us Tue, 21 Jun 2016 13:49:00 +0200 https://livesys.se/posts/luigi-tutorial/ Tue, 21 Jun 2016 13:49:00 +0200 https://livesys.se/posts/luigi-tutorial/ <p><em>This is a Luigi tutorial I held at the <a href="http://uppnex.se/events/eInfraMPS2015/" target="_blank" rel="noopener">e-Infrastructures for Massively parallel sequencing</a> workshop (<a href="https://www.youtube.com/channel/UCfLDx5VYn25QIZLmtybvdeQ/videos" target="_blank" rel="noopener">Video archive</a> ) at <a href="http://scilifelab.se/" target="_blank" rel="noopener">SciLifeLab</a> Uppsala in January 2015, moved here for future reference.</em></p> <h2 id="luigi_screenshotpng"><p class="image"> <img src="luigi_screenshot.png" alt="" /> </p> </h2> <h2 id="what-is-luigi">What is Luigi?</h2> <p><a href="https://github.com/spotify/luigi" target="_blank" rel="noopener">Luigi</a> is a batch workflow system written in Python and developed by <a href="https://erikbern.com/" target="_blank" rel="noopener">Erik Bernhardson</a> and others at <a href="http://spotify.com/" target="_blank" rel="noopener">Spotify</a> , where it is used to compute machine-learning powered music recommendation lists, top lists etc.</p> <p>Luigi is one of not-too-many batch workflow systems that supports running both normal command line jobs and <a href="http://hadoop.apache.org/" target="_blank" rel="noopener">Hadoop</a> jobs in the same (in this tutorial, we will focus only on the command line part).</p>