Workflow-Tools on Living Systems_

Setting up the Nextflow Language Server (LSP) with NeoVim

Fri, 16 May 2025 22:31:42 +0200

It turns out it is super easy to get Nextflow code intelligence to work in NeoVim now! This is thanks to the newly released Nextflow language server following the increasingly popular Language Server Protocol (LSP).

See this episode of the Nextflow podcast for a more in-depth discussion on this new functionality.

The focus has initially been on making this to work in the very popular Visual Studio Code editor, but through some recent developments by the community, setting it up in NeoVim is also pretty smooth.

Troubleshooting Nextflow pipelines

Wed, 01 Nov 2023 11:47:00 +0100

We have been evaluating Nextflow before in my work at pharmb.io , but that was before DSL2 and the support for re-usable modules (which was one reason we needed to develop our own tools to support our challenges, as explained in the paper ). Thus, there’s definitely some stuff to get into.

Based on my years in bioinformatics and data science, I’ve seen that the number one skill that you need to develop is to be able to effectively troubleshoot things, because things will invariably fail in all kinds of ways. And in the process, you will probably learn a lot about the technology stack you are using.

Make your commandline tool workflow friendly

Fri, 25 May 2018 23:59:00 +0200

Update (May 2019): A paper incorporating the below considerations is published:

Björn A Grüning, Samuel Lampa, Marc Vaudel, Daniel Blankenberg, “Software engineering for scientific big data analysis ” GigaScience, Volume 8, Issue 5, May 2019, giz054, https://doi.org/10.1093/gigascience/giz054

There are a number of pitfalls that can make a commandline program really hard to integrate into a workflow (or “pipeline”) framework. The reason is that many workflow tools use output file paths to keep track of the state of the tasks producing these files. This is done for example to know which tasks are finished and can be skipped upon a re-run, and which are not.

What is a scientific (batch) workflow?

Thu, 07 Dec 2017 00:57:00 +0100

Workflows and DAGs - Confusion about the concepts

Jörgen Brandt tweeted a comment that got me thinking again on something I’ve pondered a lot lately:

“A workflow is a DAG.” is really a weak definition. That’s like saying “A love letter is a sequence of characters.” representation ≠ meaning

– @joergenbr

Jörgen makes a good point. A Directed Acyclic Graph (DAG) does not by any means capture the full semantic content included in a computational workflow. I think Werner Gitt’s universal information model is highly relevant here, suggesting that information comes in at least five abstraction layers: statistics (signals, number of symbols), syntax (set of symbols, grammar), semantics (meaning), pragmatics (action), apobetics (purpose, result). A DAG seems to cover the syntax and semantics layers, leaving out three layers out of five.

First production run with SciPipe - A Go-based scientific workflow tool

Thu, 28 Sep 2017 19:32:00 +0200

Today marked the day when we ran the very first production workflow with SciPipe , the Go -based scientific workflow tool we’ve been working on over the last couple of years. Yay! :)

This is how it looked (no fancy GUI or such yet, sorry):

The first result we got in this very very first job was a list of counts of ligands (chemical compounds) in the ExcapeDB dataset (download here ) interacting with the 44 protein/gene targets identified by Bowes et al as a good baseline set for identifying hazardous side-effects effects in the body (that is, any chemical compounds binding these proteins, will never become an approved drug).

Tutorial: Luigi for Scientific Workflows

Tue, 21 Jun 2016 13:49:00 +0200

This is a Luigi tutorial I held at the e-Infrastructures for Massively parallel sequencing workshop (Video archive ) at SciLifeLab Uppsala in January 2015, moved here for future reference.

What is Luigi?

Luigi is a batch workflow system written in Python and developed by Erik Bernhardson and others at Spotify , where it is used to compute machine-learning powered music recommendation lists, top lists etc.

Luigi is one of not-too-many batch workflow systems that supports running both normal command line jobs and Hadoop jobs in the same (in this tutorial, we will focus only on the command line part).

Wanted: Dynamic workflow scheduling

Mon, 26 Oct 2015 21:23:00 +0100

Photo credits: Matthew Smith / Unsplash

In our work on automating machine learning computations in cheminformatics with scientific workflow tools , we have came to realize something; Dynamic scheduling in scientific workflow tools is very important and sometimes badly needed.

What I mean is that new tasks should be able to be scheduled during the execution of a workflow, not just in its scheduling phase.

What is striking is that far from all workflow tools allow this. Many tools completely separate the execution into two stages:

Workflow tool makers: Allow defining data flow, not just task dependencies

Wed, 10 Jun 2015 12:03:00 +0200

Upsurge in workflow tools

There seem to be a little upsurge in light-weight - often python-based - workflow tools for data pipelines in the last couple of years: Spotify’s Luigi , OpenStack’s Mistral , Pinterest’s Pinball , and recently AirBnb’s Airflow , to name a few. These are all interesting tools, and it is an interesting trend for us at pharmbio , who try to see how we can use workflow tools to automate bio- and cheminformatics tasks on compute clusters.

The problem with make for scientific workflows

Sat, 14 Mar 2015 20:46:00 +0100

The workflow problem solved once and for all in 1979?

As soon as the topic of scientific workflows is brought up, there are always a few make fans fervently insisting that the problem of workflows is solved once and for all with GNU make , written first in the 70’s :)

Personally I haven’t been so sure. On the one hand, I know the tool solves a lot of problems for many people. Also, there is something very attractive about buildling on a tool that you can be sure will be available on more or less every unix-like operating system, for decades to come.

Links: Our experiences using Spotify's Luigi for Bioinformatics Workflows

Thu, 12 Feb 2015 20:45:00 +0100

Fig 1: A screenshot of Luigi’s web UI, of a real-world (although rather simple) workflow implemented in Luigi:

Update May 5, 2016: Most of the below material is more or less outdated. Our latest work has resulted in the SciLuigi helper library , which we have used in production and will be focus of further developments.

In the Bioclipse / Pharmaceutical Bioinformatics group at Dept of Pharm. Biosciences att UU, we are quite heavy users of Spotify’s Luigi workflow library , to automate workflows, mainly doing Machine Learning heavy lifting.