Parsing DrugBank XML (or any large XML file) in streaming mode in Go

Thu, 15 Mar 2018 15:19:00 +0100

I had a problem in which I thought I needed to parse the full DrugBank dataset, which comes as a (670MB) XML file

(For open access papers describing DrugBank, see: [1], [2], [3] and [4]). It turned out what I needed was available as CSV files under “Structure External Links ”. There is probably still some other uses of this approach though, as the XML version of DrugBank seems to contain a lot more information in a single format. And in any case, this forced me to figure out how to parse large XML files in a streaming fashion in Go, as older tools like XMLStarlet chokes for many minutes upon the DrugBank file (trying to read it all into memory?), killing any attempt at an iterative development cycle. And, it turns out Go’s support for streaming XML parsing is just great!

Xml on Living Systems_

Parsing DrugBank XML (or any large XML file) in streaming mode in Go