ioProc - a light-weight workflow manager in Python

Abstract

ioProc ("input/output processing") is a low level workflow manager in Python providing robust, scalable and reproducible data pipelines. It is developed for and in the scientific context of energy systems analysis but widely applicable in other scientific fields. ioProc was designed with a specific user group in mind: scientists without a thorough programming education or on the beginner level. Scientists' usual prime interest lies in low-maintenance tools, yet providing adaptability solving their problems. Also, most of this user group is accustomed to script based or single file monolythic analysis pipelines and in recent years more frequently to Jupyter notebooks (Kluyver, Ragan-Kelley et al. 2016). For small to medium departments of scientist, forming a group with a shared software base, it is increasingly complicated to maintain standards for sharing code reliably across users, projects and topics. Jupyter notebooks and the provided server infrastructure provide one very specific solution for these challenges which might not fit every scientific research groups' needs, especially when it comes to migrating to a deployable application. ioProc tries to solve this issue in its own unique way, by providing a basic, easy to apply framework which enforces structuring and provides the basis for code sharing. In combination with a concurrent versions system (cvs), ioProc can easily be used to create shared, versioned code bases over scientists, projects and topics, facilitating the exchange of code in a dynamic scientific research environment. There is already a large number of workflow frameworks that have also been developed for data science purposes, such as Airflow, Kedro, Luigi, Pinball and Snakemake. Most of these frameworks, however, are more complex and require a deeper understanding of their software architecture. While this might be easy and even familiar to regular software engineers or programming inclined scientists, the average scientist might perceive these frameworks as distracting or unnecessary overhead. Especially when it comes to debugging or advanced features like parallel processing, complex frameworks create additional maintenance and learning overhead, which not everybody is inclined to. Deploying or publishing analysis software or data-flow pipelines in a scientific context, the dependency of a complicated workflow tool can be a hindrance, especially when it introduces specific hardware and software dependencies. ioProc tries to negotiate a reasonable compromise between the underlying varying requirements. It is developed for scientists (i) wanting an easy to understand and easy to use workflow manager, (ii) looking for a framework that gets into their way as little as possible, while (iii) keeping the possibility to share or publish a code base within their workgroup, department or the public. By its structure and approach, ioProc encourages its users to adhere to well established software engineering principles like the single responsibility principle or DRY (don't repeat yourself). It thus supports and benefits the maintenance of software created on top of it.