elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

ioProc - a light-weight workflow manager in Python

Fuchs, Benjamin and Riehm, Judith and Nitsch, Felix and Wulff, Niklas (2020) ioProc - a light-weight workflow manager in Python. [Other]

Full text not available from this repository.

Official URL: https://gitlab.com/dlr-ve/ioproc/-/releases/v1.0

Abstract

ioProc ("input/output processing") is a low level workflow manager in Python providing robust, scalable and reproducible data pipelines. It is developed for and in the scientific context of energy systems analysis but widely applicable in other scientific fields. ioProc was designed with a specific user group in mind: scientists without a thorough programming education or on the beginner level. Scientists' usual prime interest lies in low-maintenance tools, yet providing adaptability solving their problems. Also, most of this user group is accustomed to script based or single file monolythic analysis pipelines and in recent years more frequently to Jupyter notebooks (Kluyver, Ragan-Kelley et al. 2016). For small to medium departments of scientist, forming a group with a shared software base, it is increasingly complicated to maintain standards for sharing code reliably across users, projects and topics. Jupyter notebooks and the provided server infrastructure provide one very specific solution for these challenges which might not fit every scientific research groups' needs, especially when it comes to migrating to a deployable application. ioProc tries to solve this issue in its own unique way, by providing a basic, easy to apply framework which enforces structuring and provides the basis for code sharing. In combination with a concurrent versions system (cvs), ioProc can easily be used to create shared, versioned code bases over scientists, projects and topics, facilitating the exchange of code in a dynamic scientific research environment. There is already a large number of workflow frameworks that have also been developed for data science purposes, such as Airflow, Kedro, Luigi, Pinball and Snakemake. Most of these frameworks, however, are more complex and require a deeper understanding of their software architecture. While this might be easy and even familiar to regular software engineers or programming inclined scientists, the average scientist might perceive these frameworks as distracting or unnecessary overhead. Especially when it comes to debugging or advanced features like parallel processing, complex frameworks create additional maintenance and learning overhead, which not everybody is inclined to. Deploying or publishing analysis software or data-flow pipelines in a scientific context, the dependency of a complicated workflow tool can be a hindrance, especially when it introduces specific hardware and software dependencies. ioProc tries to negotiate a reasonable compromise between the underlying varying requirements. It is developed for scientists (i) wanting an easy to understand and easy to use workflow manager, (ii) looking for a framework that gets into their way as little as possible, while (iii) keeping the possibility to share or publish a code base within their workgroup, department or the public. By its structure and approach, ioProc encourages its users to adhere to well established software engineering principles like the single responsibility principle or DRY (don't repeat yourself). It thus supports and benefits the maintenance of software created on top of it.

Item URL in elib:https://elib.dlr.de/136084/
Document Type:Other
Title:ioProc - a light-weight workflow manager in Python
Authors:
AuthorsInstitution or Email of AuthorsAuthor's ORCID iD
Fuchs, Benjaminbenjamin.fuchs (at) dlr.dehttps://orcid.org/0000-0002-7820-851X
Riehm, JudithJudith.Vesper (at) dlr.dehttps://orcid.org/0000-0002-1783-0196
Nitsch, Felixfelix.nitsch (at) dlr.dehttps://orcid.org/0000-0002-9824-3371
Wulff, NiklasNiklas.Wulff (at) dlr.dehttps://orcid.org/0000-0002-4659-6984
Date:1 September 2020
Refereed publication:No
Open Access:No
Gold Open Access:No
In SCOPUS:No
In ISI Web of Science:No
DOI :10.5281/zenodo.4010275
Status:Published
Keywords:Software, Python, Workflow manager
HGF - Research field:Energy
HGF - Program:Technology, Innovation and Society
HGF - Program Themes:Renewable Energy and Material Resources for Sustainable Futures - Integrating at Different Scales
DLR - Research area:Energy
DLR - Program:E SY - Energy Systems Analysis
DLR - Research theme (Project):E - Systems Analysis and Technology Assessment (old)
Location: Stuttgart
Institutes and Institutions:Institute of Engineering Thermodynamics > Energy Systems Analysis
Deposited By: Nitsch, Felix
Deposited On:23 Sep 2020 08:50
Last Modified:23 Sep 2020 08:50

Repository Staff Only: item control page

Browse
Search
Help & Contact
Information
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.