Automated Processing of Pipelines Managing Now- and Forecasting of Infectious Diseases

Abstract

When faced with the challenge of now- and forecasting infectious diseases, multiple data sources and state-of-the-art models have to be considered. Automatic aggregation, processing, and publishing to relevant data sinks is paramount to achieving consistent, reproducible, and timely results given daily-reported data. To facilitate scientific collaboration and reproducibility of workflows, open and extensible architectures for compute pipelines are required. In this research, we devise an architecture realizing the seamless management and processing of reproducible pipelines. Our case-study is a daily pipeline for nowcasting the state of SARS-CoV-2 in Germany based on public data and state-of-the-art models implemented in the simulation software MEmilio. The results of our pipeline are pushed to ESID (Epidemiological Scenarios for Infectious Diseases), a user interface to epidemiological simulations. To realize the given pipeline, a workflow management system is required to ensure pipeline processing and secure access to multiple heterogeneous data storages. For this purpose, we based our work on an open-source workflow management system - Apache Airflow, which provides the orchestration, coordination and management of complex connected tasks. S3 is utilized as an intermediate data storage service for sharing data between workflow steps and persisting experiment output. We provide a comprehensive view on our work on automated, end-to-end and reproducible pipelines, with detailed commentary on use case, and its realization.