Designing a Framework for DataOps: Improving Data Quality and Pipeline Efficiency in Data Science

Abstract

Data Science (DS), with the application of methods from Data Analytics, can assist organizations in deriving value from huge amounts of data to improve performance. To be able to provide actionable insights from this resource, a significant effort in DS is attributed to the data collection, cleaning, and transformation to construct a target high-quality dataset for analytics. However, the traditional, predominantly manual approach to data preparation is considered inefficient and error-prone, failing to meet the requirements of Big Data environments and the adaptability to react to changing conditions. DataOps aims to combine best practices and technologies from DevOps to automate stages of the data lifecycle to increase the quality and reliability of data pipelines. Hence, in this paper, a framework is proposed to guide DataOps implementation, using the Design Science Research methodology. The applicability of the artifact is demonstrated in a case study in urban mobility analytics.