Using big data technologies to process operational events from field units as well as large scale railway network simulations

Abstract

Nowadays, in every industry sector, processes and systems are being automated and digitalized. Not only processes, but also the availability of free data sets over open data portals such as the mCloud in Germany is creating diverse new possibilities in correlating different data for creating new knowledge. The railway sector is also being directly involved in this process. This article describes the activities in the project Rail2X-Smart Servicesi concerning the use of a data management system composed by big data technologies to process real time data from the field as also a large scale simulation from Germany by using the open source microsimulation software SUMO and a simulation middleware which based on the simulated data from SUMO reproduces the project use cases behavior and generates events which are send to the big data cluster. The big data technologies used in this project is the Apache Kafka message broker framework, the Apache Flink streaming processing framework, the Apache Spark batch processing framework and the Apache Hadoop Framework. All these frameworks are open source tools which are used in the Rail2X-Smart Services project to process standardized messages from the field such as vehicle trajectories, measuring data and status messages. We describe which and how this open source frameworks are going to be used and the architectural build of the system.