UTILIZING MONITORING AND REPORTING TECHNIQUES IN DATA PROCESSING SYSTEMS TO IMPROVE THE VALUE ADDING OF DATA

Abstract

Every day more than 20 TB of remote sensing data is processed at DLRs German Remote Sensing Data Center (DFD) by currently more than 70 processing chains. However, the input data is of varying quality as it may be corrupted by both external and internal effects. Thus, an accurate and effective monitoring and reporting mechanism is vital to ensure reliable processed data. The purpose of this paper is to describe the monitoring and reporting mechanism employed by DFDs Monitoring & Reporting Experts for a processing chain of ionospheric data provided by the Ionosphere Monitoring and Prediction Center (IMPC) and assess its effect. The IMPC is a platform for processing and providing ionospheric data established by the DFD and the Institute for Solar-Terrestrial Physics. The IMPC uses equipment like antennas, computing clusters and a variety of special software to receive information about the ionosphere and upgrade them into forecasts and warnings about its prevailing conditions. As the IMPCs process is vulnerable to error, introduced by hardware issues or software malfunctioning, it requires monitoring and reporting to ensure operational data quality. The monitoring and reporting mechanism described herein, is applied to the IMPC system and consists of four successive steps. Firstly, agents collect, extract, and transform the input data. Secondly, the transformed input data is saved into a time-series database. Thirdly, the input data is visualized for ideal comprehension by the end-user. Fourthly, key statistics of the IMPC process and the visualized input data are reported. Based on the results of this examination, the presented monitoring and reporting techniques safeguards the maintenance of data processing, detects bottlenecks, and ensures the significant improvement the IMPCs data quality.