Archive Reload Function of the Online Data Management System for Earth Observation Data Exploitation Platforms

Abstract

The amount of exploitable remote sensing data has increased significantly in recent years. As a result, the requirements of applications for fast processing of large coverages and time series in high resolution have also increased, which confronts Earth Observation (EO) data centers with new challenges. In order to cope with these challenges, approaches such as "Bringing users to the data" have emerged. For this purpose, exploitation platforms with large online memory and processing possibilities are being developed, which provide the functionality to process the data on the platform itself instead of transferring them to the user. The German Aerospace Center DLR is developing an Online Data Management System (ODMS) which provides functionality for managing EO data in exploitation platforms. Due to the enormous amount of data, it is challenging to store all the data on the platform itself despite their large online storage capacity. From time to time, data will have to be evicted based on specific rules. However, in many exploitation use cases large data set series of historic data need to be held online to be processed into consistent time series products. Therefore the ODMS shall provide functionality to efficiently reload evicted data from the backend archives. This introduces additional complexity in the overall concept and results in particular challenges for the development of the system such as performance and optimization for bulk data reloading, handling priorities, taking into account data popularity for eviction and reload while respecting IT security constraints. Additionally, the ODMS strives for decoupling workloads for reload from traditional archiving tasks, and ensuring consistency of data holdings the online platform and the backend archives. This paper will give an overview of the architecture focusing on the archive reload functionality, describe the current state of development and performance measurements. It will introduce sample implementations in projects such as CODE-DE and DIAS for which DLR provides the backend archives for Sentinel-1, Sentinel-3 OLCI, Sentinel-2 and Sentinel-5 Precursor data.