Big data access and processing within the cloud: CODE-DE a non-commercial, operational Earth Observation platform

Abstract

The amount of Earth Observation (EO) data is increasing at an unprecedented rate. The European Copernicus program, in particular with Sentinel-1 and Sentinel-2 imagery in 10 m spatial resolution, is a major data contributor. These large volumes of data require not only sophisticated data storage and access options, but also new big spatial data analysis approaches. The German Copernicus Data and Exploitation Platform (CODE-DE) is an infrastructure for EO data access, on-demand processing and value-added product generation. CODE-DE is operational since March 2017, hosts more than 1.300 registered users with more than 100.000 downloaded data products. The rolling archive host more than 650 TB of data with und 1.2 million data products. The catalogue client provides an enhanced solution to discover, view, and download available products. While a focus is set on data from the Copernicus program are other satellite imagery integrated into the portal. The on-demand processing environment for high-performance cloud processing is operational since November 2017. The algorithms currently available are mainly based on the Sentinel tool box (SNAP) functionality, such as generic band math or dedicated time series algorithms. Processed data can optionally be mosaicked covering larger areas or a time periods. The capacity of the processing environment has been continuously exploited since March 2018 by selected applications, e.g. in the agricultural, maritime or snow and ice context, based on dedicated virtual machines or docker technology. This flexible and free processing approach supports different use cases – allowing developers, processing experts and data experts to work on the platform in support of their own processing scenarios. The CODE-DE data and IT architecture will be explained and implemented user applications presented.