elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Archive Reload Function of the Online Data Management System for Earth Observation Data Exploitation Platforms

Kunze, Markus and Kiemle, Stephan and Weiland, Nicolas and Hofmann, Matthias (2018) Archive Reload Function of the Online Data Management System for Earth Observation Data Exploitation Platforms. In: PV2018 Conference - Adding value and preserving data, p. 201. Science and Technology Facilities Council. PV2018 Conference - Adding value and preserving data, 15.-17. Mai 2018, Harwell. ISSN 1362-0231

[img] PDF
512kB

Official URL: https://epubs.stfc.ac.uk/work/37981055

Abstract

The amount of exploitable remote sensing data has increased significantly in recent years. As a result, the requirements of applications for fast processing of large coverages and time series in high resolution have also increased, which confronts Earth Observation (EO) data centers with new challenges. In order to cope with these challenges, approaches such as "Bringing users to the data" have emerged. For this purpose, exploitation platforms with large online memory and processing possibilities are being developed, which provide the functionality to process the data on the platform itself instead of transferring them to the user. The German Aerospace Center DLR is developing an Online Data Management System (ODMS) which provides functionality for managing EO data in exploitation platforms. Due to the enormous amount of data, it is challenging to store all the data on the platform itself despite their large online storage capacity. From time to time, data will have to be evicted based on specific rules. However, in many exploitation use cases large data set series of historic data need to be held online to be processed into consistent time series products. Therefore the ODMS shall provide functionality to efficiently reload evicted data from the backend archives. This introduces additional complexity in the overall concept and results in particular challenges for the development of the system such as performance and optimization for bulk data reloading, handling priorities, taking into account data popularity for eviction and reload while respecting IT security constraints. Additionally, the ODMS strives for decoupling workloads for reload from traditional archiving tasks, and ensuring consistency of data holdings the online platform and the backend archives. This paper will give an overview of the architecture focusing on the archive reload functionality, describe the current state of development and performance measurements. It will introduce sample implementations in projects such as CODE-DE and DIAS for which DLR provides the backend archives for Sentinel-1, Sentinel-3 OLCI, Sentinel-2 and Sentinel-5 Precursor data.

Item URL in elib:https://elib.dlr.de/123120/
Document Type:Conference or Workshop Item (Poster)
Title:Archive Reload Function of the Online Data Management System for Earth Observation Data Exploitation Platforms
Authors:
AuthorsInstitution or Email of AuthorsAuthors ORCID iD
Kunze, MarkusMarkus.Kunze (at) dlr.deUNSPECIFIED
Kiemle, Stephanstephan.kiemle (at) dlr.deUNSPECIFIED
Weiland, Nicolasnicolas.weiland (at) dlr.deUNSPECIFIED
Hofmann, MatthiasMatthias.Hofmann (at) dlr.deUNSPECIFIED
Date:15 May 2018
Journal or Publication Title:PV2018 Conference - Adding value and preserving data
Refereed publication:No
Open Access:Yes
Gold Open Access:No
In SCOPUS:No
In ISI Web of Science:No
Page Range:p. 201
Editors:
EditorsEmail
Conway, EstherUNSPECIFIED
Winfield, KateUNSPECIFIED
Publisher:Science and Technology Facilities Council
Series Name:PV2018 Conference
ISSN:1362-0231
Status:Published
Keywords:ODMS Copernicus Sentinel Online Data Management
Event Title:PV2018 Conference - Adding value and preserving data
Event Location:Harwell
Event Type:international Conference
Event Dates:15.-17. Mai 2018
Organizer:RAL Space
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Earth Observation
DLR - Research area:Raumfahrt
DLR - Program:R EO - Erdbeobachtung
DLR - Research theme (Project):R - Payload ground segment
Location: Oberpfaffenhofen
Institutes and Institutions:German Remote Sensing Data Center > Information Technology
Deposited By: Kunze, Markus
Deposited On:27 Nov 2018 12:28
Last Modified:31 Jul 2019 20:20

Repository Staff Only: item control page

Browse
Search
Help & Contact
Information
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.