Processing Management Tools for Earth Observation Products at DLR-DFD

Abstract

During the next years more than 50 processing systems for earth observation data are to be integrated into the Data Information and Management System (DIMS) at DLR-DFD. DIMS is a multi-mission processing and archiving and distribution facility for earth observation products. It is a development of DFD. Processing systems developed by various research groups are to be integrated successively. They will be operated for years. Thus, it seems to us to be crucial that operating is efficient and homogeneous. And it can be expected that the processing systems will themselves go through the usual software evolution processes. Experiences from earlier archiving system developments let us decide to provide tools and frameworks for processing systems right from the beginning of DIMS development. These tools are the DIMS Processing System Management (PSM) and the DIMS Operating Tool (OT). Technically speaking, the PSM provides an interface between the implemented processing algorithms on one side and the processing and archiving facility of DIMS on the other. At the DIMS side, two components are involved: DIMS Production Control that controls what to process on which processing system, and DIMS Product Library that serves input products and receives processing results for cataloguing, long term archiving or short term storage. Processing systems are heterogeneous. They vary in programming languages and platforms used, and they range from simple one step input-output systems to complex distributed multi-step systems with user interaction and elaborated workflow control. Examples of complex cases are among those for the Shuttle Radar Topography Mission (SRTM) or for the Global Ozone Monitoring Experiment (GOME). Simpler cases are the raw data processing for the Challenging Minisatellite Payload mission for gravity and magnetic field measurements (CHAMP). But there is also a lot in common among these systems, and that is what we address with the tools. For a family of products ranging from raw data received from a downlink up to high level or value added products there usually is a corresponding family of processing systems that process them. What is common to almost any family of products is that the scenarios change over time: first product "ingestion" of low level products where the inputs come from sources outside of the DIMS Product Library, then systematic processing of some levels, finally production on (user order) request only, and post-processing. All three scenarios are supported by DIMS and the PSM. Not only processing systems but also their developers are heterogeneous. It is a task to introduce certain integration procedures or guidelines. They shall help to mediate between processing algorithm know-how and the requirements of integration and operation. We do not have stable experiences with the establishment of procedures and guidelines up to now. But we know that they are necessary. This "standardisation" issue is also supported by the fact that sometimes systems tend to reside in an institution longer than their developers. So, it is a challenge to support a modular processing system design and a homogeneous user interface, by tools and by procedures. We concentrate on the tools in this paper.