The FISHNET Case Study on Implementing and Scaling a Complex Earth Observation Workflow

Kurzfassung

Earth observation (EO) scientists use sophisticated algorithms, large datasets, and high-performance data analytics (HPDA) clusters to develop and execute complex EO workflows. To improve portability and reusability within the domain, the Open Geospatial Consortium (OGC) published a set of best practices for developing EO workflows. Even though EO data products are often shared openly, representative EO workflow implementations are hardly available. In this paper, we contribute a case study on implementing and scaling a complex EO workflow, adhering to the OGC best practices and demonstrating its portability by deploying it both locally and on the HPDA platform "terrabyte" at the Leibniz Supercomputing Centre. Contentwise, the contributed workflow analyzes settlement patterns by first delineating coherent settlements derived from leveraged satellite data that maps builtup areas and subsequently calculating centrality measures to characterize the spatial arrangement and hierarchy of settlements in a given region.We demonstrate the workflow’s scalability and variance in resource demands by analyzing its time-to-result, total CPU time, and resource efficiency under different inputs and configurations. Interestingly, implicit parameters hidden in the input data semantics, like the number and area of settlements in the region of interest, significantly impact the time required to complete the processing. Concurrent processing is restricted to connected components of the settlement graph in the analysis stage, leading to an unbalanced workload distribution when analyzing large urban areas, showcasing the scalability challenges EO scientists face.