DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

ExAscale Pre- and Post-Processing Strategy and Software Technology

Basermann, Achim and Matura, Gregor and Chen, Fang and Gerndt, Andreas and Aumüller, Martin and Groen, Derek (2014) ExAscale Pre- and Post-Processing Strategy and Software Technology. Project Report.

[img] PDF


Today’s large-scale simulations deal with complex geometries and numerical data on an extreme scale. As computation approaches the exascale, it will no longer be possible to write and store full-sized result data sets. In-situ data analysis and scientific visualisation provide feasible solutions to the analysis of complex large-scale simulations. To bring pre- and post-processing to the exascale we must consider modifications to data structure and memory layout, and address latency and error resiliency. Load balancing is a crucial pre-processing task on extremely parallel systems. Here, our focus is on a load balancing strategy that supports multiple simulation phases and includes their costs to calculate a data distribution that leads to an optimal performance for the full simulation. The software library PPStee developed in CRESTA already incorporates this idea. It was explicitly designed to support multiple simulation phases. The user provides communication costs of a simulation phase represented as edge weights of a graph corresponding to the simulation data. The according computation costs of the phase are matched to vertex weights. As multiple weight sets can be included, the partitioning of the simulation data is calculated to achieve an optimal load balance covering the full simulation cycle. PPStee supports various partitioning tools, repartitioning and improved initial data distributions by exploiting knowledge from previous simulation runs. PPStee was successfully integrated into several large-scale fluid simulation codes. Here, we demonstrate the flexibility of PPStee with the hemodynamic simulation code HemeLB. In-situ processing has become a key concept in exascale data post-processing and visualisation. Waiting for a simulation to finish and writing out huge amounts of simulation output is no longer a viable solution for data analysis. Instead, visualisation and data analysis must happen when and where a certain simulation step has been carried out, as the so-called in-situ processing. Our in-situ processing system provides scalable distributed post-processing. This system supports on-the-fly data analysis and user interaction to on-going simulations. Here, we demonstrate the feasibility of our system by an online-monitoring scenario with the hemodynamic simulation code HemeLB. Remote hybrid rendering (RHR) is used to access remote exascale simulations from immersive projection environments over the Internet. The display system may range from a desktop computer to an immersive virtual environment such as a CAVE. The display system forwards user input to the visualisation cluster, which uses highly scalable methods to render images of the post-processed simulation data and returns them to the display system. The display system enriches these with context information rendered locally, before they are shown. RHR decouples local interaction from remote rendering and thus guarantees smooth interactivity during exploration of large remote data sets. Here, we discuss strategies, algorithms and techniques for RHR in exascale scenarios and present performance measurements for a prototype developed in CRESTA. For performance analysis, the prototype has been instrumented to collect timing information, compression ratios and image quality metrics.

Item URL in elib:https://elib.dlr.de/93169/
Document Type:Monograph (Project Report)
Title:ExAscale Pre- and Post-Processing Strategy and Software Technology
AuthorsInstitution or Email of AuthorsAuthor's ORCID iDORCID Put Code
Aumüller, MartinUniversität StuttgartUNSPECIFIEDUNSPECIFIED
Groen, DerekUniversity College LondonUNSPECIFIEDUNSPECIFIED
Date:October 2014
Refereed publication:No
Open Access:Yes
Keywords:Exascale computing, pre-processing, post-processing, remote hybrid rendering, partitioning, in-situ processing, interactive visualization
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Space System Technology
DLR - Research area:Raumfahrt
DLR - Program:R SY - Space System Technology
DLR - Research theme (Project):R - Vorhaben SISTEC (old)
Location: Braunschweig , Köln-Porz
Institutes and Institutions:Institut of Simulation and Software Technology > Distributed Systems and Component Software
Institut of Simulation and Software Technology > Software for Space Systems and Interactive Visualisation
Deposited By: Basermann, Dr.-Ing. Achim
Deposited On:09 Dec 2014 14:08
Last Modified:20 Jun 2021 15:46

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Website and database design: Copyright © German Aerospace Center (DLR). All rights reserved.