Array computing and machine learning on HPC-systems: the Python library Heat and an application to earth observation

Kurzfassung

Handling and analyzing massive data sets is particularly important in the field of earth observation. Nevertheless, this can be challenging, especially for researchers and developers without a background in high-performance computing (HPC). The Python library Heat (”Helmholtz Analytics Toolkit”), jointly developed by DLR, Research Center Jülich (FZJ), and Karlsruhe Institute for Technology (KIT), aims at supporting such researchers and developers by providing general-purpose, memory-distributed and GPU-accelerated array manipulation, data analytics, and machine learning algorithms in Python, targeting the usage by non-experts in HPC. Developing software for HPC systems, in particular HPC systems with GPUs, requires extensive testing and benchmarking on such hardware, of course. The easy access to the powerful resources of the Terrabyte cluster has proven to be very helpful for doing so. In this talk/poster we will provide a brief overview on Heat and its main features, followed by scaling results of various functions obtained on Terrabyte in a multi-node, multi-GPU setting. As a particular highlight, we will present recent results of Heats application within the DLR-project RESIKOAST dealing with anomaly detection in an earth observation context. For these results, the usage of a multi-GPU setting on Terrabyte was crucial as the underlying amount of data goes beyond the capabilities of a single GPU. The application in the context of RESIKOAST is joint work with W. Koslow, K. Rack, and A. Rüttgers (all DLR-SC-HPC). The development of Heat is joint work with various colleagues from DLR, FZJ, and KIT, and open-source contributors.