Large-scale coastal anomaly detection using the parallel machine learning library Heat

Kurzfassung

Large-scale Earth observation (EO) datasets are crucial for identifying environmental changes, particularly in coastal regions vulnerable to erosion. However, analyzing these massive datasets requires computational techniques beyond the capabilities of conventional workstations or single nodes of an HPC system. In this study, we demonstrate how Heat, a highly parallel, open-source, Python-based library designed for scalable machine learning and data processing, can address this issue-quasi as an off-the-shelf solution for up-scaling-when detecting anomalies in satellite imagery of the German North Sea coast. Our implementation of an unsupervised outlier-based anomaly detection algorithm uses Heats distributed arrays and vectorized map (vmap) primitives. This algorithm processes tens of millions of shoreline locations by sharing memory and computation across multi-node CPU/GPU clusters, with minimal code changes compared to NumPy/PyTorch. Weak-scaling experiments on both CPUs and GPUs demonstrate the scalability of our approach with increasing amounts of data. We thereby provide, to the best of our knowledge, the first density-based anomaly detection on large-scale EO datasets in a multi-node setting, with proven portability to different hardware architectures (x86-and ARM-based CPUs) and vendors (Nvidia and AMD GPUs). The resulting anomaly maps align with known severe weather episodes, daily anomaly counts correlate with wind metrics from coastal stations, and hotspot maps identify regions of high activity. These findings support the geophysical plausibility of the detections. Our approach is reproducible due to its deterministic algorithms, extensible to additional EO modalities (e.g. coherence), and broadly applicable to nationwide monitoring. Although the 20 m data resolution limits detection sensitivity, the method itself is resolution-agnostic.