Self-supervised learning for multispectral and hyperspectral remote sensing

Abstract

Foundation models have triggered a paradigm shift in computer vision and remote sensing, demonstrating remarkable generalizability across tasks with minimal fine-tuning. While extensive research has focused on RGB, multispectral, and radar imagery, the potential of foundation models on hyperspectral imagery remains largely untapped. Moreover, most existing works are limited to one or two sensors, despite the availability of diverse satellites capturing imagery across varying spectral ranges and spatial resolutions. Integrating different data sources can improve the versatility of foundation models and enables exploiting the complementarity of different sensors. In this work, we develop a foundation model for multispectral and hyperspectral data, leveraging the rich spectral information unique to HSI. Specifically, we pair the EnMAP-based SpectralEarth dataset with Sentinel-2 and Landsat-8 imagery to create a diverse, multi-sensor dataset for pre-training. Additionally, we design a self-supervised learning method based on masked image modeling and apply it to a vision transformer model. The employed architecture consists of a few initial sensor-specific layers to account for differences in spatial and spectral characteristics, followed by a shared vision transformer backbone that processes all modalities in a unified latent space. The pre-trained model is evaluated on downstream tasks from each sensor, including land cover and crop type classification, to assess its generalizability. Our findings highlight the effectiveness of multi-sensor pre-training and demonstrate positive transfer across modalities. This study contributes to ongoing efforts in developing generic foundation models and provides insights into the training of such models.