SpectralEarth: Training Hyperspectral Foundation Models at Scale

Abstract

Geospatial Foundation models have gained a lot of attention in recent years, demonstrating remarkable generalization capabilities with minimal fine-tuning. This line of research is supported by the abundance of publicly available satellite data from missions, such as Copernicus Sentinel-2, which provides petabytes of archive data for model training. In contrast, progress in foundation models for the hyperspectral domain has been hindered by the lack of large-scale HSI datasets. While recent efforts have begun to address this limitation, they still fall short in terms of data volume and geographic diversity to effectively train general hyperspectral foundation models. To close this gap, we introduce SpectralEarth, a large-scale dataset derived from the EnMAP mission, featuring over 538,974 hyperspectral image patches from 415,153 unique locations with global spatial distribution and cloud coverage below 10%. SpectralEarth represents an important leap in scale, being significantly larger than existing HSI datasets, as illustrated in Figure 1. Notably, about 17% of its geospatial locations include multiple timestamps, enabling multi-temporal HSI analysis. To effectively exploit the rich information in hyperspectral data, we modify conventional vision backbones such as ResNet and ViT with a spectral adapter module to capture the unique spectral characteristics of HSI. Using three popular self-supervised learning algorithms, MoCo-V2, DINO and MAE, we train these backbones on SpectralEarth, to provide a plurality of pretrained models for hyperspectral image analysis. To benchmark the efficacy of our models, we introduce four downstream datasets geospatially co-registered with EnMAP and DESIS imagery for land cover and crop type mapping. Our experiments demonstrate the potential of foundation models to efficiently generalize across various hyperspectral imaging contexts.