FAST-EO: Transforming Earth Observation Through Multi-Modal Foundation Models

Abstract

FAST-EO (Fostering Advancements in Foundation Models via Unsupervised and Self-Supervised Learning for Downstream Tasks in Earth Observation) is an ESA Phi-Lab funded project that aims to develop advanced Foundation Models (FMs) tailored to the unique demands of Earth Observation (EO). The project addresses critical environmental challenges through a transformative approach to data integration, analysis, and scalable modeling solutions. [1]. These FMs are designed to leverage the diverse and complex data streams characteristic of EO, enabling comprehensive insights into Earth's dynamic systems. At the core of FAST-EO is the 4M4EO model, an extension of the "Massively Multimodal Masked Modeling" framework [2], which integrates diverse EO data sources—optical and SAR, as well as metadata and textual data—into a unified representation space. This cohesive architecture supports robust capabilities for zero-shot learning, fine-tuning, and generative tasks, making it applicable to a broad spectrum of EO applications. The model builds upon advancements in multi-modal architectures and incorporates temporal mechanisms, enabling effective processing and analysis of time-series data, which is vital for capturing dynamic environmental changes [3]. The effectiveness of FAST-EO’s advancements is demonstrated through its application to several high-impact use cases. These include flood and wildfire monitoring, providing critical insights for mitigating climate disaster impacts; methane leak detection, supporting efforts to curb greenhouse gas emissions; and forest biomass observation, contributing to carbon management and reforestation initiatives. Additional applications include soil property estimation, which enhances precision agriculture practices; land cover change detection, offering systematic monitoring of urbanization and ecological transitions; and mining expansion assessments, which evaluate the environmental and agricultural implications of land-use changes. These diverse use cases highlight the adaptability and versatility of FAST-EO's models in addressing pressing societal and environmental challenges. A key component of FAST-EO’s progress is its use of the petascale modular supercomputer JUWELS Booster [4], along with benchmarking and preparation for JUPITER - Europe’s upcoming first exascale supercomputer [5]. This computational infrastructure enables the efficient training of large-scale, multi-modal AI foundational models by providing the capacity to handle extensive datasets with high speed and precision. The combined capabilities of JUPITER and JUWELS Booster help address the scale and complexity challenges inherent in EO data, aiming for high performance and scalability while maintaining computational efficiency. FAST-EO’s integration of advanced AI methodologies and multi-modal data processing underscores the transformative potential of Foundation Models in EO. By bridging the gap between state-of-the-art computational techniques and real-world environmental applications, FAST-EO sets a new standard for resource management and decision-making. This project not only advances the role of AI and remote sensing in tackling global challenges but also supports a future of enhanced sustainability, resilience, and informed environmental stewardship. Acknowledgements FAST-EO (Fostering Advancements in Foundation Models via Unsupervised and Self-supervised Learning for Downstream Tasks in Earth Observation) project is funded by the European Space Agency (ESA) Phi-Lab under the contract No. 4000143501/23/I-DT. References [1] Zappacosta, A., Kuzu, R. S., Dumitru, C. O., Molina, D. E., Brunschwiler, T., Kienzler, R., Jakubik, J., Blumenstiel, B., Cavallaro, G., Kesselheim, S., Sedona, R., Wijata, A., Tulczyjew, L., Marek, D., & Nalepa, J. (2024, May 7–10). Democratizing foundation models for Earth Observation applications. ESA-ECMWF ML4ESOP Workshop, ESA-ESRIN, Frascati, Italy. [2] Mizrahi, D., Bachmann, R., Kar, O., Yeo, T., Gao, M., Dehghan, A., & Zamir, A. (2024). 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36. [3] Jakubik, J., Roy, S., Phillips, C. E., Fraccaro, P., Godwin, D., Zadrozny, B., Szwarcman, D., Gomes, C., Nyirjesy, G., Edwards, B., Kimura, D., Simumba, N., Chu, L., Mukkavilli, S. K., Lambhate, D., Das, K., Bangalore, R., Oliveira, D., Muszynski, M., ... Ramachandran, R. (2023). Foundation models for generalist geospatial artificial intelligence. arXiv. https://arxiv.org/abs/2310.18660 [4] Jülich Supercomputing Centre. (2021). JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities, 7, A183. http://dx.doi.org/10.17815/jlsrf-7-183 [5] Jülich Supercomputing Centre, "JUPITER - Exascale for Europe", https://www.fz-juelich.de/en/ias/jsc/jupiter