Heidler, Konrad und Mou, LiChao und Hu, Di und Jin, Pu und Li, Guangyao und Gan, Chuang und Wen, Ji-Rong und Zhu, Xiao Xiang (2023) Self-supervised Audiovisual Representation Learning for Remote Sensing Data. International Journal of Applied Earth Observation and Geoinformation, 116, Seite 103130. Elsevier. doi: 10.1016/j.jag.2022.103130. ISSN 1569-8432.
![]() |
PDF
- Preprintversion (eingereichte Entwurfsversion)
2MB |
Offizielle URL: https://www.sciencedirect.com/science/article/pii/S1569843222003181
Kurzfassung
Many current deep learning approaches make extensive use of backbone networks pre-trained on large datasets like ImageNet, which are then fine-tuned to perform a certain task. In remote sensing, the lack of comparable large annotated datasets and the wide diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pre-trained backbone networks in remote sensing, we devise a self-supervised approach for pre-training deep neural networks. By exploiting the correspondence between geo-tagged audio recordings and remote sensing imagery, this is done in a completely label-free manner, eliminating the need for laborious manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and audio samples all around the world. Using this dataset, we then pre-train ResNet models to map samples from both modalities into a common embedding space, which encourages the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pre-trained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery. The dataset, code and pre-trained model weights will be available at this URL: https://github.com/khdlr/SoundingEarth.
elib-URL des Eintrags: | https://elib.dlr.de/191012/ | ||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Dokumentart: | Zeitschriftenbeitrag | ||||||||||||||||||||||||||||||||||||
Titel: | Self-supervised Audiovisual Representation Learning for Remote Sensing Data | ||||||||||||||||||||||||||||||||||||
Autoren: |
| ||||||||||||||||||||||||||||||||||||
Datum: | Februar 2023 | ||||||||||||||||||||||||||||||||||||
Erschienen in: | International Journal of Applied Earth Observation and Geoinformation | ||||||||||||||||||||||||||||||||||||
Referierte Publikation: | Ja | ||||||||||||||||||||||||||||||||||||
Open Access: | Ja | ||||||||||||||||||||||||||||||||||||
Gold Open Access: | Ja | ||||||||||||||||||||||||||||||||||||
In SCOPUS: | Ja | ||||||||||||||||||||||||||||||||||||
In ISI Web of Science: | Ja | ||||||||||||||||||||||||||||||||||||
Band: | 116 | ||||||||||||||||||||||||||||||||||||
DOI: | 10.1016/j.jag.2022.103130 | ||||||||||||||||||||||||||||||||||||
Seitenbereich: | Seite 103130 | ||||||||||||||||||||||||||||||||||||
Verlag: | Elsevier | ||||||||||||||||||||||||||||||||||||
ISSN: | 1569-8432 | ||||||||||||||||||||||||||||||||||||
Status: | veröffentlicht | ||||||||||||||||||||||||||||||||||||
Stichwörter: | audiovisual representation learning, geo-tagging, remote sensing imagery, | ||||||||||||||||||||||||||||||||||||
HGF - Forschungsbereich: | Luftfahrt, Raumfahrt und Verkehr | ||||||||||||||||||||||||||||||||||||
HGF - Programm: | Raumfahrt | ||||||||||||||||||||||||||||||||||||
HGF - Programmthema: | Erdbeobachtung | ||||||||||||||||||||||||||||||||||||
DLR - Schwerpunkt: | Raumfahrt | ||||||||||||||||||||||||||||||||||||
DLR - Forschungsgebiet: | R EO - Erdbeobachtung | ||||||||||||||||||||||||||||||||||||
DLR - Teilgebiet (Projekt, Vorhaben): | R - Künstliche Intelligenz | ||||||||||||||||||||||||||||||||||||
Standort: | Oberpfaffenhofen | ||||||||||||||||||||||||||||||||||||
Institute & Einrichtungen: | Institut für Methodik der Fernerkundung > EO Data Science | ||||||||||||||||||||||||||||||||||||
Hinterlegt von: | Beuchert, Tobias | ||||||||||||||||||||||||||||||||||||
Hinterlegt am: | 30 Nov 2022 14:21 | ||||||||||||||||||||||||||||||||||||
Letzte Änderung: | 19 Dez 2022 15:08 |
Nur für Mitarbeiter des Archivs: Kontrollseite des Eintrags