DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Self-supervised Audiovisual Representation Learning for Remote Sensing Data

Heidler, Konrad and Mou, LiChao and Hu, Di and Jin, Pu and Li, Guangyao and Gan, Chuang and Wen, Ji-Rong and Zhu, Xiao Xiang (2023) Self-supervised Audiovisual Representation Learning for Remote Sensing Data. International Journal of Applied Earth Observation and Geoinformation, 116, p. 103130. Elsevier. doi: 10.1016/j.jag.2022.103130. ISSN 1569-8432.

[img] PDF - Preprint version (submitted draft)

Official URL: https://www.sciencedirect.com/science/article/pii/S1569843222003181


Many current deep learning approaches make extensive use of backbone networks pre-trained on large datasets like ImageNet, which are then fine-tuned to perform a certain task. In remote sensing, the lack of comparable large annotated datasets and the wide diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pre-trained backbone networks in remote sensing, we devise a self-supervised approach for pre-training deep neural networks. By exploiting the correspondence between geo-tagged audio recordings and remote sensing imagery, this is done in a completely label-free manner, eliminating the need for laborious manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and audio samples all around the world. Using this dataset, we then pre-train ResNet models to map samples from both modalities into a common embedding space, which encourages the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pre-trained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery. The dataset, code and pre-trained model weights will be available at this URL: https://github.com/khdlr/SoundingEarth.

Item URL in elib:https://elib.dlr.de/191012/
Document Type:Article
Title:Self-supervised Audiovisual Representation Learning for Remote Sensing Data
AuthorsInstitution or Email of AuthorsAuthor's ORCID iDORCID Put Code
Heidler, KonradUNSPECIFIEDhttps://orcid.org/0000-0001-8226-0727UNSPECIFIED
Mou, LiChaoUNSPECIFIEDhttps://orcid.org/0000-0001-8407-6413UNSPECIFIED
Hu, DiUNSPECIFIEDhttps://orcid.org/0000-0002-7118-6733UNSPECIFIED
Gan, ChuangUNSPECIFIEDhttps://orcid.org/0000-0003-4031-5886UNSPECIFIED
Wen, Ji-RongUNSPECIFIEDhttps://orcid.org/0000-0002-9777-9676UNSPECIFIED
Zhu, Xiao XiangUNSPECIFIEDhttps://orcid.org/0000-0001-5530-3613UNSPECIFIED
Date:February 2023
Journal or Publication Title:International Journal of Applied Earth Observation and Geoinformation
Refereed publication:Yes
Open Access:Yes
Gold Open Access:Yes
In ISI Web of Science:Yes
Page Range:p. 103130
Keywords:audiovisual representation learning, geo-tagging, remote sensing imagery,
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Earth Observation
DLR - Research area:Raumfahrt
DLR - Program:R EO - Earth Observation
DLR - Research theme (Project):R - Artificial Intelligence
Location: Oberpfaffenhofen
Institutes and Institutions:Remote Sensing Technology Institute > EO Data Science
Deposited By: Beuchert, Tobias
Deposited On:30 Nov 2022 14:21
Last Modified:19 Dec 2022 15:08

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Website and database design: Copyright © German Aerospace Center (DLR). All rights reserved.