DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Earth Observation Image Semantic Bias: A Collaborative User Annotation Approach

Murillo Montes de Oca, Ambar and Bahmanyar, Reza and Nistor, Nicolae and Datcu, Mihai (2017) Earth Observation Image Semantic Bias: A Collaborative User Annotation Approach. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10 (6), pp. 2462-2477. IEEE - Institute of Electrical and Electronics Engineers. DOI: 10.1109/JSTARS.2017.2697003 ISSN 1939-1404

[img] PDF

Official URL: http://ieeexplore.ieee.org/document/7926346/


Correctly annotated image datasets are important for developing and validating image mining methods. However, there is some doubt regarding the generalizability of the models trained and validated on available datasets. This is due to dataset biases, which occur when the same semantic label is used in different ways across datasets, and/or when identical object categories are labeled differently across datasets. In this paper, we demonstrate the existence of dataset biases with a sample of eight remote sensing image datasets, first showing they are readily discriminable from a feature perspective, and then demonstrating that a model trained on one dataset is not always valid on others. Past approaches to reducing dataset biases have relied on crowdsourcing, however this is not always an option (e.g., due to public-accessibility restrictions of images), raising the question: How to structure annotation tasks to efficiently and accurately annotate images with a limited number of nonexpert annotators? We propose a collaborative annotation methodology, conducting image annotation experiments where users are placed in either a collaborative or individual condition, and we analyze their annotation performance. Results show the collaborators produce more thorough, precise annotations, requiring less time than the individuals. Collaborators labels show less variance around the consensus point, meaning their assigned labels are more predictable and likely to be generally accepted by other users. Therefore, collaborative image annotation is a promising annotation methodology for creating reliable datasets with a reduced number of nonexpert annotators. This in turn has implications for the creation of less biased image datasets.

Item URL in elib:https://elib.dlr.de/112428/
Document Type:Article
Title:Earth Observation Image Semantic Bias: A Collaborative User Annotation Approach
AuthorsInstitution or Email of AuthorsAuthors ORCID iD
Murillo Montes de Oca, Ambarremote sensing technology institute (imf)UNSPECIFIED
Bahmanyar, Rezareza.bahmanyar (at) dlr.deUNSPECIFIED
Nistor, Nicolaelmu, münchen, germanyUNSPECIFIED
Date:11 May 2017
Journal or Publication Title:IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Refereed publication:Yes
Open Access:Yes
Gold Open Access:No
In ISI Web of Science:Yes
DOI :10.1109/JSTARS.2017.2697003
Page Range:pp. 2462-2477
Du, Qian (Jenny)du@ece.msstate.edu
Publisher:IEEE - Institute of Electrical and Electronics Engineers
Keywords:Dataset biases, Remote sensing images, Semantic image annotation, Sensory and semantic gaps, User evaluation
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Earth Observation
DLR - Research area:Raumfahrt
DLR - Program:R EO - Erdbeobachtung
DLR - Research theme (Project):R - Vorhaben hochauflösende Fernerkundungsverfahren
Location: Oberpfaffenhofen
Institutes and Institutions:Remote Sensing Technology Institute > Photogrammetry and Image Analysis
Deposited By: Bahmanyar, Gholamreza
Deposited On:19 May 2017 13:54
Last Modified:31 Jul 2019 20:09

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.