AI4EO: Learning from Human Uncertainty

Abstract

As many other research fields, remote sensing has been greatly impacted by machine and deep learning and benefits from technological and computational advances. In the recent years, a considerable effort has been spent on deriving not just accurate, but also reliable modeling techniques. In the particular framework of image classification, this reliability is validated by e.g. checking whether the confidence in the model prediction adequately describes the true certainty of the model when confronted with unseen data. We investigate this reliability in the framework of classifying satellite images into different land cover classes. More precisely, we use the So2Sat LCZ42 data set comprised of Sentinel-1 and Sentinel-2 image pairs. Those were classified into 17 categories by a team of two labelers, following the Local Climate Zone (LCZ) classification scheme. As a novelty, we make explicit use of the so-termed evaluation set which was additionally produced by the authors of the LCZ42 data set. In this supplementary study, a subset of the initial data was re-labeled by 10 different remote sensing experts, which independently of one another re-cast their label votes for each satellite image. The resulting sets of label votes contain a notion of human uncertainty associated with the underlying satellite images. In the following, we try to explicitly incorporate this uncertainty into the training process of a neural network classifier and investigate its impact on model performance. Also, the earlier introduced definition of reliability is checked and compared to a more common modeling approach. The more common approach is using a single ground truth as label, which is derived from the majority vote of the individual expert label votes.