Assessing Predictive Uncertainties in Remote Sensing Image Classification via Conformal Prediction

Abstract

A crucial and often unrecognized aspect of machine learning models for remote sensing is the evaluation of the model's predictive uncertainty. Particularly for classification tasks, models often suffer from overconfidence, which in turn results in small uncertainty values and badly calibrated predictions. In practice, such uncalibrated remote sensing models fail to predict on unseen test data and cannot adequately express uncertainties in their predictions. To tackle this problem, recent advances have been made in the context of deep learning models building upon the theory of conformal prediction. In short, conformal prediction allows us to construct prediction sets that satisfy theoretical coverage guarantees regarding the true label for an arbitrary machine learning model. The construction of the sets here relies on a predictive uncertainty measure of the underlying model. Adding to that, the general theory only has mild assumptions regarding the data distribution. For the experimental setup, we focus our analysis on the So2Sat LCZ42 dataset, which consists of labeled Sentinel-2 imagery of 42 cities around the globe. The labeling scheme consists of 17 classes, the so-called Local Climate Zones (LCZs), describing urban conglomerates and their vegetational surroundings. We set a particular focus on a subset consisting of 10 European cities, for which a label evaluation study has been performed. For this, 10 remote sensing experts independently labeled the same images, producing a label distribution for each image. While it could already be shown that training with the label distribution instead of a single label increases model generalization and calibration performance, we now investigate whether conformal prediction can help to improve the quality of the predictive uncertainty. In particular, we aim to analyze the correlation between the coverage guarantees of the conformal prediction framework and the label uncertainty among the individual labelers of the studied dataset.