Assessing Conformal Prediction for Remote Sensing: Application to Local Climate Zone Classification

Kurzfassung

While neural networks are at the forefront of solving a variety of tasks in remote sensing, these black-box models often lack a clear sense of uncertainty. Currently experiencing a rebirth in the light of deep learning is the statistically founded uncertainty framework of conformal prediction (CP). Under certain weak assumptions, this framework provides theoretical coverage guarantees for the ground-truth value or class. The coverage guarantee holds for the predictions of the CP framework, which take the form of a prediction set containing one or more classes for classification or segmentation problems, or a prediction interval in the case of regression problems. Additionally, the size of the prediction set or interval also serves as an intuitive indicator of uncertainty. We investigate this uncertainty framework for a large-scale land cover classification dataset termed So2Sat LCZ42, which comes with ten independent expert annotations per annotated satellite image patch. We limit ourselves to the building classes of the dataset, which yield the largest proportion of expert uncertainty. The multiple annotations are here encoded into a soft label used for label uncertainty-aware training. First, we evaluate different adaptations of CP calibration methods. Next to the analysis of coverage and prediction set sizes, as a novelty, we also evaluate the methods with respect to the explainability of the conformalizer’s uncertainty toward aleatoric and epistemic uncertainty. Additionally, we construct a novel active learning (AL) framework based on CP prediction sets and show promising results in terms of expressiveness regarding predictive uncertainty. CP-based AL achieves better test performance than random querying, and yields lower predictive uncertainties than the model trained on the full training set while upholding balanced test accuracy, with only a fraction of the data needed.