On the Relevance of SAR and Optical Modalities in Deep Learning-Based Data Fusion

Abstract

When preparing SAR-optical fusion datasets, cloudy samples are often removed from the optical component if they do not contain any information for the prediction task. Although optical data contains more information that is easier to extract and SAR data is noisier, the latter is less affected by changes in the location or illumination and is not obscured by cloud coverage. By removing clouds from the dataset, the realistic situation of cloud coverage is withheld from the network during training and SAR data has less influence on the prediction than when training with cloudy data. In this work, we show on publicly available pre-trained networks and two remote sensing datasets that the effort to filter and correct clouds might not be needed. In contrast, the results of self-trained ResNet18 networks indicate that having cloudy examples in the dataset might lead to a more informative feature extraction from the SAR modality. This leads to networks that utilize the SAR modality comparatively more for predictions, which we show by an increased relevance of the SAR modality. Moreover, such networks obtain improved accuracy, not only on cloudy test samples but potentially also on clear test data.