Exploring generalization of deep neural networks for the semantic segmentation of burnt areas across Sentinel-2 and Sentinel-3 satellite imagery

Kurzfassung

Mapping burnt areas is an important task in order to be able to assess the ecological and economic consequences of wildfires. Remote sensing data is a suitable tool for this task, enabling spatial classification at pixel level (semantic segmentation). In the field of semantic segmentation, deep learning is by now on a par with rule-based approaches and offers a high degree of adaptability, but requires a large amount of labeled data. One solution to this problem is transfer learning. For the near-real-time semantic segmentation of burnt areas, it is particularly relevant to be able to access remote sensing sensors of various satellite missions in order to ensure a high temporal coverage. To study the transferability between similar optical sensors, based on the MSI sensor on board of the Sentinel-2 mission and the OLCI sensor on board of Sentinel-3, several approaches including data augmentation, fine tuning and joint learning are investigated to generalize across remote sensing sensors. This aims at answering the question if a model can generalize well to both, Sentinel-2 and Sentinel-3 data. The underlying deep learning architecture of derived models is the U-Net. The training of the models is based on the state of the art and attention is paid to a comparable setup, whereby scarce data is available. Further, the Segment Anything Model, a foundation model with extremely high generalization capabilities, is evaluated for the purpose of cross-sensor generalization. Comparisons of the applied approaches show that a model's generalization regarding a different sensor can be increased. By means of the comparable experimental setup, in which two basic models, which were trained on Sentinel-2 and Sentinel-3 data respectively, serve as a reference, it is found that the methods used proved to be of varying usefulness with respect to the research objective. Data augmentation and joint learning can reduce the domain shift with respect to Sentinel-3 data (target domain) between 16 to 25 percent, while maintaining accuracy on Sentinel-2 data (source domain) compared to the baselines. Fine tuning is also suitable for generalizing to the target domain, but at the expense of accuracy to the source domain. Overall, a higher ability for a model to generalize well can be observed when using a combination of data augmentation and joint learning. The results obtained by the Segment Anything Model, on the other hand, are significantly less accurate. There are indications that this is correlated to a decrease in spatial resolution of remote sensing data and further is related to limitations of the model in recognizing semantics.