Advancing semantic cloud segmentation in all-sky images: A semi-supervised learning approach with ceilometer-driven weak labels

Abstract

Semantic segmentation of all-sky images provides high-resolution cloud coverage information useful for applications in meteorology, climatology, optical satellite downlink operations, and solar energy. While deep neural networks are highly effective for segmentation, their performance depends on large labeled datasets to learn complex visual features. To address this challenge, we introduce a semi-supervised learning approach for semantic cloud segmentation, combining advanced techniques such as ceilometer-driven weak labeling, pseudo-labeling, and consistency regularization. At the core of this approach is CloudMix, a novel data augmentation technique tailored specifically for cloud segmentation tasks. Our method begins with assigning weak labels to over 47,000 all-sky images using ceilometer data, which are combined with 616 manually labeled images to train a segmentation model. By employing pseudo-labeling and weak-to-strong consistency regularization, the model leverages both labeled and weakly labeled data effectively. The semi-supervised model surpasses a fully supervised baseline and a state-of-the-art model in pixel accuracy and mean Intersection over Union (mIoU) across validation, test and domain-shift test dataset. In particular, the detection of mid- and high-layer clouds improves significantly, with an increase in IoU of more than 7 and 9 percentage points on the test dataset. Furthermore, on the domain-shift test dataset, the semi-supervised model achieves over 20 and 27 percentage points higher mIoU than the baseline and state-of-the-art, respectively. These results underscore the robustness and generalization capabilities of the proposed method, making it a promising solution for cloud segmentation.