Influence of multimodal training data for segmentation

Abstract

In order to protect and secure critical infrastructures, it is important to advance technologies for automated detection and mapping of crucial components like the pipes of fire suppression systems. This work explores the instance segmentation on such pipes in multimodal image media. Instance segmentation has been significantly enhanced by machine learning techniques, but faces complexities when the used RGB training set is limited and lacks diversity. In the means of improving the training efficiency, we study the influence of enhancing the training set with thermal infrared images on the models performance. In our study, we harnessed both RGB and infrared images captured from the same location. We employed Mask R-CNN with transfer learning from the COCO weights and trained multiple neural networks on different training set combinations with RGB images and thermal infrared images. In order to further enlarge each training data set, we implemented different augmentation methods in the training. Subsequently, we conducted fine-tuning and optimization procedures for the Mask R-CNN training and determined the quality of the pipe instance segmentation of the produced models on test sets containing RGB and infrared images. Using the Jaccard score we provide a quantitative measure of pipe segmentation. The results show that the addition of a comparatively low number of infrared images to the training process does not only improve the detection performance on infrared data, but also improves the models performance on RGB data. A SOTA comparison shows that the detection quality of our models is comparable with the detection results of Ultralytics YOLOv8 and Metas SAM, which are both more recently released.