Are Vision-Language Foundation Models Able to Fly?

Abstract

Safe autonomous aircraft require accurate environment perception, which can be achieved through semantic segmentation of camera images. However, training neural networks relies on large, diverse datasets that are often unavailable in aviation. Vision-language foundation models offer a promising alternative, but their accuracy for aviation tasks is an open question as the aerial perspective might not be adequately represented in the original training data. Against this background, this paper investigates the performance of two vision-language foundation models, CLIPSeg and CAT-Seg, on an aerial image dataset. Our experiments show that the models can achieve competitive semantic segmentation performance without aviation-specific training. This paper further examines prompt engineering and discusses challenges of deploying these models in aviation. While certification and runtime constraints pose significant hurdles, our findings suggest that vision-language foundation models have potential for improving environment perception in aviation and may reduce the need for extensive training data in the future.