Training Deep Learning Models on Synthetic Images

Kurzfassung

Accurate environment perception is a crucial capability for future autonomous aircraft systems, e.g., to identify safe emergency landing sites or to detect various objects of interest. Deep learning has become the state-of-the-art approach for many perception tasks, such as object detection and semantic segmentation in images. However, deep learning models require large and diverse training datasets to achieve high accuracy, which is a major challenge in safety-critical domains like aviation, where data collection is often limited by regulatory, safety, ethical, and economical constraints.

On the other hand, game engines have made significant progress in simulating visually realistic virtual environments, enabling the extraction of diverse synthetic images. Using such synthetic images for training promises to reduce the need for real-world data in many domains, but deep learning models often struggle to generalize well from synthetic to real-world images, and factors influencing this sim-to-real generalization issue are not yet fully understood.

Against this background, this work explores the potential of synthetic images extracted from game engines to train deep learning models for selected aviation perception use-cases. Additionally, it comprehensively investigates the influence of deep learning model architectures on their sim-to-real generalization capability and the ability of image augmentation techniques to improve sim-to-real generalization.

To do so, an extensive evaluation of various use-cases and factors influencing the effectiveness of synthetic data is conducted, utilizing multiple perception tasks and datasets. The evaluations employ the overall real-world performance and the sim-to-real gap as the main evaluation metrics, and the results are based on over 1,700 deep learning model variations trained on synthetic images and evaluated on real-world ones.

This work demonstrates that synthetic images can be a suitable basis for training deep learning models for various aviation use-cases, such as drogue detection during air-to-air refueling and semantic segmentation from a low-altitude aerial perspective. It shows that deep learning models differ significantly in their generalization capability and highlights the strong influence of the model backbone. Further influencing factors are identified, providing practical insights for model selection. Additionally, the application of image augmentation techniques during training, particularly color augmentations, is found to significantly improve the generalization capability of deep learning models trained on synthetic images, stressing the impact of color differences on sim-to-real generalization issues, and showcasing an easy-to-use method to reduce its effect.

Overall, this work provides novel perspectives, a deeper understanding of the potential and limitations of using synthetic images for training deep learning models, and practical insights for model selection and training. It demonstrates the potential of synthetic data to reduce the need for expensive and time-consuming real-world data collection, helping to bring advances in deep learning-based perception into future autonomous aircraft. While this work focuses on aviation, the experiments also include an automotive dataset. These broad investigations further allow the transfer of the findings to other industries with limited access to real-world data, such as maritime or space, ultimately accelerating the deployment of deep learning-based perception to new domains.