There is No Model to Beat Them All: Recommendations for Deep Learning Model Selection when Training on Synthetic Images

Abstract

Synthetic training images generated with game-engines are a promising approach to enable the use of deep learning perception models in domains that lack diverse datasets. However, previous works have shown significant performance drops when these models are deployed to real-world scenarios and definite reasons and influences are not yet found. This paper builds on previous work investigating the influence of the model architecture on the sim-to-real generalizability and extends it by addressing key limitations. Based on an extensive study of 378 trained variations of 27 semantic segmentation models on an autonomous driving and an aerial dataset as well as the current literature, this work is the first to provide practical recommendations for selecting deep learning models when training on simulation images.