Vehicle detection in aerial images using neural networks with synthetic training data

Abstract

Deep learning approaches have made great strides in pattern recognition due to their superior performance. Such approaches require a large amount of ground-truth data. However, it is a challenge to collect enough real training data and label them manually due to cost and time consumption. To overcome this problem, an alternative approach is to use synthetic data with automatically generated ground truth. By means of synthetic data generation, large amount of images can be extracted directly from a virtual scene. These simulated images can be customized according to the specific needs of the use-case. Therefore, this thesis focuses on the use of synthetic data in vehicle detection. A pipeline for generating synthetic data is based on the real-time 3D creation tool Unreal Engine and the drone simulator AirSim. Unreal Engine provides a simulation environment that allows one to simulate complex situations in a virtual world, such as data acquisition with drones. AirSim on the other hand, is a simulator for drones and cars, which works as a plugin for the Unreal Engine. By accessing the AirSim Application Programming Interfaces (APIs), we can retrieve images from a virtual scene at desired camera locations. An existing virtual scene, City Park Environment Collection, and multiple vehicle assets are downloaded and used. The main focus for this thesis is vehicle placement in the virtual scene with the goal of creating realistic scenarios. The scenarios include generating various traffic situations, such as traffic jam, normal traffic flow, and sparsely placed vehicles. The object detector used in this thesis is the Faster R-CNN detector (ReDet). The detection performance of experiments trained with a variety of training and testing datasets are compared and evaluated quantitatively and qualitatively. The evaluation metrics used are true positive rate, false negative rate, average precision, and precision recall curves. Furthermore, we provide insights about the domain similarity of each of these datasets and the effects of having a limited amount of real-world data for training. The results indicate that a domain gap exists between the synthetic and real-world data, yet with an increasing ratio of real-world data for training, the detection performance can be comparable to which trained with only real-world images. The synthetic images generation pipeline can be a guidance or implemented directly for future synthetic image generation. The experimental results of this thesis can be used as a reference when researching on the detection performance using mixed synthetic and real training data.