Multi-Object Tracking in Aerial and Satellite Imagery

Kurzfassung

Multi-pedestrian and -vehicle tracking in aerial imagery has several critical applications, including event monitoring, disaster management, predictive traffic, and transport efficiency. While some research works already studied vehicle tracking in remote sensing scenarios, pedestrian tracking has not found the necessary attraction caused by the insufficient level of detail in aerial imagery. Recently, the development of better camera systems and the possibility to capture aerial imagery at low-cost paved the way for establishing novel tracking approaches based on remote sensing. However, current state-of-the-art algorithms, including deep learning based methods, perform especially poorly with pedestrians in aerial imagery, incapable of handling severe challenges such as the large number and the tiny size of the pedestrians (e.g., 4 × 4 pixels) with their similar appearances as well as different scales, atmospheric conditions, low frame rates, and moving camera. In contrast to vehicles moving along predetermined paths, such as highways or streets, pedestrians show more difficult motion characteristics posing additional demands on the tracker. Within the scope of this master thesis, we propose AerialMPTNet, a novel regression-based deep neural network able to tackle the challenges of pedestrian and vehicle tracking in geo-referenced aerial imagery. AerialMPTNet fuses appearance features by a Siamese Neural Network with movement prediction of a Long Short-Term Memory and adjacent graphical features of Graph Convolutional Neural Network. In contrast to previous works, we encode the motion model and the adjacent neighbor modeling in an end-to-end fashion as part of the neural network. Consequently, our network can learn motion characteristics directly from the data and additionally learns to weight the influence of surrounding objects. Furthermore, to the best of our knowledge, we are the first to apply Squeeze-and-Excitation layers and Online Hard Example Mining to a regression-based deep tracker. We evaluate AerialMPNet intensively on two aerial pedestrian datasets, AerialMPT and KIT AIS pedestrian. Both datasets consist of multiple image sequences captured at two frames per second on different flying altitudes, showing different crowd densities and different terrain (e.g., open-air concerts, Munich city areas, BAUMA trade fair). Results indicate that AerialMPTNet outperforms state-of-the-art algorithms such as Tracktor++, SMSOT-CNN or DCFNet on pedestrian tracking in aerial imagery significantly. Compared to the SMSOT-CNN baseline, multiple-object tracker accuracy (MOTA) improves by 18.8 points to -16.2 and by 13.8 points to -23.4 on KIT AIS pedestrian and AerialMPT, respectively. Additionally, we evaluate our approach on the KIT AIS vehicle dataset. AerialMPTNet achieves a competitive MOTA score of 42.0. Since we fitted our method gradually for pedestrian tracking, other trackers achieve better scores here.