Multi-Vehicle Detection and Tracking in Aerial Image Sequences based on Deep Learning

Abstract

Multi-object detection is a classical challenge in the computer vision community which involves the identification of up to several objects of interest in an image and constructing bounding boxes around them to demarcate them from the background. Multi-object tracking (MOT) takes this a step further by constructing the trajectory of each of the detected objects in every image frame of a video sequence. The research community works actively on the task of MOT via the MOT benchmark that offers the most popular datasets which are generally centered around pedestrian detection and tracking on the ground. However, in this work, we tackle the relatively less worked-upon problem of detecting and tracking vehicles in both aerial imagery and ground image sequences. We pick certain state-of-the-art (SOTA) algorithms from the MOT benchmark and apply them to our domain which consists of two sharply contrasting datasets, the low FPS high resolution DLR dataset containing aerial images captured from a helicopter with large camera motion and the high FPS medium resolution A9 dataset containing ground images captured from traffic monitoring systems with no camera motion. We train several YOLOv7 based detection models and test several SOTA tracking algorithms on the two datasets to conclude that intersection-over-union (IoU) and Kalman Filter work well on the A9 dataset but not on the DLR dataset, whereas appearance features and camera motion compensation make more sense for the DLR dataset and not so much for the A9 dataset. We therefore propose a new robust tracking algorithm called ByteDe-SORT that lacks the Kalman Filter and is a combination of the IoU based ByteTrack and the appearance features based DeepSORT. Byte-De-SORT achieves a competitive (HOTA, MOTA, IDF1) score of (0.56, 0.75, 0.63) on the DLR dataset and (0.48, 0.68, 0.51) on the A9 dataset, making it the best overall method that can be applied to both the datasets. It achieves an average inference speed of 2.82 FPS on the DLR dataset and 14.34 FPS on the A9 dataset, making it also suitable for real-time tracking.