Infrastructure and Traffic Monitoring in Aerial Imagery Using Deep Learning Methods

Kurzfassung

Infrastructure and traffic monitoring are two of the most innovative applications for automatically extracting semantic information from aerial images. These applications also include urban and city planning, High-Definition (HD) mapping, parking lot usage mapping, and disaster management mapping for search and rescue operations, among others. HD mapping is also used in autonomous driving as an additional source of information, as it provides fine-grained information about the location of objects. The best way to publicly disseminate spatial details about infrastructure components such as buildings, roads, parking lots, lane markings, and vegetation is through maps. The necessary data collection in the field (on the ground) for a larger area is costly because terrestrial imagery requires the cartographer to visit the area in question. On the other hand, aerial photography offers a wealth of opportunities to remotely observe and map a large area in a short time. With appropriate camera configuration and flight altitude, the resolution of aerial imagery is a few centimeters. Infrastructure and traffic monitoring is an application of aerial image analysis that has emerged in recent decades. For example, aerial imagery can monitor traffic flow to quickly detect potential bottlenecks, accidents, congestion, and other features of interest in a large area. Additionally, automatic detection of dynamic objects can help build more efficient roads, intersections, and highways to reduce congestion and eliminate hazardous areas. The application is not limited to land transportation but can also be extended to maritime transportation. Other dynamic objects in mobility applications include bicyclists, motorcyclists, and pedestrians. A dynamic map and a static map can be combined using automatic aerial imagery analysis, resulting in a comprehensive map called a hybrid map. Initially, image analysis algorithms relied mainly on feature-driven methods. This work focuses on data-driven algorithms such as deep-learning methods that extract information with high accuracy while being transferable to other regions of interest. Objects with few pixels, complex backgrounds, different scales, low resolution, different view angles, shadows and occlusions make this task very challenging. This work aims to develop new deep learning methods to automatically extract infrastructure and traffic monitoring information from aerial images. A total of five problems are addressed in this context. Two problems are related to automatic segmentation from aerial imagery, e.g., roadway markings and other infrastructure-related objects for generating fine-grained HD maps. The other three problems are related to detecting and tracking vehicles in aerial imagery. The present work is cumulative in nature. All five problems are described in six peer-reviewed articles summarized below. Aerial LaneNet: Lane Marking Semantic Segmentation in Aerial Imagery using Wavelet-Enhanced Cost-sensitive Symmetric Fully Convolutional Neural Networks (CNNs): Conventional maps describe infrastructure mainly from the perspective of the road. In order to achieve comprehensive monitoring of the infrastructure, a detailed map is required, including, for example, detailed information about lane markings. These represent an essential and inseparable component of the road infrastructure. By automatically locating lane markings, it is possible to define road boundaries, analyze traffic behavior, and create HD maps for autonomous vehicles. A proposed method combines the wavelet transform (WT) with CNN and enables direct extraction of lane markings with high accuracy and precision at high computational speed without additional information or intermediate processing. SkyScapes - Fine-Grained Semantic Understanding of Aerial Scenes: A detailed map contains information about the different categories of infrastructure components such as buildings, sidewalks, and road markings. A new approach based on deep learning methods is presented that automatically extracts all relevant objects with pixel-level accuracy without any additional information. The algorithm is a cross-class and cross-task dense pixel-wise semantic segmentation for dense and angular segments. This work also presents a concept for the direct classification of road markings in multiple classes from aerial imagery, which is also applicable to satellite imagery. In addition, proof of concept is also provided to extract entrances, exits, and hazardous areas. The proposed method outperforms many state-of-the-art algorithms at this time. Towards Multi-class Object Detection in Unconstrained Remote Sensing Imagery: EAGLE: Large-scale Vehicle Detection Dataset in Real-World Scenarios using Aerial Imagery: In these two papers, a high-precision vehicle detector based on aerial imagery data is presented in conjunction with a robust dataset for detecting vehicles under realistic conditions. Vehicles are annotated in different classes with driving directions and are also detected automatically using the algorithm. The proposed method is a novel deep learning architecture that can localize objects with horizontal and rotated bounding boxes to determine the exact position of objects of interest. The algorithm can also be applied to other objects such as boats and ships for maritime applications. In addition to the traffic monitoring problem, the localization of individual infrastructure objects such as bridges, ports, traffic circles, tank farms, and several other classes is also demonstrated. ShuffleDet: Real-Time Vehicle Detection Network in Onboard Embedded UAV Imagery: Based on the previous work, a new algorithm for vehicle detection based on deep learning is proposed, with low computational cost and comparable performance to other complex and heavy models. The processing method must be fast enough to run on an onboard computing platform, such as Unmanned Aerial Vehicle (UAV). AerialMPTNet: Multi-Pedestrian and -Vehicle Tracking in Aerial Imagery, Using Temporal and Graphical Features: These two papers address the problem of pedestrian and vehicle tracking in aerial sequences. The task is to determine pedestrians’ and vehicles position, speed, and acceleration for a comprehensive traffic monitoring system based on aerial image data. For this last chain of a traffic monitoring system, this paper presents a new multi-object tracking algorithm that tracks single objects in aerial images and provides the position information in the current image from which the speed and orientation of each vehicle or pedestrian can be extracted.