SkyScapes - Fine-Grained Semantic Understanding of Aerial Scenes

Abstract

Understanding the complex urban infrastructure withcentimeter-level accuracy is essential for many applicationsfrom autonomous driving to mapping, infrastructure monitoring, and urban management. Aerial images providevaluable information over a large area instantaneously; nevertheless, no current dataset captures the complexityof aerial scenes at the level of granularity required byreal-world applications. To address this, we introduceSkyScapes, an aerial image dataset with highly-accurate,fine-grained annotations for pixel-level semantic labeling.SkyScapes provides annotations for 31 semantic categoriesranging from large structures, such as buildings, roadsand vegetation, to fine details, such as 12 (sub-)categories of lane markings. We have defined two main tasks onthis dataset: dense semantic segmentation and multi-classlane-marking prediction. We carry out extensive exper-iments to evaluate state-of-the-art segmentation methodson SkyScapes. Existing methods struggle to deal with thewide range of classes, object sizes, scales, and fine detailspresent. We therefore propose a novel multi-task model,which incorporates semantic edge detection and is bettertuned for feature extraction from a wide range of scales.This model achieves notable improvements over the base-lines in region outlines and level of detail on both tasks.