Multi-label Learning with ViT for Building Footprint Extraction from Off-Nadir Aerial Images

Kurzfassung

The building footprint extraction (BFE) from aerial images is important for the creation and continuous monitoring of building inventories useful for urban planning, among others. Existing methods frequently extract roofs of buildings from aerial images assuming that they overlap with the footprint. This assumption does not hold in the case of off-nadir images. This letter proposes a novel multilabel learning of oblique building features—footprint, roof, and shape—with a Vision Transformer (ViT) for accurate BFE from off-nadir aerial images. A shape calculation algorithm is developed to derive shape polygons from the existing footprint and roof polygons. The method is compared with several convolutional neural networks (CNNs) and ViTs, and a postprocessing algorithm is further devised to achieve regular building footprint polygons. The proposed method outperforms existing scores of BFE on the BONAI dataset (0.727 versus 0.643F1), and our shape calculation algorithm provides labels as accurate as Segment Anything 2 without the need for a GPU. The results conclude that models trained with shapes in addition to the footprint and roof provide consecutively higher scores (F1 score: 0.747 w/ shape versus 0.727 w/o shape versus 0.682 w/ only footprint) and substantially improve the BFE on off-nadir images. The codes and datasets are available at: https://github.com/bipulneupane/Multilabel-BONAI/.