Urban 3D Reconstruction in Remote Sensing via Deep Learning and Dataset Enhancement Strategies

Kurzfassung

Modelling the profile of a city has been widely studied by the research community, particularly in remote sensing. By using sensors located on airborne and satellite platforms, it is possible to retrieve data such as optical/infrared images, radar and laser measurements, etc. Many of these sensors can be used to compute the 3D profile of the scene. Radar and LiDAR are able to measure the distance with high accuracy, but the reconstruction might be sparse, include outliers and uses expensive technology. Images on the contrary are relatively cheaper and capture geometric details, useful for a dense reconstruction. Nonetheless, the reconstruction depends on the matching capabilities of the applied algorithm, as the depth has to be computed from the displacement of corresponding pixels in the images. Before the deep learning solutions, algorithms such as Semi-Global Matching or those based on Structure from Motion used to lead the reconstruction benchmarks. These conventional algorithms can be implemented on any set of images without any prior knowledge of the scene and the refinement process, which benefit from geometric principles to detect inconsistencies and occlusions, generate an accurate digital surface models with few remaining outliers. However, conventional approaches fail in complicated areas such as those with poor texture, repetitive patters, reflective surfaces, that are common in remote sensing imagery. In contrast, deep learning approaches deal better with complicated areas and by using contextual information, they are able to reconstruct a smooth 3D profile with few outliers and high accuracy. Yet, learning based algorithms might fail if the differences between the training and testing sets are large. In addition, neural networks require a large amount of quality data for a robust training, which is not easy to collect for remote sensing platforms. What is more, ground truth might still be obtained with laser but for smaller regions, leading to domain shifts. Hence, the first step to set a reliable framework to evaluate reconstruction algorithms is to provide high quality data. As this is expensive in a real scenario, this study proposes the use of a pipeline to generate large amounts of synthetic data to train stereo matching and multi-view stereo (MVS) networks. Since the data is rendered from software, accurate ground truth is available. Moreover, as the software allows editions of the virtual scene, the urban growth can be simulated, which helps to create data for additional tasks like change detection. A reliable dataset allows to set up experiments to evaluate the quality of the reconstruction algorithms. This dissertation considers two main research directions to design these experiments. On the one hand, it is important to explore the advantages of both the conventional and the learning based solutions, which are evaluated for the stereo matching case. On the other hand, a comparison between the stereo and MVS algorithms is conducted. Intuitively, using complementary information as MVS does might produce a more robust result, but stereo methods have been more studied and have a simplified matching case. Therefore, conventional and learnable, stereo and MVS algorithms are analysed with reliable datasets to assess how these contribute to the 3D reconstruction task. Furthermore, an alternative case to fuse height values into a final digital surface model is explored, where the confidence for the values predicted by the neural networks is estimated and used to guide the fusion. Valuable insights into the urban 3D reconstruction were obtained from the carried out experiments. The generation of datasets from real and synthetic scenarios facilitated the analysis of the capabilities of the tested algorithms. Despite the well-known problem of the domain gap, the networks trained on the generated datasets produced good reconstruction results in complex regions. Buildings and man-made structures benefit from the synthetic models, but for vegetation and natural elements the algorithms exhibit a lower performance because such elements are simplified in the 3D modelling. Among the methods tested, stereo matching approaches computed reconstructions that were less prone to outliers, while the MVS was more robust for edge discontinuities. However, learning algorithms estimate a value for each pixel in the input images, but the reliability of this estimation should still be assessed. By pre-selecting the predicted values based on a confidence estimation, the accuracy of the fusion was improved for the stereo matching case. Yet, this fusion strategy needs to be further explored to generalize to MVS methods as well