Denninger, Maximilian and Triebel, Rudolph (2023) 3D Semantic Scene Reconstruction from a Single Viewport. In: Proceedings of the 3rd International Conference on Image Processing and Vision Engineering (IMPROVE 2023), 1, pp. 15-26. SciTePress. 3rd International Conference on Image Processing and Vision Engineering, 2023-04-21 - 2023-04-23, Prague, Czech Republic. doi: 10.5220/0011747700003497. ISBN 978-989-758-642-2. ISSN 2795-4943.
PDF
14MB |
Official URL: https://dx.doi.org/10.5220/0011747700003497
Abstract
We introduce a novel method for semantic volumetric reconstructions from a single RGB image. To overcome the problem of semantically reconstructing regions in 3D that are occluded in the 2D image, we propose to combine both in an implicit encoding. By relying on a headless autoencoder, we are able to encode semantic categories and implicit TSDF values into a compressed latent representation. A second network then uses these as a reconstruction target and learns to convert color images into these latent representations, which get decoded after inference. Additionally, we introduce a novel loss-shaping technique for this implicit representation. In our experiments on the realistic benchmark Replica dataset, we achieve a full reconstruction of a scene, which is visually and in terms of quantitative measures better than current methods while only using synthetic data during training. On top of that, we evaluate our approach on color images recorded in the wild.
Item URL in elib: | https://elib.dlr.de/195220/ | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Document Type: | Conference or Workshop Item (Speech) | ||||||||||||
Additional Information: | Best Student Paper Award | ||||||||||||
Title: | 3D Semantic Scene Reconstruction from a Single Viewport | ||||||||||||
Authors: |
| ||||||||||||
Date: | April 2023 | ||||||||||||
Journal or Publication Title: | Proceedings of the 3rd International Conference on Image Processing and Vision Engineering (IMPROVE 2023) | ||||||||||||
Refereed publication: | Yes | ||||||||||||
Open Access: | Yes | ||||||||||||
Gold Open Access: | No | ||||||||||||
In SCOPUS: | No | ||||||||||||
In ISI Web of Science: | No | ||||||||||||
Volume: | 1 | ||||||||||||
DOI: | 10.5220/0011747700003497 | ||||||||||||
Page Range: | pp. 15-26 | ||||||||||||
Publisher: | SciTePress | ||||||||||||
ISSN: | 2795-4943 | ||||||||||||
ISBN: | 978-989-758-642-2 | ||||||||||||
Status: | Published | ||||||||||||
Keywords: | 3D Reconstruction, 3D Segmentation, Single View, Sim2real | ||||||||||||
Event Title: | 3rd International Conference on Image Processing and Vision Engineering | ||||||||||||
Event Location: | Prague, Czech Republic | ||||||||||||
Event Type: | international Conference | ||||||||||||
Event Start Date: | 21 April 2023 | ||||||||||||
Event End Date: | 23 April 2023 | ||||||||||||
HGF - Research field: | Aeronautics, Space and Transport | ||||||||||||
HGF - Program: | Space | ||||||||||||
HGF - Program Themes: | Robotics | ||||||||||||
DLR - Research area: | Raumfahrt | ||||||||||||
DLR - Program: | R RO - Robotics | ||||||||||||
DLR - Research theme (Project): | R - Multisensory World Modelling (RM) [RO] | ||||||||||||
Location: | Oberpfaffenhofen | ||||||||||||
Institutes and Institutions: | Institute of Robotics and Mechatronics (since 2013) > Perception and Cognition Institute of Robotics and Mechatronics (since 2013) | ||||||||||||
Deposited By: | Strobl, Dr. Klaus H. | ||||||||||||
Deposited On: | 30 May 2023 07:15 | ||||||||||||
Last Modified: | 24 Apr 2024 20:55 |
Repository Staff Only: item control page