Part semantic segmentation of complete and partial point clouds using deep learning

Kurzfassung

The availability of the 3D sensors and the computational power in the recent years has allowed the computer vision community to start using deep learning approaches for processing 3D data, inspired by the great advancements that deep learning achieved on 2D data to exploit the rich geometrical information that 3D data provides. 3D data can be easily captured and compactly stored when represented by a 3D data representation called point clouds, which describe 3D data solely using their 3D coordinates in the 3D space. Due to their unstructured nature, processing point clouds using deep learning is tricky and cannot be achieved using the traditional deep learning architectures such as CNNs. PointNet and PointNet++ are the pioneer deep learning architectures in processing raw point clouds. This thesis was done in the German Aerospace Center (DLR) in the Institute of Robotics and Mechatronics as part of a project that focuses on the development of service robots that use point clouds as the main source of visual information to perform proper grasping of household objects. Such a complicated task does not only require detailed geometrical information of the objects but also the semantic understating of these objects and their parts. Therefore, the main problem that this thesis addresses is the part semantic segmentation of point clouds. The PointNet++ architecture is capable of solving this problem by being trained, in a supervised manner, to classify the individual points in each point cloud as belonging to one object’s part or another. However, the analyses performed in this thesis show that the PointNet++ architecture fails to segment partially occluded point clouds, which is the case of real-world point clouds. Hence, this thesis mainly focuses on improving the original PointNet++ architecture, not only on partial data but also on complete data. In addition to modifying the original architecture to become translation-invariant, which has shown significant improvements on partial data, this thesis introduces three main variations of PointNet++: SkipNet, intelligent feature propagation (IFP), and point-pair similarity loss (PPSLoss), to further improve the original architecture. The results of this thesis show that combining the first two variations in a translation-invariant architecture outperforms the original architecture on both, complete and partial data. However, when the third variation is added to the new architecture, further improvements are achieved on complete data, but a slight drop in the performance is observed on partial data.