3D Mesh Segmentation Using Transformer Based Graph Operations

Kurzfassung

Deep Learning and especially convolutions have been a massive success in computer vision tasks such as Segmentation, Object Detection, and others. However, all of these are limited to 2D images, whereas the progress in the 3D domain has been limited. Extending these priors works to the 3D domain is sadly not straightforward. The biggest challenge here is the unstructured representation of 3D data such as meshes or point clouds. While other works use voxel grids, which have structured representation, they usually struggle with computation time and memory. In this thesis, the task of extending a convolution operation to unstructured data and its problems along with potential solutions is explored. In this thesis, two new methods to the task of Mesh Segmentation are proposed. Both these methods are based on Transformer networks and their components. In the first method, a first of its kind application of transformers to the task of Mesh Segmentation is proposed. In the second method, a permutation invariant Graph Convolution layer named TransConv is proposed which acts similar to a convolution operation on images and can be used in any model architecture. In addition to these methods, two extensions are proposed that improve the performance of both our methods. The first extension is to use depth encoding to add more information about the geodesic distance to the model. The second extension is to extend the concept of atrous convolutions in images to meshes. All of our methods and extensions are evaluated on two datasets and compared with other related works. The first dataset is a collection of high-resolution meshes called the Coseg dataset. The second dataset is a collection of point clouds of 3D objects called ShapeNet part annotation. Our proposed graph convolution layer TransConv outperforms other related works in both the datasets. However, our method to use transformers for mesh segmentation produced comparable results.