Hand and Object Pose Estimation using Self-Supervised Learning

Abstract

Hand and object pose estimation is an important topic in robotics and computer vision. State-of-the-art methods typically train a deep neural network on annotated dataset using supervised learning. However, precisely annotating high-dimensional poses in two dimensional image plane is very difficult and time-consuming, especially when there are severe occlusions while the human hand manipulating objects. Some researchers try to eliminate this problem by relying on rendered synthetic dataset. However, the models do not generalize well on the real images due to the domain gap between rendered and real photos. In this work, we address the issue of lacking annotated dataset for both object and hand pose estimation. We design a self- or weakly supervised learning framework respectively for estimating hand pose and object pose, which directly extract information from input images and use it as an alternative of ground truth. In object pose estimation framework, we utilize a differentiable renderer to render images with estimated poses and train the network by aligning rendered images with input images. For hand pose estimation, we weakly supervise the training process by fitting the MANO hand model to the 2D hand keypoints predicted with pretrained OpenPose hand detector. Through quantitative and qualitative evaluation, we demonstrate that with appropriate settings of objective functions, we can remove the need of pose annotation without losing much accuracy.