Representation Learning for Robot Keypoint Detection using Prior Kinematic Knowledge

Abstract

We introduce the Prior Knowledge Robot Keypoint Detection (PK-ROKED) approach for 2D keypoint detection on a robot arm. Our proposed method comprises a Deep Learning network, which learns a representation of keypoints based on prior kinematic knowledge and monocular RGB images. This allows us to provide robust visual feedback for state estimations on a robot arm pose, as this pose can be inaccurate due to imprecise forward kinematics. We incorporate the prior kinematic knowledge about potential keypoint locations into the detection network by concatenating it to the input image. These potential keypoints are derived by forward kinematics, which can be faulty with a bounded error. Hence, this additional information can only indicate and steer the detection algorithm to assumed keypoint locations in image space. Additionally, our approach approximates the uncertainty of a keypoint detection through Monte Carlo Dropout and image moments. To this end, PK-ROKED is trained on the respective synthetic data of a robot arm, which we conduct for two different robot arm models. The resulting performance is evaluated on real-world datasets. We observe our PK-ROKED approach to outperform a baseline network, which we defined for benchmarking. Furthermore, when incorporating our prior knowledge approach into the baseline network we can observe a performance increase compared to without this additional information. To test the robustness of our algorithm, we qualitatively evaluate on challenging data from a space-analogue mission, which demonstrates our approach being potentially deployable in such an environment.