Real-Time In-Hand Object Tracking and Sensor Fusion for Advanced Robotic Manipulation

Abstract

In recent years, continuous research has led to robotic hands that are perfectly suitable for in-hand manipulation. To execute such tasks and for example move a pen, the pose of the object is required. While a wide range of algorithms is available to track objects visually, most of those methods suffer if large parts of the object are occluded, which is typically the case during in-hand manipulation. To provide a robust pose estimate under such circumstances, it is desirable to fuse vision-based results with additional information from the robot hand, such as joint angles and torques. This, however, requires an estimate of the tracker’s uncertainty, which is typically not provided by state-of-the-art algorithms. In order to address those challenges, this thesis proposes a novel probabilistic tracking framework that not only provides an estimate of the current pose but also of its uncertainty. It employs both an ICP-based modality to incorporate depth information and a region-based modality to utilise contour information from RGB images. For the region-based modality, the current state of the art is significantly extended with the development of a probabilistic model that features ray correspondences and an efficient scale space implementation. Additionally, a novel mathematical proof is derived that shows that the new modality has Gaussian properties. For the resulting multi-modality tracker, a real-time capable implementation is created that considers occlusions using both explicit and implicit information from rendered masks and depth images. The tracker is integrated on the DLR robot system David and incorporated in the existing EKF-based in-hand localisation framework. Finally, a thorough validation in a wide range of experiments is conducted to evaluate all aspects of the tracker’s performance.