Closing the Loop: 3D Object Tracking for Advanced Robotic Manipulation

Abstract

In robotics, it is often assumed that the world is static, forward kinematics are perfectly known, and interactions are fully deterministic. However, while those assumptions are true for some applications, in general, they significantly limit the complexity of manipulation tasks and the design of robotic hardware. To overcome such limitations, this thesis focuses on 3D object tracking techniques for advanced robotic manipulation. The goal is to develop a flexible and efficient algorithm that uses camera data to provide continuous pose estimates for the robot's end effector and relevant objects. Given such feedback, it is possible to adopt more human-like visual servoing approaches, react to changes in the environment, and facilitate safe and efficient robot designs. The first main contribution of this work is a highly-efficient approach to region-based object tracking. It features a well-founded probabilistic model that considers image information sparsely along so-called correspondence lines. Experiments demonstrate that the algorithm performs significantly better than existing region-based methods while being approximately one order of magnitude faster. Subsequently, based on this approach, an extension to multi-modality tracking is discussed. The method allows to fuse depth, texture, and multi-region information from various cameras in a highly-modular probabilistic formulation. Again, experiments on different datasets show that the resulting algorithm is highly efficient and outperforms both conventional and deep learning-based methods by a considerable margin. Finally, the third part of this thesis considers the tracking of multi-body objects, which are composed of rigid bodies that are connected by joints. For this, a flexible framework is proposed that allows the extension of existing rigid object tracking methods to multi-body tracking. It considers both tree-like and closed kinematic structures and facilitates the flexible configuration of joints and constraints. For a detailed evaluation, a highly-realistic synthetic dataset is introduced that features a large number of sequences and various robots. Experiments demonstrate the excellent performance of the developed framework and tracker. All theoretical concepts are implemented into the multi-body, multi-modality, and multi-camera tracking library M3T, which we released as open-source software. It provides a highly-modular architecture that supports a wide range of kinematic structures, object characteristics, and camera setups. In addition, the algorithm allows the incorporation of robot joint measurements of varying accuracy and reliability. In real-world experiments on the humanoid robot David and the MiroSurge system, it is used to continuously provide the pose and configuration of manipulated objects and robot end effectors. The developed approach thereby allows to close the perception-action loop and facilitates new capabilities for advanced robotic manipulation.