Depth-Based Tracking with Physical Constraints for Robot Manipulation

Abstract

This work integrates visual and physical constraints to perform real-time depth-only tracking of articulated models, with a focus on tracking a robot's manipulators and the objects they interact with in realistic scenarios. As such, we modify an existing dense visual articulated object tracker to additionally avoid interpenetration of multiple interacting objects, and to make use of contact detection by the robot. To achieve greater stability, the tracker detects when the object is stationary relative to the table or relative to the palm and then uses information from multiple frames to converge to an accurate and stable estimate. Deviation from stable states is also detected in order to remain robust to failed grasps and dropped objects. The tracker is integrated into a shared autonomy system in which it provides state estimates used by a grasp planner and the controller of two anthropomorphic hands. We demonstrate the accuracy of the tracking by comparing the estimates with an independent information source: the color stream of the sensor. Qualitative results are also provided for a number of challenging manipulations which are made possible by the speed, accuracy, and stability of the tracking system.