Introspective Methods for Learning-enabled Robotic Perception and Planning

Abstract

Introspection refers to shaping the self-awareness of an agents internal state. This capability can be essential for building trustworthy and cognitive open-world robotic autonomy. As core components within an autonomy stack, striving for enhanced generalizability and efficiency, learning-enabled perception and planning are gaining wider adoption to contend with the complex real world. However, problems of the data-driven learning paradigm, e.g. overconfident predictions and vulnerability against Out-of-Distribution (OOD) inputs, raise serious safety concerns for these approaches applied in robotics. These problems motivate specific research challenges to be addressed, spanning from reliable uncertainty estimation and effective OOD detection to learning actively. In this thesis, we attempt to address these challenges by developing learning-based methods with improved introspective capabilities for application in robotic perception and assembly sequence planning. To this end, we first develop a method based on Bayesian Deep Learning and Probabilistic Graphical Models for reliable uncertainty estimation. This method can not only assist uncertainty-based adaptive object classification but also incorporate object co-occurrence in the scene, facilitating semantic reasoning capabilities. Secondly, towards open-world robot deployment, we introduce an efficient and flexible OOD detection method with flow-based deep generative models, where we propose to utilize an expressive base distribution in the flow to mitigate the fundamental topological constraint. This leads to a performant open-set object detector that is compatible with diverse existing architectures. We further study a similar idea for feasibility learning of an assembly in the task of Robotic Assemble Sequence Planning (RASP), for which we propose a holistic data-driven graphical approach based on Graph Neural Networks (GNNs). This work provides a promising direction to address the challenge of spatial embodiment. Lastly, to pave the way to an active and incremental learning-enabled robot, we devise an active learning pipeline for sim-to-real object detection based on uncertainty estimates from Bayesian Neural Networks (BNNs), in which we tackled the issue of label distribution shift under such conditions with a simple yet effective sampling strategy. Besides comprehensive evaluation in simulation and on self-collected and benchmark data sets, we further conduct real-robot experiments with these methods on an assistance robot and an aerial manipulator, demonstrating their practical applicability. We aim to develop methods to equip a data-driven, leaning-enabled robot with introspective capabilities for greater reliability, adaptivity, and autonomy.