Safe Reinforcement Learning for Robotics

Kurzfassung

Despite its proven potential to learn tasks from scratch across diverse domains, the requirement for a large number of training episodes, coupled with safety concerns, remains a major limiting factor for applying Reinforcement Learning (RL) in robotics. Learning skills directly on real robots is time-consuming, causes wear and tear, and risks damage to both the robot and its environment due to unsafe exploratory actions. Although learning skills in simulation and transferring them to real robots has shown promise, its success is limited by the reality gap between simulation and the physical world. This challenge is particularly pronounced for tasks involving contact with the environment, as contact dynamics are difficult to model and simulate accurately. Moreover, designing realistic simulations is itself a tedious and time-consuming process. This work addresses these challenges by developing methods that incorporate task knowledge and constraints to guide RL, thereby improving sample efficiency and safety during exploration and execution. Unknown task knowledge and actions are learned by an RL agent exploring within constrained environments, enabling learning directly on real robots, especially for contact-rich tasks. The first method in this work, Reinforcement Learning with Shared Control Templates (RL-SCT) leverages hand-designed task knowledge and safety constraints to reduce state and action spaces, enabling safe and sample-efficient learning on real robots. The second method, Kernelized Guided Reinforcement Learning (KGRL) enhances this by integrating human demonstrations to initialize policies and enforce constraints via Linearly Constrained Null-Space Kernelized Movement Primitives (LC-NS-KMP). KGRL improves task knowledge extraction over RL-SCT by deriving task completion strategies directly from demonstrations. It introduces state-dependent, uncertainty-aware exploration noise derived from demonstration variance and imposes linear inequality constraints on the robot's state to ensure safe guided exploration. The third method, Smooth Kernelized Guided Reinforcement Learning (sKGRL) extends KGRL by incorporating smooth exploration strategy, utilizing smooth kernels (e.g., radial basis functions) and robot state history to minimize abrupt, high-acceleration actions, further enhancing safety. Unlike classical RL, our methods enforce constraints explicitly, allowing for simpler reward functions without cost terms for constraint violations. Together, these contributions enable efficient and safe reinforcement learning directly on real robots, advancing RL's applicability for complex, contact-rich robotic manipulation tasks. Evaluated on real robot and simulated environments, these frameworks outperform existing classical RL methods in safety and efficiency metrics, such as reduced spillage of liquid, collisions, interaction forces and more importantly the number of training episodes. By incorporating shared control, demonstration-guided learning, and smooth exploration, this work offers a holistic approach to real-world robotic reinforcement learning. We offer extensive evaluation of these methods to showcase their ability to handle diverse manipulation tasks. The findings in this work provide novel frameworks for scalable, safe RL, advancing the field toward practical deployment in complex, contact-rich robotic manipulation tasks.