Camera-Supported Uncertainty Awareness for Guided Real-World Reinforcement Learning

Kurzfassung

Learning from Demonstration (LfD) has become an intuitive and user-friendly method for robot programming. It enables robots to imitate complex behaviors and adapt them to task-specific constraints without requiring detailed models of the environment. However, in scenarios where demonstrations do not fully capture all task dynamics, LfD can become unreliable. For contact-rich tasks, which are common in manufacturing, demonstration-based policies often struggle to generalize due to out-of-distribution states.

Reinforcement Learning (RL) has emerged as a promising solution. By training an RL policy to correct a baseline LfD trajectory based on the robot's state, the resulting behavior adheres to the demonstrations while dynamically adapting to the environment to improve reliability. In recent work, a new approach called Kernelized Guided Reinforcement Learning (KGRL) extends this idea by shaping the RL agent's exploration space according to the probabilistic properties of the demonstrated behavior. Restricting the robot to explore only states that are close to previously observed ones addresses common concerns regarding safety and sample efficiency when applying RL to real-world systems. However, KGRL assumes that the uncertainty encoded in demonstrations sufficiently captures the variability of the task, which is often not the case in practical scenarios.

In this thesis, an extension of KGRL, termed External-Uncertainty KGRL (EU-KGRL), is proposed. The key idea is to incorporate externally provided covariance information into the framework, allowing the exploration space to be shaped by additional sources of uncertainty, such as perception systems, alongside demonstrations. To this end, a markerless, camera-based pose estimation pipeline is implemented, achieving millimeter-level position accuracy. Its accuracy and precision are analyzed to characterize the resulting uncertainty.

The proposed extension is evaluated both in simulation and on a real robotic system for scenarios involving target pose uncertainty. In simulation, EU-KGRL demonstrates the ability to solve tasks that KGRL alone cannot address by enabling structured exploration in task-relevant directions. In real-world experiments, the method is applied to an Ethernet connector insertion task, where the target device pose is estimated using the implemented vision pipeline. The results show that, in certain settings, EU-KGRL can compensate for positional inaccuracies more effectively than KGRL. At the same time, the challenges of deploying EU-KGRL in real-world scenarios are highlighted, as its performance depends strongly on the availability of an accurate uncertainty model.