Adaptive Model Mediated Control Using Reinforcement Learning

Kurzfassung

Due to similarities in learning techniques, Reinforcement Learning (RL) is the closest alternative to human-level intelligence. Teleoperation systems using RL can adapt to new environmental conditions and deal with high uncertainty due to long-time delays. In this thesis, we propose a method that takes advantage of RL capabilities to extend the human reach in dangerous remote environments. The proposed method utilizes the Model Mediated Teleoperation (MMT) concept in which the teleoperator interacts with a simulated setup that resembles the real environment. The simulation can provide instant haptic feedback where the data from the real environment are delayed. The proposed approach enables haptic feedback teleoperation of high-DOF dexterous robots under long time delays in a time-varying environment with high uncertainty. In existence of time delay, when the data is received by the remote system the environment may change drastically, therefore, the attempt for task execution will fail. To prevent failure, an intelligence system is realized in two layers, the first layer utilizes the Dynamic Movement Primitives (DMP) which accounts for certain changes in the environment. DMPs can adjust the shape of a trajectory based on given criteria, for example, a new target position or avoiding a new obstacle. But in an uncertain environment, DMPs fail, therefore, the second layer of intelligence makes use of different reinforcement learning methods based on expectation-maximization, stochastic optimal control and policy gradient to guarantee the successful completion of the task. Furthermore, To ensure the safety of the system, and speed up the learning process, each learning session for RL happens in multiple simulations of the remote system and environment, simultaneously. The proposed approach was realized on DLR's haptic hand-arm user interface/exoskeleton, Exodex Adam. It has been used for the first time in this work as the master device to teleoperate a high-DOF dexterous robot. This slave device is an anthropomorphic hand-arm system combining a five-finger hand (FFH) attached to a custom configured DLR lightweight robot (LWR 4+) more closely fitting to the kinematics of the human arm. An augmented reality visualization implemented on the Microsoft Hololens fuses the slave device and virtual environment models to provide environment immersion for the teleoperator. A preliminary user-study was carried out to help evaluate the human-robot interaction capabilities and performance of the system. Meanwhile, the RL approaches are evaluated separately in two different levels of difficulty; with and without uncertainty in perceived object position. The results from the unweighted NASA Task load Index (NASA TLX) and System Usability Score (SUS) questionnaires show a low workload (27) and above-average perceived usability (71). The learning results show all RL methods can find a solution for all challenges in a limited time. Meanwhile, the method based on stochastic optimal control has a better performance. The results also show DMPs to be effective at adapting to new conditions where there is no uncertainty involved.