Guiding Reinforcement Learning with Shared Control Templates

Abstract

Purposeful interaction with objects usually requires certain constraints to be respected, e.g. keeping a bottle upright to avoid spilling. In reinforcement learning, such constraints are typically encoded in the reward function. As a consequence, constraints can only be learned by violating them. This often precludes learning on the physical robot, as it may take many trials to learn the constraints, and the necessity to violate them during the trial-and-error learning may be unsafe. We have serendipitously discovered that constraint representations for shared control -- in particular Shared Control Templates (SCTs) -- are ideally suited for guiding RL. Representing constraints explicitly (rather than implicitly in the reward function) also simplifies the design of the reward function. We evaluate the advantages of the approach (faster learning without constraint violations, even with sparse reward functions) in a simulated pouring task. Furthermore, we demonstrate that these advantages enable the real robot to learn this task in only 65 episodes taking 16 minutes.