Interactive Robot Control using Augmented Reality Optical See-Through Head Mounted Displays for Robotically Assisted Surgical Systems

Abstract

Human-machine interaction in robotically assisted surgical systems is essential for successful interventions. The surgeon requires intuitive and reliable control of the robotic manipulator, as well as better understanding of the robot's capabilities and limitations. As the technology advances, so does the complexity of each component, with human-machine interaction becoming a challenging task. This work proposes an Augmented Reality (AR) application for interactive robot control, allowing the surgeon or the surgeon assistant to interactively control a robotic manipulator, holding an endoscope. The solution provides an interface using a real-time impedance control model, that allows robot end effector control in task space via hand gesture input sent from an optical-see through head mounted display (OST-HMD). Furthermore, the proposed work integrates features of workspace visualization and endoscopic video stream transmission. In addition, the solution provides an adjustable virtual monitor intended to restore the hand-eye coordination over standard monitor in the case of minimally invasive surgery. The virtual monitor can be arbitrarily aligned in space, allowing the user to choose a pose that enables intuitive hand-eye coordination. The results of the experiments are explored with the MIRO robot and demonstrate interactive AR control via the OST-HMD. The application yields improved hand-eye coordination and enhanced precision for guiding the conventional instruments for the particular application of minimally invasive surgery.