Interpreting Manipulation Actions: from Language to Execution

Abstract

Processing natural language instructions for execution of robotic tasks has been regarded as a means to make more intuitive the interaction with robots. This paper is focused on the applications of natural language processing in manipulation, specifically on the problem of recovering from the instruction the information missing for the manipulation planning, which has been traditionally assumed to be available for instance via pre-computed grasps or pre-labeled objects. The proposed approach includes a clustering process that discriminates areas on the object that can be used for different types of tasks (therefore providing valuable information for the grasp planning process), the extraction and consideration of task information and grasp constraints for solving the manipulation problem, and the use of an integrated grasp and motion planning that avoids relying on a predefined grasp database.