DrawIt: Sketch-Based Robotic Task Programming Framework

Abstract

Robotic programming traditionally requires specialized knowledge and advanced technical skills, which creates significant barriers for non-expert users who want to utilize robotic technologies effectively. Several implementations have already been introduced to address these issues, for example, by leveraging reusable blocks of code. This approach allows users to drag and drop these blocks using a well-designed user interface. However, a more intuitive way for people to engage with robotics is through writing and drawing. This thesis introduces DrawIt, a sketch-based robotic task programming framework that simplifies the robot programming by enabling more natural and intuitive human-computer interaction. Users show their intention for the robot by drawing freehand sketches directly onto a 3D visualization of its working environment, eliminating the need for conventional coding or specialized technical interfaces. DrawIt uses advanced techniques to accurately segment users’ sketches into symbolic and textual elements, utilizing Vision-Language Model (VLM) and Large-Language Model ( LLM ) for interpretation. These models allow the system to reliably interpret diverse user inputs, translating human nature which is ambiguous and inconsistent human sketches into understandable and actionable commands. The framework extracts necessary details, such as skills, parameters, and objects. Using these to convert to structured task sequences that robots can execute directly.