Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules

Dies ist die aktuellste Version dieses Eintrags.

Kurzfassung

Explainability in robotics is vital for establishing user trust. Recently, foundation models (e.g. vision-language models, VLMs) fostered a wave of embodied agents that answer arbitrary queries about their environment and their interactions with it. However, as VLMs answer queries based on camera images instead of on internal robot components, they cannot be applied directly to existing robot architectures which represent the robot's tasks, skills, and beliefs about the state of the world. To overcome this limitation we propose RACCOON, a framework that combines foundation models' responses with a robot's internal knowledge. Inspired by Retrieval-Augmented Generation (RAG), RACCOON selects relevant context, retrieves information from the robot's state, and utilizes it to refine prompts for an LLM to answer questions accurately, bridging the gap between the model's adaptability and the robot's domain expertise.

Verfügbare Versionen dieses Eintrags

• Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules. (deposited 08 Jul 2024 20:44) [Gegenwärtig angezeigt]