Tell a friend
Print
Bustamante Gomez, Samuel and Knauer, Markus Wendelin and Jeremias, Thun and Schneyer, Stefan and Weber, Bernhard and Stulp, Freek (2024) Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules. In: RSS (Robotics: Science and Systems) conference 2024, Generative Modeling meets HRI Workshop. Generative Modeling meets HRI Workshop at the RSS (Robotics: Science and Systems) conference 2024, 2024-07-15, Delft, Netherlands.
This is the latest version of this item.
![[img]](https://elib.dlr.de/style/images/fileicons/application_pdf.png) |
PDF
4MB |
Abstract
Explainability in robotics is vital for establishing user trust. Recently, foundation models (e.g. vision-language models, VLMs) fostered a wave of embodied agents that answer arbitrary queries about their environment and their interactions with it. However, as VLMs answer queries based on camera images instead of on internal robot components, they cannot be applied directly to existing robot architectures which represent the robot's tasks, skills, and beliefs about the state of the world. To overcome this limitation we propose RACCOON, a framework that combines foundation models' responses with a robot's internal knowledge. Inspired by Retrieval-Augmented Generation (RAG), RACCOON selects relevant context, retrieves information from the robot's state, and utilizes it to refine prompts for an LLM to answer questions accurately, bridging the gap between the model's adaptability and the robot's domain expertise.
| Item URL in elib: | https://elib.dlr.de/205203/ | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Document Type: | Conference or Workshop Item (Poster) | ||||||||||||||||||||||||||||
| Title: | Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules | ||||||||||||||||||||||||||||
| Authors: |
| ||||||||||||||||||||||||||||
| Date: | 15 July 2024 | ||||||||||||||||||||||||||||
| Journal or Publication Title: | RSS (Robotics: Science and Systems) conference 2024, Generative Modeling meets HRI Workshop | ||||||||||||||||||||||||||||
| Refereed publication: | Yes | ||||||||||||||||||||||||||||
| Open Access: | Yes | ||||||||||||||||||||||||||||
| Gold Open Access: | No | ||||||||||||||||||||||||||||
| In SCOPUS: | No | ||||||||||||||||||||||||||||
| In ISI Web of Science: | No | ||||||||||||||||||||||||||||
| Status: | Published | ||||||||||||||||||||||||||||
| Keywords: | Retrieval-Augmented Generation (RAG), Large language models (LLMs), Explainability in robotics, Knowledge representation, Embodied Question-Answering | ||||||||||||||||||||||||||||
| Event Title: | Generative Modeling meets HRI Workshop at the RSS (Robotics: Science and Systems) conference 2024 | ||||||||||||||||||||||||||||
| Event Location: | Delft, Netherlands | ||||||||||||||||||||||||||||
| Event Type: | Workshop | ||||||||||||||||||||||||||||
| Event Date: | 15 July 2024 | ||||||||||||||||||||||||||||
| HGF - Research field: | Aeronautics, Space and Transport | ||||||||||||||||||||||||||||
| HGF - Program: | Space | ||||||||||||||||||||||||||||
| HGF - Program Themes: | Robotics | ||||||||||||||||||||||||||||
| DLR - Research area: | Raumfahrt | ||||||||||||||||||||||||||||
| DLR - Program: | R RO - Robotics | ||||||||||||||||||||||||||||
| DLR - Research theme (Project): | R - Intelligent Mobility (RM) [RO] | ||||||||||||||||||||||||||||
| Location: | Oberpfaffenhofen | ||||||||||||||||||||||||||||
| Institutes and Institutions: | Institute of Robotics and Mechatronics (since 2013) | ||||||||||||||||||||||||||||
| Deposited By: | Bustamante Gomez, Samuel | ||||||||||||||||||||||||||||
| Deposited On: | 08 Jul 2024 20:44 | ||||||||||||||||||||||||||||
| Last Modified: | 15 Jul 2024 11:10 |
Available Versions of this Item
- Grounding Embodied Question-Answering with State Summaries from Existing Robot Modules. (deposited 08 Jul 2024 20:44) [Currently Displayed]
Repository Staff Only: item control page
