Enabling Conversational Intelligence in Service Robots Through Multimodal Fusion and Resource-Aware Generative Models

Abstract

The integration of generative AI, particularly large language models (LLMs) and vision-language models (VLMs), into human-robot interaction (HRI) systems is unlocking new capabilities for natural, context-aware communication. This workin-progress paper presents a modular multimodal interaction framework that enables service robots to engage in socially intelligent and emotionally responsive conversations in near realtime. Our system fuses vision and speech through a unified architecture, allowing the robot to understand user intent, emotional state, and context in a laboratory environment. By leveraging quantized generative models like LLaVA and efficient deep learning components for speech recognition and face detection, the proposed framework is designed with resource-constrained deployment in mind. While deployment is in an experimental stage, Preliminary results from module integration on development setups suggest the potential of generative multimodal AI in enabling adaptive, explainable, and collaborative human-robot interactions.