Multi-Sensor and Multi-Modal Localization in Indoor Environments on Robotic Platforms

Abstract

This dissertation presents a detailed exploration of multi-sensor visual odometry systems, with a focus on enhancing robustness in indoor environments. The core contribution lies in the development of an advanced ego-state estimation framework, wherein visual odometry is bolstered by multiple visual sensors to overcome common challenges such as occlusions and textureless surfaces. By addressing frequent loss-of-tracking (LoT) events, the system ensures continuous, reliable localization in complex, cluttered indoor settings, such as households and elderly care facilities. To further augment situational awareness, sound source localization (SSL) is integrated as a complementary modality. Its fusion with visual data significantly enhances the robot’s perception of the environment, enabling the detection and identification of objects and events that may be visually occluded or otherwise undetectable. This multi-modal fusion provides a more holistic understanding of the robot’s surroundings, contributing to improved operational reliability in dynamic, human-centered environments. A key feature of this research is the introduction of IndoorMCD, a novel multisensor benchmark specifically designed to evaluate localization performance in indoor environments. Additionally, this work introduces URSim, an online real-time visual simulation framework that enables rigorous testing of multisensor localization systems under various conditions. Extensive experimental validation, using both real-world scenarios and simulated environments, demonstrates the robustness and fault tolerance of the proposed system. This research advances the state-of-the-art in robotic perception and indoor localization by providing a multi-modal, fault-tolerant approach to localization, offering valuable contributions to both theoretical understanding and practical application in robotics.