Camera Self-Health-Maintenance by means of Sensor Artificial Intelligence

Abstract

Autonomous machines require increasingly more robustness and reliability to meet the demands of modern tasks. These requirements specially apply to cameras onboard such machines, as they are the predominant sensors acquiring information about the environment to support decision making and actuation. Hence, the cameras must maintain their own functionality. This poses a significant challenge, primarily driven by the variety of existing cameras, the vast amount of potential application scenarios, and the limited machine resources, all while demanding real-time performance. Existing solutions are typically tailored to specific problems or detached from the downstream computer vision tasks of the machines, which, however, determine the requirements on the quality of the produced camera images. This thesis presents a camera self-health-maintenance framework to bridge this gap. The approach combines a generalized condition monitoring and a task-oriented decision & control unit. The monitoring is based on novel learning-based blur and noise estimators that incorporate physical knowledge about the camera to increase consistency and robustness. Especially the incorporation of camera metadata enables the system to disambiguate the contributions of different noise processes within a camera. In this manner alone, the decision & control unit can initiate appropriate countermeasures, if necessary. To this end, camera parameters are readjusted based on an empirical image task analysis to optimize performance under any situation. The framework is evaluated on synthetic and real datasets from transportation and robotic scenarios in terms of accuracy, robustness and real-time capability. Firstly, the blur and noise estimators are examined and two extensions are analyzed, which recover the estimation of combined blur/noise corruptions and reduce estimation uncertainties, respectively. Secondly, the effect of an acquired image and the camera`s metadata on noise source estimation is investigated. This method is further demonstrated on the detection of mismatches between both inputs (image and camera metadata) to quantify unexpected noise as from camera defects. Lastly, the framework is implemented and verified on a real robot system. The real demonstration on a robot shows promising results to employ the framework for arbitrary mobile machines in unknown environments. In particular, the proposed framework outperforms standard camera parameter controllers. Yet, the results also highlight current limitations that require framework extensions in future studies, such as the application to complex non-linear motion blur and scenes with high dynamic light intensity ranges.