Towards a robust real time 3D reconstruction system for dynamic objects in static scenes

Kurzfassung

The surveillance of maritime infrastructures is an important part of maritime security. Especially harbours are highly active traffic zones with many different agents and infrastructures present. Surveillance of such areas therefore requires skilled operators, making fast security-related decisions. Situational awareness of these operators can be improved through the use of 3d reconstruction. It can be used as a tool to compress scene information and provide operators with more spatial and semantic detail about agents requiring less context switching. 3d reconstruction of static scenes has been an active area of research for several decades. Research on dynamic 3d reconstruction is still novel and mostly focused on autonomous navigation where a moving observer is present. In the context of surveillance, where the observer is static and scenes are dynamic, this work can not be readily applied. Furthermore, surveillance of maritime infrastructures often requires remote surveillance locations to cover large outdoor areas from an elevated point-of-view. To reduce system complexity and network bandwidth, 3d reconstruction can be performed using edge computing. However, research on dynamic 3d reconstruction for embedded systems is sparse. This work presents a prototype system that performs dynamic 3d reconstruction of dynamic agents in static scenes using an end-to-end parallel embedded architecture. The system is designed as a modular pipeline with different stages. First, static and dynamic image parts are segmented resulting in image masks for moving targets. Then, depth estimation is performed for the masked, dynamic targets. By removing the static image parts, traditional visual odometry techniques can be used to estimate target motion. This is due to the fact that the inverse ego-motion of the dynamic object can also be used to describe virtual camera around the object. Moving the virtual camera instead of assuming object motion, allows treating dynamic objects as static. Given these assumptions, the resulting color, depth and motion data can now be fused into a volumetric representation using truncated signed distance functions. The resulting reconstructed data can be sent over a network to perform real time rendering on client-side. Using real time 3d reconstructed data can improve situational awareness by aiding scene understanding and simplifying the decision-making context in complex maritime scenarios.