3D Human Pose Reconstruction From Monocular Images for Emergency Management in Digital Twins

Kurzfassung

Multi-person pose detection is a research topic with vast potential for monitoring and decision support in emergency management. Current research outlines use cases for crowd monitoring, pose detection for automated signaling of emergencies, and detection of suspicious activities. In addition, by extracting behaviors from detected poses agent-based crowd simulations can be conducted. These use cases overlap with application areas of Digital Twins (DTs). Many DT use cases rely on movement and pose data of persons, e.g., for crowd emergency management, decision support for building evacuation, and anomaly detection in public spaces. This highlights the need to continuously include pose data of real-world detected persons in DT models for monitoring and emergency management purposes. While the detection of persons in camera images is a thoroughly researched topic, the reconstruction of a 3D poses within an environment remains a more complex challenge. In addition, there is limited research on integrating humanoid 3D poses into DTs for end-user visualization. To address this challenge, we propose a process chain that first extracts image-space poses using established deep learning methods. After this step, novel approaches are presented to reconstruct 3D poses and integrate them into a DT. A reference implementation is developed, using MMPose as foundation for pose extraction, the FIWARE Orion Broker as a backbone of the DT data model, and a novel streaming service to send pose data to the Unreal Engine for user-centric visualization. The results demonstrate that the system is capable of accurately reconstructing and visualizing continuos 3D poses of detected persons, which enables further research on DT applications relying on pose data and exact positioning of entities within the 3D space.