Towards Detecting Semantic Deviations Between Digital Twins and Reality

Kurzfassung

Digital twins (DTs) have emerged as a promising technology for improving the resilience and operational reliability of critical infrastructures. By coupling virtual representations of physical assets with real-world systems, DTs enable continuous monitoring, simulation of what-if scenarios, and enhanced decision support during both normal operation and crisis situations. However, the reliability of DT-based analyses strongly depends on maintaining an accurate and up-to-date representation of the physical system. Real-world infrastructures continuously evolve due to maintenance activities, environmental influences, and unforeseen events, leading to discrepancies between the DT and its physical counterpart. Detecting such inconsistencies is therefore essential for reliable monitoring and crisis management.

While anomaly and change detection are well-established research areas in computer vision, existing approaches typically compare observations originating from the same visual domain, such as multi-temporal surveillance or satellite imagery. In contrast, DT consistency verification requires cross-domain comparison between real-world observations and simulated or rendered DT representations. Since DTs often prioritize semantic structure and functional consistency over photorealistic rendering, substantial domain gaps arise in terms of appearance, texture, and level of detail, limiting the applicability of methods based on low-level visual similarity.

To address this challenge, we present an initial pipeline for detecting discrepancies between DTs and their physical counterparts by combining real-world visual observations with DT-derived semantic scene priors. The proposed approach enables the spatial localization of semantically relevant deviations and supports the continuous alignment between DTs and reality. Preliminary experiments conducted in representative urban scenarios demonstrate the feasibility of the concept. Although evaluated in an urban environment setting, the approach is largely application-agnostic and transferable to other domains, including drone-based monitoring of railway infrastructure.