Advancing Interaction with Massive Point Clouds in Immersive VR

Abstract

Over the past several decades, rapid advances in 3D sensing have yielded vast, detailed point cloud datasets critical for machine learning, yet traditional 2D annotation workflows remain inefficient and error-prone. This work presents a novel open-source VR-based point cloud labelling tool developed in Unreal Engine that immerses users in a fully interactive 3D environment for intuitive annotation. By modifying Unreal Engine's LiDAR Point Cloud Plugin, real-time, GPU-accelerated labelling of point clouds with over 150 million points at (86±21) FPS is achieved. A novel sphere tracing-based labelling algorithm for point selection is proposed, which dynamically adapts to complex point cloud structures at variable distances, ensuring precise annotation across diverse scenarios. Furthermore, a literature review of VR-based labelling tools is provided, detailing software environments, hardware configurations, performance metrics, and interaction techniques. These contributions expand the range of VR annotation tools and establish a new performance baseline, paving the way for future hybrid intelligence systems. The open-source nature of the modifications invites further development in immersive 3D data annotation.