Assisting the Remote Assistant: Augmenting Degraded Video Streams with Additional Sensor Data to Improve Situation Awareness in Complex Urban Traffic

Abstract

Remotely operating highly automated vehicles (HAVs, SAE 4 [1]) bears the potential to boost their large-scale deployment. A human operator supports the vehicle automation remotely in situations that exceed the automation’s capabilities. A high-quality video stream displaying the HAV’s environment is key for the remote operator to obtain and maintain situation awareness. One of the major technical obstacles is the frequent and serious limitation of connectivity between the remote operator and the HAV, particularly a drop of bandwidth. This results in a severely degraded video resolution. As a remedy, we propose the use of data transmitted from the HAV’s additional onboard sensors to augment a low-resolution video stream. A significantly lower bandwidth suffices to transmit sensor compared to video data, enabling their transmission even when the video resolution is degraded. This could help the remote operator gain situation awareness, particularly in remote assistance, a variant of remote operation in which the operator provides high-level advice to the vehicle [2]. To confirm the need for a sensor data augmented view of the traffic situation, an experimental online user study (N = 117) was conducted. The study presented short video clips of complex naturalistic urban road traffic to participants. The objective was to examine if overlaying the video stream with visualized sensor data improves a remote assistant’s situation awareness and whether the effect of overlaid sensor data depends on the video resolution. Results revealed a significant effect of video resolution on objective situation awareness. Additionally, an interaction effect between resolution and sensor-data overlay became evident on the perception level of situation awareness. Hence, sensor data augmentation of degraded video streams may support the remote assistant’s situation awareness by increasing the salience of relevant elements in a traffic situation. Future research will investigate sensor data augmentation in a standardized simulation environment.