Development of an Adaptive and User-centered HMI for Connected and Automated Vehicles

Abstract

Highly automated vehicles (HAVs; SAE level of automation 4) are likely to be an essential element of future mobility solutions. However, even with advanced automated driving systems (ADS), human intervention and support will be required for safe, reliable and inclusive driving operations. One approach to integrate a human into the system is the remote assistance of HAVs. A remote assistant supports the HAV from a distance, e.g., a remote operations center, by being displayed key information from the HAV and the surrounding traffic environment. When a need for assistance emerges, e.g., in the event of reaching ODD limits, edge, or corner cases, the remote assistant is notified and asked to aid the HAV by reviewing the situation and providing input to the ADS without executing the dynamic driving task. To interact with the ADS, the remote assistant uses a suitable human-machine interface (HMI). Moreover, an in-vehicle HMI is required to keep passengers on board of the HAV up to date on the remote assistant's intervention. One way to meet user needs in designing these HMIs is to apply user-centered design methods. The presenters' PhD projects follow the user-centered design process to develop and evaluate solutions for HMIs in the event of remote assistance, for both the remote assistant in a control center and for the passengers of remotely assisted HAVs. Focusing on the HMI for the remote assistant, a comprehensive prototype of a remote assistance workstation was initially introduced and evaluated from a Human Factors perspective. This baseline prototype yielded good results regarding performance, workload, and situation awareness (Schrank et al., 2024a). While the initial HMI design predominantly relied on video streams to provide situational information, a second PhD study explored the potential of augmenting the video stream with additional sensor data. These data were visualized to make road users that were difficult to perceive more salient, particular in situations with poor visibility such as fog. The augmentation helped to significantly reduce the number of collisions overall but especially in foggy conditions (Schrank et al., 2024b). The augmentation applied in the study was the output of an assumed adaptive system: It was only displayed for road users an interaction was likely with. Thus, the augmentation was adapted to likelihood for interaction in the foreseeable future. Focusing on the HMI for passengers of remotely assisted HAVs, an In-vehicle HMI for passengers was investigated. Results provided evidence that providing information about the remote assistance resolution process improved passengers' understanding and experience significantly (Brandt et al., 2024a). In a follow-up study, design principles for presenting this information were investigated, utilizing media-richness as a vehicle to improve user experience of passengers of remotely assisted vehicles further (Brandt et al., 2024b). In this forum session, the presenters intend to explore together with participants and experts how to extend their research on both adaptive HMIs for remote assistance and multimodal in-vehicle HMIs for passenger communication in their following PhD studies. In particular, they want to brainstorm ideas on the further refinement of the perception-based HMI for remote assistance from previous studies and conceptualizations for adaptive HMIs beyond that, including, but not limited to, intention-based HMIs. Additionally, possibilities to provide more information about the system according to passengers' needs will be brainstormed. Implications of Human Factors research for these HMI solutions will be discussed.