Multimodal Assessment of Mental Workload during Automated Vehicle Remote Assistance: Modeling of Eye-Tracking-Related, Skin Conductance and Cardiovascular Indicators

Abstract

Remote assistance for highly automated vehicles (HAVs), i.e., third-party assistance from support staff outside the vehicle in times of the need for assistance, presents a solution to extend the capabilities of HAVs by integrating a third party for decision making in uncertain situations. Similar to other control center positions, we expect the remote assistance tasks to exert high mental demands on the human operators. Therefore, we assessed impact of elevated mental workload during HAV remote assistance in a controlled environment in a user study (N = 37) with the goal of identifying cues to differentiate workload levels based on eye-tracking-related, skin conductance, and cardiovascular indicators. The results provide evidence that (A) elevated workload induced via a secondary task depreciates performance, and (B) we can identify workload levels person-independently as differences in tonic skin conductance (F(2,72) = 24.538, p < 0.001, partial η² = 0.405) and pupil dilation (F(2,72) = 13.872, p < 0.001, partial η² = 0.278), resulting in a classification accuracy of 58% in a three-class classification task. The results provide evidence that we are able to differentiate operator workload during remote assistance in a time-resolved way with the ultimate goal to provide adaptations to counteract task deficiencies.