Frustration-aware assistance systems - Assessment and Causes of In-Vehicle Frustration

Kurzfassung

Frustration is an emotion that occurs when goal-directed behavior is hindered. This frequently happens in transportation, for example in driving. Problematically, frustration can lead to risky driving behavior when a human driver is in charge of the vehicle. Furthermore, frustration may hinder the acceptance of innovative automated vehicles. The option to automatically recognize negative emotions, such as frustration, of vehicle users with so-called affect-aware systems has gained increasing attention within the last few years. These systems enable to adapt vehicle functions, such as the human-machine interface, in real-time depending on the current traveler state and corresponding needs. However, the automated recognition of emotion requires high-quality data sets to train algorithms on. These are insufficiently present in the affective computing community so far. Furthermore, a wide variety of measures for affect recognition exist, but methods to compare different modalities for measures of frustration are lacking. Previous research found that emotional expressions in the face and body form promising indicators for user frustration. Previous studies have investigated expressions of frustration in the context of driving and mobility but have neglected interindividual differences. Furthermore, knowledge of possible causes of frustration is needed to successfully mitigate frustration, which are yet unknown. To properly design frustration-aware systems and to develop methods to capture frustration, it is, therefore, necessary to 1) provide a training dataset, 2) find a method to compare different modalities of frustration recognition, 3) improve recognition of frustration by facial expressions and 4) investigate causes of frustration in driving. This dissertation did so and thereby enabled a more reliable recognition of frustration in driving. This could make manual driving safer and contribute to the development of affect-aware systems, which also have the potential to facilitate the acceptance of automated driving systems. In total, this dissertation presents three studies published in four papers. Paper 1 of this dissertation presents high-quality and continuously frustration-labeled expression data that we provide for the research community. It contains a thorough description of the data and a benchmark algorithm that automatically recognizes frustration in video data. Paper 2 found that in addition to previously described frustration-typical expressions, individual-typical expressions of frustration exist. Paper 3 presents a latent variable model that can evaluate which measurements for frustration are necessary. Finally, paper 4 investigated causes and coping strategies for frustration in driving through a diary study and a focus group study. The overall goal of this dissertation is to contribute to the underlying research necessary to develop frustration-aware assistance systems. Based on the findings of the three studies, this dissertation helps to expand our knowledge of how to measure in-vehicle frustration. The discussion highlights this dissertation’s contributions for such a development, but also points out limitations of the current studies. Ethical aspects of automated emotion recognition are discussed.