Multimodal Estimation of Frustrative Driving Situations Using a Latent Variable Model

Abstract

Within recent years, it has become popular to use physiological and expression data to determine appropriate moments for information communication in human-machine interaction. One emotion that is highly relevant to inform systems about these appropriate time points is frustration, which occurs when a user’s goal in using the system is failed to be met. In the context of transportation, frustration can lead to risky behavior when driving manually and diminished acceptance of automated cars. It is therefore crucial to mitigate this negative emotion, for example by implementing frustration-aware systems that can recognize frustration and react accordingly. Automated recognition of frustration has previously been achieved by various measures obtained from the user or driver of the vehicle via classical regression or automated classification. This paper presents a latent variable model that estimates frustration in two different contexts by continuous subjective frustration rating, facial expressions and frontal alpha asymmetry in the electroencephalogram. We then compare this full model to models with less measurement variables to evaluate which measurements can be left out. Our results show that expression frequency and subjective frustration make important contributions to the model of experienced frustration. This paper presents a proof of concept for using a latent variable model to evaluate collected measures to estimate an experienced emotion. This method can inform researchers which measurements are most informative in different circumstances. Additionally, the method can be used to evaluate how well purely objective measurements (that are the only feasible measurements in most applied settings) perform in comparison to a model including subjective ratings.