The importance of considering individual differences in expression for in-vehicle emotion recognition - a machine learning approach

Abstract

High frustration levels during driving are a critical problem for road safety in manual driving and play an essential role in the user experience in all automation levels. Therefore, the development of frustration-sensitive systems is an important step for building user focused systems that adapt its behavior to the user's needs. In this paper, we propose a fully automated process to detect frustration of users in video recordings from in-vehicle driver monitoring. This work uses a unique dataset containing frustrating driving situations that reflect real-world scenarios, which was collected in a driving simulator study with 50 participants with 6 drives each. Following each drive, the participants gave a post-hoc continuous frustration rating. Previous emotion research focused on finding universal expressions of emotion. However, scientific evidence suggests that each individual has a unique set of individual-specific expressions of frustration. Therefore, this work identifies frustration patters on an individual basis. Facial muscle movements have shown to be a promising indicator for emotion recognition. 19 facial muscle movements were detected with the commercially available software FACET and later used to train a Support Vector Machine and Gradient Boosted Tree. The method shows promising results in automated in-vehicle frustration recognition, which is a primary step towards the development of user experience sensitive driving assistance.