Graph Convolutional Networks for Frustration Recognition of Drivers

Abstract

High frustration levels during driving are a critical problem for road safety during manual driving and play an important role in the user experience on all automation levels. For this reason, detecting frustration is essential to build more user-focused systems that adapt their behaviour to the user's needs. In this thesis we propose a fully automated method to detect frustration levels of drivers. The method was trained with a unique and labeled dataset containing real life driving situations, collected in a driving simulator with 50 participants. The video recordings were labeled with a post-hoc continuous frustration rating. The first step of our method was to extract facial landmarks from the video recordings to build a graph, representing facial structures at each frame. The graph was then used to train state-of-the-art Spatio-Temporal Graph Convolutional Networks with residual links between layers. The network is formed of three input branches, containing landmark, velocity, and edge features, each processed first separately and then concatenated into one main network stream. The model was additionally trained combining the information from high frequency and low frequency data, using a two-path network, to capture both fast movements and the semantics of the data. The results show promising results for the real-time recognition of frustration levels, with an accuracy of 79.4±7.4 % when allowing a prediction difference of 0.1. The model presents a new method to get objective frustration ratings from video feeds of drivers by using subjective labels of several drivers to train our network.