A Dynamic Emotion Recognition System Based on Convolutional Feature Extraction and Recurrent Neural Network

Over the past three decades, there has been sustained research activity in emotion recognition from faces, powered by the popularity of smart devices and the development of improved machine learning, resulting in the creation of recognition systems with high accuracy. While research has commonly focused on single images, recent research has also made use of dynamic video data. This paper presents CNN-RNN (Convolutional Neural Network - Recurrent Neural Network) based emotion recognition using videos from the ADFES database, and we present the results in the arousal-valence space, rather than assigning a discrete emotion. As well as traditional performance metrics, we also design a new performance metric, PN accuracy, to distinguish between positive and negative emotions. We demonstrate improved performance with a smaller RNN than the initial pre-trained model, and report a peak accuracy of 0.58, with peak PN accuracy of 0.76, which shows our approach is very capable distinguishing between positive and negative emotions. We also present a detailed analysis of system performance, using new valence-arousal domain temporal visualisations to show transitions in recognition over time, demonstrating the importance of context based information in emotion recognition.