A Dynamic Emotion Recognition System Based on Convolutional Feature Extraction and Recurrent Neural Network

Yida Yin, Misbah Ayoub, Andrew Abel*, Haiyang Zhang

*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

1 Citation (Scopus)


Over the past three decades, there has been sustained research activity in emotion recognition from faces, powered by the popularity of smart devices and the development of improved machine learning, resulting in the creation of recognition systems with high accuracy. While research has commonly focused on single images, recent research has also made use of dynamic video data. This paper presents CNN-RNN (Convolutional Neural Network - Recurrent Neural Network) based emotion recognition using videos from the ADFES database, and we present the results in the arousal-valence space, rather than assigning a discrete emotion. As well as traditional performance metrics, we also design a new performance metric, PN accuracy, to distinguish between positive and negative emotions. We demonstrate improved performance with a smaller RNN than the initial pre-trained model, and report a peak accuracy of 0.58, with peak PN accuracy of 0.76, which shows our approach is very capable distinguishing between positive and negative emotions. We also present a detailed analysis of system performance, using new valence-arousal domain temporal visualisations to show transitions in recognition over time, demonstrating the importance of context based information in emotion recognition.

Original languageEnglish
Title of host publicationIntelligent Systems and Applications - Proceedings of the 2022 Intelligent Systems Conference IntelliSys Volume 2
EditorsKohei Arai
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages21
ISBN (Electronic)978-3-031-16078-3
ISBN (Print)978-3-031-16077-6
Publication statusPublished - 2023
EventIntelligent Systems Conference, IntelliSys 2022 - Virtual, Online
Duration: 1 Sept 20222 Sept 2022

Publication series

NameLecture Notes in Networks and Systems
Volume543 LNNS
ISSN (Print)2367-3370
ISSN (Electronic)2367-3389


ConferenceIntelligent Systems Conference, IntelliSys 2022
CityVirtual, Online


  • Convolutional neural network
  • Deep learning
  • Emotion recognition
  • Recurrent neural network
  • Visualisation


Dive into the research topics of 'A Dynamic Emotion Recognition System Based on Convolutional Feature Extraction and Recurrent Neural Network'. Together they form a unique fingerprint.

Cite this