Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals

Nusrat J. Shoumy, Li Minn Ang, Kah Phooi Seng*, D. M.Motiur Rahaman, Tanveer Zia

*Corresponding author for this work

Research output: Contribution to journalReview articlepeer-review

84 Citations (Scopus)


Affective computing is an emerging multidisciplinary research field that is increasingly drawing the attention of researchers and practitioners in various fields, including artificial intelligence, natural language processing, cognitive and social sciences. Research in affective computing includes areas such as sentiment, emotion, and opinion modelling. The internet is an excellent source of data required for sentiment analysis, such as customer reviews of products, social media, forums, blogs, etc. Most of these data, called big data, are unstructured and unorganized. Hence there is a strong demand for developing suitable data processing techniques to process these rich and valuable data to produce useful information. Early surveys on sentiment and emotion recognition in the literature have been limited to discussions using text, audio, and visual modalities. So far, to the author's knowledge, a comprehensive survey combining physiological modalities with these other modalities for affective computing has yet to be reported. The objective of this paper is to fill the gap in this surveyed area. The usage of physiological modalities for affective computing brings several benefits in that the signals can be used in different environmental conditions, more robust systems can be constructed in combination with other modalities, and it has increased anti-spoofing characteristics. The paper includes extensive reviews on different frameworks and categories for state-of-the-art techniques, critical analysis of their performances, and discussions of their applications, trends and future directions to serve as guidelines for readers towards this emerging research area.

Original languageEnglish
Article number102447
JournalJournal of Network and Computer Applications
Publication statusPublished - 1 Jan 2020
Externally publishedYes


  • Affective applications
  • Affective computing
  • Multimodal fusion
  • Sentiment analysis
  • Sentiment databases


Dive into the research topics of 'Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals'. Together they form a unique fingerprint.

Cite this