Exploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction

Guofeng Yi, Yuguang Yang, Yu Pan, Yuhang Cao, Jixun Yao, Xiang Lv, Cunhang Fan, Zhao Lv, Jianhua Tao, Shan Liang*, Heng Lu*

*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

3 Citations (Scopus)

Abstract

utilize multimodal data to predict the intensity of three emotional categories. In our work, we discovered that integrating multiple dimensions, modalities, and levels enhances the effectiveness of emotional judgment. In terms of feature extraction, we utilize over a dozen types of medium backbone networks, including W2V-MSP, GLM, and FAU, which are representative of audio, text, and video modalities, respectively. Additionally, we utilize the LoRA framework and employ various domain adaptation methods to effectively adapt to the task at hand. Regarding model design, apart from the RNN model in the baseline, we have extensively incorporated our transformer variant and multi-modal fusion model. Finally, we propose a Hyper-parameter Search Strategy (HPSS) for late fusion to further enhance the effectiveness of the fusion model. For the MuSe-MIMIC, our method achieves Pearson's Correlation Coefficient of 0.7753, 0.7647, and 0.6653 for Approval, Disappointment, and Uncertainty, respectively, outperforming the baseline system by a large margin (i.e., 0.5536, 0.5139, and 0.3395) on the test set. The final mean pearson is 0.7351, surpassing all other participants and ranking Top 1.

Original languageEnglish
Title of host publicationMuSe 2023 - Proceedings of the 4th Multimodal Sentiment Analysis Challenge and Workshop
Subtitle of host publicationMimicked Emotions, Humour and Personalisation, Co-located with: MM 2023
PublisherAssociation for Computing Machinery, Inc
Pages19-26
Number of pages8
ISBN (Electronic)9798400702709
DOIs
Publication statusPublished - 2 Nov 2023
Externally publishedYes
Event4th Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2023, In conjunction with 31st ACM Multimedia 2023 - Ottawa, Canada
Duration: 29 Oct 2023 → …

Conference

Conference4th Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2023, In conjunction with 31st ACM Multimedia 2023
Country/TerritoryCanada
CityOttawa
Period29/10/23 → …

Keywords

  • hyper-parameter search strategy
  • multimodal fusion
  • multimodal sentiment analysis

Fingerprint

Dive into the research topics of 'Exploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction'. Together they form a unique fingerprint.

Cite this