Exploiting Attention-Consistency Loss For Spatial-Temporal Stream Action Recognition

Haotian Xu, Xiaobo Jin*, Qiufeng Wang, Amir Hussain, Kaizhu Huang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)


Currently, many action recognition methods mostly consider the information from spatial streams. We propose a new perspective inspired by the human visual system to combine both spatial and temporal streams to measure their attention consistency. Specifically, a branch-independent convolutional neural network (CNN) based algorithm is developed with a novel attention-consistency loss metric, enabling the temporal stream to concentrate on consistent discriminative regions with the spatial stream in the same period. The consistency loss is further combined with the cross-entropy loss to enhance the visual attention consistency. We evaluate the proposed method for action recognition on two benchmark datasets: Kinetics400 and UCF101. Despite its apparent simplicity, our proposed framework with the attention consistency achieves better performance than most of the two-stream networks, i.e., 75.7% top-1 accuracy on Kinetics400 and 95.7% on UCF101, while reducing 7.1% computational cost compared with our baseline. Particularly, our proposed method can attain remarkable improvements on complex action classes, showing that our proposed network can act as a potential benchmark to handle complicated scenarios in industry 4.0 applications.

Original languageEnglish
Article number119
JournalACM Transactions on Multimedia Computing, Communications and Applications
Issue number2 S
Publication statusPublished - 6 Oct 2022


  • Action recognition
  • attention consistency
  • multi-level attention
  • two-stream structure


Dive into the research topics of 'Exploiting Attention-Consistency Loss For Spatial-Temporal Stream Action Recognition'. Together they form a unique fingerprint.

Cite this