TY - JOUR
T1 - Attention-Augmented Machine Memory
AU - Lin, Xin
AU - Zhong, Guoqiang
AU - Chen, Kang
AU - Li, Qingyang
AU - Huang, Kaizhu
N1 - Funding Information:
This work was partially supported by the Major Project for New Generation of AI under Grant No. 2018AAA0100400, the Joint Fund of the Equipments Pre-Research and Ministry of Education of China under Grant No. 6141A020337, the National Natural Science Foundation of China under Grant No. 61876155, the Natural Science Foundation of Shandong Province, China, under Grant No. ZR201911080230, the Jiangsu Science and Technology Programme (Natural Science Foundation of Jiangsu Province) under Grant No. BE2020006-4 and BK20181189, the Project for Graduate Student Education Reformation and Research of Ocean University of China under Grant No. HDJG19001, and the Key Program Special Fund in XJTLU under Grant No. KSF-T-06 and KSF-E-26. The authors would like to thank Zhaoyang Niu for his help in the revision of this paper.
Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.
PY - 2021/5
Y1 - 2021/5
N2 - Attention mechanism plays an important role in the perception and cognition of human beings. Among others, many machine learning models have been developed to memorize the sequential data, such as the Long Short-Term Memory (LSTM) network and its extensions. However, due to lack of the attention mechanism, they cannot pay special attention to the important parts of the sequences. In this paper, we present a novel machine learning method called attention-augmented machine memory (AAMM). It seamlessly integrates the attention mechanism into the memory cell of LSTM. As a result, it facilitates the network to focus on valuable information in the sequences and ignore irrelevant information during its learning. We have conducted experiments on two sequence classification tasks for pattern classification and sentiment analysis, respectively. The experimental results demonstrate the advantages of AAMM over LSTM and some other related approaches. Hence, AAMM can be considered as a substitute of LSTM in the sequence learning applications.
AB - Attention mechanism plays an important role in the perception and cognition of human beings. Among others, many machine learning models have been developed to memorize the sequential data, such as the Long Short-Term Memory (LSTM) network and its extensions. However, due to lack of the attention mechanism, they cannot pay special attention to the important parts of the sequences. In this paper, we present a novel machine learning method called attention-augmented machine memory (AAMM). It seamlessly integrates the attention mechanism into the memory cell of LSTM. As a result, it facilitates the network to focus on valuable information in the sequences and ignore irrelevant information during its learning. We have conducted experiments on two sequence classification tasks for pattern classification and sentiment analysis, respectively. The experimental results demonstrate the advantages of AAMM over LSTM and some other related approaches. Hence, AAMM can be considered as a substitute of LSTM in the sequence learning applications.
KW - Attention mechanism
KW - Attention-augmented machine memory (AAMM)
KW - Long short-term memory (LSTM)
KW - Machine learning
KW - Sequence classification
UR - https://www.scopus.com/pages/publications/85103170248
U2 - 10.1007/s12559-021-09854-5
DO - 10.1007/s12559-021-09854-5
M3 - Article
AN - SCOPUS:85103170248
SN - 1866-9956
VL - 13
SP - 751
EP - 760
JO - Cognitive Computation
JF - Cognitive Computation
IS - 3
ER -