Deep Learning Based on Hierarchical Self-Attention for Finance Distress Prediction Incorporating Text

Sumei Ruan, Xusheng Sun, Ruanxingchen Yao, Wei Li*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)


To detect comprehensive clues and provide more accurate forecasting in the early stage of financial distress, in addition to financial indicators, digitalization of lengthy but indispensable textual disclosure, such as Management Discussion and Analysis (MD&A), has been emphasized by researchers. However, most studies divide the long text into words and count words to treat the text as word count vectors, bringing massive invalid information but ignoring meaningful contexts. Aiming to efficiently represent the text of large size, an end-To-end neural networks model based on hierarchical self-Attention is proposed in this study after the state-of-The-Art pretrained model is introduced for text embedding including contexts. The proposed model has two notable characteristics. First, the hierarchical self-Attention only affords the essential content with high weights in word-level and sentence-level and automatically neglects lots of information that has no business with risk prediction, which is suitable for extracting effective parts of the large-scale text. Second, after fine-Tuning, the word embedding adapts the specific contexts of samples and conveys the original text expression more accurately without excessive manual operations. Experiments confirm that the addition of text improves the accuracy of financial distress forecasting and the proposed model outperforms benchmark models better at AUC and F2-score. For visualization, the elements in the weight matrix of hierarchical self-Attention act as scalers to estimate the importance of each word and sentence. In this way, the "red-flag"statement that implies financial risk is figured out and highlighted in the original text, providing effective references for decision-makers.

Original languageEnglish
Article number1165296
JournalComputational Intelligence and Neuroscience
Publication statusPublished - 2021
Externally publishedYes


Dive into the research topics of 'Deep Learning Based on Hierarchical Self-Attention for Finance Distress Prediction Incorporating Text'. Together they form a unique fingerprint.

Cite this