Visual-Textual Attention for Tree-Based Handwritten Mathematical Expression Recognition

Wei Liao, Jiayi Liu, Jianghan Chen, Qiu Feng Wang*, Kaizhu Huang

*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review


Handwritten mathematical expression recognition (HMER) has attracted much attention and achieved remarkable progress under the encoder-decoder framework. However, it is still challenging due to complex structures and illegible handwriting. In this paper, we propose to refine the encoder-decoder framework for HMER. Firstly, we propose a multi-scale vision and textual attention fusion mechanism to enhance the contexts from both spatial and semantic information. Next, most of HMER works simply regard the HMER as a sequence-to-sequence problem (i.e., Latex string), ignoring the structure information in the mathematical expressions. To overcome this issue, we utilize a tree decoder to capture such structure contexts. Furthermore, we propose a parent-children mutual learning method to enhance the learning of our encoder-decoder model. Extensive experiments on the HMER benchmark datasets of CROHME 2014, 2016 and 2019 demonstrate the effectiveness of the proposed method.

Original languageEnglish
Title of host publicationAdvances in Brain Inspired Cognitive Systems - 13th International Conference, BICS 2023, Proceedings
EditorsJinchang Ren, Amir Hussain, Iman Yi Liao, Rongjun Chen, Kaizhu Huang, Huimin Zhao, Xiaoyong Liu, Ping Ma, Thomas Maul
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages10
ISBN (Print)9789819714162
Publication statusPublished - 2024
Event13th International Conference on Brain Inspired Cognitive Systems, BICS 2023 - Kuala Lumpur, Malaysia
Duration: 5 Aug 20236 Aug 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14374 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference13th International Conference on Brain Inspired Cognitive Systems, BICS 2023
CityKuala Lumpur


  • Handwritten mathematical expression recognition
  • Mutual learning
  • Tree decoder
  • Visual-textual attention


Dive into the research topics of 'Visual-Textual Attention for Tree-Based Handwritten Mathematical Expression Recognition'. Together they form a unique fingerprint.

Cite this