Skip to main navigation Skip to search Skip to main content

Visual-Textual Attention for Tree-Based Handwritten Mathematical Expression Recognition

  • Wei Liao
  • , Jiayi Liu
  • , Jianghan Chen
  • , Qiu Feng Wang*
  • , Kaizhu Huang
  • *Corresponding author for this work
    • Xi'an Jiaotong-Liverpool University
    • Duke Kunshan University

    Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

    Abstract

    Handwritten mathematical expression recognition (HMER) has attracted much attention and achieved remarkable progress under the encoder-decoder framework. However, it is still challenging due to complex structures and illegible handwriting. In this paper, we propose to refine the encoder-decoder framework for HMER. Firstly, we propose a multi-scale vision and textual attention fusion mechanism to enhance the contexts from both spatial and semantic information. Next, most of HMER works simply regard the HMER as a sequence-to-sequence problem (i.e., Latex string), ignoring the structure information in the mathematical expressions. To overcome this issue, we utilize a tree decoder to capture such structure contexts. Furthermore, we propose a parent-children mutual learning method to enhance the learning of our encoder-decoder model. Extensive experiments on the HMER benchmark datasets of CROHME 2014, 2016 and 2019 demonstrate the effectiveness of the proposed method.

    Original languageEnglish
    Title of host publicationAdvances in Brain Inspired Cognitive Systems - 13th International Conference, BICS 2023, Proceedings
    EditorsJinchang Ren, Amir Hussain, Iman Yi Liao, Rongjun Chen, Kaizhu Huang, Huimin Zhao, Xiaoyong Liu, Ping Ma, Thomas Maul
    PublisherSpringer Science and Business Media Deutschland GmbH
    Pages375-384
    Number of pages10
    ISBN (Print)9789819714162
    DOIs
    Publication statusPublished - 2024
    Event13th International Conference on Brain Inspired Cognitive Systems, BICS 2023 - Kuala Lumpur, Malaysia
    Duration: 5 Aug 2023 → 6 Aug 2023

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume14374 LNAI
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    Conference13th International Conference on Brain Inspired Cognitive Systems, BICS 2023
    Country/TerritoryMalaysia
    CityKuala Lumpur
    Period5/08/23 → 6/08/23

    Keywords

    • Handwritten mathematical expression recognition
    • Mutual learning
    • Tree decoder
    • Visual-textual attention

    Fingerprint

    Dive into the research topics of 'Visual-Textual Attention for Tree-Based Handwritten Mathematical Expression Recognition'. Together they form a unique fingerprint.

    Cite this