HGR-ViT: Hand Gesture Recognition with Vision Transformer

Chun Keat Tan, Kian Ming Lim*, Roy Kwang Yang Chang, Chin Poo Lee, Ali Alqahtani

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

Hand gesture recognition (HGR) is a crucial area of research that enhances communication by overcoming language barriers and facilitating human-computer interaction. Although previous works in HGR have employed deep neural networks, they fail to encode the orientation and position of the hand in the image. To address this issue, this paper proposes HGR-ViT, a Vision Transformer (ViT) model with an attention mechanism for hand gesture recognition. Given a hand gesture image, it is first split into fixed size patches. Positional embedding is added to these embeddings to form learnable vectors that capture the positional information of the hand patches. The resulting sequence of vectors are then served as the input to a standard Transformer encoder to obtain the hand gesture representation. A multilayer perceptron head is added to the output of the encoder to classify the hand gesture to the correct class. The proposed HGR-ViT obtains an accuracy of 99.98%, 99.36% and 99.85% for the American Sign Language (ASL) dataset, ASL with Digits dataset, and National University of Singapore (NUS) hand gesture dataset, respectively.

Original languageEnglish
Article number5555
JournalSensors
Volume23
Issue number12
DOIs
Publication statusPublished - Jun 2023
Externally publishedYes

Keywords

  • attention
  • hand gesture recognition
  • sign language recognition
  • vision transformer
  • ViT

Fingerprint

Dive into the research topics of 'HGR-ViT: Hand Gesture Recognition with Vision Transformer'. Together they form a unique fingerprint.

Cite this