Recognition and Control of a Chinese Pinyin Sign Language Robot via a Cognitive Robotics Approach

Hanzhang Cui, Yang Luo, Fan Zhang*

*Corresponding author for this work

Research output: Contribution to conferencePaperpeer-review

28 Downloads (Pure)

Abstract

Technological advancements in AI and Robotics are facilitating the devel-opment of sign language robots, which are resolving communication barri-ers for people living with disabilities. The current work aims to develop a sign language robot that mimics how human communicate in Chinese Sign Language. Two primary developments are presented in this paper, namely the real-time translation of sign language and the movement production of a simulated dexterous hand. Real-time translation was performed using com-puter vision and deep learning techniques, and the movement production of a dexterous hand to represent the sign language gestures implemented for-ward kinematics. The robot vision system can detect and tracking human hand, allowing real-time highly accurate recognition of hand positions and shapes. Employing the VGG16 architecture and customised training da-taset, it was further developed to identify sign language gestures, achieving an accuracy rate of 80% between ambiguities. The simulation of a bionic dexterous hand was realised in MATLAB Simulink. The simulated bionic dexterous hand can provide real-time feedback on displacement data and adjusting input signals for enhanced control. The two developments of the current work could be easily integrated and prototyped as robotic hand which could behave on par with human-like communications via sign lan-guages beyond Chinese Pinyin. It could also lead to the development of bet-ter human-robot interaction systems for people living with hard-of-hearing disabilities.
Original languageEnglish
Publication statusAccepted/In press - 22 Aug 2024
EventInternational Conference on Intelligent Manufacturing and Robotics 2024 - Taicang, Suzhou, China
Duration: 22 Aug 202423 Aug 2024
https://www.xjtlu.edu.cn/en/study/departments/school-of-intelligent-manufacturing-ecosystem/icimr2024

Conference

ConferenceInternational Conference on Intelligent Manufacturing and Robotics 2024
Abbreviated titleICiMR 2024
Country/TerritoryChina
CityTaicang, Suzhou
Period22/08/2423/08/24
Internet address

Fingerprint

Dive into the research topics of 'Recognition and Control of a Chinese Pinyin Sign Language Robot via a Cognitive Robotics Approach'. Together they form a unique fingerprint.

Cite this