Recognition and Control of a Chinese Pinyin Sign Language Robot via a Cognitive Robotics Approach

Technological advancements in AI and Robotics are facilitating the development of sign language robots, which are resolving communication barriers for people living with disabilities. The current work aims to develop a sign language robot that mimics how human communicate in Chinese Sign Language. Two primary developments are presented in this paper, namely the real-time translation of sign language and the movement production of a simulated dexterous hand. Real-time translation was performed using computer vision and deep learning techniques, and the movement production of a dexterous hand to represent the sign language gestures implemented forward kinematics. The robot vision system can detect and tracking human hand, allowing real-time highly accurate recognition of hand positions and shapes. Employing the VGG16 architecture and customised training dataset, it was further developed to identify sign language gestures, achieving an accuracy rate of 80% between ambiguities. The simulation of a bionic dexterous hand was realised in MATLAB Simulink. The simulated bionic dexterous hand can provide real-time feedback on displacement data and adjusting input signals for enhanced control. The two developments of the current work could be easily integrated and prototyped as robotic hand which could behave on par with human-like communications via sign languages beyond Chinese Pinyin. It could also lead to the development of better human-robot interaction systems for people living with hard-of-hearing disabilities.