Enhancing the Communication of Speech-Impaired People Using Embedded Vision-based Gesture Recognition through Deep Learning

Communication between people is the key to delivering a message. It is easier for normal people to have a communication medium (language) known between them. A person with speech impairment or hearing difficulty cannot communicate with others like a normal human. Sign language helps people with disabilities to communicate with each other. In sign language systems, there is no de facto standard followed by all the countries in the world. It is not easy to get recognized using sign language alone. Hence, recognition systems are required to improve their communication capabilities. The rapid growth in the field of Artificial Intelligence motivated us to build a gesture recognition system based on machine learning and/or deep learning techniques for improved performance. In this chapter, an image-based recognition system for American Sign Language (ASL) is designed using 1. Handcrafted features classified by Machine Learning algorithms, 2. classification using a pre-trained model through transfer learning and 3. classification of deep features extracted from a particular layer by machine learning classifiers. Among these three approaches, deep features extracted from DenseNet and classification using K-Nearest Neighbor (K-NN) yield the highest accuracy of about 99.2%. To make this system handy, low cost, and available to needy people, the Resnet 50 model is deployed in a Raspberry Pi 3b + microcontroller.

Keywords: Speech impaired, Artificial intelligence, Communication, Deep features, Gesture recognition, Machine learning, Transfer learning, Deep learning.