Communication Aid for Deaf and Dumb
Guided by S.Indhumathi (Asst. Professor O.G)
Sabresh Mouli
ECE Department
SRM Institute of Science and Technology,
Chennai, India-600026
sabresh.mouli@gmail.com
Abstract— The Deaf and Dumb are facing many
challenges, and are craving for self-dependence. The above
mentioned cause was using flex sensors to record their
gestures. But we are taking this a step further and creating
a platform in which Deaf and Dumb can interact with a
normal person. They can use the sign language to
communicate with the normal person, and the sign
language will be converted into text and the other person’s
text message will be converted into sign language. The
technology we have used is computer vision and deep
learning which comes under the broad umbrella of
artificial intelligence. The processor which is going to be
used is Raspberry Pi. The programming language that will
be used is Python. The technology will be useful in all the
public sector place in which they can get what they want.
This will help the recruiters in recruiting them and a deaf
and dumb person can present papers in large seminars
and conferences.
Keywords—Artificial Intelligence, Computer Vision,
Deep Learning
I. INTRODUCTION
The sign language used as a means of communication between
deaf and dumb is unadapted for private communications, as
the language spoken to one is spoken to all present who
understand it. Spoken language can be whispered, or its
volume can be so reduced as to be inaudible to other ears than
those for which it is intended; but the force of the sign
language cannot thus be modified, and when private
conversations are held, written language is generally
employed. Besides the tediousness of this process cannot be
always resorted.
To date, several researchers have tried to tackle the problem of
communication for deaf and dumb but haven’t been able to
establish a two-way communication between them. Two-way
communication can be established with the help of computer
vision, deep learning and machine learning. The overall
contribution of this paper is as follows:
Mrinal Bage
ECE Department
SRM Institute of Science and Technology,
Chennai, India-600026
mrinalbage123@gmail.com
 To establish a two-way communication system using
deep learning and computer vision.
 To train the system with 26 alphabet sign language
with different hand textures and to use the same
database for the two-way communication.
 The use of artificial intelligence will make two-way
communication possible for deaf and dumb.
II. RELATED WORK
To date many researchers have proposed several methods to
solve the problem of communication for deaf and dumb.
These methods are discussed below in this section:
As the name suggests ,Wireless Bell for Deaf and Dumb. But
how can a deaf or a dumb respond to bells. A Deaf cannot
hear to the bell and a dumb cannot respond to the calling bell.
Therefore, a simple low cost feasible system can be used to
help the deaf and the dumb to respond effectively to the
person knocking at the door. As the deaf cannot hear, a device
with vibrators is attached to their body in the form of some
wrist band or a watch. Similarly, for the dumb, a kind of bell
with speakers is developed which will playback some voice to
the person standing at the door as an acknowledgement
message. The entire system is wireless and can work
effectively within a radius of 50m i.e. the range of RF
Transmitter.
Traditionally, gesture recognition method was divided into
two categories namely vision based and sensor based method.
In vision based method, the computer camera is an input
device for varies gestures of hands and figures. In sensor
based systems, gloves are used which can achieve the accurate
positions of hand gesture. Lots of studies have been done on
sensor-based approaches like gloves, helmets etc. But wearing
it continuously is not possible. Therefore further work is
concentrated on image-based approaches.
The existing system suffers from lots of challenges in
providing two-way communication. Another problem is in the
accuracy of the existing systems. The existing system suffers
from accuracy problems due to occlusion of objects and due to
different textures of the hand, there is a deviation in the output
of the system.
III. HARDWARE REQUIREMENTS
A. Raspberry Pi:
Raspberry Pi is a credit-card sized computer that connects to a
computer monitor or TV and uses input devices like keyboard
and mouse. It is capable of performing various functionalities
such as surveillance system, military applications, surfing
internet, playing high definition videos, live games and to
make data bases The paper is implemented using a Raspberry
Pi 3 board and their specifications are as follows:
1) Memory:1 GB LPDDR-900 SD-RAM (i.e. 900MHz)
2) Processor: Raspberry Pi has a Broadcom BCM2837 system
on1chip module. It has quad core ARMv8 CPU. It can run at
1.2 GHz video core for multimedia GPU.
3) Power: The Pi requires a 5V power supply.
4) GPIO (General Purpose Input-Output): GPIO is a generic
pin on an integrated circuit which can be configured as input
and output pin. The Raspberry Pi board has 40 pins, 2.54mm
expansion header.
5) SD card: Raspberry Pi has no storage on board so an
external memory is required to store the OS.
6) Connections: 4 USB ports,10/100 Mbps Ethernet and
802.11n wireless LAN connections,3.5mm jack for audio
out,BCM43438 Wi-Fi, Bluetooth low energy(BLE).
Fig 1: Raspberry Pi Version 3
B. Logitech Camera
It is a plug and play setup which is easy to apply. You can
easily make video calls on major IMs. It has a 5MP camera
with high resolution. It has built in mikes with right sounds
gives you a clear conversation without any noise. XVGA
video recording system has a reach of about 1024x768
resolution. In this project, we are using a Logitech camera
which is capturing the image and gesture control function.
Fig 2: Logitech Camera
IV. PROJECT OVERVIEW
We aim for developing the prototype model for deaf and dumb
people by employing in a single compact device. The project
provides a unique solution for these people to manage their
sites by themselves. The project is catered with the source
code of Python. It is the easiest programming language to
interface with the Raspberry Pi. The project is run by the
source code of Python to assist dumb and deaf people in a
single device which is so compact and easy for them to
manage.
Fig 3: Text to Sign conversion output
The process started with the capturing the image and cropped
the useful portion. Convert the RGB image into gray scale
image for better functioning, Blur the cropped image through
Gaussian blur function and pass it to the threshold function to
get the highlighted part of the image. Find the contours and an
angle between two fingers. By using convex hull function we
can implement the finger point. Count the number of angles
which is less than 90 degree which gives the number of
defects.
Background Extraction is done through using histograms of
oriented gradient algorithm. The histogram of oriented
gradients (HOG) is a feature descriptor used in computer
vision and image processing for the purpose of object
detection.
Random Forests are an ensemble learning method for
classification, regression and other tasks, that operate by
constructing a multitude of decision trees at training time and
outputting the class that is the mode of classes or mean
prediction of the individual trees.

V. BACKGROUND EXTRACTION
Background Extraction is done through using histograms of
oriented gradient algorithm. The histogram of oriented
gradients (HOG) is a feature descriptor used in computer
vision and image processing for the purpose of object
detection. The technique counts occurrences of gradient
orientation in localized portions of an image. This method is
similar to that of edge orientation histograms, scale-invariant
feature transform descriptors, and shape contexts, but differs
in that it is computed on a dense grid of uniformly spaced cells
and uses overlapping local contrast normalization for Fig 5: Precision of the Random Forest Algorithm
improved accuracy. The essential thought behind the
histogram of oriented gradients descriptor is that local object The precision or accuracy is 0.77 i.e 77%. The benchmark
appearance and shape within an image can be described by the value is 70% but we have obtained 7% more accurate result.
distribution of intensity gradients or edge directions. The
image is divided into small connected regions called cells, and
for the pixels within each cell, a histogram of gradient VII. CONCLUSION
directions is compiled. We have designed the prototype model for deaf and dumb
people into a single compact device. The advantage of this
device is that it can be easily portable due to its less weight
and size. This paper fulfils the hand gesture recognition
process with some limitations as both the hands cannot be
used in this technique because the result is not efficient.

VIII. FUTURE SCOPE

The system can be further expanded for the alphabet, numbers
in gesture control. The input can be also taken in the form of
videos and the videos are divided into frames and the frames
can be converted into text. We can also add grammatically
structure for sign language. The system can be made handy by
incorporating it into a mobile phone.

Fig 4: Bounding Box
VI. RANDOM FOREST ALGORITHM
Random Forests are an ensemble learning method for
classification, regression and other tasks, that operate by
constructing a multitude of decision trees at training time and
outputting the class that is the mode of classes or mean
prediction of the individual trees.
The training algorithm for random forest applies the general
technique of bootstrap aggregating, or bagging to tree learners.
Given a training set X = x1, ..., xn with responses Y=y1,…,yn
bagging repeatedly (B times) selects a random sample with
replacements of the training sets and fits the trees to the
samples.
IX. REFERENCES
[1] Yiwen He, Jianyu Yang, Zhanpeng Shao, Youfu Li, ”Salient
feature point selection for real time RGB-D hand gesture
recognition”, IEE International Conference on real- time Computing
and Robotics, 2017
[2] Rania A. Elsayed, Mohammed S. Sayed, Mahmoud I. Abdalla,
“Hand gesture recognition based on dimensionality reduction of
histogram of oriented gradients”, Japan-Africa Conference on
Electronics, Communication and Computers, 2017
[3] Himadri Nath Saha, Sayan Tapadar, Shinjini Ray, Suhrid Krishna
Chatterjee, “A Machine Learning based approach for Hand Gesture
Recognition using distinctive feature extraction”, IEEE 8th Annual
Conference on Computing and Communication, 2018
 