( ICRCICN

Real Time Human Pose Recognition Using Leap

Motion Sensor
Rajesh B. Mapari

Govind Kharat

Department of Electronics & Telecommunication Engg.

Anuradha Engineering College
Chikhli, India
rajesh_mapari2001@yahoo.com

Principal
Sharadchandra Pawar College of Engineering
Otur, India
gukharat@gmail.com
In first approach Sensor-based recognition systems
depend on instrumented gloves to acquire the
gestures information. In general, equipped sensors
measure information related to the shape, orientation,
movement, and location of the hand. Due to glove the
segmentation is easily achieved which is difficult
compared to bare hand segmentation in vision based
system. However the Glove based system is little bit
hectic as signer has to wear many sensors on wrist and
arms while performing sign.

Abstract Researchers have used one of the means

for acquiring hand Signs like Web Camera,
instrumented Glove, Depth Camera and Kinect sensor.
In this paper, an Indian Sign Language recognition
system is developed which recognizes ISL (with both
hands) using Leap Motion sensor. This sensor
overcomes the major issues in real time environment
like background, lightening condition, and occlusion.
The leap motion sensor captures the hand gesture and
gives finger position in 3D format (X, Y, Z axis values).
The positional information of five finger tips along with
center of palm for both the hand is used to recognize
sign posture based on Euclidean distance and Cosine
similarity. The system was tested for ISL (Indian Sign
Language alphabets) signs with 10 different signers.
The average recognition accuracy of ISL is 88.39 % for
Euclidean Distance method and 90.32 % for Cosine
similarity. While performing sign, the Leap Motion
Camera is kept about 10 degree inclined so that depth
information is properly extracted. Although Leap
Motion sensor tracks both the hand accurately it cant
track non manual signs which involve other body parts
and facial expressions.

In contrast, vision based system supports to both

manual and non manual signs. Here the segmentation is
color space based. So there are chances that skin color
of hand, cloth, face and surrounding environment may
be similar. So this creates difficulty in hand
segmentation. Another issue is occlusion handling
which may create when overlap with face or with
another hand.
Recently a new method which considers both the
local feature and global feature of gesture is introduced
using Kinect sensor [2]. But the problem with this
sensor is its not support minute details like shape of
hand.

Keywords American Sign Language (ASL), Indian

Sign Language (ISL), Chinese Sign Language (CSL).

I. INTRODUCTION
The number of the deaf-mute is more than 5% of
the population. Sign language is mainly employed by
deaf-mutes to communicate with each other. It is a
kind of language communicated through gestures and
visions. As sign languages are non-verbal languages,
information is conveyed visually, using a combination
of manual and non-manual means of expression.
Manual parameters are hand shape, hand posture,
hand location, and hand motion. The non manual
parameters include head and body posture, facial
expression, gaze and lip patterns. Non-manual
parameters are essential in sign language, since they
carry grammatical and prosodic information. Some
signs can be distinguished by manual parameters
alone, while others remain ambiguous unless
additional non-manual information, in particular facial
expression, is made available. According to author
sign language recognition system is defined in
following ways. According to the media of sign
language input, there are two kinds of sign language
recognition system: system based on instrumented
glove and system based on vision [1].

In our method, we have focused on local features

extraction to recognize static Signs of ISL as inputs to
the system. We have proposed a Euclidean distance [3]
and Cosine Similarity based method using five 3D key
points (five corresponding to five fingers of one hand).
The ultimate aim of our research is to enable
communication between speech impaired people and
common people who dont understand sign language.
This may work as translator [4] to convert sign language
into spoken words.
II.

RELATED WORK

Most of the research work in sign language

recognition system is concern to translation of sign
language to text or spoken word. Some systems are as
follows.
A. Instrumented glove Based system
Using instrumented glove, Gaolin Fang et al. [5]
presented a combination of HMMs [1] and SOFMs for
the
signerindependent
CSL
recognition.
In
SOFM/HMM architecture training and recognition
procedure was included. It is observed SOFM/HMM
signer independent sign recognition system was having
96.6% recognition rates in Reg. and 90.1% in Unreg.
Similarly Chunli Wang et al. [6] have recognized 2435

323

gestures in Chinese Sign Language using Parallel

HMM (PaHMMT). Two CyberGlove and a Pohelmus
3-D tracker with three receivers positioned on the wrist
of CyberGlove and the back are used as input device in
this system. 48dimensional vector obtained from
sensors used as feature which indicates hand shape,
position of hand and orientation of hand. The overall
recognition rate obtained is 84.06%. Jerome M. Allen et
al. [7] have designed a system to recognize 24 static
American Sign Language (ASL) finger spelling and
translate it to corresponding alphabet. A cyber glove
was used to collect the information and pattern
recognition technique was used with accuracy of 90%.

with 15 images of each. 10 images of each sign are

used for training the system and 5 images of each sign
are used for testing the system. Features extracted
from the training image set are used for training
the feed forward neural network. Average recognition
rate of 91.11%
C. Kinect sensor Based system
C. S. Weerasekera [4] et al.have proposed a robust
approach for recognition of bare-handed static sign
language. Local Binary Patterns (LBP) histogram
features based on color and depth information, and also
geometric features of the hand are used as features.
Linear binary Support Vector Machine (SVM)
classifiers are used for recognition. In the case of
multiple matches SVM is coupled with template
matching. An accurate hand segmentation scheme using
the Kinect depth sensor is also presented. The algorithm
is tested on two ASL fingerspelling datasets where
overall classification rate over 90% are observed. The
resulting sign language recognition system could be
employed in many practical scenarios and works in
complex environments in real-time. It is also shown to
be robust to changes in distance between the user and
camera and can handle possible variations in finger
spelling among different users. Lucas Rioux-Maldague
et al. [13] present a new feature extraction technique for
American Sign Language fingerspelling (alphabets
except J and Z) hand pose recognition using depth and
intensity images. Classification of
fingerspelling
carried using a Deep Belief Network. Results are
evaluated on a multi-user data set with two scenarios:
one with all known users and one with an unseen user.
Authors achieved 99 % recall and precision on the first,
and 77 % recall and 79 % precision on the second.

In similar type of approach, Tushar Chouhan et al.

[8] have achieved average accuracy of 96% over a data
set of 200 examples (20 examples per gesture) was
generated by taking the values for symbols 0-9
iteratively. Vasiliki E. Kosmidou et al. [9] proposed a
analysis of the surface electromyogram (sEMG) signal
for recognition of American Sign Language (ASL)
gestures. Sixteen features are extracted from the sEMG
signal acquired from the users forearm, and evaluated
by the Mahalanobis distance criterion. Discriminant
analysis is used to reduce the number of features for
classification of sign. The classification results reveal
that 97.7% of the inspected ASL gestures were
correctly recognized. Tan Tian Swee et al. [10] have
designed a system that can recognize 25 common
words signing in Bahasa Isyarat Malaysia (BIM) by
using the Hidden Markov Models (HMM) method. A
set of sensors consists of accelerometers and flexure
sensors have been setup to capture the movement or
gesture.
B. Vision based system

A.S.Elons et al. [14] have captured hands and

fingers movements in 3D digital format using Leap
motion. The sensor throws 3D digital information in
each frame of movement. These temporal and spatial
features are fed into a Multi-layer perceptron Neural
Network (MLP). The system was tested on 50 different
dynamic signs (distinguishable without non manual
features) and the recognition accuracy reached 88% for
two different persons.

Asha Thalange et al. [11] proposed a method to

detect static images of numbers 0-9 in American Sign
Language (ASL). This method is based on counting the
open fingers in the static images and extracting the
feature vector based on the successive distance between
the adjacent open fingers. Multilayered feed forward
back-propagation Neural is used for the classification.
This method is qualified to provide an average
recognition rate of 92 %. P.V.V.Kishore et al. [12]
proposed a 4 camera model for recognizing static
gestures of Indian sign language (26 alphabets and
numbers 1 to 10). Dataset consists of total 36 signs
from 4 cameras with 4 different orientations and two
different signers. Fourier descriptors [2] are used as
features to classify sign using artificial neural networks
with back-propagation training algorithm. Total number
of sign images is 144, out of which 144 are used for
training and remaining 144 sample sign images are
used for testing. The recognition rate stands at 95.1%
when tested with 288 samples.
Adithya v [3] proposed a method which uses digital
image processing techniques and artificial neural
network for recognizing different signs of ISL(Indian
sign language).The signs considered for recognition
include 26 letters of the English alphabet and the
numerals from 0-9. Database for training and testing
made in controlled environment in lab with black
background. After hand segmentation, image is in
binary format where distance transform is used as a
feature. The experiments are conducted for 36 signs

III.

SYSTEM OVERVIEW

The Leap Motion controller is a small USB

peripheral device which is designed to be placed on a
physical desktop, facing upward. Using two
monochromatic IR cameras and three infrared LEDs,
the device observes a roughly hemispherical area, to a
distance of about 1 meter [15]. Leap Motion sensor is a
small size sensor which is easy to use and of low cost
as shown in Fig 1.

Fig. 1. Leap Motion Controller

This sensor not only tracks the hand movements but

also it has the ability to distinguish the fingers' joints

324

and track their movements. This information is

supported
by
Leap
Motion
Vendors
"https://www.leapmotion.com" [14].
GUI is designed for the complete system as shown in
Fig. 2.
The working of system is as below:

Fig. 3. Gesture Assignment

Through the Button GESTURE VIEWER we can check

whether expected signs are stored or not as shown in Fig. 4.

Fig. 2. GUI for Sign Language Recognition

A. Profile Setup
Initially profile name is to be set like ASL using
button SAVE PROFILE AS, it stores the gesture
performed in ASL profile indicating sign belongs to
American Sign Language. We have to add gesture using
ADD GESTURE button as shown in Fig. 2. After that
the gesture assignment window is open as shown in
Fig. 3. First click on GRAB button. Simultaneously
user has to perform sign in front of Leap Motion. Leap
Motion has an option to visualize the hand motion as a
skeletal hand motion on screen. We have utilized this
option for getting the input coordinate values. The
program outputs 3D coordinates (x, y, z) value of in
each sign in the following format for both left and right
hand (Centre of the palm, Thumb, Index finger, Middle
finger, Ring finger, Pinky. The sign is captured and
stored by assigning gesture name and what it outs when
sign is recognized like speech.

Fig.4. Gesture Viewer

B. Recognition setup
Here first click on LOAD PROFILE button to
load expected profile like ASL, BSL or ISL. So that
sign performed in that language will be accepted and
recognized. To recognize sign click on GESTURE
RECOGNITION, now signer can perform any sign
which is stored in database (Profile).

325

(6)
However to make system more accurate some more
distance measures are added as follows.

(7)
(8)

(9)
when the signer gives perform the sign in front of
Leap Motion camera, the current hand coordinates are
used to calculate the d1,d2,d3,d4,d5,d6,d7,d8 as
mentioned above.

Fig.5. Gesture Recognition

When the signer performs sign, its coordinates as

mentioned earlier are calculated. Then using Euclidean
Distance, the distance measures are calculated
comparing all stored signs. According to the Euclidean
distance formula, the distance between two points in
the plane with coordinates (x, y) and (a, b) is given by

Different distance measures are calculated as

follow:
1) Euclidean Distance:
(10)
2) Cosine Similarity:
Given two vectors of attributes, A and B, the cosine

(1)
Whichever sign has small measure value is
output in the window as shown in Fig. 5 using text to
speech converter to get audio output.

IV. D

similarity,

d
magnitude as

ISTANCE MEASURES FOROREC

GNITION

(11)
Array1
=
(12)
ArrayCurrent
(13)

D1,D2,D3,D4,D5,D6,D7,D8]

[d1,d2,d3,d4,d5,d6,d7,d8]

(14)
For our setup we have kept the threshold as follows
For Euclidean Distance threshold is 1227. It means if
the result of distance measure is above 1227, the given
sign is considered as not recognized. The distance
measure is calculated for all store sign and the sign
which gives minimum value is considered as matched
sign.

Fig.6. Sample Hand Posture

As shown in Fig. 6 the 3D co-ordinates of sign is

stored in profile with the gesture name. Later on
distances of each finger tip with respect to center of
palm is calculated as follows.

For Cosine similarity threshold is 0.90. It means if

result for these distance measures are below threshold
value, the given sign is considered as not recognized.

(2)

The distance measure is calculated for all store sign and

the sign which gives maximum value is considered as
matched

(3)
(4)
(5)

326

sign. Fig. 7 shows postures

ndianof
sign
I Language.The ISL numbers postures are performed as presented

in [16].The recognition accuracy of the system is as

shown in table I and table II. However few signs
postures of numbers are not included in recognition
because of similarity in posture like sign posture Zero
and alphabet O, sign posture 1 and alphabet I ,
sign posture 2 and alphabet v and sign posture 7
and alphabet w .
TABLE I. CORRECT RATE OF SIGN POSTURE RECOGNITON
USING EUCLEDIAN DISTANCE

Fig.7. Postures in ISL

Correct rate of sign posture recognition
using Cosine Similarity

Correct rate of sign posture recognition

using Euclidean Distance
Recorded
Sign
Recorded
A
Sign

Total number Failure

Successfully
rate of
sign posture
recognition
of Correct
signs
number
of
recognized
Distance
presented using Euclidean
signs
signs
Total 10
number Failure
Successfully
0
10
of
signs number of
recognized
10
presented
signs 0
signs10

Correct
rate
Correct
100 %
rate

100 %

O
C

10
10

00

10
10

100
100 %
%

P
D

10
10

23

87

80
70 %
%

Q
E

10
10

30

7
10

70 %
100
%

R
F

10
10

50

5
10

50 %
100
%

S
G

10
10

5
10

5
10

50 %
100
%

T
H

10
10

07

10
3

100
%
30 %

U
I

10
10

00

10
10

100
100 %
%

V
J

10
10

00

10
10

100
100 %
%

W
K

10
10

05

10
5

100
%
50 %

X
L

10
10

00

10
10

100
100 %
%

Y
M

10
10

00

10
10

100
100 %
%

Z
N

10
10

60

4
10

40 %
100
%

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

Average correct Rate

Recorded
Sign

Total number
of
signs
presented

Failure
number of
signs

Successfully
recognized
signs

10

50 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

10

60 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

Average correct Rate

V. CONCLUSION
The Euclidean and Cosine based Indian Sign
Language Recognition system is presented in this paper.
The system is able to recognize all alphabets A to Z and
numbers 1 to 9 of

88.39 %

ISL. However number posture 0, 1, 2 and 7

are not included in recognition system due to
similarity with alphabet posture. The

327

Correct
rate

90.32 %

TABLE II. CORRECT RATE OF SIGN POSTURE RECOGNITON system has one constraint that Leap Motion Sensor has to be
USING COSINE SIMILARITY
Correct rate of sign posture recognition
using Cosine Similarity
kept little inclined while signs are performed and system

10

10

100 %

can recognize only static signs of ISL. The accuracy of

the system can be improved by using Artificial Neural
Network. However this type of system is very useful for
Hearing Impaired people of India as signer do not deed
to wear any special hardware device or do not need to
worry about environmental constraints.

10

80 %

REFERENCES

10

10

100 %

10

10

100 %

10

10

10

100 %

10

50 %

10

10

100 %

10

10

100 %

10

50 %

10

10

100 %

10

10

100 %

10

10

100 %

10

10

100 %

Recorded
Sign

Total number
of
signs
presented

Failure
number of
signs

Successfully
recognized
signs

Correct
rate

10

10

100 %

10

10

100 %

10

80 %

10

70 %

10

60 %

[1]

Wu jiangqin, Gao wen, Song yibo, Liu wei, Pang bo, a

simple sign language recognition system based on data
glove, Fourth International Conference on Signal
Processing Proceedings, Vol.2, pp. 1257 -1260 ,1998.

[2]

Chung-Lin Huung, Wen-Yi Huung, and Cheng-Chang

Lien, Sign Language Recognition using 3-D Hopfield
Neural Network, International Conference on Image
Processing, Vol. 2, pp. 611 - 614 ,1995.

[3]

Adithya V, Vinod P. R., Usha Gopalakrishnan ,"

Artificial Neural Network Based Method for Indian Sign
Language Recognition ",Proceedings of IEEE
Conference on Information and Communication
Technologies, pp. 1080-1085, 2013.

[4]

C. S. Weerasekera, M. H. Jaward, N. Kamrani, "Robust

ASL Fingerspelling Recognition Using Local Binary
Patterns And Geometric Features", International
Conference on Digital Image Computing: Techniques
and Applications (DICTA), pp.1-8,2013.

[5]

Gaolin Fang, Wen Gao, A SRN/HMM System for

Signer-independent
Continuous
Sign
Language
Recognition, Proceedings of the Fifth IEEE
International Conference on Automatic Face and Gesture
Recognition (FGR.02), pp. 312 317, 2002.

[6]

Chunli Wang, Xilin Chen, Wen Gao, Expanding

Training Set for Chinese Sign Language Recognition,
Proceedings of the 7th International Conference on
Automatic Face and Gesture Recognition (FGR06),
pp.323-328, 2006.

[7]

Jerome M. Allen, Pierre K. Asselin, Richard Foulds,

American Sign Language Finger Spelling Recognition
System in the Proceedings of the IEEE 29th Annual
Northeast Bioengineering Conference, pp.285286.2003.

[8]

Tushar Chouhan, Ankit Panse, Anvesh Kumar Voona and

S. M. Sameer, "Smart Glove With Gesture Recognition
Ability For The Hearing And Speech Impaired", Global
Humanitarian Technology Conference - South Asia
Satellite (GHTC-SAS), IEEE, pp.105-110,2014.

328

[9]

Vasiliki E. Kosmidou, Leontios J. Hadjileontiadis ,

Stavros M. Panas Evaluation of surface EMG features
for the recognition of American Sign Language gestures
Proceedings of the 28th IEEE EMBS Annual
International Conference New York City, USA, pp. ,
6197-6200, 2006.

[10] Tan Tian Swee, Sh-Hussain Salleh, A.K. Ariff, CheeMing Ting, Siew Kean Seng, and Leong Seng Huat,
Malay Sign Language Gesture Recognition System,
International Conference on Intelligent and Advanced
Systems, ICIAS 2007, pp. 982-985, 2007.
[11] Asha Thalange, Dr.Shantanu Dixit, "Effect of Thinning
Extent on ASL Number Recognition Using Open-finger
Distance Feature Measurement Technique", International
Conference on Signal Processing And Communication
Engineering Systems (SPACES), pp.39-43, 2015.
[12] P.V.V.Kishore, M.V.D.Prasad, Ch.Raghava Prasad,
R.Rahul,"4-Camera Model for Sign Language
Recognition Using Elliptical Fourier Descriptors and
ANN", International Conference on Signal Processing
And Communication Engineering Systems (SPACES),
pp.34-38, 2015.
[13] Lucas
Rioux-Maldague,
Philippe
Gigu`ere, "Sign
Language
Fingerspelling Classification from Depth and Color
Images using a Deep Belief Network", Canadian
Conference on Computer and Robot Vision, pp. 92-97,
2014.
[14] A.S.Elons, Menna Ahmed, Hwaidaa Shedid and
M.F.Tolba Arabic Sign Language Recognition Using
Leap Motion Sensor 9th International Conference on
Computer Engineering & Systems (ICCES), pp. 368673, 2014.
[15] Wikipedia. Leap Motion Wikipedia.org. [Online].
Available
https://en.wikipedia.org/wiki/Leap_Motion
[Last Modified: 26 July 2015, 07:16].
[16] Neha V. Tavari, Prof. A. V. Deorankar, Indian Sign
Language Recognition based on Histogram of Oriented
Gradient, International Journal of Computer Science
and Information Technologies, Vol. 5 (3) , pp. 36573660, 2014.

329