JOURNAL OF TELECOMMUNICATIONS, VOLUME 30, ISSUE 2, JUNE 2015

11

Development of a Sign language Tutoring

System for People with Hearing Disability
Bakpo, F.S and Ezugwu, O.A.
AbstractThis paper presents the development of a sign Language tutoring system using Markov Decision process model.
The work provides computer based support for the deaf and hard-to-hearing people and is valuable in places where the number
of qualified sign Language interpreters or technological assistance which enhanced accessibility is grossly inadequate or
absent. The Markov Decision Process (MDP) model is applied to process sign movement signals and to verify if a trainee has
closely replicated a human teachers sign movement. The system is designed using OOAD methodology and implemented
using visual C# programming Language. An empirical study was then conducted using questionnaire to 50 trainees and 4
human teachers on usability. The result shows that both human interpreters (teachers) and trainees (students) were satisfied
with the computer based approach than with traditional human based.
Index Terms Sign language, Markov decision mode, tutoring system, people with disability

1 INTRODUCTION
language learning dates back to 18 century. The
SIGN
first school for the deaf,the Institute Royal Des Soudsth

Muets (Royal Institute for the Deaf and Mute) located in

Paris, France tutored many pioneers of the modern day
sign languages. Prompted by the deaf daughter of a close
friend who wished to go to school, one of the old boys of
the French school, Thomas Gallaudet established the first
American school for the deaf in Hatford, Connecticut, in
1817. This gave rise to the first American Sign Language
(ASL) as it is referred to today. Other famous sign languages include the British sign language (BSL), the Swiss
German sign language and many others. Furthermore,
oral communication is a big issue for the most hard hearing, late defend and oral people around the world today.
This is because; it is a fundamental capability that pervades a persons life. Communication problem among the
deaf cannot be overemphasized; this informed the renewed interest to this subject. Although linguistic still
have much to learn about the world sign languages, it has
become clear that hundreds, if not thousands of sign languages exist around the world. Many people in deaf
communities around the world use sign languages as
their primary means of communication. This community
includes both deaf and hearing people who converse using sing languages. According to [1], sign language is a
communication method which adopts gestures that are
interpreted visually. Sign language can also be used as
alternative means of communication as described in [2].
The language is not restricted to deaf alone; although to
them, it serves as a primary or native language. Reference
[1] asserted that sign language creates strong sense of
social and cultural identity to the deaf. Due to the main

streaming policy of the United State Government, deaf

children no longer attend residential schools for the deaf.
Although some school of thought argued that this development isolated the deaf students from their fellow deaf
thereby depriving them of social and emotional comforts
that peer group can provide [2]. Others argued that the
advent of computer interaction has created more jobs for
private sign language tutors, software developers like
computer scientist and even given the deaf more opportunities to express themselves. According to [2], culturally deaf people no longer need to visit clubs in order to
meet and talk with each other. Instead, deaf people can
call each other using a special telephone device known as
teletypewriter (TTY) and many deaf people communicate
by computer as well as learning the sign language. In Nigeria, there is no much standard local sign language for
the deaf community. In [3] a method was proposed which
promoted the use of Nigerian signs to supplement the
existing sign language in classroom instructions for the
deaf and dumb. This paper describes the use of a Markov
Decision Process (MDP) to model and learn the students
knowledge of a sign language and choose the optimal
mode of instruction. The MDP framework described is
very flexible and contains a number of parameters available to modify the behaviour of the MDP to mimic various
pedagogical techniques. The paper defers in the implementation of a specific teaching method but focuses on
the fundamental framework that allows the use of an
MDP to implement an adaptive tutoring system. The rest
of the work is structured as follows in section 2 sign language is discussed. In section 3, we looked at the system
analysis and implementations of sign language tutor. In
section 4, we did performance analysis and in section 5,
we concluded the work.

F.S.Bakpo is a staff of Computer Science, University of Nigeria, Nsukka.

O.A.Ezugwu is a staff of Computer Science, University of Nigeria, Nsukka.

SIGN LANGUAGE

The keywords that form subject matter in this section are

12

sign and language. Sign refers to a gesture or movement

made with the hand or head to give information or command. According to [4] language is a system of sounds
and words used by humans to express their thoughts and
feeling. A language is used to convey meaning from one
party in communication say A to another party say B. A
sign language is one which, instead of acoustically conveying sound patterns as in spoken languages, conveys
meaning by simultaneously combining hand shapes, orientation and movement of the hands, arms or body, and
facial expression to fluidly express a speakers thoughts
[5]. There exist many sign language dialects across the
globe. Most predominant sign languages are the American Sign Language (ASL) and the British Sign Language
(BSL). American Sign Language is the Native or first language of deaf people in the continental United States (and
some part of Canada). The British Sign language is the
sign language used in the United Kingdom (UK), and is
the first or preferred language of deaf people in UK. It
makes use of space and involves movement of the hands
body, face and head. It was invented by Thomas Braidwood. The main different between sign languages and
spoken languages is that while spoken languages use
units of sounds to produce words, sign languages use
units of form. These units are composed of four basic
hand forms: hand shape, such as an open hand or closed
fist; hand location, such as the middle of the forehead or
in front of the chest; hand movement, such as upward or
downward; and hand orientation, such as the palm facing
up or out. In spoken languages units of sound combine to
make meaning whereas sign languages contain units of
forms that by themselves hold no meaning, but when
combined create a word. Spoken languages differ in the
way these units combine to make words, however. In
spoken languages, units of sound and meaning are combined sequentially. In sign language, units of form and
meaning are typically combined simultaneously. Figure 1
shows a typical communication scenario in Sign Language.

Fig. 1. People using sign language

Fig. 2. The American manual alphabet. Letters with asterisks are shown from the side (*L left or *R right).
ASL possesses a set of 26 signs known as the American
manual alphabet, which can be used to spell out words
from the English language. These signs make use of the
19 handshapes of ASL. For example, the signs for 'p' and
'k' use the same handshape but different orientations.

SIGN LANGUAGE TUTOR

Computer based teaching systems have a long history

dating back to the 1970s starting with the SCHOLAR [1]
program in 1970. However, it was not until the late
1980s and 1990s that widespread research and realistic
deployments were realized. Some well known intelligent
tutoring systems include the PUMP Algebra Tutor [9],
ANDES [10] physics tutor, and the computer literacy
software AutoTutor [4]. In this work, we shall consider
the design of a sign language tutor using the Markov Decision Process as a learning model.

3.1 Markov Decision Process (MDP)

A Markov Decision Process may be defined as a tuple (S,
A, Psa, !, R), where S is a set of possible states; A is the set
of possible actions; Psa is the state transition probabilities;
! is the discount factor and is between 0 and 1; R is the
reward function. On executing action a in state s the
probability of transitioning to state s is given by Psa(S)
and the reward is given by R(s). A policy is the expected sum of discounted rewards obtained by following
the policy from the current state forward. The primary
goal in MDPs is to find an optimal policy that maximizes
the value function in each state of the MDP. The value
function for a state s is defined in terms of the well known
Bellman equations:

V ( s) = R( s) +

s (s)

s s

2015 JOT
www.journaloftelecommunications.co.uk

( s )V ( s )

13

V * ( s) = V * ( s) = R( s) + max
aA

s (s)

( s )V ( s )

s s

There are a number of methods used to solve this nonlinear equation including dynamic programming, policy,
iteration and value iteration. This paper will utilize the
commonly used value iteration method as follows:
Given a MDP (S, A, Psa, !, R), the value iteration algorithm is used to find the optimal value function and is as
follows:
1.
For each state s, initialize V(s) to some starting
value
2.
Repeat until convergence
{
For each state s, update

V ( s) = R( s) + max Ps ( s ) ( s )V ( s )
aA

s s

}
The updates to the value function for each state can be
done all at once in a synchronous update or each time the
value function is calculated for a given state in an asynchronous manner. Either method will cause the value
functions to converge to the optimal value.

reached after answering a question correctly and the IC

state represents answering a question incorrectly. The set
of all states for a sign MDP is given by S: (L1, C1, IC1, ,
Ln, Cn, ICn) for n supporting concepts. The action set for
the MDP utilized in this paper contains two actions for
each sign movement: Teaching (T) and Questioning (Q).
Thus for a given sign, the set of actions is given by A:{T1,
Q1, Tn, Qn }for n supporting concepts.

SYSTEM DESIGN AND IMPLEMENTATION

4.1 System Design

The design of the system is presented using the use
case diagram. The Use Case Diagram which is an integral
aspect of this design method describes the functionality
provided by a system in terms of actors and their actions;
their goals represented use cases, and any dependencies
among those cases. It describes the relationships between
actors (the program and the user in this case). We have
carefully provided a video coverage of each of the sign
alphabets shown in Figure 2 and a trainee or student is
allowed to closely imitate or replicate the sign movement
shown in the video. The MDP is applied to validate each
students attempt at practicing the sign alphabet and to
obtain the optimal value.

3.2 knowledge models

The knowledge model is based on two fundamental assumptions. The first assumption is that one way to show
mastery of a subject is the ability to answer random questions from the given subject area. The second assumption
is that a statement in a language is normally broken down
into a set of supporting words. Figure 3 shows an example of a simple knowledge model of hand movement.

Left_Up

Hand_Movement

Left_Down

Right_Up

Right_Down

Fig. 3. Example of Tutoring Knowledge Model of Hand
movement

3.3 Sign Language Tutoring Using Markov

Decision Process
The sign language tutoring using MDP is designed to
teach a student the sign language vocabularies consisting
of multiple hand movements. The sign language tutor
enables the trainee to watch and learn new signs, practice
and validate their performance. Through reinforcement
learning it derives a student knowledge model for each of
the hand movement and uses that model to determine
what hand movement to spend time on and whether to
teach or quiz the student. The tutoring MDP contains a
tuple of states (L, C, IC) for each sign. The state labeled L
is the learning state for a given sign. The C state is

Fig. 4. Use case diagram

Learn alphabet: here a user is given the opportunity to learn the basic symbols that make up
the standard ASL alphabet.
Video tutorial: Here the user is granted access to
multiple video tutorials of words and how they
are used to form phrases and clauses.
Test yourself: Owning to the fact that humans
will at some point doubt whether they know or
not. This portion allows the user to test his or her
self. The program will in responds grade the user

14

in such a test.

4.2 System Implementation

System implementation involves converting the designs developed in the analysis stage into a particular
programming language. Basically the choice of programming language is based on the application software
features and the extent of our knowledge on the chosen
language. We implemented prototype using Visual Basic
10.0 Programming Language and Microsoft Access Relational Database (MS Access 2007). These tools were chosen for the following reasons: (i) user friendly forms for
interacting with the user, (ii) Easy to access since almost
every PC has Microsoft Office installed and having MS
Access as an Integral Member (iii) It is an Object Oriented
Programming Language hence inherits all the merits of
any OOP language (iv) Runs comfortably in Window Operating System which is a widely used operating system
in the target market.

TABLE 2
COMPARISON OF HUMAN TUTOR WITH AUTOMATED SIGN
LANGUAGE TUTOR
Criteria for
Comparison

Automated Sign
Language Tutor

Human
Tutor

Clarity of understanding

76

75

Distraction

90

87

Easy to use

82

80

Self explanatory

90

85

Self study

95

78

PERFORMANCE ANALYSIS

This package was used to administer tutorial to a class of

50 students and conducted in two phases. The class was
further subdivided into two groups. In the first phase,
the first group was tutored by the automated software
while the second group was tutored by a human tutor or
interpreter. In the second phase, the tutoring system was
swapped in turn. At the end of the process, the students
were asked to fill a questionnaire. The questionnaire consists of definite criteria such as clarity of understanding,
distraction, easy to use, self study and self explanatory.
At last, the questionnaires were collected and the score
obtained are as shown in Table 1a and 1b, respectively.
Consequently, the summary of the experiment is as
shown in Figure 5a and Figure 5b, below.

Fig. 5a. Experimental Results obtained from comparing automated

sign language tutor with human tutor

TABLE 1
COMPARISON OF HUMAN TUTOR WITH AUTOMATED SIGN
LANGUAGE TUTOR
Criteria for
Comparison

Automated Sign
Lang. Tutor

Human
Tutor

Clarity of understanding

80

75

Distraction

90

89

Easy to use

85

75

Self explanatory

90

85

Self study

95

90

Fig. 5b. Experimental Results obtained from comparing automated

sign language tutor with human tutor

The results from figures 5a and 5b showed that in all the

tested criteria which were clarity of understanding, distraction, ease of use, self-explanatory and self study the
automated sign language tutor out-performed the human
tutor. Thus we found that a lot of people prefer the automated software as it encourages self study, promotes repetition, and void of distraction.

15

CONCLUSION

In a nutshell, this paper is a report of an ongoing project work currently carried out in the Department of
Computer Science in University of Nigeria Nsukka which
aims at developing an automated sign language tutor for
People with special needs. Having identified most of the
problems associated with various human sign language
tutors, we have been able to design a computer based
approach, which will play a key role in improving sign
language learning. Existing human sign language interpreters are not productive enough since even the absent
or death of human sign tutor may have devastating effects. We further tested the performance with some interesting users. Quantitative results show that the automated sign language tutor is able to teach sign language.

REFERENCES
[1]
[2]
[3]
[4].
[5]
[6]
[7]
[8]
[9]

S. Wilcox, Sign Language. Microsoft Encarta 2009 [DVD]. Redmond,

WA: Microsoft Corporation, 2008.
C.A. Padden, Deaf Culture. Microsoft Encarta 2009 [DVD].
Redmond, WA: Microsoft Corporation, 2008.
P.A. Ajavoni, Incorporation of Nigerian Sign in Deaf Education in Nigeria.
Oxford 2003.
A.S. Hornby, Oxford Advanced Learners Dictionary of Current English.
Oxford University Press 1998.
Sign language, downloaded from htpp://en.m.wikipedia.org/wiki/. Edited last
2008.
Erden, Z., Erkmen, A. M. and Erden, A (1998), A Petri Net-Based Design
Network with Applications to Mechatronic Systems SDPS Transactions,
Journal of Integrated Design and Process Science, Vol. 2, No 3, pp. 32 -48.
Kelly, M. Stack, Tiers and Syllable Structure in American Sign Language:
Evidence from Phonotactics. Curled from
http://roseavenue.net/kelly/HummetFiles/thesis.htm,1988.
Corbett, A. T, Anderson, J. R.and Koedinger, K. R. Intelligent Tutoring
systems. Handbook of Human-Computer Interaction, Second, Completely
revised Edition, Chapter 37, 1997.
Vanlehn, C., Lynch, Schulze, K. The Andes Physics Tutoring system: Les
sons Learned. International Journal of Artificial Intelligence in Education,
15(3), 2005.

Prof. Bakpo Francis S. received his M Sc degree in Computer Science and Engineering from Kazakh National Technical University,
Almaty (formerly, USSR) in 1994 and PhD degree in Computer engineering in 2008 from Enugu State University of Science and Technology (ESUT), Agbani. He joined the Department of Computer Science at University of Nigeria Nsukka as lecturer in June 1996. His
current research interest includes Computer architectures, computer
communications networking, artificial neural network applications,
intelligent software agents and Petri net theory. He is dully registered
professionally as computer scientist and has also published a number of excellent journal papers, books and conference proceeding
papers in his field.