Sector: Science and Technology Education

An augmented Human Computer Interaction, self learning and communication Tool for hearing impaired persons
Suman Deb Computer Science & Eng. Department, National Institute of Technology Agartala sumandebcs@gmail.com Abstract Hearing and speech are two basic senses through which human communicate to the world. Unfortunately some people are debarring of this basic sensory organ. In this paper we have tried to illustrate an inexpensive supplement to this hearing disability in the form of text to sign generation mechanism. This is an interactive software tool that can be used as Indian Sign Language (ISL) learning system, sign language dictionary and communication system through which hearing impaired persons can understand speech of other people. The same system can be interfaced with speech to text engine, where it can directly transfer any text received from a speech to text application and then to relevant gesture. This human computer interaction system takes any sentence in text (ASCII) form and uses two step sentence splitting mechanism which searches first words and their corresponding gestures in a predefined knowledgebase and if not found respective match then it splits the word into letters and the words is expressed by individual letter sign. These gestures of corresponding letter and sign are designed as per Indian Sign Language Dictionary[1]. Keywords: Text to gesture, hearing aid, Assistive technology, gesture dictionary, alternative communication. 1. Introduction: Disable but not Unable -people with disability may be different but they have the same need as everyone else. Given a helping hand, they can live independently and contribute to society [3]. This is the theme line which drove us to think of a system suitable for hearing impaired. During the development of this system we visited few institutions related to hearing disabilities. There we found schematic, animation or motion pictures are best suited for learning with disability. But there are almost no low cost computerized training system available suitable to Indian Sign Language Training. Studies have revealed not only for disabilities but for usual class room teaching also hypermedia can bring out the best. Hypermedia instruction is one way in which teachers can make their lessons more accessible for the hearing impaired students. When text is presented on a hypermedia content based, it allows the reader to get the information in several ways. In this paper the proposed text to sign or icon generation engine illustrates how technology within the classroom can benefit the hearing impaired child or enhance their self learning. 2. System Architecture Text to sign or icon generation engine(TSIGE) is a two step sentence splitter. Sentence can be entered through the interface by typing or text files can be browsed and feed to the system. Then the user passes the command to translate the given sentence or text file to sequence of corresponding gestures. This is a rule based system. After getting sentence it splits the whole sentence into words considering the

space and put them in a queue. For users ease of operation these words are shown in a visual list format. Then after a defined delay it starts picking up one word form the queue and searches the corresponding gesture image in the database. The images are in scanned JPEG format and indexed with the same name as words. If the match is found then in a gesture window the image is shown for user define period and if there is no match, the word is spited into letters following the same algorithm used for splitting sentence. And then letters are sequentially shown. Here a demon runs besides the main application which monitors whether showing a letter or a word phrase. Depending on usability it automatically
Start

synchronizes the delay between letters or words. The system is equipped with all amendable controls and any time user can customize the set of words with selection of subject domain. Searching can be done on separate domain or on whole knowledgebase. This separation of searching makes it reasonably faster. With every word an added parameter is used treated as use frequency counter. It is an integer type number and does relative counting of used words. In knowledgebase these words are kept in descending order of their used frequency. In case of same frequency the alphabetical sorting is applied.

Take one word/Letter from queue

Input any text from keyboard, text file or speech to text engine

Split the sentence in words

Put the words in a queue

Put the parsed letters in a queue

Parse the words in letter

Is a Word?

No

Yes

No

Search match for the word from queue to data base. Is there any corresponding match for the word?

Fig:2(a) Process flow of Text to gesture generation Yes Show the corresponding image

Yes End

Is the word queue empty?

No

3. Int terface of Software s S subsystem: 4. 4 Areas of applicatio f on:

Fig:3(a) Main in F nterface/start w window

This text to sign or icon g T generation engine(TSIG e GE) system can be use as both m ed indoor and o door app i out plication. Du uring class room teachi hearing impaired pe r ing eople can directly get t lessons t d the through spee to text ech and a then tex to gestur system. T same xt re The can c be treat as electr ted ronic diction nary with wildcard search facility where pe w y, eople can look for any particular w l y word and tog gether can see s the corr responding gesture. So learning o can c be more interesting and fast. A as out e And door applica d ation unit ca be used at rail way an station, airpo and daily life places major of s ort y s transaction. t 5. 5 Conclusio on: Presently ext to sign or icon g P generation engine(TSIG e GE) system works only on 2-D m images whic actually scanned im i ch mages and having a one to o h one mappi ing with correspondin words. T system does not c ng The consider wo or phrasa grouping. This is a c ord al one way co o ommunicatio This sy on. ystem can show text to gesture on Work pr s o nly. rogress is going on to do the reve g erse i.e. gene eration of speech or te form ges s ext sture of hum man, word grouping, ph g hrasal repres sentation an context nd based intelli b igent gestur generation During re n. the t developm ment of this system we took help from Speech and heari impaired Ashram f h ing d (Abhoynaga Agartala). Who helpe us with ( ar, ed the t portraya of present system of teaching, al t training and study mate t d erials. With practical h use u of the text to sign or icon g n generation engine(TSIG e GE)system among hearing impaired p i people we got very positive response bo from u r oth user and in nstructors. Work progr W ress is goin on to m ng make the system vers s satile and m more robust with 3-D modeling of gesture and motion gen m f d neration. References: R :
[1] [ Indian Sign Language Dicti ionary, Ramkris shna Mission Vidyalaya, Inte V ernational Hum man Resource D Development Center(IHRDC) for the disabled Coimbatore :-6 C d. 641070. [2] [ http://www.me edindia.net/news/v view_news_main

Fig: 3(b) Browse a text file/zoom/delay contro ol

Fig: 3(c) Word-gesture add/d delete/modify

.asp?t=gn&x=5 5576

[3] [ http://www.dis sability.org.sg/hea aringdisability /ind dex.asp [4] [ http://www.a aiishmysore.com m/