ISSN: 2349-2783
Normalizing the Hindi Text: A Survey
Lovely Sharma
Department of Computer Science,
Doaba College
Abstract
All areas of language and speech
technology, directly or indirectly, require
handling of real (unrestricted) text. For
example, Text-to-Speech systems directly
need to work on real text, whereas
Automatic Speech Recognition systems
depend on language models that are
trained on text. This survey paper describes
about how to normalize Hindi Text using
standard rules. We studied the different
papers that used for different languages like
Bangla, Python, French etc. and even Hindi
also. Different papers had used different
types of algorithms for the different fields
like the NLP and DIP etc. Based on the
related work we will develop an algorithm
using Hindi Standard Rules for standardizing
the spelling variants in Hindi Text.
1. INTRODUCTION
Hindi Text Normalization (HTN) works on spelling
standardization issues, thereby resulting in multiple
spelling variants for the same word. The major
reasons for this phenomenon can be attributed to
the phonetic nature of Indian languages and multiple
dialects, transliteration of proper names, words
borrowed from foreign languages, and the phonetic
variety in Indian language alphabet. The variety in
the alphabet, different dialects and influence of
foreign languages has resulted in spelling variations
of the same word. Such variations sometimes can be
International Journal of Innovation and Research in Computer Science
Abhishek Goel
Department of Computer Science,
B.D. Arya Girls College
treated as errors in writing. While some are very
widely used to be called as errors. We consider all
types of spelling variations of a word in the
language.
We will develop a solution to come up with a set of
rules specific to language which can handle such
variations, which could result in more precise
performance.
Therefore apart from the Indian language words we
should also be able to handle proper names and
English words transliterated in Indian languages
since they form substantial percentage of words.
For example we found widely used spelling
variations for the Hindi word `angrezi' as shown
below.
In real text, many non-standard representations of
words appear, for e.g., numbers (year, time, ordinal,
cardinal, floating point), abbreviations, acronyms,
currency, dates, URLs. All these non-standard
representations must typically be normalized, or in
other words converted to standard words, which
would then be processed in various applications.
Some examples of the problems faced by natural
language understanding systems:
37

 Text Segmentation: Some written languages like
Chinese, Japanese and Thai do not have single word
boundaries either, so any significant text parsing
usually requires the identification of word
boundaries, which is often a non-trivial task.
 Word Sense Disambiguation: Many words have
more than one meaning; we have to select the
meaning which makes the most sense in context.
 Hindi Text Normalization: Many Words in Hindi
language written in different ways. We work on the
Normalization of those Hindi words.
 Syntactic Ambiguity: The grammar for natural
languages is ambiguous, i.e. there are often multiple
possible parse trees for a given sentence. Choosing
the most appropriate one usually requires semantic
and contextual information. Specific problem
components of syntactic ambiguity include sentence
boundary disambiguation.
1.1 Main cause behind Hindi Text Normalization:-
The issue why Normalization is required on Hindi
Text is due to Devanāgarī (दे वनागरी) Script.
Devanāgarī (दे वनागरी) is an abugida script used to
write several Indo-Aryan languages, including
Sanskrit, Hindi, Gujarati, Marathi, Sindhi, Bihari, Bhili,
Marwari, Konkani, Bhojpuri, Pahari (Garhwali and
Kumaoni), Santhali, Nepali, Newari, Tharu and
sometimes Kashmiri and Romani. The Devanāgarī
Italic text writing system can be called an abugida, as
each consonant has an inherent vowel (a), which can
be changed with the different vowel signs.
Devanāgarī is written from left to right. A top line
linking characters is thought to represent the line of
the page with characters historically being written
under the line. In Sanskrit, words were originally
written together without Media: spaces, so that the
top line was unbroken, although there were some
exceptions to this rule. The break of the top line
primarily marks breath groups. In modern languages,
word breaks are used. When reading Sanskrit
written in Devanāgarī, the pronunciation is
completely unambiguous. Similarly, any word in
Sanskrit is considered to be written only in one
manner (discounting modern typesetting variations
in depicting conjunct forms). However, for modern
languages, certain conventions have been made (e.g.
International Journal of Innovation and Research in Computer Science
ISSN: 2349-2783
truncating the vowel form of the last consonant
while speaking, even as it continues to be written in
full form). There are also some modern conventions
for writing English words in Devanāgarī.
2. NEED OF TEXT
NORMALIZATION
2.1 Normalization of Non-standard Words Real text
contains a variety of "non-standard" token types,
such as digit sequences; words, acronyms and letter
sequences in all capitals; mixed case words (WinNT,
SunOS); abbreviations; Roman numerals; URL's and
e-mail addresses. Many of these kinds of elements
are pronounced according to principles that are
quite different from the pronunciation of ordinary
words. Furthermore, many items have more than
one plausible pronunciation, and the correct one
must be disambiguated from context: IV could be
"four", "fourth", "the fourth", or "I.V. Normalizing or
rewriting such text using ordinary words is an
important issue for several applications more
sophisticated text normalization will be an important
tool for utilizing the vast amounts of on-line text
resources. Normalized text is likely to be of specific
benefit in information extraction applications.
2.2 Text normalization challenges
The process of normalizing text is rarely
straightforward. Texts are full of heteronyms,
numbers, and abbreviations that all require
expansion into a phonetic representation. There are
many spellings in English and many other languages
which are pronounced differently based on context.
For example, “IV could be "four", "fourth", "the
fourth", or "I.V. ".
Most text-to-speech (TTS) systems do not generate
semantic representations of their input texts, as
processes for doing so are not reliable, well
38
 ISSN: 2349-2783

understood, or computationally effective. As a

result, various heuristic techniques are used to guess
the proper way to disambiguate homographs, like
examining neighboring words and using statistics
about frequency of occurrence. 3. LITERATURE SURVEY

Similarly, abbreviations can be ambiguous. For

example, the abbreviation "in" for "inches" must be
differentiated from the word "in", and the address
"12 St John St." uses the same abbreviation for both
"Saint" and "Street". TTS systems with intelligent
front ends can make educated guesses about
ambiguous abbreviations, while others provide the
same result in all cases, resulting in nonsensical (and
sometimes comical) outputs.
2.3 Application of Text Normalization
All areas of language and speech technology, directly
or indirectly, require handling of real (unrestricted)
text [1]. For example, Text-to-Speech (TTS) systems
directly need to work on real text, whereas
Automatic Speech Recognition (ASR) systems
depend on language models that are trained on text.
In real text, many non-standard representations of
words appear, for e.g., numbers (year, time, ordinal,
cardinal, floating point), abbreviations, acronyms,
currency, dates, URLs. All these non-standard
representations must typically be normalized, or in
other words converted to standard words, which
would then be processed in various applications.

Text Normalization is one of the most important 3.1 Related Work

tasks in text processing and text to speech
conversion. There are two major components in a
text to speech system: (1) text normalization (2)
Firoj Alam, S. M. Murtoza Habib, Mumit Khan
speech generation. Before discussing the issues
(2009) describes about a process of text
related to text processing, let us briefly discuss the
normalization system of Bangla language by
nature of the Indian languages scripts for which the
identifying the semiotic classes from Bangla text
synthesis systems are built. The basic units of the
corpus. After identifying the semiotic classes a set of
writing system in Indian languages are Aksharas,
rules were written for tokenization and
which are orthographic representations of speech
verbalization. This study is important for Text-To-
sounds. An Akshara in Indian language scripts is a
Speech (TTS) system and as well as in language
syllable and can be typically of the following form: V,
model for speech recognition. It must first be pre-
CV, CCV and CCCV where C is a consonant and V is a
processed to remove the ambiguities and convert
vowel. All Indian language scripts have a common
some Non Standard Words (NSWs) into their
phonetic base, and a universal phone set consisting
standard word pronunciations. This study develops a
of about 35 consonants and about 18 vowels. The
method to normalize Bangla text using a rule based
pronunciation of these scripts is almost
system rather than using a decision tree and a
straightforward. There is more or less one to one
decision list to ambiguous tokens. Many little works
correspondence between what is written and what
has been done on under resourced languages such
is spoken. However, in languages such as Hindi and
as Bangla. The basic similarity among the work done
Bengali the inherent vowel (short /a/) associated
is that each involves repeated sequences of
with a consonant is not pronounced depending on
tokenization, token classification, token sense
the context. It is referred to as inherent vowel
disambiguation and standard word generation to get
suppression or schwa deletion.

International Journal of Innovation and Research in Computer Science 39


the normalized text. According to semiotic classes a
lexical analyser was designed to tokenize each NSW
by regular expression using tool JFlex. We assigned a
tag for each token according to semiotic classes. The
outputs of the tokenization are then used in the next
step i.e. token expander. According to the assigned
tag token verbalization and disambiguation was
performed by the token expander. In Semiotic class
identification, we identified a set of semiotic classes
which belongs to the Bangla language. To do this, we
selected a news corpus, forum and blog, then we
proceeded in two steps to identify the semiotic
classes: (i) Python [4] script was used to identify the
semiotic class from news corpus and we manually
checked it in the forum and blog (ii) we defined a set
of rules according to context of homographs or
ambiguous tokens. The result is a set of semiotic
classes in Bangla text as shown in table1.
Figure 3.1 Possible token type in Bangla Text
We defined each semiotic class to a specific tag and
assigned this tag to each class of token. The
tokenization undergoes three levels such as: i.
Tokenizer ii. Splitter and iii. Classifier. Like English
and other South Asian scripts Bangla also uses
whitespace to tokenize a string of characters into a
separate token. The output of this work is the list of
words in a normalized form. The performance of this
rule based system is 90% for ambiguous tokens such
as float, time and currency. The accuracy of these
three types of token is: floating point 100%, currency
100% and time 62% [5].
International Journal of Innovation and Research in Computer Science
ISSN: 2349-2783
Yuxiang Jia, Dezhi Huang, Wu Liu, Yuan Dong,
Shiwen Yu, Halia Wang (2008) develops taxonomy
of NSWs on the basis of a large scale Chinese corpus,
and proposes a two-stage NSWs disambiguation
strategy, Finite State Automata (FSA) for initial
classification and Maximum Entropy (ME) classifiers
for subclass disambiguation. Typical methods for
text normalization are based on handcrafted rules.
But such hand-crafted rules are difficult to write,
maintain and adapt to new domains. On the other
hand, in view of homograph disambiguation, many
machine learning methods are employed and have
shown their advantages. Decision tree and decision
list are used in English and Hindi text normalization
[1]. The text normalization approach proposed in
this paper does not need word segmentation
process. Finite state automata detect NSWs from the
real text and make an initial classification and then
maximum entropy classifiers are used for further
classification. The process flow is outlined in Fig.1.
Figure 3.2 Flow chart of Text Normalization
Finite State Automata (FSA) are designed to detect
NSWs and give an initial classification based on
NSWs formats. Next is the subclass disambiguation
process which is needed to determine the true
pronunciations in certain contexts with the help of
Maximum Entropy classifier. Last module is the
Standard Word Generation of text normalization. It
is a generation step while former steps are analysis
40

steps. The input of this module is NSW itself and its
class tag. The output is its corresponding Chinese
words. The conversion is a one-one correspondence
and finite state transducers are applicable here. This
paper makes an extensive investigation of Chinese
text normalization. NSWs taxonomy is developed
based on a large scale corpus. After a systematic
analysis of the taxonomy, a two stage NSWs
classification strategy is proposed, finite state
automata for initial classification and maximum
entropy classifiers for further classification.
Experiment results show that this approach achieves
a good performance and generalizes well to new
domains. In addition, this approach is character-
based, no need of word segmentation preprocess.
However, some error occurs in experiment such as
number sequence error [7].
Conghui Zhu, Jie Tang, Hang Li, Hwee Tou Ng, Tie-
Jun Zhao (2007) describes about the issue of text
normalization, an important yet often overlooked
problem in natural language processing. By text
normalization, we mean converting ‘informally
inputted’ text into the canonical form, by eliminating
‘noises’ in the text and detecting paragraph and
sentence boundaries in the text. Previously, text
normalization issues were often undertaken in an
ad-hoc fashion or studied separately. This paper first
gives a formalization of the entire problem. It then
proposes a unified tagging approach to perform the
task using Conditional Random Fields (CRF). The
paper shows that with the introduction of a small set
of tags, most of the text normalization tasks can be
performed within the approach. The accuracy of the
proposed method is high, because the subtasks of
normalization are interdependent and should be
performed together [9].
Gralioski Filip, Jassem Krzysztof, Wagner Agnieszka,
Wypych Mikołaj (2006) describes some problems of
text normalization for an inflected language like
Polish. It is shown that text normalization, being
usually one of the first steps of text-to-speech
synthesis, is a complex process, referring to several
levels of analysis: morphological, syntactical and
semantic. It is claimed here that text normalization
International Journal of Innovation and Research in Computer Science
ISSN: 2349-2783
may be treated in a similar way to Machine
Translation: The tools and algorithms developed for
Machine Translation may be used for text
normalization, with the “spoken language” being
treated as the target language [10].
K.Panchapagesan, Partha Pratim Talukdar,
N.sridhar Krishna, Kalika Bali, A.G. Ramakrishnan
(2004) proposed a novel approach to text
normalization, where in tokenization and initial
token classification are combined into one stage,
followed by a second level of token sense
disambiguation [2]. The architecture of the proposed
approach is shown in Figure 1. Tokenization and
Initial Token Classification are performed using a
lexical analyzer that is derived from various token
definitions in the form of regular expressions. For
the second level of token sense disambiguation,
traditionally, text normalization is viewed as an
engineering issue and is conducted in a more or less
ad-hoc manner.
Figure 3.3 Stages involved in Text Normalization.
In this approach to text Normalization, tokenization
and initial token classification / identification are
achieved in a single step. We have used Flex, an
automatic generator tool for high-performance
scanners; Flex takes a set of regular expressions as
input and generates a scanner as output that will
scan an input stream for the tokens represented by
the regular expression. A scanner works as a lexical
analyzer, recognizes lexical patterns in the input
text, and thereby groups input characters into
tokens. Tokens are specified using patterns (regular
expressions). In token sense disambiguation, once
the tokens are extracted from the input text, the
41

category of each token need to be identified. This is
accomplished by the lexical analyser itself when
there is no ambiguity arising from the format(s) of
the token. In case of ambiguity, the token with
possible token types is output to facilitate further
disambiguation. Identification of token category
involves high degree of ambiguity. For example,
‘1960’ could be of the type ‘Year’, or of the type
‘Cardinal Number’, and ‘1.25’ could be of the type
‘Float’, or of the type ‘Time’. Disambiguation is
generally handled by hand-crafted context-
dependent rules. We have used decision tree based
data-driven techniques to address the same. When a
token is input to the tree for disambiguation, a
decision is made by traversing the tree starting from
the root node, taking various paths satisfying the
conditions at intermediate nodes, till the leaf node is
reached. Decision lists are a special class of decision
trees; they can use for representing a wide range of
classifiers. A decision list can be viewed as hierarchy
of rules, when a classification is needed; the rule in
the hierarchy is addressed. If that rules fails to
classify as well, the third rule is addressed, and so
on. They are basically if-then else statements [3]. For
example:
if condition1 (x) is true then output = output1 (x)
else if condition2 (x) is true then output = output2
(x)
. . .
. . . closely related to the speech recognition accuracies
. . . normalizations of up to 185M words of newspaper
Else output = default_output (x)
Steve Atwell, Hahn Koo, Liam Moran, and Tae-Jin
Yoon (2004) describes the implementation of a
system in the programming language Python that
normalizes texts into a form that resembles how a
human might read it out loud. Classified ads are the
target domain of the normalization system. For a
given input text, a sequence of letters can either be
treated as a single token or a group of subtokens
that can be split further. For example, character
International Journal of Innovation and Research in Computer Science
ISSN: 2349-2783
sequences are usually divided into subtokens using
the presence of commas, hyphens, or slashes within
them. The algorithm in the Perl splitter is that the
input texts are first bracketed into groups, and then
the texts are split based on split points, while
keeping the groups intact. Next is the classifier which
operates in two stages. The first stage handles all
tags except for ASWD, EXPN, and LSEQ, which were
combined into a single ALPHA tag. If a token is
tagged as ALPHA, the alphabetic classifier classifies
the token as one of ASWD, EXPN, or LSEQ. The
classifier generates features for a token using the
two preceding tokens, the token itself, and the two
following tokens. The general classifier has 128
features. Other features include the length of the
token and context tokens, whether the target and
context tokens have proper name capitalization, the
number of splits the splitter would make on the
target token, and some very basic context
disambiguation scoring. The result is not satisfactory
due to the performance of the classifier, since the
low recall rate evidently lowers the accuracy rate [4].
Gilles Adda, Martine Adda-Decker, Jean-Luc
Gauvain, Lori Lamel (1997) describes a quantitative
investigation into the impact of text normalization
on lexica and language models for speech
recognition in French. The text normalization
process defines what is considered to be a word by
the recognition system. Depending on this definition
we can measure different lexical coverage’s and
language model perplexities, both of which are
obtained on read newspaper texts. Different text
texts are presented along with corresponding lexical
coverage and perplexity measures. Some
normalizations were found to be necessary to
achieve good lexical coverage, while others were
more or less equivalent in this regard. The choice of
normalization to create language models for use in
the recognition experiments with read newspaper
texts was based on these findings. Best system
configuration obtained an 11.2% word error rate in
the AUPELF ‘French-speaking’ speech recognizer
42
 ISSN: 2349-2783

evaluation test held in February 1997. We use two
large French dictionaries: BDLEX and DELAF [6].
4. RULES USED FOR
NORMALIZING TEXT
Unicode contains 10 consonant characters with
nukta (a dot under consonant) and one nukta
character itself. We delete all occurrences of nukta
character and replace all consonants with nuktas
with their corresponding consonant character. This
would equate words like the ones shown below.

Indian language words face spelling standardization

issues, thereby resulting in multiple spelling variants
for the same word. The major reasons for this
phenomenon can be attributed to the phonetic
nature of Indian languages and multiple dialects,
transliteration of proper names, words borrowed
from foreign languages, and the phonetic variety in
Indian language alphabet. Given such variations in
Halanth deletion:-
spelling it becomes difficult for web Information
Retrieval applications built for Indian languages.
India is a multi-language, multi-script country with
22 official languages and 11 written script forms. Hindi and many other Indian languages face the
About a billion people in India use these languages problems of 'schwa' (the default vowel 'a' that
as their first language. Some of the rules used are: occurs with every consonant) deletion. Lots of
spelling variations occur due to 'schwa' deletion. In
order to normalize such words we delete all the
halanth characters in the given word before making
Mapping chandrabindu to bindu:- a string match. This operation would normalize
words as shown in the example below.

Often people tend to use chandrabindu (a half-

moon with a dot) and bindu (a dot on top of
alphabet) interchangeably. Lots of confusion exists in
common language usage on which to use when. In
order to equate all such words we convert all
occurrences of chandrabindu to bindu, which would
equate all the words shown below.
Mapping character ‘ न ् ’ to Bindu:-

In Hindi language this problem occurs because

various people use ‘Bindu’ in place of “न ् “. In our
application, we convert ‘न ् ‘to Bindu (ं). Using this
Nukta deletion:-
feature we can normalize words as shown in this
example:

International Journal of Innovation and Research in Computer Science 43


So, these are some of the rules that we will used for
standardizing the spelling variants in Hindi Text.
5. CONCLUSION AND FUTURE
WORK
We have studied papers on various languages such as
Bangla, Python, etc. and come to conclusion that
normalization of Hindi Text is an important part of
Natural Language Processing (NLP). Hindi Rules
should be used for standardizing the spelling variants
in Hindi Text. For Future Work, we will develop
more rules and Secondly, Development of dictionary
automatically from Hindi corpus for such words and
depending upon the frequency of words in text,
choosing one among them as standard. Using above
rules, standardizing the given Hindi Text
automatically.
Implementation will be done in Microsoft .Net as
Front-End and MS-Access as Back-End.
REFERENCES
[1]. Richard Sproat, Alan Black, Stanley Chen, Shankar
Kumar, Mari Ostendorfand Christopher Richards, “Normalization
of Non-Standard Words”, Computer Speech and
Language, 15(3):287.333, 2001.
[2]. Manish Sinha, Mahesh Kumar Reddy .R, Pushpak
Bhattacharyya, Prabhakar Pandey, Laxmi Kashyap, “Hindi Word
Sense Disambiguation”, International Symposium on Machine
Translation, Natural Language Processing and Translation Support
Systems, Delhi, India, November, 2004.
[3]. K.Panchapagesan, Partha Pratim Talukdar, N.sridhar
Krishna, Kalika Bali, A.G. Ramakrishnan, “Hindi Text
International Journal of Innovation and Research in Computer Science
ISSN: 2349-2783
Normalization” Fifth International Conference on Knowledge Base
Computer Systems (KBCS), Hyderabad, India, 19-22 December
2004.
[4]. Steve Atwell, Hahn Koo, Liam Moran, and Tae-Jin Yoon,
“Text Normalization in Python”
www.linguistics.uiuc.edu/grads/moran/papers/TextNorm.pdf.
[5]. Firoj Alam, S. M. Murtoza Habib, Mumit Khan, “Text
Normalization system for Bangla” Paper accepted in Conference
on Language and Technology 2009 (CLTO9), 22-24 January 2009.
[6]. Gilles Adda, Martine Adda-Decker, Jean-Luc Gauvain,
Lori Lamel, “Text Normalization and Speech Recognition in
French” Proc. ESCA Eurospeech'97,
ftp://tlp.limsi.fr/public/euro97bref.ps.Z
[7]. Yuxiang Jia, Dezhi Huang, Wu Liu, Yuan Dong, Shiwen
Yu, Halia Wang, “Text Normalization in Mandarin Text-To-Speech
System” Acoustics, Speech and Signal Processing, 2008. ICASSP
2008. IEEE International Conference on Volume, Issue, March 31
2008-April 4 2008 Page(s):4693 – 4696
[8]. Pingali, P., Jagarlamudi, J., and Varma, V. 2006,
“WebKhoj: Indian language IR from multiple character encodings”,
In Proceedings of the 15th International Conference on World
Wide Web (Edinburgh, Scotland, May 23 - 26, 2006), WWW '06.
ACM Press, New York, NY, 801-809.
[9]. Conghui Zhu, Jie Tang, Hang Li, Hwee Tou Ng, Tie-Jun
Zhao, “A Unified Tagging Approach to Text Normalization”, in
Proceedings of ACL.2007
[10]. Gralioski Filip, Jassem Krzysztof, Wagner Agnieszka,
Wypych Mikołaj, “Text Normalization as a Special Case of Machine
Translation”, Proceedings of the International Multiconference on
Computer Science and Information Technology pp. 51–56.
44