Recognizing Arabic Handwritten Script uSIng

Support Vector Machine classifier

Mohamed Elleuch Roussem Lahiani

National School of Computer Science (ENSl), University of National School of Electronics and Telecommunications,
Manouba, Tunisia Univesrity of Sfax, Sfax, Tunisia
Advanced Technologies for Medicine and Signals (ATMS), lahianihoussem@gmail.com
University of Sfax, Tunisia
mohamed.elleuch.2015@ieee.org
Monji Kherallah
Faculty of Sciences, University of Sfax, Tunisia
monji.kherallah@gmail.com

Abstract- Handwriting recognition ranks among the issue, Arabic handwritten script recognition has been deeply
highest and the most triumphant applications in the pattern investigated for years in the handwriting recognition
recognition domain. Despite being a developed field, many domain. Different algorithms like Support Vector Machine
enquiries are still needed and still represent a defiance mainly (SVM), Multi-Layer Perceptron (MLP), Artificial Neural
for the Arabic Handwritten Script (AHS). Recently, more Networks (ANN), Hidden Markov Model (HMM) and
regard has been given to Support Vector Machines (SVM) Convolutional Neural Networks (CNN), etc., have been
classifier for script recognition. Nevertheless, it has not been
exploited by researchers who have attained a lot of
put in application yet to the handwritten Arabic field if
favorable results. Performance and accuracy have been
compared with the other methods like ANN, CNN, RNN and
demonstrated by the systems in a large field of applications.
HMM. SVMs for AHS recognition is examined in this paper.
Alike, these systems have acquired a great triumph in
Handcrafted feature is handled as input by the suggested
method and gets going with a supervised learning algorithm.
Optical Character Recognition (OCR) in Latin and Asian
We chose the Multi-class Support Vector Machine with an
languages [5, 6]. A lot of researchers have utilized Hidden
RBF kernel and we tested it on Handwritten Arabic
Markov Models for offline Arabic handwritten and printed
Characters Database (HACDB) as well. It was proven that the recognition. It is noticeable that these models have proved
proposed method was effective thanks to the simulation results. their forcefulness [7, 8, 9, 10]. Y et, some elastic models like
We compared the well-functioning of this method with HMM are still considered a great hinder in Arabic
character recognition reliabilities coming from state-of-the-art handwriting recognition because of due to the huge
Arabic OCR which resulted in commendatory outcomes. variability and distortions of patterns.

Keywords-SVM; arabic handwritten recognition; handcraft CNN is made up of a number of convolutional and sub
feature; HACDB sampling layers which are non-mandatory accompanied by
Fully Connected Layers. The Fully Connected Layers are
the same as to the layers in a standard Multi-Layer
I. INTRODUCTION
Perceptron. Yet, MLP shows two restrictions in
During the two last decades and within the context of classification tasks: to begin with, there is not any
signal processing and pattern recognition, offline and online theoretical relationship between both the classification task
data classification has been granted great attention. It has and the MLP structure. Next, MLP derive hyper-planes
been widely applied to a variety of research fields naming as separation surfaces, in feature representation space, which
examples vision recognition task [I, 2], Automatic Speech are not optimal in terms of margin between the examples of
Recognition (ASR) [3] and analysis of two different classes.
Electroencephalograms (EEGs) signals [4] and
classification. It plays a progressively important perspective On the other hand, being considered as one of the
strongest and most vigorous algorithm in machine learning
in the Human-Computer Interaction field. Besides,
computer feigning and machine simulations of human invented by Vapnik [ I I] Support vector machines (SVM),
reading/writing/gesture have been given great importance have turned into being a popular approach used in various
for the last years. domains [12, 13, 14], like pattern classification, recognition,
and image processing.
The writing recognition belongs to the wider field that is
the pattern recognition. Mostly, because of being a detying Using SVM, Byun and Lee [IS] who handed out survey
applications of pattern recognition reviewed seven

978-1-4673-8709-5/15/$31.00 2015 IEEE 551

categories in relation with their aims like object recognition,
handwritten character/digit recognition, face
detection/verification and many others. Likewise, Chen et
al. [16] presented a recognition system exploiting SVM.
They proved that Gabor features are more effective than the
previous used features techniques.
Till recently, SVM method has not been practiced yet on
the handwritten Arabic field. In this paper, we have
suggested a system built on SVM classifier for offline
Arabic handwritten text recognition. For this reason, the
plausible advantages of the proposed SVM classifier have
been studied. Fig. 1 showed our suggested system
architecture among which the pre-processing steps, next Fig. 2. Principle of SVM; two-class hyper-plane example, SV are Support
feature extraction step and ultimately the SVM takes the Vectors and Ho is the optimal hyper-plane defined as that which maximizes
handcraft features as a feature vector for classification. the margin,that is, the distance to the classes Class 1 and Class 2

The organization of the rest of the paper is the Determining an optimal separating hyper-plane
following. Tn Section IT, we describe the basic concepts (equation 2) or deciding a surface by adopting an
behind Support Vector Machine (SVM). Our experimental unconventional technique founded on mapping the sample
study using this model is next presented in the Section m. points into a high-dimensional feature space and categorized
Finally, Section TV discusses the overall results of the model utilizing a nonlinear transformation <D, even with non
and presents some concluding remarks. linearly separable data is the fundamental concept of SVM .
The most favorable hyper-plane is acquired by solving a
quadratic programming problem subject to regularization
parameters (equation 4). This transformation has been
! executed by kernel functions stating as examples linear,
Pre-processing Radial Basis Function, Sigmoid and Polynomial kernel
types which are defined as;

The linear kernel: K(x, y) = x x y

Feature E .1..'iractioD
The polynomial kernel: K(x, y) = [(x x y) + I]d
The Sigmoid kernel: K(x, y) = tanh (o x y + l)
D RBF kernel (Radial Basis Function):
Handcraft Features K(x, y) = exp(- y Ilx Y112) _

L' 1 1 1 1 1 y= 1 / ()2
[I 1 1 1 I I
I I 1 ..... 1 I I With d, o, 1' and yare parameters that will be determinate
empirically.

SVM
1 L-. f(x) = W T <D(x) + b (2)
I Classifier I -
Where W E: R" , b E: R and <D(x) is a feature map.

Fig. I. Proposed system overview. The generalized optimal separating hyper-plane is

determined by the vector w that minimizes the functional:

IT. SUPPORT VECTOR MACHINE (SVM)

(3)
Being developed by Vapnik [11] and Cortes [17],
Support Vector Machine is a strong discriminative classifier
which has been largely used with success for many pattern Where Si ;:: 0, which measure the miss-classification errors,
c1assification/recognition tasks [18]. It's regarded as be the and C are constants.
state-of-the-art tool for resolving linear and non-linear (see
I 1 I
Fig. 2) classification problems [11], thanks to its flexibility, Maximize W(a) "
a JaaJ
-- "y.y K(x x)l' J
(4)
i=1
=

.1=1
1 1 1

parsimoniousness, prediction capacity and the global 2

optimum character. Their formulation is based on the
y,ai with C >= ai > = 0, i=l, ... ,1
i=!
structural risk minimization, better than the empirical risk Subject to = 0
minimization traditionally used in artificial neural networks
Where Yi represents the label for i-th training sample, I is
[11]. The goal is to find a parameter setting that minimizes
the risk given by:
the number of training samples and C is real parameter
which is varied through a wide range of values.
Rem/a) = 21
1
IYi - f(xi,a)1
I
(1)
The most commonly used methods are founded mainly
on the fundamental SVM (binary model) which is for two
Where Yi is the expected output, Xi represents the given class problem, named, the one-versus-one and the one
input and a is a set of parameters. versus-all approach (see Fig. 3). The problem is sorted out

552 2015 15th International Conference on Intelligent Systems DeSign and Applications (ISDA)
by these approaches once the number of classes exceeds 2 training set of 5.280 images and a test set of l.320 images.
classes, called multi-class SVM. Work in [19, 20] provides Details of the class for each shape are presented in Table 1.
more detailed information.
Optimal choice of kernels is crucial for determining how
<Y"
to carry out the projection into higher-dimensional space. In saad
this paper, because of the linearly inseparable of the feature
'J
space, an alteration was put in an application by mapping
Lam Alif
the input data (Xi, Yi) into a higher dimensional feature space
utilizing a nonlinear operator <!lex). Hence we can define,
the optimal hyper-plane (Ho) as the following:
Lam mim ha
(5)
..,jl J,\. J).. j.) Jl .J\ j Jj j,t
Where K(Xi' x) = exp(-Yllxi - x112) is the kernel function Alif lam ha
based on a Radial Basis Function (RBF), and sgn(.) is the
sign function. This classifier model named RBF kernel
SVM is used to do classification for Arabic Handwritten
Text. Fig. 4. Samples from the HACDB database written by 10 different
writers.
y

TABLE T. CLASS FOR EACH SHAPE OF AN ARABIC SCRIPT

Arabic
Shape class Arabic Script Shape class
Script

Aeen (t)
t- I
J- 34

Class 2 .J.. 2
L 35

0
3 Lam_Alif (':!) 36

(a)
x
t: 4 )f 37

5 38
l::J. Class 3 Lam .Teem
y l::J. / .....Q.... 6 -.d 39
(..."J)
l::J. l::J.
l::J. , ...... """
\
l::J. f13(X}
fd x } l::J. Alif (l) 7 Lam_Mem (...J) ...(, 40
l::J../
"- /
l 8
Lam Mem .Te
e m () 41

Alif Lam .Te

em (.:,JI) -
JI, 9 Meem () r 42

Baa (y) L-' 10 ------Zl 43

Class 1
----> \I
r 44

c....N 12 ----0- 45
x
(b) ----"- Mem .Teem
13
( ) 46
Fig. 3. Approach's of Support Vector Machine; (a) one-versus-all
method,(b) one-versus-one method Daal(o) 14 Noon (0)
U 47

...I...- 15 U 48
m. EXPERIMENTAL RESULTS AND DISCUSSION Faa (w) C9 16 Raa (.)) J 49
We performed our experimental studies using SVM
model for recognizing offline Arabic character. We
-..9 17 Y 50

LA...- 18 Saad ()o"') IY' 51

examined the suggested system on HACDB database [21].
Outcomes are itemized and discussed in the following -1Z--- 19 --+' 52
subdivision.
Haa (,) 0 20 Lf- 53

A. HACDB database 21 -4'- 54

The HACDB database [21] contains 6.600 shapes of c:L 22 Seen (lY') V 55
handwritten characters written by 50 persons (Fig. 4). Each
writer has generated two forms for 66 shapes: 58 shapes of --8-" 23 (,)- 56

characters and 8 shapes of overlapping characters 24 57

(representing 24 basic characters/overlapping characters
Hamza (.) .c 25 -----1 58
without dots). The images are normalized 28 by 28 pixels
and are in the gray scale. The dataset is divided into a .Teem (d L 26 cr- 59

2015 15th International Conference on Intelligent Systems DeSign and Applications (ISDA) 553
 27 60 <I> denotes the orientation of a Gabor filter,
o"x, O" represent the standard deviation in x- and y
t- 28 Taa (.b)
61 y
direction, respectively.
../:.--- 29 62

Kaaf (.)
30 Waao (J)
63
ll(u,,; A,", "" "
,
J c exp - 2, '
{ ( ( J ,, : c; - + ";(F,)' )}
i:,

31
)- 64

Laam (J)
J 32 Yaa (c.
05 65 (7)

J 33
0-
B. Experiments setting
[n order to evaluate the efficiency of the suggested
system based on SVM classifier, we investigated its
performance for training and recognizing characters of
HACDB database. To effectively train the model on more
data so as to perfectly handle the variability of handwriting,
the size of the training set is extended ten times by the
elastic deformation technique suggested by Simard et al
[22]. Technical execution specifics of the selected system
are given in the next subdivision.
Pre-processing: The pre-processing phase where the
database utilized in this experimental study does not
require to be normalized. It is noticeable that a few
fundamental pre-processing activities are vitally to be
conducted throughout the database development.
Parameters setting: For the setting architecture, we must
determine the optimal kernel parameter and penalty
parameter of SVM. The value of the tradeoff parameter
C and a parameter in SVM are chosen empirically.
Feature extraction: Features are the information
extracted from the image of a word or character, and they
are used to build classifiers for classification. The
challenge is to determine which features are more
suitable for classification. Tn this paper, Gabor filter [16]
was used in this experiment.
Gabor filters are used widely in image processing for its
good characteristics as; the optimal joint spatial/spatial
frequency localization and the fitness to simulate the
receptive fields of simple cells in the visual cortex [23, 24].
They depict an image in different levels of frequency.
Therefore, various features are extracted, depending on
everyone's response of filter and its frequency. Tn [25]
Daugman incited the use of Gabor filters as a representation
for receptive fields and their exploit in recognition systems.
Two-dimensional Gabor filter is a complex sinusoidally
modulated by a Gaussian function with the response in the
spatial domain (equation 6) and in frequency domain
(equation 7), defined as follows [16, 26]:
66 Where FJ , Fz and C are:
F; = ucosl/l+vsinl/l
F2 = -usinrjJ + vcosrjJ
And C is a constant.
C. Experiments using SVM model
Tn our experiments, we investigated the performance of
the SVM model for training and recognizing Arabic
characters. For the setting architecture, we need to
determine about SVM classifier essentially two parameters
of the RBF kernel; Gamma (y) and C.
We selected the pertinent parameters for SVM model on
the basis of empirical tests. We inaugurated an experimental
study so that we can assess the suggested model. Our
selection of parameters is on the basis of the criterion of the
error classification rate on the train dataset. We also used the
one-versus-all method with 66-way for the multi-class RBF
kernel SVM as it provides a more valuable discrimination
than the linear kernel. Yet, a less parameter than the
polynomial kernel was utilized.
Tn this work, Gabor features is the input to the SVM
classifier. Nevertheless, the SVM assumes the handcraft
features in the role of a feature vector to recognize the
unrevealed handwriting text. And consequently by
analyzing the error classification rate on the Arabic
handwritten character classification task.
We indicated that in our experiments, LTBSVM [27] tool
is considered to set up multi-class SVM classifier with RBF
kernel. LTBSVM implements the SMO-based algorithm. We
utilized a soft-margin SVM marked by the first parameter C
which controls the margin from a class boundary and the 0"
parameter which controls the variance in the RBF kernel.
The parameter C is the tradeoff between training set and the
error in the separation.
The choice of parameters (C, 0") was proved to be
experimentally effective by applying a grid search with the
5-fold stratified cross validation method. We synthesized the
most favorable usefulness of principal parameters obtained
after the tests on the training Arabic handwritten text

h(x,y;:t,,<Jx,<Jy)
1
exp
{I [
- - -2 +-2
R\ 2 R ]} x exp
[ 27fR\ ]
.
/--
database HACDB as follows: a = 0.05 and C = 30.
The suggested system founded on RBF kernel SVM
27f<Jx<Jy 2 <Jx <Jy :t
already described, provides an error classification rate
(6) (ECR) of 11.23% on the testing dataset with 66 classes (see
fig. 5) while SVM with polynomial kernel attained an error
Where R1 , R2 ,Ie, <1>, ax and ay are explain as: classification rate of 29.18%.
R1 = xcosr/J+ ysinl/l

R2 = -xsinl/l+ ycosl/l

Ie denotes the wavelength of a Gabor filter,

554 2015 15th International Conference on Intelligent Systems DeSign and Applications (ISDA)
 to increase the recognition rate.

TABLE Ill. PERFORMANCE COMPARISONS USING HANDWRITTEN

ARABIC DATABASE

Authors Methods Data bases (class) ECR

Present work SVM HACDB (66) 11.23%
Arabic characters
LawgaJi et al 2 1 . 18% (5600
ANN (old version of
[28] shapes)
HACDB)
Elleuch et al
CNN HACDB (66) 14.7 1 %
[29]
Chen et al AMA Arabic Dataset
SVM 17.3%
[ 1 6] (34) [30]
Azeem and
HMM IFNIENIT [3 1] 7%
Ahmad [ 1 0]

Fig. 5. Samples random of 149 incorrectly classifIed characters using RBF

kernel SVM
TV. DISCUSSION AND CONCLUSION
The suggested system was juxtaposed with some other
already suggested methods. Tn table IT, SVM with RBF
kernel was shown to be performing better than polynomial
kernel once experimented on the selfsame database
HACDB. It was clear that there was a noteworthy profit in
error classification rate (ECR) in caparison with polynomial
kernel SVM in which the complete identification rate was
enhanced by 18.05% with the suggested RBF kernel SVM
for the 66 class problem. The ECR gained with the RBF
kernel SVM model (with 66 classes) on the HACDB
database is equivalent to 11.23%. Compared with character
recognition reliabilities got from state-of-the-art OCR
systems (see Table TIT), this rate has a statistically
significant importance.
TABLE IT. ECR FOR OUR PROPOSED SYSTEMS APPLIED ON HACDB
DATABASE
ECR
Approach
HACDB database (66 classes)
RBF kernel SVM 11.23 %
polynomial kernel SVM 29. 18 %
A comparative study of our system with other
techniques utilizing handwritten Arabic database (see Table
TIT) was also realized. Our SVM model with RBF kernel
still performs better than hand-crafted features-based
approach like ANN [28], SVM [16] and HMM [10] methods
and automatic method of feature extraction such as CNN
[29].
Tn general, we deduce that our suggested system
performs better than the other current methods by producing
better results. RBF kernel SVM is quite a promising
classification method in the handwriting recognition
domain. As perspective, the expansion of our suggested
system by integrating other handcraft features with Gabor
filter so that we can heighten the recognition rate is vital.
Yet, we have to combine our proposed SVM architecture
with a further classifier such as CNN or Deep CNN in order
REFERENCES
[I] H. Lee. R. Grosse, R. Ranganath, A Y. Ng, "Unsupervised learning
of hierarchical representations with convolutional deep belief
networks," Communications of the ACM, 54( 10) (20 1 1) 95- 103.
[2] G.-B. Huang, H. Zhou, X. Ding, R. Zhang, "Extreme Learning
Machine for Regression and Multiclass Classification," IEEE
Transactions on Systems, Man, and Cybernetics - Part B:
Cybernetics,vol. 42(2), pp. 5 13-529,20 12.
[3] H. Lee, PT. Pham, Y. Largman, AY. Ng, "Unsupervised feature
learning for audio classifIcation using convolutional deep belief
networks," Advances in Neural Information Processing Systems
(NIPS),pp. 1096- 1 104,2009.
[4] Y. Ren, Y. Wu, "Convolutional Deep Belief Networks for Feature
Extraction of EEG Signal," International Joint Conference on Neural
Networks (TJCNN),pp. 2850-2853,20 14.
[5] D.C. Ciresan, U. Meier, J. Schmidhuber, "Transfer Learning for
Latin and Chinese Characters with Deep Neural Networks," In
Proceedings of International Joint Conference on Neural Networks,
20 12.
[6] D.C. Ciresan, J. Schmidhuber, "Multi-Column Deep Neural
Networks for Oftline Handwritten Chinese Character Classification,"
In Proceedings of CoRR. 20 13.
[7] R. AI-Hajj, L. Likforman-Sulem, C. Mokbel, "Combining
slantedframe classifIers for improved HMM-based arabic
handwriting recognition," IEEE Trans. Pattern Anal. Mach. Intell.,
voI 3 1(7),pp. 1I65- 1 177,2009.
[8] J.H. AIKhateeb, J. Ren, J. Jiang, H. AI-Muhtaseb, "Offline
handwritten Arabic cursive text recognition using hiddenMarkov
models and re-ranking," Pattern Recognit. Lett., vol. 32(8),20 1 1.
[9] M. Hamdani, H. EI Abed, M. Kherallah, M. Alimi Adel,
"Combining multiple HMMs using online and offline features for
offline Arabic handwriting recognition," In: Proceedings of the 10th
International Conference on Document Analysis and Recognition
(ICDAR),2009.
[ 1 0] S.A Zeem, H. Ahmed, "Effective technique for the recognition of
oftline Arabic handwritten words using hidden Markov models,"
TJDAR,vol. 16(4),pp. 399-4 12,20 13.
[II] V. Vapnik, "Statistical Learn Theory," John Wiley,New York, 1998.
[ 1 2] H. Byun, S.-W. Lee, "A survey on pattern recognition applications
of Support Vector Machines ", International Journal of Pattern
Recognition and ArtifIcial Intelligence,vol. 17,pp. 459-486,2003.
[ 1 3] D. Gorgevik, D. Cakmakov, V. Radevski, "HandwTitten digit
recognition by combining support vector machines using rule-based
reasoning," Proc. 23rd Int. Conf Information Technology Interfaces
(ITT),pp. 139- 144,200 1.
[ 1 4] G. Guo, S. Z. Li, K. Chan, "Face Recognition by Support Vector
Machines," Proc. 4th IEEE IntI. Conf on Face and Gesture
Recognition,pp. 196-20 1,2000.
[ 1 5] H. Byun, S.-W. Lee, "Applications of Support Vector Machines for
Pattern Recognition: A Survey," In Proceedings of the First
International Workshop,SVM 2002,pp. 2 13-236,2002.

2015 15th International Conference on Intelligent Systems DeSign and Applications (ISDA) 555
[ 1 6] J. Chen, H. Cao, R. Prasad, A Bhardwaj, P. Natarajan, "Gabor
features for offline arabic handwriting recognition," In Proceedings
of the 9th IAPR International Workshop on Document Analysis
Systems (DAS),pp. 53-58,20 10.
[ 1 7] C. Cortes, V. Vapnik, "Support vector networks," Machine Learning,
vol. 20,pp. 273-297, 1995.
[ 18] C. Burges, "A tutorial on support vector machines for pattern
recognition," Data Mining Knowledge Discovery,vol. 2(2), pp. 1 2 1 -
167, 1998.
[ 1 9] B. Scholkopf, A Smola, "Learning with Kernels," MIT Press,
Cambridge, USA,2002.
[20] J. Shawe-Taylor, N. Cristianini, "Kernels Methods for Pattern
Analysis," Cambridge University Press, Cambridge,2004.
[2 1] A Lawgali,M. Angelova, A Bouridane, "HACDB: Handwritten
Arabic characters database for automatic character recognition,"
EUropean Workshop on Visual Information Processing (EUVIP), pp.
255-259,20 13.
[22] P. Simard, D. Steinkraus, 1. C. Platt, "Best Practices for
Convolutional Neural Networks Applied to Visual Document
Analysis," International Conference on Document Analysis and
Recognition (ICDAR),pp. 958-962,2003.
[23] 1.G. Daugman, "Uncertainty relation for resolution in space, spatial
frequency, and orientation optimized by two dimensional visual
cortical fIlters," J. Opt. Soc. Am.,vol. 2 (7), 1985.
[24] AK. Jain, F. Farrokhnia, "Unsupervised texture segmentation using
Gabor fIlters," Pattern Recog.,vol. 24( 12),pp. 1 167- 1 186, 199 1 .
[25] 1 . Daugman, "Complete discrete 2 D Gabor transforms b y neural
networks for image analysis and compression," IEEE Transactions
on Acoustics, Speech and Signal Proc., vol. 36(7), pp. 1 169- 1 179,
1988.
[26] X. Wang,X. Ding,and C. Liu,"Gabor fIlter-based feature extraction
for character recognition," Pattern Recognition,vol. 38,pp. 369-379,
2005.
[27] c.c. Chang, C.J. Lin, "LTBSVM: A Library for Support Vector
Machines," Software Available at
http://www.csie.ntu.edu.tw/-cjlin/libsvm. 200I.
[28] A Lawgali, A Bouridane, M. Angelova and Z. Ghassemlooy,
"Handwritten Arabic character recognition: Which feature extraction
method?," International Journal of Advanced Science and
Technology,vol. 34,pp. 1 -8,20II.
[29] M. Elleuch,N. Tagougui,and M. Kherallah, "Towards Unsupervised
Learning for Arabic Handwritten Recognition Using Deep
Architectures," Neural Information Processing - 22nd International
Conference, ICONIP 20 15, Istanbul, Turkey, part. (I), pp. 363-372,
20 15.
[30] Applied Media Analysis, Arabic-Handwritten-I.O,
http://appliedmediaanalysis.comlDatasets.htm. 2007.
[3 1] M. Pechwitz,S. S. Maddouri, V. Margner,N. Ellouze,and H. Amiri,
"IFNIENIT database of handwritten Arabic words," In Colloque
International Francophone sur l'Ecrit et Ie Document (CIFED), pp.
127- 136,2002.

556 2015 15th International Conference on Intelligent Systems DeSign and Applications (ISDA)