Noname manuscript No.

(will be inserted by the editor)

Facial expression recognition using SVM-CHARM

Ariel Esaú Ortiz Esquivel · Claudia Cruz

Martı́nez · José Francisco Martı́nez
Trinidad · Jesús Ariel Carrasco Ochoa

Received: date / Accepted: date

Abstract Human expression recognition is a topic in computer vision and

image processing widely studied for multiple researchers using different ap-
proaches. In this works we classify seven different basic human facial expres-
sions (angry, contempt, disgust, fear, happy, sadness, surprise) through face
images from CK+ database. The proposed method begins with face detection,
after that we propose to use the WND-CHARM method for extracting differ-
ent features on important regions of the image as the eyes, mouth, nose and
eyebrows, which contain information about the expression shows the person
at that moment. The final representation of each image is a feature vector
with 7125 numeric attributes, then the main features are extracted by Reli-
efF algorithm allowing select the most important values and shrink the vector.
The proposed classification method accuracy of human expressions get 84.71%,
outperforming other proposed approaches.

Ariel Esaú Ortiz Esquivel

Instituto Nacional de Astrofı́sica, Óptica y Electrónica
Puebla, México
E-mail: arieleo@ccc.inaoep.mx
Claudia Cruz Martı́nez
Instituto Nacional de Astrofı́sica, Óptica y Electrónica
Puebla, México
E-mail: ccmartinez@ccc.inaoep.mx
José Francisco Martı́nez Trinidad
Instituto Nacional de Astrofı́sica, Óptica y Electrónica
Puebla, México
E-mail: fmartine@inaoep.mx
Jesús Ariel Carrasco Ochoa
Instituto Nacional de Astrofı́sica, Óptica y Electrónica
Puebla, México
E-mail: ariel@inaoep.mx
2 Ariel Esaú Ortiz Esquivel et al.

1 Introduction

The study of human emotions recognition has been covered for different appli-
cations in areas such as human-computer interaction (HCI), security systems,
robotics and computer vision. The human face is very expressive and each of
its states are relatively easily recognizable by a human. However address this
problem automatically is very complex because it requires specialized tech-
niques of image processing, pattern recognition and classification for offer a
satisfactory solution.

In this paper, we propose to use the Cohn-Kanade database extended ver-

sion (CK +) [1]. Which comprises of 210 adults, between 18 and 50 years
old, with 68% women, 81% Euro-American, 13% Afro-American and 6% other
ethnic groups. Each expression is recorded in 23 facial series, beginning and
ending with a neutral expression. Each face includes seven different emotions
(angry, contempt, disgust, fear, happy, sadness, surprise).

In order to detect the human face region within each image, we used Con-
strained Local Model (CLM) technique [2] which allows detection of key points
belonging to the face. To make the feature vector corresponding to each image,
we propose to use the feature set generated by Weighted Neighbor Distances
using a Compound Hierarchy of Algorithms Representing Morphology (WND-
CHARM) [3] on 7 keys face regions that according to [4] those are the most
characterize for facial expressions.

The result after the extraction of the characteristics set is a vector with
7125 values for each human face, witch represent a high computational cost
for the classification task. We proposed to reduce the amount of elements in
the vector using the ReliefF algorithm, witch make a characteristics selection
without lose information and present what are the most important character-
istics for this problem. Support Vector Machine (SVM) is used for expression
classification, as proposed in [2], where the same image set of human emotions
is used. In the results section a comparison with this work is done.

The proposed method is called SVM-CHARM because use SVM, for facial
expression classification, and WND-CHARM, for the feature extraction part
(which originally used the Weighted Neighbor Distances approach to data clas-
sification method).

1.1 Feature set WND-CHARM

WND-CHARM is a defined feature set that can be extracted from any picture
using a lot of filters and transformations from the original data, where the cal-
culated values can be used in any classification problem based on feature vec-
Facial expression recognition using SVM-CHARM 3

tors. Its name is derived from the combination of the classification algorithm
Weighted Neighbor Distances (WND) and feature extraction process called
Compound Hierarchy of Algorithms Representing Morphology (CHARM)[3].

There are two versions of this set: a) the extended features set, consisting
of 2046 values and, b) the reduced features set, with 1025. In this paper, the
reduced version of WND-CHARM is used to carry out tests on images. In
general, the extracted features with this method are relate to shape, texture
and appearance of the analysed image. WND-CHARM aims to create a feature
set that allows to separate objects of each class defined by a specific problem.
When this features are extracted from the human face images, it is expected
to be able to make a distinction between each of the expression mentioned.

1.2 Ranking features using ReliefF algorithm

ReliefF is a feature selection algorithm proposed in [5]. This technique assumes

that one of the attributes used in the classification problem is relevant if it can
distinguish between elements of different classes more easily. It is assigned a
weight to each value used in the feature vector. Later, given a random instance
of any class, ReliefF search the nearest neighbors of the same class and the
closest elements of other classes. Then updates the values of weights for each
feature using the difference between the values of the selected instance and
its closest neighbors, assigning a positive value when there is big difference
between different classes and subtracting the difference between elements of
the same class. Finally, ReliefF produces a weight vector corresponding to the
feature set, indicating the importance of each feature in classification problem.

2 Description of the proposed method

The general process of proposed classification method for facial expressions is

shown in Figure 1. To perform the experiments, the database used is proposed
and described in [1]. The algorithms and techniques used were implemented
using Matlab .
R

As a first step, it is necessary to align all faces for further processing, in

this step the key points are detected in the face for each image and a process of
geometric transformation on these points is applied, after that all images have
the same orientation. Following proposed in [4], the face division is performed
in seven relevant regions for the feature vector generation CHARM, ending
this phase with the selection of the most important attributes in vector by
ReliefF algorithm.
4 Ariel Esaú Ortiz Esquivel et al.

Fig. 1: Flowchart of classification expression system proposed.

2.1 Detection and alignment of key points

The CK+ database [1] contains images divided into seven categories of basic
emotions: angry, contempt, disgust, fear, happy, sadness, surprise. In each of
the images it is required to delimit the region corresponding to the human
face. We propose to use CLM which identifies key facial points in an image
[2].

The faces in each image has a different position compared to the others,
therefore it is necessary to align them to coincide. The resulting model for
each emotion was calculated by a geometric transformation. Figure 2 shows
the result of applying CLM on one of the images. After this, we segment the
region of interest for the human face to be analysed.

2.2 Segmenting face regions and feature vector extraction

The next step is the extraction of feature vector. According to [4], we make
a division of the face in seven relevant regions. These regions defined the ex-
pressed emotion at that instant and correspond to: eyes, mouth, nose and
eyebrows (Figure 3).

For each of the seven regions CHARM feature set is calculated, resulting
in a vector of 7125 values. Because the vector has a lot of elements, it is com-
putationally difficult to treat. Therefore, a selection process of attributes was
applied using ReliefF algorithm. Thus, it is possible to remake the feature
Facial expression recognition using SVM-CHARM 5

Fig. 2: Application of CLM for detecting key points

Fig. 3: Regions of interest

vector using a subset of the original full feature set without being affected the
results of the classification process.

2.3 Classification using Support Vector Machines

Support Vector Machines (SVM) have proven useful in different pattern clas-
sification tasks, which include the recognition of facial expressions in images
[4]. The classification usually involves separating the large amount of data into
two sets: a) the training set and, b) the test set. Each instance in the test set
6 Ariel Esaú Ortiz Esquivel et al.

contains a target value (labeled class) and various attributes (character-

istics or observations). SVM tries to find the hyperplane that maximizes the
margin between positive and negative remarks for a specific class. The main
objective is to produce a model based on the training data, which predicts
values objectives test data attributes giving only test data model. This classi-
fication is defined as an observation test untagged x∗ by:

wT x∗ > b
< f alse

where w is the normal vector to the separating hyperplane y b is the bias [6,1].

Based on the work [1], the linear SVM classification is possible due to their
ability to generalize the data as vector values.

3 Experiments and Results

Classification experiments performed for facial expressions using CK + database

[1], which contains a total of 327 different samples of seven different emotions
of 118 people. Table 1 shows the number of images available for each of the
emotions referred to. The test for training and classification are performed by
cross validation leave-one-subject-out.

Table 1: Number of images per facial expression in CK+ database

Emotion Number of images

Angry (An) 45
Contempt (Co) 18
Disgust (Di) 59
Fear (Fe) 25
Happiness (Ha) 69
Sadness (Sa) 28
Surprise (Su) 83

In order to determine the percentage of features to be acquired in feature

vector using ReliefF, tests are performed varying the percentage of most im-
portant values. Tables 2 (using 100% of data), 3 (using 90% of data), 4 (using
80% of data), 5 (using 70% of data), 6 (using 60% of data), 7 (using 50% of
data), 8 (using 40% of data), 9 (using 30% of data), 10 (using 20% of data)
y 11 (using 10% of data) shows the confusion matrix of different tests. The
best accuracy result is obtained by using 20% of the major features, reaching
84.71%.
Facial expression recognition using SVM-CHARM 7

Table 2: Confusion matrix for facial expression classification using 100% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 71.11 0.0 0.0 0.0 0.0 28.89 0.0
Di 3.39 86.44 0.0 0.0 0.0 10.17 0.0
Fe 0.0 0.0 56.0 4.0 0.0 36.0 4.0
Ha 0.0 0.0 0.0 95.65 0.0 4.35 0.0
Sa 10.71 0.0 0.0 0.0 39.29 50.0 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 3: Confusion matrix for facial expression classification using 90% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 71.11 0.0 0.0 0.0 0.0 28.89 0.0
Di 0.0 89.83 0.0 0.0 0.0 10.17 0.0
Fe 0.0 0.0 56.0 4.0 0.0 36.0 4.0
Ha 0.0 0.0 0.0 94.20 0.0 5.80 0.0
Sa 14.29 0.0 0.0 0.0 35.71 50.0 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 4: Confusion matrix for facial expression classification using 80% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 68.89 0.0 0.0 0.0 0.0 31.11 0.0
Di 1.69 88.14 0.0 0.0 0.0 10.17 0.0
Fe 0.0 0.0 56.0 4.0 0.0 36.0 4.0
Ha 0.0 0.0 0.0 95.65 0.0 4.35 0.0
Sa 14.29 0.0 0.0 0.0 35.71 50.0 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

The best performance for detecting emotions is shown in Table 10 where

20% of the most important features is used. It shows that emotions such as
Happiness, Surprise and Disgust had good performing in terms of ranking
compared to other emotions. This result can be explained because these emo-
tions are visually easier to identify than others. The accuracy rate for each
8 Ariel Esaú Ortiz Esquivel et al.

Table 5: Confusion matrix for facial expression classification using 70% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 71.11 0.0 0.0 0.0 0.0 28.89 0.0
Di 0.0 89.83 0.0 0.0 0.0 10.17 0.0
Fe 0.0 0.0 56.0 4.0 4.0 32.0 4.0
Ha 0.0 0.0 0.0 95.65 0.0 4.35 0.0
Sa 14.29 0.0 0.0 0.0 39.29 46.43 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 6: Confusion matrix for facial expression classification using 60% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 71.11 0.0 0.0 0.0 0.0 28.89 0.0
Di 0.0 93.22 0.0 0.0 0.0 6.78 0.0
Fe 0.0 0.0 64.0 4.0 4.0 28.0 0.0
Ha 0.0 0.0 0.0 95.65 0.0 4.35 0.0
Sa 14.29 0.0 3.57 0.0 39.29 42.86 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 7: Confusion matrix for facial expression classification using 50% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 68.89 0.0 0.0 0.0 0.0 31.11 0.0
Di 0.0 91.53 0.0 1.69 0.0 6.78 0.0
Fe 0.0 4.0 68.0 4.0 0.0 24.0 0.0
Ha 0.0 0.0 0.0 94.2 0.0 5.8 0.0
Sa 14.29 0.0 3.57 0.0 42.86 39.29 0.0
Su 0.0 0.0 0.0 0.0 0.0 96.39 3.61
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

emotion are: 66.67% (Anger ), 86.44% (Disgust ), 76% (Fear ), 94.2% (Happi-
ness), 53.57% (Sadness), 96.39% (Surprise) and 94.44% (Contempt ).

Next, we present the results of the related works in order to make a com-
parison between they and our proposed work using a confusion matrix. Table
Facial expression recognition using SVM-CHARM 9

Table 8: Confusion matrix for facial expression classification using 40% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 68.89 0.0 0.0 0.0 0.0 31.11 0.0
Di 1.69 89.83 0.0 0.0 0.0 8.47 0.0
Fe 0.0 4.0 64.0 4.0 0.0 28.0 0.0
Ha 0.0 0.0 0.0 94.2 0.0 5.8 0.0
Sa 14.29 0.0 3.57 0.0 46.43 35.71 0.0
Su 0.0 0.0 0.0 0.0 0.0 97.59 2.41
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 9: Confusion matrix for facial expression classification using 30% of

SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 66.67 0.0 0.0 0.0 0.0 33.33 0.0
Di 1.69 91.53 0.0 0.0 0.0 6.78 0.0
Fe 0.0 4.0 60.0 0.0 4.0 32.0 0.0
Ha 0.0 0.0 0.0 92.75 0.0 7.25 0.0
Sa 14.29 0.0 3.57 0.0 50.0 32.14 0.0
Su 0.0 0.0 0.0 0.0 0.0 97.59 2.41
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 10: Confusion matrix for facial expression classification using 20% of
SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 66.67 4.44 0.0 0.0 0.0 28.89 0.0
Di 6.78 86.44 0.0 0.0 0.0 6.78 0.0
Fe 0.0 4.0 76.0 0.0 4.0 16.0 0.0
Ha 0.0 0.0 0.0 94.2 0.0 5.8 0.0
Sa 14.29 0.0 3.57 0.0 53.57 28.57 0.0
Su 1.2 0.0 0.0 0.0 0.0 96.39 2.41
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

12 show the results reported by [1], witch it use Similarity Normalized Shape
(SPTS) for the characteristics extraction. We observed that [1] has a good
performance in emotions like Fear and Surprise, but the proposed method has
a better performance in emotions like Sadness, Angry, Fear and Contempt
.
10 Ariel Esaú Ortiz Esquivel et al.

Table 11: Confusion matrix for facial expression classification using 10% of
SVM-CHARM feature vector.

An Di Fe Ha Sa Su Co
An 68.89 4.44 0.0 0.0 2.22 24.44 0.0
Di 6.78 81.36 0.0 1.69 0.0 10.17 0.0
Fe 0.0 4.0 60.0 0.0 12.0 24.0 0.0
Ha 0.0 0.0 1.45 89.86 0.0 8.7 0.0
Sa 14.29 0.0 3.57 0.0 50.0 32.14 0.0
Su 1.2 0.0 0.0 0.0 1.2 95.18 2.41
Co 0.0 0.0 0.0 0.0 0.0 5.56 94.44

Table 12: Confusion matrix for facial expression classification using Similarity
Normalized Shape (SPTS).

An Di Fe Ha Sa Su Co
An 35.0 40.0 0.0 5.0 5.0 15.0 0.0
Di 7.9 68.4 0.0 15.8 5.3 0.0 2.6
Fe 8.7 0.0 21.7 21.7 8.7 26.1 13.0
Ha 0.0 0.0 0.0 98.4 1.6 0.0 0.0
Sa 28.0 4.0 12.0 0.0 4.0 28.0 24.0
Su 0.0 0.0 0.0 0.0 0.0 100.0 0.0
Co 15.6 3.1 6.3 0.0 15.6 34.4 25.0

Table 13 show the results of [1], but using Canonical Appearance (CAPP)
as characteristics extraction. This algorithm gets better in emotions like An-
gry and Sadness. However, it still having a less classification rate in Fear and
Contempt, where our method have a better performances.

Table 14 show the results using the combination of the previous methods
(STPS+CAPP). It present a better success rate in all emotions. Nevertheless
our method keeps a better performance in Fear, Happiness, Surprise and Con-
tempt.

Table 15 present the results of [4]. In that work, do not considered the
Contempt emotion, therefore we only make a comparison with the other emo-
tions. Our proposed method gets a better performance in the classification of
the most of emotions, no matter how many acquired characteristics gets from
the ReliefF algorithm.
Facial expression recognition using SVM-CHARM 11

Table 13: Confusion matrix for facial expression classification using Canonical
Appearance features.

An Di Fe Ha Sa Su Co
An 70.0 5.0 5.0 0.0 10.0 5.0 5.0
Di 5.3 94.7 0.0 0.0 0.0 0.0 0.0
Fe 8.7 0.0 21.7 21.7 8.7 26.1 13.0
Ha 0.0 0.0 0.0 100.0 0.0 0.0 0.0
Sa 16.0 4.0 8.0 0.0 60.0 4.0 8.0
Su 0.0 0.0 1.3 0.0 0.0 98.7 0.0
Co 12.5 12.5 3.1 0.0 28.1 21.9 21.9

Table 14: Confusion matrix for facial expression classification using

STPS+CAPP

An Di Fe Ha Sa Su Co
An 75.0 7.5 5.0 0.0 5.0 2.5 5.0
Di 5.3 94.7 0.0 0.0 0.0 0.0 0.0
Fe 4.4 0.0 65.2 8.7 0.0 13.0 8.7
Ha 0.0 0.0 0.0 100.0 0.0 0.0 0.0
Sa 12.0 4.0 4.0 0.0 68.0 4.0 8.0
Su 0.0 0.0 0.0 0.0 4.0 96.0 0.0
Co 3.1 3.1 0.0 6.3 3.1 0.0 84.4

Table 15: Confusion matrix for facial expression classification using Robust
Metric Learning

An Di Fe Ha Sa Su
An 34.2 20.1 10.5 5.2 7.8 7.4
Di 2.4 60.4 6.5 2.2 0.0 8.1
Fe 19.7 15.3 40.4 0.0 4.2 0.0
Ha 0.7 4.9 0.0 94.4 0.0 0.0
Sa 0.2 0.0 0.0 0.0 93.5 0.0
Su 1.1 0.0 4.2 5.2 0.0 89.5

4 Conclusions

This paper presents a method for facial expression classification. Using the
feature set proposed by WND-CHARM on specific regions of the human face
regions is adequate to address this problem. The application of this technique
involves generating a large number of features and arrange them into a vector
12 Ariel Esaú Ortiz Esquivel et al.

of values, which implies a high computational cost, due to this, a selection

process of the main attributes ReliefF is applied in order to generate a vec-
tor easier to handle, as well as improving the final accuracy of classification
of emotions. We propose the use of SVM classifier, replacing the method of
Nearest Neighbor classification WND proposed in [3] getting performance im-
provements of the system when is tested on the CK+ database.

As future work, we proposed a experimentation in other databases for com-

parison with other related works. Likewise, we proposed to verify, based on the
features extracted by ReliefF, whether we can define methods that conform
more to the problem itself. Finally we proposed to try some other method
of detection of areas on the face, since in this work a better result from the
image to 7 patches, unlike other jobs using full face was found. To this, we pro-
posed to optimize the calculation of the benchmarks in the face for improved
computational performance without leaving the classification performance.

References

1. P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, in Computer

Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society
Conference on (IEEE, 2010), pp. 94–101
2. X. Zhu, D. Ramanan, in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE
Conference on (IEEE, 2012), pp. 2879–2886
3. N. Orlov, L. Shamir, T. Macura, J. Johnston, D.M. Eckley, I.G. Goldberg, Pattern recog-
nition letters 29(11), 1684 (2008)
4. S. Wan, J. Aggarwal, Pattern Recognition 47(5), 1859 (2014)
5. M. Robnik-Šikonja, I. Kononenko, Machine learning 53(1-2), 23 (2003)
6. C.J. Lin, C.W. Hsu, C.C. Chang, National Taiwan U., www. csie. ntu. edu. tw/cjlin/pa-
pers/guide/guide. pdf (2003)