1

M. Abdulrahman, T. R. Gwadabe, F. J. Abdu & A. Eleyan

Department of Electrical & Electronics Engineering

Mevlana University Konya, Turkey

Presented by

MUZAMMIL ABDULRAHMAN

2013

 Introduction
Application

of FER
Basic Steps of FER
Principal Component Analysis
Local Binary Pattern
Gabor Wavelet Transform
Classification
Simulation Results
Conclusion
References
3

DEFINITION

Face Recognition (FR) can be described as classifying a

face either known or unknown, after comparing it with
known individuals stored in a database

Facial Expression Recognition (FER) system is a

computer application for automatically identifying or
verifying peoples emotions reflected on their faces from
a digital image or a video frame from a video source by
comparing it with database.

Facial recognition utilizes distinctive features of the

face such as distinct micro elements like: Mouth,
Nose , Eye, Cheekbones, Chin, Lips, Forehead, Ears

The distance between the eyes, the length of the nose,

and the angle of the jaw give rise to the type of
expression. Below are the 7 Facial Expressions types
Angry

Disgust

Fear

Happy

Neutral

Sad

Surprise

Human computer interaction

Automated

Video

access control

surveillance
6

 Games

Security
Patient

condition monitoring

FER involve the following steps:

Face detection
Facial expression data extraction
Facial expression classification

The following algorithms can be used in a Holisticbased approach to extract the facial expression
features:
Principal Component Analysis PCA
Linear Discriminant Analysis LDA
Local Binary Patterns LBP
Discrete Wavelet Transform DWT
Gabor Wavelet Transform GWT
Discrete Cosine Transform DCT

The aim of the PCA is to reduce the

dimensionality of the raw data (features)
while retaining as much as possible of the
variation present in the dataset.
Speeds up the computational time.

PCA

10

The database
a1

a
2

a
2
N

b1

b
2

b
2
N

a1 b1 L h1

r 1 a2 b2 L h2
m
,
M
M M M

a
2 b 2 L h 2
N
N
N

c1

c
2

c
2
N

d1

d
2

d
N 2

where M 213

11

Then subtract it from the training

faces
a m
c m
d
b m

r b2 m2
r a2 m2
r
, c
am
, bm
m
M
M

M
M

aN 2 mN 2
bN 2 m N 2

m1

r d 2 m
c2 m2
2

, dm
,

M
M
M
M

c N 2 mN 2
d N 2 mN 2

e1 m1

r e2 m2
em
,
M
M

eN 2 mN 2

g1 m 1
h1

r h2
g 2 m2
, hm
M
M
M

g N 2 m N 2
hN 2

f1 m1

r f 2 m2
r
, g
fm
m
M

f N 2 mN 2

m1

m2
M

m 2
N

12

Now we build the matrix which is N2 by M

The covariance matrix which is N2 by N2

r r r r r r r r
A am bm cm d m em f m g m hm

Cov AA

Find eigenvalues of the covariance matrix

The matrix is very large
The computational effort is very big
We are interested in at most M eigenvalues
We can reduce the dimension of the matrix
13

Compute another matrix which is M by M

A
A
Find the M eigenvalues and eigenvectors
Eigenvectors of Cov and L are equivalent

Build matrix V from the eigenvectors of L

Eigenvectors of Cov are linear combination of image space
with the eigenvectors of L

U AV r

V is Matrix of
Eigenvectors

r
r r r r r r
A am bm cm d m em f m g m hm

Eigenvectors represent the variation in the faces

14

A: collection of the
training faces

U: Face Space /
Eigen Space

Compute for each face its projection onto the face space

r
1 U am , 2 U
r
5 U em , 6 U

r
r
bm , 3 U cm , 4 U
r
r
f m , 7 U g m , 8 U

r
dm ,
r
hm

15

To recognize a Facial Expression

r1

r2

M

rN 2

Subtract the average face from it

r1 m1

m
r 2
2
rm
M M

rN 2 mN 2
16

Compute its projection onto the face space U

r
U rm

17

Different illumination

18

Different head pose

Different alignment
Different facial expression

19

The LBP operator was originally designed for texture

description. The operator assigns a label to every pixel of an
image by thresholding the 3x3-neighborhood of each pixel
with the center pixel value and considering the result as a
binary number.

233

224

150

200

173

185

120

128

20

1
T h re s h o ld

1
0

0
1

B in a ry : 1 0 0 0 1 0 1 1
D e c im a l: 1 3 9

20

21

Uniform Pattern: An LBP is called uniform if the binary pattern contains at most
two bitwise transitions from 0 to 1 or vice versa when the bit pattern is
considered circular
Example
The patterns 00000000 (0 transitions)
01110000 (2 transitions)
are uniform
11001111 (2 transitions)

The patterns 11001001 (4 transitions) and 01010011 (6 transitions) are not uniform.
Advantages of Uniform LBP
P
Save memory: With a non uniform pattern there is
Possible combinations while for uniform LBP there are patterns
P( Pof 1) 2

Uniform LBP detects only the important local textures like spots,
edges and corners

22

Divide the examined face image to cells

For each pixel in a cell, compare the pixel to each of
its neighbors. Follow the pixels along a circle, i.e.
clockwise or counter-clockwise.
Where the center pixel's value is greater than the
neighbor, write "1". Otherwise, write "0". This gives an
8-digit binary number (which is converted to decimal).
Compute the histogram, over the cell, of the
frequency of each "number occurring.
Optionally normalize the histogram.
Concatenate normalized histograms of all cells. This
gives the feature vector for the face image.

23

A GW filter is an essential tool used to extract

local features which can be applied on images to
extract features aligned at particular angles
(orientations).
The GWs filter captures significant visual features
such as spatial localization, orientation selectivity,
frequency selectivity, and phase relationship
The GWs kernel can be defined by the following
equation:

1
x, y, ,
e
2
2

X ' Y ' 2
(
)
2
i X '
2

24

where (x,y) denote the pixel position in

the spatial domain
,
is the central
frequency of a sinusoidal plane wave,
is the orientation of the Gabor filter and
is the standard deviation along x and
y directions. The parameters and can
be defined by the following equations:

X ' X cos sin , ' X sin cos

25

26

Having an input image I(x,y) of size MxN and a

Gabor wavelets kernel
u ,v ( x,of
y , , )
The Gabor feature representation is obtained by
convolving the input image with 40 Gabor wavelet
kernels given by
u , v ( x, y ) I ( X , Y ) u , v ( x, y , , )
Concatenate the magnitude of the convolved output
images of all the 40 feature vectors for each input
face image
Optionally before concatenation each image output
is down-sample by a factor of 16 or 32 and
normalized to zero mean and unit variance.
Apply Any dimensionality Reduction Algorithm to
reduce the size of the feature vector.
27

Gabor Wavelet Transform posses many properties

which make them attractive for many applications.
Directional selectivity
Invariance to shifts and rotations
Insensitive and robust to facial expression changes
Insensitive to illumination variations

Despite many advantages of Gabor wavelet based

algorithms in face recognition, it has major
disadvantages.
High computational complexity
High memory capacity requirement
Feature vectors dimensions are extremely large

28

Compute the Euclidian distance in the

face space between the test face and all
faces in the Training data
i
2
i

for i 1.. M

The expression with the minimum

distance from Test face to the Training
will be matched as the best expression
of the Test face.
29

JAFFE facial expression database was used to conduct our

experiments.
It contains 213 images of 10 different females each with 7
expressions posed by 3 or 4 examples of each of the seven
facial expressions under different illumination and head
position.
The images are of the size 256x256
Each original image has been aligned by normalizing it.
A total of 137 images (64%) were used as training data,
while the remaining 76 images(36%) as testing data
The K-nearest neighbour, Euclidean distance (L2) was

31

FRR(%) Comparisons For Different FER

Technique Using JAFFE

Experiment

PCA [1]

PCA+LDA [1] ASM & HMM [2] LBP [3]

SVM [4]

PCA NMF LNMF[5]

Recognition

80.00

95.11

88.79

85.57

94.5

63.25 65.50 64.50

128x96

230X250

64X64

44X32

Rate (%)
Face Dimension 128x96

40X30

32

Gabor wavelets were used as a pre-processing stage followed by

dimensionality reducing using PCA/LBP for facial expression
recognition in this paper.
Experimental evaluations the proposed approach were conducted on
JAFFE database.
The results obtained showed that pre-processing with Gabor
wavelets improves the performance of directly applying both PCA
and LBP.
Also the variation in illumination, hair and head position affect the
facial recognition rate.
Facial expression recognition proposed in this paper has an
improved performance when compared with the previous works
using different algorithms using the same JAFFE database as seen in
tables.

33

[1] H. Deng, L. Jin And L. Zhen, A New Facial Expression Recognition

Method Based On Local Gabor Filter Bank And PCA Plus LDA,
International Journal Of Information Technology Vol. 11 No. 11 2005, pp. 93
[2] W. Zhao And J. Zhang, Using ASM-Optical Flow Method And Hmm In
Facial Expression Recognition, IERI International Conference On Affective
Computing And Intelligent Interaction, Lecture Notes In Information
Technology, Vol.10, 2012 Pp. 268.
[3] S. Liao, W. Fan And D. Yeung, Facial Expression Recognition Using
Advanced Local Binary Patterns, Tsallis Entropies And Global Appearance
Features, IEEE, 2006 pp. 668.
[4] A. Bouzerdoum, S.L. Phung And P. Li, Feature Selection For Facial
Expression Recognition, IEEE, 2 nd European Workshop On Visual
Information Processing USA, 2010 pp. 39
[5] I. Buciu And I. Pitas, Application Of Non-Negative And Local Non
Negative Matrix Factorization To Facial Expression Recognition, IEEE
Proceedings Of The 17th International Conference On Pattern
Recognition , 2004 1051-4651.

34

35