A Novel Hybrid Bioinspired Algorithm for Facial
Emotion Recognition Using CSO-GA-PSO-SVM
T V, Vivek
Department of Information Technology
National Institute of Technology Karnataka
Surathkal, Mangalore, INDIA
vivekam101@gmail.com
Abstract — Human-Computer Interaction gets more natural
when the machine can detect human emotions faster and
accurate. A lot of research is being carried out in the field of
affective computing in order to improve the accuracy with speed.
Bio-inspired algorithms for feature extraction and classification
stages, has improved accuracy and speed further. In this paper,
we propose a novel hybrid algorithm using CSO (Cat Swarm
Optimization) with PSO (Particle Swarm Optimization) and GA
(Genetic Algorithm) for emotion recognition (ER). This novel
bioinspired algorithm in conjunction with the support vector
machine (SVM) will find an optimal feature set from a bigger set.
Results from CK+ (Cohn Kanade) [1] dataset demonstrate that
our proposed method using CSO-GA-PSOSVM outperforms
Emotion Recognition System with CSOSVM by 10.5% in accuracy.
This paper also proposes a new E-Learning [2] system to
demonstrate its effectiveness and efficiency in real-time scenario.
The proposed algorithm is applied over the facial characteristics
captured from students in teaching-learning environment. The
optimized feature vector obtained is passed to the SVM classifier
for classification. Experimental results yield 99% classification
accuracy in a person dependent mode with six basic emotions
namely Happy, Sad, Anger, Disgust, Surprise and Neutral.
Keywords- Emotion Recognition; Bioinspired Algorithms; Cat
Swarm Optimization; Genetic Algorithm; Particle Swarm
Optimization; Support Vector Machines; Hybrid Bioinspired
Algorithms; E-Learning.
I. INTRODUCTION
Emotion recognition from different media faces severe
performance degradation and instability due to its huge
variation in training database and that in real time
application. On training stage, we have mainly two steps:
feature extraction and classification. The features extracted
from multimedia are too big that it is adversely affecting the
classification stage. Accuracy can be improved by using
better features for classification. We aimed at solving this
NP-Hard feature subset problem in emotion recognition
using the effective use of bio-inspired heuristic algorithms.
The main challenge here is to choose the best heuristic
algorithm in conjunction with classifier for emotion
recognition. Bio-inspired approaches are of either iterative
Guddeti, Ram Mohana Reddy
Department of Information Technology
National Institute of Technology Karnataka
Surathkal, Mangalore, INDIA
profgrmreddy@nitk.ac.in
improvement of a population of solutions (as in Evolutionary
algorithms, Swarm Intelligence based algorithms) or a single
solution (Tabu Search) and mostly employs randomization
and local search to solve a given optimization problem. Size
and relevance of features in the feature vector play a vital
role in emotion recognition system. Time complexity is the
other factor that is vital for emotion recognition. Emotions
vary frequently and a perfect interaction is not possible
without timely classification.
The core idea behind our proposed method is to make use
of swarm intelligence algorithm in combination with
machine learning algorithms in order to reduce the execution
time and improving the accuracy of emotion recognition
system. Our research is on finding the appropriate hybrid
swarm intelligence algorithm devised from effective
algorithms such as PSO [3], CSO [4] and GA [5]. Further to
test and validate the efficiency of the system, we propose a
real time E-Learning system with a feedback mechanism.
Following are the key contributions of our proposed work:
 To the best of our knowledge, this is the first paper
which proposes a novel hybrid bio-inspired algorithm in
conjunction with SVM [6] for emotion recognition.
 To the best of our knowledge, this is the first approach
where a new E-Learning system is devised with hybrid
bioinspired algorithm and tested the efficiency in real
time for six basic emotions.
The rest of this paper is organized as follows: Section 2 deals
with the Related Work; Section 3 focuses on the Proposed
Methodology; Section 4 discusses Experimental Results and
Discussion; Finally the Concluding remarks with future
directions are given in Section 5.
II. RELATED WORK
Several studies are being carried out in the field of
evolutionary computing to improvise the existing algorithms
according to a vast variety of problems. Cat Swarm
Optimization has been studied as important one among them.
Some of recent existing works are summarized in Table I.
TABLE I. SUMMARY OF EXISTING HYBRID BIO-INSPIRED SYSTEMS design optimization and employ local surrogate models that
approximate the outputs of a computationally expensive
Work Advantages Limitations Euler solver. Another important contribution in this area is
Vibration mutation cat given by Israa Hadi et al. [13]. They introduced a new
Yan Zhang et swarm targeting the Convergence is not algorithm based on Hybrid Cat Swarm Optimization
al. [7] diversity in solution considered (HCSO) to reduce the number of search locations in Block
space Matching (BM) process. The conducted simulations indicate
Yuanmei Wen Convergence and the proposed method gives better result than other BM
Parallel CSO approach
and Yanyu
augmented with SVM
Divergence are not algorithms in terms of accuracy and computation time. Table
Chen [8] considered I summarizes the merits and demerits of existing works.
Average-Inertia
Maysam Weighted CSO (AICSO)
Divergence is not III. PROPOSED METHODOLOGY
Orouskhani et concentrating on
considered
al. [9] convergence in trace
mode A. Feature Extraction
We used Constrained Local Model (CLM) tracker as
Targets problems are of
Pei-wei Tsai et Parallel cat swarm
small population and facial recognizer. It is provided by Saragih et al. [14] where
al. [10] optimization (EPCSO) features are extracted based on the locality and shape
iterations
constraints. It is carried out in two steps i.e. Model building
and search process. Model building has two steps called
Israa Hadi et Hybrid CSO and Parallel
Problem dependent shape and patch model building. Shape model is created
al. [13]. approach using Principal Component Analysis (PCA) and gives the
mean shape and shape constraint. Mean shape is used for
initialization of new shape. CLM patch model creation is
very similar to Active Appearance Model with difference of
Yan Zhang et al. [7] came up with a variation in CSO using a set of patches of different features instead of using
called vibration mutation cat swarm, or VMCSO which triangular patches. A linear SVM is used to train the patch
targets on increasing the diversity in global search. They model. It is trained with MUCT database [15] which is
compared results using benchmark functions and showed a having over 3700 faces. We used Viola-Jones face detection
good improvement in accuracy. Yuanmei Wen and Yanyu algorithm [16] which is based on Haar like features, integral
Chen [8] used support vector machine (SVM) model with image, adaboost algorithm and cascade classifier. In facial
modified parallel cat swarm optimization (MPCSO) to emotion detection module, we capture images from live
forecast next-day cooling load in district cooling system video and then we identify 66 feature points in x, y
(DCS). Eigen value is extracted from data and Principal coordinates.
Component Analysis (PCA) algorithm is used to reduce
complexity in data sequence. Maysam Orouskhani et al. [9] B. Hybrid Bio-Inspired-Machine Learning Algorithm
proposed a new algorithm of CSO namely, Average-Inertia
Weighted CSO (AICSO). They introduced a new parameter
to the position update equation as an inertia weight and used
a new form of velocity update equation in the tracing mode
of algorithm. They concentrated on the convergence rather
than divergence part as done by Yan Zhang et al. [7].
In another approach, Pei-wei Tsai et al. [10] investigated a
parallel structure of cat swarm optimization. In the
experiments, comparison is done with Particle Swarm
Optimization (PSO). Parallel approach CSO converges fast
on small dataset and results were good. Further, Pei-wei Tsai
et al. [11] introduced an enhanced form of parallel cat swarm
optimization (EPCSO) method for solving numerical
optimization problems. This method is devised to solve
numerical optimization problems under the conditions of a
small population size and a few iteration numbers. The
Taguchi method is widely used in the industry for optimizing
the product and the process conditions. By adopting the
Taguchi method into the tracing mode process of the PCSO
method, they improved the accuracy and computation time.
Further, Y. S. Ong et al. [12] presented an evolutionary
algorithm hybridized with a gradient-based optimization Figure 1: Framework of proposed ER System
technique in the spirit of Lamarckian learning for efficient
 As shown in Figure 1, visual features are extracted and
processed with the help of our proposed hybrid bioinspired
system. In the training phase, we used 66 feature points
from each face of CK+ dataset; our proposed algorithm in
conjunction with SVM Classifier as fitness function
identified 15 relevant features giving the best accuracy. The
modified CSO algorithm with divergence by GA and
convergence by PSO-GA combination is described below.
The relevant features are used to classify to six basic
emotions using proposed hybrid algorithm and the details
are shown in Algorithm 1.
Algorithm 1: Hybrid Algorithm using CSO-GA-PSO-SVM
Input: Training feature set [each feature vector contains 66
facial features]
Output: Feature vector indices [It will list out the indices of
feature vector which provide best accuracy]
1: Randomly initialize cat’s position and speed
2: For each cats until required accuracy is obtained or
termination requirement satisfied
3: Check whether to choose cat’s current characteristic
set
4: Derive characteristic subsets for cats
5: Compute SVM fitness value for subsets found in
Step 4
6: Execute Modified CSO algorithm
7: end
8: Optimal characteristic subset
Proposed hybrid algorithm is a modification of most
recent swarm intelligence algorithm: CSO algorithm which
is developed based on the common behavior of cats. In this
proposed version, the location of each particle is represented
as vector xi= (xi1, xi2, xi3… xin) taking each bit xij (with j
in {1,N} binary values 0 or1. For our problem this xij
represents the feature and whole vector is the feature set.
Cats are very observant and it spends most of its time
observing its surroundings rather than running behind things
which leads to wastage of energy. In algorithm we represent
this behavior by Seeking mode and Tracing mode.
1) Seeking Mode Operations: Resting and Observing
Seek mode represents the cats behavior in resting time. This
mode decides the best position to move in next iteration. We
are using GA operation ‘Mutation’ here for good
divergence. This mode has four main parameters as below:
Seek Memory Pool (SMP), mutation probability, counts of
dimensions to change (CDC) and Self position
consideration (SPC). The modified process of seeking mode
is described by the following steps:
Step1: Make j copies of the present position of catk, where
j=SMP. If the value of SPC is true, let j = (SMP-1), then
retain the present position as one of the candidates.
Step2: For each copy, according to CDC, randomly apply
mutation on its feature set having 66 bits. Each bit
represents feature is present or not.
Step3: Calculate the fitness values (FS) of all candidate
points.
Step4: If all FS are not exactly equal, calculate the selecting
probability of each candidate point by (2); otherwise set all
the selecting probability of each candidate point is 1.
Step5: Randomly pick the point to move to from the
candidate points, and replace the position of cat k.
𝑆𝑆𝐸𝑖−𝑆𝑆𝐸𝑚𝑎𝑥
𝑃𝑖 = 𝑆𝑆𝐸𝑚𝑎𝑥−𝑆𝑆𝐸𝑚𝑖𝑛
If the goal of the fitness function is to find the minimum
solution, FSb = FSmax, otherwise FSb = FSmin.
2) Tracing Mode : Running after a Target
Tracing mode is the other mode in CSO which models the
moving behavior of cats. We are including PSO+GA
combination for a better convergence. PSO operators
namely the subtraction and addition operators are used
along with GA operators like mutation, crossover and
selection. Particle movement is as per the following
equation (1):
𝑋 𝑘 (𝑡 + 1) = (𝑃𝑘 (𝑡) − 𝑋 𝑘 (𝑡)) + [𝑋 𝑡 (𝑡)]′ (1)
The subtraction operator represents the crossover operation
between two individuals. [𝑋 𝑡 (𝑡)]′ represent random
mutation operation on 𝑋 𝑡 (𝑡) , and the addition represents
selection operation from the individuals. Here 𝑃𝑘 (𝑡)
represent position of global best particle. We have tried with
adaptive convergence rates based on SVM accuracy. We
applied 90% convergence rate till cross validation accuracy
reach 70% and then reduced to 85% to increase the
divergence. The whole process is summarized in the
following figure 2.
C. Evaluation Function
Here each particle position represents a feature vector
and the evaluation of each particle is carried out by means
of the SVM classifier to assess the quality of the represented
feature set. The fitness of a particle xi is calculated by
applying a 10-fold cross validation (10FCV) method to
calculate the rate of correct classification accuracy of SVM
trained with this feature subset. In 10FCV, the data set is
divided into 10 subsets. Each time one of the 10 subsets is
used as the test set and the other 9 subsets are put together
forming training subset. Then the average error across all 10
trials is computed. The complete fitness function is IV. EXPERIMENTAL RESULTS AND DISCUSSION
described in the Equation 2.
A. Experimental Setup
100 Proposed algorithm is coded in Python language
𝑓𝑖𝑡𝑛𝑒𝑠𝑠(𝑥) = 𝛽. ( ) + 𝛾. #𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 (2)
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 using threads for parallel implementation. The multilabel
SVM classifier is implemented using the LibSVM toolkit
Where 𝛽 and 𝛾 are weight values set to 0.75 and 0.25 [17]. RBF kernel function is used in SVM as it is giving
respectively to keep 10FCV accuracy value takes priority better result as compared to linear and polynomial kernels.
over the subset size. The objective here consists of All the experiments are carried our using a PC with Linux
maximizing the accuracy and minimizing the subset size. O.S ( Ubuntu 14.04 kernel 3.13.0-32.57) 4TH GEN INTEL
PENTIUM QUAD CORE with 8 GB RAM. For comparison
with individual bioinspired algorithms, we ran CSO, PSO
and GA individually and in combination 10 times over each
dataset to reach a statistically meaningful conclusion.

Real time experiment is done with 20 postgraduate students

from IT Department, NITK Surathkal, Mangalore, India in
an E-Learning environment. It acts like a feedback for E-
learning providers to know how the student is performing in
a particular subject of teaching-learning process. Students
are provided a web interface with webcam and microphone.
Audio and facial signals are captured while student is in the
class room. These signals are processed by our proposed
system and the corresponding emotion is displayed based on
the duration. In the end, a questionnaire is provided to each
of 20 students to assess the performance of the student.
Experimental results are promising and motivated us to
carry out further research. Figure 3 shows the E-Learning
process using the proposed hybrid method.

Figure 2: Modified CSO Flow

D. Dataset
We have used instances from two well known
datasets for facial emotion recognition.

 The Cohn-Kanade AU-Coded Expression Database

Version 2, referred to as CK+, includes both posed and non-
posed (spontaneous) expressions and additional types of
metadata. For posed expressions, the number of sequences
is increased from the initial release by 22% and the number
of subjects by 27%. For action unit and expression
recognition, support vector machine (SVM) classifier with
leave-one-out subject cross-validation was used.

Figure 3: E-learning Process using Proposed Method

 are shown in Figure 5. CSO and combination with CSO
takes more computational cost since the seeking mode of
B. Parameter Settings
cats create duplicates to look for neighborhood positions
The parameters used in our proposed hybrid algorithm are and evaluation of these duplicates is computationally
given in Table II. These parameters were selected after expensive. In case of PSO and GA, this mode is not
several test evaluations of each dataset several times until it available. Further our proposed hybrid method is little more
reaches quality solutions and computational effort. computationally expensive than other CSO combinations as
TABLE II. PARAMETERS FOR DESGINING PROPOSED HYBRID ALGORITHM divergence is created using mutation process and
convergence using combination of PSO-GA process
Parameter Value or Range involving crossover, mutation and select operations. Further
with the selected optimal features we classified the facial
SMP 5
characteristics to six basic emotions like Anger, Happy,
Mutation probability 10% Surprise, Sad, Disgust and Neutral. Confusion matrix
CDC 80% generated from this classification is shown in Table III.
MR 2%
Cross Over Probability 90%

SMP: Seeking Memory Pool

CDC: Counts of dimensions to change
MR: Mixed Ratio to decide seeking mode and tracing mode

C. Results and Discussion

Figure 4 shows the comparison of our proposed method
against other bioinspired algorithms and it shows an
improvement with clear margin. We ran each algorithm for
100 iterations for comparison purpose. Further to find the
best possible result from each algorithm we tried each
algorithm 10 times using same dataset.

Figure 5: Time taken by each bioinspired algorithms for

100 iterations

TABLE III. CONFUSION MATRIX OBTAINED USING CSO-GA-PSOSVM

Emotion Ang Hap Surpr Sad Disgu Neut

Ang 100 00.0 00.0 00.0 00.0 00.0

Hap 00.0 100.0 00.0 00.0 00.0 00.0

Surpr 00.0 5.27 92 00.0 1.42 1.31

Sad 00.0 00.0 00.0 90 5.5 4.5

Figure 4: Cross validation accuracy obtained using various Disgu 00.0 00.0 00.0 5.3 93.3 1.4
combinations of bioinspired algorithms
Neut 00.0 00.0 00.0 3.93 8.07 88
Similarly, we analyzed the computational cost of each
algorithm with the same setup as mentioned above. Results Total 93.88
 V. CONCLUSION AND FUTURE WORK
In the proposed work, we developed a novel Hybrid
Bioinspired algorithm for Emotion recognition using CSO-
GA-PSO-SVM. Further, we tested this system with person
dependent and independent scenarios and results are
encouraging. We used CK+ dataset considering 75% of data
for training the system and remaining for testing. It gives an
average accuracy of 93.8% considering visual cues alone.
This is an improvement of 10.5% accuracy when compared
to ER system with CSOSVM alone. Further, we demonstrated
the performance of our proposed algorithm in a real time
scenario. Emotions in collaborative work places like office,
school, college, university are generally expressed in the
form of text, gesture and body movements. In future, we will
extend the work on hybrid bio-inspired system while
considering these multimodal features and thereby widening
the scope.
REFERENCES
[1] Lucey, Patrick, et al. "The Extended Cohn-Kanade Dataset (CK+): A
complete dataset for action unit and emotion-specified
expression." Computer Vision and Pattern Recognition Workshops
(CVPRW), 2010 IEEE Computer Society Conference on. IEEE,
2010.
[2] Zhu, Aiqin, and Qi Luo. "Study on speech emotion recognition
system in E-learning." Human-Computer Interaction. HCI Intelligent
Multimodal Interaction Environments, Springer Berlin Heidelberg,
2007. 544-552.
[3] Binitha, S., and S. Siva Sathya. "A survey of bio inspired
optimization algorithms." International Journal of Soft Computing
and Engineering 2.2 (2012): 137-151.
[4] Chu, Shu-Chuan, Pei-Wei Tsai, and Jeng-Shyang Pan. "Cat swarm
optimization." PRICAI 2006: Trends in Artificial Intelligence.
Springer Berlin Heidelberg, 2006. 854-858.
[5] Back, Thomas. "Evolutionary algorithms in theory and practice,
1996."
[6] Tong, Simon, and Daphne Koller. "Support vector machine active
learning with applications to text classification." The Journal of
Machine Learning Research2 (2002): 45-66.
[7] Y. Zhang and Y. Ma, “Cat Swarm Optimization with a Vibration
Mutation Strategy”, International Journal of Machine Learning and
Computing, vol. 4, no. 6, (2014) December, pp. 510-514.
[8] Y. Zhang and Y. Ma, “Cat Swarm Optimization with a Vibration
Mutation Strategy”, International Journal of Machine Learning and
Computing, vol. 4, no. 6, (2014) December, pp. 510-514.
[9] Y. Wen and Y. Chen, “Modified Parallel Cat Swarm Optimization in
SVM Modeling for Short-term Cooling Load Forecasting”, Journal of
Software, vol. 9, no. 8, (2014) August, pp. 2093-2104.
[10] P. W. Tsai, J. S. Pan, S. M. Chen, B. Y. Liao and S. P. Hao, “Parallel
Cat Swarm Optimization”, In Proceedings of the 7th International
Conference on Machine Learning and Cybernetics, (2008), pp. 3328-
3333.
[11] P. w. tsai, J.-S. Pan, S.-M. Chen and B.-Y. Liao, “Enhanced parallel
cat swarm optimization based on the Taguchi method”, vol. 39, no. 7,
(2012) June 1, pp. 6309–6319.
[12] Y. S. Ong, K. Y. Lum and P. B. Nair, “Hybrid evolutionary algorithm
with Hermite radial basis function interpolants for computationally
expensive adjoint solvers”, Computational Optimization and
Applications, Springer US, vol. 39, no. 1, (2008) January, pp. 97-119.
[13] Hadi, Israa, and Mustafa Sabah. "An Enhanced Video Tracking
Technique Based on Nature Inspired Algorithm." International
Journal of Digital Content Technology and its Applications
(JDCTA) 8.3 (2014): 32-42.
[14] Saragih, Jason M., Simon Lucey, and Jeffrey F. Cohn. "Face
alignment through subspace constrained mean-shifts." Computer
Vision, 2009 IEEE 12th International Conference on. IEEE, 2009.
[15] Milborrow, Stephen, John Morkel, and Fred Nicolls. "The MUCT
landmarked face database." Pattern Recognition Association of South
Africa 201.0 (2010).
[16] Viola, Paul, and Michael J. Jones. "Robust real-time face detection."
International journal of computer vision 57.2 (2004): 137-154.
[17] Chang, Chih-Chung, and Chih-Jen Lin. "LIBSVM: a library for
support vector machines." ACM Transactions on Intelligent Systems
and Technology (TIST)2.3 (2011): 27.