Sie sind auf Seite 1von 4

Seminar Paper: Emotion Recognition from Physiological signals

using Bio-sensors
Revathi priya Muthusamy,
University of Fribourg, Switzerland,
Revathipriya.muthusamy@unifr.ch

Abstract:
Electrocardiogram (ECG). An electrocardiogram
This paper presents an introduction to various bio-signals used in (ECG) is a test which measures the electrical activity of the heart.
Emotion recognition such as Electromyography (EMG) ECG is used to calculate the rate and regularity of heartbeats. Can
electrocardiogram (ECG), electrodermal response (EDR), Skin be recorded from above the chest or limbs. ECG measurements
temperature, and Blood volume pulse (BVP) and respiration from the limbs is less inconvenient but more vulnerable to
changes. This paper also discusses about various kinds of bio- artifacts. [1]
sensors used to detect the Physiological signals. In the remaining
of the paper we describe the Multimodal emotion elicitation and We can measure heart rate (HR) and inter-beat intervals (IBI) and
Acquisition of emotion-specific physiological signal database. It thus we can determine the heart rate variability (HRV). A low
also deals with feature extraction technique based on the Hilbert- HRV can indicate a state of relaxation, whereas an increased HRV
Huang Transform (HHT) method and classification techniques can indicate frustration or state of mental stress.
such as Support Vector Machine. the highest recognition accuracy
is obtained by using the fission based HHT features method with
28 features, 76% of test cases are correctly classified.
Keywords:
Bio-sensors, Multimodal emotion elicitation, Hilbert-Huang
Transform, Support Vector Machine
Fig 2: (a) heart with Nervous system intervention (b) Heart rate
Variability HRV
1. Introduction to Biomedical Signals Electromyography (EMG) is a technique used to record
Most physiological processes produce signals of several electrical potential created by muscle membranes when there is an
types: electrical or neurological triggering. For example a high muscle
tension is created when there is stress or frustration. These signals
 Biochemical -in the form of hormones.
can be measured by placing the bio sensors over face or hands. [3]
 Electrical - in the form of current, and
 Physical - in the form of pressure and temperature. [1]
For example when we get afraid of something our heart
races, our muscles tense, our mouth becomes dry. These changes
are controlled by automatic nervous system, which manages heart
muscles, smooth muscle and various glands in our body. [7]
These bodily reactions can be measured and monitored.
Fig 3. Anatomy of hand (left), typical EMG signal (right)
These signals are called as bio-signals. We can monitor the body
reactions from outside by use of Bio sensors and classify the bio Electrodermal response (EDR)-also called as skin conductivity
signals corresponding for each emotion. (SC) measures the conductivity of the skin. For example EDR
increases if the skin is sweaty. This signal is a sensitive indicator
of stress and for other stimuli. It is also used to classify between
conflicts-no conflict situations or classify between anger and fear.
The main disadvantage of this signal is that it is influenced by
external factors like outside temperature. Hence reference
measurements and calibrations are recommended. [3]

Fig 1: The Autonomic nervous system and the organs controlled Figure 4: Anatomy of skin (left), typical EDR signal (right)
by it.
Skin temperature can be measured by determining the
We study the following Bio-signals for Emotion recognition: temperature on the surface of the skin. For example under strain,
muscles are tensed, the blood vessels will be contracted and hence
the temperature will decrease. Similar to the Electro dermal

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition


response the skin temperature also depends on external factors. adopted for many psychophysiological studies involving emotion
Furthermore it is a relatively slow indicator of changes in induction.
emotional state.
Blood volume pulse (BVP) is a measured by determining the
amount of blood currently running though the vessels, e.g. in the
finger of a test subject. A photoplethysmograph (PPG) is used to
measure BVP, it consists of a light source and photo sensor. It is
attached to the skin and the amount of reflected light, which
depends on the amount of blood, is measured. [1]
Respiration sensors measure how deep and fast a person is
breathing. This is measured by applying a rubber band around the
chest. Fast and deep breathing can indicate excitement such as
anger or fear but sometimes also joy. Rapid shallow breathing can
indicate tense anticipation including panic, fear or concentration.
Slow and deep breathing indicates a relaxed resting state while
slow and shallow breathing can indicate states of withdrawal, Fig.6 Illustration of example of emotion induction protocols'. Its"
passive like depression or calm happiness.
purpose is to induce status of 'sadness'[3]
We study the protocol utilizing a multimodal (audio, visual and
2. The Bio-Sensors cognitive) approach to evoke specific targeted emotional statuses,
and it was developed in collaboration with specialists from the
A Bio Sensor is an analytical device used to convert a
field of cognitive and physiological psychology (YANG et al.,
biological response into electrical signal.
2000). The emotion induction protocols are summarized in Table.
A preliminary test of the protocols was performed for 80 subjects
aged from seven to eight years. [3]

Figure 5. Main components of Biosensor

Figure 5. Examples of Bio-sensors.


Figure [5.1] shows the sensors used for measuring EMG. We
measure the muscle activity of the masseter muscle, [7] the
muscle movement is described to be reliable in that part of body.
Figure [5.2] shows a standard ECG sensor. In Figure [5.3] the
respiration sensor is shown applied to the chest while in Figure
[5.4] the skin conductivity, BVP and temperature sensor can be
seen applied to the fingers of the left hand (the non-dominant
hand should be used for measuring).[7]

3. Data Collection Experiments


Acquisition of emotion-specific physiological signal database: 4. Feature Extraction
The physiological signals were acquired using the MP100 system.
4.1 Preprocessing, waveform detection and feature extraction:
The sampling rate was fixed at 256 samples s 1 for all the
channels. Appropriate amplification and bandpass filtering were The first necessary step was the detection of the characteristic
performed. The subjects were requested to be as relaxed as waveform and extraction of useful information-bearing features
possible during this period. Subsequently, emotional stimulus was for pattern classification. As shown in Fig. 7, the baseline values
applied and debriefing and recovery followed. [3] of each component of the feature vectors were subtracted before
they were given to the classifier. Here, the baseline values mean
Emotion induction protocol: We study that visual stimulation
the components of feature vectors extracted from 50 s segments of
using still images was not sufficient for effective emotion
signals that were acquired without stimulus. [3]
induction, and we did not study the international affective picture
system (IAPS) developed by LANG et al. (1988), despite its being

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition


Fig 10. Feature extraction from EMG signal [4]
4.1.3 Electrodermal activity:

Fig. 7 Overall structure of emotion recognition system [3] [4]


4.1.1. From ECG signals - RR interval and heart rate
variability:

Fig 11 (a) Typical waveform of EDA under emotional


stimulation.
(b) Output signal from detection module in (a).
(c) Block diagram of SCR detection module. Sampling rate of
traces in (a) and (b) is 21.3 samples s z.[3]
Electrodermal activity (EDA) was obtained by measurement of
the voltage between two electrodes across which a low-level
current was applied. Fig. 11 a shows a typical waveform of EDA
under emotional stimulation (after subtraction of mean EDA
level). Important features of EDA include the DC level and the
Fig.8 Block diagram of information extraction module from ECG distinctive short waveforms that are indicated by arrows in Fig.
Fig.8 above shows the block diagram of the feature extraction 11. [3]
module for the heart rate. Heart rate variability (HRV) contains 4.1.4 Skin temperature variation:
abundant information on the status of the autonomic nervous
system and can be derived from ECG or PPG. [3] Frequency- No special signal processing was necessary for the feature
domain features of HRV have also been considered to be extraction from the skin temperature (SKT). Although frequency-
significant for the exploration of the autonomic nervous system in domain analysis of the time-varying SKT has been reported
many previous studies for cardiac function assessment and psycho (SHUSTE~MAN and BARNEA, 1995), here the mean and
physiological investigation (DRUMMOND and QUAH, 2001; maximum values within 50 s intervals were used as the features of
MCCRATY et al., 1995).[3] SKT.[3]

4.1.2 EMG Recordings and Data Reduction


5. Classification Experiments
After extracting the features as described in the previous section,
statistical classifier is used for learning the corresponding emotion
for a set of features with which it is presented. There are different
options for building such a classifier. Hilbert Haung transform,
wavelet transform, neural network classifier are some of the
widely used Classification methods.
5.1 Hilbert Haung transform
Huang et al. have proposed a new tool for the analysis of
Fig 9 supercilii and the zygomaticus muscles for EMG signal nonlinear and non-stationary data called as the Hilbert-Huang
acquisition. [5] transform method (HHT)[2]. Fission process- Empirical Mode
Decomposition (EMD) decomposes a signal into basis functions
Facial EMG was measured by bipolar attached electrodes placed
which are finite called the intrinsic mode functions. Fussion
over the corrugator supercilii and the zygomaticus major on the
process- After this fission process, the intrinsic mode functions
left side of the face, according to the guidelines given by Fridlund
(IMFs) of interest are combined in an ad hoc or automated
and Cacioppo (Figure 9). [5]
fashion in order to provide a greater knowledge. [2].

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition


video analysis, motion detection or emotion recognition from
speech signals to bring a real emotional dialog system to work.

References:
1. Rangaraj M. Rangayyan; Biomedical Signal Analysis – A
Case-Study Approach; IEEE Press 2002
2. C. Zong and M. Chetouani, "Hilbert-Huang transform based
Fig 12. HHT Fission and Fusion methods physiological signals analysis for emotion recognition", in proc.
Fission process- Each signal is decomposed into a finite of IEEE International Symposium on Signal Processing and
set of AM-FM mono components by using the Empirical Mode Information Technology, pp. 334-339, 2009.
Decomposition (EMD) which is the important part of the HHT. 3. K. H. Kim I S.W. Bang 2 S.R. Kim 2Emotion recognition
Fusion process- For the next classification stage, the information system using short-term monitoring of physiological signals, 1
components of interest are then combined to create feature vectors Department of Biomedical Engineering, College of Health
[2]. Science, Yonsei University, South Korea. 2 Human-computer
Interaction Laboratory, Samsung Advanced Institute of
Technology, South Korea
6. Results
4. Cheng-Ning Huang Chun-Han Chen Hung-Yuan Chung, The
Review of Applications and Measurements in Facial
Electromyography,Department of Electrical Engineering, National
Central University, Chung-Li, Tiwan, 320 ROC
5. Anton van Boxtel,Facial EMG as a Tool for Inferring Affective
States, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The
Netherlands, a.vanboxtel@uvt.nl, Tilburg University, Tilburg,
The Netherlands,
6. Marco Tamietto , Emotional Contagion for Unseen Bodily
Expressions: Evidence from Facial EMG, Cognitive and Affective
Neurosciences Laboratory, Department of Psychology, University
of Torino, Torino, Italy, M.Tamietto@uvt.nl
Table 2. SVM Classification rates of four emotions for four bio 7. Andreas Haag, Silke Goronzy, Peter Schaich, “Emotion
signals. (TD : Time Domain ; FD : Frequency Domain)[2] Recognition Using Bio-Sensors: First Steps Towards an
The features from all methods ( the state-of-the-art method, the Automatic System” Jason Williams,Sony Corporate Laboratories
fission process and the fusion process based HHT features Europe, Hedelfinger Str. 61,D-70327 Stuttgart, Germany {Haag,
method) for four bio signals are combines and classified using Goronzy, Schaich, Williams@Sony.de}
Support Vector Machine (SVM). The classification results are as 8. FATMA NASOZ, KAYE ALVAREZ, “Emotion Recognition
listed below from Physiological Signals for Presence Technologies”, In
International Journal of Cognition, Technology, and Work -
Special Issue on Presence, Vol.6(1), 2003.
9.Lang, P.J.,Bradley, M.M., & Cuthbert, B.N. (2001).
International affective picture system (IAPS):Instruction Manual
and Affective Ratings. Technical Report A-5. Gainesville, FL.
The Research in Psychophysiology, University of Florida.
Table 3. Classification results [2].
10. Center for the Study of Emotion and Attention [CSEA-NIMH]
(2001). The international affective picture system: Digitized
photographs. Gainesville, FL: The Center for Research in
7. Conclusion Psychphysiology, Univerity of Florida
In this paper we have surveyed research works using 11. Barreto A., Heimer M., and Garcia M., “Characterization of
Physiological sensors to recognize human emotions. In this Photoplehtysmograpic Blood Volume Pulse Waveforms for
article, we discus about characteristic waveform detection, feature Exercise Evalution,” Proceedings 14th Southern Biomedical
extraction and pattern classification stages. The HHT Hilbert- Engineering Conference, Shreveport, Louisiana, April, 1995, pp.
Huang transform method classified signals by 71 % for Baseline 220-223
features and 76% for fission method and 62% for fusion method.
[2]. Future Research scope: There are clearly more steps to take in 12. Chrisite, “Multivariate Discrimination of Emotion-specific
this research area. Research has to be done to improve the Autonomic Nervous System Activity”, Master Thesis in Science
accuracy of the network. This could be done either by adding of Psychology, Virginia
some meaningful features (the full feature set) that are computed 13. Healy J. and Picard R., “Digital processing of Affective
from the existing sensor signals or by using different bio-signals Signals”, ICASSP 98
such as EEG data. Also research has to be done to for
incorporation of different sources for emotion recognition such as

Submitted for Research Seminar on Emotion Recognition on 15.02.2012 http://diuf.unifr.ch/main/diva/teaching/seminars/emotion-recognition