Analysis of Spontaneous Emotional

Expressivity in Adults with Traumatic

Brain Injury
Samuel James
Edgemont High School

James 1

Acknowledgements
I would like to thank my teacher, Ms. Maria DeCandia, for all her help during the past three
years while I was in the Science Scholars program. I would also like to thank my mentor, Dr.
Preeti Raghavan, for providing me with the opportunity to work at the RUSK Rehabilitation Lab.
I am also grateful for all her time and the guidance she has given me during my research. I would
also like to thank Dr. Aluru Viswanath for his guidance during the course of the study, as well as,
the other researchers at the RUSK Lab. Finally, and most importantly, I would like to thank my
mother, Lovetta James, and my father, Joji James, for their extensive support.

James 2

Abstract
A growing number of research studies have shown that people with Traumatic Brain Injury (TBI)
have difficulties in identifying emotions. However, little research has been done to determine if
this impairment has an impact on the emotive response of subjects with TBI. Do individuals with
TBI have difficulty expressing their emotions? How different is their emotional expression
compared to individuals without TBI? With advancements in Automatic Facial Action Coding
(AFAC), the facial expressions of subjects with TBI can be analyzed quickly and thoroughly. In
this exploratory study we analyzed the facial expressions of subjects with TBI as they watched
disgust and amusement-eliciting film clips. Subjects were videotaped as they watched the clips.
The subjects also completed a Post-Film Questionnaire (PFQ), which self-identified the emotions
they experienced while watching the film clips. The software, Attention Tools FACET Module
(InMotion, MA), was used to analyze the facial expressions of the subjects. We then compared
the emotions that the subjects claimed to have experienced (PFQ) with the emotions expressed
on their faces (FACET). The results suggest that subjects with TBI are impaired in expressing
their emotions. Lack of emotional expressivity may contribute to dysfunctional inter-personal
relationships after TBI.

James 3

Table of Contents
List of Tables..1
Introduction.2
Methods...3
Results.9
Discussion..13
References..15

James 4

List of Tables

TABLE 1

Emotion-eliciting movie clips shown to participants

List of Figures
FIGURE 1

FACET evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 2

PFQ evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 3

FACET evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 4

PFQ evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 5

FACET evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 6

PFQ evidence for TBI Subject during Disgust and Amusement evoking video
clips

11
FIGURE 7

FACET evidence for Control Subject during Disgust and Amusement evoking
video clips
12

FIGURE 8

PFQ evidence for TBI Subject during Disgust and Amusement evoking video
clips

12
FIGURE 9

FACET evidence for Control Subject during Disgust and Amusement evoking
video clips
12
James 1

FIGURE 10

PFQ evidence for Control Subject during Disgust and Amusement evoking video
clips

12
FIGURE 11

FACET evidence for Control Subject during Disgust and Amusement evoking
video clips
12

FIGURE 12

PFQ evidence for Control Subject during Disgust and Amusement evoking video
clips

12

James 2

Introduction
Traumatic Brain Injury (TBI) is the result of sudden forceful movement of the brain against the
inner walls of the rigid skull. TBI can be caused by closed or open head injury from motor
vehicle accidents, falls, and sports-injuries. While sports-related head injuries are commonly
called concussions (Levine, 2010 & Anderson 2006), they are in effect mild traumatic brain
injuries that can cause cumulative brain dysfunction.

It is known that TBI can affect a persons social and emotional behavior (Brown, 2011). More
recent research has shown that TBI can cause deficits in facial emotion perception. When
presented with facial models that exhibited various emotions, TBI subjects had difficulty
identifying the particular emotion. They especially had trouble recognizing negative emotions
such as disgust, fear and sadness (Cauldon,2000;Croker et al., 2005). However, little is known
about whether a TBI also alters an individuals own emotive response. This is important because
an inability to express ones feelings effectively may lead to unresolved frustration, anger and
depression, which can have dire behavioral consequences, including suicide (Mckay et.al, 2009).
The purpose of this study was to determine whether individuals with TBI suffer from deficits in
spontaneous emotional expression when compared with healthy individuals.

James 3

Methods
Participants

The study included six subjects; three post-TBI subjects, who complained of emotional
dysfunction six months following their brain injury, and three healthy age-matched control
subjects. They were equally divided in gender. Exclusion criteria for subjects included comorbid
psychiatric disease (e.g. schizophrenia), active substance abuse, history of craniectomy, active
infection, seizure activity beyond one week post-TBI, confounding medical complications (e.g.
hypertension, diabetes, epilepsy or demyelinating disease, implanted neuromodulatory or
electronic device) and pregnancy. Also, only those subjects with the capacity to understand the
terms of the study and provide informed consent were enrolled.

Procedure

Subjects sat in a well lit room facing a computer monitor. They were informed that they were
going to watch a series of eight film clips that would be 2-5 minutes long. However, they were
not made aware of the content or the emotion evoking qualities of the clips. Before each clip, the
subjects watched a set of instructions for 10 seconds informing them to relax and watch a +
sign on the computer screen for the next minute. This technique was used to return subjects back
to baseline prior to each new film clip. The clips were shown in a specific order to balance out
the negative and positive emotions. This prevented the subjects from being overloaded with any
single emotion.
James 4

There were two sets of film clips; each set included clips that elicited the following emotions neutral, amusement, sadness, sexual amusement, fear and disgust. These clips were selected
from a database of film clips validated to elicit specific emotions (Schaefer, A., Nils, F. et al. ,
2010). Unlike the film clips used in previous studies, which only included facial expressions, the
clips we selected have a background setting which provides subjects with contextual cues. This is
relevant since a TBI subject may have difficulty identifying facial expressions, but they may still
be able to identify emotions from contextual cues (Calder et al., 2000). Furthermore, the fil clips
not only convey the spontaneity of emotions, but may also show head and facial movement
which are also markers of emotions (Mcdonald, 2005). Another benefit of videos over static
images is the addition of audio. Both individuals with TBI and without TBI have been shown to
identify emotions better when there is a conjunction of both video and audio (Calder et al.,
2000).

The subjects watched one film clip each that elicited neutral, amusement, sadness, sexual
amusement and fear. They were told, We will now be showing you a short film clip. It is
important to us that you watch the film clip carefully. Subjects then watched three clips
depicting disgust. They were instructed to watch the first disgust film clip just as they had the
previous clips. For the second disgust clip, subjects were told to suppress their emotional
response. Watch the film clip carefully. If you have any feelings as you watch the film, please
try your best not to let those feeling show. For the third disgust clip, subjects were told to
amplify their emotional response. Watch the film clip carefully. If you have any feelings as you
watch the film clip, try your best to let those feelings show. After watching each film clip, the
James 5

subjects were asked to complete a questionnaire to self-rate the intensity of their feelings on
scale of 0 (none) to 10 (extreme).

James 6

Table 1. Emotion-eliciting movie clips shown to participants

Video
Target
Movie title & description
emotion
Set
Neutral
Sticks accumulating on a black background (no audio).
Amusement Something about Mary man fights with a small dog.
Sadness
City of Angels A woman dying in a mans arms.
Sexual
When Harry Met Sally A woman loudly simulates sexual
amusement excitement in a crowded diner.
Set 1
Fear
The Shining A boy plays alone in an empty hallway
Trainspotting A man defecates then dives into a filthy toilet.
Amputation noncommercial recording of arm amputation
Disgust
Bat Taste A group of young men eat vomit.
Neutral
Sticks accumulating on a black background (no audio).
Amusement Benny and Joon A man clowns around in a diner.
Sadness
The Champ A young boy cries as his father lays dead.
Sexual
A Fish Called Wanda A woman and man are caught while in a
amusement comedic tryst
Set 2
Fear
Silence of the Lambs A chase scene followed by a dead corpse.
Vampires Kiss A man eats a cockroach
Black Swan A woman peels back a hangnail.
Disgust
Pink Flamingoes A woman eats dog feces.

Time
(min)
4.6
4.33
4.35
3.75
2.5
2.38
2.3
2.96
4.6
3.33
3.83
4.08
4.6
1.65
1.87
2.5

Note. Adapted from Uy, et. al. (2013)

James 7

IMOTION FACET Module

Motivated by advancements in facial action coding technology, this study is one of the first to
collect facial video data on TBI subjects during spontaneous emotional expression. Analysis of
the subjects facial expressions from the video was performed using The Facial Action Coding
System (FACS) (G. Littlewort et al., 2011). The Facial Action Coding System is a database
created by Paul Ekman in the 1970s, and was expanded in 2001. The original database was a
collection of Ekmans own emotional expressions formed by varying facial muscle activity.
Eckman used this data for his own study of emotion-related muscle response (Eckman &
Friesen, 1976). To score a face according to the FACS, the rater examines different Action Units
(AUs), as well as combinations of different AUs. Each AU corresponds to a different muscle
movement. For example, if someone were to raise their inner eyebrow they would be exhibiting
AU1. AUs can be broken down into varying emotions. For Example AUX1, AUX2, AUX4, may
be prominent when someone is expressing fear (Eckman, Friesen and Hager, 2002).

IMOTIONs FACET Module is an automated facial expression recognition software that can
analyze video in real time. Its predecessor, the Computer Expression Recognition Toolbox
(CERT) has already been used in research. The FACET module is able to detect faces from the
live feed and then code the faces using Paul Eckmans Facial Action Coding System (FACS). It
can identify and measure the intensity of 19 Action Units (AU) {AUs 1, 2, 4, 5, 6, 7, 9, 10, 12,
14, 15, 18, 20, 23, 24, 25, 26, and 28}. The FACET Module differentiates nine different
emotions, including the six basic emotions: Joy, Sadness, Fear, Anger, Disgust, and Surprise and
three other advanced emotions Contempt, Frustration, and Confusion (Eckman & Friesen, 1976).
James 8

Face and Feature Detection

FACET employs different boosting algorithms to identify both the face and facial features. The
facial features corresponding to the different AUs. The booting algorithms output the likelihood
that a feature is present at a pixel in 2D Cartesian space.

Facial Warping
In order to maximize facial detection performance the faces are cropped and warped to fit in a
96x96 frame. This process helps to normalize the face by adjusting for any pitch, yaw and size
difference among the faces. The cropped face then undergoes multiple Gabor Filters for edge
detection. The output is a single feature vector.

Action Unit Recognition

The feature vector is input into linear Support Vector Machines (SVP) so as to estimate the
intensity of each AU. The SVP was trained using multiple databases such as the Cohn Kascade
Database which includes coded videos and static images. The SVP output is the distance
between the cropped face and the separating hyperplane. This value correlates with the
intensities of each emotion. In turn, the intensity score allows for a reliable and consistent way to
measure the intensity of AUs, a task which is difficult to accomplish manually.

James 9

Analysis

Although the subjects watched the entire clip, specific attention was given to the Interval of
Interest (IOI), which is 30 seconds of the clip that contains the most emotion eliciting content.

During the IOI, the Flatness and Inappropriateness of the subjects face was also measured.
Flatness is a measure of how stoic a persons face is in response to a situation. For example, if
someone were to see a close relative or friend pass away, one can expect them to show signs of
remorse such as crying. Their face would be flat if they showed no emotion at all.
Inappropriateness is a measure of how appropriate the reaction is in a situation. Laughing or
smiling in a sad situation would be inappropriate.

Flatness was calculated by first finding the number of frames where the AU evidence for all the
AUs was less than 0.2. This value was then divided by the total number of frames to produce a
decimal value. Inappropriateness was calculated by first finding the number of frames where the
targeted emotion was not the dominantly expressed emotion. This number was then divided by
the total number of frames to get a decimal value. The values ranged from 0.001 to 1.

Results

Thus far, the FACET and PFQ data for the Amusement and Disgust (neutral phase) film clips
have been analyzed for the TBI and Control subjects.
James 10

TBI
For this group, Amusement was the lowest evident emotion on facial expression analysis with
FACET (Figures 2, 4, 6). However, it was the highest-rated emotion experienced as per the postfilm questionnaires (PFQ, Figures 1, 3, 5). Anger was not at all evident in and disgust was
minimal in both PFQ and FACET data. Confusion was evident in two of the three subjects PFQ
scores and in all their FACET scores (Figure 1-6). Overall, the PFQ scores did not correspond
with their FACET scores (Figures 1-6).

Control
The target emotion on the film clip showed the most prominent facial expression in the Control
group (Figures 8, 10, 12). The target emotion was also rated most highly on the PFQ (Figures 7,
9, 11). During the Disgust evoking clips, the Control group however also showed high confusion
on both the FACET and PFQ (Figures 7-12).

Disgust and Amusement

During the sequence of Disgust clips, the TBI group evidenced less disgust, and more anger and
confusion than the Control group. Furthermore, there was more evidence of sadness and surprise
for TBI subjects. During the Amusement clips, the TBI group showed more evidence of anger
and confusion, and less evidence of surprise than the Control group. There was no difference in
the expression of sadness for the TBI group, when watching the Disgust and the Amusement clip.

James 11

TBI Subject 1 FACET

TBI Subject 1 PFQ

6.0

10

3.0

0.0
Amusement Anger
FACET Disgust

Fear

Frustration

0
Amusement Anger

FACET Amusement

PFQ Disgust

Figure 1: FACET evidence during Disgust and Amusement

during Disgust and Amusement
video clips for TBI Subject 1.

TBI Subject 2 FACET

10

FACET Disgust

Fear

Frustration

FACET Amusement

Frustration

PFQ Amusement

Figure 2: PFQ evidence

video clips for TBI Subject 1.

TBI Subject 2 PFQ

0
Amusement Anger

Fear

0
Amusement Anger
PFQ Disgust

Figure 3: FACET evidence during Disgust and Amusement

during Disgust and Amusement
video clips for TBI Subject 2.

Fear

Frustration

PFQ Amusement

Figure 4: PFQ evidence

video clips for TBI Subject 2.

James 12

TBI Subject 3 FACET

TBI Subject 3 PFQ

10

0
Amusement Anger
FACET Disgust

Fear

Frustration

0
Amusement Anger

FACET Amusement

PFQ Disgust

Figure 5: FACET evidence during Disgust and Amusement

Disgust and Amusement
video clips for TBI Subject 3.

Fear

Frustration

PFQ Amusement

Figure 6 PFQ evidence during

video clips for TBI Subject 3.

Control Subject 1 FACET Control Subject 1 PFQ

6

10

0
Amusement Anger
FACET Disgust

Fear

Frustration

0
Amusement Anger

FACET Amusement

PFQ Disgust

Fear

Frustration

PFQ Amusement

Figure 7: FACET evidence during Disgust and Amusement

Figure 8: PFQ evidence
during Disgust and Amusement
video clips for Control Subject 1.
video clips for Control Subject 1.

Control Subject 2 FACET Control Subject 2 PFQ

6

10

0
Amusement Anger
FACET Disgust

Fear

Frustration

FACET Amusement

0
Amusement Anger
PFQ Disgust

Fear

Frustration

PFQ Amusement

James 13

Figure 9: FACET evidence during Disgust and Amusement

Figure 10: PFQ evidence
during Disgust and Amusement video video clips for Control Subject 2.
clips for Control Subject 2.

Control Subject 3 FACET Control Subject 3 PFQ

6

10

0
Amusement Anger
FACET Disgust

Fear

Frustration

0
Amusement Anger

FACET Amusement

PFQ Disgust

Fear

Frustration

PFQ Amusement

Figure 11: FACET evidence during Disgust and Amusement

Figure 12: PFQ evidence
during Disgust and Amusement video video clips for Control Subject 3.
clips for Control Subject 3.

Discussion
In this study, we analyzed the facial expressions of subjects with TBI and age-matched Control
subjects as they watched validated film clips that were intended to evoke target emotions. The
subjects faces were videotaped as they watched the film clips. They were also asked to complete
questionnaires after watching each clip where they rated their emotional experiences.

Social Dysfunction
It is known that TBI affects social relationships. Relatives have noted that subjects showed
changes in self-control and social perception, creating distress in their relationships (Henry,
2006). They had difficulty understanding basic emotions, subtle inferences and intentions of
James 14

those they interacted with (McDonald & Flanagan, 2004). Subjects also have an increased
divorce rate following their injury (Brooks, 1986). In this study, TBI subjects had difficulty
expressing an appropriate emotional response. They were confused when watching video clips
that should have evoked either disgust or amusement. Their measure of disgust, when watching a
disgust-evoking film clip, in expression and feeling was less than that of the Control groups. This
was also the case for the amusement-evoking film clip. It was reflected in their responses,
collected by the Post Film Questionnaires (PFQ), as well as identified from FACET.
Studies have noted that TBI patients are challenged in identifying emotions in others (Caulder,
2000; Croker et al., 2005). This study identifies the added inability of TBI subjects to express
appropriate emotional responses to external stimuli, in this case, emotion evoking film clips. The
inability to understand emotions in others and within themselves can be a direct precursor to
developing poor social relationships. It can develop feelings of frustration and anger. This can
lead to sudden aggression such as verbal threats and even more serious physical harm
(Gertler,2006).
In conclusion, even though this was an exploratory study with a small number of subjects, the
findings are relevant to the continued study of TBI patients in this area. Data collection for this
study included a wider range of target emotion for TBI and Control subjects. This data continues
to be examined. The FACET software was a useful tool in analyzing emotive response of TBI
subjects.

James 15

References
Brooks, N., L. Campsie, C. Symington, A. Beattie, and W. Mckinlay. "The Five Year Outcome of
Severe Blunt Head Injury: A Relative's View."Journal of Neurology, Neurosurgery & Psychiatry
49.7 (1986): 764-70. Web.
Brown, C. (2011). Emotion Recognition and Traumatic Brain Injury (Masters Thesis). Arizona
State University, Arizona.
Calder, A., Keane, J., Manes, F., Antoun, N., & Young, A. (2000). Impaired recognition and
experience of disgust following brain injury. Nature Neuroscience, 3(11), 1077-1078.
Croker, V., & Mcdonald, S. (2005). Recognition of emotion from facial expression following
traumatic brain injury. Brain Injury, 19(10), 787-799.
Ekman, P., & Friesen, W. (1976). Measuring facial movement. Environmental Psychology and
Nonverbal Behavior, 1(1), 56-75.
Ekman, P., & Friesen, W. & Hager, J. (2002). Facial action coding system. Salt Lake City, Utah:
A Human Face.
Henry, Julie D., Louise H. Phillips, John R. Crawford, Magdalena Ietswaart, and Fiona
Summers. "Theory of Mind following Traumatic Brain Injury: The Role of Emotion Recognition
and Executive Dysfunction."Neuropsychologia 44.10 (2006): 1623-628. Web.
Gertler, Paul. "Angry Young Men: Aggressive Behaviour after Traumatic Brain Injury." The
Australian Psychological Society (2006): n. pag. Web.
G. Littlewort , J. Whitehill , T. Wu , I. Fasel , M. Frank , J. Movellan and M. Bartlett The
computer expression recognition toolbox (cert). Proc. IEEE Int. Conf. Autom. Face Gesture
Recog., pp.298 -305 2011
J. Hamm, A. Pinkham, R. Gur, R. Verma, and C. Kohler, Dimensional Information-Theoretic
Measurement of Facial Emotion Expressions in Schizophrenia. Schizophrenia Research and
Treatment, vol. 2014
Mcdonald, S. (2005). Are You Crying or Laughing? Emotion Recognition Deficits After Severe
Traumatic Brain Injury. Brain Impairment, 6(1), 56-67.
McDonald, S., and S. Flanagan. "Social Perception Deficits after Traumatic Brain Injury: The
Interaction between Emotion Recognition, Mentalising Ability and Social
Communication."Neuropsychology 18 (2004): 572-79. Web.
James 16

Uy, Catherine C., Iain A. Jeffrey, Matthew Wilson, Viswanath Aluru, Anita Madan, Ying Lu, and
Preeti Raghavan. "Autonomic Mechanisms of Emotional Reactivity and Regulation."
Psychology 04.08 (2013): 669-75. Web.

James 17