Professor Hussein Al Osman 1

2
 Emotion and Affect will be used interchangeably in this course
 Emotional state refers to the internal dynamics associated with emotions
 Emotional state cannot be directly observed by another person – it is inferred

 Emotional experience refers to the conscious perception of your own emotional

state
 Emotional expression refers to what is revealed to others, either voluntarily (e.g.
smile) or involuntarily (e.g. nervous twitch)
 Mood refers to a longer-term affective state. The precise duration is not well
defined.

3
 Affective computers are machines that recognize and express emotions

 How can a computer recognize emotions?

 How can a computer express emotions?

 What is the definition of emotions?

4
 Emotions are thoroughly investigated in the field of psychology
 Classic theories of emotion:

 James-Lange theory: Emotions are physical processes. Emotions are “the experience of
sets of bodily changes that occur in response to emotive stimuli” 1.

 Cannon-Bard theory: Emotions are cognitive processes originating in the brain.

Physiological changes in response to an emotion are a consequence rather than the
instigator of the emotion.

1 - LeDoux, Joseph. "The emotional brain, fear, and the amygdala." Cellular and molecular
neurobiology 23.4-5 (2003): 727-738.
5
 Sensing physiological changes in the body gives rise to emotions

 Example: “So, if we meet a bear in the woods, it is not the case that we feel frightened
and run; rather, running away follows directly from our perception of the bear, and our
experience of the bodily changes involved in running is the emotion of fear.”1

1 - LeDoux, Joseph. "The emotional brain, fear, and the amygdala." Cellular and molecular
neurobiology 23.4-5 (2003): 727-738.
6
 Understanding a situations gives rise to emotions
 Example: “That was an important goal to me and you prevented me from attaining it;
therefore, I am angry.”

 Cannon and Bard found that decorticated cats were prone to make unexpected and
unwarranted anger attacks

7
 Today, most researchers agree that the brain and body interact in the generation of
emotion and its experience

 Thoughts can lead to emotions

 Changes in bodily chemistry can lead to emotions

8
“Rafe's friends always describe him as a happy person. He likes to play tennis
and finds great enjoyment in watching the top professionals play the game.
After watching his favorite player win in the semifinals of a grand prix tennis
tournament, Rafe contentedly stood in line under a hot August sun waiting to
get a cool drink. As the glow of his vicarious victory faded, the heat and
humidity became more and more oppressive. Suddenly, Rafe felt a piercing
pain from a blow to his lower back. Rafe turned rapidly with an angry
expression and clenched fist. Rafe saw that he had been hit by Rebecca, a
person with hemiplegia whose wheelchair had gone out of control and caused
her to crash into Rafe and to spill her drink on her dress. Rafe's understanding
that the cause of his pain was an uncontrollable even that had embarrassed
Rebecca immediately changed his anger to sadness and sympathy. Though
still in pain, his happy nature surfaced, and he began helping Rebecca recover
from the accident.” 2
9
 Will affective computers act irrationally?
 In popular culture, acting emotionally is synonymous with acting irrationally with poor
judgement

 Imagine a computer throwing an angry fit after the user enters the input incorrectly
several times
 Should we give computers affective abilities at all?

10
 Will computers ever exhibit innate emotional capacities or will they simply
simulate (fake) this capability?
 Is there a difference between the two?

11
 An affective computer can infer the emotional state of an agent it is interacting with
from the observation of its emotional expression

 Recognition of emotion may require the use of various sensors:

 Camera
 Microphone
 Physiological sensors

12
 The term affective computing was coined in the late 1990s by
Rosalind Picard

 Concepts related to what we term affective computing today

precede the existence of the term

 For example, the film 2001: A Space Odyssey (released in

1968) showcases the HAL 900 computer endowed affective
capabilities2

Rosalind Picard

2 Daily, Shaundra B., et al. "Affective computing: historical foundations, current applications,
and future trends." Emotions and Affect in Human Factors and Human-Computer Interaction. 13
Academic Press, 2017. 213-231.
 To judge emotional expression, humans rely in general on verbal and nonverbal
channels
 Verbal channels: speech (tonal and semantic information)
 Nonverbal: eye gaze and blink, facial and body expression

 Facial expression and speech intonation are believed to be the most relied upon by
humans for emotions’ interpretation
 These channels are likely rich in informational cues about the affective state

14
 Social psychologists have remarked that emotional expression can be consciously
regulated to convey a calculated self-presentation
 Nonverbal emotional channels tend to be less vulnerable to deliberate manipulation

 When verbal behavior conflicts with nonverbal comportment, nonverbal

expressions may be more reflective of the true affective status
 Speech intonation may be the least consciously controllable modality

15
 The visual modality is rich in relevant informational content
and includes:
 Facial expression
 Eye gaze
 Pupil diameter
 Blinking behavior
 Body expression (posture, gesture, gait)

16
 By far, the most studied nonverbal affect-recognition method
 Facial expressions may be the most intuitive indicators of affect
 Children draw simplistic faces that convey various emotions by manipulating the
forehead creases, eyebrows, and mouth

 We use emojis in digital textual communications that convey emotions through simple
facial-expression depictions

17
 Facial expressions result from the contraction of facial
muscles resulting in the temporary deformation of the
neutral expression
 The deformations are typically brief and last mostly between
250ms and 5s
 Charles Darwin was one of the early researchers to explore
the evolutionary foundation of facial-expressions display
 According to Darwin:
 Facial expressions are universal across humans
 They are habitual movements associated with certain states of
the mind
 The habits were favored through natural selection and inherited
across generations Charles Darwin

18
 Anger Anger Anger Anger
Sadness Sadness Sadness Sadness
Disgust Disgust Disgust Disgust
Happiness Happiness Happiness Happiness
19
 Paul Ekman conceived the facial action coding system
(FACS)
 It describes all possible perceivable facial muscle movements in
terms of predefined action units (AUs)
 All AUs are numerically coded and facial expressions
correspond to one or more AUs
 FACS can be used to describe facial muscle activation
regardless of the underlying cause.

Paul Ekman

20
 AU4: Brow Lowerer
AU6: Cheek Raiser AU7: Lids Tightener

AU44: Squint

AU9: Nose Wrinkler

AU27: Mouth Stretch

AU26: Jaw Drop

AU15: Lip Corner Puller

21
 Ekman agrees with Darwin that Facial Expressions are universal

 He contends that facial expressions are modulated by cultural display rules

 Display Rules:
 Growing up, we learn when to show emotions and to whom
 Facial emotional expressions can be suppressed, de-amplified, or exaggerated
depending on the social context

 This creates a challenge for affective computers that assess facial expressions to
infer the emotional state

22
 What facial expression is displayed by this subject?

Image obtained from the The Japanese Female Facial Expression (JAFFE) Database -
http://www.kasrl.org/jaffe.html
23
 Paul Ekman still contends that facial expressions are universal, although modulated
by display rules

 In a 2016 study, Crivelli et al. found that adolescents from the Trobriand islands of
Papua New Guinea interpreted a gasping face as conveying anger and threat 3
 The same facial expression is seen by western subjects as conveying fear and submission

 For the purpose of this course, we will suppose that facial expressions are universal
 I believe this assumption stands for the majority of the world populations, despite cultural
changes in display rules

3 - Crivelli, Carlos, et al. "The fear gasping face as a threat display in a Melanesian society."
Proceedings of the National Academy of Sciences 113.44 (2016): 12403-12407.
24
Questions? 25