Beruflich Dokumente
Kultur Dokumente
brill.com/artp
E-motions
Rossana Actis-Grosso and Daniele Zavagno
Universit degli Studi di Milano-Bicocca, Dipartimento di Psicologia,
Piazza dellAteneo Nuovo, 1, 20126 Milano, Italy
Received 30 April 2013; accepted 1 November 2013
Abstract
An experiment is presented, aimed at preliminary testing the hypothesis according to which facial
expressions related to specific emotions, such as anger, fear, and joy, incorporate a sense of dynamicity and are used to enhance the representation of motion in static artworks. Since a growing body
of evidence shows that motion is one of the core components of emotion, and the representation of
motion in art is often achieved by portraying unstable poses, we hypothesize that the visually more
dynamic emotions are those with unstable facial expressions, i.e., expressions that imply muscular
tensions that cannot be held for long (e.g., rage, grief, amazement: E-motions) whereas static emotions are those which can last and even represent a constant facial feature in some people. To test this
hypothesis we chose eleven static artworks from which we derived twelve human figures that convey
different emotions in different proportions. Images were manipulated to produce two sets of stimuli:
headless bodies (Set 1) and bodiless heads (Set 2). Participants were asked to rate perceived dynamicity of stimuli from Set 1 on a 7-point Likert scale (Session 1) and to rate each stimulus from Set 2 for
joy, sadness, surprise, disgust, anger, fear, serenity, puzzlement and dynamicity (i.e., eight emotions
and dynamicity). As expected, we found that some facial emotions (i.e., disgust, anger and fear) are
positively related to the dynamicity attributed to the artworks: those emotions are the more sudden
ones and thus the more unstable. We also found that serenity is negatively related to dynamicity.
Contrary to our expectations, we found instead that joy is statistically different from dynamicity, a
result that calls for further investigation.
Keywords
Emotions, motion, dynamicity, visual arts
1. Introduction
Since the seminal study by Ekman and Friesen (1971), it is accepted that facial
expressions associated with basic emotions (i.e., Happiness, Surprise, Fear,
*
DOI:10.1163/22134913-00002022
54
Sadness, Disgust and Anger) are universal: humans not only express their
emotions in the same way, but they are also able to identify the expressions of
emotions reliably in photographs portraying people from other cultures (e.g.,
Russell, 1994).
There is a growing body of evidence showing a link between motion and
emotion, with several studies specifically focused on facial expression. For
instance, in an attempt to reduce non-motion cues in video sequences, several studies examining the role of facial movement have used an adaptation
of Johanssons (1973, 1977) point-light technique to isolate information provided by facial motion from feature-based information (Bassili, 1978, 1979;
Bruce and Valentine, 1988). Bassili (1978) showed that observers could recognize several basic emotions on the basis of facial movement alone; however,
accuracy rates increase significantly when the faces of the actors are fully visible. Bassili (1979) proposed that, under good viewing conditions, the observer
has access to a redundant system of facial information that can assist emotion
recognition when viewing conditions are poor. Although results of these pointlight studies show that facial motion increased the likelihood of the recognition
of basic expressions, some authors (Bould and Morris, 2008; Fiorentini and
Viviani, 2011; Knight and Johnston, 1997) consider these results as controversial, given that in point-light studies (a) the outline of features could still
be detected, thus providing the observer with additional information on the
spatial structure of facial features related to emotional expression and (b) both
the spatial information about the facial structure and the dynamic information
about the changes in the facial structure are degraded, whereas, ideally, if one
wishes to rely on dynamic cues, only spatial information should be degraded,
while leaving cues from image motion intact.
However, facial expressions are not the only source of input that conveys
emotionally relevant information. In everyday situations, other sources (such
as the communicators body language or bodily kinematics) are equally important, especially when facial expressions are inconsistent or unavailable
to the observer. Regarding bodily kinematics, the link between motion and
emotions is far more clear: at a perceptual level it has been demonstrated
that biological motion i.e., a point-light motion display (Johansson, 1973,
1977) is sufficient for the perception of emotions (Atkinson et al., 2004;
Clarke et al., 2005), while investigations of dance movements with point-light
displays indicate that some emotions can be recognized above chance levels
(Brownlow et al., 1997; Dittrich et al., 1996; Walk and Homan, 1984). Moreover, arm movements alone, performing simple actions, have been shown to
convey information about affect (Pollick et al., 2001). This may imply that
people are able to perceive emotions from kinematic patterns without having
to compute the detailed shape of the human form first.
55
It was recently found that specific motion patterns increase perceived intensity and arousal related to emotional faces. Chafi et al. (2012) presented
emotional faces encrusted in a disk; the disk could move according to three
different motion patterns: translational (leftright and rightleft) parabolic and
wave-like. Those three motion patterns are known to be cognitively associated
with different emotions (Podevin, 2009); Chafi et al. (2012) demonstrated that
they are also perceptually associated with emotional faces, given that wavelike motion increases perceived intensity and arousal related to positive faces
(i.e., surprised and happy), while parabolic motion has the same effect on
sad faces. These results thus suggest that motion could change the perception of emotional faces even when the face is not moving (or at least it is not
moving with the normal deformation of its surface while changing expression).
Relying on these findings, we hypothesize that some emotional facial expressions (EFEs) convey a stronger impression of dynamicity or, in other
words, that they are more motion-related than other EFEs, and that consequently these expressions embody a stronger sense of motion. In saying this
we are somehow reversing the logical path underlying the literature described
above. All the studies regarding the link between motion and emotions are
mainly interested in the recognition of different emotions and how this can
be improved by taking into consideration the motion of the whole pattern;
or how motion per se can improve the recognition of emotions. Here instead
we are interested in investigating whether EFEs contribute to the perception
of motion or dynamism within static representations of the human figure. Our
hypothesis is that pictures of emotional faces may convey information with respect to implied motion: namely the fact that a still photograph of an object in
motion may convey dynamic information about the position of the object immediately before and after the photograph was taken. Psychophysical studies
suggest that observers extract this kind of dynamic information by extrapolating an objects future position from the motion implied in a static photograph
even when the experimental task does not require such an operation (Freyd,
1983). In addition, it has been demonstrated that brain regions involved in
the visual analysis of motion are also engaged in processing implied dynamic
information from static images (Kourtzi and Kanwisher, 2000).
Focusing on the facial expression of emotions, our working hypothesis is
that all emotions can be classified in terms of inherent dynamism. With this
term we do not want to contrast the multidimensionality that characterizes
the experience of emotions, nor subvert any of the existing models on emotions. Instead we want to focus on the inherent dynamic content that might
be a visible trace within the facial expression of an emotion. Hence, on the
dynamic side of a staticdynamic scale, dynamic emotions (i.e., emotions that
might convey implied motion information) are mostly those with unstable
56
facial expressions, i.e., expressions that imply muscular tensions that cannot
be held for long (e.g., rage, desperation, amazement), whereas static emotions
are those that can last and even represent a constant facial feature in some
people (e.g., serenity, apprehension, boredom). In other words, our hypothesis
is that the facial expression of some emotions implies motion, allowing the
observer to extract dynamic information from their static representations. We
therefore re-propose the play on words e-motion (Dittrich et al., 1996) to
emphasize the hypothetical relevant presence of a motion (or dynamic) component embodied in specific EFEs.
If our hypothesis makes sense, then the representation of a human figure
with a specific EFE could appear as more or less dynamic, depending on the
emotion on her face.
As a preliminary test to our hypothesis, and to determine which emotions
are more motion related, we chose to start focusing on the visual arts, which
devised a number of different strategies aimed at coding and representing the
dynamic content of an event (Actis-Grosso and Zavagno, 2008).
The representation of motion in art is often achieved by portraying unstable
poses (Arnheim, 1969; Massironi, 2002), e.g., a posture representing a transition from state A to state B, or part of a possible series of transitions. In other
words, implied motion was very well known by painters, who often chose to
represent motion with poses that allow the observer to anticipate the future
position a moment after the actual position, or to extrapolate dynamic information about the position of the portrayed object immediately before and after
the portrayed position. As stated earlier, if our hypothesis on e-motions holds
true, then it should be possible for a visual artist to make use of facial expressions of e-motions to enhance the representation of motion in static artworks.
The question thus being whether EFEs contribute to build the dynamic content
of figurative artworks in which the representation of humans is involved.
Since our hypothesis implies that e-motions enhance the representation of
motion in static artworks, it should be possible to classify emotions based on
the dynamicity conveyed by and attributed to the paintings where they
are portrayed. In this study we preliminarily tested this possibility by limiting
our investigation to a single person portrayed in a painting, in order to verify
whether participants are able to consistently rate (a) the type of emotion depicted on the face and (b) the dynamicity conveyed by a face. At this stage, we
did not consider artistic style, which, however, may be rather influential, and
hence we handpicked a number of artworks that appeared to us as displaying
a high degree of dynamicity or and/or of emotional states.
An EFE, however, may work in two ways: it may enhance the dynamic
appearance of a human figure (the body and the EFE are both either ranked low
or ranked high on a staticdynamic scale) or it may contrast it (body high/EFE
57
low, or body low/EFE high). For this reason we decided to consider in our
study not only the face of the portrayed person, but also the body.
We decided to focus our analysis on the following eight emotions: joy, sadness, surprise, disgust, anger, fear, serenity and puzzlement. We chose not to
limit our analysis to the emotions suggested by Ekman and Friesen (1971),
but to extend it to a wider range of emotions or emotion schemas (see for example Izard, 2009), so as to comprise also emotions related to a lower degree
of muscular tension. Our hypothesis concerning e-motions comprises a wide
range of emotional experiences, as classified, for example, by Kissin (1986)
or Plutchik (1980). Our working hypothesis is that some of these emotions
(i.e., puzzlement and serenity) should be negatively correlated to dynamicity,
as the expression of these emotions does not request a high degree of muscular tension (and normally they do not imply action), whereas joy, surprise,
anger, disgust, and fear should be positively correlated with dynamicity. We
have no particular expectations about sadness: this emotion is usually related
to a general slow-down in terms of reactivity to environmental stimulation, but
it does not necessarily imply absence of motion (and its facial expression is
not necessarily unstable).
Given the novelty of our hypothesis, there are steps that need to be taken to
approach a full investigation. First of all, we need to test whether our working
hypothesis has some ground. Does it make sense to categorize different EFEs
on a staticdynamic scale?
2. Materials and Methods
2.1. Participants
Twelve participants voluntarily took part in this experiment (2329 years old;
mean age: 23.08; female: 5). They all had normal or corrected-to-normal vision and were all nave as to the purpose of the experiment.
2.2. Stimuli
To test our hypothesis we chose eleven static artworks (Fig. 1) from which
we derived twelve human figures. In these artworks emotions are conveyed
either by facial expression or by body pose, or by a combination of both facial
expression and body pose in different proportions. In addition, we selected
four drawings, two headless bodies and two bodiless heads, that should be on
the opposite ends of a dynamicity scale.
Images were manipulated to produce two sets of stimuli: headless bodies
(Set 1) and bodiless heads (Set 2). Each set of stimuli comprised 14 images.
58
Figure 1. Artworks from which stimuli for headless bodies and bodiless heads were derived.
(1, 4, 6, and 11) Caravaggio; (2) Delacroix; (3, 5, and 12) Giotto; (7) Guido Reni; (8 and 10)
Nicol dellArca; (9) Masaccio; (13a and 14a) Leonardo da Vinci; (13b and 14b) Michelangelo
Buonarroti.
59
2.3. Procedure
The stimuli were displayed on a 19-inch LCD monitor (Samsung Syncmaster943; res.: 1280 1024; dynamic contrast: 50000:1). Stimuli presentation
was controlled by E-prime 2. Each participant sat at a comfortable viewing
distance from the screen (about 60 cm) in a dimly lit room. The experiment
was divided into two sessions. In the first session participants were asked to
rate perceived dynamicity of stimuli from Set 1 (i.e., headless bodies) on a
7-point Likert scale by pressing a key (1 = not dynamic; 7 = very dynamic).
In the second session participants were asked to rate each stimulus from Set 2
(i.e., bodiless heads) on 7-point Likert scales corresponding to the following
nine dimensions: joy, sadness, surprise, disgust, anger, fear, serenity, puzzlement and dynamicity (i.e., eight emotions and dynamicity).
Within each experimental session stimuli were randomly displayed. In the
first session each stimulus lasted until the response was given. In the second
session, a written reminder (e.g., anger) was displayed on top of each figure
for each dimension to be rated; when a response was given for a given dimension (by pressing keys 1 to 7), another written reminder appeared above the
figure. When all dimensions for a given stimulus were rated, a new stimulus
appeared. Participants were always asked to first rate a stimulus with respect to
the eight emotions (in random sequence) and lastly with respect to dynamicity.
With this experimental design we aimed at: (1) measuring the dynamicity
conveyed by the body alone; (2) measuring the dynamicity conveyed by the
face alone; (3) understanding whether the faces we chose deliver strong impressions of specific emotions; (4) comparing the dynamicity ratings for the
body with those for the face.
2.4. Results
Figure 2a and b display the results for both sets of images; the big graph
combines dynamicity ratings for headless bodies (dark grey bar) and bodiless
heads (light grey bar); while the small graph next to each head shows scores
for emotions averaged across participants The graph is organized from the
lowest face dynamicity scores to the highest. Except in one case (image 2), all
other scores for face dynamicity are either lower or equal to the corresponding
scores for body dynamicity.
Paired t-tests between scores, averaged across images, attributed to each
emotion and the mean score for face dynamicity (bodiless heads) indicates
that joy, sadness, perplexity, and serenity are statistically different from face
dynamicity (respectively t (167) = 5.99, t (167) = 5.73, t (167) = 3.93, and
t (167) = 4.97, p < 0.0001), while all other emotions are not.
Table 1 shows Pearson correlations (12 subjects 14 images). Considering
ratings for emotions, correlations show internal consistency between types of
60
(a)
Figure 2. Results. The big graphs display mean scores for dynamicity for headless bodies (light
grey bars) and bodiless heads (dark grey bars). The small graph next to each head displays the
heads scores for the eight emotions.
61
(b)
Figure 2. (Continued.)
emotions. For instance, joy correlates positively with serenity and negatively
with sadness, anger, fear, and perplexity, while perplexity correlates positively
with surprise (and negatively with serenity). Considering the ratings for each
emotion and the ratings for face dynamicity, we found that the last correlates
Joy
Sadness
Surprise
Disgust
Anger
Fear
Perplexity
Serenity
DynFace
DynBody
1.91
4.15
3
2.98
2.71
3.11
3.84
1.98
2.97
3.72
Mean
1.51
2.1
2.02
2.13
1.97
2.08
2.04
1.6
1.73
1.92
St. dev.
Sadness
Surprise
Disgust
Anger
Fear
1
0.353 0.061
0.103
0.262 0.288
1
0.104
0.247
0.270
0.315
0.353
0.061
0.104
1
0.253
0.096
0.342
0.253
1
0.452
0.383
0.103
0.247
0.262
0.270
0.096
0.452
1
0.534
0.288
0.315
0.342
0.383
0.534
1
0.318 0.004
0.156
0.008
0.079
0.079
0.205 0.307 0.426
0.692 0.398 0.116
0.011
0.050
0.337
0.235
0.214
0.426
0.146
0.211
0.023
0.377
0.505
0.452
Joy
Table 1.
Pearson correlations (12 participants 14 images). p < 0.01 (two-tailed). p < 0.05 (two-tailed)
0.318
0.004
0.156
0.008
0.079
0.079
1
0.183
0.141
0.097
Perplexity
DynFace
DynBody
0.692
0.011
0.146
0.398
0.050
0.211
0.116
0.337
0.023
0.205
0.235
0.377
0.307
0.214
0.505
0.426
0.426
0.452
0.183 0.141
0.097
1
0.187 0.282
0.187
1
0.245
0.282
0.245
1
Serenity
62
R. Actis-Grosso, D. Zavagno / Art & Perception 3 (2015) 5366
63
positively with surprise, disgust, anger, and fear, and negatively with serenity.
Finally, ratings for the dynamicity of the face correlate positively with ratings
for the dynamicity of the body, meaning that overall the dynamic content provided by facial expressions was consistent enough with the dynamic content
expressed by the body, though only in few cases ratings were actually comparable (see Fig. 2).
In fact, paired t-tests between scores for each image for face dynamicity
and body dynamicity conveyed by corresponding images proved to be significant for images 2 (t (11) = 0.56, p = 0.001), 3 (t (11) = 2.72, p = 0.020),
4 (t (11) = 4.42, p = 0.001), 5 (t (11) = 2.86, p = 0.015), 6 (t (11) =
3.93, p = 0.020), 8 (t (11) = 2.76, p = 0.018), 10 (t (11) = 3.09, p =
0.010), and 12 (t (11) = 3.76, p = 0.003).
3. Discussion and Conclusions
Despite the preliminary nature of our study, results appear promising: EFEs
can be distinguished on a dynamicity scale. In particular, facial expressions
related to specific emotions, such as anger, fear, or disgust, appear to incorporate a higher sense of dynamicity. As expected, ratings for such emotions
were positively correlated to the dynamicity attributed to the bodiless faces;
in fact those emotions are characterized as abrupt, which adds to their relative
unstableness.
Contrary to our expectations, we did not find a correlation between scores
for face dynamicity and joy, while we found a negative correlation between
face dynamicity and serenity. The last was expected, given our assumption that
serenity corresponds more to an inner state that is not necessarily manifested
in a dynamic way.
With regards to joy, instead, it can be conceptualized as the opposite of sadness, which also did not correlate with face dynamicity. It could be that these
two terms imply a lower degree of arousal (and therefore of dynamicity) than
what we expected (Kissin, 1986). In fact, looking at faces 1 and 8 (with high
scores for joy and sadness, respectively; see Fig. 2), the impression is that
words such as joy and sadness do not fully describe the emotions depicted
by the images; terms like cheerfulness and despair appear more suitable, as
they imply a higher degree of arousal. Joy and sadness are by nature ambiguous as far as arousal is concerned: speaking about joy, the term may stand
for a steady sense of happiness, or for cheerfulness or even for euphoria and
enthusiasm, emotions that are usually accompanied by sudden motion. Similarly, sadness may indicate a sense of gloom, but also refer to despair. Of
course similar observations can be advanced for every emotion. As several authors pointed out (e.g., Plutchik, 1980), every emotion can be conceptualized
along a continuum of arousal. What we think makes the difference between
64
joy and sadness and all the other emotions in this study is of course related to
dynamicity: fear, disgust and anger commonly imply actions of avoidance or
aggression, even when they are experienced at a mild degree (i.e., with a very
low arousal), whereas joy and sadness are connected to actions mostly when
they are characterized by a high degree of arousal. Thus, we speculate that the
scores for joy and sadness are significantly different for those for dynamicity
basically because those two terms can stand for a relatively large array of emotions that are differently related to dynamicity, depending also on how they are
conceptualized in terms of arousal.
Finally, we also evaluated the correlation between the dynamicity conveyed
by the body and that conveyed by the face for each pair of stimuli (Sets 1
and 2). Only negative correlations were found for image pairs 2 and 6, which
may depend on the contrast between face and body dynamicity (respectively
highly dynamic facial expression with a static body, and vice versa). In general, it emerges that faces are scored as less dynamic than their corresponding
bodies, but this may depend on the fact that motion for faces translates into a
deformation of facial features and not into translation in space. It is therefore
possible that people do not find natural, in principle, the request to rate the dynamicity of a face, while it might appear more natural to rate the dynamicity
of a body even in static images.
Hence, while we think our results are promising, there are still many issues that need to be addressed more carefully in future studies, while a whole
lot of work awaits ahead of us in view of a full range study on the impact
of EFEs on perceived dynamicity in figurative artworks. For instance, it appears necessary to have participants rate the implicit dynamicity inherent to
the terms we intend to use in future experiments, without any connection to
visual stimuli. This step is necessary because positive and negative emotions
may be conceptualized in different ways in terms of dynamicity based on the
degree of arousal with which they are usually experienced. Secondly, we need
to build a rich database of EFEs and corresponding headless bodies derived
from artworks. In particular, the bodiless heads showing EFEs should be normalized in size and color palette and comprise only frontal and three-quarter
portrait poses. Thirdly, we need to refine our experimental design by employing a forced-choice method where participants are asked to identify the main
emotion expressed by a face derived from an artwork, given that the fact that
EFEs are universally recognized is still debated (Jack et al., 2012). Finally, we
need to verify the consistency of the results emerging from the forced-choice
design by asking another group of participants to rate the intensity of the emotion expressed in each artwork. With these steps and sets of information we
can finally test our working hypothesis, on which our idea concerning the role
of EFEs in conveying implicit dynamicity information is based, which we hy-
65
66
Izard, C. E. (2009). Emotion theory and research: Highlights, unanswered questions, and emerging issues, Annu. Rev. Psychol. 60, 125.
Jack, R., Garrod, O. G. B., Yu, G., Caldara, R. and Schyns, P. (2012). Facial expressions of
emotion are not culturally universal, Proc. Natl Acad. Sci. U.S.A. 109, 127132.
Johansson, G. (1973). Visual perception of biological motion and a model for its analysis, Percept. Psychophys. 14, 201211.
Johansson, G. (1977). Studies of perception of locomotion, Perception 6, 365376.
Kissin, B. (1986). Conscious and Unconscious Programs in the Brain. Plenum Press, New York,
NY, USA.
Knight, B. and Johnston, A. (1997). The role of movement in face recognition, Vis. Cogn. 4,
265273.
Kourtzi, Z. and Kanwisher, N. (2000). Activation in human MT/MST by static images with
implied motion, J. Cogn. Neurosci. 12, 4855.
Massironi, M. (2002). The Psychology of Graphics Images. Lawrence Erlbaum Associates,
Hillsdale, NJ, USA.
Plutchik, R. (1980). A general psychoevolutionary theory of emotion, in: Emotion: Theory,
Research, and Experience, Vol. 1. Theories of Emotion, R. Plutchik and H. Kellerman (Eds),
pp. 333. Academic Press, New York, NY, USA.
Podevin, G. (2009). Influence des Mouvements Elmentairese sur les processus cognitifs emotionnels. Thse de Doctorat, Universit de Lille Nord de France UDL3, France.
Pollick, F. E., Paterson, H., Bruderlin, A. and Sanford, A. J. (2001). Perceiving affect from arm
movement, Cognition 82, B51B61.
Russell, J. (1994). Is there universal recognition of emotion from facial expression? A review
of the cross-cultural studies, Psychol. Bull. 115, 102141.
Walk, R. D. and Homan, C. P. (1984). Emotion and dance in dynamic light displays, Bull.
Psychonom. Soc. 22, 437440.