PREPRINT

Non-verbal persuasion and communication in an

affective agent
Elisabeth Andre, Elisabetta Bevacqua, Dirk Heylen, Radoslaw Niewiadoski,
!atherine Pelachaud, !hristo"her Peters, Isabella Po##i, $atthias Reh
Abstract This cha"ter deals with the counication o% "ersuasion& 'nly a sall "ercenta#e o% counication involves
words( as the old sayin# #oes, )it*s not what you say, it*s how you say it+& ,hile this likely underestiates the i"ortance
o% #ood verbal "ersuasion techniques, it is accurate in underlinin# the critical role o% non-verbal behaviour durin# %ace-to-
%ace counication& In this cha"ter we restrict the discussion to body lan#ua#e& ,e also consider ebodied virtual a#ents&
As is the case with huans, there are a nuber o% %undaental %actors to be considered when constructin# "ersuasive
a#ents& In "articular, one who wishes to "ersuade ust a""ear credible, trustworthy, con%ident and non-threatenin#&
.nowin# how not to behave is also a vital basis %or e%%ective "ersuasion& This includes resolvin# task constraints or other
%actors with the social "erce"tion considerations& These social virtual a#ents %ace any o% the sae "robles as huans
have in controllin# and e/"ressin# theselves in an a""ro"riate anner so as to establish and aintain "ersuasive
interaction& All alon# the cha"ter, uch o% our discussion will handle conce"ts a""licable both to a#ent and huan
behaviour&
Elisabeth Andre
0niversity o% Aus#bur#, 1erany e-ail( andre@informatik.uni-augsburg.de
Elisabetta Bevacqua
0niversity o% Paris 2, now at !NR3, Teleco-ParisTech, 4rance e-ail( elisabetta.
bevacqua@telecom-paristech.fr
Dirk Heylen
0niversity o% Twente, The Nederland e-ail( d.k.j.Heylen@ewi.utwente.nl
Radoslaw Niewiadoski
0niversity o% Paris 2, now at Teleco-ParisTech, 4rance e-ail( radoslaw.niewiadomski@
telecom-paristech.fr
!atherine Pelachaud
0niversity o% Paris 2, now at !NR3, Teleco-ParisTech, 4rance e-ail( catherine.
pelachaud@telecom-paristech.fr
!hristo"her Peters
0niversity o% Paris 2, now at !oventry 0niversity e-ail( Christopher.Peters@
coventry.ac.uk
Isabella Po##i
0niversity o% Roe Tre, Italy e-ail( poggi@uniroma3.it
$atthias Reh
0niversity o% Aus#bur#, 1erany e-ail( matthias.rehm@informatik.
uni-augsburg.de
1 Introduction
Persuasion is a way to in%luence other "eo"le, that is, to ake the do actions, "ursue #oals, that they would have not
otherwise& But it di%%ers %ro other ways to in%luence %or three reasons( 56 it does not i"ly the use o% %orce, but akes an
a""eal to the Persuadee*s %ree choice7 86 it clais that the #oal9action "ro"osed by the Persuader is in the interest o% the
Persuadee in that it is a sub#oal to the Persuadee*s #oals, and :6 it ais at in%luencin# throu#h counication( by lettin# the
Persuadee know that the Persuader wants hi to do so& Persuasion necessarily #oes throu#h counication( in "art throu#h
ar#uentation, that is, words and sentences7 but only a sall "ercenta#e o% counication involves words( as the old
sayin# #oes, )it*s not what you say, it*s how you say it+& ,hile this likely underestiates the i"ortance o% #ood verbal
"ersuasion techniques ;see re%( 3ec& 2, Handbook<, it is accurate in underlinin# the critical role o% non-verbal behaviour
durin# %ace-to-%ace counication( 4or no atter how correct the ar#uent or essa#e, it is unlikely to be listened to
PREPRINT
unless the nonverbal lan#ua#e is con#ruent, counicatin# credibility, con%idence and trustworthiness& 4or e/a"le, in
e/"erients concernin# the e/"ression o% attitudes and %eelin#s, $ehrabian ;$ehrabian=5>?56< %ound that when there was
incon#ruence in verbal and non-verbal essa#es bein# counicated, the relative i"ortance o% essa#es was ?@ based
on verbal likin#, :2@ on vocal likin# and AA@ on %acial likin#( receivers tended to "redoinantly %avour the non-verbal
as"ects, in contrast to the literal eanin# o% the words, durin# the counication o% attitudes and %eelin#s& ,hile the
environent, ha"tics, state o% the "ersuadee and a""earance o% "ersuader are also o% i"ortance, here we restrict the
discussion to body lan#ua#e& Body lan#ua#e is certainly not a new to"ic in the doain o% scienti%ic study ;Darwin=52?86<,
althou#h its use in "ractical "ersuasive situations has increasin#ly been the %ocus o% studies in arketin#, "olitical
ca"ai#nin# and courtroo scenarios ;Bernstein et al&=5>>B6< and is becoin# "o"ularised throu#h the "ublication
o% nuerous easily accessible books ;Pease and Pease=8CCD6<& A relatively new and novel doain %or the evaluation and
a""lication o% body lan#ua#e research is that o% ebodied virtual a#ents& I% ebodied a#ents reseble huans in
a""earance, then they can take advanta#e o% these e/tra odalities to enrich and sooth the huan-co"uter
interaction "rocess& These social virtual a#ents there%ore %ace any o% the sae "robles as huans have in controllin# and
e/"ressin# theselves in an a""ro"riate anner so as to establish and aintain "ersuasive interaction - as such, uch o% our
discussion will handle conce"ts a""licable both to a#ent and huan behaviour&
As is the case with huans, there are a nuber o% %undaental %actors to be considered when constructin# "ersuasive
a#ents& In "articular, one who wishes to "ersuade ust a""ear credible, trustworthy, con%ident and non-threatenin#&
.nowin# how not to behave is also a vital basis %or e%%ective "ersuasion& This includes resolvin# task constraints or other
%actors with the social "erce"tion considerations& 4or e/a"le, not lookin# at "eo"le due to consultin# one*s notes ay #ive
the i"ression o% dishonesty7 slouchin# due to tiredness could be isinter"reted as a si#n o% a lack o% interest& 1ood
behaviour does not a""ly solely to the s"eaker o% course& 4or e/a"le, the back#round non-verbal behaviours ade when
one does not hold the %loor can also have a lar#e e%%ect on credibility ratin#s ;3eiter et al&=8CCD6<& In the ne/t section o% this
cha"ter we re"ort on several studies on the role o% nonverbal behaviours in "ersuasion& ,hile 3ection : "resents studies
%ro the stand "oint o% the s"eaker, 3ection B e/aines the e%%ects o% "ersuasive non-verbal behaviours on listener*s
"erce"tion o% s"eaker& 3ection A continues to look at listener& It addresses the issues o% backchannel si#nals eitted by the
listener as a way o% assessin# the e%%ect o% "ersuasion on the listener& In the last section o% this cha"ter we "resent works on
virtual a#ents endowed with social ca"abilities, in "articular the ca"acities o% controllin# and ana#in# their %acial
behaviours and #estures&
2 Persuasion and Emotion
To "ersuade the other to do soethin# we ust convince hi that this is the ri#ht thin# to do7 and we can do so, as Aristotle
"ut it, throu#h lo#os =rational ar#uents6, ethos =the 'rator*s character6, and "athos =the audience*s eotions6( in other
words, to "ersuade we ust 56 "rovide #ood reasons to do what we "ro"ose 86 win the interlocutor*s trust, and :6 induce or
evoke eotions& The %irst as"ect, lo#os, is the obEect o% the science o% ar#uentation, which studies the "ossible ar#uents
and counter-ar#uents that can be ade e/"licit throu#h verbal lan#ua#e7 the second re%ers to the need %or the Persuader to
show an ia#e o% a "erson who is both a6 co"etent in the to"ic dealt with =co"etence6, in such a way that his ar#uents
are taken at %ace value, and b6 one who really wants to ake the interests o% the Persuadee =benevolence6, so that the
Persuadee does not %ear to be deceived, and to #et har %ro doin# what the "ersuader "ro"oses& In this work we do not #o
dee" into the issue o% ar#uentation, nor in the as"ect o% the "ersuader*s ethos& Here we %ocus on the role o% eotions in
"ersuasion& ,hy and how can eotions be an instruent o% "ersuasionF Eotions are linked to #oals at least in three ways(
5& eotions onitor #oals& The %unction o% eotions is to onitor huans* ada"tive #oals, since an eotion is %elt any tie
an ada"tively i"ortant #oal is, or is likely to be, %ul%illed or thwarted =;Darwin=52?86, 4riEda=5>2D6, !astel%ranchi=8CCC6<6&
4ear onitors the #oals o% survival and sa%ety, an#er the #oal o% Eustice, shae the #oals o% ia#e and sel%-ia#e
8& eotions activate #oals( they have a hi#h otivatin# "ower in that they tri##er #oals& Each eotion, alon# with %eelin#s,
"hysiolo#ical arousal, e/"ressive "attern, includes a readiness to action, that is, a #oal o% hi#h "riority that "resses to be
ur#ently %ul%illed& 4ear activates %i#ht or %li#ht, an#er tri##ers a##ression, co"assion tri##ers hel"&
:& eotions can becoe #oals( "leasant eotions #enerate a #oal o% %eelin# the a#ain, while un"leasant eotions a #oal o%
not %eelin# the ne/t tie& I% I %elt "roud because I ade y hoe assi#nent well, I*ll have the #oal to study hard a#ain,
i% I %elt ashaed a%ter havin# a##ressed a %riend I*ll have the #oal o% not doin# so anyore& Thus, eotions result in a
learnin# echanis& As we entioned, "ersuasion is a way to in%luence "eo"le, that is, to lead the to "ursue soe #oals,
and to induce the Persuadee to do so the Persuader ust convince hi that the "ro"osed #oal is a #oal o% hi#h value, also
because it is a eans %or other very valuable #oals& But to activate a #oal in a "erson only throu#h rational versus also
throu#h an eotional echanis is not the sae& 3u""ose the sae course is tau#ht by two teachers( one si"ly teaches
e the disci"line, but the other also #ives e enthusias %or that disci"line7 I will "robably choose the latter, because
PREPRINT
attendin# the %orer course only %ul%ils y co#nitive #oal, while this also %ul%ils y a%%ective #oals& In the sae vein, in
"ersuasion attachin# the "ro"osed #oal only to rational #oals o% the Persuadee is not as e%%ective as attachin# it also to
eotional #oals& 3ince eotions are linked to biolo#ically ada"tive #oals, they add a hi#her value to other #oals&
3 The Persuasive Bod
The i"ortance o% body behaviour in "ersuasive discourse has been stressed ever since ancient Rhetoric& !icero in )De
oratore+ and Guintilian in )Institutiones 'ratoriae+ showed how #esture, %ace, #aHe and "osture are an i"ortant "art o%
)Actio+ =discourse delivery6& Es"ecially #estures were studied, due to their ca"acity o% suonin#, "roisin#, e/hortin#,
incitin#, "rohibitin#, a""rovin#, and to their ease in e/"ressin# eotions, showin# attitudes, indicatin# obEects o% the
orator*s thou#ht& Guintilian*s work about #estures is "artly #uided by norative intents - he o%ten stresses what #estures
should not be used by an orator, while they are - and Eust because they are - ty"ical o% coic actors7 but he does so on the
basis o% a dee" and detailed knowled#e o% the #estures* %ors and eanin#s& 4or every #esture he also tells us in what
se#ent o% the rhetorical structure o% discourse it can be used, which eans that also #estures, as words, are subEect to rules,
and their distribution is deterined by conte/t& $oreover, %ro his descri"tion one can see that "articular cobinations o%
oveents with the sae handsha"e quite "recisely convey the eanin# o% s"eci%ic s"eech acts - %or e/a"le, we )lower
;our hands< in a"olo#y or su""lication =&&&6 or raise the in adoration, or stretch the out in deonstration or invocation+
=55A6 - or e/"ress eotions( )we soeties clench the hand and "ress it to our heart when we are e/"ressin# re#ret or
an#er+ =5CB6& 4inally, Guintilian acknowled#es that soeties throu#h #estures we ay induce "ersuasive e%%ects(
)3la""in# the thi#h =&&&6 is becoin# as a ark o% indi#nation, while it also e/cites the audience+ =58:6&
3.1 Persuasion and gesture
In "resent ties, while a hu#e quantity o% studies addresses the use o% #esture and other bodily si#nals in everyday
conversation, only "art o% the is devoted to analysin# the in "ersuasive discourse& 3oe overview soe as"ects o% the
body*s relevance in "olitical counication ;Atkinson=5>2B6<, or %ocus on the synchroniHation o% #estures with "auses and
intonation and other rhetorical devices, %requently used to quell the a""lause ;Bull=5>2D6<& 'thers investi#ate the Audience*s
"hysiolo#ical, co#nitive and eotional reactions to the "oliticians* %acial e/"ression and other vocal and bodily behaviours
;4rey=8CCC6, Bucy and Bradley=8CCB6<& 3oe recent works %inally "rovide detailed or"holo#ical and seantic
descri"tions o% #estures and on this basis #ive "ertinent insi#hts about the relation between #esture and "ersuasive discourse
;!albris=8CC:6, .endon=8CCB6, 3treeck=8CC26<& In her book )The #estural e/"ression o% a "olitician*s thou#ht+
!albris=8CC:6<, !albris analyses the "olitical discourses by Iionel Jos"in since July 5>>? throu#h A"ril 5>>2 - his %irst
onths as a Prie $inister& Throu#h an insi#ht%ul analysis o% the eta"hors e/"loited by his anual behaviour, she
deonstrates how Jos"in*s #estures, or as"ects o% their e/ecution - %or e/a"le the sha"e o% the hand, or even which hand is
used, ri#ht or le%t - can e/"ress abstract notions like e%%ort, obEective, decision, balance, "riority, "rivate or "ublic stance& At
the sae tie, thou#h, they %ul%il discursive %unctions( they can deliit or stress, enuerate or e/"licate the to"ics o%
discourse& .endon ;.endon=8CCB6< analyses #estures in di%%erent cultures and di%%erent ty"es o% interaction, by "eo"le that
tell about their "ast li%e or coent on everyday li%e events, soeties also with an ar#uentative intent, and distin#uishes
three ain %unctions o% co-verbal #estures( a re%erential %unction, o% conveyin# "arts o% the "ro"ositional content o% an
utterance7 an interactive %unction, o% hel"in# the turn-takin# ana#eent7 and a "ra#atic %unction7 within this, a #esture
has a "er%orative %unction i% it clari%ies the ty"e o% s"eech act that is bein# "er%ored7 a odal %unction i% it alters the
inter"retation o% the utterance, e&#&, throu#h ne#ation or intensi%ication7 and a "arsin# %unction( arkin# the syntactic or
te/tual structure o% a sentence or discourse& Then .endon analyses soe #esture %ailies, by sin#lin# out, %or each s"eci%ic
hand sha"e and orientation, ty"ical conte/ts o% use and %indin# uni%yin# seantic thees& 3oe o% the #estures he analyses
can well have a "ersuasive use( %or e/a"le, the )rin#+ #estures that bear a eanin# o% Kakin# "recise* or Kclari%yin#*, are
used every tie this clari%ication is i"ortant )in #ainin# the a#reeent, the conviction or the understandin# o% the
interlocutor+ in Italian culture ="& 8B56&
3.2 Persuasion, body language and politics
3treeck ;3treeck=8CC26< analyses the #estural behaviour o% the Deocratic candidates durin# the "olitical ca"ai#n o% 8CCB
in 03A& They do not use any di%%erent #estures, as to hand sha"e and oveent "attern, and the #estures they use
are very rarely iconic( "artly because, as antici"ated by Guintilian, iconics look too uch a "o"ular style o% #esturin#7 and
"artly because their %unction in "olitical discourse is not to convey re%erential but ainly "ra#atic in%oration& Let, aon#
the candidates* #estures with "ra#atic %unctions, 3treeck aintains they ainly %ul%il a "arsin# %unction, ore than a
"er%orative one7 and they do not unequivocally indicate which s"eech act is bein# "er%ored, since they do not i"ly a
%i/ed %or-%unction relationshi"& 4or e/a"le, 3treeck doubts that the )rin#+ always has a eanin# o% "recision, or that the
)"ower #ri"+ o% ovin# the %ist always conveys an assertion o% "ower& In aintainin# that the candidates* #estures ainly
PREPRINT
have a "arsin# %unction, 3treeck ;3treeck=8CC26< %urther ar#ues that the te"o o% a #esture, ore than other as"ects o% it, is
a clear cue to the discourse structure( %or e/a"le, the alternation o% ra"id and slow beats, or whether the stroke cobines
with the "eak syllable or with all stressable syllables, distin#uishes between back#round and %ore#round in%oration&
Po##i M Pelachaud ;Po##i and Pelachaud=8CC26< analysed %ra#ents o% three "re-electoral debates throu#h an annotation
schee that described #estures as to their #lobal eanin# and to the eanin# borne by their e/"ressivity "araeters, and
classi%ied the eanin#s as to their "ersuasive i"ort& Then they co"uted the quantity o% "ersuasive #estures in the two
candidates and their "ercenta#e o% distribution across the various "ersuasive strate#ies& The "ersuasive strate#ies ado"ted are
soewhat di%%erent between the two "oliticians& They di%%er as to the "ro"ortion o% "athos and lo#os7 %or both the aEority
o% #estures "ursue an ethos strate#y and both tend to "roEect ore an ia#e o% co"etence than one o% benevolence, but one
o% the two candidates the "re%erence %or the ia#e o% co"etence is uch hi#her than that o% benevolence& $oreover, the
"ersuasive strate#ies are not always conveyed by the #esture sha"e, but in soe cases only by the #esture e/"ressivity or
si"ly by an indirect eanin# o% the #esture& Pathos is contained ore ty"ically =e/clusively, %or one candidate6 in the
e/"ressivity o% #esture, while ethos in both orators is ore o%ten conveyed by the #esture as a whole& 4urther, the "athos
strate#y coes out only at the indirect level in both "oliticians, while they di%%er in their level o% indirectness&
4ro the analysis it resulted that the di%%erences how uch each "ersuasive strate#y is e/"loited and how it is conveyed
=throu#h whole #esture, or e/"ressivity, direct or indirect eanin#6 is con#ruent with either the conte/t o% the s"eci%ic
debate under analysis, or the #eneral "olitical strate#y o% the candidates, or %inally with his #eneral counicative style&
Po##i M NincHe ;Po##i and NincHe=8CC26< analysed the #aHe behaviour o% two orators, Roano Prodi and 3Oe#olPene
Royal, in "olitical debates and interviews& The study analysed the #eneral "ersuasive structure o% soe %ra#ents in ters
o% hierarchy o% #oals, and the ites o% #aHe throu#h an annotation schee that analysed their direct and indirect "ersuasive
i"ort& Prodi*s #aHe was %ound to use a co"etence strate#y uch ore %requently than other strate#ies and ore than
Royal =D?@ vs& 8?@6, while he never used a benevolence strate#y& Royal, instead, used the lo#os #aHe ost %requently
=AB@6& Also in this study it was %ound that the "attern o% #aHe "ersuasive strate#ies o% each orator was con#ruent with the
"oliticians* #lobal "olitical strate#y and with the conte/t o% the s"eci%ic debate& This sees to show that durin# "ersuasion
all body odalities coherently coo"erate to one and the sae "ersuasive #oal&
3.3 Quality of persuasive body behaviours
In a recent study, Po##i M Pelachaud ;Po##i and Pelachaud=8CC26< wondered i% it is "ossible to sin#le out soe #esture that
one could de%ine as )"ersuasive #estures+& Actually, there could be soe - very rare - #estures that one could call
"ersuasive( %or e/a"le a #esture o% incitation& Let, it is ore %requent that "ersuasiveness in a #esture does not dwell in the
#lobal eanin# o% the #esture "er se, but rather in the e/"ressivity o% #esture ;Hartann et al&=8CC86<( its s"atial e/tent,
te"oral e/tent, %luidity, "ower and re"etition& Iater, Po##i M NincHe ;Po##i and NincHe=8CC26< "ut the sae question
about the e/istence o% )"ersuasive #aHe+ =as described in "revious sub-section6&
But what does it ean that a #esture or a #aHe are )"ersuasive+F Accordin# to Po##i M Pelachaud ;Po##i and
Pelachaud=8CC26<, a #esture is "ersuasive when either the #lobal eanin# o% the #esture or the eanin# o% soe "arts or
as"ects o% it convey soe o% the seantic content that are contained in the "ersuasive structure o% a discourse& Po##i M
Pelachaud ;Po##i and Pelachaud=8CC26< and Po##i M NincHe ;Po##i and NincHe=8CC26< ado"t the odel o% "ersuasion o%
Po##i ;Po##i=8CCA6<, accordin# to which "ersuasion is an act o% social in%luence brou#ht about counication,
throu#h which a Persuader A ais to convince a Persuadee B =that is, to ake hi believe with a hi#h level o% certainty6
that a #oal 1A "ro"osed by A is a sub#oal to soe #oal 1B that B already has7 thus, A ust convince B that 1A is a
#oal o% hi#h value, "ossibly o% hi#her value than alternative #oals 1! or 1D, and ust do so by e/"loitin# the three
Aristotelian strate#ies o% lo#os =rational ar#uentation6, "athos =induce eotions in B6, and ethos =look credible and reliable
to B6& $ore s"eci%ically, he ust show that he is certain o% what he is sayin# =certainty6, that he "ro"oses #oal 1A not out o%
her own concern but in order to the #oals o% B =benevolence6, and that he has #ood ca"acities o% action and "lannin# to
"ursue his own #oals and to advice #oals to others =co"etence6& There%ore, the #esture and #aHe ites that have a
"ersuasive i"ort are those that convey the %ollowin# ty"es o% in%oration(
5& I"ortance& The si#nals conveyin# the eanin# )i"ortant+ ention the hi#h value o% a "ro"osed #oal, thus tryin# to
convince the Persuadee to "ursue it& This eanin# is ty"ically contained in soe "er%orative #estures, like incitation and
request %or attention, or other #estures like .endon*s =8CCB6 )#ra""olo+ =)%in#er bunch+6, that convey a notion o%
i"ortance as their very eanin#7 but )i"ortant+ is also the core eanin# o% beats, since every beat stresses a "art o% a
sentence or discourse, hence counicatin# )this is the i"ortant "art o% the discourse I want you to "ay attention to and to
understand+& 4inally, this Non-verbal "ersuasion and counication in an a%%ective a#ent > can be the eanin# o% soe "articular
as"ects o% the e/"ressivity o% #esture, %or instance irre#ularity or discontinuity in oveent ;!ha%ai et al&=8CC?6<&
8& !ertainty& To induce certainty in the Persuadee, the Persuader ust show sel%con%ident and certain about what he is
PREPRINT
sayin#& This is why #estures that convey hi#h certainty, like the )rin#+ entioned by .endon ;.endon=8CCB6< and 3treeck
;3treeck=8CC26< ay be "ersuasive& Let, since "ersuadin# can ean either to convince to believe soethin#, or to convince
to do soethin#, the #estures that convey a hi#h de#ree o% certainty #enerally "ersuade to believe, and only indirectly
"ersuade to do soethin#&
:& Evaluation& To e/"ress a "ositive evaluation o% soe obEect or event i"lies that it is a use%ul eans to soe #oal7 thus,
to brin# about that event or to obtain that obEect becoes desirable, a #oal to be "ursued& In a arket"lace, to convince
soeone to buy a %ood, a )check screw+ =rotatin# the ti" o% the inde/ %in#er on cheek6, that eans )#ood+, )tasty+, ade by
an Italian #rocer, would be a #ood e/a"le o% "ersuasive #esture& '% course, we cannot %ind an e/a"le like this
in our %ra#ents, due to obvious reason o% social re#ister& However, as we see below, a "ersuader, to "ursue an ethos
strate#y, can ake #estures that induce a "ositive evaluation o% hi&
B& 3ender*s benevolence& In "ersuasion not only the evaluation o% the eans to achieve #oals is i"ortant, but also the
evaluation o% the Persuader( the 3ender*s ethos& I% I a benevolent to you, you can trust e, so i% I tell you that a #oal
is worthwhile you should "ursue it& A #esture driven by the ethos strate#y o% showin# one*s oral reliability is, %or e/a"le,
"uttin# one*s hand on one*s breast, that eans )I a noble, I a %air+& This #esture is quite %requent in various cor"ora o%
"olitical counication =see %or e/a"le ;3erenari=8CC:6<, ;Po##i and Pelachaud=8CC26<6&
A& 3ender*s co"etence& I% I a an e/"ert in the %ield I a talkin# about, i% I a intelli#ent, e%%icient, you i#ht Eoin e and
"ursue the #oals I "ro"ose& 4or e/a"le, in a "re-electoral debate a candidate, in talkin# o% quite technical thin#s concernin#
ta/es rotates his ri#ht hand curve o"en, with "al to le%t, twice& This #esture eans that he is "assin# over these
technicalities, "ossibly di%%icult %or the audience7 but at the sae tie the rela/ed a""earance o% his oveent lets the
in%er that he is sart because he is talkin# o% such di%%icult thin#s easily, unconstrained& This "rovides an ia#e o%
co"etence in what he is talkin# about&
D& Eotion& I% I e/"ress an eotion, and this is trans%erred to you throu#h conta#ion ;I=8CCB6<, since eotions tri##er #oals,
a #oal will be activated in you, thus i"leentin# a "athos strate#y& Another candidate says( )I cannot "retend to act in a
country di%%erent %ro what it is+, re%errin# to his country, Italy& The oveent o% his %orear shows low s"atial e/tent and
%luidity =it is short and Eerky6 and hi#h "ower and velocity, thus conveyin# an eotion load that ais to transit a sense o%
"ride o% bein# Italian, and thus to elicit the Italians* desire to vote %or hi& Another candidate in 4rance observes that her
o""onent*s "olitics towards the une"loyed is soewhat "unitive& ,hile sayin# )sanctionner les chQoeurs+ ="unish the
une"loyed6 she raises the internal "arts o% eyebrows, thus "er%orin# a #aHe o% sadness& Thus she e/hibits an eotion and
ais to transit it to the audience( a use o% the "athos strate#y&
! The Effects of Persuasion
3o %ar we have seen what "ersuasion is on the "art o% the Persuader7 now we can wonder about the e%%ects o% body
behaviour on "ersuasion, and about how "ersuasive e%%ects can be assessed durin# interaction& As to what are the e%%ects o%
#estures on "ersuasion, several authors ake the hy"othesis that a "ersuasive discourse is ore or less e%%ective de"endin#
on the ty"e o% #estures used& Ty"ically, as Henley ;Henley=5>??6<, Bur#oon, et al&;Bur#oon et al&=5>>C6< , and !arli et al&
;!arli et al&=5>>A6< aintain, sel%-ada"tors - the #estures o% touchin# one*s body - see to have a ne#ative e%%ect on
"ersuasion& However, "resuably it is not so uch the ty"e o% #esture - whether sybolic, iconic, sel%-ada"tor, obEect
ada"tor or the like - that akes the di%%erence in "ersuasiveness& 'ur hy"othesis is, instead, that it is not "ersuasiveness
away %ro discourse, but the #esture*s eanin#& ,hy, %or e/a"le, are sel%-ada"tors ty"ically sel%-de%eatin# in "ersuasive
#estureF Probably because what a sel%-ada"tor lets you in%er is that the orator is not very sel%-con%ident, that he needs to
reassure hisel%7 and this lowers trust in the "ersuader& 'ne case in which the very eanin# o% a #esture had a ne#ative
e%%ect on "ersuasion concerns an Aerican candidate, Howard Dean, who used to ake a sin#le #esture, )inde/ u"+, or
)%in#er wa#+, which seeed to have a subtly sel%-de%eatin# e%%ect& Accordin# to 3treeck =8CC?6, this #esture dis"layed the
3"eaker*s clai that what he was sayin# was i"ortant and instructive7 but since Dean was enactin# this )hierarchical act+
in "eranence, he i#ht have #iven the i"ression he was "resentin# hisel% as one o% )su"erior knowled#e+, thus
s"oilin#, with a body behaviour soehow conte"tuous toward the audience, the ascendancy credited to hi by his early
te/tual "resence& Also in this case, there%ore, what has ne#ative e%%ects on "ersuasion is not si"ly the ty"e o% #esture used,
but its s"eci%ic eanin#&
" Assessing the effect of persuasion# the role of Bac$channel
The latter issue to investi#ate is( how can the "ersuader onitor i% and how uch his "ersuasive e%%ort has been e%%ectiveF
To this #oal, the Interlocutor*s %eedback ust be taken into account& ,hen conversin#, we try to #et our interlocutors to
en#a#e with us in a Eoint "roEect& ,e utter words, so that they will be heard, listened to, understood and that the other "erson
en#a#es in the "ro"er acts that we are solicitin#( attendin#, understandin# and reactin# a""ro"riately( answerin# our
questions, believin# our stateents, takin# u" our orders, etcetera& !onversational actions are undertaken to en#a#e the
PREPRINT
addressees in takin# actions in turn& ,hen advisin# "eo"le, we ho"e "eo"le take the advice to heart& ,hen cheerin#
the u", we ho"e they %eel ha""y %or a while& ,hen we ar#ue in %avour o% a certain "ro"osition, we ho"e that the other will
becoe convinced o% what we believe to be true&
In "ersuasive or ar#uentative conversations, Eust like with any action, it is i"ortant %or s"eakers to check to what de#ree
these actions are success%ul on all levels& In the case o% conversations these checks involve onitorin# the interlocutor
to see his or her u"take o% the Eoint "roEects "ro"osed by the s"eaker& Throu#h ty"ical behaviours involvin# #aHe, %acial
e/"ressions, head oveents, "osture and vocal backchannels listeners show that they are en#a#ed in the conversation,
"ayin# attention, showin# they are interested in what is bein# said and, in the case o% "ersuasive dialo#ue, whether they are
startin# to #et convinced or not by the ar#uents& The interaction, between s"eaker and audience is claied to be essential
in ar#uentative discourse accordin# to Nan Eeeren and 1rootendorst =;van Eeeren and 1rootendorst=5>2B6<&
In ordinary conversational situations soe s"eech acts by s"eakers are s"eci%ically calculated to elicit %ro listeners certain verbal =and "ossibly
also non-verbal6 res"onses in which they indicate understandin# and =in "articular6 acce"tance& In our view this a""lies "reeinently to the
ar#uentation advanced durin# a discussion or debate& This eans that to a certain e/tent ar#uents in debates are desi#ned to achieve "recisely
de%ined verbally e/ternaliHed illocutionary and "erlocutionary e%%ects that are iediately related to the s"eech acts "er%ored& ;"& 8B<
A "articular ty"e o% res"onse to s"eech is "er%ored throu#h backchannels& These were ori#inally characteriHed by Ln#ve
=;Ln#ve=5>?C6<6 as %ollows&
;B<oth the "erson who has the turn and his "artner are siultaneously en#a#ed in both s"eakin# and listenin#& This is because o% the e/istence o%
what I call the back channel, over which the "erson who has the turn receives short essa#es such as )yes+ and )uh-huh+ without relinquishin#
the turn& The "artner, o% course, is not only listenin#, but s"eakin# occasionally as he sends the short essa#es in the back channel& The back
channel a""ears to be very i"ortant in "rovidin# the onitorin# o% the quality o% counication&
As a subset o% %eedback e/"ressions, these backchannels si#nals in this view are short vocal essa#es that have an
i"ortant structural %unction =onitorin# the quality o% the counication6& .endon ;.endon=5>D?6< talks about a siilar
cate#ory o% e/"ressions which he ters acco"anient si#nals( )short utterances that the listener "roduces as an
acco"anient to a s"eaker, when the s"eaker is s"eakin# at len#th&+ He divides these into two %unctional #rou"s( attention
si#nals =in which one a""ears to si#nal no ore than that one is attendin#6 and assentin# si#nals that e/"ress K"oint #ranted*
or Ka#reeent*& In #eneral listener res"onses can serve several %eedback %unctions& Allwood, Nivre and AhlsOen =;Allwood et
al&=5>>:6Allwood, Nivre, and AhlsOen<6 distin#uishes %our kinds( contact, "erce"tion, understandin# and attitude&
5& !ontact( si#nals that show whether the interlocutor is willin# and able to continue the interaction
8& Perce"tion( behaviours that indicate whether the interlocutor is willin# and able to "erceive the essa#e
:& 0nderstandin#( actions that dis"lay whether the interlocutor is willin# and able to understand the essa#e
B& Attitude( reactions that tell whether the interlocutor is willin# and able to react and =adequately6 res"ond to the essa#e,
s"eci%ically whether he9she acce"ts or reEects it&
The attitude-%eedback %unctions, o% which the assentin# si#nals discussed by .endon ;.endon=5>D?6< %or an i"ortant
subset, are interestin# %or the study o% "ersuasive discourse as they i#ht indicate the success achieved by the "erson who
is tryin# to "ersuade& In real li%e e/a"les it is not always easy to say which %unction is served by a "articular %eedback
e/"ression and any will cobine several %unctions& ,ithin the annotations o% the A$I cor"us ;Jaies et al&=8CC?6<, a
di%%erence was ade between di%%erent ty"e o% s"eech acts that were ainly or e/clusively used as listener res"onses(
backchannels that acknowled#e rece"tion, CommentAboutUnderstanding that indicate understandin# or "robles with
understandin#, and assessments& The latter are de%ined as %ollows&
An assessment is any coent that e/"resses an evaluation, however tentative or inco"lete, o% soethin# that the #rou" is discussin#& ;&&&<
There are any di%%erent kinds o% assessent7 they include, aon# other thin#s, acce"tin# an o%%er, e/"ressin# a#reeent9 disa#reeent or any
o"inion about soe in%oration that*s been #iven, e/"ressin# uncertainty as to whether a su##estion is a #ood idea or not, evaluatin# actions by
ebers o% the #rou", such as drawin#s& ;&&&< It can be very short, like +yeah+ and +ok+&
Assessents are clearly used %or the attitudinal reactions, where the s"eaker e/"resses his stance towards what is said, either
acce"tance or reEection5& It thus a""ears that there is uch "olysey8 in a verbal backchannel si#nal that can only in
"art be resolved by the conte/t& In 3ection D&5 we take a look at work on nonverbal backchannels that are i"ortant in a
"ersuasive conte/t %or an E!A&
% To&ard Persuasive Embodied 'onversational Agents
Ebodied !onversational A#ents, E!As, are virtual huan-like entities, with the ability to counicate with users and9or
virtual a#ents& They are endowed with verbal and non-verbal counicative eans, such as s"eech, voice intonation,
#esture, body "osture, %acial e/"ression, #aHe, etc& Huan users have a tendency to a""ly the sae "rinci"les when
PREPRINT
5 A question looked at by ;Heylen and o" den Akker=8CC?6< was whether it is "ossible to distin#uish an utterance containin# Kyeah* which
e/"resses a stance =o% "artial a#reeent, i&e& an assessent in ters o% the A$I annotation schee6 %ro an utterance that is si"ly eant as a
backchannel& They achieved correct classi%ication only %or DC@ o% the cases when not takin# into account the s"eech act o% the "revious utterance
and 2C@ i% they did& 8 ,e avoid the use o% the ter abi#uity here as the distinction between the various cate#ories is not strict and
acknowled#ents can easily shi%t into an assessent&
counicatin# with other huans or with E!As ;Reeves and Nass=5>>D6<& E!As are o%ten used as dialo# "artners& They
can be used as "ersonal assistant, web "resenter, co"anion, tutor and so on& As "ointed out in "revious sections, "ersuasion
ay be relevant in each o% these a""lications& 3everal odels o% E!As have been i"leented& ,hile %ew odels have
been elaborated %or "ersuasive E!As, several works have been undertaken to endow E!As with counicative qualities
that allow the to be "ersuasive in #iven situations& In the %ollowin# we "resent a state o% the art o% such odels& As
described above, we have hi#hli#hted : qualities that are i"ortant in "ersuasion, naely #esture and e/"ressive #estures,
"ersuasion and eotion, and %inally "ersuasion and backchannel&,e now "resent the e/istin# odels in relation to these
three qualities&
6.1 ECA persuasion and gesture
Not uch work has been done so %ar in i"leentin# #esture %eature %or "ersuasion in E!A& ,e re"ort works done in
odellin# #esture style and e/"ressive #estures that could be used in "ersuasive strate#ies as described in 3ection :&
Ruttkay and her collea#ues "ro"osed a behaviour re"resentation to enco"ass styles ;Ruttkay et al&=8CC:6<& An E!A is
described over a lar#e set o% diensions ran#in# %ro its culture and "ro%ession to its eotional and "hysical state&
All these diensions a%%ect the way an E!A oves and #esticulates& E$'TE ;!hi et al&=8CCC6< i"leents the Iaban
annotation schee %or dance to chan#e, throu#h a set o% "araeters, the way a #esture looks de"endin# on values such as
the stren#th o% the #esture and its te"o& E$'TE works as a "ost-%ilter a%ter a #esture aniation has been co"uted and
adds e/"ressivity to its %inal aniation& A odel o% behaviour e/"ressivity usin# a set o% si/ "araeters that act as
odulation o% behaviour aniation has been i"leented ;Hartann et al&=8CCA6<& ,hen a""lied at these di%%erent levels,
e/"ressivity ay convey di%%erent %unctions in the discourse conte/t( it can attract the attention ;!ha%ai et al&=8CC?6<,
"ersuade the addressee ;Po##i and Pelachaud=8CC26<, indicate eotional state ;$artin et al&=8CCAb6<& In these studies they
were interested in understandin# and odellin# how eotion as well as "ersuasion can be conveyed qualitatively, in
"articular throu#h #esture e/"ressivity&
8D cartoons were anually annotated to understand how aniators used characters* oveent e/"ressivity to call %or the
attention o% the s"ectators& Two ty"es o% odulation o% behaviour e/"ressivity =irre#ularities and discontinuities6
were %ound to be used by the aniators& They were inte#rated in an E!A syste ;!ha%ai et al&=8CC?6<& An evaluation study
was conducted to see i% these odulations "layed a role in attractin# the user*s attention when conversin# with the a#ent&
The results o% the evaluation con%ir that e/"ressivity s"eci%ied at #esture "hase level ay "lay a s"eci%ic %unction7 naely
here, it can act to attract the attention o% the interlocutor at "recise oents o% the dialo#&
E/"ressivity "araeters were e/tracted either anually ;$artin et al&=8CCAa6, Devillers et al&=8CCA6< or autoatically
;!aridakis et al&=8CCD6< over a whole sequence and "layed back by an a#ent& The "ur"ose o% these studies was to understand
which eleents o% behaviours, in "articular behaviour e/"ressivity, "lay a role %or "erceivin# an eotional state&
%(1(1 Persuasion and bac$channel
In a series o% studies ;Bevacqua et al&=8CC?a6< and ;Heylen et al&=8CC?6<, looked at the "erce"tion o% %acial e/"ressions o%
attitudes that a conversational a#ent dis"layed when listenin#& They looked at a "articular subset o% e/"ressions that i#ht
be relevant to use in "ersuasive dialo#ues, where the a#ent as listener could indicate its understandin# or acce"tance o% the
utterances o% the s"eaker& In a "erce"tion test, subEects were asked to watch the virtual character 1reta ;Bevacqua et al&
=8CC?b6< dis"layin# a cobination o% %acial si#nals and head oveents& The %ollowin# behaviours were "resented in the
5B video cli"s shown in a %irst e/"erient =;Bevacqua et al&=8CC?a6<6(
5& a sin#le head nod =N6
8& a head nod with a sile =N36
:& a head nod and a raise o% the eyebrows =NRE6
B& a head shake
A& a head shake and a %rown =346
D& a head shake, a %rown and a tension in the li"s that ti#hten and #et thinner =34T6
?& a %rown and a tension in the li"s that ti#hten and #et thinner =4T6
2& a raise o% the le%t eyeb rows =RIE6
>& the eyes roll in the head =ER6
PREPRINT
5C& a head tilt to the le%t with sad eyebrows =T3E6
55& a head tilt to the le%t and a %rown =T46
58& a head tilt to the ri#ht and a raisin# o% the eyebrows =TRE6
5:& a head tilt to the ri#ht, #aHe turns down to the ri#ht =T16
5B& eyes wide o"en =E,'6
The dis"lays are "rototy"ically associated with a nuber o% eanin#s in a conversational settin# as listener res"onses that
are either used as conventional si#nals or as e/tensions o% the eanin#s these dis"lays have in, %or instance, eotional
e/"ressions&
The subEects had to associate one or ore eanin#s to these si#nals& They could choose %ro a close list o% eanin#s& In
"articular we have investi#ated how well these e/"ressions atched various classes o% "er%orative dis"lays and e"isteic
and a%%ective states( a#reeent, disa#reeent, acce"tance, reEection, interest and boredo, believe, disbelieve,
understandin# and lack o% understandin#& These are assued to be i"ortant attitudinal reactions in a "ersuasive dialo#ue&
'ne could assue that si#nals containin# nods and siles would be ostly associated with the "ositive states whereas
shakes, %rowns and li" tension would be associated with ne#ative states ;Bassili=5>?>6<& The %ollowin# table shows %or each
behaviour the eanin# that was ost o%ten assi#ned to it& In soe cases there were two eanin#s that were equally o%ten
chosen&
Nod acce"tance
Nod 3ile likin#
Nod Ri#ht Eyebrow a#reeent
Raise Ie%t eyebrow disbelieve
Tilt 3ad Eyebrows not understand
Eye roll disbelieve
Tilt, #aHe down boredo
3hake, %rown, tense li"s disa#reeent9reEection
4rown, tense li"s disa#reeent
Head shake reEection
3hake, %rown disa#reeent, reEection
Tilt, %rown disa#reeent, disbelieve
Tilt ri#ht, raise brows believe ;is this correct<
Eyes wide o"en disbelieve
This study ;Heylen et al&=8CC?6< shows that %or ost attitudes %airly clear "rototy"ical visual e/"ressions can be %ound& A
one-to-any a""in# can be established between a %acial si#nal and a backchannel eanin#& The results o% the study are
in line in e/istin# literature& Nods, %or e/a"le, are ostly used in the conte/ts o% acknowled#eents or a#reeents or as a
"ositive answer substitutin# or coincidin# with the utterance Kyes* or an equivalent& 3hakes, on the other hand, have a
ne#ative eanin# ;.endon=8CC:6<& Head tilts o%ten su##est interest& 3iles occur a lot in conversations& They ostly show
soe "ositive attitude, e/ce"t in cases such as a sarcastic #rin& Eyebrow oveents have a ran#e o% eanin#s& 'ne o% the
eanin#s that Ekan ;Ekan=5>?>6< "oints out is one where it can %unction as an a#reeent res"onse indicatin# that the
listener is attendin# but also understands and does not disa#ree with what is bein# said& Raisin# o% the eyebrows to#ether
with a head nod or an a#reeent word is a ty"ical a#reeent listener res"onse& However, raised eyebrows can also %i#ure in
dis"lays o% sur"rise& Also a %rown can %unction as an e/"ression o% "er"le/ity but is also associated with e/"ressions o%
an#er and dis#ust& The "er"le/ity that is o%ten bein# e/"ressed by the %rown can %unction as a res"onse to the s"eaker
indicatin# lack o% understandin# or indicatin# that one thinks what is bein# said does not ake sense&
6.2 Persuasion and e!otion
In "ersuasion one has to convince with lo#os =rational ar#uentation6, athos =the "ersuader*s credibility and reliability6, and
"athos =the a""eal to the eotions o% the interlocutor6& In both tryin# to be convincin# and in transittin# eotions can one
need to e/hibit a %ake "ersonality and9or %ake eotionsF In huan-huan counication, "eo"le o%ten try to hide their real
eotions because the social situation requires it& Ty"ical e/a"le are e/cuses, such as )I would love to Eoin you, but
&&&+& Eotions are the nuber-one to"ic that "eo"le lie about and studies show that u" to :C@ o% social interaction lon#er
than 5C inutes contain such dece"tions ;DePaulo et al&=5>>D6<& Endowin# technical systes like ebodied conversational
a#ents with the ability to detect, re"resent, #enerate and9or show eotions, it is thus indis"ensable to investi#ate the crucial
questions how to handle %alse eotional e/"ressions %ro the user and how and when to create %alse eotional e/"ressions
in the E!A&
PREPRINT
%(2(1 Persuasion and deception
Narious atte"ts have been ade to create synthetic a#ents that deliberately o""ress or e/"ress a certain eotion& De Rosis
and collea#ues ;deRosis et al&=8CC:6< as well as Prendin#er and collea#ues ;Prendin#er and IshiHuka=8CC56< have develo"ed
a#ents that are able to control their eotional dis"lays i% the social situation requires it& 4or instance, i% the social distance
between an a#ent and its conversational "artner is hi#h, Prendin#er*s a#ent would not show an#er to the %ull e/tent&
The virtual tutor !'3$' develo"ed by Iester and collea#ues ;Iester et al&=8CCC6< intentionally "ortrays eotions with the
#oal o% otivatin# students and thus increasin# the learnin# e%%ect&
Earlier a""roaches start %ro the assu"tion that the a#ent is able to "er%ectly hide eotions i% the social or "eda#o#ical
situation requires it& However, huans are not always ca"able o% co"letely concealin# their true eotions& 4or instance,
askin# siles cannot entirely override the uscular "ro#ra o% the ori#inal eotion because not every %acial uscle can
be consciously controlled& As a consequence, such a ask will always include se#ents o% one*s %elt eotion& The question
arises o% how to handle situations in which the a#ent decides to dis"lay an eotion which is in con%lict with its internal
a""raisal "rocesses& In soe situations, it i#ht be desirable to e"loy a#ents that "er%ectly convey )wron#+ eotions with
the ai to convince the interlocutor& !onsider a sales a#ent on the web that has to advertise a "roduct o% inor quality& I% it
does not succeed in concealin# its ne#ative attitude towards the "roduct, a decrease o% the sales i#ht be the consequence&
'n the other hand, a#ents in social settin#s ay coe across as little believable or cold i% they are always able to "er%ectly
hide their true eotions& In addition, the dis"lay o% i/ed eotions ay even lead to a "ositive res"onse %ro the
interlocutor& 4or instance, students ay %eel sy"athy towards a virtual teacher that des"erately tries to hide its ne#ative
eotions caused by the students* bad "er%orance& Iast but not least, the eulation o% dece"tive behaviours ay enrich our
interactions with synthetic a#ents - es"ecially in #ae-like environents&
4ollowin# these considerations, Reh and AndrOe ;Reh and AndrOe=8CCAa6< %ocused on synthetic a#ents that ay e/"ress
eotions that are in con%lict with their a""raisal "rocesses& 0nlike earlier work, they odelled situations in which the a#ent
%ails to entirely conceal its )%elt+ eotions& They develo"ed an a#ent whose behaviours ay re%lect "otential con%licts
between )%elt+ and deliberately e/"ressed eotions& Their work concentrated on %acial e/"ressions o% dece"tion which have
been "ro%oundly researched in the "sycholo#ical literature& Accordin# to Ekan and collea#ues ;Ekan et al&=5>226Ekan,
4riesen, and '*3ullivan<, there are at least %our ways in which %acial e/"ressions ay vary i% they acco"any lies and
dece"tions( icro-e/"ressions, asks, tiin#, and asyetry&
5& $icro-e/"ressions( A %alse eotion is dis"layed, but the %elt eotion is unconsciously e/"ressed %or the %raction o% a
second& The detection o% such icroe/"ressions is "ossible %or a trained observer&
8& $asks( The %elt eotion =e&#&, dis#ust6 is asked by a non- corres"ondin# %acial e/"ression, in #eneral by a sile&
Because we are not able to control all o% our %acial uscles, such a askin# sile is in soe way de%icient& Thus, it reveals
at least in "art the ori#inal eotion&
:& Tiin#( 4acial e/"ressions acco"anyin# true eotions do not last %or a very lon# tie& Thus, the lon#er an e/"ression
lasts the ore likely it is that it is acco"anyin# a lie& A s"ecial case sees to be sur"rise, where elon#ated onand
o%%set ties are a #ood indicator o% a %alse eotion&
B& Asyetry( Noluntarily created %acial e/"ressions like they occur durin# lyin# and deceivin# tend to be dis"layed in an
asyetrical way, i&e&, there is ore activity on one side o% the %ace than on the other&
To odel the non-verbal behaviour, they e"loyed the 1reta a#ent syste develo"ed by Pelachaud and collea#ues
;Bevacqua et al&=8CC?b6<& 4or e/a"le, 4i#& 5 shows the %acial dis"lays %or #enuine Eoy, #enuine dis#ust and dis#ust %aked
by Eoy& In the case o% natural Eoy =le%t screenshot6, the corners o% the outh and cheeks are oved u" syetrically& In the
case o% natural dis#ust =iddle screenshot6, the eye brows are contracted and the u""er li" is oved u"& To %ake dis#ust by
Eoy =ri#ht screenshot6, the eye oveents o% dis#ust and the outh oveents o% Eoy have been cobined& Di%%erent
de#rees o% askin# are cobined with di%%erent de#rees o% asyetry o% the %acial dis"lays resultin# in :8 "ossible %acial
e/"ressions&
)ig( 1 1reta showin# #enuine Eoy =le%t6, #enuine dis#ust =iddle6 and dis#ust %aked by Eoy =ri#ht6
PREPRINT
A co"utational odel o% co"le/ %acial e/"ression was develo"ed by NiewiadoskiM Pelachaud ;Niewiadoski and
Pelachaud=8CC?a6, Niewiadoski and Pelachaud=8CC?b6<& It is also based on Ekan*s work ;Ekan=5>?A6, Ekan=8CC:6<&
!o"le/ %acial e/"ressions are obtained usin# a %ace "artitionin# a""roach where %acial e/"ression is de%ined by a set o%
ei#ht %acial areas( %orehead9eyebrow, u""er-eyelid, eyes, lowereyelid, cheek, nose, li" =oveent6, li" tension& In "articular,
in a case o% co"le/ %acial e/"ressions di%%erent eotions can be e/"ressed on di%%erent areas o% the %ace& The co"le/
%acial e/"ressions are co"osed o% the %acial areas o% in"ut e/"ressions usin# a set o% %uHHy rules& Di%%erent rules have been
i"leented %or the su"er"osition o% two %elt eotions =see 4i#ure 86 and %or the askin# o% a %elt eotion by a
%ake =i&e& non-%elt6 eotion =see 4i#ure :6& The rules ake use o% the results showin# that e/"ressions o% %elt eotions are
si#nalled by characteristic =also called reliable6 %eatures =such as the crow %eet %or %elt ha""iness6 ;Ekan=5>?A6,
Ekan=8CC:6< and that "ositive eotions are ainly "erceived %ro the lower "art o% the %ace =sile %or ha""iness6 and
ne#ative eotions %ro the u""er %ace =such as %rown %or an#er6 ;Bassili=5>?>6<&
An e/a"le o% such a rule %or the su"er"osition o% two %elt eotions =sadness and an#er6 is( the ore one o% the in"ut
e/"ressions is =siilar to6 an#er and the other in"ut e/"ression is =siilar to6 sadness, the ore certain is that the %inal
e/"ression contains brows, u""er eyelids, eyes, and the lower eyelids o% the %irst e/"ression and the outh area rest o% the
second =see 4i#ure 86& The out"ut o% this odule is a %acial e/"ression co"osed o% "arts o% the %acial e/"ression o% one
eotion and "arts o% the %acial e/"ression o% the other eotion& They re%er to this %inal e/"ression as a co"le/ e/"ression
as it is co"osed o% two e/"ressions and thus shows two eotions&
)ig( 2 3u"er"osition o% an#er and sadness& 4ro the le%t to ri#ht( an#er a6, sadness b6, su"er"osition
o% an#er and sadness c6 su"er"osition o% an#er and sadness with si#ni%icant areas arked d6&
4i#ure : "resents the a#ent dis"layin# an e/"ression o% sadness, that is asked by a %ake ha""iness& Accordin# to these rules
%acial areas o% %orehead9eyebrows, and u""er eyelids cover the %eatures o% %elt sadness =red circles in 4i#ure :d6 that leak
over the ask o% a Eoy =yellow circles in 4i#ure :d6& As a consequence, they can be observed in a %inal e/"ression&
To %ind out how dece"tive clues are subEectively "erceived by a huan user and to what e/tent users are able to correctly
inter"ret the, AndrOe and Reh ;Reh and AndrOe=8CCAa6< conducted two e/"erients& In the %irst e/"erient, a
dece"tive
PREPRINT
)ig( 3 3adness asked by ha""iness& 4ro the le%t to ri#ht( sadness a6, ha""iness b6, sadness
asked by ha""iness c6 sadness asked by ha""iness with si#ni%icant areas arked d6&
and a non-dece"tive version o% the 1reta a#ent "resented ovie reviews to the users& That is user did not interact with the
a#ent and in%oration was "rovided by the a#ent in ters o% a onolo#ue& The study revealed that the non-deceivin#
a#ent is "erceived as bein# ore reliable, trustable, convincin#, credible, and ore certain about what it said& Althou#h
"eo"le reacted to %acial clues o% deceit when they had the o""ortunity to care%ully watch and co"are di%%erent instances o%
a#ents, they were not able to nae the reasons %or these reactions& In the second e/"erient, the a#ent was servin# as a
#ae "artner in a #ae o% dice& To win the #ae, "layers had to lie to the other "layers and to catch liars& In the #ae, the
a#ent tried to islead the other "layers by "ortrayin# %acial e/"ressions that did not corres"ond to her actual eotional
state& 4or instance, she e/"ressed %alse Eoy to ake her #ae "artners believe that she achieved a hi#h score& Nevertheless,
the a#ent did not lie in a "er%ect anner, but still revealed her dece"tive behaviours by subtle %acial cues& The %irst study
indicated that even subtle e/"ressions o% dece"tion ay have an un%avourable i"act on the user*s "erce"tion o% the
a#ent& The results o% the %irst study could, however, not be con%ired %or the second scenario in which the e/"eriental
conditions were uch less controlled& $ost likely the "layers were too uch en#a#ed in the #ae to "ay attention to the
a#ent*s dece"tive cues& In a ore natural and en#a#in# %ace-to-%ace situation, subEects tend to disre#ard dece"tive clues&
%(2(2 Persuasion and politeness
I% you want to "ersuade soeone throu#h inducin# "ositive eotions in hi9her a #ood way to do so ay be "oliteness&
,hen huans interact with each other, they risk continuously threatenin# the %ace o% their conversational "artners, %or
e/a"le by showin# disa""roval or by "uttin# the other "erson under "ressure& To iti#ate such %ace threats, huans
usually rely on "oliteness tactics& 4or instance, instead o% %orulatin# a direct request )3olve the equation&+, a teacher i#ht
su##est ),hy don*t we solve the equation to#ether&+&
In their seinal work, Brown and Ievinson ;Brown and Ievinson=5>2?6< analyHe verbal strate#ies o% "oliteness& Positive
"oliteness seeks a""roval o% the addressee, %or e/a"le, by akin# hi a co"lient be%ore a criticis is delivered&
Ne#ative "oliteness e"hasiHes the hearer*s %reedo o% action, %or e/a"le, by %orulatin# a su##estion instead o% a
request& '%%-record stateents are va#ue and the addressee has to in%er the e/act eanin# o% the s"eaker leavin# hi in the
"osition to isunderstand the s"eaker and thus to not %eel o%%ended& Previous work has concentrated %or the ost "art on
verbal eans to iti#ate %ace threats& An e/ce"tion is an e"irical study by Trees and $anusov ;Trees and $anusov=5>>26<
who %ound that non-verbal behaviours, such as "leasant %acial e/"ressions and ore direct body orientation ay hel" to
iti#ate %ace threats evoked by criticis& Bavelas et al& ;Bavelas et al&=5>>A6< "rovide a classi%ication o% #estures soe o%
which can be directly a""ed onto Brown and Ievinson*s strate#ies o% "oliteness& 3hared in%oration #estures ark
aterial that is "art o% the interlocutor*s coon #round& !itin# #estures re%er to "revious contributions o% the addressee
and ai at conveyin# the i"ression that the interlocutors share a coon o"inion& Elli"tical #estures ark inco"lete
in%oration that the addressee should au#ent %or hi- or hersel% and ay take on a siilar %unction as o%% record
PREPRINT
strate#ies& 3eekin# a#reeent #estures directly corres"ond to Brown and Ievinson*s a""roval oriented strate#ies&
Turn o"en #estures can be re#arded as atte"ts to satis%y the addressee*s desire %or autonoy& Iin#uistic eans to deliver
%ace threatenin# acts have "artly becoe "art o% the #raar and Bavelas classi%ication o% #estures su##ests that there
i#ht be siilar "rinci"led and standardiHed connection between non-verbal eans o% counication and "oliteness
strate#ies&
To shed li#ht on the question o% how %ace threats are iti#ated by non-verbal eans, Reh and AndrOe ;Reh and Andr
Oe=8CCAb6< conducted a cor"us study with huan s"eakers& This ultiodal cor"us consists o% sta#ed interactions o%
inherently %ace threatenin# situations& They devised a scenario that %orced the "artici"ants to use their =unconscious6
knowled#e o% "oliteness strate#ies by con%rontin# the with an inherently %ace threatenin# situation& To code "oliteness
strate#ies, they %ollowed ,alker et al&*s ;,alker et al&=5>>?6< cate#oriHation o% direct, a""roval oriented, autonoy oriented,
and o%%-record strate#ies& The results o% their cor"us analysis indicate that #estures are indeed used to stren#then the e%%ect
o% verbal acts o% "oliteness ;Reh and AndrOe=8CCAb6<& In "articular, va#ueness as a eans o% "oliteness is not only
re%lected by verbal utterances, but also by #estures& Iconic and deictic #estures were overwhelin#ly used in ore direct
criticis while there was a hi#h %requency o% eta"horic #estures in o%% record strate#ies& 'bviously, the subEects did not
atte"t at co"ensatin# %or the va#ueness o% their s"eech by usin# ore concrete #estures&
,alker et al& ;,alker et al&=5>>?6< have "resented one o% the %irst a""roaches to i"leent "oliteness strate#ies based on the
theory by Brown and Ievinson ;Brown and Ievinson=5>2?6< as a eans to ore %le/ible dialo#ue control& They
describe a selection echanis that is based on the variables "ower, social distance, and rankin# o% s"eech act& Johnson et
al& ;Johnson et al&=8CCB6< investi#ated the "otential bene%its o% "oliteness in a tutorin# syste& E/ainin# the interactions
between a real tutor and his students, they cae u" with a set o% te"lates each o% which is annotated accordin# to the
aount o% redress that tactic #ives to the learner*s %ace& Reh and AndrOe ;Reh and AndrOe=8CCAb6< equi""ed the 1reta
a#ent with "oliteness behaviours based on the cor"us study "resented above& Niewiadoski M Pelachaud ;Niewiadoski
and Pelachaud=8CC?b6< analysed the sae video-cor"us o% Reh and AndrOe ;Reh and AndrOe=8CCAb6< in order to %ind
relations between "oliteness strate#ies and %acial behaviour& They considered %our ty"es o% %acial dis"lays( e/"ression o% the
true eotional state, inhibited, asked, and %ake e/"ression& They analysed the %requency o% occurrence %or each o% the
and %ound that di%%erent ty"es o% %acial e/"ressions were not evenly distributed alon# the di%%erent strate#ies o% "oliteness&
They used this in%oration to build their odel o% %acial dece"tive behaviour ana#eent in inter"ersonal relations& In their
odel, they considered three variables to enco"ass the characteristics o% the interaction ;Brown and Ievinson=5>2?6< and
the eotional state o% the dis"layer, naely( social distance, social "ower, and eotion valence& Based on the value o% these
variables they de%ined which %acial ana#eent is the ost a""ro"riate, e# when a %elt eotion should be asked or
inhibited&
* +ecommendations to build a persuasive E'A
To conclude this cha"ter, we "rovide recoendations %or nonverbal behaviours to be considered when constructin# a
"ersuasive E!A&
".1 #a$e behaviour
Eye contact is a "ower%ul tool %or "ersuasive "ur"oses& It can be used to acknowled#e "resence, show interest, "rovide
%eelin#s o% i"ortance and hel" convey an attitude that the Persuader likes the Persuadee, soethin# also in%luenced
by lar#e "u"il siHe& In courtroo situations, akin# eye contact with Eurors has been re%erred to as bein# the ost i"ortant
ty"e o% counication "ossible ;Bernstein et al&=5>>B6<& 3"eakers who en#a#e in eye-contact are "erceived as bein#
ore credible ;Hesley and Doob=5>?26< than those who continually avert their #aHe, the latter a""earin# untrustworthy or
uncon%ident& However, it is i"ortant to establish a balance between a""earin# con%ident by en#a#in# in su%%icient eye
contact, while at the sae tie avoidin# to be "erceived as over"owerin# or threatenin# by aintainin# eye contact %or too
lon#& This is a di%%icult "ro"osition in "ractice, as it ust account %or any %actors related to the conte/t o% the interaction,
the Persuader and the Persuadee, such as culture and #ender& At a hi#her-level, #aHe strate#ies such as lo#os #aHe,
co"etence and benevolence strate#ies =see 3ection 86 can be ado"ted to su""ort "ersuasion throu#h rational ar#uents,
establishin# trust and invokin# eotions ;Po##i and NincHe=8CC26<& In these situations, a key %actor should be to ensure
con#ruence between #aHe and other odalities in order to "rovide coherent "ersuasive behaviour consistent with a sin#le
strate#y& The behavioural realis o% the #aHe behaviour is also a %actor %or consideration, as virtual huans hi#her in
behavioural realis have been %ound to be ore in%luential ;1uada#no et al&=8CC?6<&
".2 %acial E&pressions
4acial e/"ressions convey a lar#e aount o% nonverbal in%oration( a sile or a %rown ade at a certain oent can s"eak
PREPRINT
volues& In #eneral, %acial e/"ressions should a""ear to be rela/ed in order to hel" convey a con%ident but non-threatenin#
deeanour( tension should not be "erce"tible in the brows, Eaw, outh or shoulders so that the "ersuader a""ears to be in
control o% the situation at all ties&
".3 'ho(ing Attention and )nterest
Related to eye contact and %acial e/"ression is the i"ortance o% showin# interest in what the other has to say& 4or e/a"le,
as described in ;Ievine=8CCD6<, based on his e/"eriences workin# on the docuentary Thin Blue Line, interviewer Errol
$orris hi#hli#hted the i"ortance o% a""earin# to be en#rossed and show interest in what his #uests had to say at all costs&
In soe situations, $orris even su##ested not actually becoin# too en#rossed in the story, so one could concentrate totally
on "rovidin# si#nals o% interest, which he re#arded as bein# o% the utost i"ortance %or elicitin# disclosure o% %acts %ro
interviewees&
".* +ands and ar!s
1reat s"eakers use hand #estures ore than on avera#e and #estures can #ive the listener con%idence in the s"eaker as well
as hel" the s"eaker aintain their turn& It is "articularly i"ortant that hand #estures kee" out o% the "ersonal s"ace o% the
other and es"ecially away %ro their %ace& The ars should rein%orce the i"ression o% o"enness on the "art o% the a#ent by
bein# uncrossed& As with %acial e/"ressions, ar oveents should not a""ear to be tense and should be o"en and
e/"ansive&
Huans have a natural tendency to "rotect vulnerable or#ans es"ecially in threatenin# or stress%ul situations - not doin# so
can ake us %eel e/"osed and vulnerable,but can "roEect a coura#eous "osture, re%lectin# sel%-con%idence& 4urtherore,
hands should ideally be visible rather than "laced in "ockets( '"en "als are a #ood way to e/"ress honesty and
trustworthiness& 3el%-touchin# behaviours are to be avoided, ainly because they ay convey a need %or sel%-reassurance&
"., #eneral posture and barriers
As with the other %actors, it is i"ortant to "roEect an o"en, honest, coo"erative "osture, as this can be "erceived to re%lect a
"sycholo#ical o"enness& In contrast, a "oor, de%lated "osture is associated with a lack o% con%idence or a lack o% interest& The
orientation o% the body is o% i"ortance here too( as the ter Kcold shoulder* su##ests, when we %eel unco%ortable with a
"erson o% situation, we ay tend to orientate our body sideways, showin# aversion o% tiidity& 'rientin# towards the
audience, and not allowin# obstructions between the and the "ersuader, is o% i"ortance in establishin# a direct
connection with the& Ars, obstacles and instruents ay be used to create barriers, severin# this connection and
"ortrayin# a de%ensive "osture - thus, ars should not be "laced at rest in %ront o% the abdoen or chest&
+eferences
;Allwood et al&=5>>:6Allwood, Nivre, and AhlsOen< J& Allwood, J& Nivre, and E& AhlsOen& 'n the seantics and "ra#atics
o% lin#uistic %eedback& Semantics, >=56, 5>>:&
;Atkinson=5>2B6< $& Atkinson& Our Masters Voices. The Language and Body Language o !olitics. Iondon, $ethuen,
5>2B&
;Bassili=5>?>6< J&N& Bassili& Eotion reco#nition( the role o% %acial oveent and the relative i"ortance o% u""er and
lower areas o% the %ace& "ournal o !ersonality and Social !sychology, :?=556(8CB>R8CA2, 5>?>&
;Bavelas et al&=5>>A6< J&B& Bavelas, N& !hovil, I& !oates, and I& Roe& 1estures s"ecialiHed %or dialo#ue& !ersonality and
Social !sychology Bulletin, 85=B6( :>BRBCA, 5>>A&
;Bernstein et al&=5>>B6< I&E& Bernstein, $&E& Deorest, and 3&P& Eberhardt& A co"utational a""roach to analyHin#
sentential s"eech "erce"tion( Phonee-to"honee stiulus-res"onse ali#nent& "ournal o the Acoustical Society o
America, 5>>B&
;Bevacqua et al&=8CC?a6< E& Bevacqua, D& Heylen, !& Pelachaud, and $& Tellier& 4acial %eedback si#nals %or ecas& In #n
!roceedings o A#SB$%& Artiicial and Ambient #ntelligence, Newcastle 0niversity, Newcastle u"on Tyne, 0., A"ril
8CC?a&
PREPRINT
;Bevacqua et al&=8CC?b6< E& Bevacqua, $& $ancini, R& Niewiadoski, and !& Pelachaud& An e/"ressive eca showin#
co"le/ eotions& In !roceedings o the A#SB Annual Con'ention, "a#es 8C2R85D, Newcastle, 0., 8CC?b&
;Brown and Ievinson=5>2?6< B& Brown and 3&!& Ievinson& !oliteness& some uni'ersals on language usage& !abrid#e
0niversity Press, 5>2?&
;Bucy and Bradley=8CCB6< E&P& Bucy and 3&D& Bradley& Presidential e/"ressions and viewer eotion( !ountere"athic
res"onses to televised leader dis"lays& Social Science #normation, B:(A>R>B, 8CCB&
;Bull=5>2D6< P&E& Bull& The use o% hand #esture in "olitical s"eechers( 3oe case studies& "ournal o Language and Social
!sychology, A(5C8R552, 5>2D&
;Bur#oon et al&=5>>C6< J&.& Bur#oon, T& Birk, and $& P%au& Nonverbal behaviors, "ersuasion, and credibility& (uman
Communication )esearch, 5?(5BCR5D>,
5>>C&
;!albris=8CC:6< 1enevive !albris& L*+,ression gestuelle de la ,ense dun homme ,oliti-ue& !NR3 !ounication, 8CC:&
;!aridakis et al&=8CCD6< 1& !aridakis, A& RaouHaiou, .& .ar"ouHis, and 3& .ollias& 3ynthesiHin# #esture e/"ressivity based
on real sequences& In .orksho, on multimodal cor,ora& rom multimodal beha'iour theories to usable models/
L)*C 0$$1 Conerence, 1enoa, Italy, 8B-8D $ay 8CCD&
;!arli et al&=5>>A6< I&I& !arli, 3&I& Ia4leur, and !&!& Ioeber& Nonverbal behavior, #ender, and in%luence& "ournal o
!ersonality and Social !sychology, D2(5C:CR5CB5, 5>>A&
;!astel%ranchi=8CCC6< !& !astel%ranchi& A%%ective a""raisal versus co#nitive evaluation in social eotions and interactions&
In A&$& Paiva, editor, Aecti'e #nteractions& To2ards a 3e2 4eneration o Com,uter #nteraces, "a#es ?DR5CD& 3"rin#er-
Nerla#, 8CCC&
;!ha%ai et al&=8CC?6< N&E& !ha%ai, !& Pelachaud, and D& PelOe& A case study o% #esture e/"ressivity breaks& #nternational
"ournal o Language )esources and *'aluation/ S,ecial issue on Multimodal Cor,ora or Modelling (uman Multimodal
Beha'ior, 8CC?&
;!hi et al&=8CCC6< D&$& !hi, $& !osta, I& Shao, and N&I& Badler& The E$'TE odel %or e%%ort and sha"e& In .urt Akeley,
editor, Siggra,h 0$$$/ Com,uter 4ra,hics !roceedings, "a#es 5?:R528& A!$ Press 9 A!$ 3I11RAPH 9 Addison ,esley
Ion#an, 8CCC&&
;Darwin=52?86< !& Darwin& The e+,ression o the emotions in man and animals& John $urray, 52?8&
;DePaulo et al&=5>>D6< Bella $& DePaulo, Deborah A& .ashy, 3usan E& .irkendol, $elissa $& ,yer, and Jenni%er A&
E"stein& Iyin# in Everyday Iive& "ournal o !ersonality and Social !sychology, ?C=A6(>?>R>>A, 5>>D&
;deRosis et al&=8CC:6< 4& deRosis, !& Pelachaud, I& Po##i, N& !aro%i#lio, and B& De!arolis& 4ro 1reta*s ind to her %ace(
$odellin# the dynaics o% a%%ective states in a conversational ebodied a#ent& #nternational "ournal o (uman5
Com,uter Studies, A>=5-86(25R552, 8CC:&
;Devillers et al&=8CCA6< I& Devillers, 3& Abrilian, and J&-!& $artin& Re"resentin# real li%e eotions in audiovisual data with
non basic eotional "atterns and conte/t %eatures& In 6irst #nternational Conerence on Aecti'e Com,uting and #ntelligent
#nteraction 7AC##0$$89, "a#es A5>RA8D, BeiEin#, !hina, 8CCA&
;Ekan=5>?A6< P& Ekan& Unmasking the 6ace. A guide to recogni:ing emotions rom acial clues& Prentice-Hall, Inc&,
En#lewood !li%%s, New Jersey, 5>?A&
;Ekan=5>?>6< P& Ekan& About bro2s& emotional and Con'ersational signals, "a#es 5D>R8C8& !abrid#e 0niversity
Press9Editions de la $aison des 3ciences de l*Hoe, 5>?>&
PREPRINT
;Ekan=8CC:6< P& Ekan& The 6ace )e'ealed& ,eiden%eld M Nicolson, Iondon, 8CC:&
;Ekan et al&=5>226Ekan, 4riesen, and '*3ullivan< P& Ekan, ,& 4riesen, and $& '*3ullivan& 3iles when lyin#& "ournal
o !ersonality and Social !sychology, AB=:6, 5>22&
;4rey=8CCC6< 3& 4rey& ;ie Macht des Bildes. ;er *inluss der non'erbalen <ommunikation au <ultur und !olitik& Bern,
Huber, 8CCC&
;4riEda=5>2D6< N& 4riEda& The *motions& !abrid#e 0niversity Press, 5>2D&
;1uada#no et al&=8CC?6< R&E& 1uada#no, J& Blascovich, J&N& Bailenson, and !& $c-!all& Nirtual huans and "ersuasion( The
e%%ects o% a#ency and behavioral realis& Media !sychology, 5C(5R88, 8CC?&
;Hartann et al&=8CC86< B& Hartann, $& $ancini, and !& Pelachaud& 4orational "araeters and ada"tive "rototy"e
instantiation %or $PE1-B co"liant #esture synthesis& In Com,uter Animation$0, 1eneva, 3witHerland, 8CC8& IEEE
!o"uter 3ociety Press&
;Hartann et al&=8CCA6< B& Hartann, $& $ancini, and !& Pelachaud& I"leentin# e/"ressive #esture synthesis %or
ebodied conversational a#ents& In 4esture in (uman5Com,uter #nteraction and Simulation/ 1th #nternational 4esture
.orksho,/ 4. 0$$8/ Berder #sland, "a#es 522R5>>, 8CCA&
;Hesley and Doob=5>?26< 1&D& Hesley and A&N& Doob& The e%%ect o% lookin# behaviour on "erce"tions o% a
counicator*s credibility& "ournal o A,,lied Social !sychology, 2(5:DR5BB, 5>?2&
;Henley=5>??6< N&$& Henley& Body ,olitics& !o2er/ se+/ and non'erbal beha'ior& En#lewood !li%%s, NJ( Prentice-Hall,
5>??&
;Heylen et al&=8CC?6< D& Heylen, E& Bevacqua, $& Tellier, and !& Pelachaud& 3earchin# %or "rototy"ical %acial %eedback
si#nals& In !& Pelachaud, J&-!& $artin, E& AndrOe, 1Oerard !hollet, .ostas .ar"ouHis, and Danielle PelOe, editors,
!roceedings o the %th #nternational Conerence on #ntelligent Virtual Agents 7#VA9, volue B?88 o% Lecture 3otes in
Com,uter Science, "a#es 5B?R5A:& 3"rin#er, 8CC?&
;Heylen and o" den Akker=8CC?6< Dirk Heylen and Rieks o" den Akker& !o"utin# backchannel distributions in ulti-"arty
conversations& In Justine !assell and Dirk Heylen, editors, !roceedings o the ACL .orksho, on *mbodied Language
!rocessing, volue ,C?-5>, "a#es 5?R8B, Pra#ue, !Hech Re"ublic, 8CC?& Association o% !o"utational Iin#uistics&
;I=8CCB6< Po##i I& Eotions %ro ind to ind& In !roceedings o the .orksho, .=>& *m,athic Agents. Third
#nternational Conerence on Autonomous Agent Systems/ AAMAS 0$$?, "a#es 8C2R85D, New Lork, 8CCB&
;Jaies et al&=8CC?6< AleEandro Jaies, HervOe Bourlard, 3teve Renals, and Jean !arletta& Recordin#, inde/in#,
suariHin#, and accessin# eetin# videos( An overview o% the ai "roEect& In 6ourth #nternational Conerence on #mage
Analysis and !rocessing 7#C#A! 0$$%9 and .orksho, on Visual and Multimedia ;igital Libraries 7VM;L 0$$%9, "a#es A>R
DB, 8CC?&
;Johnson et al&=8CCB6< ,&I& Johnson, P& RiHHo,,& Bosa, 3& .ole, $& 1hiEsen, and H& van ,elber#en& 1eneratin# socially
a""ro"riate tutorial dialo#& In #SCA .orksho, on Aecti'e ;ialogue Systems, "a#es 8ABR8DB, 8CCB&
;.endon=5>D?6< A& .endon& 3oe %unctions o% #aHe direction in social interaction& Acta !sychologica, 8D(88RD:, 5>D?&
;.endon=8CC:6< A& .endon& 3oe uses o% head shake& 4esture, 8(5B?R528, 8CC:&
;.endon=8CCB6< A& .endon& 4esture & Visible Action as Utterance& !abrid#e 0niversity Press, 8CCB&
;Iester et al&=8CCC6< J&!& Iester, 3&1& Towns, !&B& !allaway, J&I& Noeran, and P&J& 4itH1erald& Deictic and eotive
counication in aniated "eda#o#ical a#ents& In 3& Prevost J& !assell, J& 3ullivan and E& !hurchill, editors, *mbodied
Con'ersational Characters, "a#es 58:R5AB& $IT"ress, !abrid#e, $A, 8CCC&
PREPRINT
;Ievine=8CCD6< R& Ievine& The !o2er o !ersuasion, cha"ter ,ho do we trustF E/"erts, Honesty and Iikability& ,iley,
8CCD&
;$artin et al&=8CCAa6< J& $artin, 3& Abrilian, and I& Devillers& Annotatin# ultiodal behaviors occurrin# durin# non basic
eotions& In !roceedings o the irst #nternational Conerence on Aecti'e Com,uting and #ntelligent #nteraction 7AC##9,
Pkin, !hine, 8CCAa&
;$artin et al&=8CCAb6< J&-!& $artin, 3& Abrilian, and I& Devillers& Annotatin# ultiodal behaviors occurrin# durin# non
basic eotions& In =st #nternational Conerence on Aecti'e Com,uting @ #ntelligent #nteraction 7AC##0$$89, BeiEin#,
!hina, 'ctober 88-8B 8CCAb&
;$ehrabian=5>?56< A& $ehrabian& Silent Messages& ,adsworth Publishin# !o"any, 5>?5&
;Niewiadoski and Pelachaud=8CC?a6< R& Niewiadoski and !& Pelachaud& 4uHHy siilarity o% %acial e/"ressions o%
ebodied a#ents& In !& Pelachaud, J&-!& $artin, E& AndrOe, 1Oerard !hollet, .ostas .ar"ouHis, and Danielle PelOe, editors,
!roceedings o the %th #nternational Conerence on #ntelligent Virtual Agents 7#VA9, volue B?88 o% Lecture 3otes in
Com,uter Science, "a#es 2DR>2& 3"rin#er, 8CC?a&
;Niewiadoski and Pelachaud=8CC?b6< R& Niewiadoski and !& Pelachaud& $odel o% %acial e/"ressions ana#eent %or an
ebodied conversational a#ent& In A& Paiva, R& Prada, and R&,& Picard, editors, Second #nternational Conerence on
Aecti'e Com,uting and #ntelligent #nteraction 7AC##9, volue B?:2 o% Lecture 3otes in Com,uter Science, "a#es 58R8:&
3"rin#er, 8CC?b&
;Pease and Pease=8CCD6< Barbara Pease and Allan Pease& The ;einiti'e Book o Body Language& Banta Books, 8CCD&
;Po##i=8CCA6< I& Po##i& The #oals o% "ersuasion& !ragmatics and Cognition, 5:( 8>2R::A, 8CCA&
;Po##i and Pelachaud=8CC26< I& Po##i and !& Pelachaud& Persuasion and the e/"ressivity o% #estures in huans and
achines& In I& ,achsuth, $& IenHen, and 1& .noblich, editors, *mbodied Communication in (umans and Machines,
"a#es :>5RB8B& '/%ord, '/%ord 0niversity Press, 8CC2&
;Po##i and NincHe=8CC26< Isabella Po##i and Iaura NincHe& Persuasive #aHe in "olitical discourse& In !roceedings o the
A#SB Annual Con'ention, Aberdeen, 3cotland, 0., 8CC2&
;Prendin#er and IshiHuka=8CC56< H& Prendin#er and $& IshiHuka& 3ocial role awareness in aniated a#ents& In !roceedings
o the ith international conerence on Autonomous agents, "a#es 8?CR8??, $ontreal, Guebec, !anada, 8CC5&
;Reeves and Nass=5>>D6< B& Reeves and !& Nass& The media e-uation& (o2 ,eo,le treat com,uters/ tele'ision/ and ne2
media like real ,eo,le and ,laces& !abrid#e 0niversity Press, 5>>D&
;Reh and AndrOe=8CCAa6< $& Reh and E& AndrOe& !atch e i% you can R e/"lorin# lyin# a#ents in social settin#s& In
4rank Di#nu, Nir#inia Di#nu, 3ven .oeni#, 3arit .raus, $unindar P& 3in#h, and $ichael ,ooldrid#e, editors,
the !roceedings o #nternational Aoint Conerence on Autonomous Agents and Multi5Agent Systems 7AAMAS9, "a#es >:?R
>BB, 0trecht, The Netherlands, 8CCAa& A!$&
;Reh and AndrOe=8CCAb6< $& Reh and E& AndrOe& In%orin# the desi#n o% ebodied conversational a#ents by analysin#
ultiodal "oliteness behaviors in huan-huan counication& In .orksho, on Con'ersational #normatics
or Su,,orting Social #ntelligence and #nteraction, 8CCAb&
;Ruttkay et al&=8CC:6< S& Ruttkay, N& van $o""es, and H& Noot& The Eovial, the reserved and the robot& In ,roceedings o the
AAMAS$> .s on *mbodied Con'ersational Characters as #ndi'iduals, $elbourne, Australia, 8CC:&
;3eiter et al&=8CCD6< J&3& 3eiter, H&J& .inHer, and H&,e#er Jr& Back#round behavior in live debates( the e%%ects o% the i"licit
ad hoine %allacy& Communication )e,orts, 5>=56(A?, A"ril 8CCD&
PREPRINT
;3erenari=8CC:6< $& 3erenari& E/a"les %ro the berlin dictionary o% everyday #estures& In I Po##i $ Rector and Ntri#o,
editors, 4estures. Meaning and Use& Edicoes 0niversidade 4ernando Pessoa, 8CC:&
;3treeck=8CC26< J& 3treeck& A case study o% the deocratic "residential candidates durin# the 8CCB "riary ca"ai#n&
)esearch on Language and Social #nteraction, B5(5ABR52D, 8CC2&
;Trees and $anusov=5>>26< A&R& Trees and N& $anusov& $ana#in# %ace concerns in criticis& (uman Communication
)esearch, 8B=B6(ADBRA2:, 5>>2&
;van Eeeren and 1rootendorst=5>2B6< 4&H& van Eeeren and R& 1rootendorst& S,eech Acts in Argumentati'e ;iscussions&
,alter de 1ruyter and 4oris, Berlin and Dordrecht, 5>2B&
;,alker et al&=5>>?6< $&A& ,alker, J&E& !ahn, and 3&J& ,hittaker& I"rovisin# lin#uistic style( 3ocial and a%%ective bases %or
a#ent "ersonality& In 6irst international conerence on Autonomous agents, "a#es >DR5CA, $arina del Rey, !ali%ornia,
0nited 3tates, 5>>?&
;Ln#ve=5>?C6< N&H& Ln#ve& 'n #ettin# a word in ed#ewise& In !a,ers rom the si+th regional meeting o the Chicago
Linguistic Society, "a#es AD?R??, !hica#o( !hica#o Iin#uistic 3ociety, 5>?C&