Accepted Manuscript

Title: From gaze cueing to dual eye-tracking: Novel

approaches to investigate the neural correlates of gaze in
social interaction

Author: Ulrich J. Pfeiffer Kai Vogeley Leonhard Schilbach

PII: S0149-7634(13)00188-7
DOI: http://dx.doi.org/doi:10.1016/j.neubiorev.2013.07.017
Reference: NBR 1815

To appear in:

Received date: 10-4-2013

Revised date: 16-7-2013
Accepted date: 26-7-2013

Please cite this article as: Pfeiffer, U.J., Vogeley, K., Schilbach, L., From gaze
cueing to dual eye-tracking: Novel approaches to investigate the neural correlates
of gaze in social interaction, Neuroscience and Biobehavioral Reviews (2013),
http://dx.doi.org/10.1016/j.neubiorev.2013.07.017

This is a PDF file of an unedited manuscript that has been accepted for publication.
As a service to our customers we are providing this early version of the manuscript.
The manuscript will undergo copyediting, typesetting, and review of the resulting proof
before it is published in its final form. Please note that during the production process
errors may be discovered which could affect the content, and all legal disclaimers that
apply to the journal pertain.
From gaze cueing to dual eye-tracking: Novel approaches to

investigate the neural correlates of gaze in social interaction

t
ip
cr
Ulrich J. Pfeiffer1*, Kai Vogeley1,2, Leonhard Schilbach1

us
an
M
1
Neuroimaging Group, Department of Psychiatry, University Hospital Cologne, Kerpener Strasse
d

62, 50937 Cologne

te

2
Institute of Neuroscience and Medicine – Cognitive Neuroscience (INM3), Research Center
p

Juelich, Leo-Brandt Strasse, 52428 Juelich

ce
Ac

1
Page 1 of 57
*Correspondence should be addressed to:

Ulrich J. Pfeiffer, Neuroimaging Group, Department of Psychiatry, University Hospital Cologne,

Kerpener Strasse 62, 50937 Cologne, mail: ulrich.pfeiffer@uk-koeln.de, phone : +49 (0)221 478

t
87145

ip
Abstract

cr
Tracking eye-movements provides easy access to cognitive processes involved in visual and

us
sensorimotor processing. More recently, the underlying neural mechanisms have been examined

an
by combining eye-tracking and functional neuroimaging methods. Apart from extracting visual

information, gaze also serves important functions in social interactions. As a deictic cue, gaze can
M
be used to direct the attention of another person to an object. Conversely, by following other

persons’ gaze we gain access to their attentional focus, which is essential for understanding their
d

mental states. Social gaze has therefore been studied extensively to understand the social brain. In
te

this endeavor, gaze has mostly been studied from an observational perspective using static
p

displays of faces and eyes. However, there is growing consent that observational paradigms are
ce

insufficient for an understanding of the neural mechanisms of social gaze behavior, which

typically involve active engagement in social interactions. Recent methodological advances have
Ac

allowed increasing ecological validity by studying gaze in face-to-face encounters in real-time.

Such improvements include interactions using virtual agents in gaze-contingent eye-tracking

paradigms, live interactions via video feeds, and dual eye-tracking in two-person setups. These

novel approaches can be used to analyze brain activity related to social gaze behavior. This

review introduces these methodologies and discusses recent findings on the behavioral functions

and neural mechanisms of gaze processing in social interaction.

2
Page 2 of 57
Keywords

Social interaction, gaze, eye-tracking, joint attention, hyperscanning, mentalizing, autism

t
ip
cr
us
an
1. Introduction

1.1 The special case of gaze in social cognition

M
Whenever we look at a face, the eyes are the primary and most consistent target of our visual
d

attention from early infancy on (Haith et al., 1977; Walker-Smith et al., 1977). Despite the
te

emergence of other means to navigate the social world (e.g. language), gaze remains a crucial cue
p

system for our understanding of others throughout adulthood and serves a variety of social-
ce

cognitive functions beyond mere visual detection (e.g. Frischen et al., 2007). It has been related

to the seeking of information, signaling interpersonal attitudes, regulating the synchronicity of

Ac

speech during dialogue, and it plays a significant role in the regulation of interpersonal distance

(Argyle et al., 1973; George and Conty, 2008). Furthermore, the eye region provides information

related to a person’s identity, emotional state, and focus of visual attention. The latter is a result

of the unique morphology of the human eye (Kobayashi and Kohshima, 2001): Non-human

primates generally have darkened eyes which makes it difficult to distinguish the iris from the

surrounding sclera. In contrast, the human eye is characterized by an exposed, depigmentated

sclera surrounding the dark-colored iris. The development of these features facilitates the
3
Page 3 of 57
detection of the gaze direction of other individuals (Emery, 2000), which provides an important

cue to the thoughts of others regarding entities in a shared environment and thereby to their

mental states. Accordingly, social gaze has been termed a ‘window into social cognition’

(Shepherd, 2010). Hence, the behavioral functions and neural mechanisms of gaze behavior are

t
ip
of interest not only for neuroscience, but for a wide range of disciplines encompassing social

psychology, linguistics, human-computer interaction, as well as developmental and evolutionary

cr
psychology.

us
an
1.2 Non-interactive methods to study social gaze

Until recently, gaze behavior in social contexts has been studied using comparably static
M
methods. A large body of the literature has relied on gaze-cueing paradigms in which the

influence of a static gaze cues on attentional processing is examined (e.g. Frischen et al., 2007;
d

Langton et al., 2000). In these paradigms, an attentional cue towards a certain object or location
te

is exerted by a picture of a real face (e.g. Hood et al., 1998; Mason et al., 2005), a virtual agent
p

(e.g. Nuku and Bekkering, 2008; Pelphrey et al., 2003), or simply a symbolic drawing (e.g.
ce

Friesen and Kingstone, 1998). This approach has proven extremely helpful in unraveling the

effects of social orientation on attentional processing. Another class of studies has used more
Ac

dynamic stimulus materials including videos of real persons or virtual agents moving their eyes

towards or away from a participant (e.g. Kuzmanovic et al., 2009; Pelphrey et al., 2004;

Schilbach et al., 2006; von dem Hagen et al., 2013). These experiments have often been

complemented by concurrent recordings of participants’ eye-movements using eye-tracking

devices which can be applied in behavioral as well as functional imaging settings. While such

‘diagnostic’ eye-tracking (Duchowski, 2007) provides insights about a person’s focus of visual

4
Page 4 of 57
attention, further important cues to cognitive and affective processes can be obtained by

pupillometry (Granholm and Steinhauer, 2004; Laeng et al., 2012). For example, it has been

shown that increases of pupil diameter correlates with switches of attention (e.g. Karatekin et al.,

2004), increases in cognitive load and task difficulty (e.g. Kahneman and Beatty, 1966), motor

t
ip
preparation (e.g. van der Molen et al., 1989), and emotional processing (e.g. Steinhauer et al.,

1983). As all these processes play a role in social interactions, pupil diameter is an important

cr
source of information in studies of social cognition. However, all the methods mentioned above

us
have in common that they do not capture the interactive nature of real-world social cognition

(Schilbach et al., in press). One major aim of this review is therefore to present novel

an
methodological developments which allow studying gaze in truly interactive settings and to

integrate the findings obtained with these methods with previous research.
M
d

1.3 Motivation of this review

te

In the past, social cognition has been studied from a detached, observational perspective in tasks
p

involving inert social stimuli (offline cognition), which has led to a situation in which social
ce

cognition is studied without actual social interaction in what has been termed ‘isolation

paradigms’ (Becchio et al., 2010). Recent claims emphasized that the active engagement with
Ac

others in interaction (online cognition) plays a particular role in understanding other minds

(Schilbach et al., in press) and might underlie the development of our ability to think about

others, rather than the other way round (Reddy and Morris, 2004). A growing number of

researchers has therefore postulated the study of social interaction from an interactor’s instead of

an observer’s point of view (e.g. Becchio et al., 2010; De Jaegher and Di Paolo, 2007; De Jaegher

et al., 2010; Froese and Fuchs, 2012; Hobson, 1991; Konvalinka and Roepstorff, 2012; Marsh et

5
Page 5 of 57
al., 2009; Pfeiffer et al., 2013; Reddy and Morris, 2004; Riley et al., 2011; Schilbach et al., in

press). The importance of studying behavior and neural activity in truly interactive contexts is

particularly important in studies of social gaze, because social gaze always involves two

individuals who are engaged with one another face-to-face and in real-time. In the last years, the

t
ip
wealth of literature based on applying ‘conventional’ (i.e. non-interactive) methods to the study

of social gaze has been complemented by exciting methodological developments which allow

cr
studying gaze in interaction These developments and related new insights into the neurobiology

us
of social gaze will be the focus of this review.

The structure of this review is as follows. In chapter 2, a brief overview is provided

an
regarding the function and processing of gaze in social contexts. Results reviewed in this part
M
have been obtained with conventional (i.e. non-interactive) methods. Emery (2000) has identified

mutual gaze, gaze aversion, gaze-following, joint attention, and shared attention as the core
d

processes constituting what will be termed social gaze throughout this article. This chapter does
te

not represent a comprehensive review but rather provides an overview and update of key aspects

of social gaze that have been discussed in more detail elsewhere (e.g. Carlin and Calder, in press;
p
ce

Emery, 2000; George and Conty, 2008; Haxby et al., 2002; Itier and Batty, 2009; Langton et al.,

2000; Nummenmaa and Calder, 2009; Shepherd, 2010). In chapter 3, the major focus of this
Ac

article will then be on novel research methods which can be used to disentangle the neural

mechanism underlying gaze behavior in dynamic, real-time social interactions – an aspect that is

greatly underrepresented in the current literature. In chapter 4, novel findings obtained with

interactive techniques are discussed and integrated with previous research to provide a detailed

picture of the neural mechanisms supporting joint attention, which has been described as the most

pivotal non-verbal skill enabling an understanding of other minds (e.g. Clark, 1996; Mundy and

Newell, 2007; Tomasello et al., 2005). Chapter 5 discusses new insights into impairments of
6
Page 6 of 57
gaze-based social interactions in autism spectrum disorder (ASD). ASD is a developmental

disorder which is characterized by deficits in communication and social interaction (Baron-Cohen

and Belmonte, 2005). In particular, impairments in social gaze and the underlying neural

mechanisms are characteristic for ASD (e.g. Dalton et al., 2005; Pelphrey et al., 2005; von dem

t
ip
Hagen et al., 2013). The study of interactive gaze behavior in ASD might hence provide a key to

understanding the core deficits of this disorder. Eventually, chapter 6 discusses a putative

cr
neurofunctional model of gaze in interaction on the basis of the previous chapters before a brief

us
outlook to future research is given to conclude the review.

an
M
2. The core processes of social gaze
d

2.1. Mutual gaze

te

Any gaze-based interaction starts with two individuals looking at each other, a situation referred
p

to as mutual (or direct) gaze (Fig. 1A). Mutual gaze illustrates a key feature of social gaze,
ce

namely that perception and action are coupled in single acts of looking (Gibson and Pick, 1963).

Moreover, it represents a situation in which we mutually perceive an action through an identical

Ac

action. Consistently, numerous studies have found that direct gaze is a powerful modulator of

cognition – a phenomenon referred to as the ‘eye-contact effect’ (Senju and Johnson, 2009). For

example, it has been shown that faces displaying direct gaze are spotted faster among distractor

items (von Grünau and Anston, 1995) and memorized better than faces with averted gaze (Mason

et al., 2004). Direct gaze also speeds up the identification of faces and facial expressions (Adams

Jr. and Kleck, 2005) and has a positive effect on our judgment of the attractiveness of other

persons (Mason et al., 2005), and the likeability of objects associated with them (Strick et al.,
7
Page 7 of 57
2008). Most importantly, however, mutual gaze serves as an ostensive cue: it signals

communicative intent and ‘opens the channel’ for social interaction (Cary, 1978). A recent model

by Senju and Johnson (2009) proposes that on the neural level the eye contact effect is brought

about by a subcortical ‘fast-lane’ via the amygdala and low-level visual areas including the

t
ip
superior colliculus and the pulvinar. This subcortical mechanism is supposed to modulate social

brain areas involved in the detection of gaze direction, such as the superior temporal sulcus

cr
(STS), and intentionality, such as the medial prefrontal cortex (mPFC) and the temporo-parietal

us
junction (TPJ). The dorsolateral prefrontal cortex (dlPFC) administrates current task demands

further manipulating this system in a top-down manner (see Fig. 2 for an overview of brain

regions involved in the processing of social gaze).

an
M
2.2. Averted gaze and gaze-following

The perception of averted gaze is a prominent cue signaling that another individual’s attention is
d

not directed at oneself but at another person or object in the environment (Fig. 1A). Studies using
te

Posner-task-like gaze cueing paradigms have demonstrated that the detection of averted gaze
p

results in a shift of visuo-spatial attention by means of reflexive gaze-following behavior (Fig.

ce

1B) even when a gaze cue is counter-predictive of a target (cf. Shepherd, 2010). The detection of

gaze direction is associated with activation of the STS, which is commonly separated into an
Ac

anterior and a posterior region (a/pSTS). Neuroimaging studies have consistently reported the

pSTS to encode the perceived gaze direction of other individuals (cf. Nummenmaa and Calder,

2009). In a seminal study, participants watched video sequences in which an animated virtual

agent walked toward them and either expressed mutual or averted gaze when passing them

(Pelphrey et al., 2004). Mutual gaze resulted in greater activation of the pSTS than averted gaze,

thereby suggesting that this region encodes approach and avoidance associated with direct and

8
Page 8 of 57
averted gaze, which might play a role in the detection of communicative intent. However, in

recent years it was shown that the aSTS encodes gaze direction in a more fine-grained fashion

than the pSTS (Carlin and Calder, in press). While the pSTS responds rather generally to the

direction of attention and is not only activated by perceived gaze direction but also by other

t
ip
forms of biological motion (Redcay, 2008), the aSTS contains neural populations encoding gaze

direction independently of head direction and physical features of faces (Carlin et al., 2012,

cr
2011). It has therefore been suggested that there is a posterior-to-anterior specialization for gaze

us
direction which is increasingly invariant to gaze-irrelevant features (Carlin and Calder, in press).

There is evidence that in addition to its role in the general processing of gaze and head direction,

an
the pSTS is also sensitive to the social context in which a gaze shift occurs. This has been shown

in an experiment, where participants watched brief video clips in which a visual stimulus
M
appeared to the left or right side of a virtual agent looking at them. The agent then either shifted
d

its gaze towards the stimulus or towards the blank side. Whenever the agent’s response was
te

incongruent, activity in the right pSTS was increased. As a visual stimulus should evoke more

interest than a blank space, the shift away from the stimulus violated participants’ expectations
p

regarding the agent’s behavior. It was therefore concluded that the pSTS is involved in analyzing
ce

intentions expressed by gaze shifts (Pelphrey et al., 2003).

Ac

2.3. Joint attention

Following another individual’s gaze to a novel focus of visual attention creates a situation of joint

attention (JA, Fig. 1C). In contrast to a situation of coincidental looking, the follower is aware of

sharing the gazer’s focus of attention in joint attention. However, according to Emery (2000), JA

does not require the gazer to be aware of the follower’s reaction. Based on this, a distinction has

9
Page 9 of 57
been made between joint and shared attention (SA), with the latter requiring both individuals to

be aware of focusing on the same object and on each other (Fig. 1D). However, most researchers

commonly use the term JA to denote situations that would actually fall into Emery’s definition of

SA (e.g. Carpenter et al., 1998; Redcay et al., 2012b; Saito et al., 2010; Schilbach et al., 2010;

t
ip
Tomasello et al., 2005). To avoid confusion, in the present article we will also use the term JA to

denote situations involving mutual awareness. JA is a particularly interesting aspect of social

cr
gaze. While mutual gaze and gaze-following represent dyadic processes involving two

us
individuals, it represents a triadic interaction involving a ‘referential triangle’ of two individuals

and some third entity (e.g. object, person, location etc.) in the environment (Carpenter et al.,

an
1998). As people look where they attend and where they intend to act, JA is considered a

fundamental developmental milestone and essential to an understanding of other minds (Fig. 1E).
M
In light of the importance of this capacity, two previous functional neuroimaging studies have
d

addressed the neural basis of JA, but have done so without the use of interactive eye-tracking
te

technologies. Materna et al. (2008) found that engaging in JA by following another individual’s

gaze cue to an object recruited the pSTS, which is consistent with studies on the perception of
p

gaze direction. More importantly, these authors demonstrated that this activation is specific for
ce

gaze cues, while symbolic cues recruit the IPS which is more generally implicated in re-directing
Ac

spatial attention. Another study reported activity in the mPFC while participants engaged in JA

coincidentally (Williams et al., 2005). The mPFC is a well-known key region of the ‘social brain

network’ consistently activated across a wide range of mentalizing tasks (Amodio and Frith,

2006), thereby suggesting a direct link between JA and the inference of mental states. An

important distinction has been introduced between responding to a bid for JA (RJA) by following

someone’s gaze and initiating JA (IJA) by making another individual follow one’s own gaze

(Mundy and Newell, 2007). Importantly, IJA is not to be confused with initiating behavioral
10
Page 10 of 57
regulation (IBR) which has been numbered among joint attentional skills in infants by some

researchers (e.g. Leavens and Racine, 2009). IBR refers to the operationalization of others to

achieve a behavioral goal by protoimperative gestures – e.g. when infants point towards an object

that is out of reach (Bates et al., 1975). Although IBR represents an expression of personal

t
ip
intentions, it does not necessarily require shared intentionality as it is the case in IJA. Whereas

IBR and RJA can also be observed in non-human primates, IJA is believed to represent a

cr
uniquely human capacity (Moll and Tomasello, 2007; Tomasello and Carpenter, 2005; but see

us
Leavens and Racine, 2009 for a different interpretation). Moreover, children with autism are

relatively intact in their ability to follow gaze, while they do not initiate the sharing of perceptual

an
experience themselves (Mundy, 2003). Despite a wealth of developmental studies, neuroimaging

literature on JA is still sparse. Due to the lack of interactive technologies, the problem researchers
M
were facing until today was that they could only address RJA by using gaze-cueing procedures in
d

static experimental designs. However, this did neither capture the reciprocal nature of JA, nor did
te

it allow for disentangling differences in the neural mechanisms subserving IJA and RJA. The

next chapter shall therefore introduce the use of interactive eye-tracking paradigms in the study
p

of JA before the neural mechanisms of JA are discussed.

ce
Ac

3. Gaze in interaction: Novel methods and findings

Until present, the majority of studies on social gaze made use of gaze-cueing paradigms and other

experimental designs in which static stimuli were presented to participants. While these studies

provided important insights into the neural mechanisms of social gaze, they failed to capture an

essential aspect of real-life social encounters, in which a person cannot only react to another
11
Page 11 of 57
individual’s gaze behavior, but also has the opportunity to initiate social interaction and to

observe reactions to their own gaze behavior (Schilbach et al., in press). In sum, real-time social

interactions in everyday life are characterized by reciprocity and interdependence of actions

rather than by simple stimulus-response patterns. The following chapters describe four types of

t
ip
experimental setups, which allow the study of gaze-based interaction face-to-face and in real-time

as well as the findings obtained using these approaches: In chapter 3.1, the application of virtual

cr
reality techniques for the study of interactive gaze is introduced. Subsequently, chapter 3.2

us
discusses findings obtained with live video-feeds of real interaction partners. Results from fMRI

and EEG hyperscanning studies which allow acquisition and analysis of neural activity of both

an
interaction partners are discussed in chapters 3.3 and 3.4, respectively. In chapter 3.5, the

different approaches are compared and evaluated.

M
3.1. Gaze-based interaction in virtual environments
d

The first group of setups makes use of virtual reality technologies to create more realistic
te

interactions. Although such gaze-contingent eye-tracking has already been used in human-
p

computer interaction and marketing previously (Duchowski et al., 2004), it has only recently
ce

found its way into social neuroscience: Wilms and colleagues (2010) created a paradigm that

allows participants to interact with anthropomorphic virtual agents (i.e. virtual characters with
Ac

realistic human features) on a computer screen via their eye-movements (Fig. 3A). This allows

participants not only to react to the agent, but also to observe the agent moving its eyes in a gaze-

contingent fashion – that is, in response to their own gaze behavior as this would occur in

everyday social encounters. Various studies have demonstrated that the interaction with

anthropomorphic agents results in comparable reactions and social behaviors as real human

interactions. For example, socially induced inhibition as well as facilitation of task performance

12
Page 12 of 57
cannot only be observed in the presence of another person, but also in the presence of human-

controlled avatars (Hoyt et al., 2003). Likewise, participants’ regulation of interpersonal distance

and approach behavior is comparable in immersive virtual environments and real social

encounters (Bailenson et al., 2003). It was also demonstrated that participants displayed empathic

t
ip
concern for virtual agents in distressful situations, thereby demonstrating that encounters with

virtual characters readily elicit prosocial behavior (Gillath et al., 2008). Another study revealed

cr
that the simulation of gaze behavior by virtual characters in face-to-face interactions results in

us
similar experiences of social presence and intimacy as real gaze behavior (Bente et al., 2007).

Taken together, these studies demonstrate the usability of virtual agents in research on social

interaction.

an
M
In a seminal study, this method of applying gaze-contingent social stimuli was used to

investigate the neural correlates of IJA and RJA as well as self- and other-initiated non-joint
d

attention (NJA) in a combined eye-tracking and fMRI experiment (Schilbach et al., 2010).
te

Participants received the instruction that they would participate in an interactive game with

another person outside the scanner. They were informed that the eye-movements of the other
p
ce

were displayed by a virtual agent and that the other would see a similar visualization of their own

gaze behavior (Fig. 3A). The experiment employed a 2x2 design with the factors ‘initiation’ (self
Ac

vs. other) and ‘gaze reaction’ (JA vs. NJA). In the self-condition, their task was to direct the gaze

of the other to an object by fixating it. The virtual agent was programmed to either follow their

gaze or avert its gaze to another object, thereby engaging in JA or NJA, respectively. In the

other-condition, the agent initiated situations of JA and NJA. Importantly, the algorithm required

participants to establish mutual gaze at the beginning of each trial. Prior to each interaction block,

participants were cued whether it was their turn to initiate or to respond. Independent of who

initiated the interaction, JA was associated with activation of the mPFC, while NJA recruited a
13
Page 13 of 57
fronto-parietal network related to the voluntary control of visual attention. Interestingly,

activation of the mPFC was higher during RJA as compared to IJA. This is consistent with results

of a previous study which demonstrated that the reaction to another person’s gaze cues recruits

the same area (Williams et al., 2005). The mPFC is consistently recruited whenever we reason

t
ip
about other person’s mental states (Amodio and Frith, 2006). The observed increase in mPFC

while we follow another person’s gaze to a location of her interest is therefore likely to reflect an

cr
increased demand for mentalizing. Most intriguingly, however, results indicated that the

us
successful initiation of JA recruited the ventral striatum, a pivotal component of the brain’s

reward system. Moreover, the strength of the BOLD signal in this region correlated with

an
pleasantness ratings of IJA in a post-experiment questionnaire. Interestingly, it has previously

been argued that JA is not only characterized by its functional role of establishing perceptual
M
common ground, but also by an intrinsic motivation to share (Tomasello et al., 2005). The finding
d

that the engagement of others in JA recruits the reward system can therefore be seen as providing
te

first-time evidence for a neural mechanism underlying this intrinsic motivation.

In the meantime, several behavioral studies have applied interactive eye-tracking

p
ce

paradigms to study cognitive processes underlying self-initiated gaze behavior. These are

sketched briefly here because of their potential to inform future neuroimaging studies. For
Ac

instance, in what the authors called a ‘non-verbal Turing test’, it was recently shown that

participants discriminate a human interaction partner from a computer program based on the

degree a virtual character engages in JA, thereby demonstrating that we have a natural

expectation of others following our gaze and to engage in JA (Pfeiffer et al., 2011). Another

study used a gaze-contingent setup to explore the ‘microstructure’ of JA and SA (Pfeiffer et al.,

2012). Results demonstrated that the latency of a gaze-following reaction needs to fall into a

certain temporal range to be perceived as contingent upon one’s own behavior. Furthermore, the
14
Page 14 of 57
self-initiated establishment of SA requires more gaze shifts between an agent and an object as

compared to JA, thereby arguing that these phenomena are indeed two different cognitive

processes as suggested by Emery (2000). It has also be shown that the successful initiation of JA

resulted in differences in dwell time on faces, facilitated looking back at the faces, and resulted in

t
ip
preference for faces that had previously engaged in JA in comparison to faces that had not

(Bayliss et al., 2012). Finally, novel studies have also started to investigate individuals’ sense of

cr
agency and self-regulation of social gaze in virtual social contexts (Grynszpan et al., 2011, 2012).

us
In sum, these behavioral approaches and findings provide a wealth of opportunities for further

neuroimaging studies based on gaze-contingent social stimuli.

an
M
d
te

3.2. Gaze-based interaction via live video feeds

The use of virtual agents in interactive eye-tracking paradigms was paralleled by the development
p
ce

of experimental setups allowing face-to-face interactions between a subject inside an MRI

scanner and an experimenter located outside the scanner in real-time (Redcay et al., 2010). This
Ac

was enabled by a live video feed of the face of an experimenter to a participant inside the scanner

whose gaze was monitored by an eye-tracker to enable the experimenter to monitor the

participant’s eye-movements on a dedicated screen (Fig. 3B). Similar to gaze-contingent eye-

tracking, live video setups had been previously used in other areas of research without making

their way into social-cognitive neuroscience yet (e.g. Murray and Trevarthen, 1985; Wellens,

1982).

15
Page 15 of 57
 In the first of two crucial experiments, participants either engaged in live interactions with

the experimenter, or saw recordings of the experimenter’s behavior from earlier interactions. In

the live interactions, participants saw the experimenter and two containers with different objects

in them. Their task was to use their eye-movements either to prompt the experimenter to show

t
ip
them the content of one of two containers, or to choose the correct container for an object the

experimenter was showing to them. Results indicated that live interactions were associated with

cr
greater activity in areas that have been implicated in mentalizing, attention, and reward. Among

us
the mentalizing areas were the right TPJ, and anterior as well as posterior regions of the STS.

Additional activation of the anterior cingulate cortex (ACC), of the cuneus and of the cerebellum

an
suggested an increased demand of visual attention during live interaction. Furthermore, the

ventral striatum and the amygdala were activated, which suggests that live interaction recruits the
M
reward system. While this is generally consistent with the finding of increased striatal activation
d

during IJA, live interaction resulted in activation of the putamen rather than the nucleus
te

accumbens as describe earlier in the study by Schilbach and colleagues (2010). This points

towards potential differences in the exact nature of the reward processes involved in these two
p

situations (Delgado, 2007). In a second experiment comparing joint and solo attention,
ce

participants engaged in a game in which they had to find an item in one of four locations on the
Ac

screen. In the JA condition, they received a gaze cue to the correct location by the experimenter.

In the solo attention condition, participants had to locate the object while the experimenter

disengaged from the interaction by closing her eyes. JA evoked greater activity in the pSTS and

the TPJ and to a lesser extent also in the aSTS and the mPFC. The activation of the pSTS is

consistent with previous studies on the detection of gaze direction (as discussed above), which is

a crucial component of responding to another individual’s bid for JA and has also been reported

in a previous study on responding to JA (Materna et al., 2008).

16
Page 16 of 57
 In another study, Redcay and colleagues directly compared self- and other-initiated JA

using the same setup in which participants were instructed to play a game together with the

experimenter (Redcay et al., 2012b). Their task was to locate an object on the screen following

the gaze cue of the experimenter, or to help the experimenter to find the object by providing them

t
ip
with a gaze cue. Again, a solo attention condition served as the control condition. The most

consistent activation for JA as compared to solo attention was observed in the right pSTS. This is

cr
in line with previous studies that demonstrated the involvement of this region in RJA (Materna et

us
al., 2008; Redcay et al., 2012b). As it has been described above, the pSTS is crucially involved in

decoding the direction of another person’s gaze direction as well as the intention behind a gaze

an
shift, particularly in the context of social interactions (Nummenmaa and Calder, 2009). When

compared to a solo attention condition, both IJA and RJA also showed greater activation of the
M
dorsal mPFC (dmPFC) which is consistent with the results of other studies (Schilbach et al.,
d

2010; Williams et al., 2005) and is likely to reflect the sharing of attention that is necessary for
te

the inference of another individual’s mental states. During RJA, however, the ventral mPFC

(vmPFC) was activated to a greater extent as compared to IJA. This activation extended in the
p

mOFC, which is considered to be a principal component of the brain’s reward system. As the
ce

gaze cue of the experimenter was a cooperative cue in the framework of a collaborative game
Ac

involving a joint intention (i.e. ‘finding the object together’), the authors argued that this

activation might be related to the anticipation of rewards. However, although there is some

evidence for the involvement of the mOFC in reward anticipation, a recent meta-analysis

demonstrated that it is more specifically involved in the subjective hedonic experience of reward

(Peters and Büchel, 2010). This activation could therefore rather be related to the experience of a

cooperative cue by another person per se. IJA recruited a fronto-parietal attention network

including the inferior parietal cortices and the inferior and middle frontal gyri, which is explained
17
Page 17 of 57
by greater attentional demands attributed to the voluntary initiation of JA in comparison to

following someone’s gaze. In the study cited above, this network was only recruited during NJA

(Schilbach et al., 2010). Although this discrepancy cannot be entirely clarified here, it might be

related to the explicit instruction to engage in NJA that participants received in the study by

t
ip
Schilbach and co-workers (2010), while the control condition in the studies using live video feeds

was a solo attention condition in which the experimenter closed her eyes and thereby disengaged

cr
from the interaction.

us
an
3.3. Gaze-based interaction in fMRI hyperscanning setups

The approaches discussed so far provide important insights into the neural mechanisms
M
supporting the engagement in gaze-based interactions within the brain of a single person.

However, it is conceivable that the neural processes within the brains of two interacting persons
d

are interdependent (Konvalinka and Roepstorff, 2012). A more complete understanding of the
te

mind in interaction would therefore require an analysis of two data sets collected at the same
p

time. This has been made possible by so-called hyperscanning paradigms, in which the brain
ce

activity of two (or more) persons is measured simultaneously while they are engaged in social

interaction (Montague et al., 2002). Very recently, an fMRI hyperscanning setup using live video
Ac

feeds was introduced for the study of JA (Saito et al., 2010). In this setup, two interconnected

fMRI scanners were equipped with eye-tracking systems including infrared cameras capturing

participants’ eye regions (Fig. 3C). These video images were transferred to the upper half of the

screen of the other participant in real-time. The lower half was used for stimulus presentation

which consisted of two targets that were presented on the left and the right side of the screen.

Stimulus presentation, video transmission, and fMRI data acquisition were synchronized

18
Page 18 of 57
throughout the experiment. In a 2x2 design, participants’ task was to shift their gaze either

according to a gaze cue by the other participant or due to a color change by one of the targets (i.e.

gaze-cueing vs. target-cueing). In so-called concordant blocks, they had to look in the same

direction, whereas discordant blocks required a gaze shift to the opposite target. A condition in

t
ip
which participants were instructed to engage in mutual gaze in the absence of the targets served

as a high-level baseline.

cr
A first analysis aimed at unraveling the effects of gaze- versus target cueing without

us
differentiating between JA and NJA. Apart from visual association areas, dmPFC and right

inferior frontal gyrus (rIFG) showed increased activity during all gaze-cued trials. When focusing

an
on RJA, however, the only region that was differentially activated was the intraparietal sulcus
M
(IPS). These results are somewhat at odds with previous studies on joint attention. For example,

other studies provide converging evidence that the mPFC is specifically activated in situation of
d

JA but not NJA, consistent with the previously described role of this region in inferring mental
te

states (Bristow et al., 2007; Materna et al., 2008; Redcay et al., 2010, 2012b; Schilbach et al.,

2010; Williams et al., 2005). Furthermore, the unspecific involvement of the rIFG appears
p
ce

puzzling. The IFG is a component of the putative mirror neuron system which plays a role both

in the execution and in the observation of actions and has therefore been related to the
Ac

understanding of the intentions behind actions by means of shared representations (Rizzolatti and

Sinigaglia, 2010). Obviously, situations of JA result from subsequent gaze-shifts to the same

target by two individuals and thus might be considered to involve an imitative component which

potentially requires mirroring. If the IFG was specifically involved in gaze-cued shifts of

attention, this should become particularly obvious in situations of JA but not NJA. However, this

was not the case as the analysis of neural activity during RJA exclusively revealed activation of

the intraparietal sulcus (IPS). Materna and colleagues (2008) reported a similar IPS activation,
19
Page 19 of 57
but were able to show that this was independent of the type of cue which triggered participants’

gaze shifts (i.e., arrow vs. gaze). This finding is consistent with the domain-general function of

the IPS in reflexive and controlled shifts of attention (Corbetta and Shulman, 2002). In sum, the

activations reported by Saito and colleagues cannot be interpreted unequivocally in the light of

t
ip
previous findings and hence warrant further investigation.

cr
One potential reason for the discrepancies observed in the hyperscanning study and other

investigations of JA might be the application of different experimental designs (see also Redcay

us
and Saxe, 2013). Interestingly, however, this study is the first study on interactive gaze behavior

pointing towards an involvement of the putative mirror neuron system in joint attention. To

an
further explore this possibility, Saito and colleagues analyzed between-subject synchronization of
M
brain activity during the baseline condition in which participants engaged in mutual gaze. This

analysis indicated the rIFG as a locus of neural synchronization during the baseline task. This is
d

highly plausible considering that mutual gaze is a situation in which the observed and the
te

executed action are identical. The authors hence argue that rIFG activity could be the neural

correlate that is required for the establishment of communicative intent, thereby providing some
p
ce

kind of ‘readiness potential’ for subsequent gaze-based interactions. Indeed, it has been shown

that the rIFG is more active during complimentary as compared to imitative actions (Newman-
Ac

Norlund et al., 2007). Although mutual gaze necessarily involves an imitative component it can

be assumed that it is understood as a complimentary action due to its social salience (Senju and

Johnson, 2009) and relevance in initiating communication (Cary, 1978) and joint actions (e.g.

Sebanz et al., 2006; Vuilleumier et al., 2005). Interestingly, also the duration of direct gaze

displayed by an interaction partner modulates rIFG activity. A recent study reported greater rIFG

activity when direct gaze was expressed for 4 seconds as compared to 1 second (Kuzmanovic et

al., 2009). In addition, these authors showed that likeability increases with increasing duration of
20
Page 20 of 57
direct gaze. The duration of baseline trials in the hyperscanning study was 4.5 seconds, which

might be taken to suggest that situations of mutual gaze need to persist for a certain time until the

mirror neuron system is activated. This might explain the lack of rIFG activation in other studies

of joint attention, in which mutual gaze was only briefly established (or not at all in the case of

t
ip
mere gaze-cueing studies) before participants engaged in JA or NJA. Alternatively, it is

conceivable that rIFG activity might only be detected by analyzing between-brain

cr
synchronization but not by analyzing brain activity of single subjects (Konvalinka and

us
Roepstorff, 2012). The findings by Saito and collaborators (2010) provide exciting first insights

into the neural mechanisms that are related to the actual actions involved in JA rather than to the

an
re-orienting of (social) attention or higher-order mentalizing processes. Nevertheless, due to the

uniqueness of these findings more research is needed in characterizing the role of the mirror
M
neuron system in social gaze (Schilbach, 2010).
d
p te

3.4. Gaze-based interaction in dual EEG setups

ce

A more general drawback of fMRI scanning and hyperscanning is the low temporal resolution
Ac

due to the inertia of the BOLD response. Accordingly, it has been argued that this might make an

informative analysis of the neural processes supporting minute interactions between mutually

coordinated agents very difficult (Konvalinka and Roepstorff, 2012). For this reason, a recent

study has applied dual electroencephalography (dual-EEG or EEG hyperscanning) to study the

temporal dynamics of JA in live interactions (Lachat et al., 2012). This setup has the major

advantage over fMRI studies that it allows true social interactions in which two participants are

seated face-to-face with each other. Stimuli consisted of a circle of differently colored LEDs that

21
Page 21 of 57
were visible to both participants. Similar to the fMRI hyperscanning task, a 2x2 design was

chosen in which participants either engaged in JA or NJA in response to a gaze cue by their

interaction partner, or in response to one of the LEDs lighting up. EEG data acquisition and

stimulus presentation were synchronized throughout the experiment. Results revealed that gate-

t
ip
cued JA resulted in the modulation in the alpha- and mu-rhythms, which have frequencies

between 11 and 13 Hz and are measured over parieto-occipital and centro-parietal regions,

cr
respectively. Specifically the suppression of oscillation in the mu frequency band has previously

us
been related to activity in the mirror neuron system (Pineda, 2005), thereby suggesting that the

mirroring of gaze shifts is a crucial process involved in joint attention. As the analysis of dual

an
EEG studies is computationally demanding, there have been only few studies addressing social

interaction using this method. Interestingly, however, these studies suggest that mu rhythm
M
suppression is generally associated with inter-individual coordination in social interaction, for
d

instance during spontaneous imitation (Dumas et al., 2010) or during the coordination of finger
te

movements (Naeem et al., 2012; Tognoli et al., 2007).

Unfortunately, while providing exciting tools to study realistic interactions, none of the
p
ce

hyperscanning studies contributed to disentangling the neural mechanisms subserving IJA and

RJA. In the fMRI study, this factor was not included as a variable of interest. In the dual EEG
Ac

study, the preselected unit of analysis was the time participants’ gaze dwelled on the object of

joint focus which made it impossible to address this aspect. Studies using a more fine-grained

experimental design and/or different temporal regions of interest would therefore be required to

exhaust the full potential of truly interactive setups. Finally, in order to unify data from fMRI and

EEG studies, a better localization of the anatomical sources of neuronal oscillations is needed.

Very recently, a dual magnetoencephalography (MEG) setup was developed for the study of real-

time social interaction (Baess et al., 2012). This provides exciting opportunities and might
22
Page 22 of 57
possibly help to close the gap between temporal and spatial resolution in studies on gaze-based

interaction.

t
3.5. Evaluation of interactive approaches to the study of social gaze

ip
All four approaches to the study of dynamic gaze-based interactions that have been discussed in

cr
this article have advantages and disadvantages. FMRI hyperscanning has the advantage that it

us
allows face-to-face interactions while neural activity is recorded simultaneously from two brains.

A general problem is the methodological complexity and the costs involved in hyperscanning.

an
More specifically, a potential problem of the study by Saito and colleagues (2010) is that the

methods used for data analysis do not rely on general linear models commonly used in fMRI
M
studies, but instead on the inter-subject correlation of what the authors refer to as ‘innovations’.

These represent “the residual time courses of the neural activities obtained by modeling out the
d

task-related effects and other confounding effects” (Saito et al., 2010, p.2) and have previously
te

been used to model correlations between regional activity within single brains (Riera et al.,
p

2004). While this method has the advantage of incorporating the structure of the background
ce

noise relating to intrinsic spontaneous activity, it has not been used before with the aim of

detecting inter-subject neural synchronization. Therefore, the results of the hyperscanning study
Ac

have to be taken with some caution and might not be easily comparable to the results of other

studies. Notwithstanding, the hyperscanning setup provides an exciting opportunity for studies on

JA in a highly realistic fashion.

At the moment, the only systems allowing true face-to-face interactions between two

participants are EEG hyperscanning setups (Babiloni and Astolfi, in press). Although the use of

live video feeds allows ecologically valid gaze-based interactions, no comparable possibilities

23
Page 23 of 57
exist for fMRI, which presently represents the most common method to study the neural

mechanisms of social interaction. However, the study of gaze behavior in interactions with live

faces is prone to substantial noise because gaze direction is only one - albeit powerful - cue out of

multiple facial cues conveying non-verbal information (Grant, 1969). Even slight changes in the

t
ip
muscles controlling the eye region or the labial angles can result in differences in the perception

of a person (Ekman and Oster, 1979) and it has often been demonstrated that facial features and

cr
facial configuration bias our impression of others (Berry and McArthur, 1986). Studies focusing

us
on gaze-based aspects of social interaction hence require strict control of these features which is

very difficult when an experimenter or actor functions as the interaction partner.

an
These complications can be overcome by using anthropomorphic virtual agents that have
M
been suggested to provide a highly efficient tool in the study of social interaction (Vogeley and

Bente, 2010). The use of virtual agents has the advantage that cues of interest can be isolated
d

while confounding variables can be neutralized (Fox et al., 2009). This enables examining the
te

effects of gaze reactions while other facial features such as emotional expressions or

attractiveness are kept constant. Furthermore, these features can be modulated in a step-by-step
p
ce

fashion, for instance by pairing congruent gaze reactions with a smile, a frown, or an eye-blink.

Despite their usefulness, however, it is clear that the use of virtual reality techniques intrinsically
Ac

entails an abstraction from reality that cannot be neglected. A more specific caveat concerns the

potential application of virtual faces in research including persons with ASD. A study examining

the influence of stimulus type on visual exploration demonstrated that persons with ASD do not

show differences during the exploration of real and virtual faces (Hernandez et al., 2009).

However, a second study investigating changes in pupil size during the exploration of real faces,

virtual faces, and objects in children with ASD and healthy controls found pupil dilation during

the exposure to real faces but not virtual faces or objects in the ASD group, which has been
24
Page 24 of 57
attributed to increases in cognitive load (Martineau et al., 2011). Such changes in cognitive load

in the absence of behavioral differences indicate that caution needs to be applied when using

virtual agents to study social gaze in ASD.

Although both approaches have limitations, the decision between virtual and real

t
ip
interactions is at the moment predominantly a decision between taking a bottom-up or a top-

cr
down approach to the study of gaze-based interaction, respectively. A symbiosis of both

approaches might promise exciting possibilities. Recently, a dual eye-tracking setup was

us
introduced in which two individuals can engage in multimodal interactions via gaze cursors,

mouse movements, and verbal communication. While it does not provide a realistic visualization

an
of gaze, a more integrative approach to the integration of several channels of communication is
M
taken by this setup (Carletta et al., 2010). This system has inspired work in our group resulting in

the development of a system which enables two individuals to engage in gaze-based interaction
d

with each other while their gaze behavior is visualized by virtual agents (Barisic et al., 2013). For
te

a quantification of gaze parameters beyond gaze direction, this platform allows very fine-grained

behavioral measurements and allows describing interactive gaze behavior in terms of direction,
p
ce

scan path length and area, number of saccades and fixations, fixation duration, and saccade

velocity. In comparison to other systems described here, this platform has the advantage that real-
Ac

time interactions between two individuals can be combined with the benefits and experimental

control provided by virtual reality methods and can possibly be used in future imaging studies.

4. The neural mechanisms of joint attention revisited

As interactive methods have in particular been used to study JA, the underlying neural
25
Page 25 of 57
mechanisms are re-evaluated in the following. This re-evaluation points towards commonalities

as well as discrepancies with respect to those mechanisms. For instance, only two of the studies

reported here have shown an involvement of the right TPJ (Redcay et al., 2012b, 2010). Studies

by Saxe and colleagues (for an overview see Saxe, 2006) emphasized the relevance of the TPJ in

t
ip
differentiating the actual content of other people’s mental states by an inference of transient goals, beliefs,

or desires. Although it is possible that this area might have been recruited because participants interacted

cr
with a live human whose face was fully visible and whose gaze behavior might not have been controlled

us
to the same extent a virtual character’s gaze could have been, this needs to be put to the test in future

studies. The same is true for the involvement of the mirror neuron system during JA which has

an
been suggested by both hyperscanning studies discussed above (Lachat et al., 2012; Saito et al.,

2010). These findings are neither consistent with results from passive viewing studies of JA
M
(Materna et al., 2008; Williams et al., 2005), nor are they comparable to other interactive eye-

tracking studies of JA (Redcay et al., 2012b, 2010; Schilbach et al., 2010). A vital question for
d

future studies will thus be to determine the parameters which influence recruitment and possibly
te

co-activations of cortical midline structures and regions of the mirror neuron system.
p

A region critically involved in JA is the mPFC, which is commonly divided into the
ce

dmPFC and the vmPFC. The most reliable activation during JA was observed in the dmPFC

(Redcay et al., 2012b, 2010; Saito et al., 2010; Schilbach et al., 2010; Williams et al., 2005)
Ac

which is involved across a wide range of social-cognitive tasks requiring the inference of mental

states. Despite early evidence that the dmPFC is consistently active during mental state

attributions (e.g. Gallagher et al., 2000; Vogeley et al., 2001), later studies indicated that the

dmPFC is also activated during social interactions which do not require explicit mental state

attribution (Schilbach et al., 2010). One study measured neural activity in situations in which

participants were addressed either by direct gaze or by a voice calling their name showed that the

26
Page 26 of 57
intention to communicate engages the dmPFC independent of the modality used to convey the

intention (Kampe et al., 2003). Another study showed that the dmPFC was more active when

participants inferred other individuals’ communicative intentions than when they reasoned about

private intentions (Walter et al., 2004). The dmPFC was also differentially related when

t
ip
participants observed animations of virtual agents directing their gaze at them and displaying

socially salient facial expression. This was not the case when participants observed the agent

cr
directing this behavior at another person (Schilbach et al., 2006). In one study, the dmPFC was

us
more active during IJA as compared to RJA (Redcay et al., 2012b), which might be related to the

establishment of communicative intent. Taken together, the activation of the dmPFC appears to

an
provide a neural substrate enabling communication about the shared environment and hence

supports a ‘meeting of minds’ (Amodio and Frith, 2006). In contrast, the vmPFC was more active
M
during RJA than during IJA, which might reflect that being intentionally directed to a certain
d

location by another person is likely to involve more mentalizing as compared to leading

te

someone’s gaze (Redcay et al., 2012b; Schilbach et al., 2010). Nevertheless, the role of this

region is less clear at the moment, although the extension of the vmPFC cluster into the mOFC in
p

the second study by Redcay et al. (2012b) might be indicative of reward-related processes
ce

triggered by the cooperative nature of the interaction partner’s gaze cues in the game-like context
Ac

of their study.

The involvement of the pSTS is less clear as only some of the studies examining JA have

shown increased activation of this region (Materna et al., 2008; Redcay et al., 2012b, 2010) while

others did not (Saito et al., 2010; Schilbach et al., 2010; Williams et al., 2005). This absence of

pSTS activation is surprising given the essential role of the pSTS in the decoding of the direction

as well as the underlying intentions of gaze shifts (Pelphrey and Morris, 2006). Redcay and

colleagues argue that this might be due to dissimilarities in the experimental designs – in
27
Page 27 of 57
particular in the control conditions – used in the different studies of JA (Redcay et al., 2012b). In

the studies by Redcay, the control condition consisted of disengagement of the experimenter from

the interaction, while there was an explicit instruction to look at the opposite object in the study

by Saito et al. (2010) or to look at one of two different objects in the study by Schilbach and co-

t
ip
workers (2010). Despite the requirement of an anti-saccade to complete the task, participants still

need to coordinate their own behavior with a gaze cue presented by another person. Considering

cr
the previously described role of the pSTS in processing other’s gaze direction in socially relevant

us
contexts, NJA might therefore result in equal recruitment of this area as JA. This suggests that the

pSTS is not specifically recruited for JA, but rather encodes the direction of gaze shifts bearing

an
communicative meaning. This is consistent with the broader role of the pSTS in the processing of

communicative cues that – depending on sensory modality and task demands – might be
M
modulated by network co-activations (Redcay, 2008). This has been taken to suggest that future
d

studies should focus on connectivity analyses to detect similarities and differences in STS
te

activation between different tasks (Hein and Knight, 2008).

Taken together, the mPFC and the pSTS appear to be central for the establishment of JA.
p
ce

The activation of these areas may represent the cognitive aspects of JA, such as the detection of

gaze direction, the re-orientation of attention, and the inference of mental states. However, it has
Ac

been argued that JA requires a combination the cognitive mechanisms enabling us to engage in

JA and an intrinsic motivation to share attention and establish common ground with another

person – i.e. ‘a special motivation to feel and act and perceive together’ which is uniquely human

and develops during the first year of life (Tomasello et al., 2005, p.8). This emphasizes the

significance of the ventral striatal activation observed for self-initiated JA in one of the studies

(Schilbach et al., 2010). As the activation of the reward-system by a successful engagement of

others in JA might indeed foster the development of social interactions, this has been interpreted
28
Page 28 of 57
as the neural mechanism supporting the motivational aspect of JA. Consistent with this idea,

Redcay et al. demonstrated a role of the reward system during live versus pre-recorded

interactions (Redcay et al., 2010). As other studies on JA failed to report reward-related

neurocircuitry during JA, a final conclusion about the intrinsic motivational forces driving JA

t
ip
cannot be drawn at present. In the following chapter, differences in the JA network in persons

with ASD are discussed.

cr
us
an
5. Impairment of gaze-based interaction in autism spectrum disorders

Autism spectrum disorders (ASD) are characterized by deficits in communication, and social
M
interaction (Baron-Cohen and Belmonte, 2005). For instance, there is evidence that autistic
d

persons avoid the eye region during the visual inspection of faces (Pelphrey et al., 2002). Eye-
te

tracking experiments have revealed that autistic persons spent significantly less time fixating the

eye region of people as compared to non-autistic controls in passive viewing studies involving
p

social scenes (Klin et al., 2002). Furthermore, they have difficulties with interpreting gaze as a
ce

non-verbal cue supporting the disambiguation of social scenes, thereby suggesting a more general
Ac

problem in using gaze as a tool to infer the mental states of others (Boraston and Blakemore,

2007). Interestingly, autistic children appear to be able to follow someone’s gaze, but tend to

spend less attention to congruent objects in a gaze-following task (Bedford et al., 2012). This

suggests that while core processes of social gaze can be functional, they might be driven by

different motives than in non-autistic individuals, possibly indicating changes in the functional

connectivity of autistic brains (e.g. Kleinhans et al., 2008; Murphy et al., 2012). A more detailed

discussion of the specificity of gaze behavior in ASD can be found elsewhere (Falck-Ytter and
29
Page 29 of 57
von Hofsten, 2011).

Pelphrey and colleagues repeated their fMRI study on the processing of congruent and

incongruent gaze shifts (Pelphrey et al., 2003) in autistic persons in order to disentangle

differences in the neurobiology of gaze processing (Pelphrey et al., 2005). Results demonstrated

t
ip
that the STS did not distinguish between congruent and incongruent gaze reactions in participants

cr
with ASD. This lack of a neural mechanism supporting the inference of intentions from gaze cues

might contribute critically to the deficits in social gaze behavior. These observations are

us
corroborated by a recent attentional cueing task comparing the neural responses to gaze versus

arrow cues. Results indicate that gaze cues result in greater activation of a fronto-parietal

an
attentional network only in non-autistic controls but not in subjects with ASD (Greene et al.,
M
2011), which suggests that a neural preference for gaze cues is absent in autistic persons.

Nevertheless, neuroimaging literature on social gaze in autism is still scarce. Two recent
d

studies have addressed differences in the neural mechanisms of JA using interactive techniques.
te

Redcay and co-workers used their live video feed setup to unravel neural differences underlying
p

IJA and RJA in autism (Redcay et al., 2012a). Results indicated that – unlike in controls – JA was
ce

not differentiated from solo attention by activity in the dmPFC and the pSTS. Between-group

comparisons revealed that this absence of neural differentiation was greatest in the RJA
Ac

condition. Despite the previously described lack of motivation for spontaneous initiation of JA,

there were no differences in the neural activity underlying IJA (Mundy and Newell, 2007), which

is likely to reflect the explicit instruction to cue the interaction partner’s attention to a target

location. Additional analyses showed hyper-activation of the pSTS in the ASD relative to the

control group, which possibly indicates an atypical development of this region in ASD. These

data are supported by a recent hyperscanning study in which dyads comprising an ASD and a

30
Page 30 of 57
control participant or two controls engaged in a gaze- versus target-cued RJA task (Tanabe et al.,

2012). This yielded a reduction of inter-individual coherence of intrinsic activity fluctuations in

ASD-control as compared to control-control pairs in the right IFG possibly related to decreased

mirroring of gaze behavior in these dyads. Surprisingly, intra-individual coherence of activated

t
ip
brain regions was indifferent in ASD and control subjects. However, connectivity analyses

revealed a substantial reduction of effective functional connectivity between right aSTS and the

cr
IFG in controls who were paired with ASD participants which correlated with their accuracy in

us
establishing JA following a gaze cue. Albeit speculative, the authors advocate that this is due to

the lack of shared intention in ASD-control dyads that might be reflected by the decrease of inter-

individual coherence of IFG activity.

an
M
Taken together, the study of social gaze in ASD using interactive paradigms is likely to

foster our understanding of the underlying communicative deficits in ASD, whilst simultaneously
d

providing valuable information about the neural mechanisms underlying neurotypical dynamic
te

social interactions.
p
ce
Ac

6. Towards a neurofunctional model of social gaze in interaction

This review demonstrates how methodological advances in the experimental paradigms used to

investigate social gaze have led to an increasingly refined understanding of the functional

neuroanatomy of gaze perception and social interaction. Based on these findings, we indicate

31
Page 31 of 57
directions towards a putative neurofunctional model of social gaze in interaction. Figure 2

illustrates the presumed components of such a model.

Initial studies have highlighted the importance of brain regions for perceptual processes

concerned with invariant features of faces and the eye region (e.g. Hoffman and Haxby, 2000;

t
ip
Tong et al., 2000). Later studies then began using more dynamic stimuli to also investigate

action-related aspects of gaze processing (e.g. Kuzmanovic et al., 2009; Pelphrey et al., 2004;

cr
Ramsey et al., 2011). These studies, indeed, were successful in delineating a ‘social attention

us
network’ (Nummenmaa and Calder, 2009; Nummenmaa et al., 2010) which comprises a variety

of brain regions, but may be subdivided into two subsystems thought to subserve different

an
functions: the 'dorsal attention network’ (including fronto-parietal cortical regions) has been

described as the neural correlate of attentional top-down control mechanisms, which are assumed
M
to bias the processing of stimulus features in such a way that they allow a selection of them based
d

on internal goals or expectations and a link to appropriate motor responses (Corbetta et al., 2008).
te

The 'ventral attention network’ (including IFG and anterior insula) is thought to serve as a circuit

breaker, which helps to reconfigure attention-related networks and can thereby promote rapid
p

attentional reorienting in order to respond to behaviorally relevant targets (Sridharan et al., 2008).
ce

However, a network purely dedicated to social attention is insufficient as it neither

Ac

incorporates affective components nor social-cognitive functions of gaze-based interaction.

Interestingly, the social attention network also bears some resemblance with those two large-scale

neural networks that have gained center stage as the neural substrates of social cognition, i.e. the

‘mentalizing network’ and the ‘mirror neuron system’ (Van Overwalle and Baetens, 2009).

Mainly due to methodological constraints, previous research, however, has not been able to

address the precise roles of these two large-scale neural networks during ongoing social

interactions (Schilbach, 2010). Consequently, it has remained unclear how activity in these
32
Page 32 of 57
networks might be modulated by the extent to which a person feels to be part of an ongoing

interaction and whether the networks may play complementary or mutually exclusive roles in this

case. Also, it is not entirely clear whether real-time behavioral parameters of interpersonal

coupling such as gaze recurrence (Richardson and Dale, 2005) are correlated to activity changes

t
ip
in these networks and what their relation could be to models of social cognition that draw upon

the notion of predictive coding (Zaki, 2013). The methodological advances and studies discussed

cr
in this review provide means to circumvent these problems and to elucidate the precise roles of

us
these networks during online interactions. In particular, these studies have shown that gaze in

social interaction can serve as a powerful social reward, whose experience can be pleasant and

an
motivating. Having an effect on someone else's gaze behavior has concomitantly been shown to

activate reward-related neurocircuitry, such as the ventral striatum (Schilbach et al., 2010).
M
Furthermore, the ventral striatum is also closely connected to hubs of the networks described
d

above: for instance, measures of connectivity between VS and mPFC have been related to self-
te

evaluations in comparison to conspecifics (Yamada et al., 2013) and are likely to represent a

driving force behind the human motivation to engage in social interaction (Lebreton et al., 2009).
p

Recent studies also show inverse relationships between activity in the VS and another
ce

brain region that has been implicated in gaze processing, the amygdala. Whereas VS appears to
Ac

be more relevant for processing the hedonic aspects of gaze, the amygdala - apart from its more

general function as a salience detector - has been linked to processing aversive aspects of gaze.

VS and amygdala might, therefore, implement a subcortical system of 'checks and balances',

which biases processing of the affective and motivational aspects of social gaze. Recent studies

are also beginning to provide insights concerning the anatomical connections of brain regions

relevant for gaze processing. Ethofer et al. (2011) have been able to show that dynamic gaze

shifts toward a human observer not only increase activity in the pSTS, but also functional
33
Page 33 of 57
connectivity with the anterior insula cortex. For this connection, Ethofer and colleagues were also

able to demonstrate direct fiber connections by means of DTI measurements. In addition to this,

other recent work has related connectivity differences in the 'social attention' network to

differences in autistic traits in the neurotypical population (Nummenmaa et al., 2012; von dem

t
ip
Hagen et al., 2011) while even more pronounced structural and neurofunctional differences have

been detected in individuals with ASD (e.g. Greene et al., 2011; Pitskel et al., 2011; von dem

cr
Hagen et al., 2013). Intriguingly, based on the social motivation theory of autism (Chevallier et

us
al., 2012), recent findings also implicate alterations of reward-related neurocircuitry in autism

(e.g. Dichter et al., 2012; Kohls et al., 2012) but have so far not been able to connect this to the

an
literature on social attention and gaze processing. Here, a more comprehensive network model

targeting social gaze in interaction might help to provide a more parsimonious account.
M
In sum, the work presented in this review builds upon, but also extends previous accounts of
d

a neural network dedicated to social attention by taking into account interaction-specific

te

additions to this model, which highlight the involvement of additional brain regions related to

affective and motivational processes and relevant connectivity patterns that may be crucial for
p

successful participation in real-time social interaction.

ce
Ac

7. Conclusion

The present review illustrates that the development and application of interactive methodologies

to study social gaze provides an innovative and powerful tool in social-cognitive neuroscience. It

enables scientists to go beyond studying social cognition as a detached and observational process

34
Page 34 of 57
by obtaining behavioral and neural measures while participants are active participants in

dynamic, real-time social interactions. While this work is still in its infancy, the various different

implementations of interactive gaze paradigms which have been developed over the past years

provide a sound basis for uncovering the neural mechanisms sustaining social gaze behavior. Due

t
ip
to the dynamic nature of social gaze, this will provide broad access to the neural basis of our

ability to engage in social interaction. While first implications for network models of social gaze

cr
have been discussed in the previous chapter, future studies will hopefully enable the development

us
and test of a more complete neurofunctional model of social gaze in interaction while

simultaneously fostering our understanding of the core deficits underlying ASD and possibly

an
other psychopathologies involving impairments of social interaction. Moreover, the application

of interactive technology may not only have diagnostic value, but could also be used to provide
M
therapeutic tools for the training of social interactions in these disorders (e.g. Bellani et al., 2011;
d

Lahiri et al., 2011).

p te
ce
Ac

Acknowledgments

35
Page 35 of 57
This work is partly based on the PhD thesis by U.P., which is supported by grants of the Köln

Fortune Program of the Medical Faculty at the University of Cologne and the Volkswagen

Foundation to L.S.

t
ip
cr
us
an
M
d
p te
ce
Ac

36
Page 36 of 57
 t
ip
cr
us
References

an
Adams Jr., R.B., Kleck, R.E., 2005. Effects of Direct and Averted Gaze on the Perception of

Facially Communicated Emotion. Emotion 5, 3–11.

M
Amodio, D.M., Frith, C.D., 2006. Meeting of minds: the medial frontal cortex and social

cognition. Nat. Rev. Neurosci. 7, 268–277.

d

Argyle, M., Ingham, R., Alkema, F., McCallin, M., 1973. The Different Functions of Gaze.
te

Semiotica 7, 19–32.
p

Babiloni, F., Astolfi, L., in press. Social neuroscience and hyperscanning techniques: Past,
ce

present and future. Neurosci. Biobehav. Rev.

Baess, P., Zhdanov, A., Hirvenkari, L., Mäkelä, J.P., Jousmäki, V., Hari, R., 2012. MEG dual
Ac

scanning: a procedure to study real-time auditory interaction between two persons. Front.

Neurosci. 6, 83.

Bailenson, J.N., Blascovich, J., Beall, A.C., Loomis, J.M., 2003. Interpersonal distance in

immersive virtual environments. Pers. Soc. Psychol. Bull. 29, 819–833.

Barisic, I., Timmermans, B., Pfeiffer, U.J., Vogeley, K., Schilbach, L., 2013. Using dual eye-

tracking to investigate real-time social interactions, in: Proceedings of the ACM SIGCHI

37
Page 37 of 57
 Conference on Human Factors in Computing Systems. Presented at the ACM SIGCHI

Conference on Human Factors in Computing Systems, Paris.

Baron-Cohen, S., Belmonte, M.K., 2005. Autism: a window onto the development of the social

and the analytic brain. Annu. Rev. Neurosci. 28, 109–126.

t
ip
Bates, E., Camaioni, L., Volterra, V., 1975. The acquisition of performatives prior to speech.

Merrill Palmer Quart. 21, 205–226.

cr
Bayliss, A.P., Murphy, E., Naughtin, C.K., Kritikos, A., Schilbach, L., Becker, S.I., 2012. “Gaze

us
Leading”: Initiating Simulated Joint Attention Influences Eye Movements and Choice

Behavior. J. Exp. Psychol. Gen.

an
Becchio, C., Sartori, L., Castiello, U., 2010. Toward You: The Social Side of Actions. Curr. Dir.

Psychol. Sci. 19, 183–188.

M
Bedford, R., Elsabbagh, M., Gliga, T., Pickles, A., Senju, A., Charman, T., Johnson, M.H.,
d

BASIS team, 2012. Precursors to social and communication difficulties in infants at-risk
te

for autism: gaze following and attentional engagement. J. Autism Dev. Disord. 42, 2208–

2218.
p

Bellani, M., Fornasari, L., Chittaro, L., Brambilla, P., 2011. Virtual reality in autism: state of the
ce

art. Epidemiol. Psychiatr. Sci. 20, 235–238.

Ac

Bente, G., Eschenburg, F., Krämer, N.C., 2007. Virtual Gaze. A Pilot Study on the Effects of

Computer Simulated Gaze in Avatar-Based Conversations, in: Shumaker, R. (Ed.),

Virtual Reality. Springer, Berlin, Heidelberg, pp. 185–194.

Berry, D.S., McArthur, L.Z., 1986. Perceiving character in faces: The impact of age-related

craniofacial changes on social perception. Psychol. Bull. 100, 3–18.

Boraston, Z., Blakemore, S.-J., 2007. The application of eye-tracking technology in the study of

autism. J. Physiol. 581, 893–898.

38
Page 38 of 57
Bristow, D., Rees, G., Frith, C.D., 2007. Social interaction modifies neural response to gaze

shifts. Soc. Cogn. Affect. Neurosci. 2, 52–61.

Carletta, J., Hill, R.L., Nicol, C., Taylor, T., de Ruiter, J.P., Bard, E.G., 2010. Eyetracking for

two-person tasks with manipulation of a virtual world. Behav. Res. Methods 42, 254–265.

t
ip
Carlin, J.D., Calder, A.J., in press. The neural basis of eye gaze processing. Curr. Opin.

Neurobiol.

cr
Carlin, J.D., Calder, A.J., Kriegeskorte, N., Nili, H., Rowe, J.B., 2011. A head view-invariant

us
representation of gaze direction in anterior superior temporal sulcus. Curr. Biol. 21, 1817–

1821.

an
Carlin, J.D., Rowe, J.B., Kriegeskorte, N., Thompson, R., Calder, A.J., 2012. Direction-sensitive

codes for observed head turns in human superior temporal sulcus. Cereb. Cortex 22, 735–
M
744.
d

Carpenter, M., Nagell, K., Tomasello, M., 1998. Social cognition, joint attention, and
te

communicative competence from 9 to 15 months of age. Monogr. Soc. Res. Child 63, i–

vi, 1–143.
p

Cary, M.S., 1978. The Role of Gaze in the Initiation of Conversation. Soc. Psychol. 41, 269.
ce

Chevallier, C., Kohls, G., Troiani, V., Brodkin, E.S., Schultz, R.T., 2012. The social motivation
Ac

theory of autism. Trends Cogn. Sci. 16, 231–239.

Clark, H.H., 1996. Using language, 2nd ed. Cambridge University Press, Cambridge, UK.

Corbetta, M., Patel, G., Shulman, G.L., 2008. The reorienting system of the human brain: from

environment to theory of mind. Neuron 58, 306–324.

Corbetta, M., Shulman, G.L., 2002. Control of goal-directed and stimulus-driven attention in the

brain. Nat. Rev. Neurosci. 3, 201–215.

39
Page 39 of 57
Dalton, K.M., Nacewicz, B.M., Johnstone, T., Schaefer, H.S., Gernsbacher, M.A., Goldsmith,

H.H., Alexander, A.L., Davidson, R.J., 2005. Gaze fixation and the neural circuitry of

face processing in autism. Nat. Neurosci. 8, 519–526.

De Jaegher, H., Di Paolo, E., Gallagher, S., 2010. Can social interaction constitute social

t
ip
cognition? Trends Cogn. Sci. 14, 441–447.

De Jaegher, H.D., Di Paolo, E.D., 2007. Participatory sense-making. Phenom. Cogn. Sci. 6, 485–

cr
507.

us
Delgado, M.R., 2007. Reward-Related Responses in the Human Striatum. Ann. N. Y. Acad. Sci.

1104, 70–88.

an
Dichter, G.S., Richey, J.A., Rittenberg, A.M., Sabatino, A., Bodfish, J.W., 2012. Reward

circuitry function in autism during face anticipation and outcomes. J Autism Dev Disord
M
42, 147–160.
d

Duchowski, A.T., 2007. Eye tracking methodology: theory and practice. Springer, London.
te

Duchowski, A.T., Cournia, N., Murphy, H., 2004. Gaze-Contingent Displays: A Review.

Cyberpsychol. Behav. 7, 621–634.

p

Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., Garnero, L., 2010. Inter-brain
ce

synchronization during social interaction. PLoS ONE 5, e12166.

Ac

Ekman, P., Oster, H., 1979. Facial Expressions of Emotion. Annu. Rev. Psychol. 30, 527–554.

Emery, N.J., 2000. The eyes have it: the neuroethology, function and evolution of social gaze.

Neurosci. Biobehav. Rev. 24, 581–604.

Ethofer, T., Gschwind, M., Vuilleumier, P., 2011. Processing social aspects of human gaze: a

combined fMRI-DTI study. Neuroimage 55, 411–419.

Falck-Ytter, T., von Hofsten, C., 2011. How special is social looking in ASD: a review. Prog.

Brain Res. 189, 209–222.

40
Page 40 of 57
Fox, J., Arena, D., Bailenson, J.N., 2009. Virtual reality. J. Media Psychol. 21, 95–113.

Friesen, C.K., Kingstone, A., 1998. The eyes have it! Reflexive orienting is triggered by

nonpredictive gaze. Psychon. Bull. Rev. 5, 490–495.

Frischen, A., Bayliss, A.P., Tipper, S.P., 2007. Gaze cueing of attention: visual attention, social

t
ip
cognition, and individual differences. Psychol. Bull. 133, 694–724.

Froese, T., Fuchs, T., 2012. The extended body: a case study in the neurophenomenology of

cr
social interaction. Phenom. Cogn. Sci. 11, 205–235.

us
Gallagher, H.L., Happé, F., Brunswick, N., Fletcher, P.C., Frith, U., Frith, C.D., 2000. Reading

the mind in cartoons and stories: an fMRI study of “theory of mind” in verbal and

nonverbal tasks. Neuropsychologia 38, 11–21.

an
George, N., Conty, L., 2008. Facing the gaze of others. Neurophysiol. Clin. 38, 197–207.
M
Gibson, J.J., Pick, A.D., 1963. Perception of another person’s looking behavior. Am. J. Psychol.
d

76, 386–394.
te

Gillath, O., McCall, C., Shaver, P.R., Blascovich, J., 2008. What can virtual reality teach us about

prosocial tendencies in real and virtual environments? Media Psychol. 11, 259–282.
p

Granholm, E., Steinhauer, S.R., 2004. Pupillometric measures of cognitive and emotional
ce

processes. Int. J. Psychophysiol. 52, 1–6.

Ac

Grant, E.C., 1969. Human Facial Expression. Man 4, 525–692.

Greene, D.J., Colich, N., Iacoboni, M., Zaidel, E., Bookheimer, S.Y., Dapretto, M., 2011.

Atypical neural networks for social orienting in autism spectrum disorders. Neuroimage

56, 354–362.

Grynszpan, O., Nadel, J., Martin, J.-C., Simonin, J., Bailleul, P., Wang, Y., Gepner, D., Le

Barillier, F., Constant, J., 2011. Self-Monitoring of Gaze in High Functioning Autism. J.

Autism Dev. Disord.

41
Page 41 of 57
Grynszpan, O., Simonin, J., Martin, J.-C., Nadel, J., 2012. Investigating social gaze as an action-

perception online performance. Front. Neurosci. 6, 94.

Haith, M.M., Bergman, T., Moore, M.J., 1977. Eye contact and face scanning in early infancy.

Science 198, 853–855.

t
ip
Haxby, J.V., Hoffman, E.A., Gobbini, M.I., 2002. Human neural systems for face recognition and

social communication. Biol. Psychiatry 51, 59–67.

cr
Hein, G., Knight, R.T., 2008. Superior temporal sulcus--It’s my area: or is it? J. Cogn. Neurosci.

us
20, 2125–2136.

Hernandez, N., Metzger, A., Magné, R., Bonnet-Brilhault, F., Roux, S., Barthelemy, C.,

an
Martineau, J., 2009. Exploration of core features of a human face by healthy and autistic

adults analyzed by visual scanning. Neuropsychologia 47, 1004–1012.

M
Hobson, R.P., 1991. Against the theory of “Theory of Mind”. Br. J. Dev. Psychol. 9, 33–51.
d

Hoffman, E.A., Haxby, J.V., 2000. Distinct representations of eye gaze and identity in the
te

distributed human neural system for face perception. Nat. Neurosci. 3, 80–84.

Hood, B.M., Willen, J.D., Driver, J., 1998. Adult’s Eyes Trigger Shifts of Visual Attention in
p

Human Infants. Psychol. Sci. 9, 131–134.

ce

Hoyt, C.L., Blascovich, J., Swinth, K.R., 2003. Social inhibition in immersive virtual
Ac

environments. Presence-Teleop. Virt. 12, 183–195.

Itier, R.J., Batty, M., 2009. Neural bases of eye and gaze processing: the core of social cognition.

Neurosci. Biobehav. Rev. 33, 843–863.

Kahneman, D., Beatty, J., 1966. Pupil diameter and load on memory. Science 154, 1583–1585.

Kampe, K.K.W., Frith, C.D., Frith, U., 2003. “Hey John”: signals conveying communicative

intention toward the self activate brain regions associated with “mentalizing,” regardless

of modality. J. Neurosci. 23, 5258–5263.

42
Page 42 of 57
Karatekin, C., Couperus, J.W., Marcus, D.J., 2004. Attention allocation in the dual-task paradigm

as measured through behavioral and psychophysiological responses. Psychophysiology

41, 175–185.

Kleinhans, N.M., Richards, T., Sterling, L., Stegbauer, K.C., Mahurin, R., Johnson, L.C.,

t
ip
Greenson, J., Dawson, G., Aylward, E., 2008. Abnormal functional connectivity in autism

spectrum disorders during face processing. Brain 131, 1000–1012.

cr
Klin, A., Jones, W., Schultz, R., Volkmar, F., Cohen, D., 2002. Defining and quantifying the

us
social phenotype in autism. Am. J. Psychiatry 159, 895–908.

Kobayashi, H., Kohshima, S., 2001. Unique morphology of the human eye and its adaptive

an
meaning: comparative studies on external morphology of the primate eye. J. Hum. Evol.

40, 419–435.
M
Kohls, G., Schulte-Rüther, M., Nehrkorn, B., Müller, K., Fink, G.R., Kamp-Becker, I., Herpertz-
d

Dahlmann, B., Schultz, R.T., Konrad, K., 2012. Reward system dysfunction in autism
te

spectrum disorders. Soc. Cogn. Affect. Neurosci.

Konvalinka, I., Roepstorff, A., 2012. The two-brain approach: how can mutually interacting
p

brains teach us something about social interaction? Front. Neurosci. 6, 215.

ce

Kuzmanovic, B., Georgescu, A.L., Eickhoff, S.B., Shah, N.J., Bente, G., Fink, G.R., Vogeley, K.,
Ac

2009. Duration matters: dissociating neural correlates of detection and evaluation of

social gaze. Neuroimage 46, 1154–1163.

Lachat, F., Hugueville, L., Lemaréchal, J.-D., Conty, L., George, N., 2012. Oscillatory Brain

Correlates of Live Joint Attention: A Dual-EEG Study. Front. Neurosci. 6, 156.

Laeng, B., Sirois, S., Gredebäck, G., 2012. Pupillometry A Window to the Preconscious?

Perspect. Psychol. Sci. 7, 18–27.

43
Page 43 of 57
Lahiri, U., Warren, Z., Sarkar, N., 2011. Design of a gaze-sensitive virtual social interactive

system for children with autism. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 443–452.

Langton, S.R.H., Watt, R.J., Bruce, V., 2000. Do the eyes have it? Cues to the direction of social

attention. Trends Cogn. Sci. 4, 50–59.

t
ip
Leavens, D., Racine, T.P., 2009. Joint Attention in Apes and Humans: Are Humans Unique? J.

Conscious. Stud. 16, 240–267.

cr
Lebreton, M., Barnes, A., Miettunen, J., Peltonen, L., Ridler, K., Veijola, J., Tanskanen, P.,

us
Suckling, J., Jarvelin, M.-R., Jones, P.B., Isohanni, M., Bullmore, E.T., Murray, G.K.,

2009. The brain structural disposition to social interaction. Eur. J. Neurosci. 29, 2247–

2252.

an
Marsh, K.L., Richardson, M.J., Schmidt, R.C., 2009. Social Connection Through Joint Action
M
and Interpersonal Coordination. Topics Cog. Sci. 1, 320–339.
d

Martineau, J., Hernandez, N., Hiebel, L., Roché, L., Metzger, A., Bonnet-Brilhault, F., 2011. Can
te

pupil size and pupil responses during visual scanning contribute to the diagnosis of autism

spectrum disorder in children? J. Psychiat. Res. 45, 1077–1082.

p

Mason, M.F., Hood, B.M., Macrae, C.N., 2004. Look into my eyes: gaze direction and person
ce

memory. Memory 12, 637–643.

Ac

Mason, M.F., Tatkow, E.P., Macrae, C.N., 2005. The look of love: gaze shifts and person

perception. Psychol. Sci. 16, 236–239.

Materna, S., Dicke, P.W., Thier, P., 2008. Dissociable roles of the superior temporal sulcus and

the intraparietal sulcus in joint attention: a functional magnetic resonance imaging study.

J. Cogn. Neurosci. 20, 108–119.

Moll, H., Tomasello, M., 2007. Cooperation and human cognition: the Vygotskian intelligence

hypothesis. Phil. Trans. R. Soc. B 362, 639–648.

44
Page 44 of 57
Montague, P.R., Berns, G.S., Cohen, J.D., McClure, S.M., Pagnoni, G., Dhamala, M., Wiest,

M.C., Karpov, I., King, R.D., Apple, N., Fisher, R.E., 2002. Hyperscanning: simultaneous

fMRI during linked social interactions. Neuroimage 16, 1159–1164.

Mundy, P., 2003. Annotation: The neural basis of social impairments in autism: the role of the

t
ip
dorsal medial-frontal cortex and anterior cingulate system. J. Child Psychol. Psychiatry

44, 793–809.

cr
Mundy, P., Newell, L., 2007. Attention, joint attention, and social cognition. Curr. Dir. Psychol.

us
Sci. 16, 269–274.

Murphy, E.R., Foss-Feig, J., Kenworthy, L., Gaillard, W.D., Vaidya, C.J., 2012. Atypical

an
Functional Connectivity of the Amygdala in Childhood Autism Spectrum Disorders

during Spontaneous Attention to Eye-Gaze. Autism Res. Treat. 2012, 652408.

M
Murray, L., Trevarthen, C., 1985. Emotional regulation of interaction between two-month-olds
d

and their mothers, in: Field, T., Fox, N. (Eds.), Social Perception in Infants. Ablex
te

Publishing, New Jersey, pp. 101–125.

Naeem, M., Prasad, G., Watson, D.R., Kelso, J.A.S., 2012. Electrophysiological signatures of
p

intentional social coordination in the 10-12 Hz range. Neuroimage 59, 1795–1803.

ce

Newman-Norlund, R.D., van Schie, H.T., van Zuijlen, A.M.J., Bekkering, H., 2007. The mirror
Ac

neuron system is more active during complementary compared with imitative action. Nat.

Neurosci. 10, 817–818.

Nuku, P., Bekkering, H., 2008. Joint attention: Inferring what others perceive (and don’t

perceive). Conscious. Cogn. 17, 339–349.

Nummenmaa, L., Calder, A.J., 2009. Neural mechanisms of social attention. Trends Cogn. Sci.

13, 135–143.

45
Page 45 of 57
Nummenmaa, L., Engell, A.D., von dem Hagen, E., Henson, R.N.A., Calder, A.J., 2012. Autism

spectrum traits predict the neural response to eye gaze in typical individuals. Neuroimage

59, 3356–3363.

Nummenmaa, L., Passamonti, L., Rowe, J., Engell, A.D., Calder, A.J., 2010. Connectivity

t
ip
Analysis Reveals a Cortical Network for Eye Gaze Perception. Cereb. Cortex 20, 1780–

1787.

cr
Pelphrey, K.A., Morris, J.P., 2006. Brain Mechanisms for Interpreting the Actions of Others

us
From Biological-Motion Cues. Curr. Dir. Psychol. Sci. 15, 136–140.

Pelphrey, K.A., Morris, J.P., McCarthy, G., 2005. Neural basis of eye gaze processing deficits in

autism. Brain 128, 1038–1048.

an
Pelphrey, K.A., Sasson, N.J., Reznick, J.S., Paul, G., Goldman, B.D., Piven, J., 2002. Visual
M
scanning of faces in autism. J. Autism Dev. Disord. 32, 249–261.
d

Pelphrey, K.A., Singerman, J.D., Allison, T., McCarthy, G., 2003. Brain activation evoked by
te

perception of gaze shifts: the influence of context. Neuropsychologia 41, 156–170.

Pelphrey, K.A., Viola, R.J., McCarthy, G., 2004. When Strangers Pass Processing of Mutual and
p

Averted Social Gaze in the Superior Temporal Sulcus. Psychol. Sci. 15, 598–603.
ce

Peters, J., Büchel, C., 2010. Neural representations of subjective reward value. Behav. Brain Res.
Ac

213, 135–141.

Pfeiffer, U., Schilbach, L., Timmermans, B., Jording, M., Bente, G., Vogeley, K., 2012. Eyes on

the mind: Investigating the influence of gaze dynamics on the perception of others in real-

time social interaction. Front. Psychol. 3, 537.

Pfeiffer, U.J., Timmermans, B., Bente, G., Vogeley, K., Schilbach, L., 2011. A non-verbal Turing

test: Differentiating mind from machine in gaze-based social interaction. PLoS ONE 6,

e27591.
46
Page 46 of 57
Pfeiffer, U.J., Timmermans, B., Vogeley, K., Frith, C.D., Schilbach, L., 2013. Towards a

neuroscience of social interaction. Front. Neurosci. 7, 22.

Pineda, J.A., 2005. The functional significance of mu rhythms: Translating “seeing” and

“hearing” into “doing”. Brain. Res. Rev. 50, 57–68.

t
ip
Pitskel, N.B., Bolling, D.Z., Hudac, C.M., Lantz, S.D., Minshew, N.J., Vander Wyk, B.C.,

Pelphrey, K.A., 2011. Brain mechanisms for processing direct and averted gaze in

cr
individuals with autism. J. Autism Dev. Disord. 41, 1686–1693.

us
Ramsey, R., Cross, E.S., Hamilton, A.F. de C., 2011. Eye can see what you want: posterior

intraparietal sulcus encodes the object of an actor’s gaze. J. Cogn. Neurosci. 23, 3400–

3409.

an
Redcay, E., 2008. The superior temporal sulcus performs a common function for social and
M
speech perception: implications for the emergence of autism. Neurosci. Biobehav. Rev.
d

32, 123–142.
te

Redcay, E., Dodell-Feder, D., Mavros, P.L., Kleiner, M., Pearrow, M.J., Triantafyllou, C.,

Gabrieli, J.D., Saxe, R., 2012a. Atypical brain activation patterns during a face-to-face
p

joint attention game in adults with autism spectrum disorder. Hum. Brain Mapp.
ce

Redcay, E., Dodell-Feder, D., Pearrow, M.J., Mavros, P.L., Kleiner, M., Gabrieli, J.D.E., Saxe,
Ac

R., 2010. Live face-to-face interaction during fMRI: A new tool for social cognitive

neuroscience. Neuroimage 50, 1639–1647.

Redcay, E., Kleiner, M., Saxe, R., 2012b. Look at this: The neural correlates of initiating and

responding to bids for joint attention. Front. Neurosci. 6, 169.

Redcay, E., Saxe, R., 2013. Do you see what I see? The neural bases of joint attention., in:

Metcalfe, J., Terrace, H.S. (Eds.), Agency and Joint Attention. Oxford University Press,

USA, New York.

47
Page 47 of 57
Reddy, V., Morris, P., 2004. Participants Don’t Need Theories Knowing Minds in Engagement.

Theory Psychology 14, 647–665.

Richardson, D.C., Dale, R., 2005. Looking To Understand: The Coupling Between Speakers’ and

Listeners’ Eye Movements and Its Relationship to Discourse Comprehension. Cognitive

t
ip
Sci. 29, 1045–1060.

Riera, J., Bosch, J., Yamashita, O., Kawashima, R., Sadato, N., Okada, T., Ozaki, T., 2004. fMRI

cr
activation maps based on the NN-ARx model. Neuroimage 23, 680–697.

us
Riley, M.A., Richardson, M.J., Shockley, K., Ramenzoni, V.C., 2011. Interpersonal Synergies.

Front. Psychol. 2.

an
Rizzolatti, G., Sinigaglia, C., 2010. The functional role of the parieto-frontal mirror circuit:

interpretations and misinterpretations. Nat. Rev. Neurosci. 11, 264–274.

M
Saito, D.N., Tanabe, H.C., Izuma, K., Hayashi, M.J., Morito, Y., Komeda, H., Uchiyama, H.,
d

Kosaka, H., Okazawa, H., Fujibayashi, Y., Sadato, N., 2010. “Stay tuned”: inter-
te

individual neural synchronization during mutual gaze and joint attention. Front. Neurosci.

4, 127.
p

Saxe, R., 2006. Uniquely human social cognition. Curr. Opin. Neurobiol. 16, 235–239.
ce

Schilbach, L., 2010. A second-person approach to other minds. Nat. Rev. Neurosci. 11, 449.
Ac

Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., Vogeley, K., in

press. Toward a second-person neuroscience. Behav. Brain Sci.

Schilbach, L., Wilms, M., Eickhoff, S.B., Romanzetti, S., Tepest, R., Bente, G., Shah, N.J., Fink,

G.R., Vogeley, K., 2010. Minds made for sharing: initiating joint attention recruits

reward-related neurocircuitry. J. Cogn. Neurosci. 22, 2702–2715.

48
Page 48 of 57
Schilbach, L., Wohlschlaeger, A.M., Kraemer, N.C., Newen, A., Shah, N.J., Fink, G.R., Vogeley,

K., 2006. Being with virtual others: Neural correlates of social interaction.

Neuropsychologia 44, 718–730.

Sebanz, N., Bekkering, H., Knoblich, G., 2006. Joint action: bodies and minds moving together.

t
ip
Trends Cogn. Sci. 10, 70–76.

Senju, A., Johnson, M.H., 2009. The eye contact effect: mechanisms and development. Trends

cr
Cogn. Sci. 13, 127–134.

us
Shepherd, S.V., 2010. Following gaze: gaze-following behavior as a window into social

cognition. Front. Neurosci. 4, 5.

an
Sridharan, D., Levitin, D.J., Menon, V., 2008. A critical role for the right fronto-insular cortex in

switching between central-executive and default-mode networks. Proc. Natl. Acad. Sci.
M
U.S.A. 105, 12569–12574.
d

Steinhauer, S.R., Boller, F., Zubin, J., Pearlman, S., 1983. Pupillary dilation to emotional visual
te

stimuli revisited. Psychophysiology 20.

Strick, M., Holland, R.W., van Knippenberg, A., 2008. Seductive eyes: attractiveness and direct
p

gaze increase desire for associated objects. Cognition 106, 1487–1496.

ce

Tanabe, H.C., Kosaka, H., Saito, D.N., Koike, T., Hayashi, M.J., Izuma, K., Komeda, H.,
Ac

Ishitobi, M., Omori, M., Munesue, T., Okazawa, H., Wada, Y., Sadato, N., 2012. Hard to

“tune in”: neural mechanisms of live face-to-face interaction with high-functioning

autistic spectrum disorder. Front. Neurosci. 6, 268.

Tognoli, E., Lagarde, J., DeGuzman, G.C., Kelso, J.A.S., 2007. The phi complex as a

neuromarker of human social coordination. Proc. Natl. Acad. Sci. U.S.A. 104, 8190–

8195.

49
Page 49 of 57
Tomasello, M., Carpenter, M., 2005. The Emergence of Social Cognition in Three Young

Chimpanzees. Monogr. Soc. Res. Child 70, i–152.

Tomasello, M., Carpenter, M., Call, J., Behne, T., Moll, H., 2005. Understanding and sharing

intentions: The origins of cultural cognition. Behav. Brain Sci. 28, 675–691.

t
ip
Tong, F., Nakayama, K., Moscovitch, M., Weinrib, O., Kanwisher, N., 2000. Response properties

of the human fusiform face area. Cogn. Neuropsychol. 17, 257–280.

cr
Van der Molen, M.W., Boomsma, D.I., Jennings, J.R., Nieuwboer, R.T., 1989. Does the heart

us
know what the eye sees? A cardiac/pupillometric analysis of motor preparation and

response execution. Psychophysiology 26, 70–80.

an
Van Overwalle, F., Baetens, K., 2009. Understanding others’ actions and goals by mirror and

mentalizing systems: a meta-analysis. Neuroimage 48, 564–584.

M
Vogeley, K., Bente, G., 2010. “Artificial humans”: Psychology and neuroscience perspectives on
d

embodiment and nonverbal communication. Neural Netw. 23, 1077–1090.

te

Vogeley, K., Bussfeld, P., Newen, A., Herrmann, S., Happé, F., Falkai, P., Maier, W., Shah, N.J.,

Fink, G.R., Zilles, K., 2001. Mind reading: neural mechanisms of theory of mind and self-
p

perspective. Neuroimage 14, 170–181.

ce

Von dem Hagen, E.A.H., Nummenmaa, L., Yu, R., Engell, A.D., Ewbank, M.P., Calder, A.J.,
Ac

2011. Autism spectrum traits in the typical population predict structure and function in the

posterior superior temporal sulcus. Cereb. Cortex 21, 493–500.

Von dem Hagen, E.A.H., Stoyanova, R.S., Rowe, J.B., Baron-Cohen, S., Calder, A.J., 2013.

Direct Gaze Elicits Atypical Activation of the Theory-of-Mind Network in Autism

Spectrum Conditions. Cereb. Cortex.

Von Grünau, M., Anston, C., 1995. The detection of gaze direction: a stare-in-the-crowd effect.

Perception 24, 1297–1313.

50
Page 50 of 57
Vuilleumier, P., George, N., Lister, V., Armony, J., Driver, J., 2005. Effects of perceived mutual

gaze and gender on face processing and recognition memory. Vis. Cogn. 12, 85–101.

Walker-Smith, G.J., Gale, A.G., Findlay, J.M., 1977. Eye movement strategies involved in face

perception. Perception 6, 313 – 326.

t
ip
Walter, H., Adenzato, M., Ciaramidaro, A., Enrici, I., Pia, L., Bara, B.G., 2004. Understanding

intentions in social interaction: the role of the anterior paracingulate cortex. J. Cogn.

cr
Neurosci. 16, 1854–1863.

us
Wellens, D.A.R., 1982. Applying computerized interactive video technology to nonverbal

communication research. J. Nonverbal Behav. 7, 121–123.

an
Williams, J.H.G., Waiter, G.D., Perra, O., Perrett, D.I., Whiten, A., 2005. An fMRI study of joint

attention experience. Neuroimage 25, 133–140.

M
Wilms, M., Schilbach, L., Pfeiffer, U., Bente, G., Fink, G.R., Vogeley, K., 2010. It’s in your eyes
d

- Using gaze-contingent stimuli to create truly interactive paradigms for social cognitive
te

and affective neuroscience. Soc. Cogn. Affect. Neurosci. 5, 98–107.

Yamada, M., Uddin, L.Q., Takahashi, H., Kimura, Y., Takahata, K., Kousa, R., Ikoma, Y.,
p

Eguchi, Y., Takano, H., Ito, H., Higuchi, M., Suhara, T., 2013. Superiority illusion arises
ce

from resting-state brain networks modulated by dopamine. Proc. Natl. Acad. Sci. U.S.A.
Ac

110, 4363–4367.

Zaki, J., 2013. Cue Integration A Common Framework for Social Cognition and Physical

Perception. Perspect. Psychol. Sci. 8, 296–312.

51
Page 51 of 57
 t
ip
cr
us
Figure Legends

Figure 1: The core processes of social gaze.

an
M
This figure indicates the core components of social gaze behavior (adapted from Emery, 2000).
d

Green arrows indicate mutual awareness, blue arrows indicate unidirectional awareness. See
te

chapter 2 for details on distinct processes.

p
ce

Figure 2: Brain regions involved in the processing of social gaze.

Ac

Regions depicted in blue are related to motivational and affective components of social gaze.

Regions involved in mentalizing are depicted in orange, while components of the mirror neuron

system are represented in green. A: The amygdala receives inputs from low-level visual

processing areas and is activated by behaviorally relevant cues such as direct gaze. IFG: The

inferior frontal gyrus is part of the mirror neuron system. Hyperscanning revealed synchronous

activity during mutual gaze, which might indicate that this region provides a neural substrate of

common psychological ground. mPFC: The ventral portion of the medial prefrontal cortex
52
Page 52 of 57
(vmPFC) is involved in the detection of communicative intent and the processing of the affective

value of the actions of other individuals. The dorsal portion (dmPFC) is crucially involved in the

inference of mental states during social interactions. STS: The superior temporal sulcus is

functionally divided in an anterior and posterior region. The pSTS is involved in decoding the

t
ip
direction of general social attention involving head orientation and body posture, while the aSTS

is more specifically tuned to the processing of gaze-related information. TPJ: The temporo-

cr
parietal junction is known to play a general role in the encoding of the content of mental states.

us
VS: The ventral striatum serves as an intrinsic motivator of engaging others in joint attention and

drives prefrontal regions involved in mentalizing and valuation.

Figure 3: Types of interactive gaze setups.

an
M
A) Interactive eye-tracking using virtual agents. B) Gaze-based social interactions using live

video feeds (taken from Redcay et al., 2012b). C) A combination of fMRI hyperscanning and
d

dual eye-tracking (taken from Saito et al., 2010). D) Live gaze interaction in combination with
te

EEG hyperscanning (taken from Lachat et al., 2012). See chapter 3 for details on the different
p

methodologies.
ce
Ac

53
Page 53 of 57
Highlights

Gaze behavior is central to social interaction and the understanding of others

Besides passive observation, active participation in interaction must be studied

Interactive gaze paradigms allow assessing face-to-face interactions in real-time

t
Novel studies reveal neural systems supporting active social engagement

ip
Interactive approaches are the key to understanding autism spectrum disorders

cr
us
an
M
d
p te
ce
Ac

54
Page 54 of 57
Figure

i
cr
us
an
M
ed
pt
ce
Ac

Page 55 of 57
Figure

i
cr
us
an
M
ed
pt
ce
Ac

Page 56 of 57
Figure

i
cr
us
an
M
ed
pt
ce
Ac

Page 57 of 57