Multi-Modal Gesture

Charles Goodwin
Applied Linguistics UCLA
cgoodwin@humnet.ucla.edu
http://www.sscnet.ucla.edu/clic/cgoodwin.htm

A primordial site for the study of gesture consists of a situation in which multiple participants are
carrying our courses of action together while attending to 1) each other; 2) the detailed
organization of the talk in progress; 3) relevant phenomena in the environment and 4) the larger
activities that they are engaged in. Within such a framework gesture does not stand alone as a
self-contained system that can be analyzed in isolation from the other semiotic resources and
meaning practices that participants are using to build action in concert with each other. Instead
gesture occupies an interstitial position within a larger ecology of sign systems that build
meaning and action by mutually elaborating each other. This is manifest in the organization of
gesture in a variety of different ways. For example talk can be incomplete without being
problematic by virtue of the way in which part of what is being said and treated as relevant
within the utterance is accomplished through gesture. Such mutual interdependence between
gestures and other co-occurring sign systems is relevant in other ways as well. For example the
placement of gesture in relationship to other sign systems, such as the participation frameworks
constituted through the visible orientation of the actors bodies, provides methods and practices
for constituting the communicative or non-communicative status of a speakers hand movements.
In short the prototypical place where gesture emerges in the world is in the midst of a dense and
consequential set of other semiotic practices being used by participants to build relevant action in
concert with each other. This environment is not simply a haphazard collection of other semiotic
resources that happen to be present when a gesture occurs, but instead a genuine ecology of sign
systems that shape each other through their mutual interaction. Like other ecologies the
organization of these sign systems in relationship to each other can dynamically change to
maintain function and stability with confronted with a disturbance in both moment to moment
interaction and on much larger time scales. Moreover, the way in which gesture is lodged within
such an ecology demonstrates why it has a structure that is not only different from, but
complementary to that of other sign systems such as language.
In the longer presentation at the Austin 2002 conference Gesture: the Living Medium these
arguments about the structure and organization of gesture were demonstrated through analysis
drawn from videotapes recorded in a number of different settings including 1) an archeological
field excavation where gestures linking phenomena in the dirt being excavated to relevant talk
were crucial to the constitution of the professional vision of new archaeologists, that is their
mastery of how to see as an archaeologist as a form of public practice; 2) interaction between
young girls playing hopscotch where contingencies in their interaction led to rapid changes in the
structure of both gesture and the other sign systems it was linked to; 3) conversations in home of
a man with severe aphasia where gesture was crucial but lodged within an organization of sign
systems quite different that of fully fluent speakers (for example someone other than the gesturer
produced the talk that elaborated the gesture and publicly established its locally relevant
meaning). 4) lawyers gesturing at a videotape while making arguments to a jury about the guilt

and innocence of the policemen seen on the tape beating a motorist, Rodney King; and 5)
gestures within family interaction and interaction between friends talking on the street. Because
of the present space limitations only a single, quite simple strip of data will be examined here.
The following was recorded in an American home. Talk is transcribed using the system created
by Gail Jefferson (Sacks, Schegloff, & Jefferson, 1974: 731-733).
It is impossible to grasp just what the speaker is telling his recipient from the talk alone. Clearly
a major reason for this is the use in the talk of deictic terms (this and that) that instruct the hearer
to attend to phenomena beyond the stream of speech. Indeed each of these terms indexes a
gesture. Characteristically gesture is analyzed by linking what a hand is doing to the structure of
the talk in progress. Here however that is inadequate. When the gesturing hands alone are taken
into account what exactly is being talked about is still not visible:
Figure 1 Gesturing Hands

To grasp what the speaker is saying and demonstrating a hearer must take into account an object
being held by the speaker and being presented and demonstrated through the gesture. The object
here is a pitcher for a blender that the speaker has ordered over the Internet. The speaker is
telling his addressees that while the pitcher was shipped he did not receive either the top for the
pitcher, or its screw-in base. While this is not made visible through gesture and its
accompanying talk alone, it becomes vividly clear when a larger multi-modal sign complex that
encompasses not only talk and gesture, but also objects in the world is taken into account
(Streeck, 1996).
Figure 2 Gesture Linked to an Object in the World

As the speaker begins this utterance (more specifically over the word sold) his hands
noticeably grasp the pitcher. He is not grasping the pitcher to hold it (it is already well supported
by his other hand) but instead to prominently display the object to his addressees. One might
think of this hand movement as a gestural practice for presenting or indicating something, that is
as an action similar to a pointing gesture. However it is crucial to not restrict analytic focus to the
gesturing hand, but to also take into account the object in the world being grasped. As is
demonstrated a moment later this object forms a crucial part of the multi-modal signs that
display the missing parts of the blender. The gesturing hands alone fail to make visible the absent
base and lid (see Figure 1).
The co-occurring talk is equally crucial in that it formulates what is being done as describing
something absent that can be inferred from the structure of the object being held. The general
importance of the talk that elaborates a gesture is made particularly clear when the party
producing the gesture cant speak. Rather than being immediately, transparently clear, a gesture
such as this unaccompanied by relevant talk can set off a long sequence devoted to figuring out
what a speaker suffering from aphasia is trying to say to his interlocutors through the gesture
(Goodwin, 1995, 2002a).
In short what one finds here is a small ecology in which different signs in different media (talk,
the gesturing body and objects in the world) dynamically interact with each other. Each
individual sign is partial and incomplete. However, as part of a larger complex of meaning
making practices they mutually elaborate each other to create a whole, a clear statement, that is
not only different from its individual parts, but greater than them in that no sign system in
isolation makes clear what is being said.
Gestures coupled to phenomena in the environment are pervasive in many settings
(archaeological field excavations, weather forecasts, pointing to overheads in academic talks, etc.
-- consider how many computer screens are smeared with fingerprints). Gestures linked to the

environment would thus seem to constitute a major class of gesture. However with a few notable
exceptions (for example Goodwin, 2000, 2002b; Haviland, 1995; Haviland, 1998; Heath &
Hindmarsh, 2000; Hutchins & Palen, 1997; LeBaron, 1998; LeBaron & Streeck, 2000; Nevile,
2001; Streeck, 1996) multi-modal sign complexes that encompass both gesture and phenomena
in the world have been largely ignored by students of gesture. This neglect may result from the
way in which such gestures slip beyond theoretical frameworks focused on either ties between
gesture and psychological processes inside the mind of the individual speaker, or exclusively on
the talk and bodies of participants in interaction. In essence an invisible analytic boundary is
drawn at the skin of the participants. However, rather than being something that can be studied in
isolation as a neat, self contained system, gesture is an intrinsically parasitic phenomenon,
something that gets its meaning and organization from the way in which it is fluidly linked to the
other meaning making practices and sign systems that are constituting the events of the moment.
Human cognition and action are unique in the way in which they use as resources both the details
of language, and physical and cultural environments that have been shaped by human action on
an historical time scale. Typically these different kinds of phenomena are studied in isolation
from each other by separate disciplines (for example linguistics and archaeology). However
gestures interstitial position as something than links the details of language use to structure in
the environment provides a key analytic point of entry for investigation of the rich interdigitiaton
of different kinds of semiotic resources that human beings use to build relevant action in the
consequential settings that define the lifeworld of a society.
========
This is a brief report from a larger analysis that was presented at the University of Texas at
Austin conference Gesture: The Living Medium, June 5-8, 2002. The video data being analyzed
here was collected by the UCLA Center on Everyday Lives of Families, a Center on Working
Families funded by the Alfred P. Sloan Foundation. I thank Scott Phillabaum for the line
drawings used here.

References Cited
Goodwin, C. (1995). Co-Constructing Meaning in Conversations with an Aphasic Man.
Research on Language and Social Interaction, 28(3), 233-260.
Goodwin, C. (2000). Action and Embodiment Within Situated Human Interaction. Journal of
Pragmatics, 32(1489-1522).
Goodwin, C. (2002a). Conversational Frameworks for the Accomplishment of Meaning in
Aphasia. In C. Goodwin (Ed.), Situating Language Impairments Within Conversation (pp.
90+116). Oxford, New York: Oxford University Press.
Goodwin, C. (2002b). Pointing as Situated Practice. In S. Kita (Ed.), Pointing: Where Language,
Culture and Cognition Meet (pp. 217-241). Hillsdale, NJ: Lawrence Erlbaum Associates.
Haviland, J. (1995). Mental Maps and Gesture Spaces: Land, Mind and Body in Two Contexts.
Paper delived at the Conference on Gesture Compared Cross-Linguistically, Linguistic Institute,
University of New Mexico, July 7-10, 1995.
Haviland, J. B. (1998). Early Pointing Gestures in Zincantn. Journal of Linguistic
Anthropology, 8(2), 162-196.
Heath, C., & Hindmarsh, J. (2000). Configuring Action in Objects: From Mutual Space to Media
Space. Mind, Culture and Activity, 7(1&2), 81-104.

Hutchins, E., & Palen, L. (1997). Constructing Meaning from Space, Gesture, and Speech. In L.
Resnick & R. Slj & C. Pontecorvo & B. Burge (Eds.), Discourse, Tools and Reasoning: Essays
on Situated Cognition (pp. 23-40). Berlin, Heidelberg, New York: Springer-Verlag.
LeBaron, C. (1998). Building Communication: Architectural Gestures and the Embodiment of
Ideas. Ph. D. dissertation, Department of Communition,The University of Texas at Austin.
LeBaron, C. D., & Streeck, J. (2000). Gestures, Knowledge, and the World. In D. McNeill (Ed.),
Gestures in Action, Language, and Culture (pp. 118-138). Cambridge: Cambridge University
Press.
Nevile, M. (2001). Beyond the Black Box: Talk-in-Interaction in the Airline Cockpit. Ph.D.
dissertation, Department of Linguistics, Australian National University, Canberra.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A Simplest Systematics for the Organization
of Turn-Taking for Conversation. Language, 50, 696-735.
Streeck, J. (1996). How to Do Things with Things. Human Studies, 19, 365-384.