Available online at www.sciencedirect.

com

Journal of Pragmatics 41 (2009) 14591463

www.elsevier.com/locate/pragma

Book review

New Directions in the Analysis of Multimodal Discourse

Terry D. Royce and Wendy L. Bowcher (Eds.), Mahwah, NJ: Lawrence Erlbaum, 2007. ISBN 0-8058-5106-2 (cloth).
403 pp.

A rapidly waxing flow of multimodality floods publishers catalogues, journals, and conferences. This is
understandable, since discourses that draw on more than just the good old language modality present themselves in
ever more emphatic ways, due to developments such as the growing multifunctionality of cell phones, the rise of
computer games, and the huge popularity of Youtube and Internet-surfing in general. In this monomodal review I
discuss a book that promises innovative perspectives on multimodality, Royce and Bowchers edited volume New
Directions in the Analysis of Multimodal Discourse.
The coherence between the twelve chapters in the book resides mainly in the adherence to the Systemic
Functional Linguistics (SFL) approach rooted in Hallidayan grammar, while Kress and Van Leeuwens (1996/2006)
pioneering application of this approach to the visual realm is cited in every chapter. I should say straightaway that I
have no first-hand knowledge of Hallidays work. This has the drawback that I cannot verify how true to the letter or
spirit the authors are to their primary sources of inspiration, and the advantage that I can assess with relative
freshness what promising tools SFL has to offer for analyzing multimodality. Let me further add that my own
primary affiliation is to Cognitive Linguistics and Relevance Theory models, as developed by such scholars
as Lakoff, Johnson, Gibbs, Kovecses, Turner, Fauconnier, and Sperber and Wilson. However, I consider myself
a critical and eclectic follower of these paradigms. The tree is known by its fruits, and I will happily make use
of any theoretical insight that can help understand multimodality, irrespective of its source. I will first briefly, and
with no claim to exhaustiveness, discuss each chapter separately, and end by charting and evaluating shared
tendencies.
The 60 page-long first chapter by Matthiessen is devoted to The multimodal page: A Systemic Functional
exploration. After some preliminary remarks, the author proposes first to examine multimodality within language
(p. 11), because allegedly the ideational resources of language provide us with a range of strategies for modeling [. . .]
all modes and media of expression (p. 21). Then I more or less lose track. I cannot help feeling that the many
technical terms Matthiessen introduces come as a substitute for explanation, rather than as its prerequisite. The
complex panels and figures (figure 1.14 has more than 80 boxes!) are not particularly encouraging either. And while
the author makes some interesting observations on articulation and prosody as distinguishable modalities in
spoken language, and on the rhetorical patterns in verbal texts, he has very little to say about the visual realm, let alone
about interrelations between word and image.
Royce usefully begins his chapter Intersemiotic complementarity: A framework for multimodal discourse
analysis by briefly outlining the three metafunctions Halliday distinguishes in language: the ideational (pertaining
to meaning as it arises out of the relation between text and world), the interpersonal (pertaining to the relation
between communicator and addressee), and the textual (pertaining to the internal coherence of a discourse and the
way it links up to extratextual context). Royce argues that these metafunctions are applicable to non-verbal
communication as well and uses an Economist article, accompanied by a cartoon and a diagram, to demonstrate that
repetition, antonymy, hyponymy, meronymy, and collocation are concepts that define relations both
within and straddling the verbal and visual modalities on the ideational level. He comes up with similar modality-
independent concepts for the interpersonal and textual levels. Moreover, he introduces the notion of visual message
elements (VMEs), visual features which carry semantic properties [. . .] potentially realized by a variety of visual

0378-2166/$ see front matter # 2009 Elsevier B.V. All rights reserved.
doi:10.1016/j.pragma.2009.01.007
1460 Book review

techniques at the disposal of the visual designers (p. 70). The in-depth analysis of a single case-study in the light
of clearly outlined theoretical concepts provides a useful starting point for applications by other scholars to other
(types of) texts.
Thibaults chapter focuses on the visual dimensions of language. He proposes a typology of graphic dimensions of
verbal elements (e.g., a grapheme is the visual expression of the verbal letter, and logogram corresponds to
word). This is insightful, but it is a pity that the author tends to be rather verbose, and to use an overdose of jargon to
explain sensible but uncomplicated ideas. He goes on to illustrate how the levels of content and graphic expression
combined create meaning in a Duracell ad, drawing on Gestalt theory concepts (figure-ground, spatial orientation,
distance, similarity, continuity of direction, directionality or orientation, closure). Thibault ends his chapter by
showing how visually deviant language (as in comics and cartoons) can resonate with visuals in a way that enhances a
sense of rhythm that is difficult to capture in words alone.
In a fine chapter, Bateman, Delin, and Henschel convincingly expound how the impact of content shifts when it is
presented in different media. They present lists of verbal, visual, and production/consumption-related dimensions that
need minimally to be taken into account for an adequate description of how multimodal documents communicate, and
that allow for comparison between documents belonging to the same genre. What I particularly like in this research,
which is part of a larger Genre and multimodality project, is that it combines commonsense theoretical concepts
with honest, systematic data analysis, the central idea being that one can only discover interesting patterns after
identifying and quantifying salient variables. The case study, which compares traditional and electronic versions of
newspapers, elegantly underscores this point.
Baldry reports about the benefits and prospects of the Multimodal Corpus Authoring (MCA) system he
helped develop. He, too, emphasizes the importance of corpus research, and advances proposals for the labelling and
description of pertinent variables in multimodal discourse indispensable for enabling generalizations and
formulating testable hypotheses. Among other things, he discusses the issue of segmenting a text, taking into account
linguistic, prosodic, and visual features. A case study exemplifies the approach, and further cooperation is invited via
the MCA, accessible online.
Lim takes his cue from fellow contributor Martins earlier work to develop visual counterparts of verbal
organization mechanisms, specifically visual taxonomy, visual configuration, visual reference, and visual taxis, and
applies them to comics. He relies on McClouds (1993: 7072) six types of frame-transition, even though these types
are sometimes difficult to distinguish. Among the concepts he develops and discusses are the Associating Element
(e.g., a cornflakes package connotes breakfast and a tree connotes outdoor scene), and the visual linking
device (VLD), the latter enabling a viewer to understand how the action in one panel relates to that in a previous
panel, e.g., by the repetition of (part of) a character or scene.
Martin and Stenglin describe how spatial organization and layout in a post-colonial museum in New
Zealand, bridging indigenous and white perspectives on history and culture, is made communicatively
significant. Among the tools they deploy in their analyses are the given-new dichotomy, bound versus
unbound spaces, and orbital versus serial navigation paths. The authors focus on a single modality only,
but persuasively argue that this modality space needs to be taken into account in all communication that
involves physical motion.
Bowcher draws on her co-editors model for A multimodal analysis of good guys and bad guys in Rugby League
Week. Her analysis, of a newspaper article, is dauntingly detailed. Although mainly focusing on verbal aspects, the
author also discusses photographs and headings, and their location on the page. Importantly, she tries to show how
information in the verbal and visual modalities are connected. Her tentative claim that the direction in which the rugby
players in the photographs look correlates with where in the verbal text goodness and badness are discussed
is suggestive. But even if correct many more case-studies will have to be conducted to assess whether such
gaze-direction mechanisms are in any sense generalizable.
Mohan et al., claiming that the mere fact that similar graphics occur in textbooks [i.e., scientific textbooks in
different languages, ChF] does not mean that readers of those textbooks interpret them in the same way (p. 277),
asked Chinese, English, and Japanese readers to respond in their own language to a diagram representing the water
cycle. A sample of the texts was subsequently analyzed in more detail, revealing that some participants produced
texts which had a sequential line of meaning, while others produced texts that explicitly reflected causal relationships
(p. 278). The authors then hypothesize that causal line texts exemplify the features of scientific discourse more than
do time line texts (ibid.). The hypothesis is confirmed but what has been proven? Surely there is something circular
 Book review 1461

going on here: the only conclusion we can draw is that subjects who produced causal texts (Y happens because of X)
understood the water cycle better than those who produced timeline texts (Y happens after X), and that language and
cultural background had no effect on this. Ergo, some English, Japanese, and Chinese subjects, but not others, would pass
their Physics test . . .
Japanese semiotic vernaculars in ESP multiliteracies projects is Ferreiras report of a classroom task performed
by Japanese students in two consecutive years to produce a multimodal tourist brochure aimed at an English-speaking
audience. He notes, among other things, that Japanese right-left writing and the possibilities for vertical layout of text
can have (sometimes: unintended) effects on the visuals in the brochure.
Unsworth investigates how stories are affected by their translation into a different medium. Here, this classic
type of remediation research assumes some urgency because the target audience consists of children. Unsworth
emphatically presents the comparative multimodal analyses of Stellaluna, Shrek, and The Little Prince and their
CD-ROM/filmic counterparts in terms of fostering a critical pedagogic practice (p. 333). He unveils a number of
salient differences, both in the language and visuals used, which particularly in Shrek reveal that much of the prickly
unconventionality of the original has been smoothed out in the big-screen version.
Royces programmatic Multimodal communicative competence in second language contexts would have made a
good opening chapter. He takes his inspiration from the New London Group to promote the teaching of multimodal
competences by extending Dell Hymes four criteria for effective communication (grammatical/formal correctness,
ease of production and processing, appropriateness to context, and effects in the real world). For the analysis of graphic
elements he draws on a list by Donis Dondis which consists of dot, line, shape, direction, tone, color, texture, scale/
proportion, and dimension/motion. He ends the chapter by presenting a number of classroom tasks that can aid
multimodal communicative competence.
Let me signal and comment on some themes and patterns in the book. First, the positive and promising
developments I discern:

1. Most chapters insist that teaching multimodal communicative competence is a necessary component in (secondary)
education. Having to teach multimodality forces researchers and educators to develop clear-cut tools and models,
whose usefulness is demonstrated by sample analyses but which are flexible enough to be applicable to new texts.
Focusing on multimodal discourses also sharpens pupils awareness of the relative strengths and weaknesses of
different modalities. At the very least, this work outlines an important roadmap for the humanities curriculum of
the near future.
2. The laudable use of corpora, though sometimes small ones, surfaces in several chapters. By definition, detecting
patterns requires comparing things. Even comparing two things using the same measuring tool is already infinitely
more useful for theorizing than an application to a single case-study. Comparing more than two items (dozens,
hundreds) is better, of course, but given that qualitative research of multimodal texts can be very time-consuming,
this is not always feasible. A bonus of a corpus approach is that, given a clear theoretical framework, comparative
work can be divided over a group of students who are to earn credits for course work.
3. In the visual modality, spatial dimensions are crucial. Moreover, given that space involves movement it is
not surprising that several chapters mention embodiment a central concept in the Cognitive Linguistics
approach. Specifically the source-path-goal schema (discussed by Mohan et al.) has overlap with the importance
attributed to vectors in the SFL framework, and provides fruitful angles for analyzing audiovisual modalities
(see Johnson, 1987).
4. An often-quoted term in the volume is Gibsons (1979: chapter 8) notion of affordance (roughly: that what you
can do with a thing or concept). It makes sense to add its opposite, constraints (that what you can not do with a
thing or concept). My view is that this pair is very helpful in the assessment of the opportunities and impossibilities
pertaining to media, modalities, and genres (see Forceville, 2006a).

I also see serious problems:

1. Many chapters subscribe to the idea that the visual modality constitutes a system comparable to that exemplified
in the grammar and vocabulary of the verbal modality. This confidence, typical of old-style semiotics, is not
warranted, at least not to the degree voiced in this book. The notion of a visual grammar (Kress and Van
Leeuwen, 1996/2006) needs to be used with extreme caution. The term betrays the extent to which work on
1462 Book review

multimodality is undertaken by scholars trained as linguists, and entails the risk that claims about the need to
acquire visual competence are either exaggerated or misdirected the former because understanding pictures
comes to humans a lot more naturally than understanding language; the latter because the language model may
blind researchers to affordances characteristic of other modalities and media that, by definition, are absent or
downplayed in verbal communication. Moreover, while many of the concepts proposed by Kress and Van Leeuwen
(1996/2006), such as given-new, centre-margin, and vectors, are exciting and potentially fruitful, I am worried that
they are simply considered as facts that can be unproblematically used to prove things, whereas in reality these
concepts are at present no more than hypotheses requiring further critical evaluation as well as empirical testing
(see Forceville, 1999).
2. Apart from the fact that modality itself is not adequately defined, this book focuses predominantly on (the
interrelations between) two modes only: the verbal and the visual. Martin and Stenglin are absolutely right to point
out that the two-dimensional page affords representing language and (static) pictures at the expense of, for
instance, space, sound, music, moving images, and gestures. We thus run the risk of being blind to the specificities
of these other modalities, and of dangerously narrowing the concept of multimodality.
3. Even the visual modality gets little attention, compared to the verbal modality. As to the verbal modality:
analysis in this vein continues (uncited) work in the venerable tradition of stylistics, pioneered by Leech and
Short (Leech, 1966, 1969; Leech and Short, 1981), and associated with the journal Language and Literature. As
to the short shrift paid to the visual modality: to some extent, this is inevitable, since there are few tried-and-
tested tools for the analysis of contemporary visual culture (film theory is an exception; see Bordwell and
Thompson (2008), while McClouds work (1993, 2006) holds promise for comics). But in order to discuss how
different modalities interact in multimodal discourse, basic knowledge of how they function on their own is
indispensable. Hard work is needed to develop this non-linguistic knowledge; otherwise what is bound to
happen a problem surfacing in many of the chapters of the current book is that the visual modality is seen as
always merely supportive of the verbal.

For my own recent attempts to practise what I preach, see Forceville (2005a,b, 2006b), Forceville and Jeulink
(2007), Eggertsson and Forceville (in press), and Forceville et al. (forthcoming).

References

Bordwell, David, Thompson, Kristin, 2008. Film Art: An Introduction, 8th ed. McGraw-Hill, Boston.
Eggertsson, Gunnar Theodor, Forceville, Charles, in press. The HUMAN VICTIM IS ANIMAL metaphor in horror films. In: Forceville, C., Urios-Aparisi, E.
(Eds.), Multimodal Metaphor, Mouton de Gruyter, Berlin, pp. 423444.
Forceville, Charles, 1999. Educating the eye? Kress & Van Leeuwens Reading Images: the grammar of visual design. Language and Literature 8,
163178.
Forceville, Charles, 2005a. Visual representations of the Idealized Cognitive Model of anger in the Asterix album La Zizanie. Journal of Pragmatics
37, 6988.
Forceville, Charles, 2005b. Addressing an audience: time, place, and genre in Peter Van Straatens calendar cartoons. Humor 18, 247278.
Forceville, Charles, 2006a. Non-verbal and multimodal metaphor in a cognitivist framework: agendas for research. In: Kristiansen, G., Achard,
M., Dirven, R., Ruiz de Mendoza Ibanez, F. (Eds.), Cognitive Linguistics: Current Applications and Future Perspectives. Mouton de
Gruyter, Berlin, pp. 379402.
Forceville, Charles, 2006b. The source-path-goal schema in the autobiographical journey documentary: McElwee, Van der Keuken, Cole. The New
Review of Film and Television Studies 4, 241261.
Forceville, Charles, Jeulink, Marloes, 2007. The source-path-goal schema in animation film. Paper Presented at the International Cognitive
Linguistics Conference, Jagiellonian University Krakow, Poland, 1520 July.
Forceville, Charles, Veale, Tony, Feyaerts, Kurt, forthcoming. Balloonics: the visuals of balloons in comics. In: Goggins, J., Hassler-Forest, D.
(Eds.), Out of the Gutter, McFarland, Jefferson NC.
Gibson, James J., 1979. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston.
Johnson, Mark, 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. University of Chicago Press, Chicago.
Kress, Gunther, Van Leeuwen, Theo, 1996/2006. Reading Images: The Grammar of Visual Design. Routledge, London.
Leech, Geoffrey N., 1966. English in Advertising: A Linguistic Study of Advertising in Great Britain. Longman, London.
Leech, Geoffrey N., 1969. A Linguistic Guide to English Poetry. Longman, London.
Leech, Geoffrey N., Short, Michael H., 1981. Style in Fiction: A Linguistic Introduction to English Fictional Prose. Longman, London.
McCloud, Scott, 1993. Understanding Comics. Paradox, New York.
McCloud, Scott, 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. HarperCollins, New York.
 Book review 1463

Charles Forceville studied English language and literature at the Vrije Universiteit, Amsterdam, where he subsequently taught in the Departments
of English, comparative literature, and Word & Image. Since 1999 he has been employed in the Department of Media Studies of the Universiteit van
Amsterdam, where he is now associate professor and directs the Departments Research Master programme. In 1996, he published Pictorial
Metaphor in Advertising (Routledge); the volume Multimodal Metaphor (Mouton de Gruyter), co-edited with Eduardo Urios-Aparisi, is
forthcoming. His teaching and research interests focus on the structure and rhetoric of multimodal discourse in various media and genres
(documentary, animation, comics, cartoons). Between 1987 and 2007 Forceville wrote some 200 literary reviews for the Dutch newspaper Trouw.

Ch.J. Forceville*
Department of Media Studies, University of Amsterdam Turfdraagsterpad 9,
1012 XT Amsterdam, The Netherlands

*Tel.: +31 20 5254596; fax: +31 20 5254599.

E-mail address: c.j.forceville@uva.nl.
URL: http://home.medewerker.uva.nl/c.j.forceville/