You are on page 1of 8


Lars Ellestrm

Let us take a look at something. Let us see what can be found in something
that could be anything, for instance, something like an X. What is it? Is it
a sign? Yes, at least it can be a sign. The moment we decide that it means
something, it becomes a sign. The easiest way to make the X mean some-
thing is to put it in some sort of context, for instance, a conventional sign
system: the Greek or the Latin alphabets, the sign systems of mathematics or
logic, or some other more or less settled scheme. We know what to do with
X in the word mix and we know that X, in an equation such as 3 + X = 5,
means the unknown number.
There are very many alphabets, systems and schemes, however, that can
make signs readable and, furthermore, signs often become signs not in
determined sign systems but in contexts that are tied only marginally or
indirectly to the prevailing conventions. Perhaps it is not even a very good
idea to say that all signs are read, since the acts of interpretation are much
diversified. If we stubbornly persist in taking a look at the X, we have to
admit that even a simple little sign such as this one leads a rather compli-
cated double or triple life. Actually, both well-defined and temporary signs,
as well as sign systems and specific media productions and works of art,
can be said to share a lot of properties with and, hence, also overlap with
other signs, systems and types. The general characteristics of X, shared by
innumerable other signs, can be summarized as follows.
It has a material interface. The reader of this Introduction is probably
confronted with some sort of flat surface: a paper, hopefully in a book, or
a computer screen. This kind of material interface allows for a variety of
appearing content. You can put almost anything in a book, as long as it
is flat and static, and you can put almost anything on a screen, including
things that move, as long as it is flat. This flat surface is, first and foremost,
seen by the reader, spectator or interpreter. Of course, it is not forbidden
to use the other senses as well when interacting with the displayed con-
tent of screens and pages, but in this case nothing very significant would be
added by way of, for instance, trying to taste the X that can be seen in this

L. Ellestrm (ed.), Media Borders, Multimodality and Intermediality
Palgrave Macmillan, a division of Macmillan Publishers Limited 2010
2 Introduction

sentence. Nevertheless, many signs that are part of conventional systems

have sounds attached to them. When reading this English text, most people
form the sound eks, silently or aloud, when seeing the mark X. This sound
is, however, a secondary aspect since it is not part of the material interface
but a result of the interpretative interaction with it. In a Greek context, the
appearance of the visually identical symbol X would generally lead to the
sound chi (pronounced as the German ich backwards).
Spatiotemporally, this text is static since it remains the same once it has
been mediated into a printed or electronically fixed product. It takes time
to read it, however, and the alphabetic form invites the reader to decode
it according to certain rules including, among other things, standards for
sequential reading. On the level of what it represents, this text also cre-
ates temporal relations. As regards my writing about primary and secondary
sensorial experiences, for example, it is implied that in the described, con-
ceptual world that the linguistic expressions refer to, the visual comes first
and the aural at a later stage in the interpretative process. There is thus some
sort of virtual time in the description. The manner in which the equation
3 + X = 5 is outlined also implies a temporal sequence mirrored by the
reading order: first Linda catches three fishes and then an unknown number
of fishes, and when she returns at the end of the day she has five fishes.
The X, as a written sign, is obviously spatial, both in itself and as part of
conventional sign systems or unfixed context. As a visual sign, it stretches
out in four directions with a junction in the middle and it is generally con-
ceived as a combination of two intersecting lines. Its spatiality is normally,
and certainly in the context of this Introduction, two-dimensional, but also
three-dimensional objects, such as Ronald Bladens minimalistic but never-
theless very large aluminium sculpture The X, can be seen as parts of the
wide sign type of X. X-like figures that we see in paintings, drawings and
photographs can acquire virtual, three-dimensional spatiality, but the X in
Bladens sculpture, as well as the letters of many alphabetical toy bricks, is
three-dimensional in itself. It must also be recognized that there is an impor-
tant aspect of spatiality involved in the activity of taking a look at X. To take
a look at X means both actually to process the information perceived by our
photosensitive receptors and to form cognitive, spatial structures when con-
sidering the many aspects of what X can be understood to be and how this
being can be understood. To see is to form mental structures in cognitive
The X actually means nothing, however, until it is interpreted as some
kind of sign. When reading about mixed vegetables in a cookbook most
people automatically and rationally see the little X as a conventional sign
that is part of a word with a specific meaning: blended, composed of dif-
ferent elements and so forth. If, in the same cookbook, the word mixed
is shaped by various vegetables with a stalk of asparagus and a stick of cel-
ery forming the X, there is an iconic element attached to the conventional
Lars Ellestrm 3

sign: the word mixed is no longer arbitrarily connected to its meaning since
it actually also resembles what it represents: mixed vegetables. If the cap-
tion of an image of mixed vegetables forming the words mixed vegetables
says that mixed vegetables are good for your health we assume that the
text, because of its closeness to the image, can be read as a part of what the
image expresses. Moreover, we can assume also that vegetables that are not
depicted, but are nevertheless closely related, are actually as healthy as the
ones depicted, or we can assume that the ones that occur more frequently are
actually healthier than the others. All in all, the interpretative acts that form
the meaning of this page in the cookbook, that is, the interpretative acts that
actually put the sign-functions into action, or even create the signs, can be
said to be based on convention, resemblance and contiguity.
We must now remember that the X is merely an example. It is something
that could be anything. Yet, considering its conventional attributes and sig-
nificance, it is a funny little sign. When we ponder on, say, Ronald Bladens
X or the many visually striking Xs in popular culture (graffiti, record sleeves,
movie posters and so forth), a whole range of potential meanings pops up:
the X can represent meetings and mixes, but it also implies erasure and the
forbidden. Nonetheless, X is also the sign of presence. It can mean that some-
thing is valid or present and it is the only alphabetic letter that may also be
written analphabetically? Even if you cannot read the X, you can see its
iconic potential. The X always carries visual aspects that may be activated
as iconic significance as soon as the interpreter finds an opportunity. When
there is a point in it, the X looks like a pair of crossed fingers, branches or
vegetables, and since the wings of the windmill definitely look like an X,
the X also looks like the wings of a windmill. Ian Hamilton Finlay is said to
have written a poem called The Windmills Song that reads like this: X.
The poem certainly represents the same kind of windmills that can be seen
in, say, many paintings by Jan Brueghel, both the elder and the younger. It
would thus be easy to put the wings into the sequence of conventionally
formed letters. Don Quijote, as we know, had windmills on his mind, and
if we spell his name in the original way, and add a slight emphasis by way
of capitalizing the crucial letter, we suddenly have something that might be
called a concrete poem: Don QuiXote. In this version of his name, we can
actually see, iconically, that he is full of windmills.
The X, whether seen as a letter, a conventional, non-alphabetical sign, a
poem, an image, a sculpture, a part of a linguistic sequence, a graphic mark
on a surface, a sound, a part of a work of art or a media production, or
a sign used in completion of the income-tax return, is obviously part of a
context where all texts and systems, in the widest senses of the words,
overlap. It would be very wrong to say that all systems are the same; that
there is no difference at all between the way meaning is produced when we
find the X in an exhibition hall and in a form respectively, but if we are to
talk about borders we are better off talking about border zones rather than
4 Introduction

strictly demarcated borders. Hence, I would say that all kinds of sign systems
and also specific media productions and works of art must be seen as parts
of a very wide field including not least the material, sensorial, spatiotempo-
ral and semiotic aspects. In my essay in this volume, I call these the four
modalities of media. By investigating the modalities one clearly sees that
all forms of art, media, languages, communication and messages have some
characteristics in common which make it possible for something like the X
to hover between different systems and simultaneously be part of various
frameworks without losing its relative stability.
For various reasons it is convenient to talk about media when discussing
these issues. The notion of a medium is fairly inclusive and offers a way to
bring together scholarly efforts within a considerable number of disciplines.
The phenomenon whereby the properties of all media partly intersect and
the study of this same phenomenon are called intermediality. As might be
expected, however, the term intermediality has been used in many ways that
are not always entirely compatible. Sometimes the study of intermediality is
seen as the study of a group of media and art forms that fall outside the
borders of the normal or established media. In this collection of essays,
however, most of the writers, at least most of the time, see intermedial-
ity as the precondition for all mediality. Even so, it may occasionally be
appropriate to highlight the fact that some media may be perceived as more
border-crossing than others.
Our small survey of the X has hopefully demonstrated that media actually
cross each other rather than border each other. Presumably we will never
stop talking about media borders, but in this volume they will be partly
crossed out.

Media Borders, Multimodality and Intermediality is a collection of essays that

have found their final forms in dialog with each other. The initial ques-
tion posed was simply what is intermediality? To answer this question, one
must also ask what a medium is and where we find the gaps that inter-
mediality bridges. Clearly, the supposedly crossed borders must be described
before one can proceed to the inter of intermediality. We also set out to
investigate how the two notions of intermediality and multimodality, rarely
discussed together, are related. The aim was to facilitate communication and
theoretical cross-fertilization over the borders between the aesthetic disci-
plines, media and communication studies, linguistics, and so forth. That
turned out to be a lucky strike, I would say, but the edges of the concep-
tual tools had to be sharpened in order to work properly. Further questions
that we all had in common from the beginning, and that are clearly vis-
ible in many of the essays, concerned the relations between notions such
as technical medium, medium, art form and genre. We wanted to inves-
tigate what happens, from a historical and social point of view, when
new media and art forms emerge and are delimited and to determine the
Lars Ellestrm 5

theoretical and practical implications of describing something as intermedial

or multimodal.
The essays thus have common starting-points and they have been influ-
enced by each other. We have sought to avoid confusing conflicts between
terminologies and the research angles are relatively compatible, but there
is no absolute harmony between the essays. They are written by individual
scholars who rarely agree on the best way to deal with tricky queries. They
are, however, focused on a coherent range of questions and topics that are
believed to be important.
Of course, the titles of the volumes sections only partly mirror the con-
tent of the essays, but they might give a hint of the main focus. The first
essay, The Modalities of Media: A Model for Understanding Intermedial
Relations, is my own attempt to build a theoretical framework that explains
how all media are related to each other: what they have in common and
in which ways they differ. The key to this understanding is to be found in
the four modalities that were briefly mentioned earlier in this Introduction.
I also find it imperative to distinguish between three aspects of the notion of
medium. Basic media are simply defined by their modal properties whereas
qualified media are also characterized by historical, cultural, social, aesthetic
and communicative facets. Technical media are any objects, or bodies, that
realize, mediate or display basic and qualified media.
Part II, Media Borders of Qualified Media, contains three essays. Irina
O. Rajewskys Border Talks: The Problematic Status of Media Borders in the
Current Debate about Intermediality discusses the many critical approaches
that make use of the notion of intermediality. Rajewsky focuses on the
assumption of tangible borders between individual media and the recent
questioning of precisely this fundamental premise of discernible media bor-
ders. Yet, Rajewsky argues, media borders and medial specificities are of
crucial importance for art practice, and she specifies how media can be
conceived as distinct and how they actually come into play in concrete
intermedial practices. In Intermedial Topography and Metaphorical Inter-
action, Axel Englund also highlights the tendency to think of arts and
media in terms of geographic areas delineated by definable borders, and
consequently of intermedial studies as a kind of topographical description.
Focusing on relations between music and literature, his essay points to some
of the implications of this topographical model, and contrasts them with
another way of conceptualizing intermedial relations, namely as metaphor-
ical phenomena. Englund argues that many musico-literary artefacts can be
successfully read as a metaphorical interaction between their musical and
verbal elements. Christina Ljungbergs semiotically-oriented essay Inter-
medial Strategies in Multimedia Art proposes that intermediality either
concerns the transgression of the borders between conventionally distinct
media of communication or the iconic enactment of one medium within
another. Ljungberg argues that both of these instances of intermediality are
6 Introduction

highly performative, as we are confronted with hybrid forms that gener-

ate something new and unique, that they are strongly self-reflexive, since
they focus attention both on their own mode of production and on their
own semiotic character, and that they constitute a highly effective com-
munication strategy, as they give receivers access to different levels of
Part III, Combinations and Integrations of Media, includes three essays
that mainly draw attention to media as multimodal products of combined
and integrated modalities and other media aspects. In Media before
Media Were Invented: The Medieval Ballad and the Romanesque Church,
Sigurd Kvrndrup approaches the medieval ballad as an intermedial art form
and proposes a new theory of one aspect of its origin. Kvrndrup builds
on Marshall McLuhans theories about the development of mass media
during the Middle Ages and relates these to Romanesque church-building.
Hkan Sandgrens The Intermediality of Field Guides: Notes Towards a The-
ory is an essay informed by ecocriticism that investigates the field guide to
birds, a hitherto unexplored area for the scholar of intermediality. In the
field guide, descriptive prose is combined with images, maps and transcrip-
tions of birdsong. A particular form of transcription offers examples of how
the author tries to transcribe bird song to text, an endeavour that some-
times approaches the domain of concrete poetry. In Sami Sjbergs Media
on the Edge of Nothingness: Visual Apostrophes in Lettrism, the author is
interested in how the Lettrists set out to challenge the limits of media and
representation. The techniques applied by this avant-garde group resulted
in works blending poetry, narrative fragments and imaginary signs. Sjberg
proposes two new concepts for analysis: the visual apostrophe locates omis-
sions in the text, where invented signs either replace the omitted part of
the text or supplement the work in an ambiguous relation to language, and
meontologization refers to the dynamics of signification and nothingness
engaged in the artwork.
The volumes fourth and most extensive part is called Mediations and
Transformations of Media and it includes seven essays that deal with the
large and multifaceted field of media transformations. Beginning with the
model of the much-discussed relation between creator, work and beholder,
Siglind Bruhn suggests a triangular concept of expressive intent in her essay
Penrose, Seeing is Believing: Intentionality, Mediation, and Comprehen-
sion in the Arts. Regarding the rarely considered discrepancy between the
human minds limited capacity to grasp the various media equally and
the near-unlimited scope of creative options, Bruhn draws attention to
areas where the chain of mediation is less than straightforward in cases of
both extreme and ostensibly quite accessible messages. Finally, she presents
an unusual three-tiered transmedialization that proceeds from a visual
through a verbal and on to a musical representation. In Beyond Definition:
A Pragmatic Approach to Intermediality, Valerie Robillard argues that defi-
nitions, although essential in laying out common terms of discourse, do not
Lars Ellestrm 7

fully contribute to our understanding or articulation of the various types

and degrees of medial interaction. The purpose of her essay is to demon-
strate the need to employ ontological systems in the delineation of medial
types, systems that have proven indispensable to other disciplines (such as
the natural sciences and linguistics) in determining the relative positions of
concepts and categories with respect to one another. Robillard proposes the
use of a pragmatic intertextual model to delineate and differentiate types of
medial interaction and to demonstrate the difference between the textual
aspects of intermediality (the message) and the medial aspects (materials).
Regina Schobers essay Translating Sounds: Intermedial Exchanges in Amy
Lowells Stravinskys Three Pieces Grotesques, for String Quartet calls
attention to notions such as transcription and intermedial translation that
imply dynamic processes of movement and transfer between medial bor-
ders. More specifically, Schober discusses a verbal representation of a musical
work by American modernist poet Amy Lowell as a paradigmatic example
of an experimental transformation of one medium into another. Following
Roman Jakobsons term intersemiotic translation as well as recent attempts
to broaden the terms translation and transcription, she comprehends
Lowells poem as both a literary and a cultural translation of Stravinskys
Three Pieces for String Quartet from 1914. In Transgenic Art: The Biopo-
etry of Eduardo Kac, Claus Clver investigates a very different sort of media
transformation resulting in poetry: the transgenic artwork Genesis (1999)
by the Brazilian poet and artist Eduardo Kac, who has radically explored the
possibilities of contemporary media technology for art-making. Genesis was
based on an artificial gene created from a phrase from the biblical Genesis
represented in Morse code that was then converted into a DNA sequence
according to a special code, mutated and eventually retranslated. In his
essay, Clver sets out to explore the functions of the work, the demands it
makes on the receiver and its implications for the discourse on media and
intermediality and for the notion of poetry.
In Katalin Sndors essay, Photo/graphic Traces in Dubravka Ugreics
The Museum of Unconditional Surrender, Ugreics novel is interpreted as a
text very much concerned with, shaped by and disrupted into (textual)
photographs, albums, museums, archives, cards, collections and memories.
Sndor focuses mainly on those passages which deal with photographic rep-
resentation and practices, as well as with a specific photograph incorporated
both visually and textually into the exile-narrative. She looks for the medial
traces of photography and explains how the photographic medium becomes
a kind of apparatus for the textual process of remembrance. In The Dance
of Intermediality: Attempt at a Semiotic Approach of Medium Specificity
and Intermediality in Film, Hajnal Kirly also analyses a specific work of
art: Hungarian director Bla Tarrs Satans Tango, a seven-and-a-half-hour
movie merging two cinematographic trends beginning with the 1990s that
have raised the problematic notions of medium specificity and intermedial-
ity in films: the so-called writers movies and the contemplative, extremely
8 Introduction

slow-paced movies, mostly coming from the Far East, defying all the com-
plex narratorial accomplishments of the film medium. Kirly argues that the
first kind of movie systematically overturns the strict delimitation between
literature and film, the idea of conceptuality of the first and visuality of the
latter, and that the second kind of movie continuously turns the running
time of the narrative into an almost static, plastic visual work of art: a
picture. In an essay by gnes Peth o, Media in the Cinematic Imagination:
Ekphrasis and the Poetics of the In-Between in Jean-Luc Godards Cinema,
the work of Godard, another film director, is the centre of attention. Peth o
states that his films have long been associated with the idea of cinematic
intermediality and that both his fiction films and his cinematic essays can
be considered a sort of direct theory, or archeology of cinema as a medium.
The essay attempts an application of the notion of ekphrasis, traditionally
understood as verbal descriptions of images, to the medium of cinema. A few
major conditions for the perception of cinematic ekphrasis are outlined and,
from the variety of intermedial relations in Godards films that can be called
ekphrastic, four types are charted and exemplified.
The fifth and last part of the volume is called The Borders of Media
Borders. The title is an attempt to capture the more general perspectives
introduced by the sections two essays. In his contribution, Heteromedial-
ity, Jrgen Bruhn proposes a widened concept of intermediality constructed
on the principle of multimodality. He suggests we begin considering all cul-
tural texts as mixed media and that we start understanding the particular
constellations of modalities as tensional relationships in the sense that the
mixing of modalities, and consequently of media, always refers to a wider
historical and ideological context. An adoption of this heteromedial view-
point will, he argues, transfer studies of intermediality to the very centre
of future humanistic studies. In Intermediality Revisited: Some Reflections
about Basic Principles of this Axe de pertinence, Jrgen E. Mller asserts that
the research axis of intermediality actually keeps numerous scholars at uni-
versities and research centres all over the globe busy. The variety of aspects
of the concept of intermediality makes it very difficult or almost impossible
to present some sort of general overview with regard to all options, Mller
argues, without opening a sort of academic bookkeeper discourse on differ-
ent terminological, theoretical, methodological and historical items. Instead,
he develops some aphorisms on the state of affairs of intermedial studies
and some perspectives for a historical approach. Mller is concerned with
the question of when a new medium becomes a new medium and, aided
by the test case of television, he discusses intermediality as a process. Fur-
thermore, the concept of intermediality is compared with intertextuality,
interartiality and hybridity. Finally Mller asks what the substance of inter-
medial studies actually is. Hopefully, not only Mller but also the volume as
a whole provides a good answer to that question.