The Multisensory Film Experience: A Cognitive Model of Experiental Film Aesthetics

Chapter 1

The Multisensory Film Experience, Experiential Film Aesthetics, Cinema and the Senses

However much the spectator may be engaged by plot or genre, subject matter or thematic implication, the texture of the film experience depends centrally upon the moving images and the sound that accompanies them. The audience gains access to story or theme only through that tissue of the sensory materials. […] However unaware spectators may be of it, style is working at every moment to shape their experience.

(Bordwell, 1997: 7−8)

Sensations of smell, equilibrium, or touch are, of course, never conveyed in a film through direct stimuli, but are suggested indirectly through sight. […] [The filmmaker] eliminates entire areas of sensory perception, and thereby brings others into higher relief, ingeniously making them take the place of those that are missing.

(Arnheim, 1933: 34)

Let us assume that, unlike the other types of pictures, film images affect primarily the spectator’s senses, engaging him physiologically before he is in a position to respond intellectually.

(Kracauer, 1960: 158)

Although this statement may be an audacious way to open this book, I am convinced that there is no such thing as a purely visual—or purely audiovisual—experience of film. What we see, or what we call seeing, is multisensory in its nature and remains multisensory, even in the final stage of what we consider consciousness: that supposed moment when perception lightens and the marvel of awareness comes about. The perceptions resulting from much of the visual and auditory information in the external sensory world are multisensory. They are not visual, auditory or audiovisual, but multisensory.

When this idea first began to germinate in my mind, I had smaller ambitions and aimed only to show that it is possible to have a multisensory experience through an audiovisual medium such as film and that our brains can perceive an audiovisual medium in a multisensory manner. As my research advanced, though, I realized that not only can our brains perceive an audiovisual medium in a multisensory way, but they must do so because there is no other way for perception to occur. Our natural, not exceptional or synaesthetic, way of perceiving is multisensory. This form of perception remains multisensory when we experience a film, whatever the configuration of the apparatus used to watch the film: a dark theatre room, a cosy living room with a television set or even a small tablet. The natural way for the brain to operate is multisensory, and even if we desired purely visual experiences, we would be frustrated and incapable to force our brains to block out and inhibit some of the neural connections between the senses.

The idea of multisensoriality in film is often based on the intellectual and phenomenal capacity of spectators to make associations creatively through imagination and memory (Marks, 2000, 2002; Sobchack, 1992, 2004). I do not wish to refute such a capacity and possibility, but rather wish to show a level of multisensoriality that is perceptual—not intellectual, imagined or remembered—in principle. This multisensoriality presents itself even before our consciousness has the capacity to make any sense or intellectual and phenomenal associations in a synaesthetic manner. To reiterate Siegfried Kracauer’s words cited at the beginning of this chapter, a film engages the spectator […] physiologically before he is in a position to respond intellectually (Kracauer, 1997: 158). In point of fact, a multisensory experience is the natural and common way for all of us to perceptually experience film instead of a mere synaesthetic, exceptional capacity to make high-order, intellectual and phenomenal associations among ideas of a multisensory nature.

My analysis of this issue is a matter of perception, not phenomenology. The multisensory film experience derives from hard-wired, low-level mechanisms of perception that are not within our conscious control. These take place in the milliseconds of the time window of sensory integration, that is, when the senses are integrated. We can divide the levels of our perceptual control of a film through the concepts of autonomic and somatic responses, where somatic response refers to a process that does not require conscious and effortful processing but can nevertheless be inhibited or controlled and autonomic response refers to a process on which we cannot exert control. This layout of a multisensory film experience contradicts the idea of a voyeur spectator who leans back comfortably and visually watches and enjoys in a detached way and in control of his private audiovisual cinematic experience.

Because the primary goal of this book is to support the idea that there are no visual, auditory or audiovisual experiences of film—only a multisensory experience—I am much more concerned with the autonomic level of our film perception than with the somatic level. Furthermore, I aim to add an examination of film aesthetics based on multisensoriality to this perceptual investigation of the senses. The resulting corollary of this combination of perception and aesthetics is materialized by what I specifically call the experiential film aesthetics of Gus Van Sant, Ki-Duk Kim and Knut Erik Jensen (see also Antunes, 2015a, 2015b, 2015c). My analysis of the work of these three directors within a multisensory frame of reference is tantamount to stylistic, narrative and emotional elements that together shape the concrete realm of the multisensory film experience using thermoception (perception of temperature), nociception (perception of pain) and the vestibular sense (perception of orientation and balance) as sense modalities not captured by the discussion of the classic five senses or of the traditional senses associated with film perception and film phenomenology.

The multisensory film experience is the conceptual formulation that evolved from my idea—and the apparent contradiction—that film is an audiovisual medium that is perceptually experienced by spectators in a multisensory fashion. This conceptual distinction between a medium source of stimuli and the resulting perceptual experience is paramount to my claims and aims to shift the burden of proof from having to show that it is possible to perceive an audiovisual medium in a multisensory manner to having to corroborate an orthodox view of film as an audiovisual medium and experience. My proposed shift of the burden of proof invites sceptics to investigate the notion that when we experience a film, we can consciously or unconsciously have exclusively visual or, at most, audiovisual experiences of film, not multisensory experiences, without the influence of the other senses.

In my model of the multisensory film experience, seeing, or what we call seeing, is not only a visual experience but also an experience that involves all of our senses, including hearing in a discriminate manner, specifically through the most relevant sensory modalities explored by certain experiential aesthetics, such as those of the three directors analysed in this book. Considering film a visual experience is not a perspective of specific film scholars but a truly common perspective of nearly all of them; thus, my arguments widely engage with a variety of studies both within and outside the film literature, without a critical focus on particular film scholarship.

In the multisensory film experience, I assume different degrees of sensitivity and individual and subjective perceptual contents of spectators’ experiences. Although the exact perceptual contents change from one individual to another and from one sensory context to another, the nature and general principles of our multisensory perception remain common to us all. In other words, the principles of multisensory processing and integration, or the idea that the brain works in a multisensory manner, are universal principles, but the exact perceptual contents of our experiences certainly vary by individual and sensory context. Thus, the multisensory film experience does not deal with the perceptual contents but with the perceptual principles of film perception.

The idea of experiential film aesthetics, as I define it, implies that film is not only the result of formal and compositional elements of style, narrative and themes but also the result of the intersection of those elements (especially film style created through camerawork, editing, light, colour and sound design) and our perceptual, multisensory nature as spectators. Our perceptual experience of a film is thus not a mechanistic method of receiving, processing and integrating a film’s sensory information, but rather an active, dynamic set of perceptual processes that are proactive and creative (although its autonomic levels are not within the reach of our conscious control). This is a constructivist frame of reference very much in line with many conceptions of cognitive film theory, however, with specific stress on the role of the senses, rather than on the more explored roles of emotion and empathy. The bottom line, however, is widely recognized, namely, that aesthetics and perception are not separate, but rather common and dialogical layers.

For instance, colour and lighting can have different expressive, formal roles, but in the realm of Jensen’s films, they are not merely sensory features of the visual domain; they also have the potential to evoke, provoke and elicit emotional, perceptual and thematic experiences associated with thermoception. Similarly, the same applies to Kim’s articulation of sound with nociception and Van Sant’s articulation of camerawork and editing with the vestibular sense. Colour and lighting, along with all other stylistic elements, can be connected to specific senses in ways that can be understood only in relation to the authorial use of each specific director. Experiential film aesthetics result, then, from a dialogical relationship between style, narrative, themes, the characters’ bodies and our perceptual experience because the elements of the style cue certain perceptual experiences and because the nature of our human perception simultaneously has the ability to relate the sensory information of each specific sense modality. A matrix of dialogical relationships finds different configurations across each of the sensory modalities that I examine in this book.

The films I have selected from the oeuvres of Van Sant, Kim and Jensen are based primarily on non-verbal communication. This criterion facilitates a focus on the experiential nature of their films, but it does not presuppose that verbal language is not part of our multisensory human perception. Verbal language may not be a sensory modality on its own, but it is a cognitive function with a perceptual basis and a strong capacity to generate multisensory imagery. Verbal language covers many aspects of human perception and cognition and also deserves an in-depth examination in future studies of the multisensory film experience. However, here, I focus on specific levels of sensory perception, and examining non-verbal cinema simplifies a methodological focus on thermoception, nociception and the vestibular sense. Otherwise, we could be led to believe that the verbal language of these films was the element prompting an associative experience of the senses. For example, in a scene from Darren Aronofsky’s The Fountain (2006), I failed to notice that the character Isabel/Izzi Creo (Rachel Weisz) sat shoeless on her rooftop while it snowed until Tomas/Tommy/Tom Creo (Hugh Jackman) joined her and made verbal mention of this fact. This is a case in which it is fair to say verbal language can play a significant role in inducing our perceptual experiences, as opposed to the almost-mute films of Van Sant, Kim and Jensen.

In addition to the minimal use of verbal language, these films display minimal narratives. There are still traces of narrative and storylines, but they are minimal or fragmented, and they are often conducted not through dialogue and verbal communication but rather through the characters’ actions, the film style and information from which we can make perceptual inferences. They are offered to spectators in forms that connect to the ineffable or to the idea that many of the contents of film—for example, thematic segments (Antunes, 2013), story events, character motivations and even some moral alignment between spectators and characters—belong to a non-verbal realm. Instead of being told what happens and hearing dialogue about the motivations behind the actions and events in the film’s story, these films resort to a level of experiential aesthetics.

Additionally, these films venture one step further from the motto Show, don’t tell (Lubbock, 1957) to, I would risk saying, Don’t just show, deliver to the senses. They make spectators active perceptual-participants, not mere observers of an unfolding narrative experience. In this sense, these films’ minimal narrative nature is still based on the serendipity principles of any other film narrative, but their perceptual nature is based on the active interaction with a participant’s perceptual experience and is constructivist, given the creative ways in which our perception can be engaged. Downplaying verbal language and narrative helps us to see the role and importance of the pure sensory experience, but verbal language is not, I stress, incompatible with any variable of the multisensory film experience.

The films of Van Sant, Kim and Jensen also have the advantage of reflecting cinematic approaches across different cultures and geographies in a sample of world cinema that includes films from North America (Van Sant), East Asia (Kim) and Northern Europe/the Arctic (Jensen). I do not intend to delve into universalist claims that experiential film aesthetics arises regardless of spectators’ cultural background or that the senses are universal. Certainly, there are culturally and individually shaped differences in perception. Instead, I note only that my claim of a multisensory film experience is not contingent upon a specific geography or culture but can be found not only across a range of aesthetics and thematic sensibilities but also across different cinematic and geographic cultures. I do, however, assume that despite these films’ textures and cultural nuances, they appeal to a wide range of audiences outside of their respective home cultures, as corroborated by their greater success overseas than that in their home countries. By using non-verbal communication and the senses, these films capture the interest of various audiences. The experiential appeal of these films is universal, but spectators’ actual perceptual experiences will certainly vary according to cultural specificities, individuals and sensory contexts (e.g., a small television set in living room versus a large, dark theatre with surround sound). Their films represent a universally appealing language of the senses with room for individual and subjective experiences.

Although the multisensory film experience is, in the aforementioned sense, a universal form of appeal offered by film, its cultural and geographic influences vary. It is given that any film can and will cue forms of experiential engagement across a number of senses: however, the following questions remain: (1) how? and (2) taking advantage mainly of which sensory modalities? Experiential immersion is present in all films with different degrees of intensity and quality. For instance, an extreme example opposed to the vestibular sense could be My Dinner With André (Louis Malle, 1981),¹ in which two men sit at a restaurant table, dining and talking throughout the film. In that case, there is not much room for salient responses to experiential immersion on the level of orientation and balance. The experience takes place more on the level of verbal language than anything else. That, however, does not mean that we do not have some spatial awareness of the characters and the room in which they are located: we need at least some sort of orientation. Even a film with such an extreme lack of movement constructs a spatial awareness through its editing, shot-scale, camera positions and even sound design. However, My Dinner With André is in opposition to, for instance, the roller coaster ride of the three-screen projection of This is Cinerama! (Various directors, 1952), where a visceral experience based on the vestibular sense is basically what is at stake and where words are replaced by screams, body contortion, mugging, vertigo and, if one has the full experience, some motion sickness.

Although all films have a foundation of experientiality (they are a form of sensory experience), some are more closely based on non-verbal forms of communication that more directly call for an immersive experience through the senses. This use of experiential film aesthetics within the realm of non-verbal communication seems to represent a strong tendency in contemporary filmmaking. In addition to the directors discussed in this book, directors as diverse as Andrea Arnold (Antunes, 2015d), Terrence Malick (Antunes, 2014), Ang Lee, Jim Jarmusch, Sofia Coppola, Jane Campion, David Cronenberg, Jacques Audiard, Jean-Pierre and Luc Dardenne, Nuri Ceylan, Steve McQueen, Kar-Wai Wong, Andrew Dominik, Ming-liang Tsai, Abdellatif Kechiche, Cristian Mungiu and Albert Serra are among those who I consider most representative of world cinema’s contemporary tendency towards experiential film aesthetics. Their cinematic projects are exceedingly challenging to our notions of film genre and even film narrative because although these films offer stories, they do so without the verbal communication to which we have become accustomed from the more common forms of film narrative based on verbal interaction, such as the screwball comedy or the film noir, as the most obvious examples.

This contemporary type of filmmaking across various geographical and cultural contexts and authorial voices, giving primacy to the experiential aesthetics of film, defies the general presupposition that Eastern arts have a more experiential nature that is opposed to the more narrative forms of the West:

[…] the Japanese aesthetic tradition opens possible areas for inquiry not often explored in modern Western aesthetics. They include the relationship between the aesthetic and its moral and spiritual considerations in the sense explained above, creative activity that is object/material-centered, and the appreciation of multisensory experiences and those qualities that are typically depreciated.

(Saito, 2009: 386)

In my view, the contemporary cinema of the senses and this contemporary search for experiential film aesthetics is not a product of Asian cinema, but represents a generalized tendency across European and North American cinema as much as in Asian cinema.

The broader idea of experiential aesthetics (applied not to film, but to the arts in general) has been used to describe art as an experience outside objectivism. According to experiential aesthetics, compositional aspects of an object of art matter only insomuch as they relate to how humans perceive and construct them; they are not immutable objective stylist features. Experiential aesthetics are then born from this dialogical relationship between the subjective and the objective, the style and perception. However, it has only been applied to film to a very limited extent because there exists what I think of as an orthodox view, which assumes that film is a visual (and at most, an audiovisual) experience, and its aesthetics are therefore audiovisual, as well. In this manner, it has been difficult to conceive how film can be experiential in the same way as an interactive art installation in a museum, which directly convokes haptic contact between the perceiver and the work of art.

This orthodox view accepts that our perceptual experience of film is experiential insofar as it is an audiovisual experience because the direct sensory link with the other sensory modalities is absent in the form of sensory stimuli. This orthodox view considers that this sensory link of a haptic nature is lacking in film. Film in its audiovisual form has not, then, been considered an experiential medium with the same immersive and interactive potential but instead one in which spectators do not actually participate perceptually in its full properties because some sensory modalities are supposedly shut off. Although the expression experiential aesthetics seems to have come to use only a few years later (by Beiswanger, 1937),² the foundations behind the concept were, nevertheless, established by the influential John Dewey in his groundbreaking Art as Experience (Dewey, 1934).³

Because of the previously mentioned lack of a sensory link that directly shows haptic contact between the medium of film and spectators, Dewey has not been very influential in film studies. Instead, Dewey has a more direct influence on studies of art performances and installations, architecture and video games, in which it is conceptually easy to see the haptic connection between medium and spectator. These are media that are more obviously experiential than film. Although part of film’s experientiality has been overlooked due to the traditional form of understanding film as an experience with neither locomotive action nor haptic contact, I see worthy reasons to consider our perceptual experience of a film equally (in extent), though diversely (in quality), experiential, even if it is true that spectators are typically seated and do not have haptic contact with the temperature or the noxious stimuli from the film’s world.

Dewey’s idea of experientiality has had a much stronger impact on the context of art media involving haptic contact that therefore display multisensory stimuli, which is in contrast to film, in which sense-to-sense correspondence governs the idea of a strictly audiovisual experience. However, Dewey relocates the idea of aesthetics from the realm of the objective and explanatory power of art theory to the realm of the human experience. More than belonging to the history of art, art belongs to actual life-experience (Dewey, 1934: 3). Dewey’s idea of art as experience repositioned art as an experience that is as essential as the art object itself. According to this view, we cannot aim to fully understand art without fully understanding many other trans-disciplinary levels of human existence, such as biology and cognition. We must understand perceivers to understand art and/or film. Furthermore, Dewey’s program is based on a shift from a paradigm of visual art forms to art that encompasses all of the sensory realms of human experience: "All senses are equally on the qui vive" (Dewey, 1934: 19).⁴ The question that remains open with respect to film is the exact involvement of the senses given the idea that film spectators do not have haptic contact with much of the sensory information from the world of the film.

My response to the objection of considering film aesthetics as experiential and multisensory is that there are other types of motor actions and haptic checking involved in our perceptual experience of a film, such as using our hands to check parts of our bodies, which create the missing sensory link.⁵ Adding to that, even what we think of as a motorway neutral position (e.g., sitting in the theatre) is never neurally and motorway neutral. Furthermore, a common misunderstanding is that spectators have to have haptic contact with the actual stimuli from the film’s world, when in fact they only need to have some sort of sensory contact (audiovisual, in this case) with referents of that film’s world, mediated to give access to other sensory modalities. It is, then, useful to track the concept of experiential aesthetics and other related concepts, such as sensory experience and embodiment.

Some of the modern discussions on the concept of experientiality in art may date back to the 1930s; but only in the 1980s and 1990s did experiential aesthetics burst into the discussion of new media, particularly as a central topic about the relationship between realism and mediated experience.⁶ In film studies, however, the expression never entered the discussion until Anna Powell’s Deleuze and Horror Film (2005), and even so, it did not seem to gain traction. At any rate, David Bordwell glimpses, if only indirectly, this perceptual and experiential nature of film in the quotation cited at the beginning of this chapter: The audience gains access to story or theme only through that tissue of the sensory materials. […] However unaware spectators may be of it, style is working at every moment to shape their experience (Bordwell, 1997: 7−8).

As Bordwell suggests, the senses are the gateway to our experience of film. They not only give us access to an experiential understanding of its narrative, themes and even emotions but also actively shape them. The question is not so much about the existence of an actual perceptual experience, but the extent and quality of this experience when applied to film. Do spectators have audiovisual experiences of film, given that visual and aural information are the medium’s sensory cues? If so, do these spectators therefore imagine all the levels of sensory experience within the world of the film that touches other sensory modalities, such as thermoception, nociception and the vestibular sense? Imagination, memory and phenomenal association are the concepts used by film phenomenologists to describe the possibility of experiencing a film across senses beyond sight and hearing in what they consider to be mainly a synaesthetic type of experience.

At the core of all these questions and problems lies the lack of conceptual distinction between medium and experience among film phenomenologists. One of the most influential film phenomenologists to address this question, Laura Marks, approaches hapticity from the perspective of The Skin of the Film (2000), in an allusion to the idea that the grainy quality of video films in the cinema of exiled filmmakers who recollect memories of their inaccessible home cultures through their films elicits associative memories of a haptic essence. This means that Marks’ concept of hapticity is attached to the haptic qualities of the medium itself, in its materiality—in this case, in the grainy aspect of the image quality–and not the denoted multisensory qualia potentially brought out by that medium. However, although this specific phenomenological issue may be relevant, we can understand experiential film aesthetics from a purely perceptual level in a manner that could explain our potential multisensory experience of any film, independent of whether the film possesses an idiosyncratic image quality, as Marks’ analysed films do. This is why I posit that the concept of multisensory film experience is not residually and idiosyncratically confined to a specific set of films, but a principle common to all films.

Concepts such as sensory experience and multisensory experience have received particular attention within discussions of film realism. The idea of sensory experience seems to be intuitively connected to the notion of realism. In Realism of the Senses in World Cinema (de Luca, 2014), Tiago de Luca considers the sensory experience in film as a form of enhanced realism. However, I believe this is more closely connected to the constructed notions of realism through concepts of slow cinema associated to the specific authorial languages of the directors explored by de Luca (e.g., Carlos Reygadas, Gus Van Sant, Ming-liang Tsai, Béla Tarr, Lisandro Alonso, Apichatpong Weerasethakul, Nuri Bilge Ceylan) than to any intrinsic realism value of the senses. To de Luca, that which confers upon the work of these directors a quality of realism is connected to how their use of the long take shifts the role of the characters’ bodies in relation to the cinematic landscape to occupy a central position. In other words, instead of focusing on story development, these directors of slow cinema create the temporal conditions for a prolonged film experience based on an observation of how the characters’ bodies perform in different landscapes. De Luca considers this use of sensory experience a form closer to realism.

However, although this linkage between realism and the senses may be valid in some cases, it should nevertheless be stressed that realism is not an intrinsic value of sensory experience. In the films of other directors—for example, Terrence Malick and Kim—who have worked around the central idea of sensory experience in a vein similar to that of slow cinema, there are events that evoke and request the participation of our senses in an experiential manner that is nevertheless extremely unrealistic. One example among many is a scene from The Tree of Life (Malick, 2001) in which Mrs. O’Brien (Jessica Chastain) begins to float in soft, gentle movements. The scene does not evoke any impression of realism but a dream-like state and it conveys a feeling of subjectivity and fantasy from the perspective of her son; in what we commonly call reality, we do not expect to see bodies floating in front of us. Nonetheless, her floating body is an event with an intense appeal to the senses—in this case, to our proprioceptive sense, which provides awareness of the relative position of the limbs and trunk.

This example illustrates that sensory perception does not connect to a necessary sense of realism as it has been traditionally defined—at least not in an absolute manner. De Luca’s take on this relationship between the senses and realism should be seen as motivated by the aesthetics and work of the specific directors he discusses. Although de Luca refers to a cinema of the senses, he does not specifically identify which senses are involved and how they are involved. This lack of differentiation is a shared point with film phenomenology. Addressing the senses in a holistic fashion without differentiating the sensory modalities involved in our perceptual experience works as a form of semantic substitute for the similarly general concept of embodiment.

De Luca identifies slowness (through the long take) with the senses, noting that the cinema of the senses precludes narrative interaction in favour of phenomenological and sensory experience (de Luca, 2011: 43). Although I understand the need to differentiate among the layers of narrative, themes and sensory experience, I do not find it necessary to assert that the relationships between these elements are mutually exclusive in the sense that we can only experience one without the others, that is, that a film must be narrative oriented or experiential. A director such as Kim shows that these levels of narrative, themes and sensory experience can