Rodrigo F.
Cdiz
Music Technology Program
School of Music, Northwestern University
711 Elgin Road
Evanston, Illinois 60208-1200 USA
rcadiz@northwestern.edu
Recent technological developments have enabled
us to synthesize images and sounds concurrently
within single computers, even in real time, giving
birth to novel and genuinely integrated audiovisual
art forms (Hunt et al. 1998). But how should we or-
ganize and compose such works? Given a certain
soundscape, what would form an appropriate se-
quence of images to that soundscape? Given a cer-
tain sequence of images, what soundscape is
appropriate to it? If the image sequence and the
soundscape are being created concurrently, how
should we compose them?
Authors have proposed different approaches to
these questions (Whitney 1980; Hunt et al. 1998;
Lokki et al. 1998; Rudi 1998; Kim and Lipscomb
2003; Gerhard and Hepting 2004; Yeo et al. 2004).
These approaches differ significantly, and they are
based on diverse principles, such as correspondence
of aural to visual harmony, audiovisual modeling of
mathematical principles, audiovisual rendering,
data sonification, algorithmic control, and parame-
ter space exploration. It is important to notice that
there is no easy or correct solution, because the
problem we must deal with lies in combining two
entirely different media in time (Hunt et al. 1998).
A fuzzy-logic approach to the challenge of com-
posing both sound and moving image within a co-
herent framework is proposed here as an alternative
solution. This approach is based on a fuzzy-logic
model that enables a flexible mapping of either au-
ral or visual information onto the other, and it is
able to generate complex audiovisual relationships
by very simple means. This mapping strategy is in-
spired by two fundamental ideas: isomorphism and
synaesthesia. Isomorphism applies when two com-
plex structures can be mapped onto each other
based on the fact that changes in one modality con-
sistently cause changes in another modality (Hof-
stadter 1999). The word synaesthesia comes
directly from the Greek syn (together) and asthe-
Computer Music Journal, 30:1, pp. 6782, Spring 2006
2006 Massachusetts Institute of Technology.
A Fuzzy-Logic Mapper for
Audiovisual Media
sis (perceive; Van Campen 1999), thus meaning
a union of the senses. Synaesthesia occurs when
stimulation in one sensory modality automatically
triggers a perception in a second modality in the
absence of any direct stimulation to this second
modality (Harrison and Baron-Cohen 1997).
This article is structured as follows. First, the
motivations for this work are presented and dis-
cussed, including discussions of audiovisual do-
mains, synaesthesia, and isomorphisms. Second,
fuzzy logic is introduced, including its main fea-
tures. Third, details of the proposed fuzzy-logic
mapper are presented. Fourth, ID-FUSIONES
(2001) and TIME EXPOSURE (2005), computer
musicvideo works that use the proposed model,
are discussed as actual implementations of the ap-
proach described in this article. Finally, conclusions
and directions for future work are addressed.
Motivation
To understand how the composition of an audiovi-
sual work should be addressed, it is important to
consider how presenting music with visuals affects
the listener differently from presenting music or
visuals alone.
Experiencing the Audiovisual
There is substantial empirical evidence to support
the common subjective experience that music and
moving images interact in powerful and effective
ways (Bullerjahn and Guldenring 1994; Iwamiya
1994; Lipscomb and Kendall 1994; Rosar 1994; Sir-
ius and Clarke 1994). However, as Finaas (2001) sug-
gests, it is often difficult to predict the exact
influence of the visual stimuli that relate to audio
stimuli. The visual elements and their relationship
to the music can vary tremendously. Hunt et al.
(1998) suggest that combining music and visuals
produces combinatorial relationships of such com-
Cdiz 67
plexity that it forces composers to develop exten-
sive algorithmic control to maintain stylistic con-
sistency. Sirius and Clarke (1994) composed music
and used computer-generated moving images to in-
vestigate the interaction of different visual and mu-
sical parameters. Their findings show that the
effects of music in the rating of visual images are
usually additive and that there are no interactions
among specific musical styles and particular visual
images. In other words, no specific audiovisual
combinations acquire particular semantic charac-
teristics.
Finaas (2001) addressed the question of whether
presenting music live, audiovisually, or only aurally
makes any difference for listeners experiences. Au-
diovisual presentations were categorized in three
different submodes: simple documentary, which is
just a video recording of the live performance; TV-
type, in which live performance is alternated with
images from various perspectives, close-ups, and
images of details; and non-documentary, which
does not aim to be a faithful description of the per-
formance. Non-documentary is the type of presen-
tation that best describes the approach presented
here, and it also generally corresponds to Rudis
(1998) idea of concert videos in which the moving
images do not necessarily resemble the performance
or the score of the music being presented. Finaas re-
viewed eight studies in the literature finding that,
in the case of non-documentary modes of presenta-
tion, the results showed that no mode of presenta-
tion was clearly favored and that non-documentary
visual material had some tendency to create impres-
sions of activity and complexity, while its effect on
preference was unpredictable.
Lipscomb and Kendall (1994) investigated the re-
lationship between the musical soundtrack and vi-
sual images in the motion picture experience, which
can usually be classified as non-documentary. They
selected five scenes from commercial motion pic-
tures along with their musical scores. Each sound-
track was paired with every visual excerpt, resulting
in a total of 25 audiovisual composites. In one of the
experiments, subjects were asked to rate all 25 com-
posites on semantic differential scales. The findings
show that music exercised a strong and consistent
influence over the subjects responses to an audio-
68
visual composite, regardless of visual stimulus. In
other words, a musical soundtrack can change the
meaning of a film presentation.
As a summary, there is significant evidence sup-
porting the idea that music and visuals interact in
powerful ways when combined together, and that in
the case of non-documentary modes of presenta-
tion, the relationship between aural and visual ma-
terial can achieve high levels of complexity. These
kinds of audiovisual interactions may resemble the
perceptual phenomenon known as synaesthesia.
Synaesthesia
Synaesthesia is technically defined as an involun-
tary physical experience of a cross-modal associa-
tion. In other words, a crossing of the senses occurs;
having one sense stimulated would cause stimula-
tion in another sense as well. For those who possess
synaesthesia it is an obvious and integral part of
their sense perception. For a synaesthetic person
there is no question about using synaesthesia or
not, because it is simply there. But to those who
do not possess synaesthesia, it remains complex and
very often misunderstood (Berman 1999). The most
common type of synaesthesia is called colored
hearing and corresponds to the phenomenon of
seeing colors when hearing music or vowels (Van
Campen 1999). A good example of colored hearing
is given by Marks (1997), as he describes the case of
a synaesthete that saw a brown strip against a dark
background when presented with a tone pitched at
50 Hz with an amplitude of 100 dB.
Synaesthesia consists not of random associations
between isolated phenomena or qualities of two
sensory domains, but rather expresses correlated di-
mensions or attributes (Marks 1997). In this sense,
synaesthesia acts like a mapping from one modal
dimension to another. The proposed fuzzy mapper
operates on the same principle.
Artistic works that fuse the senses are often re-
ferred to as synaesthetic art (Hertz 1999). This is
possibly owing to the appropriation of the term
synaesthesia by many art historians who have
written about the interrelationships of music and
art (Berman 1999).
Computer Music Journal
 However, it is very important to distinguish the
term synaesthesia from the term synaesthetic
art. As Hertz (1999) suggests, Synaesthetic art is a
deliberate contrivance, the product of an artistic as-
piration, and we should not confuse it with the neu-
rological phenomenon of synaesthesia. Persons, not
artworks, are synaesthetic. Also, these kinds of
artworks are usually created for an audience that is
not synaesthetic. Cytowic (1997) also states that
there is a sharp demarcation of synaesthesia as a
sensual perception, as distinct from a mental object
like cross-modal associations in non-synesthetes,
metaphoric language, or even artistic aspirations to
sensory fusion. The fuzzy-logic mapper described
here allows the creation of synaesthetic art by pro-
viding a framework that permits the mapping of
information from one dimension, either visual or
aural, into the other, resembling the perceptual phe-
nomenon of synaesthesia.
Isomorphism
Isomorphism is a very general concept that appears
in several areas, not only in mathematics. The word
derives from the Greek iso, meaning equal, and
morphosis, meaning to form or to shape. In a
strict mathematical sense, a morphism is a map be-
tween two objects in an abstract category. General
morphisms are called homomorphisms. A homo-
morphism that is a one-to-one mapping is called an
isomorphism. Informally, an isomorphism is a map
that preserves sets and relations among elements
(Weisstein 2005). In other words, if A and B are iso-
morphic, then there are elements or features of A
that map onto B, and vice versa, even though A and
B may appear different.
Hofstadter (1999) affirms that isomorphism ap-
plies when two complex structures can be mapped
onto each other in such a way that to each part of
one structure there is a corresponding part in the
other structure, where corresponding means
that the two parts play similar roles in their re-
spective structures. According to DiMaggio and
Powell (1983), in 1968 Hawley defined isomorphism
(in a sociological context) as a constraining process
that forces one unit in a population to resemble
other units that face the same set of environmental
conditions.
Fuzzy Logic
Fuzzy logic is a concept derived from the branch of
mathematics called fuzzy sets (Zadeh 1965). In a nar-
row sense, it refers to a logical system that generalizes
traditional two-valued logic for reasoning under un-
certainty, allowing multiple values of truth. In a broad
sense, it refers to all the theories and technologies
that employ fuzzy sets (Yen and Langari 1999). In gen-
eral, when fuzzy logic is applied to computers, it al-
lows them to emulate the human reasoning process,
quantify imprecise information, and make decisions
based on vague and incomplete data (Kosko 1993).
Binary Logic versus Fuzzy Logic
In binary logic, variables are either true or false,
black or white, 1 or 0. Aristotles law of the ex-
cluded middle holds: we can only have A or not-A
(Kosko 1993). A given object cannot be part of both
A and not-A at the same time. If so, there is a con-
tradiction. In contrast, fuzzy logic is defined with
uncertain terms, and partial values of truth are ad-
mitted. Things are not true or falseblack and
whiteanymore; they can be partially true or false
or any shade of gray. Mathematically, fuzzy logic ac-
cepts values between 1 and 0.
Fuzzy Sets
A fuzzy set is a set whose members belong to it to
some degree. In contrast, a standard or non-fuzzy
set contains its members all or none (Kosko 1993).
Theoretically, a fuzzy set F of a universe of discourse
X = {x} is defined as a mapping, mF(x) : X [0, a] by
which each x is assigned a number in the range [0, a].
When a = 1, which is the usual, the set is called nor-
mal. In the extreme case where the distribution is of
zero width, the membership function is reduced to
singularities, i.e., the fuzzy set reduces to a crisp set.
(A crisp set is one in which boundaries are sharply
Cdiz 69
defined and membership is all or none.) If the sin-
gularities are of two possibilities, we have binary
logic. Here, mF is the degree of membership (or de-
gree of truth) of x. Fuzzy sets can have any shape,
and no shape has been proven to be the best (Mitaim
and Kosko 2001). However, the most commonly
used shapes are triangular and Gaussian functions.
Fuzzy Rules and Fuzzy Patches
Fuzzy logic attempts to emulate the way humans
reason with vague rules of thumb and common
sense. Malki and Umeh (2000) propose the follow-
ing example: If the weather is fine, then we may de-
cide to go out. If the forecast says the weather will
be bad today, but fine tomorrow, then we might de-
cide not to go today and postpone it until tomorrow.
A fuzzy rule is a conditional of the form if X = A
then Y = B, where A and B are fuzzy sets (Kosko
1993). Each fuzzy rule in a system determines a
fuzzy patch, defined as the product (A B) of the
fuzzy sets A and B in the system input-output state
space (the set of all possible combinations of input
and outputs). These patches can have a simple
geometry in the system state space, for example, an
ellipsoid (Dickerson and Kosko 1996).
The Fuzzy Approximation Theorem
Using the concept of fuzzy patches, Kosko (1994)
proved what he called the Fuzzy Approximation
Theorem, which states that a fuzzy system can uni-
formly approximate any real continuous function
on a compact domain to any degree of accuracy. The
idea is to approximate the target function by cover-
ing its graph with fuzzy patches in the input-output
state space and averaging patches that overlap. Ow-
ing to this theorem, fuzzy systems are considered
universal approximators (Ying et al. 1999).
Fuzzy Systems
Fuzzy systems estimate input-output functions.
Unlike statistical estimators, they estimate a func-
70
tion without a mathematical model of how outputs
depend on inputs. They are model-free estimators.
They learn from experience with numericaland
sometimes, linguisticsample data (Kosko 1992).
A fuzzy system is defined as a system with oper-
ating principles based on fuzzy information pro-
cessing and decision making. In such a system,
inputs are classified into fuzzy sets, and outputs are
de-classified from fuzzy sets to scalars or crisp sets.
The process of classification is defined as fuzzifica-
tion and the de-classification as defuzzification.
Fuzzification and defuzzification are critical opera-
tions in the design of fuzzy systems, as both of these
operations provide nexus between the fuzzy set do-
main and the real-value scalar domain (Roychowd-
hury 2001).
Figure 1 shows the fuzzification of the concept of
loudness. The goal of the fuzzification process is
to classify a variable or concept into different fuzzy
regions or sets by calculating degrees of member-
ship in each fuzzy set. In this particular case, there
are three fuzzy sets or membership functions to
which loudness can belong, labeled LOW,
MEDIUM, and HIGH. The membership func-
tions are normalized, and the universe of discourse
X contains all possible values for loudness levels in
the range 30120 dB.
If we take, for example, a 90-dB loudness level, we
can determine the degree of membership in each
one of the three membership functions by finding
the intersection of the scalar value with each fuzzy
set. As it is possible to observe, 90 dB belongs 70.69%
to the fuzzy set HIGH, 24.97% to the fuzzy set
MEDIUM, and 0% to the fuzzy set LOW.
Once the inputs to a fuzzy system are fuzzified,
one or more fuzzy rules are computed in parallel to
produce the corresponding outputs. This process is
called fuzzy inference. Different inference methods
are used depending on the purpose and nature of the
fuzzy system. The most common method is the
Mamdani method and was proposed in 1975 by
Mamdani and Assilian (Ross 2004). There are sev-
eral variants of this method; the min-max method
represents one of the most popular. Using the min-
max Mamdani method of inferenceif, for ex-
ample, we have two inputs and one outputthe
fuzzy rules have the form
Computer Music Journal
Figure 1. A fuzzy represen-
tation of loudness.

1
LOW MEDIUM HIGH and geology (Demicco and Klir 2004). However,
0.9
fuzzy logic has seldom been used in the artistic and
creative fields, which is very surprising, given that
0.8
other related theories and techniques such as neural
0.7
0.7069
networks and genetic algorithms have been widely
used for artistic purposes. In the specific case of
Degree of membership

0.6
0.5
0.4
0.3
0.2497
0.2
0.1
0 of human behavior. Parallel computation of fuzzy
30 40 50 60 70 80
Universe of discourse (in dB)
if x1 is A1k and x2 is Ak2 then yk is Bk for k = 1, 2, . . .
where A1k and Ak2 are input fuzzy sets, and Bk is the
desired output. Given a set of r disjunctive fuzzy
if-then rules, the aggregated (fuzzy) output will be
given by
Bk(y) = maxk [min [Ak1(input(1)), Ak2(input(2))]]
k = 1, 2, . . . , r
After the fuzzy inference process has taken place,
the aggregated outputs are then defuzzified so that
the desired numeric variables are obtained. Several
methods and techniques for defuzzification have
been proposed in the literature, each one with its
advantages and disadvantages (Ross 2004). One of
the most often used is the center of mass or cen-
troid method, in which the value of the center of
gravity of the aggregated fuzzy output is used as the
resulting scalar value.
A Fuzzy-Logic Mapper
Fuzzy logic systems have been widely used in a va-
riety of fields, most prominently engineering and
control applications (Kosko 1993; Klir and Yuan
1995), but they have also been applied to other areas
as diverse as data analysis (Bandemer and Gottwald
1995), economics, business and finance (Von Al-
trock 1997), sociology (Dimitrov and Hodge 2002),
music, despite the fact that Landy (2001) included
fuzzy logic as one of the potential areas for music
research in the future, only a small number of appli-
cations related to creative activities have been re-
ported in the literature.
Fuzzy systems provide several advantages for cre-
ative applications. Fuzzy systems are powerful and
work in a way that resembles some characteristics
90 100 110 120
rules usually reduces the computation time com-
pared to a traditional mathematical approach. Fuzzy
systems, owing to the fuzzy approximation theo-
rem, allow the approximation of highly non-linear
systems with any desired degree of precision. Fuzzy
systems are model-free estimators; thus, it is not
necessary to know a mathematical model in ad-
vance to approximate any system. Fuzzy rules can
be easily specified in the form of if-then state-
for
ments, allowing the building of fuzzy systems with
simple linguistic terms.
According to Ross (2004), the fact that fuzzy sys-
tems are universal approximators is a result of the
isomorphism between algebra and the structure of a
fuzzy system, which is composed of an implication
between actions and conclusions, or antecedents
and consequents. The reason for this isomorphism
is that both entities (algebra and fuzzy systems) in-
volve a mapping between elements of two or more
domains. Just as an algebraic function maps an in-
put variable to an output variable, a fuzzy system
maps an input group to an output group. With fuzzy
systems, these groups can be linguistic propositions
or other forms of fuzzy information.
Ross (2004) emphasizes that it will be the conse-
quence of this isomorphism that fuzzy systems will
become more popular in the future. In the spirit of
this idea, this article proposes a fuzzy mapper as a
way to produce isomorphic mappings between aural
and visual information. This can be done in the fol-
lowing way: if we have two aural elements A1 and
A2, using the fuzzy mapper we can generate their

Cdiz 71
counterparts in the visual domain, V1 and V2. In
other words, given A1, the fuzzy mapper will al-
ways produce V1, assuming that the rules are time-
invariant. Consequently, given A2, the fuzzy
mapper will always produce V2. If there is any kind
of relationship between A1 and A2, there will also
be some relationship (not necessarily the same) be-
tween V1 and V2. If we accept a general definition
of isomorphism as a map that preserves sets and re-
lations among elements, then it could be argued
that the proposed fuzzy-logic mapper is isomorphic.
The fuzzy mapper discussed in this article is basi-
cally a Mamdani-type fuzzy system in which visual
or aural parameters are used as inputs or outputs.
The inputs to the system are obtained by any kind
of compositional process, and then the fuzzy map-
per is used to obtain outputs in the corresponding
domain. All the defuzzification for generating out-
put is performed using the center-of-mass method
of defuzzification.
ID-FUSIONES and TIME EXPOSURE, two audio-
visual works based on the proposed fuzzy mapper,
are presented now as actual implementations of this
system. In ID-FUSIONES, aural parameters were
used as inputs, and visual parameters were obtained
by the mapper. In TIME EXPOSURE, the opposite
situation occurred: visual parameters were used as
inputs, and aural parameters were obtained as a re-
sult of applying the fuzzy mapper.
Example 1: ID-FUSIONES
ID-FUSIONES is an audiovisual work composed by
the author in collaboration with visual artist Luz
Mara Cury in the period May 2000March 2001.
This work consists of 206 audiovisual events that
occur in a time frame of approximately eight min-
utes. Each event is unique and has eight variables
(four visual and four aural) that define it completely.
In this particular piece, aural parameters were con-
sidered inputs and visual parameters were consid-
ered outputs, although this could vary with different
applications of the fuzzy model. The relationship
could be inverted (visual as input and aural as out-
put), as it happens in the second example described
later in this article, or even a mixed approach could
72
be used (some visual and aural as inputs, some visual
and aural as outputs).
At this point, it is important to mention that
these parametersonly eight altogetherwere
chosen to keep the simplicity of the compositional
process. In theory, the proposed model can take any
number of input and output parameters with no
limitation other than computational power. Some
of these parameters, especially the aural ones, coin-
cide with those identified to be perceptually rele-
vant in the literature. Seashore (1967) mentions
that sound waves have four, and only four, char-
acteristics; namely, frequency, amplitude, duration,
and form. Sounds of every conceivable sort, from
pure tone to the roughest noise, can be recorded
and described in terms of these four. (p. 16) By the
term form, Seashore refers to timbre. Lipscomb
(1995) found pitch, loudness, and timbre to be per-
ceptually relevant in the aural domain; location,
shape, and color were found relevant in the visual
domain.
In terms of audiovisual relationships, Lipscomb
and Kim (2004) investigated the relationship be-
tween auditory and visual components of an audio-
visual composite. Data analysis revealed significant
between-subjects differences in responses to audio-
visual material, suggesting the following primary
relationships: pitch relates to vertical location,
loudness relates to size, and timbre relates to shape.
Duration did not pair as a best match with any vi-
sual parameter. Color matched equally with pitch
and loudness. The proposed fuzzy mapper is flexible
enough to produce not only these primary relation-
ships but also any kind of relationship between the
audiovisual parameters. The chosen parameters for
this piece, almost the same ones used by Lipscomb
and Kim in their study, are described below; these
consist of frequency, intensity, duration, and noisi-
ness in the aural domain and color, shape, size, and
motion in the visual domain.
Aural Parameters
The Frequency parameter corresponds to the actual
primary or fundamental frequency component of
the spectrum of each sound event, measured in
Computer Music Journal
 Figure 2. First eight events
of the score of ID-
FUSIONES (2001).

Hertz. Sonic events in this piece are assumed to be

static, in the sense that the fundamental frequency
does not change in time for a given event, although
the overall spectrum can. The second parameter,
Intensity, corresponds to the maximum intensity
level of the attack portion of an ADSR (attack-
decay-sustain-release) envelope, measured in deci-
bels of sound pressure level (dB SPL). In this work,
levels between 60 and 100 dB SPL were used to
cover the usual dynamic range for most musical
works.
Each event has a temporal duration that deter-
mines its existence. The Duration parameter of
each event is measured in seconds. Although this
parameter is considered an input to the fuzzy map-
per and an aural variable, it also determines the du-
ration of the events in the visual plane. The fourth
and final aural parameter used was called Noise
(Spectrum). Sound events were considered noisy
or not noisy according to the shape of their spec-
trum. The flatter the spectrum, the noisier the
sound. This parameter is measured in a normalized
scale from 0 (not noisy) to 1 (noisy).

Visual Parameters

Initially, the entire visible light spectrumred to

violet was considered as possible values for the
Color parameter. This range was then subdivided
into 16 colors plus two distinct shades of gray (a re-
striction imposed by the software that was used to
generate the visual material). This parameter is mea-
sured by the corresponding wavelength of each color,
measured in meters (using a scale factor of 10 7).
Each event has a certain Shape, classified on a dis-
crete complexity scale between 0 and 1. For example,
a simple straight line would have a 0 score in this
scale, a simple curved line a 0.1, a triangle around a
0.3 or 0.4, and a highly complex shape with several
line segments and different types of lines a 1.
Each event has a certain Size on the screen that
could be small, medium or large. This parameter is
measured on a normalized discrete scale from 0 to 1.
The fourth and final visual parameter considered
was Motion. All visual events of the work are in
constant movement. Motion, in this context, is
equivalent to velocity, because it could have a cer-
tain direction and a certain speed. Fifteen arbitrarily
chosen different types of motion were considered
for this parameter.
Figure 2 shows the first eight events of the score
of ID-FUSIONES. Each event comprises two rows:
the first row shows the aural parameters, or input
vector, obtained from the compositional process,
and the second row displays the corresponding vi-
sual parameters, obtained from the fuzzy mapper,
for that particular event.
Fuzzy Sets
Once the eight parameters were defined, it was nec-
essary to classify them into fuzzy sets or member-
ship functions. The majority of the variables were

Cdiz 73
Figure 3. (a) Membership SPL shown on the x axis; given on the x axis. In each
functions for Frequency, (c) membership functions plot, VL refers to very
with frequency in Hz for Duration, with units in low, L to low, M to
shown on the x axis; (b) sec on the x axis; (d) mem- medium, H to high,
membership functions for bership functions for Noise and VH to very high.
Intensity, with units of dB (Spectrum), with noisiness

(a) (b)
Membership functions for Frequency Membership functions for Intensity

VL L M H V H VL L M H VH
1 1

0.8 0.8
Degree of membership

Degree of membership
0.6 0.6

0.4 0.4

0.2 0.2

0 0

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 50 55 60 65 70 75 80 85 90 95 100

(c) (d)
Membership functions for Duration Membership functions for Noise

VL L M H VH NO YES
1 1

0.8 0.8
Degree of membership

Degree of membership

0.6 0.6

0.4 0.4

0.2 0.2

0 0

0 1 2 3 4 5 6 7 8 9 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

classified into very low (VL), low (L), medium (M), distribution of the size of the various sets, which
high (H), and very high (VH) regions. When possible, looks like a logarithmic function. This was done to
the parameters were fuzzified taking into considera- resemble the actual perception of pitch by the audi-
tion some of their perceptual characteristics. tory system.
Figure 3 displays the input fuzzy variables, and Figure 3b shows the membership functions for In-
Figure 4 shows the output variables. Figure 3a tensity. As with frequency, triangular shapes were
shows the membership functions for the aural pa- used and the uneven size distribution was done to
rameter of Frequency. As can be seen, frequency is reflect, roughly, the perception of loudness (in dB
classified into five different fuzzy sets. Triangular SPL). Figure 3c shows the membership functions for
functions were used for each set. Note the uneven Duration. In this case, triangular functions were used,

74 Computer Music Journal

Figure 4. (a) Membership functions for Shape; (d)
functions for Color, with membership functions for
wavelength (5 10 7) in me- Size. Units of measure-
ters on the x axis; (b) mem- ment on the x axis for (b)
bership functions for (d) are arbitrary, as ex-
Motion; (c) membership plained in the text.

(a) (b)
Membership functions for Color Membership functions for Motion

VL L M H VH VL L M H VH
1 1

0.8 0.8
Degree of membership

Degree of membership
0.6 0.6

0.4 0.4

0.2 0.2

0 0

4 4.5 5 5.5 6 6.5 7 0 5 10 15

(c) (d)
Membership functions for Shape Membership functions for Size

VL L M H VH S M L
1 1

0.8 0.8
Degree of membership
Degree of membership

0.6 0.6

0.4 0.4

0.2 0.2

0 0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

but they were evenly spaced. A time scale from 010 Figure 4a shows the membership functions for
sec was used. Events lasting more than 10 sec were Color. Gaussian functions were used to emulate the
considered very long (VH). Figure 3d shows the mem- actual frequency distribution of the visible spectrum.
bership functions for Noise (Spectrum). Sounds were Figure 4b shows the membership functions for Shape,
classified into noisy or not-noisy. Note that this re- in which evenly spaced triangular functions were
sembles more a binary classification, with two large considered. Figure 4c shows the membership func-
yes or no areas, than a fuzzy one (although it tions for Size. Only three fuzzy sets were used in this
still is considered fuzzy). This is another advantage case, classifying size into small, medium, and big. Fig-
of the fuzzy approach, because it allows bivalent ure 4d shows the membership functions for Motion,
and multivalent variables equally. in which evenly spaced Gaussian curves were used.

Cdiz 75
 Figure 5. Fuzzy mapper op-
eration showing (a) input
(aural) parameters and (b)
output (visual) parameters.

Table 1. Example of Fuzzy Rules from ID-FUSIONES (a)

1. If the frequency is very low and the sound is not noisy,
then the color is high, the motion is very high, and the
shape is very low.
2. If the intensity is very high and the sound is not noisy,
then the shape is low and the size is large.
3. If the frequency is very low and the intensity is very
high and the sound is not noisy, then the color is
medium.
4. If the intensity is high and the duration is very low,
then the size is medium.

Fuzzy Rules

The relations between the different variables were

regulated by means of 26 fuzzy inference rules of de-
cision that control the behavior of the entire mapper.
These rules relate one or more input variables with
one or more output variables and have the form of (b)
if-then statements. As an example, four of the 26
rules used in ID-FUSIONES are shown in Table 1.

Operation of the Fuzzy Mapper

Figure 5 shows a graphical view of the operation of

the fuzzy mapper given a particular input vector and
using only the four rules detailed in Table 1. In this
example, the input vector is [275 95 1 0.5] (represent-
ing the Frequency, Intensity, Duration, and Noise
parameters, respectively), and the output vector ob-
tained by the fuzzy mapper is [6.16 13.5 0.217 0.548]
(for Color, Motion, Shape, and Size). As stated ear-
lier, Frequency is measured in Hz, Intensity in dB
SPL, Duration in sec, Color (wavelength) in meters
(scaled by 107), and the rest of the parameters are
measured in discrete ordered scales depending on
their nature.
The rules are displayed as rows, numbered from
14. The four columns in Figure 5a show the input
(aural) variables; the four columns in Figure 5b show is affected by each rule is shown. All four rules are
the output (visual) variables. The input vector values computed in parallel, using the Mamdani method
are shown in the text row above the first rule, next of inference, and four fuzzy output variables are
to each input name, and they are also represented obtained, shown in the four rectangles below the
by vertical lines that cross the input variables. In- fourth row. These fuzzy outputs are then defuzzi-
side each rectangle, the membership function that fied using the center of mass (or centroid) method of

76 Computer Music Journal

Figure 6. (a) Screen shot
ID-FUSIONES at 2:10; (b)
at 5:29; (c) at 8:02; (d) at
10:42.
(a)
(c)
defuzzification to obtain scalar values for the out-
put (visual) parameters. The obtained scalar values
are shown in the text row above each output col-
umn and next to the output names and also repre-
sented as vertical lines on the four rectangles at the
bottom of the last row. Because the output parame-
ters are measured in discrete scales, when non-
integer values are obtained for the output variables,
they are rounded to the nearest integer to obtain an
appropriate output value.
Figure 6 shows four screen shots of ID-FUSIONES
taken at different times. Additional screenshots and
(b)
(d)
excerpts are available online at www.rodrigocadiz
.com/idfusiones. [Editors note: Also see the forth-
coming DVD accompanying Vol. 30, No. 4 of the
Journal.]
Example 2: TIME EXPOSURE
TIME EXPOSURE (2005) is an interactive multi-
media work that organically incorporates several
forms of artistic creation by means of both tradi-
tional techniques and new technologies. This work
Cdiz 77
 Figure 7. (a) Painting at
month 1; (b) at month 6;
(c) at month 12; (d) two-
dimensional spatial-
frequency spectrum of
painting at month 1; (e) at
month 6; (f) at month 12.

is a collaboration between the author and Carmen (a)

Gonzlez, a visual artist and researcher. TIME EX-
POSURE integrates traditional painting, music
composition, and video art and technology to create
a new artwork that uses time and its physical ef-
fects as artistic material.
Central to this project is the idea that, in the de-
struction of art, its own creation is implicit; to aban-
don a work of art for its own sake constitutes indeed
a way to regenerate it. Time is here conceived as the
only agent capable of creating and destroying art at
the same time: it can destroy an artwork but at the
same time will generate a new one.
The constructive sequence of this project con-
templated the creation of six paintings, especially
created for this project using traditional techniques, (d)
as its point of departure. Each of these six initial
works has a predominant color drawn from the pri-
mary and secondary colors (red, green, blue, yellow,
violet, and orange).
These paintings were then exposed outdoors over
the time span of one year, and the changes to both
their appearance and structure provoked by the ex-
posure to the environment were observed. During
this interval of time, the exposed paintings were dig-
itally recorded daily to document the changes they
endured. These recordings were then edited and
mounted, creating different visual sequences in dig-
ital video format. These resulting visual sequences
also provide the source of musical sequences that
add an audible plane to the visuals on each of the
six video sequences.
A computer program developed by the author an- Figures 7ac show three different shots (published
alyzed the visual sequences and computes their spa- in grayscale) of the blue painting taken at different
tial frequency and color content as they changed times (months 1, 6, and 12, respectively). Figures
over time. Each of the individual recorded images 7df show the spatial-frequency domain (obtained
of the original paintings had a different spatial fre- by taking the two-dimensional Fourier transform)
quency and color structure as a result of environ- of each shot. Here, it is possible to appreciate the
mental exposure. This three-dimensional visual change in frequency content of the images caused
structure (two spatial dimensions and one color di- by outdoor exposure.
mension) was mapped to a three-dimensional sonic Each one of these shots constitutes a video frame,
structure (pitch, intensity, and density), by means of and each frame lasts 1 sec in performance time.
the fuzzy-logic mapper described in this article. There are about 300 frames in total for each paint-
Consequently, the music also shows a kind of de- ing, covering a time span of a little more than a
composition in analogy to the original destructive year in physical time. The change in the color and
trajectory of the paintings. More details of this ap- brightness of the three figures is especially notice-
proach are provided below. able. Figure 7a was taken on a rainy day, and the

78 Computer Music Journal

(b)
(e)
color is dark because the canvas was wet. Figure 7b
shows some clear signs of de-coloration after 6
months of outdoor exposure. Figure 7c was taken on
a sunny day, and some shadows and a brighter im-
age can thus be seen.
It is also possible to notice some significant
changes in the spatial-frequency structure of the
three images. Figure 7d shows much more high-
frequency content than Figure 7e. Based on this in-
formation, it is possible to observe a low-pass filter
effect that takes place over the images as time pro-
gresses. Figure 7f show more high-frequency con-
tent than Figure 7e, owing primarily to the presence
of sunlight and the shadow. Also, the predominant
frequencies in the three images change, as can be
(c)
(f)
seen by diagonal lines that appear in the spectrum
at different angles.
Figure 8 shows the average power spectrum
across the x and y axes of the three images. Clearly,
important differences in the high-frequency content
of Figures 7a and 7c compared to Figure 7b can be
observed, most likely owing to the environmental
exposure.
Visual Parameters
Figure 8 also shows horizontal lines representing
the average of the average spectrum in both direc-
tions. These averages were used as inputs to the
Cdiz 79
Figure 8. (a) Average power
spectrum across the X di-
mension; (b) average
power spectrum across the
Y dimension.

(a) (b)
0 0
Month 1 Month 1
Month 6 Month 6
Month 12 10 Month 12

20
20

30
40

40

60 50

60

80
70

80
100

90

120 100
1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1
Normalized spatial frequency Normalized spatial frequency

fuzzy mapper. Other measurements such as the Table 2. Average RGB Components for Figure 7ac
standard deviations of the average spectra, the spec-
Image R G B
tral centroid in both directions, and the average val-
ues of the RGB components of each frame (shown in Painting at month 1 51 110 153
Table 2 for the case of Figures 7ac) were used as in- Painting at month 6 69 100 140
puts to a fuzzy-logic mapper very similar to the one Painting at month 12 107 123 124
described for ID-FUSIONES. If we inspect the aver-
age values of the blue component of the RGB com-
posite for each frame, we can notice a significant This work is available in DVD format but also as
loss of blue color that this painting suffers as time a computer-executable version that allows the
passes. viewer to control in real time the speed of the flow
of physical time. Additional screen shots and video
excerpts are available online at www.rodrigocadiz
Aural Parameters .com/timeexposure. [Editors note: Also see the
forthcoming DVD accompanying Vol. 30, No. 4 of
The outputs of the fuzzy mapper in this case are au- the Journal.]
ral frequency, intensity, and sonic density. These
parameters were used to control synthesis algo-
rithms that generate the soundscape for the piece. It Conclusions and Future Work
is important to emphasize that each one of the six
visual sequences used the same fuzzy mapper to The composition of audiovisual media is not an
generate the sonic content. This means that each easy task. Several authors have proposed very differ-
one of the visual sequences generated a different ent approaches with different results. The interaction
soundscape, obtained from the specific spatial fre- of music and visual material in non-documentary
quency and color structure of the visual material, modes of presentation is a very complex process
but derived from the same set of fuzzy rules, bring- (Lipscomb and Kendall 1994; Hunt et al. 1998;
ing a meta-coherence to the overall work. Finaas 2001).

80 Computer Music Journal

 A fuzzy-logic mapper for audiovisual media has
been proposed and successfully used in two com-
puter music-video pieces. The main advantage of
this approach is its simplicity of design and use.
The mapper is based on if-then-like fuzzy infer-
ence rules that resemble human thinking and com-
mon sense. The input and output parameters for the
mapper, the rules, and their quantities are not re-
stricted in any way and can be freely chosen by the
composer.
There are, however, some limitations. The pre-
sented implementation does not work in real time,
and it can only generate non-interactive mappings.
A very important direction for future work in this
area would be to extend this system to a real-time
based framework. This would allow, for instance,
generation of visual images on the fly in reaction
to live music or vice versa. Another possible direc-
tion for future research would be to develop a
neuro-fuzzy mapping model. Such a system could
be able to learn how to react to a given unknown
input, either aural or visual, and generate map-
pings based on the knowledge acquired through
past experiences.
Acknowledgments
Thanks to Luz Mara Cury and Carmen Gonzlez
for all their help and support. ID-FUSIONES was
funded by a grant for artistic creation and research
of the Andes Foundation, Santiago, Chile. TIME EX-
POSURE was funded by a grant from the Center for
Interdisciplinary Research in the Arts (CIRA) of
Northwestern University, Illinois, USA. Thanks
also to Dr. Richard Ashley, Dr. Gary S. Kendall, and
Dr. Scott D. Lipscomb for their valuable comments
and suggestions.
References
Bandemer, H., and S. Gottwald. 1995. Fuzzy Sets, Fuzzy
Logic, Fuzzy Methods with Applications. Chichester,
UK: Wiley.
Berman, G. 1999. Synesthesia and the Arts. Leonardo
32:1522.
Bullerjahn, C., and M. Guldenring. 1994. An Empirical
Investigation of Effects of Film Music Using Qualita-
tive Content Analysis. Psychomusicology 13:99118.
Cytowic, R. E. 1997. Synaesthesia: Phenomenology and
NeuropsychologyA Review of Current Knowledge.
In S. Baron-Cohen and J. E. Harrison, eds. Synaesthesia:
Classic and Contemporary Readings. Cambridge, Mas-
sachusetts: Blackwell, pp. 1739.
Demicco, R., and G. J. Klir. 2004. Fuzzy Logic in Geology.
Boston: Elsevier.
Dickerson, J. A., and B. Kosko. 1996. Fuzzy Function Ap-
proximation with Ellipsoidal Rules. IEEE Transac-
tions of Systems, Man, and Cybernetics 26(4):542560.
DiMaggio, P. J., and W. W. Powell. 1983. The Iron Cage
Revisited: Institutional Isomorphism and Collective
Rationality in Organizational Fields. American Socio-
logical Review 48:147160.
Dimitrov, V., and B. Hodge. 2002. Social Fuzziology:
Study of Fuzziness of Social Complexity. Heidelberg,
Germany: Physica-Verlag.
Finaas, L. 2001. Presenting Music Live, Audiovisually, or
AurallyDoes It Affect Listeners Experiences Differ-
ently? British Journal of Music Education 18(1):5578.
Gerhard, D., and D. H. Hepting. 2004. Cross-Modal Para-
metric Composition. Proceedings of the 2004 Interna-
tional Computer Music Conference. San Francisco,
California: International Computer Music Association,
pp. 505512.
Harrison, J. E., and S. Baron-Cohen. 1997. Synaesthesia:
An Introduction. In S. Baron-Cohen and J. E. Harrison,
eds. Synaesthesia: Classic and Contemporary Read-
ings. Cambridge, Massachusetts: Blackwell, pp. 316.
Hertz, P. 1999. Synesthetic ArtAn Imaginary Num-
ber? Leonardo 32:399404.
Hofstadter, D. R. 1999. Gdel, Escher, Bach: An Eternal
Golden Braid. New York: Basic.
Hunt, A., et al. 1998. A Generic Model for Composi-
tional Approaches to Audio Visual Media. Organised
Sound 3(3):199209.
Iwamiya, S. 1994. Interactions Between Auditory and Vi-
sual Processing When Listening to Music in an Audio
Visual Context: 1. Matching; 2. Audio Quality. Psy-
chomusicology 13:133153.
Kim, E., and S. D. Lipscomb. 2003. An Investigation into
the Relationship Between Auditory and Visual Signals
in a Multimedia Context. Poster presented at the 2003
Conference of the Society for Music Perception and
Cognition, Las Vegas, Nevada, 1619 June.
Klir, G. J., and B. Yuan. 1995. Fuzzy Sets and Fuzzy Logic:
Theory and Applications. Upper Saddle River, New Jer-
sey: Prentice Hall.
Cdiz 81
Kosko, B. 1992. Neural Networks and Fuzzy Systems.
New Jersey: Prentice-Hall.
Kosko, B. 1993. Fuzzy Thinking: The New Science of
Fuzzy Logic. New York: Hyperion.
Kosko, B. 1994. Fuzzy Systems As Universal Approxima-
tors. IEEE Transactions on Computing 43(11):1329
1333.
Landy, L. 2001. From Algorithmic Jukeboxes to Zero-
Time Synthesis: A potential AZ of Music in Tomor-
rows World (A Conference Provocation). Organised
Sound 6(2):9196.
Lipscomb, S. D. 1995. Cognition of Musical and Visual
Accent Structure Alignment in Film and Animation.
Ph. D. thesis, University of California, Los Angeles.
Lipscomb, S. D., and R. A. Kendall. 1994. Perceptual
Judgment of the Relationships Between Musical and
Visual Components in Film. Psychomusicology
13:6069.
Lipscomb S. D., and E. Kim. 2004. Perceived Match Be-
tween Visual Parameters and Auditory Correlates: An
Experimental Multimedia Investigation. Proceedings
of the 8th International Conference on Music Percep-
tion and Cognition. Evanston, Illinois: Society for Mu-
sic Perception and Cognition, pp. 7275.
Lokki, T., et al. 1998. Real-Time Audiovisual Rendering
and Contemporary Audiovisual Art. Organised Sound
3(3):219233.
Malki, A. H., and C. G. Umeh. 2000. Design of a Fuzzy
Logic-Based Level Controller. Journal of Engineering
Technology 17(1):3239.
Marks, L. E. 1997. On Colored Hearing Synaesthesia:
Crossmodal Translations of Sensory Dimensions. In
S. Baron-Cohen and J. E. Harrison, eds. Synaesthesia:
Classic and Contemporary Readings. Cambridge, Mas-
sachusetts: Blackwell, pp. 4998.
Mitaim, S., and B. Kosko. 2001. The Shape of Fuzzy Sets
in Adaptive Function Approximation. IEEE Transac-
tions on Fuzzy Systems 9(4):637656.
Rosar, W. H. 1994. Film Music and Heinz Werners The-
ory of Physiognomic Perception. Psychomusicology
13:154165.
82
Ross, T. 2004. Fuzzy Logic with Engineering Applica-
tions. Chichester, UK: Wiley.
Roychowdhury, S. 2001. An Inquiry into the Theory of
Defuzzification. In W. Pedrycz, ed. Granular Comput-
ing: An Emerging Paradigm. Heidelberg: Physica-
Verlag, pp. 143162.
Rudi, J. 1998. Computer Music Animations. Organised
Sound 3(3):193198.
Seashore, C. E. 1967. Psychology of Music. New York:
Dover.
Sirius, G., and E. F. Clarke. 1994. The Perception of Au-
diovisual Relationships: A Preliminary Study. Psy-
chomusicology 13:119132.
Van Campen, C. 1999. Artistic and Psychological Experi-
ments with Synesthesia. Leonardo 32:914.
Von Altrock, C. 1997. Fuzzy Logic and Neurofuzzy Appli-
cations in Business and Finance. Upper Saddle River,
New Jersey: Prentice Hall.
Weisstein, E. W. 2005. Morphism. From MathWorld
A Wolfram Web Resource. Available online at
http://mathworld.wolfram.com/Morphism.html.
Whitney, J. 1980. Digital Harmony: On the Complemen-
tarity of Music and Visual Art. Peterborough, New
Hampshire: McGraw-Hill.
Yen, J., and R. Langari. 1999. Fuzzy Logic: Intelligence,
Control, and Information. Upper Saddle River, New
Jersey: Prentice Hall.
Yeo, W. S., J. Berger, and Z. Lee. 2004. SonART: A Frame-
work for Data Sonification, Visualization, and Net-
worked Multimedia Applications. Proceedings of the
2004 International Computer Music Conference. San
Francisco: International Computer Music Association,
pp. 180184.
Ying, H., et al. 1999. Comparison of Necessary Condi-
tions for Typical Takagi-Sugeno and Mamdani Fuzzy
Systems As Universal Approximators. IEEE Transac-
tions on Systems, Man, and Cybernetics 29(5):508514.
Zadeh, L. A. 1965. Fuzzy Sets. Information and Con-
trol 8:338353.
Computer Music Journal