DESIGN OF PERFORMANCE AND THE NEW TECHNOLOGICAL LANDSCAPE

Music & tangible user interface

Maria Rosaria Digregorio

matriculation number > 261564

Submitted on 19/09/2007 as the examined output

of the Telecomunicazioni course (Gillian Crampton Smit, with Philip Tabor)
at the Facolt di Desing e Arti, IUAV University, Venice

1
typeface > Solex, designed by Zuzana Licko in 2000
http://www.emigre.com

2
INDEX

1 > Music: from analog to digital page 4

2 > Input/Output page 6

3 > Interface: graphical vs tangible page 8

4 > From Block Jam to... page 10

5 > From ReacTable to... page 14

6 > The good, the bad and the future page 18

6 > Sources page 20

3
1 > MUSIC: FROM ANALOG TO DIGITAL binary decimal
0 0
1 1
10 2
11 3
100 4
101 5
110 6
111 7
1000 8
Sound is a mechanical signal produced by a vibrating 1001 9
object and transmitted through a medium (for 1010 10
instance the air) as waves of alternating pressure.
The ears detect fluctuations in the medium pressure
and translate them into an electrical signal that the
brain understands.
When sounds and silence are structured by particular
laws or conventions, they are perceived as music.

A sound can be also recorded and reproduced, not

only transmitted when its really played.

In analog recording the original sound is translated

into a physical signal like the groove of a gramophone
disc or the iron filings of a magnetic tape.
In digital recording, instead, the original sound is
measured at intervals and converted into an electrical
signals, on/off pulses encoded as binary numbers
(only 0 and 1).

4
A digital signal allows to transmit music in the form
of numerical informations rather than as physical analog recording
objects.
It means that a digital sound can be separated from
its physical source and linked to something able to
decode numerical informations such as computers or
other electronic devices.

Potentially, with the help of proper technologies,

anything could became able to decode numerical
informations, anything could be used to play music...

digital recording

5
2 > INPUT/OUTPUT

In digital recording the sound is translated in binary

numbers.
So the recorder is a transducer, because it converts a
kind of energy (mechanical) in another (electrical).
But all input and output devices translate signals from
the world in numbers understandable for a computer,
so they all can be considered as transducers.

An input device can be considered as a sensor,

because it receives informations from outside and
translates it in numbers.

An output device, on the contrary, can be considered

as an actuator, because it receives numbers from
inside and translates it in signals transmitted outside.

input devices (sensors)

transducers >
output devices (actuators)

6
 input devices (sensors) output devices (actuators)

keyboard
voice light

hands mouse monitor

sound boxes
mycrophone printer
eyes

7
3 > INTERFACE: GRAPHICAL VS TANGIBLE

Generally a traditional musical instrument has a

good affordance: looking at its shape its easy to
understand how it works because its mechanism is
explicit and theres a natural correspondence between
gestures and sounds coming out.
A little bit harder is to start to play seriously because
the performer needs technical skills that cant be
acquired quickly.

To play with a computer, instead, seems not so

spontaneous because the computer works with binary
numbers and the performer needs to manipulate
them.
It could be easier playing this way if technical skills
required would be faster to acquire then those
necessary for a traditional music instrument.
So the critical point is the interface: how the
performer interacts with the computer, what he needs
to know and to do to play.

8
How can be translated numbers?
In a graphical user interface (GUI) binary numbers
are translated into icons, labels, menus, windows
or other graphical elements shown on a screen and
controllable with a mouse, a keyboard or other input GUI
devices.
In this way set of sounds can be collected as digital
data, selected, composed and modified or generated
in real time, all by interacting with their intangible
representation.
Even the interface has a good affordance, the limit is
that the interaction in not always so direct as with a
traditional musical instrument. input > remote control output > arrow

On the contrary in a tangible user interface (TUI)

binary numbers are represented physically, as real informations encoded as binary numbers
objects linked to digital informations that can be
controlled directly by grabbing and manipulating
them. TUI

It means that a tangible representation could allow

to restore a more intuitive interaction between
performer and electronic music devices using human
natural knowledge about how things work in everyday
life.
input > control output > sound
To play music could became easier like playing with
building bricks, moreover all the physical world could
turn into a musical instrument...

informations encoded as binary numbers

9
4 > FROM BLOCK JAM TO...
< Block Jam is a musical interface controlled by the
arrangement of 25 tangible blocks. By arranging the
blocks musical phrases and sequences are created,
allowing multiple users to play and collaborate.
The system takes advantage of both graphical and
tangible user interfaces. Each block has a visual
display and a combination of a gestural input and
a click-able input. Each Block metaphorically
10
contains a sound group that can be chosen via the
gestural input, the click-able input changes a block
functionally (e.g. start or stop a sequence). Thus,
musically complex and engaging configurations can
be rapidly assembled. The tangible nature of the
blocks and the intuitive interface promotes face-to-
face collaboration, and the presence of the GUI allows
for remote collaboration across a network. > [1]
 < All the blocks consist of:
> an LED display;
> a switch mechanism under the display, so by
pushing the display the user can toggle modes;
> connectors on each side, which provide power, and
serial data;
> magnets on the side so that the blocks easily snap
together;
> infrared opto-reflector* sensors embedded into the
display to detect simple gestures.> [2]

*Opto-reflector detects gestures by mesuring

the change of light reflected between an infrared
transmitter like LED and an infrared receiver.
It works good with black and white surfaces and for
short distance between the sensor and the reflective
surface.

[1] > Block Jam website

11 [2] > Block Jam website
12
< Percussa AudioCubes are a collection of cubes *Infrared light is an electromagnetic radiation
made of white translucent plastic, containing outside the range of human vision.
a rechargeable battery and sound processing/
communication electronics, and a full colour light
source.
[...]
Each side of the AudioCube except the bottom and
top sides have a communication port capable of
transmitting and receiving digital or analog signals,
using infrared light*, to and from other cubes nearby.
Cubes can communicate with each other through
this port and can exchange sound signals or data
depending on the software running in the cube.
A computer can be connected to the USB port, and
can tell the cube to run a software plugin, change
a parameter in the software plugin, or change its
colour. > [3]

13 [3] > Percussa AudioCubes manual

5 > FROM REACTABLE TO...
< The ReacTables main user interface consists of
a translucent table. Underneath the table is situated
a DV video camera, aimed at the underside of the
table and inputing video to a personal computer.
There is also a video projector under the table, also
connected to the computer, projecting video onto
the underside of the table top that can be seen
from the upper side as well.
Placed onto the table are the tangibles that have
fiducials attached to their underside which are seen
14
through the table by the camera. Fiducials are printed
black and white images, consisting of circles and dots
in varying patterns, optimized for use by reacTIVision.
ReacTIVision then uses the fiducials to understand the
function of a particular tangible.
Most of the tangibles are flat, with one fiducial
on the underside.
Some other tangibles are cubes, with fiducials
attached to several sides, allowing those tangibles
to serve multiple functions. > [4]
 Fiducials

15 [4] > Wikipedia web site

 Zoundz is composed by a support with speakers on
which move little objects who contain RFID* tags.
The support contains three RFID readers and when you
start moving the objects they change effect.

16
*RFID (Radio Frequency IDentification) allows
recognition at distance by using radio waves.
It needs a tag, to attach to something to read,
and a reader.
The tag contain a circuit to store informations and
an antenna to communicate with the reader.
Every tag as an own ID, so everything tagged
can be distinguish from the rest.

17
6 > THE GOOD, THE BAD AND THE FUTURE

BLOCK JAM PERCUSSA AUDIOCUBES

interface > GUI + TUI interface > TUI

technology > gestures recognition with infrared light technology > communication with infrared light

componentes > limited components > limited

space > limited space > unlimited

all the compontents need to be phisically cubes can communicate even if they are distant
connected to be able to play

By mixing all the good solutions lets try If it seems over the top, we can think smaller.
to immaginate the future. RFID readers start to be embedded into mobile
phones, and RIFD identification is always more
Tangible user interfaces and tecnologies able used instead of barcode to control the goods.
to recognize both tags and human beings. Its not so hard to imagine to play with a mobile
Components unlimited: everything can be tagged phone, reading tagged object in a supermarket,
or recognized from its shape or movement. everyone with a particular sound held in its tag...
Space unlimited: what today happens on a table
tomorrow will happen in the environment.
In this way everything could became
the representation of a sound and every place would
able to receive signals and output music.

18
REACTABLE ZOUNDZ

interface > TUI interface > TUI

technology > video recognition technology > RFID recognition

components > unlimited components > limited

tags can be pasted to every objcet zoundz tags are unavailable

space > limited space > limited

video recognition of the tags works only on the table components are programmed to work
only on the table

19
SOURCES

textes > Gillian Crampton Smith + Philip Tabor, Telecomunicazioni courses pdfs
textes > Bill Moggridge, Designing interactions

textes > Ross Bencina and Martin kaltenbrunner,

textes ReacTIVision: a computer-vision framework for table-based tangible interaction
textes > James Gibson, Hiroaki Nakano and Henry Newton-Dunn,
textes Block Jam: a tangible interface for interactive music
textes > Percussa Audio Cubes, manual
textes > Zoundz, Use & care manual

websites > Music Technology Group, ReacTable

websites > http://mtg.upf.edu/reactable
websites > Sony Computer Science Laboratories, Block Jam
websites > http://www.sonycsl.co.jp/IL/projects/blockjam/index.html
websites > Wikipedia
websites > http://www.wikipedia.org
websites > Percussa, Audio Cubes
websites > http://www.percussa.com
websites > Zizzle, Zoundz
websites > http://www.zizzle.com/products/product-zoundz.html

20