Shadow Triggered Interaction
in a wayfinding scenario
ABSTRACT
A wave of experimentation in the emerging field of pervasive
computing appears to be challenged and driven by the physicality
of the human body. Some designers and researchers have
acknowledged the richness, subtlety and a high degree of
coordination which an interaction enabled through the use of
bodily gestures can only offer. A gesture based interaction
exhibits a range of intangible but intuitive affordances before its
participants. It is a design space where “to perform an act” on an
artifact does not necessarily require “to touch” it.
On the same lines, the project began exploring the idea of finding
how gestures can actually influence interactions. How do gestures
transform from just being able to communicate a notion to being
able to affect an interaction. This paper would illustrate our
probes in to these simple questions. We would try to bring the
discourse which has resulted in recognizing shadows as direct
transformations of human gestures. Within the definition of
pervasive computing, the causal relationship between human
gestures and a human shadow deserves to be a basis of a sustained
and an intuitive interaction between a participant and an
intelligent environment. This thought is extended by analyzing its
relevance in the context of wayfinding.
We present that wayfinding as an activity of reaching to a
destination demands more attention. The success of a design
solution aiming to aid a user in finding a way critically depends
on easing anxieties, building trust and confidence. Shadows being
very personal to a human being can help in doing the same.
Specifically, the paper sketches a hypothetical wayfinding
scenario where the interactions between a user and the system are
triggered by the user’s own shadow. On the basis of this scenario,
we identify various issues, corresponding merits and demerits
regarding shadows and their interactions.
Keywords
Interaction Design, Pervasive Computing, Gestures, Shadow
based interactions.
1. INTRODUCTION
The Scientific American (1991) published a seminal paper [11] by
Marc Weiser describing his vision of Ubiquitous Computing, now
also called pervasive computing. In his paper, Weiser claimed that
the most profound technologies are those that disappear. They
weave themselves in to the fabric of everyday life until they are
indistinguishable from it. He embraced a new way of thinking
around technology, one that draws its groundings from natural
human environments and which allows the computers themselves
to merge with the background [1]. This vision is still quite alive
and contemporary in the current context of fast paced
technological growth around human beings. Today with a deep
penetration of global networks, wireless technologies and an ever
growing computing power, it is imperative to try and visualize
alternative ways of letting users interact intuitively [12] with their
surroundings. We see this as a process requiring long established
metaphors to change, a process which poses a newer set of
challenges and a process which yields a bundle of closely knit but
context adapted services addressing human needs.
Major universities and industry have come forward to establish
focus groups on pervasive computing viz. project Aura [14] at
Carnegie Melon, Endeavour [15] at University of California,
Berkley, Oxygen [17] at MIT and Portalano [16] at Washington.
Another group of early torch bearers includes Ars Electronica in
Austria, ZKM in Karlsruhe, ISEA in Netherlands and New Media
Institute in Frankfurt. A study of their project documentation
reveals that there are two major directions being pursued: One
which talks about taking this change on to our skins/ bodies (e.g.
wearable technologies and smarter gadgetry) and the other which
talks about making the environments we inhabit more intelligent
and smarter but intuitive. This paper accepts the latter as its
premise of investigation. It raises a sequential set of enquiries
beginning from understanding the relevance of human gestures,
their ability to inflict changes in shadows and finally how
shadows can be used to initiate smarter interactions in the
particular case of wayfinding.
2. RELATED WORK
A gamut of related projects in the field of pervasive computing
exists. However, to preserve the focus we mention the studies
specifically regarding a participant interacting with an image (of
self or of others) on a screen. This interaction is often enabled
through the recognition and digitization of human shadow and
then using this signature to solve a problem or to arrive at a
research understanding.
An important set of experiments were carried out by Myron W.
Krueger [10]. His Video Place and Video Desk systems [10, 9,
and 7] made use of silhouettes for communication between the
remote participants. A camera placed at an angle captured the
image of the participant as well as his silhouette. The digitized
versions of these silhouettes were then made available for
interaction with the participants. The participants performed
various gambits like winning graphic objects or balancing to
maintain one’s own silhouette on a screen. These ideas did not
seem to have any immediate applications at that time but they
1
draw invaluable insights into the behavior of human beings with
their silhouettes and shadows. The researcher found out that users
had a natural ease in interacting with their shadows on the screen.
They were also aware of the relative change in the sizes of their
silhouettes depending on their movements on the floor.
In Krueger’s work the projections used were a result of
orthographic arrangement of lights. Garth Shoemaker [7] in
“Shadow Reaching” extended it by using a perspective projection
and then used the shadow (larger than life in this case) to reach to
a distant point on a large interactive screen. The participant could
then interact with a graphic element on the screen by moving on
the floor.
A more immediate response is also seen from the research [9] on
“Computer Supported Collaborative Work (CSCW)” employing a
white board metaphor. Interactions were designed to bring group
awareness while a number of participants worked together to edit
a document. These participants were distantly placed but provided
with a basic mechanism to capture their shadows and broadcast
the same to the rest of the participants. Individual participants
could see each other’s activities in the form of dynamic
silhouettes appearing in a portion of their screens. The silhouettes
were thus an essential part of the shared document being created.
Another system using a shadow to support collaborative drawing
between two sites is the Video White Board system by Tang and
Minneman [18].
These examples provide strong evidence for the usefulness of the
shadows and their silhouettes in alternative and innovative modes
of human computer interaction. One can infer the following points
from these projects:
• Shadow based interactions support a natural ease in
understanding the system. These, in turn, promote
better user participation with a smooth and fast
learning curve.
• Shadows have been proved instrumental in interacting
with large interactive display surfaces (LIDS). These
interactions involve reaching to a distant part of the
screen, virtually picking up a graphic interface element,
and then interacting with the same to perform a
particular task. Reaching to a distant part of the screen
requires a participant to vary his position on the floor
relative to the light. While an act like interacting with a
graphic interface element on the screen results out of
shadow’s inherent ability to follow human gestures.
• In a collaborative set up, shadows help in creating
awareness and establish individual identities.
With this understanding, in the following section we would delve
a little deeper to know gestures and their relevance in improving
human computer interactions.
3. GESTURES: MEANING AND
RELEVANCE
Gestures are fundamental to human cognition and form a large
part of non verbal communication. Almost all living creatures
including human beings in particular are bestowed with an
inherent ability of being able to communicate using gestures,
often at times in absence of any language vocally or in writing.
Gestures are thus more about expressing and conveying rather
than necessarily telling about a thought or an action. Infants
communicate their feeling using gestures [19] when they cannot
speak during a part of natural language development. For adults,
gestures aid in cognition and linguistic communication. While one
speaks, gestures help verbalize difficult thoughts [20] and ease in
conceptually planning the speech [21]. A similar set of findings
suggests that gestures lighten cognitive load for both adults and
children [22].
Relevance of gestures in future human computer interactions
The field of human computer interactions emphasizes on making
the system intuitive. This is done to facilitate easy interactions
without expecting users to go through a steep learning curve.
Gestures with their known positive effects on the human mind
[20, 21 and 22] can help in achieving the said performance level.
Users can interact in a pervasive computing environment with an
incomparable flexibility using their own gestures.
Use of gestures frees an interaction from the limitation of using a
single interface element for multiple tasks. In other words, it
negates the homogenization of physical performance. In a
conventional desktop set up, there is no difference between the
task of editing an image, writing a document or sending a
greeting. All these tasks and many others are invariably confined
around a screen, a keyboard and a mouse. Such a system hinders a
user’s thinking and communication. McLuhan [23] identifies this
as an amputation of human senses. Fortunately, gestures can just
do the opposite!
4. INSIGHTS INTO HUMAN SHADOW
We now know from the stated studies that gestures, shadows and
human interactions are closely coupled entities. They seem to
complement each other and can make innovative and intuitive
interactions in pervasive computing a reality. However as we
bring the context of wayfinding later in this paper, we want to
analyze the nature of relationship that exists between shadows and
human beings. We avoid any digressions into deeper
interpretations of shadows available in psychology, culture and
physics but we prefer a brief mention of these to preserve the
diversity of the subject.
Carl Jung [4] describes shadows as “composed of the dark
elements of the personality, having an emotional and primitive
nature which resists moral control”. Though his explanations
seem a little less motivating in context of utilizing shadows in
interaction design but prove that there are strong linkages between
human beings and their shadows around their individual identities
and existence.
At a creative front in most parts of Asia, shadows [3] are used by
puppeteers to tell stories before their audience. It is important to
see a puppeteer performing on the stage through the shadows. He
can sense the ground on the stage through the feet of the puppet.
Another important facet of a shadow is its dynamics with respect
to the object it is formed of and the light it is resulted of. In case
the object is a non living entity, its shadow would expand or
2
contract depending on the relative position of the source of light.
However, in case of a human being (and other living creatures),
this interplay gets richer as the size and form of the shadow now
also depends on the position of the human being relative to the
light.
Each of these explanations may look insufficient in parts but as a
whole they offer few significant outcomes which are:
Identity: A user can easily identify with his own shadow. Shadows
bring individual awareness,
Performance: Notice the way a puppeteer performs a shadow
play. He successfully remains in the background manipulating
leather puppets by his hands to cause a desirable and a specific
change in the shadows on the screen. This illustrates a high degree
of performance in a physical interaction,
Scalability: Puppeteers know exactly where the shadows of their
puppets are landing on the screen. They vary the distance of their
puppets from the screen in order to change the size of their
shadows on the screen, and
Simplicity: The fact that shadows gain more definition with a
brightening source of light while diffuse with a dull one presents
these artists an opportunity to follow almost a cinematic language
in these plays. This can bring smoothness to interactions and can
lessen cognitive loads.
5. CONNECTING NODES
Based on accompanying discussions, we have seen that there
exists possibilities of harnessing gestures and in turn a
participant’s ability to interact with an intelligent system using
shadows as an interaction tools. These interactions will not only
be intuitive and novel but will reflect a graceful integration of
technology in to the human environments. The latter is a core
thought in Weiser’s vision of pervasive computing [11].
However, we also note that there seems to be no apparent design
context in the research and work done related with image assisted
interactions. In some cases if it exists [9], it is either too
immediate with a lesser scope or too technical to be included in a
design discussion. In this paper, we try to bring context awareness
to such experimentations. We believe that gestures and shadows
have a lot to influence the future use of technology. To
demonstrate this belief of ours, we analyze different observations
and findings gathered till this stage in the light of a wayfinding
activity.
6. WAYFINDING IN A PERVASIVE
COMPUTING ENVIRONMENT
Wayfinding can be viewed as an activity of reaching to a
destination by an individual in a familiar or an unknown
environment. This is also one of the earliest human activities
necessary for survival. For an individual in an unknown
environment, it combines initial fears of getting lost, a mix of
conflicting trust and confidence and a dire need of relevant
information in a comprehensible format. Shadows, owing to their
closeness to human existence, can address some of the critical
issues related with sustaining identity, trust and confidence in a
user’s conscious in a wayfinding scenario.
Contemporary design solutions aiding an individual in reaching to
a preferred destination are still developed around smarter gadgetry
in the form of software applications. This is not the direction
which we ever intend to pursue in contrast to enormous
interaction possibilities lying in the field of pervasive computing.
Pervasive computing embraces a complete disappearance of
technology from a user’s mind.
Our user persona supports mobility, a nomadic life style, a
freedom from gadgets (one or many) and a ubiquitous availability
of information. In the scenario, the user is not expected to possess
any prior knowledge of the system. He is simply invited to explore
this system. He can further interact with the system to know some
relevant information or simply leave the same at any time.
Scenario
The participant has come to visit a friend’s family in Bangalore
for few days. It’s his first visit to the city and he has active plans
to see all the different tourist attractions. He prefers walking on
city streets, clicking photographs and exploring new places. But
today was a little unusual for him. In an attempt to catch a play at
a local theatre called “Ranga Shankara” in the evening, he
missed picking up the city map guide. He is now stuck in the midst
of roads and dead ends. He looks for someone to ask for
directions only to discover that this is not a good approach as he
does not share a common language with the local residents.
Fig A: Proposed set up includes a light source, a light sensitive
floor and a camera (not shown) to record a participant’s shadow
and gestural inputs. It might also include some printing devices to
print the directions and routes on the participant’s request.
Wondering what to do next, he walks aimlessly a few steps.
Fortunately, he sees an information board inviting him to stand at
a particular location to know the detailed directions to his
destination with respect to his current location. He finds this
exciting and decides to participate. He stands on a light sensitive
floor, under a source of light. His shadow as shown in the fig A is
formed on the floor and is gradually transformed in to an
interactive map (fig B). The shadow thus defines the boundaries
of this interactive map. The participant sees a bird’s eye view of
the entire city in this map and his current location. But the
interactive map as contained within the boundary of his shadow
shows only a part of the entire city. He however notices that the
3
interactive map responds to a change in his body movement (fig
C). He explores this further to know that;
• If he leans backwards, the map zooms out and vice-
versa,
• In case he leans sideways, the map moves in a
complementary direction, sort of paginating to show
more locations,
• The underlying system is smart enough to pick up the
location name as audio inputs and then project the
same on this map, and
• He then prints a map of the route he needs to follow to
reach the theatre.
He finds this method of interacting with a map pretty exciting and
would not actually mind telling this to his friends.
Fig B: As soon as the shadow of the participant falls on the floor,
an interactive map emerges within the boundary of the silhouette.
The red spot shows participant’s location on the map.
Fig C: A participant can interact with the map through gestures.
7. DISCUSSION
While we anticipate a user’s reaction and decide over the
implementation feasibility in the stated scenario, we note the
following:
Easy affordance
Previous work done in this field suggests that users immediately
understand how to interact during such interactions. This
reinforces the thought that shadows provide intuitive intangible
affordances to the users in carrying out a task. It seems easy for
the users to understand the significance of the change in shadow
size as they move their body on the light sensitive floor. Also, a
gradual formation of an interactive map gives users a sense of
entering into the context.
Context visibility
Currently the scenario is written with a single user in focus.
However if we consider the boundaries of this interaction space to
be open and a multiuser engagement, we note that it is beneficial
to let a lot of activities remain visible to other users and
onlookers. This will bring its own challenges around personal
privacy and related issues. At the same time, an active
collaboration seems quite paying in a case like Wayfinding where
an inability to understand foreign language could also be a design
constraint. Collaboration is known to improve learning of the
system variables and to facilitate coordination among the users.
Trust and confidence
Maris and Shirley [5] define trust and confidence in relation to the
field of Human Computer Trust (HCT) as:
Trust is the extent to which a user is confident in and willing to
act on the basis of recommendations, actions and decisions of an
artificial intelligent aid, while Confidence arises as a result of
specific knowledge; it is built on reason and fact.
Based on above definitions, we can say that a user entering the
interactive system as explained in the scenario is more likely to
experience a better degree of trust and confidence while
interacting with the system. It is seen that a user in a wayfinding
scenario exhibits a high degree of anxiety caused out of
disorientation and a feeling of being lost. Confronting something
personal and known, like one’s own shadow is likely to ease a
user in this situation. We thus make the proposal of showing an
interactive map bound within the shadow of the participant in the
stated scenario. We would also like to examine our claim here on
grounds of studies done in the field of human computer trust.
Apart from these, the proposed scenario also shares an unresolved
set of implementation issues with similar work done in the field.
These includes issues related with (a) image rendering, (b)
customization in terms of color coded images rather than
monochromatic ones, (c) background cancellation and a proper
shadow definition, (d) performance issues during a continual
interaction, (e) image correction using dynamic lights rather than
fixed ones, (f) scalability for being truly a pervasive technology,
and (f) high energy demands.
8. CONCLUSION
This paper is not intended to conclude by establishing shadow
triggered interactions as the only solution in a wayfinding
scenario. Instead, it calls to stretch our thinking about the human
computer interaction with a new vision for pervasive computing.
We acknowledge the experimental work done in this field but
realize an absence of a firm design context defined around basic
human needs and nature.
4
We argue that gestures have a potential to influence interactions [10] Myron W. Krueger, Thomas Gionfriddo and Katrin
in their newer incarnations within the scope of pervasive Hinrichsen
computing. These should be put at the core rather than at the VIDEOPLACE – An Artificial Reality
periphery of such an interaction. We further present shadows as CHI ’85 USA
an integral part of human identity. It is also illustrated that they
can be made to closely follow human gestures in an intelligent [11] Weiser, M.
environment. This becomes our basis to sketch a scenario The Computer for the 21st Century.
involving shadow triggered interactions to ease the activity of Scientific American , September, 1991
wayfinding.
[12] Richard Saul Wurman
The scenario presented in the paper is unique as it claims that Information Anxiety
shadow based interactions would bring more trust and confidence
in a user’s conscious. [13] Donald A. Norman
Design of Everyday Things
REFERENCES
[14] Project Aura
[1] M. Satyanarayanan http://www.cs.cmu.edu/~aura/
Pervasive Computing: Vision and Challenges Accessed: Nov 2009
IEEE Personal Communications, 2001
[15] Project Endeavour
[2] Jack Zhao, Andrew Vande Moere http://endeavour.cs.berkeley.edu/
Embodiment in Data Sculpture:A Model of the Physical Accessed: Nov 2009
Visualization of Information DIMEA’08, September 10–12, 2008,
Athens, Greece [16] Project Portalano
http://portolano.cs.washington.edu/
[3] Scott R. Klemmer, Björn Hartmann, Leila Takayama Accessed: Nov 2009
How Bodies Matter: Five Themes for Interaction Design
DIS 2006, June 26–28, 2006, University Park, Pennsylvania, [17] Project Oxygen
USA http://oxygen.csail.mit.edu/
Accessed: Nov 2009
[4] Carl Jung
The Shadow [18] Tang, John C., Minneman, Scott L. (1991):
Understanding Dreams P 112 VideoWhiteBoard: Video Shadows to Support Remote
Collaboration,
[5] Maria Madsen, Shirley Gregor Proceedings of ACM SIGCHI 1991, pp 315-322
Measuring Human-Computer Trust
Central Queensland University Gladstone, Australia [19] Iverson, J. and S. Goldin-Meadow
Gesture paves the way for language development.
[6] Desney S. Tan, Randy Pausch
Psychological Science 16(5). pp. 367-71, 2005
Pre-emptive Shadows:Eliminating the Blinding Light from
Projectors [20] Church, R. B. and S. Goldin-Meadow
CHI 2002, April 20-25, 2002, Minneapolis, Minnesota, USA The mismatch between gesture and speech as an index of
transitional knowledge.
[7] Garth Shoemaker, Anthony Tang, and Kellogg S. Booth Cognition 23. pp. 43-71, 1986
Shadow Reaching: A New Perspective on Interaction for Large
Wall Displays [21] Alibali, M. W., S. Kita, and A. Young
UIST'07, October 7-10, 2007, Newport, Rhode Island, USA
Gesture and the process of speech production: We think,
[8] Hiroshi Ishii and Brygg Ullmer
therefore we gesture.
Tangible Bits: Towards Seamless Interfaces Language and Cognitive Processes 15. pp. 593-613, 2000
between People, Bits and Atoms
CHI ‘97, Atlanta GA USA [22] Goldin-Meadow, S., H. Nusbaum, S. D. Delly, and S.
Wagner
[9] Mark Apperley, Laurie McLeod, Masood Masoodian, Explaining math: Gesturing lightens the load. Psychological
Lance Paine, Malcolm Phillips, Bill Rogers and Kirsten Thomson Science 12(6). pp. 516-22, 1991
Use of Video Shadow for Small Group Interaction Awareness
on a Large Interactive Display Surface
[23] Marshall McLuhan
Fourth Australasian User Interface Conference (AUIC2003),
Adelaide, Australia Understanding media: the extensions of man
Edition 2 - 2001