Virtual reality and human

ACKNOWLEDGEMENTS
I would like to thank Prof.G.Potdar for allowing me to talk on
this particular subject, Mr. Shyam Deshmukh for his encouragement and
enthusiasm, inaluable comments.
!hanks are also due to all my colleagues and friends in the
Department of "omputer #ngg. in P.I.".!. who hae helped me along the way.
Pune Institute of "omputer !echnology Page $o. %

Virtual reality and human
INDEX
"hapter %& Introduction ''(
"hapter )& "omputer Graphics *endering ''+
"hapter (& ,isual Perception in *ealistic Image
Synthesis''-
(.% .uman ,isual Perception ''-
(.) "omparing *eal and )D "omputer Graphics
Scenes '.../
"hapter +& Interactie "omputer Graphics Scenes'...0
+.% 1rigins of ,irtual #nironments''0
+.) ,irtual #nironment !echnology Issues'...%2
+.( .uman 3actors Issues''%%
"hapter -& Simulation 3idelity for Dynamic Scenes'..%)
-.% 4sability #ngineering for ,irtual #nironment
5pplications ''%)
-.) "omparing *eal and Simulated ,irtual
#nironments''%(
"hapter 6& Memory and Perception''%/
6.% Mental Imagery ''%/
"hapter /& "ase study of 4.S. Military training '')2
"hapter 7& Summary''))
Pune Institute of "omputer !echnology Page $o. )

Virtual reality and human
Chapter 1
INTRODUCTION
!he aim of interactie computer graphics scenes and ,irtual
#nironment 8,#9 technologies for simulation and training is the creation of accurate,
high:;uality imagery and interaction interfaces that faithfully represent a real:world
task situation. *eliable fidelity ealuation techni;ues are essential in order to assess
releant implementations. 5 commonly employed strategy is to compare task
performance in the ,# in relation to the real world scene being represented. Spatial
perception tasks are often incorporated in benchmarking processes as such, since spatial
awareness is crucial for human performance efficiency. Such approaches, howeer, are
not sufficient to assess the fidelity of ,# systems. !hey are often limited to a specific
application and are not based on formal frameworks but in most cases on arbitrary
selected spatial perception tasks.
Pune Institute of "omputer !echnology Page $o. (

Virtual reality and human
Chapter 2
COMPUTER GRAPHICS RENDERING
<hat is a realistic image, whether it is painted, photographed or
computer:generated= !he term >realistic? is used broadly to refer to an image that
captures and displays the effects of light interacting with physical objects. !hus,
realistic images are treated as a continuum and the techni;ues used to create them are
characterised as >more? or >less? realistic.
Images referred to as >photorealistic? attempt to synthesi@e the field of
light intensities that would be focused on the film plane of a camera aimed at the
objects depicted. !he ;uest for isual realism for static 8)D9 or interactie 8(D9
enironments is a challenge for the field of computer graphics. !he production
8rendering9 of realistic imagery re;uires a precise generation of lighting effects, which
inoles the simulation of physical phenomena including light emission, propagation
and reflection as well as the simulation of physical entities as objects and space.
*endering systems can now approAimate the physical distribution of light in an
enironment. .oweer, physical accuracy of light and geometry does not guarantee that
the displayed images will hae authentic appearance to the human eye or that they will
seem >real?.
Pune Institute of "omputer !echnology Page $o. +

Virtual reality and human
Chapter 3
VISUAL PERCEPTION IN REALISTIC IMAGE
SYNTHESIS
In recent years, research in realistic image synthesis has included
perceptually based rendering, considering aspects of the .uman ,isual System 8.,S9
to produce faster or more realistic computer graphics images. !he basic goal of realistic
rendering is to create images perceptually indistinguishable from real scenes. Since the
human obserer judges the fidelity and ;uality of the resulting images, the perceiable
differences between the appearance of a computer graphics image and its real world
counterpart should be minimised. !hus, isual perception issues are clearly inoled in
realistic rendering and should be considered at arious stages of computation, rendering
and displaying. !his specific research direction has gained much attention of the
computer graphics research community, motiated by the progress in physiology,
psychophysics and psychology in proiding computational models of the .,S. In this
section, principles of human isual perception that are employed are going to be
reiewed.
5lso, research that utilises principles of human isual perception
towards efficient computer graphics algorithms and image ;uality metrics is going to be
mentioned. It is useful to demonstrate that such knowledge is inaluable for the
progress of the computer graphics field.
(.% .uman ,isual Perception
!he ability of a person or animal to detect fine spatial pattern, therefore,
resole detail in an image, is eApressed as isual acuity. !he human eye is more
sensitie to intermediate changes in brightness as opposed to gradual or sudden
changes. 5cuity decreases with increase in distance. 5t any instant, the human eye
samples a relatiely large segment of the optic array 8the peripheral field9 with low
acuity and a much smaller segment 8the central, or foeal field9 with high acuity.
Saccadic moements are rapid jumps of eye position in order to focus to an object.
Smooth and saccadic eye moements shift this high:acuity segment about rapidly so
that acute ision oer a wide angle is achieed.
Pune Institute of "omputer !echnology Page $o. -

Virtual reality and human
If the distance of an object from the obserer changes, conergence
moements keep it fiAated by the foeal field of both eyes. !he .uman 3ield of ,ision
83o,9 is normally around )22 degrees.
311 Per!ept"a# !$%&ta%!' refers to the fact that we perceie a surface as haing a
constant appearance despite changes in the spectral composition of light reflected from
it. 5 number of perceptual constancies hae been identified
3111 Colour constancy refers to the fact that the .uman ,isual System perceies
surface as haing a constant colour despite changes in the type of illumination, e.g.
spectral distribution of illumination. 5n eeryday eAample of colour constancy occurs
when we moe from daylight to an artificially lit room. !he illumination from a
common electric bulb is different from that of sunlight and is relatiely richer in long
waelengths. !he same surface will therefore reflect more long waelength light under
a light bulb than in sunlight, and, if our perception of colour depended on waelength
alone, it would appear redder. In fact, the colour actually perceied remains largely
constantB a white sheet of paper, for eAample, does not appear redder indoors and bluish
outdoors. "olour constancy is not always perfect, depending on the indiidual but
generally does not ary a lot, howeer, those with a >good? eye for colour account for
this when, for eAample, shopping for clothes under artificial light.
3112 Lightness constancy refers to the ability of the .,S to perceie surface
lightness as constant despite the changes in illumination.
3113 Shape constancy refers to the ability of the .,S to perceie objects as haing
the same shape despite changes in their orientation. 5 number of researchers suggest
that our usual ability to recognise objects across a range of iewpoints arises as a result
of our eAperiencing and storing different iewpoints
312 Rogers& >It has long been observed that the appearance of three
dimensions in a picture is more striking under certain viewing conditions, such as
viewing the picture with one eye only and by using a peephole or a lens in a reduction
screen or viewbox. Numbers of old viewing devices make use of one method or another
to produce their sometimes powerful effects. The usual explanation is that the restricted
view enhances the effectiveness of pictorialdepth information by reducing the
conflicting flatness information that specifies a pictures ob!ective surface. "eephole
viewing is monocular and head motion is prevented, and this should enhance perceived
depth. # peephole itself also restricts the observers view of the picture itself hiding the
frame and surrounding surfaces. $oss of the visible frame and discontinuous
surrounding surfaces reduce information for the picture as a flat ob!ect %perhaps even
for the presence of a surface at all&, potentially enhancing the illusion of depth in the
picture. This is the oft'cited reason for the success of the various picture viewing
devices.?
Pune Institute of "omputer !echnology Page $o. 6

Virtual reality and human
(.) "omparing *eal and )D "omputer Graphics Scenes
!o actually eAperience the foreground occlusion effect, a simple
techni;ue is to hold up a cardboard boA with a hole in it which just blocks the edges of
a monitor. 3or best results, one could play on the monitor a scene with considerable
moement. "ompare one?s impressions looking through the hole with one eye to an
unobstructed iew of the monitor at the same distance, again with only one eye.
"omparing *eal and )D "omputer Graphics Scenes 5s the goal of realistic image
synthesis is to generate accurate representations of real world scenes, synthetic images
should be compared to their real counterparts using human obserers and taking into
account aspects of the .,S.
Since this is the general methodology employed , research that focuses
on comparisons between real:life and computer graphics simulations is going to be
reiewed. In this section, research that attempts to compare a real:life scene with a
computer graphics static scene is presented, *esearch focusing on comparisons of
interactie computer graphics scenes with real:world task situations, displayed on more
sophisticated displays is going to be reiewed. !he first research effort to compare real
and simulated static scenes side by side was attempted by Meyer et al.. !hey used a -:
sided cube as their test enironment, which was placed in a dark room 8"ornell boA9.
*adiometric alues predicted using a radiosity rendering of a basic scene were
compared to physical measurements of radiant fluA densities in the real scene. !he
results of the radiosity calculations were transformed to the *GC alues from display
following the principles of colour science.
!he study done by Ms. Mania on some participants under irtual
enironment gae results as following.
Measurements of irradiation were made at )- locations in the plane of
the open face for comparison with the simulations. Meyer et al. then proceeded by
transforming the alidated simulated scene to alues displayable on a teleision
monitor. !wenty participants were asked to differentiate between a real enironment
and the displayed image, both of which were iewed through the back of a iew
camera. !hey were asked which of the images was the real scene. +-D of the
participants indicated that the simulated image was actually the real scene, a result
close to chance, e.g. oerall random selections. Participants considered the oerall
match and colour match to be good, howeer, some weaknesses were noticed in the
sharpness of the shadows, mostly due to the brightness of the ceiling panel, caused by
the orientation of the light source. 5lthough there is strong support for the perceptual
alidity of the simulation and display process in this study, there were also some
shortcomings& !he scene was ery simple and the results showed that the participants
could hae just guessed. 5nother approach towards comparing real and simulated static
scenes takes a captured image of the real scene in ;uestion and uses numerical
techni;ues to determine the perceptual differences between the two. *ushmeier et al.
introduced components of a perceptually based metric using idea from the image
Pune Institute of "omputer !echnology Page $o. /

Virtual reality and human
compression literature. Image compression techni;ues seek to minimise storage space
by saing only what will be isible in an image.
!he goal of this study was to obtain results from comparing two images
using these models that were large if large differences between the images eAist and
small when they are almost the same. !he model inspired by Daly had good results and
was also the only one that considers human limits in dark adaptation. It was noted in
this study that the biggest challenge is to take insights into human perception and apply
them to isual simulation directly, computing only as much as is needed to satisfy the
obserer. !ranslating a perceptual model into a progressie calculation is not a
straightforward task, howeer, this is not the focus of this .

Mys@kowski completed a comprehensie alidation and calibration of
the ,isible Differences Predictor 8,DP9 response, using psychophysical eAperiments.
.e subse;uently used the ,DP local error metric to steer decision making in adaptie
mesh subdiision and in isolating regions of interest for more intensie global
illumination computations. !he ,DP was tested to determine how close ,DP
predictions come to subjectie reports of isible differences between images by
designing two human psychophysical eAperiments.
*esults from these eAperiments showed a good correspondence between
human obserations and ,DP results. 5 series of psychophysical eAperiments
conducted by Drago E Mys@kowski. Participants were asked to rate the degree of
realism between two computer graphics renderings and a photograph of a real scene9B
all of them representing the atrium of the 4niersity of 5i@u in Fapan. Subjectie
responses of perceied realism ."omparing real and synthetic scenes using human
judgements of lightness perception. 1ne of the renderings was rendered accurately in
terms of the illumination, howeer, artists were called to contribute towards the second
rendering according to their aesthetic recommendations. *esults showed that the
rendering of accurate illumination was ranked higher 8in terms of the perceied degree
of realism9 in relation to the rendering artistically composed. In a more recent approach,
Mc$amara et al. introduced a method for measuring the perceptual e;uialence
between a real scene and a computer simulation of the same scene. !he model
deeloped in this research is based on psychophysical eAperimentsB more specifically
on human judgements of lightness when iewing a real scene ,a photograph of the real
scene and nine different computer graphics simulations including a poorly meshed
radiosity solution and a raytraced image. *esults were produced through a study of
ision from a human rather than a machine ision point of iew. !hey showed that
certain rendering solutions, as the tone:mapped one, were of the same perceptual
;uality as a photograph of the real scene.
Pune Institute of "omputer !echnology Page $o. 7

Virtual reality and human
Chapter (
INTERACTIVE COMPUTER GRAPHICS SCENES

5ccording to Carfield and 3urness, a ,irtual #nironment 8,#9 is
defined as a representation of a computer model or database that can be interactiely
eAperienced and manipulated by the ,# participants. #llis characteri@ed a ,irtual
#nironment as a >synthetic, interactie, illusory enironment perceied when a user
wears or inhabits appropriate apparatus proiding a coordinated presentation of sensory
information mimicking that of a physical enironment?. 5lso, in ,irtual #nironments
are defined as interactie irtual image displays enhanced by non:isual display
modalities, such as auditory and hap tic, to conince users that they are immersed in a
synthetic space. 5 irtual image is defined as the isual, auditory, tactile and
kinaesthetic stimuli that are communicated to the sensory receiers of the participants
such that they seem to originate from within the three:dimensional space. 5n interface
is a communication medium between a human and the functional elements of a
machine. 5 irtual interface is a system of signals coming from software or hardware
that form an interactie medium through which, firstly, Information is communicated to
the user?s senses in the form of three:dimensional images and possibly tactile and
kinaesthetic feedback or sound, and secondly, participants? actions are monitored
manipulating the irtual enironment.
+.% 1rigins of ,irtual #nironments
#Aperiencing imagined enironments has been rooted in the human
mind originating as early as human cae art. Gewis "arroll?s >!hrough the Gooking
Glass? describing the adentures of 5lice in the <onderland, is a more modern
eAample of this fascination. 5n enironment which would allow its >inhabitants? to
moe about and manually interact with computer graphics objects was enisioned in
the science fiction plot of <illiam Gibson?s >$euromancer? in %07+. Het, Ian
Sutherland made the first actual implementation of such a system using a head:mounted
stereo display possible much earlier, in the middle %062?s. .e deeloped fast graphics
hardware in %06-, specifically for eAperiencing computer:synthesi@ed enironments
through head:mounted graphics displays& >Don?t think of that thing as a screen, think of
it as a window, a window through which one looks into a irtual world. !he challenge
to computer graphics is to make that irtual world look real, sound real, moe and
respond to interaction in real:time and een feel real.? Myron Irueger deeloped
another early implementation of a synthetic, interactie enironment in the %0/2?s.
4nlike the deice deeloped by Sutherland, this enironment called ,ID#1PG5"#
was projected onto a wall:si@ed screen. <hen a participant touched an object on the
screen, the system generated a graphics or auditory response.
Pune Institute of "omputer !echnology Page $o. 0

Virtual reality and human

!his system did not limit participants? positioning or moement.
Generally, one of the most significant sources of ,irtual #nironment technology
comes from preious work in the deelopment of realistic ehicle simulators, primarily
for aircraft as well as for automobiles and for ships. 1perators of related technology
should be highly trained and since ac;uiring releant training on actual settings could
be either dangerous or costly, simulators could >proide? the content and dynamics of
the control enironment for this purpose. !hese systems are usually ery eApensie and
the applications? properties are speciali@ed. !hey hae recently inoled the use of
.elmet Mounted Displays, howeer, declining computer hardware and display costs
hae allowed for the implementation of personal simulators for eeryday use.
+.) ,irtual #nironment !echnology Issues
In recent days, according to the latest ,# technology reiew by
3rederick Crooks, ,irtual #nironment technologies >barely work?. "heaper high:
resolution irtual enironment systems hae become commercially aailable and some
,# applications are routinely operated for the results they produce. 3ollowing the
reiew by 3rederick Crooks, there are four technologies that are significant for ,#
systems.
!he isual, aural or hap tic displays that immerse the participant in the
irtual world and that block out contradictory sensory information from the real world.
Display technologies include .ead Mounted Displays 8.MDs9, "5,#s, workbenches
and panoramic displays. !he "5,# is an implementation of the 4niersity of Illinois
8it stands for "ae 5utomatic ,irtual #nironments9. It consists of a four:sided 8or siA:
sided9 >room? or >cae? with separate interactie computer graphics projections on each
side, each one drien by one of a set of coordinated image generation systems. !he
workbench configuration lays a rear:projection screen flat and positions the projector
so that the workbench?s length approAimates that of a human body. 1ne, two or more
iewers each perceie a custom:generated image. 5 panoramic display employs one or
more screens that are arranged in a panoramic configuration. !his arrangement suits
group iewers. 5dditionally, >fish tank irtual reality? refers to the use of a desktop
monitor to achiee interactie stereo, head:tracked scenes. !he graphics rendering
system that generates the interactie computer graphics imagery at a specific frame
rate. !he tracking system that continually reports the position and orientation of the
user?s head and limbs. <ith wide:range trackers aailable, though, the arrangement of
related wires has become highly troublesome. !here is commonly, a significant latency
between user motion and its representation to the isual system. !he database
construction and maintenance system for building and maintaining detailed and realistic
models of the irtual world.
Pune Institute of "omputer !echnology Page $o. %2

Virtual reality and human
+.( .uman 3actors Issues
,irtual enironments are enisioned as being systems that will increase
the human computer communication bandwidth by simulating natural human
interaction through appropriate stimulation of sensory channels. !his section proides
an oeriew of human factors issues related to ,# technologies, including .uman
performance efficiency in the ,# which could be affected by user and task
characteristics, human sensory and motor physiology, multi:modal interaction, current
interface and technology limitations and a need for noel interaction metaphors. .ealth
and safety issues related to inaccurate mapping of the human perceptual system with
the irtual interface of which simulator sickness is perceied as the most significant.
Pune Institute of "omputer !echnology Page $o. %%

Virtual reality and human
Chapter )
SIMULATION *IDELITY *OR DYNAMIC SCENES
Simulation fidelity is characteri@ed as the eAtent to which a ,irtual
#nironment 8,#9 and interactions with it are indistinguishable from a real
enironment and releant interactions. !he mapping from the real world enironment
to the computer graphics enironment is mediated by enironmental or isual fidelity.
!he term isual fidelity refers to the degree to which visual features in the virtual
environment conform to visual features in the real environment. Interface or interaction
fidelity refers to the degree to which the simulator technology 8isual and motor9 is
perceied by the trainee to duplicate the operational e;uipment and the actual task
situation. !raining, for instance, in a ,# with maAimum fidelity would yield transfer
e;uialent to real world training since the two enironments would be
indistinguishable. Increases in fidelity, though, could proe to be computationally
demanding. !he effectieness of ,# technology results in a trade:off between
economic and technological ariables, related to e;uipment and computational power.
It is, therefore, important to gain an understanding of which technological or
computational ariables could be sacrificed without degrading performance. !his is
focused on simulation fidelity metrics that are based on a methodology which takes
into account the cognitie processes participants employ to achiee a task in a real
world task situation in comparison to its (D simulation counterpart. In this section,
research related to the sense of >presence? is reiewed since presence is perceied as a
metric for ,# effectieness and fidelity that is not linked to task performance.
Importantly, research that compares a real world task situation to its (D simulation is
gien below in this chapter 8*ef. -.)9.
-.% 4sability #ngineering for ,irtual #nironment 5pplications
4sability has gone far from being an issue only in the design of P":
based software applications and office:based peripherals and deices to a central issue
in the design of a ast range of technologies, particularly handheld and mobile personal
systems and irtual and augmented reality applications. !here is now a ;uestion mark
whether established usability engineering methodologies will be suited to emerging
technologies. ,arious studies hae been conducted to assess the simulation fidelity of
computer graphics imagery and releant interface technology employed for a specific
,# application. !hese studies hae been almost uniformly focused on human
performance efficiency in the ,#, since for an application to be successful, this should
be maAimi@ed and often compared to the real world.
Pune Institute of "omputer !echnology Page $o. %)

Virtual reality and human
Gabbard et al. presented a methodology of ealuating a irtual
battlefield ,# based on
(ser task analysis& identifying a complete description of tasks and
subtasks re;uired to use a system.
)xpert guidelines based evaluation& identifying potential usability
problems by comparing a user interaction design to established usability design
guidelines.
*ormative user'centred evaluation& obserational ealuation method
that includes participants testing the design from early stages.
+ummative comparative evaluation& empirical assessment of an
interaction design in comparison with other maturing interaction designs for performing
the same user tasks.

Ms. Mania& JGabbard strategy borrows elements of traditional
evaluation techni,ues for -raphical (ser Interfaces %-(Is& evolved for the evaluation
of V) systems. -enerally, in order to maximi.e the efficiency of a V) implementation it
is often stated that a set of /design principles should be identified which would enable
intuitive interaction with the V). 0ften content'free re,uirement is ever possible or
useful since application goals differ and generic guidelines normally do not address
specific issues. 1ue to the complex nature of human'V) interaction, V) performance
measures need to focus on more than task outcome to be effective. -enerally, usability
has started to emerge as a necessary component of specific technology
implementations. In recent literature it appears to be closely interconnected to the
design of the system or interaction method in ,uestion.K
-.) "omparing *eal and Simulated ,irtual #nironments
Generally, ,# applications could fall into two categories& *ealistic
,irtual <orlds and Magical ,irtual <orlds . *ealistic ,irtual <orlds are meant as
simulations of the real world, mainly for training purposes such as, for instance, flight
simulators. Magical ,irtual <orlds support actiities that don?t follow >reality? rules
and hae no implications outside of the irtual domain, including, for eAample, data
isuali@ation. 5lthough, this is ;uite a broad and simplified categori@ation with often
shady boundaries, it is a significant separation to be considered for ,# usability
engineering research. *ealistic ,# must take account of means oer ends towards
achieing the application goals for simulations that are meant for transfer of training in
the real world. .oweer, Magical ,* is less constrained by training effects and can
therefore supply techni;ues which are impossible in the real world, for instance,
eAploiting >lack of graity? in a ,# or pointing and teleporting to remote objects. 3or
*ealistic ,#s, one way of getting an objectie baseline for effectieness of an
application is to ealuate that against the real world. ,arious ,# techni;ues could be
compared against that baseline.
Pune Institute of "omputer !echnology Page $o. %(

Virtual reality and human
1ne could argue that, generally, matching real world performance is not the ultimate
goal of ,# research as it runs the risk of limiting the implementation of imaginatie
interaction techni;ues and design in order to match the limitations of the real world.
.oweer, when a ,# application re;uires a high leel of simulation fidelity, comparing
against the real world in a controlled way, proes to be a useful benchmark. !his
methodology is the important element of this. 5 collection of studies that compare an
interactie computer graphics scene to its real world counterpart will be presented in
this section in detail.
$emire et al argued that measurements within a ,# application
could be compared with those in the corresponding real enironment to determine how
effectiely the ,# communicates the conditions of the real enironment. !his study
inestigated spatial orientation in a irtual enironment and compared results with
those obtained from an analogous eAperiment performed in a physical enironment.
!he authors argue that this comparison proides a measure for assessing how a irtual
enironment promotes the spatially related physiological and psychological responses
ordinarily obtained in a physical enironment. In particular, the influence of a pitched
optic array on the perception of graity referenced eye leel 8G*#G9 was inestigated.
Different spatial references may lead to different judgements of eye leel. *esults
indicated that a physical array biased G*#G more than the e;uialent geometrically
identical pitched array. !he addition of two sets of orthogonal parallel lines to the
irtual pitched array resulted in the same bias as that obtained with the physical pitched
array. !hey concluded that knowing which display parameters are essential for task
performance will enable software designers to make informed decisions about the leel
of abstraction that is necessary for performance, thereby resering computational and
display resources for information that impacts the most knowledge about the scene. !he
participants in this study had a slaed head position. Mi@ell et al. inestigated whether
immersie ,# enhance performance oer more conentional displays when isuali@ing
compleA geometry for transfer of training in the real world. Participants of a first
eAperimental study were shown an abstract rod sculpture in real:life as well as on a
typical display with a joystick and on a head tracked .MD. !he participants had to
physically demonstrate their understanding of a compleA structure by trying to make a
real:life duplicate based on iewing the real:life model or a (D model under the
different conditions. *esults showed a significant superiority of the physical
representation oer the two computer graphics representations, but no statistically
significant difference between the monitor:joystick and the .MD head tracked
conditions. Cecause of certain confounds on the eAperimental design, a second study
was conducted. !he iewing conditions were monocular or stereo imagery, different
types of tracking and aried si@e of the geometry displayed, all conducted in a "5,#.
!he head:tracked immersie ,# was shown to proide a statistically significant
adantage oer joystick interfaces, especially in the case where the displayed structure
was shown in super scale surrounding the subject. !here was no difference between
stereo and mono imagery. !his indicates that technological immersion makes a
difference when the geometry is in some sense >immersie? since the natural way to
isualise a set of geometry is to be surrounded by it. 5 study conducted by Slater et al.
compares behaiour in small groups.
Pune Institute of "omputer !echnology Page $o. %+

Virtual reality and human
!he group members perform a task in a ,# and then continue the same
task in a similar real world enironment. !he purpose of the eAperiment was to
compare arious aspects of the social interactions among the group members in the two
enironments and didn?t eAamine task performance. !en groups of three people who
had neer met before were introduced in a shared ,# and carried out a task that
re;uired locating pu@@les on the walls of the (D space and subse;uently soling them.
!he group then continued the same task in the real world. In each group, there was one
participant wearing a .MD with head tracking and two participants who eAperienced
the same enironment displayed on a typical monitor.
!he results suggest that the participant wearing the .MD tended to
emerge as the leader of the group in the ,#, but not in the real meeting that followed
after the ,# eAposure. !he study also reealed a positie correlation between presence
and co:presence 8the sense of >being? with other people9. <itmer E Sadowski also
suggested that comparing human performance in a ,# with performance in the real
world could indicate which aspects of ,# technology need improement. 4sing a
techni;ue alidated to measure real world distance judgements accurately, the authors
compared releant performance in a real world enironment with performance
employing a (D model of that enironment. !he participants in the ,# condition were
placed on a treadmill using a stereoscopic C11M:type monochromic head:tracked
display. !his techni;ue re;uired participants to walk without ision to a target after
iewing it for %2 seconds. ,# accurate distance judgements aeraged 7-D of the target
distance whereas real world judgements aeraged 0)D. !he magnitude of the relatie
errors in the ,#s was twice that in the real world indicating that the ,# degraded
distance judgements.
!he ,# was rendered low ;uality monochrome and, thus, differed
substantially to the illumination in the real world. 5 study was carried out by 4soh et
al. to assess if two ;uestionnaires, widely employed by the research community to
assess the leel of presence, could differentiate between real and irtual eAperiences. It
is argued that releant ;uestionnaires should score higher for the real eAperiences
compared to their computer graphics counterparts.
1ne case interesting study is gien below.
1ne group of %2 participants searched for a boA in a real enironment.
5nother group of %2 participants used a (D replica of the enironment to carry out the
same task. Coth of the ;uestionnaires did not reeal an oerall statistically significant
effect across conditions. !he Slater et al. presence ;uestionnaire showed a significant
difference in two out of the siA ;uestions related to presence in the ;uestionnaire. 5
similar methodology of comparing a real world application with its computer graphics
simulation counterpart is adopted here towards alidating a measure of presence.
!he purpose of the inestigation by <itmer et al was to ealuate how
well a (D replica of a compleA office building displayed on a head:tracked .MD
trained participants to naigate in the actual real world building simulated.
Pune Institute of "omputer !echnology Page $o. %-

Virtual reality and human
SiAty participants studied route directions and landmark photographs,
and then rehearsed the route using the (D model, the actual building or erbal
directions and photographs. !he participants? route knowledge was then assessed in the
actual building. !he results indicated that ,# training produced superior route
knowledge than erbal training, but less than the training in the actual building. It was
argued that ,#s ade;uately representing real world compleAity can be employed as
training media towards learning compleA routes in buildings and should be considered
when the real world space is not suited for training.
*eleant studies focused on spatial perception will be thoroughly
analy@ed. 5s shown in this section, the research community has resulted in comparing
real world task situations to computer graphics simulations for reasons ranging from
comparisons of performance towards ealuating a system, alidation of presence
measures and ;ualitatie assessment of participants? behaior in shared ,#s
eAperiences. !his is proen to be a useful process.
Pune Institute of "omputer !echnology Page $o. %6

Virtual reality and human
Chapter +
MEMORY AND PERCEPTION
.uman Memory is a system for storing and retrieing information
ac;uired through our senses . !he briefest memory store lasts for only a fraction of a
second. Such sensory memories are perhaps best considered as an integral part of the
process of perceiing. Coth ision and hearing, for instance, appear to hae a temporary
storage stage, which could be termed short:term auditory or isual memory and that
could last for a few seconds. In addition to these, though, humans clearly retain long:
term memory for sights and sounds. Similar systems eAist in the case of other senses
such as smell, taste and touch. In this section, theories of mental imagery and memory
are reiewed in detail offering a complete illustration of the elements employed
towards constructing a methodology which will aid ,# systems? simulation fidelity
ealuation.
6.% Mental Imagery
.ow many rooms eAist in one?s house= <hat kind of clothes did one
wear yesterday= !hese ;uestions eoke isual mental imagery. In order to answer the
first ;uestion, one would normally isuali@e hisLher house and then >scan? each room
with an internal >eye?. Similarly, if asked how a sound differs from another sound, one
would report >hearing? the sounds in their mind?s ear. Such tasks eoke auditory mental
imagery. ,isual mental imagery is >seeing? in the absence of the appropriate immediate
sensory input. 5uditory mental imagery is >hearing? in the absence of the appropriate
immediate sensory input. Imagery is distinct from perception that is defined as the
registration of physically present stimuli. Mental images, generally, are iewed as a
form of mental representation generated internally without ade;uate eAternal stimulus.
!hese produce behaioral effects similar to those obtained with corresponding
perceptual representations. .oweer, demonstrating the relationship between mental
images and percepts leaes open ;uestions about the appropriate format for either sort
of representation, at leels deeper than that of conscious eAperience. !he eAamples
aboe illustrate that imagery plays an important role in memory and spatial reasoningB
howeer, imagery also plays a role in abstract reasoning, skill learning and language
comprehension. #en within a single sensory modality imagery is not a single
undifferentiated ability, but rather, imagery inoles a host of processes working
together. !he image of one?s house must be initially formedB it then must be scanned
and also maintained while this is occurring. Moreoer, in many situations an imaged
object must be transformed. 3or eAample, when asked what can be found on the
backside of an object, participants often report >mentally rotating? the object in ;uestion
and >looking? at the releant side of it.
Pune Institute of "omputer !echnology Page $o. %/

Virtual reality and human
Interest in mental imagery can be traced back to the time of Plato who
thought that memories were based on images.
5ccording to his theory, memories are cared into the mind much like
pictures can be cared on a waA tablet. Plato een took account of indiidual
differences in terms, for instance, of the >purity? of the waA and in the ease of caring
figures into it. In more recent days, a lack of sophisticated methodologies for studying
internal mental eents followed a wide criticism on the alue of imagery research.
.oweer, as the limitations of approaches perceiing the study of behaior as the only
scientific method became apparent, researchers started inestigating internal eents.
Subse;uent research showed that mental imagery interferes more with like:modality
perception than with different modality perception, e.g. isual imagery interferes with
isual perception more than with auditory perception and ice ersa for auditory
imagery. 5lso, research showed that despite the fact that images are not actual objects
that must obey the laws of physics, objects in mental images often behae like actual
objects . !he difficulties in characteri@ing the way images are represented and
processed led researchers towards using neuropsychological data to inform theories of
the structure of the processing system. Major areas of research for the study of mental
imagery are learning and memory, perception and action, information processing and
reasoning. Gearning and Memory !he ancient Greeks discoered that one?s memory for
a set of objects could be greatly enhanced if one isualised them interacting in some
way. 1nce one has isualised a scene and encoded it into memory, it can then be
recalled in an image. Much of the mnemonic power of imagery comes from its ability
to represent associations between distinct objects. #en if people do not set out to store
images intentionally, they will often use imagery later to recall information. Imagery is
used to recall the shape, colour, si@e and teAture of objects or spatial relations in scenes
that cannot be inferred from facts associated strongly with them. 3or eAample, imagery
is used to determine whether somebody wears a blue jacket but not whether the same
person has two legs. "ooper states that in arious studies, instructions to employ mental
imagery as a mediator or mnemonic deice could greatly enhance performance on
standard tests of memory. Perception and 5ction Imagery draws on mechanisms used
not only in perception but also in motor control. People sometimes speak of >seeing?
objects in isual images or >hearing? sounds in auditory images. It is also possible that
the motor system plays a significant role in imagery, specifically in >image
transformation? processes as noted aboe. Goldenberg et al. reported the case of a
brain:damaged patient who had cortical blindness but still had isual mental imagery.
!his patient denied that she was blind and this belief could hae been based on her
confusing isual mental images for actual percepts. !he authors note that this patient?s
imagery could hae been triggered by tactile or acoustic perception. 1nce her ision
recoered in the portion of the field that was registered by the presered corteA, non:
isual input no longer appeared to induce the illusion of seeing. !hese findings
suggested that connections from the nonisual corteA play a compleA role in eoking
imagery. Information Processing *epresentations of images, in which information is
stored, are processed in arious ways. !he system in which images are processed can
be described in terms of these major abilities& Image inspection is the ability to eAtract
information that is depicted in an image.
Pune Institute of "omputer !echnology Page $o. %7

Virtual reality and human
In the course of inspecting imaged objects, participants of an
eAperimental study report haing to >@oom in? to >see? small details, and, in fact,
participants re;uire more time to inspect imaged objects of small si@es than objects
imaged at larger si@es.
Image generation is the ability to retain sensory input and actiate long:
memory mechanisms. Many mental images are noel combinations of objects or
characteristics that were encoded at different times and places. Many studies hae
shown that image generation inoles serial processing and thus, the time to form
images typically increases linearly for each additional part of the to:be:isuali@ed
object. !he time to isuali@e a part often can be predicted by the order in which
participants typically draw the parts.
Image transformation is the ability to apply certain transformations to
mental imagery. Image transformations often presere the time course of the
corresponding actual transformation. Image retention refers to the ability of
maintaining an image for imagery tasks that re;uire considerable time to complete.
*easoning mental imagery is significant since this is a means by which information is
learned, stored and retrieed. "onsider how one decides what is the best route to get to
work at rush hour in the morning or if a piece of furniture would fit in one?s home. In
both cases, imagery is used to carry out a kind of >mental simulation?. 5 major
bottleneck in using mental imagery is the capacity of working memoryB this capacity
hinges on properties of a passie store and properties of actie imagery operations.
Mental imagery is also likely to be employed when for eAampleB one isuali@es
mentally a house following a erbal description of its arrangement, or een in abstract
reasoning. Generally, current distinctions between differences in isualLspatial
representations emerging from research in cognitie science and neuroscience may
apply to mental images as well. ,isual imagery like isual representation or memory
8analy@ed in the neAt section9 may not proe to be a unitary concept. It should more
accurately be iewed as referring to distinguishable subsystems speciali@ed for
performing particular aspects of cognitie tasks. Distinctions among subsystems of
imagery speciali@ed for particular tasks will become increasingly sophisticated, as
researchers come to understand better the arieties of isual representation.
Pune Institute of "omputer !echnology Page $o. %0

Virtual reality and human
Chapter ,
CASE STUDY O* US MILITARY TRAINING
Months before 4.S. pilots took the skies aboe 5fghanistan last
1ctoberB they had pretty good idea what they would see there. 5lready they had logged
many hours doing irtual fly M thoughts oer rugged mountain terrain , using a mission
rehearsal system known as !opscene 8tactical operational scene 9 .Cuilt for the 4.S.
department of defense by 5nteon "orp.,3air3aA ,,a.!opscene combines aerial photos ,
satellite images , and intelligent data to create high resolution (D databases of region .
In this !opscene system pilot could isuali@e flying from ground leel to
%)222 meters, at speed up to ))-2 kmLh. the detailed rendering , showing roads ,
buildings , and een ehicles , helped them plot the best approach , scout for landmarks
, and identify designated targets.
*e;uirements of such powerful simulation tools that create irtual
reality effects are sophisticated computer modeling and graphics, faster processor
speeds, and adances in artificial intelligence.
3or such high ;uality irtual enironment re;uires ery high inestment,
and technology. 5ccording to the trade publication military training and simulation
$ews, the 4.S. department of defense spends about 4S N+ billion each year on
simulation and training e;uipments.
Casically the limits of lie training desere to hae ,I*!45G <5*
training. Gien lie training?s limitations, the military is relying more and more on
irtual simulation. Such system hae proen effectie for enhancing motor control ,as
in driing a tank or firing a rifleB decision making as in calculating the resources needed
for combatB and leadership, as in responding to ambush.
Cut maintaining such a ast training infrastructure is costly .Fust
transporting a unit to remote location for training can run seeral million dollars. !he
e;uipments are also pricey. 5 single round for the infantry carried Faelin antitank
weapon costs oer N(2,222.5 Faelin simulator, in contrast costs about the same but
can be fired again and again.
Such trainings are ery beneficial for all military staff. 1er two weeks
course of the eAercise, 4.S. central command staff, based at Mac Dill 5ir force base in
!ampa, 3la., endured all emotional highs and lows Mwhat irtual reality researchers call
JpresenceK.
Pune Institute of "omputer !echnology Page $o. )2

Virtual reality and human
"reating a irtual reality enironment that coninces the eyes and ears is
ery much important. Institute of creatie technology>s 8I"!9 computer graphics rely
on a techni;ue pioneered by Paul Deberec when he was at the uniersity of "alifornia
at Cerkley in mid:%002s known. 5s Image based rendering 8IC*9,it generates images
directly from photos and other images rather than building them from geometric
primities such as polygons . 3or military simulator, IC* could proe useful for
;uickly rendering high resolution terrain and creating realistic urban enironment.
Pune Institute of "omputer !echnology Page $o. )%

Virtual reality and human
"hapter 7
S4MM5*H
"learly, the ultimate test of such research will be not just how
sophisticated the technology is, but hoe well it helps the human. It is, the methodology
based on the memory awareness states, for assessing the simulation fidelity of a
computer graphics simulation of a real world situation is incorporated. !his
methodology is focusing on the cognitie mental processes humans employ in order to
complete a memory task in the real world, as opposed to using a computer graphics
representation of the real world space displayed on a typical monitor as well as on a
monocular .MD. !he memory task includes a isual and a non:isual component. !he
goal of the ,irtual *eality study was to deelop a robust framework based on the
memory semantics methodology that could be subse;uently applied to the main studies
that utili@e a formal eAperimental design. !he leel of reported presence was higher in
the real world compared with the desktop, .MD and audio:only conditionB howeer,
there was no significant difference for presence between those conditions.
5 consistent positie correlation between presence and task
performance was not reealed by researchers. Ms. Mania says,K Spatial memory recall
and confidence scores did not proe to be significantly different across conditions.
Memory awareness states? analysis, howeer, for the spatial task gae an inaluable
insight into participants? strategies of retrieal, most interestingly across specific
conditions where results for presence and accurate memory recall were not proen to be
significantly different. In particular, statistically significant differences were reealed
for the >remember? awareness state linked with mental imagery and eent based
recollections and for the >familiar? awareness state linked with awareness of an eent or
an image that can not be accurately placed in time. 1ne of the most interesting results
showed a significantly higher proportion of correct >remember? responses for the .MD
condition compared with the real:world condition, reealing a high amount of isual
mental imagery as a strategy of retrieal for this condition. !he result is a formal
eAperimental design incorporating a spatial perception task with more elements and
compleAity. 5lso, sophisticated display technologies are incorporated such as stereo
imagery and head tracking whilst allowing the inestigation of the effects of
preconception on spatial memoryK.
3inally in real world conflicts, "an human innoate, understand the
conse;uence of ,irtual reality= "an human accept ,irtual #nironment= Is it possible
to apply Jirtual enironmentK to all human aspects=
Pune Institute of "omputer !echnology Page $o. ))

Virtual reality and human
-I-LIOGRAPHY
Re.ere%!e& .r$/ 0$"r%a#&1
I### Spectrum March )22)
We2 &3te& 43&3te51
%. http&LLwww).discreet.comLindeA:nf.html
). http&LLwww.lightscape.comL
(. http&LLwww.sense7.comL
+. 5uto"5D. http&LLwww(.autodesk.comL
-. !he concept of Presence. F"M"
http&LLwww.ascusc.orgLjcmcLol(Lissue)Llombard.html
6. ,irtual *eality Modelling Ganguage 0/,
http&LLwww.web(d.orgLtechnicalinfoLspecificationsLrml0/LindeA.htm
/. http&LLwww.google.comL
Pune Institute of "omputer !echnology Page $o. )(

Virtual reality and human
PUNE INSTITUTE O* COMPUTER TECHNOLOGY
P4$# M +(
Seminar *eport
1n
Virtual Reality and Human
"resented by
,aibha S. Iulkarni
Pune Institute of "omputer !echnology Page $o. )+

Virtual reality and human
APPENDIX
%. "5,# "ae 5utomatic ,irtual #nironments
). 3o, 3ield of ,ision
(. G*#G Graity referenced eye leel
+. G4I Graphical 4ser Interfaces
-. .MD .ead Mounted Displays
6. .,S .uman ,isual System
/. IC* Image based rendering
7. I"! Institute of creatie technology>s
0. !opscene !actical operational scene
%2.,DP ,isible Differences Predictor
%%.,# ,irtual #nironment
Pune Institute of "omputer !echnology Page $o. )-