Introduction to

Augmented Reality
and Studierstube

Dieter Schmalstieg

First part based on material by Ron Azuma

 What is Augmented Reality?
Fantasy Game
„ Virtual Reality
„ Completely replace
real world
Augmented Reality and Studierstube
City Guide
„ Augmented Reality
„ combine virtual + real
„ Supplement reality
„ Photorealism not a goal
2 Dieter Schmalstieg
 Definition of Augmented Reality
1. Blends real and virtual, in real
environment
2. Real-time interactive
3. Registered in 3-D
„ Applies to all senses (auditory, haptic?)
„ Not an HMD-specific definition
„ Includes idea of removing part of real
environment (a.k.a. mediated or
diminished reality)
Augmented Reality and Studierstube 3 Dieter Schmalstieg
 Milgram’s Continuum
Mixed Reality

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Reality - Virtuality (RV) Continuum

Adapted from Milgram, Takemura, Utsumi, Kishino.
Augmented Reality: A class of displays on the reality-virtuality continuum
Augmented Reality and Studierstube 4 Dieter Schmalstieg
 Why are researchers interested?
„ Enhance perception of and interaction with
the real world
„ Potential for productivity improvements in
real-world tasks
„ Relatively new field with many problems, but
much progress has occurred recently

Augmented Reality and Studierstube 5 Dieter Schmalstieg

 A Brief (and incomplete) History
of AR (1)
„ 1960’s: Sutherland /
Sproull’s first HMD
system was see-through

Augmented Reality and Studierstube 6 Dieter Schmalstieg

A Brief (and incomplete) History of
AR (2)
„ Early 1990’s: Boeing coined the term “AR.”
Wire harness assembly application begun.
„ Early to mid 1990’s: UNC ultrasound
visualization project
„ 1994: Motion stabilized display [Azuma]
„ 1994: Fiducial tracking in video see-through
[Bajura / Neumann]

Augmented Reality and Studierstube 7 Dieter Schmalstieg

A Brief (and incomplete) History of
AR (3)
„ 1996: UNC hybrid magnetic-vision tracker
(first compelling environment)
„ 1998: Dedicated conferences begin
„ Late 90’s: Collaboration, outdoor, interaction
„ Late 90’s: Augmented sports broadcasts
„ 1998 - 2001: Mixed Reality Systems Lab
„ 2000: Custom see-through HMDs

Augmented Reality and Studierstube 8 Dieter Schmalstieg

 Applications: medical
„ “X-ray vision” for surgeons
„ Aid visualization, minimally-invasive
operations. Training. MRI, CT data.
‹Ultrasound project, UNC Chapel Hill.
‹ARAS, VRVis

Video 01
Courtesy UNC Chapel Hill Courtesy VRVis
Augmented Reality and Studierstube 9 Dieter Schmalstieg
 Assembly and maintenance
pictures (1)

Courtesy Andrei State, UNC

Chapel Hill
Boeing wire harness assembly.
Adam Janin wearing HMD.
Courtesy David Mizell, Boeing
Augmented Reality and Studierstube 10 Dieter Schmalstieg
 Assembly and maintenance
pictures (2)

Columbia
University

© 1993 S. Feiner, B. MacIntyre, &

D. Seligmann, Columbia University

Augmented Reality and Studierstube 11 Dieter Schmalstieg

 Applications: annotating
environment
„ Public and private annotations
„ Aid recognition, “extended memory”
‹Libraries, maps [Fitzmaurice93]
‹Windows [Columbia]
‹Mechanical parts [many places]
‹Reminder notes [Sony, MIT Media Lab]
‹Navigation and spatial information access

Augmented Reality and Studierstube 12 Dieter Schmalstieg

 Annotation pictures

Columbia
University

© 1993 S. Feiner, B. MacIntyre, © 1997 S. Feiner, B. MacIntyre,

M. Haupt, & E. Solomon, T. Höllerer, & A. Webster,
Columbia University Columbia University

HRL

Augmented Reality and Studierstube 13 Dieter Schmalstieg

 Application: broadcast
augmentation
„ Adding virtual content to live sports
broadcasts
‹“First down” line in American football
‹Hockey puck trails, virtual advertisements
‹National flags in swimming lanes in 2000
Olympics
„ Commercial application
‹Princeton Video Image is one company

Augmented Reality and Studierstube 14 Dieter Schmalstieg

 Displays for Augmented Reality
„ Optical and Video see-through HMDs
„ Video monitor Augmented Reality
„ Projector based Augmented Reality

Augmented Reality and Studierstube 15 Dieter Schmalstieg

Optical see-through head-mounted
display
Virtual images
from monitors

Real
World
Optical
Combiners

Augmented Reality and Studierstube 16 Dieter Schmalstieg

 Examples of optical see-through
HMDs
Virtual Vision VCAP

Sony Glasstron

Augmented Reality and Studierstube 17 Dieter Schmalstieg

 Video see-through head-mounted
display
Video
cameras Video
Graphics

Monitors Combiner

Augmented Reality and Studierstube 18 Dieter Schmalstieg

Example of video see-through HMD

MR Laboratory’s COASTAR HMD

(Co-Optical Axis See-Through Augmented Reality)
Parallax-free video see-through HMD
Augmented Reality and Studierstube 19 Dieter Schmalstieg
 Video monitor Augmented Reality

Video (Stereo
cameras Monitor glasses)

Video

Graphics Combiner

Augmented Reality and Studierstube 20 Dieter Schmalstieg

Projector-based Augmented Reality

User (possibly
head-tracked)

Projector

Real objects Examples:

with retroreflective Raskar, UNC Chapel Hill
covering Inami, Tachi Lab, U. Tokyo

Augmented Reality and Studierstube 21 Dieter Schmalstieg

 Example of projector-based AR

Ramesh Raskar, UNC Chapel Hill

Augmented Reality and Studierstube 22 Dieter Schmalstieg

 Strengths of optical AR
„ Simpler?
„ Direct view of real world
‹Full resolution, no time delay (for real world)
‹Safety
‹Lower distortion
„ No eye displacement (but COASTAR video
see-through avoids this problem)

Augmented Reality and Studierstube 23 Dieter Schmalstieg

 Strengths of video AR
„ True occlusion (but note Kiyokawa optical
display that supports occlusion)
„ Digitized image of real world
‹Flexibility in composition
‹Matchable time delays
‹More registration, calibration strategies
„ Wide FOV is easier to support

Augmented Reality and Studierstube 24 Dieter Schmalstieg

 Optical vs. video AR summary
„ Both have proponents
„ Video is more popular today?
‹Likely because lack of available optical products
„ Depends on application?
‹Manufacturing: optical is cheaper
‹Medical: video for calibration strategies

Augmented Reality and Studierstube 25 Dieter Schmalstieg

 STUDIERSTUBE
„ “study room”
„ A long term research
project to develop a
powerful collaborative
AR platform
„ 1996-today
Faust and Mephisto
in the study room

Augmented Reality and Studierstube 26 Dieter Schmalstieg

 Application: Medical Visualization

Liver
Planner
„ Developed at
„ ICG.TU-Graz

Augmented Reality and Studierstube 27 Dieter Schmalstieg

 Application: Industrial Design

Spacedesign
„ Collaboration with
„ Fraunhofer IGD
„ Darmstadt

Augmented Reality and Studierstube 28 Dieter Schmalstieg

 Application:
Mediated Reality for Design

„Paint“ the real world

Augmented Reality and Studierstube 29 Dieter Schmalstieg

 Application: Math Education
Teach
geometric
concepts

Roles:
teacher sees
solution,
students don’t

Augmented Reality and Studierstube 30 Dieter Schmalstieg

 Collaborative Augmented Reality
„ = virtual objects + natural communication

„ A user interface for

computer supported collaborative work!
Augmented Reality and Studierstube 31 Dieter Schmalstieg
 AR Videoconferencing

Collaboration with
Fiducial Tracking
Systems in Motion,
+ Real-time Volume Rendering
Norway
+ Low-Bandwith Videoconferencing
Augmented Reality and Studierstube 32 Dieter Schmalstieg
 STUDIERSTUBE
„ “study room”
„ Space as the
user interface
„ → Pervasive Computing

Augmented Reality and Studierstube 33 Dieter Schmalstieg

 Pervasive Computing

„ Make computer support Users

input output
available everywhere
„ Space shared by many
people
Networked pool
of computers
and I/O devices

Augmented Reality and Studierstube 34 Dieter Schmalstieg

 User Interface Design
„ AR = 3D Graphical User Interface
for Pervasive Computing

„ Smart Environments „ Mobile Computing

„ Stationary Devices „ Wearable Devices
SD SD SD SD SD WD WD WD WD

Wireless
Access
Augmented Reality and Studierstube 35 Dieter Schmalstieg
 Smart Environments (1)
„ Application Migration
„ Distributed graphics system
„ System knows about position of users,
virtual + real objects
„ Graphical apps migrate from host to host

Augmented Reality and Studierstube 36 Dieter Schmalstieg

 Smart Environments (2)
„ Tiled Projector Display
„ Scalable display area
„ PC real-time rendering
cluster
„ Software edge-blending
„ Software frame-lock
„ Currently 3x2 mono or
3x1 passive stereo

Augmented Reality and Studierstube 37 Dieter Schmalstieg

 Mobile Computing
Mobile Augmented Reality
Inertial
sensor
camera
GPS tracked HMD
touchpad

notebook Wide Area Tracking

WLAN - DGPS (outdoors)
GPRS phone - Vision-based (indoors)
Augmented Reality and Studierstube 38 Dieter Schmalstieg
 Application: Signpost
„ Wayfinding in
building

Heads-up display of room geometry world-in-miniature

+ direction arrows on tablet

Augmented Reality and Studierstube 39 Dieter Schmalstieg

 Appliction: Library Agent

„ Finding books made

easy...

Augmented Reality and Studierstube 40 Dieter Schmalstieg

Application: Signpost Outdoors
„ Wayfinding in open spaces

Augmented Reality and Studierstube 41 Dieter Schmalstieg

 Ergonomic Considerations

How long until wearable AR

becomes ergonomically and
socially acceptable?
Augmented Reality and Studierstube 42 Dieter Schmalstieg
 Surprising Answer
„ We can start right now!
„ „Handheld Augmented
Reality“
„ Platform: PocketPC 2003,
Intel XScale at 400MHz,
CF plug-in camera, WLAN
„ Fully self contained vision
tracking and 3D graphics
„ 4-5 fps for 3D navigation app.

Augmented Reality and Studierstube 43 Dieter Schmalstieg

Handheld AR
„ Three tracking modes:
‹ Stand-alone
‹ Compressed image via
WLAN to server
‹ Outside-in IR tracking
(passive target)
„ OpenGL compatible
renderer
„ Current problem:
slow camera
„ Application: Signpost
again
„ Shares most data + code
with PC version

Augmented Reality and Studierstube 44 Dieter Schmalstieg

 Some Concluding Remarks (1)

„ Computers are already ubiquitous

„ For most people, the computer is the UI
→ Better UI will be a significant improvement
„ AR leverages real world as UI
→ Users are free to go everywhere
„ 3D Graphics are the most successful interface
→ The World is seen in 3D!
„ What could be a better UI than the real world?

Augmented Reality and Studierstube 45 Dieter Schmalstieg

 Some
Concluding
Remarks (2) EN
D
Maybe a
21st ct. century
version of
Faust‘s tale
would be an
entirely different
story?
Augmented Reality and Studierstube 46 Dieter Schmalstieg