INTERNATIONAL ORGANISATION FOR STANDARDISATION

ORGANISATION INTERNATIONALE DE NORMALISATION

ISO/IEC JTC 1/SC 29/WG 11
CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC 1/SC 29/WG 11m m23798

February 2012, San Jose, CA
Title: Additional Use Cases for ARAF
Authors: Marius Preda, Ivica Arsov (on behalf of the TOTEM project)

1 Mobile Mixed Reality/Augmented Reality Games use case

Augmented Reality and, in general, Mixed Reality combines the real world with a virtual scene and creates an
immersive and interactive real-time environment. The possible applications are almost limitless and encompass
areas like learning, architecture or industrial production. In the gaming area, AR/MR offers unique opportunities.
Unlike with traditional video games, these games are not imprisoned onto the screen but interact with the real world.
They incorporate real locations and objects into the game, therefore tapping into already existing thoughts and
emotions of the player, which in turn creates a potentially much richer gaming world and experience. Mobile
AR/MR Gaming even goes a step further as the playing area becomes borderless and games can be played literally
anywhere and anytime. With the current advancement of modern cell phone technology, we are identifying the
potential of Mobile AR/MR Games being played by anyone.

2 Background and state of the art

Digital games employ digital technology as part of the game experience. In contrast to non computer games, digital
games facilitate the play of games that would be too complicated in a non computerized context.[Salen, 2003].
Game developers can create rich, interactive virtual worlds that real people can inhabit. Through digital technology
and the possibility of representing complex behavior, computer games can deliver new and intense experiences to
players. Mobile or pervasive games reach beyond desktop- or console- based gaming. They overcome spatial,
temporal, and social boundaries of conventional computer games by making the real environment
intrinsic.[Montola, 2005] They bridge the gap between traditional non computer games and existing video games by
offering direct social interaction [Magerkurth, 2005]. Thus, mobile games integrate social components of traditional
games into digital games, extending them into our everyday activities. Location-based or -aware games emphasize
these aspects of pervasive games [Bell, 2006]. The game content and the players interaction opportunities depend
on the players current real-world location. The users local environment (including buildings, plants, people, and so
on) might contain items relevant for game play. Moreover, interaction with some of these items might be part of the
game. This interaction extends to other players who might compete against one another or cooperate as part of the
game when sharing a certain areasimilar to common real-world activities. Finally, pervasive or mobile mixed
reality (MR) games [Broll, 2006] represent distinctive location-aware pervasive gamesthat is, they use MR
technology to enhance or modify the players real environment with virtual content. Instead of being represented by
an avatar like in many traditional computer games, players are their own avatar moving in the game world by
moving in their real environment. Most pervasive MR games are purely outdoor. However, some of them facilitate
combined indoor and outdoor activities. Pervasive MR games typically combine real and virtual content,
storytelling, and the users imagination to create a new type of entertainment experience.

Such systems have the potential to create an ultimate gaming experience [Stapleton, 2003] One of the first mobile
MR (or Augmented Reality) games was ARQuake, developed in 2002 [Piekarski, 2002]. ARQuake is the AR
version of the famous first-person shooter Quake. The aim is identical to the desktop version where the player shoots
monsters and collects items. In the AR version, the players wear AR systems consisting of a laptop (in a backpack)
attached to several sensors for tracking position such as differential GPS for the orientation and a camera for
marker-based computer vision tracking. In 2004, another AR variant of a famous computer game was developed.
The story of Human Pacman [Cheok, 2004] is the same as that of the original arcade game Pacman: Pacman
(represented by one player), has to collect all the cookies in the world before he is caught by a ghost (represented by
another player). The games system consists of a central server, the wearable client system, a Bluetooth device, and
a helper laptop. The client and server communicate via WiFi, and the client is connected to the physical devices via
Bluetooth. The users viewing orientation is tracked by a 3-degrees-of-freedom orientation sensor built into the
helmet and the position is determined via GPS. A special sensor detects whether the player is touched by the enemy
player. NetAttack [Broll, 2006] also developed in 2004, is a scavenger hunt game. The basic idea and technique of
ARQuake was enhanced by having outdoor as well as indoor game play. The outdoor player wears a mobile AR
system and is tracked by GPS and marker-based computer vision tracking. The outdoor player is supported by an
indoor player. The partner inside, sitting at a desktop-based survey system, communicates via Wi-Fi with the
outdoor player, guides him, and helps him find game items. In 2004, AR Soccer [Reimann, 2004] was the first
handheld-based AR game. AR Soccer is a computer vision-based smart phone game. Players shoot a penalty by
kicking a virtual ball into a virtual goal with their real feet. The smart phones camera tracks the players foot.
Instead of an optical flow algorithm, the foot is detected and tracked using a 2D edge extraction approach. In 2005,
AR Tennis [Henrysson, 2005] and The Invisible Train [Wagner, 2005] were developed as marker-based games on
handheld devices. AR Tennis is a multiuser-supported smart phone game that also uses computer vision-based
tracking. In AR Tennis, the smart phones are equipped with markers, which can be detected by the cameras of the
other players phones. The tracking data is transmitted via a peer-to-peer Bluetooth connection, thus enabling the
two players to play tennis virtually with their smart phones. The aim of the multiuser game The Invisible Train is to
steer a train over a wooden railroad track. The player can interact over the touch screen by changing the speed of the
trains and the switches. The Invisible Train is a synchronized multiuser game in which PDAs are connected via Wi-
Fi. The tracking is realized by a marker-based computer vision approach. In Capture the Flag [Cheok, 2006]
developed in 2006, a team catches the opponents flag, which is presented by a small wooden box. This box is
equipped with a Bluetooth device and a touch sensor. The touch sensor tracks whether the participants pick or move
the box. The box and the smart phone are connected via Bluetooth. Mr. Virtuoso is an educational game played as
video-see-through AR on a PDA [Wagner, 2006]. The objective is to sort artwork according to their date of creation
along a timeline. The timeline consists of fiducial markers. Each marker carries one artwork. The virtual expert Mr.
Virtuoso provides information to the artworks. The PDAs are synchronized via WiFi using a server. In 2007,
Epidemic Menace [Lindt, 2007] the successor of NetAttack, was developed. Epidemic Menace uses multiple devices
including PDAs and smart phones. It also takes place indoors and outdoors, and the players communicate via
smartphones. The smart phones display a small map of the game area and the users current position. The PDA
supports position tracking. The outdoor players are optionally equipped with mobile AR systems. TimeWarp
[Herbst 2008] is a Mobile Mixed Reality game set in the oldtown of Cologne. Players travel in time and space to
meet and help the legendary Heinzelmnnchen elves while they have to interact with virtual and real objects alike.
The Alchemists [Wetzel 2008] takes a simpler approach where the players have to collect magical ingredients from
the environment to brew potent potions (see Figure 2). What makes this game unique is the fact that it was built with
the so-called Magic Lens Box, an attempt at providing an authoring environment for Augmented Reality games.
The input features exploited in the games presented above are the location/orientation (thanks to a GPS system, or
NFC tags/QR-codes spread all over the field), the accelerometer, the Bluetooth/GSM/WiFi connection, and the
image/video captured by the mobile phone's camera. One objective of the project is to build (based on existent APIs)
a unified framework to manage and process these inputs.

A second category of information ready to be exploited in mobile mixed reality games is the user/player context. It
is used to trigger the right service/action/reaction in the right moment and is at the heart of two very active research
subjects named Pervasive Media and Autonomic Computing. Context is derived from sensors or from digital
resources (presence services such as CellID, user input, ) and may refer to physical, virtual or social
circumstances. Existent games such as ContextKing [Koolwaaij 2009] already exploit context management
frameworks [Bhm 2008] for collection/aggregation/distribution of context information. For the localization data, as
the main context feature, several solutions were implemented to ensure its computation whichever form of sensor
used [Dey 2001, Roman 2002, Capra 2003, DaRocha 2006, Julien 2006]. To improve the accuracy or to provide
higher level interpretation, sensor fusion is also an active domain reported in the literature [Fox 2001]. Some
commercial efforts, while basic today and mainly based on self-reporting, are available to collect low- (e.g location)
or high-level (e.g. current action) contextual data. One of the most popular is Twitter, a web service based on the
question "What are you doing?" However, the high dimensionality and the very heterogeneous nature of context
information are still posing technical challenges that remains to be addressed: provisioning, discovery, routing,
privacy/trust/protection, diversity [Stenton 2008].

Probably the highest barrier in the development of mobile mixed reality games will remain the authoring of the
content. Mainly there are today two class of approaches: the rapid prototyping toolkits providing re-usable
components-widgets [10 day] or end-user programming tools, more or less intuitive to use [Hull 2004, Barrenho
2006, Biswas 2006]. In both cases, the authoring is performed at the desktop and not in real gaming environment.
Other examples for MR authoring tools include MARS [Gueven 2003], AMIRE [Doerner 2002] or DART
[MacIntyre 2003] all of which are not gaming specific.

Concerning user-created content typical Web2.0 applications like blogs, wikis, and video and photo community sites
are already a big step in this direction. The latest developments, in the form of easily manageable and configurable
web services such as mashups, go beyond the pure management of non-computational media towards creating your
own programs. In the area of video games, user-created games or user-modified ones have become rather popular.
Apart from the modding community of first person shooters [Kcklich, 2005] this trend has also moved over to
other gaming genres as seen in games like World of Warcraft, The Sims or Spore. One of the main aspects of the
latter is the encouragement of user-created content (e.g. creatures) for this game which goes so far that people pay to
use the creature editor and enhance the game world with their creations.

Pervasive games like Mobile Mixed Reality games usually require a huge amount of effort not only to produce the
game but also to stage it. If part of this job could be done by the community, the popularity and economical
feasibility of such endeavors would most likely increase. One way is to make the set-up of the game part of the
actual game play, as it is done for example done in the game Hitchers [Drozd, 2006] where players create game
content themselves and place it in the environment. Full games of course are not possible to create with such an
approach.

Other examples from the non-video gaming area which have managed to build a large community around them
include Geocaching (http://www.geocaching.com), where users create quizzes for others based on GPS technology
and Lego Mindstorms (http://mindstorms.lego.com), where users can script and define the behavior of Lego robots.

3 Technologies for Mobile MR/AR Games

Technical challenges in several domains related to Mobile MR/AR games exist: interaction mechanisms, context
framework and games authoring by community. In addition, several technologies have to be deployed to ensure
content (images/videos/graphics/scripts/metadata) management in the chain (capture, compression, transmission,
storage, adaptation and presentation) and multi-user functionalities (game sessions, lobby servers, IDs).
In order to create compelling Mobile MR/AR Games, tracking plays an important role as the actual position of the
player in the real world translates to his virtual position: The player becomes his own avatar. There are already
several tracking mechanisms well established ranging from GPS tracking, inertial sensor tracking, visual marker-
based tracking to more complex like feature based tracking.
Deriving high level context information from relatively low level sensor data, ready to be exploited in applications
such as games, is still an open issue for mobile multiplayer games. High-level context abstraction is needed, as well
as methods of interpreting and translating sensor data to infer meaningful context situations. Such abstraction level
allows as well classifying the information and uses only the one most appropriate for a specific situation of the game
play. Compact representation, i.e. coding of this contextual information is also necessary, since it is expected to
transmit it to the other players.
In general, the games are one of the most complex multimedia applications. They represent a complete ecosystem,
being composed of media assets (images/graphics/videos) but also a programming layer implementing the behaviour
during the game-play and tools to capture and interpret different kind of inputs. Authoring games is thus a complex
task, involving different participants and to keep the development consistent, a certain level of orchestration is
necessary. Opening this task to the community requires making available powerful yet simple tools and
mechanisms.
Since the main target of the project is to let users/players to create games, including the multimedia assets,
interoperability is an issue. Moreover, the data to be transmitted should be as compact/compressed as possible.
MPEG-4 already provides a set of tools for representing audio/video and graphics assets. Context data as well as
relevant sensory data could be digitized and encapsulated as well in MPEG-4/MPEG-V streams. Such integrated
approach ensures multimedia delivery synchronization and use of standard transport protocols.
Other than on the level of media assets, interoperability should be ensured as well for the game behaviour, scene
description, input mechanisms and communication protocols.
Creation and playing processes will involve several collaborating users. To take care of their data communication,
existent middleware can be used and integrate them with the MPEG Player.

4 Acknowledgement
The Mobile Mixed Reality/Augmented Reality use case was prepared together with Fraunfoher FIT as part of the
TOTEM project (www.totem-games.org), financially supported by German Federal Ministry of Education and
Research and by the French National Research Agency.

5 References

Ahn, L. von, Dabbish, L., Labeling images with a computer game, Conference on Human Factors in Computing
Systems archive, Proceedings of the SIGCHI conference on Human factors in computing systems table of contents,
Vienna, Austria, pp 319 326, 2004, ISBN:1-58113-702-8
Barrenho, F., et al. (2006). InAuthoring environment: interfaces for creating spatial stories and gaming activities.
Proc. 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, ACM.
Bell, W. et al., Interweaving Mobile Games with Everyday Life, Proc. SIGCHI Conf. Human Factors in
Computing Systems (CHI 06), ACM Press, 2006, pp. 417426.
Biswas, A., et al. (2006). Assessment of mobile experience engine, the development toolkit for context aware
mobile applications. Proc. 2006 ACM SIGCHI international conference on Advances in computer entertainment
technology, ACM.
Bhm, S., Koolwaaij, J., and Luther, M., Share whatever you like , In Proceedings of the First International
DisCoTeC Workshop on Context-aware Adaptation Mechanisms for Pervasive and Ubiquitous Services
(CAMPUS08), June 2008
Broll, W., Lindt, I., Ohlenburg, J., Herbst, I., Braun, A.-K., and Wetzel, R.: Toward Next-Gen AR Games, IEEE
Computer Graphics & Applications, Vol. 11, No. 4, 2008, Special Issue on Mobile Graphics, IEEE Computer
Society, Piscataway, NJ, USA, 24 - 32.
Broll, W. et al., Meeting Technology Challenges of Pervasive Augmented Reality Games, Proc. ACM NetGames
06, ACM Press, 2006, article no. 28; http://doi.acm.org/10.1145/1230040.1230097.
Broll, W., Herling, J., and Blum, L.: Interactive Bits: Prototyping of Mixed Reality Applications and Interaction
Techniques through Visual Programming, in Proc. of the 3rd IEEE International Symposium on 3D User Interfaces
2008 (IEEE 3DUI 2008), Reno, Nev., USA, March 8-9, 2008, IEEE Computer Society, Piscataway, NJ, USA, 109-
115.
Capra, L., Emmerich, W., and Mascolo, C. CARISMA: Context-Aware Reflective Middleware System for Mobile
Applications. IEEE Transactions on Software Engineering 29, 10 (Oct. 2003), 929945.
Cheok et al., Capture the Flag: Mixed-Reality Social Gaming with Smart Phones, IEEE Pervasive Computing,
vol. 5, no. 2, 2006, pp. 6269.
Cheok et al., Human Pacman: A Mobile, Wide-Area Entertainment System Based on Physical, Social, and
Ubiquitous Computing, Personal Ubiquitous Comput., vol. 8, no. 2, 2004, pp. 7181.
Conan, D., Rouvoy, R. and Seinturier, L. Scalable Processing of Context Information with COSMOS. Proc. 7th IFIP
International Conference on Distributed Applications and Interoperable Systems, Springer-Verlag, Lecture Notes in
Computer Science Volume 4531, pages 210-224, Paphos, Cyprus, June 2007
Couderc P. and Bantre M., Beyond RFID: The Ubiquitous Near-Field Distributed Memory, ERCIM News, no 76,
January 2009, pp 35-36.
da Rocha, R., and Endler, M. Context Management in Heterogeneous, Evolving Ubiquitous Envrionments. IEEE
Distributed Systems Online 7, 4 (Apr. 2006), 113.
Dey, A. K., D., Abowd, G. D., & Salber, D. (2001). A conceptual framework and a toolkit for supporting the rapid
prototyping of context-aware applications. Human-Computer Interaction, 16 (2-4):97-166.
Doerner, R., Geiger, C., Haller, M., Paelke, V. Authoring Mixed Reality. A Component and Framework-Based
Approach. In First International Workshop on Entertainment Computing (IWEC 2002), May 2002.
Drozd, A., et al.: Hitchers: Designing for Cellular Positioning. In proceeding of UbiComp 2006: Ubiquitous
Computing. Lecture Notes in Computer Science Volume 4206/2006, pp. 279-296.
Fox, D., Hightower, J., Liao, L., Schulz, D., and Borriello, G. (2003). Bayesian filtering for location estimation.
IEEE Pervasive Computing, 2(3): 24-33.
Gueven, S., Feiner, S.. Authoring 3D Hypermedia for Wearable Augmented and Virtual Reality. In Proceedings of
ISWC03, pages 118126, 2003.
Henrysson, M. Billinghurst, and M. Ollila, Face to Face Collaborative AR on Mobile Phones, Proc. Intl
Symposium Mixed and Augmented Reality (ISMAR 2005), IEEE CS Press, 2005, pp. 8089.
Herbst, I., Braun, A., McCall, R., and Broll, W. 2008. TimeWarp: interactive time travel with a mobile mixed reality
game. In Proceedings of the 10th international Conference on Human Computer interaction with Mobile Devices
and Services (Amsterdam, The Netherlands, September 02 - 05, 2008). MobileHCI '08. ACM, New York, NY, 235-
244.
Hu, P., Indulska, J. and Robinson, R. An Autonomic Context Management System to Pervasive Computing. In Proc.
6th IEEE International Conference on Pervasive Computing and Communications, pages 213223, Hong Kong,
March 2008.
Hull, R., Clayton, B., and Melamed, T. (2004). Rapid Authoring of Mediascapes. Proc. UbiComp 2004.
Julien, C., and Gruio-Catalin, R. EgoSpaces: Facilitating Rapid Development of Context-Aware Mobile
Applications. IEEE Transactions on Software Engineering 32, 5 (May 2006), 281298.
Kephart, J.O. and Chess, D.M. The Vision of Autonomic Computing. IEEE Computer, 36(1), January 2003.
Koolwaaij, J., Wibbels, M., Bhm, S., and Luther, M., ContextKing: Virtual Kingdoms in Real Life, In Proceedings
of the first IEEE International Conference in Games and Virtual Worlds for Serious Applications (IEEE VS-
Games'09), March 23 - 24, 2009, Coventry, UK, March 2009.
Kcklich, Julian: Precarious Playbour: Modders and the Digital Games Industry. In Fibreculture Journal, ISSN:
1449 1443 Issue 5(2005), Australia
Lindt et al., A Report on the Crossmedia Game Epidemic Menace, ACM Computers in Entertainment (CIE), vol.
5, no. 1, 2007, article no. 8; http://doi.acm. org/10.1145/1236224.1236237
Lindt, I., Broll, W.: NetAttack First Steps Towards Pervasive Gaming, ERCIM NEWS Special Issue on Games
Technology, No. 57 (April 2004,). 49-50
MacIntyre, B., Gandy, M., Bolter, J., Dow, S. and Hannigan, B. "DART: The Designers Augmented Reality
Toolkit", In Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality, Tokyo, Japan,
2003, pp. 329-330.
Magerkurth et al., Pervasive Games: Bringing Computer Entertainment Back to the Real World, Comput.
Entertain., vol. 3, no. 3, 2005, p. 4.
McCarthy, C., International flavor comes to OpenSocial with translation app, CNET News, September 24, 2008,
available online at http://news.cnet.com/8301-17939_109-10050621-2.html
Montola, M., Exploring the Edge of the Magic Circle. Defining Pervasive Games, Proc. Proc. Of Digital
Experience: Design, Aesthetics, Practice Conf., 2005; http://users.tkk.fi/~mmontola/exploringtheedge.pdf.
Paspallis, N., Rouvoy, R., Barone, P., Papadopoulos, G.A., Eliassen, F. and Mamelli, A. A Pluggable and
Reconfigurable Architecture for a Context-aware Enabling Middleware System. In Proc. 10th International
Symposium on Distributed Objects and Applications, volume 5331 of Lecture Notes in Computer Science, pages
553570, Monterrey, Mexico, November 2008. Springer-Verlag.
Piekarski, W. and Thomas, B., ARQuake: The Outdoor Augmented Reality Gaming System, Commun. ACM, vol.
45, no. 1, 2002, pp. 3638.
Reimann, C., Paelke, V., and Stichling, D., Foot-Based Mobile Interaction with Games, Proc. ACM SIGCHI Intl
Conf. Advances Computer Entertainment Technology (ACE 04), vol. 74, ACM Press, 2004, pp. 321324.
Romn, M., Hess, C., Cerqueira, R., Ranganathan, A., Campbell, R., and Nahrstedt, K. Gaia: A Middleware
Infrastructure to Enable Active Spaces. IEEE Pervasive Computing 1, 4 (Oct. 2002), 7483.
Rouvoy, R., Conan, D. and Seinturier, L. Software Architecture Patterns for a Context Processing Middleware
Framework
IEEE Distributed Systems Online, Volume 9, Number 6, June 2008.
Salen, K. and E. Zimmermann, E., Rules of Play. Game Design Fundamentals, MIT Press, 2003.
Stapleton, C.B. and Hughes, C.E., Interactive Imagination: Tapping the Emotions through Interactive Story for
Compelling Simulations, IEEE Computer Graphics and Applications, vol. 24, no. 5, 2003, pp. 1115.
Stenton, Phil; Hull, Richard; Morgan, Jeff; Kindberg, Tim; Pollard, Stephen; Hunter, Andrew; Reid, Josephine;
Hawkes, Rych; Wainwright, Nick Pervasive Media: Delivering 'the right thing in the moment', HP Laboratories,
HPL-2008-124, Submitted to 2008 NEM submit, St Malo, France, Oct 13, 2008, available on-line at
http://www.hpl.hp.com/techreports/2008/HPL-2008-124.html
Sterritt R., Parashar M., Tianfield, H. and Unland, R. A concise introduction to autonomic computing. Advanced
Engineering Informatics, 19(3):181187, July 2005.
Tran S.M., Preda M., Prteux F., Fazekas K. Exploring MPEG-4 BIFS features for creating multimedia games
Proceedings IEEE International Conference on Multimedia and Expo(ICME'03), Baltimore, WA, Vol. 1, 2003, p.
429-432.
Wagner, D. et al., Towards Massively Multi-User Augmented Reality on Handheld Devices, Proc. 3rd Intl Conf.
Pervasive Computing, Springer, Berlin/Heidelberg, 2005, pp. 208219.
Wagner, D., Schmalstieg, D., and Billinghurst, M., Handheld AR for Collaborative Edutainment, Proc. 16th Intl
Conf. Artificial Reality and Telexistence (ICAT), Springer, Berlin/Heidelberg, 2006, pp. 8596.
Wetzel, R., Lindt, I., Waern, A., and Johnson, S. 2008. The magic lens box: simplifying the development of mixed
reality games. In Proceedings of the 3rd international Conference on Digital interactive Media in Entertainment
and Arts (Athens, Greece, September 10 - 12, 2008). DIMEA '08, vol. 349. ACM, New York, NY, 479-486.
Wittkaemper, M., Broll, W., Lindt, I. Proceedings of MRUI 2006 - First Workshop on Mixed Reality User
Interfaces - Approaches, Languages, Standardization" at IEEE Virtual Reality 2006, pp. 20-24, also in DVD Proc. of
IEEE VR 2006, Alexandria, Virginia, USA, 2006.