Networked Virtual Environments

Helmuth Trefftz
htrefftz@eafit.edu.co Eafit University - Medelln, Colombia

Introduction
Why are you here?

Helmuth Trefftz
Associate Professor at Eafit University, Medelln, Colombia. Director of Virtual Reality lab at Eafit. Involved in NVEs since 1996. Ph.D. in Computer Engineering in 2001 (Rutgers University, NJ). Ph.D. thesis: Heterogeneity in Virtual Environments.

Agenda
Introduction (15) History (10) Networking Concepts (25) Architectures (20) Shared State (45) Scalability and Performance (45) Research Topics (30) Toy application (30) Summary and Conclusions (10)

Introduction
Why Networked Virtual Environments? 1. Technology is here:
Powerful commodity 3D graphics accelerators on every PC. Internet is almost everywhere (wired or wire-less).

2. Need for applications is here:

Multiplayer games Military combat training Design teams across continents

Introduction
What is a Networked Virtual Environment?
A Networked Virtual Environment is a software system in which multiple users interact with each other in real-time, even though those users may be located around the world [Singhal/Zyda 1999]

http://www.crg.cs.nott.ac.uk/research/systems

Introduction
Collaborative Virtual Environments (CVEs) are online digital places and spaces where we can be in touch, play together and work together, even when we are, geographically speaking, worlds apart In CVEs we can share the experience of worlds beyond the physical [Churchill/Snowdon/Munro 2001]

Introduction
Key elements:
A shared sense of space (common virtual space) A shared sense of presence (avatars) A shared sense of time (real-time) A way to communicate (text, voice,) A way to interact (among users and with the virtual world) TELEPRESENCE

Introduction
Key components
Graphic engines and displays Control devices (keyboardtrackers) Processing Systems Data Network

Introduction

Common Virtual World

Current challenges
Network Bandwidth/Latency Heterogeneity Distributed Interaction (real-time) Resource Management - Scalability

History - Military
SIMNET (Simulator networking)[Miller/Thorpe95]
Distributed military environment developed for DARPA between 1983 and 1990. Aimed at:
Building high-quality, low cost simulators How to network the simulators together.

Created 11 sites with 50 - 100 simulator each.

History - SIMNET

History - Military
SIMNET Architecture
Object-event architecture Autonomous nodes (each entity is responsible for informing its state to others) Dead Reckoning

Average messages:
Slow moving vehicles: 1mess/sec. Fast moving vehicles: 3mess/sec.

History - Military
SIMNET Modules
Network Interface Other-vehicle state table Controls Own-vehicle dynamics Sound generation

History - Military
SIMNET
Multicast groups: each exercise is assigned a multicast address. Multiple exercises can co-exist

Scalability
Exercise in March 1990:
850 objects 5 sites Network: T-1

History - Military
DIS (Distributed Interactive Simulation) [IEEE93]
Started in 1989 as an effort to document the comm. protocol of SIMNET. Became an IEEE Standard 1278.

Definition of 27 PDUs (Protocol Data Unit). Examples:

Entity State: position, orientation, velocity changes. Fire Detonation Collision

Each 5 seconds: a heart-beat state PDU.

History - Military
DIS - Traffic:
96% Entity State PDUs Rest:
Fire Detonation Collision Others

History - Military
DIS - Scalability
Designed for 300-500 users

History - Military
DIS - Fully distributed and Heterogeneous
Each machine that can read/write DIS PDUS and manages the state, can participate. Heterogeneity can pose problems.
Slow machines can fall behind processing messages from fast machines. Inconsistencies!

History - Games
SGI Flight
Flight simulator demo program used from 1984 to 1992. Created in 1983. Networked version shown in SIGGRAPH 1984.

History - SGI Flight

Download source code from:

http://www.berkelium.com/OpenGL/readme.html

History - Games
Doom [idSoftware]
Released in December 1993. Initial version: no dead-reckoning, messages at frame-rate. 10s of millions of downloads. Source code was available.

History - Doom

History - Academia
NPSNET [NPSNET]
The longest continuing academic research effort in networked virtual environments [Singhal/Zyda 1999] Focus on software technology for implementing large-scale virtual environments (LSVE).

History - Academia

Courtesy of Naval Postgraduate School.

History - Academia
NPSNET
Contracted initially to handle SIMNET terrain. Afterwards, improvements on protocols:
Protocol for Lans only Bridges for Lans and Wans Implementation of IP Multicast for Wans vrtp proposal.

History
DIVE (Distributed Interactive Virtual Environment) [Hagsand96] [DIVE]
Built at the Swedish institute of Computer Science. World Database:
Distributed Fully Replicated New objects can be added/modified in a RELIABLE fashion. Distributed lock mechanism.

History - DIVE

History - DIVE

History - DIVE

History - DIVE
Videos

History
DIVE
Because of the reliable multicast implementation they use, DIVE does not scale well beyond 32 participants. DIVE 3 uses a basic communications library based on IP multicast and Scalable Reliable Multicast (SRM).

History
MR - TPP Minimal Reality Toolkit Peer Package [Shaw/Green93]
MR-TTP is based on UDP (unreliable). Lost packages are ignored. Instead of heartbeat: sends packages at the rate of the input device. Topology: Complete Graph Connection. Number of users: 4 or less.

History
MASSIVE-1
[MASSIVE][Greenhalgh/Benford9
5]

Developed at Nottinham Universitys CRG (Computer Research Group), led by Steve Bendford and Chris Greenhalgh.

History
MASSIVE-1
Multi-user distributed V.R. system Runs on Sun and SGI platforms Textual, graphical and audio client programs Interaction controlled by aura, focus and nimbus Connection oriented networking Scalability: MASSIVE usually works with up to about 10 users (from Massive-1 home page).

History - MASSIVE
Video

History
MASSIVE 2 and 3
Scalability and Distribution based on Locales Current networking via TCP client-server style (to be combined with multicast) The agent that creates an environment acts as distribution server for that environment Persistency Management of audio links.

History
Summary
A number of successful Networked VEs has been created:
Military Gaming Research

Differ greatly:
What can the users do. Technological decisions.

Networking Concepts
Latency
Amount of time to transfer a bit of data from one point to another. Latency has a direct impact on interaction inside the virtual world. The designer cannot really reduce latency. It is possible to hide it or reduce its impact.

Networking Concepts
Latency - causes:
Physical limitations: speed of electromagnetic waves in the transmission material (aprox. 8.25 msec per time zone). Delays introduced by the endpoint computers. Delays introduced by the network itself (routers).

Networking Concepts
Bandwidth
Rate at which the network can deliver data to the destination host. Examples:
Modem: 56Kbits per second. Ethernet: 10 or 100 Mbits per second. Fiber-optic: 10 Gbps or more.

Networking Concepts
Reliability
How much data is lost by the network and how that loss is handled. Causes:
Routers discard some of the messages (up to 50% in some cases). Data gets lost in the communication media.

Networking Concepts
Reliability - How to handle data that is lost?
Detect/Correct: Error-correcting codes. Detect: ACKS, NACKS.

Networking Concepts
TCP: Point-to-point connection to an application running in another machine. Data Checksum for integrity. Flow-control: to avoid flooding of messages, including acknowledgements. Keeps strict ordering and consistency of packages. Is this necessary in NVEs?

Networking Concepts
UDP: Lightweight protocol. Differences with TCP:
Connectionless transmission. Information about the state of the connection is not kept. Best-effort delivery. No guarantee of delivery or order of messages. Each packet is auto-contained.

Networking Concepts
UPD Advantages:
No overhead for connection oriented semantics. Packages can be transmitted/received immediately, no need for queues. It is possible to send information to groups of users (multicasting).

Networking Concepts
Unicast
One computer sends information to only another one computer.

Networking Concepts
Broadcast
One computer sends information to every computer in a subnet.

Networking Concepts
Multicast
Only computers listening to a specific multicast group receive the message.

Networking Concepts
Multicast: Similar to distribution of newspapers:
Subscribers explicitly request the newspaper. Duplicate copies are not sent down the same distribution tree. The publisher does not need to know the subscribers.

Networking Concepts
Mulsticast is emerging as the recommended way to build large-scale NVEs. BUT many routers do not handle multicast messages still.

Networking Concepts
TCP
Small number of users Limited data requirements Typically client-server configuration Higher data requirements Used both in client-server and peer-to-peer configurations. Small peer-to-peer Net VEs with high data requirements and time sensitive delivery. Large peer-to-peer NetVEs, be careful with routers.

UDP

IP Broadcasting IP Multicasting

Networking Concepts
Summary
Available networking infrastructure is a very important factor when designing a NetVE. Networking-related decisions have a big impact on:
Complexity of implementation Overall performance Scalability

Architectures
Client-Server Systems
logical architecture
Server

Client 1

Client 2

Client n

Architectures
Client-Server Systems
physical architecture with phone lines
Server

Phone Line

Phone Line

Phone Line

Client 1

Client 2

Client n

Architectures
Client-Server Systems
physical architecture on a LAN
Server

Client 1

Client 2

Client n

Architectures
Client-Server Systems
The Server can become a bottleneck. What are the advantages? The server can decide::
Which clients should receive a message. What protocol to use with different clients. Sub-sample messages to slow users. Keep statistics. ...

Architectures
Multiple-Server Architectures
Client 1 Client 2 Client n

Server 1

Server 2 Client 1 Client 2 Client n

Architectures
Multiple-Server Architectures
Several servers have the following advantages:
System scales better. Communication between clients attached to different servers takes longer. Key issue: how to assign clients to servers?

Architectures
Peer-to-peer

NETWORK Client n Client 1

Client 2

Architectures
Peer-to-peer
Network will be:
Broadcast One or multiple multicast groups.

In the case of multicast groups:

Area of Interest Management: assign different users to different multicast groups, based on some criteria.

Architectures
How many participants can co-exist in a virtual world? From a Network infrastructure point of view:
Infinite compute power at each node Network does not saturate Packet size: 144 bytes PDUs per second: 5 - 30

Architectures [Dawson98]
Technology Speed (Kbps) Min # players Max # players

Modem DSL T-1 10BT

56 1500 1500 10,000

1 39 39 263

6 163 163 1085

100BT

100,000

2630

10851

Shared State
Main idea in a NVE: Provide users with the illusion that every participant is seeing the same things and interacting with each other. Things in the NVE change. Without dynamic shared state, the illusion of sharing space and time breaks.

Shared State
Definition of the problem - The consistency-throughput tradeoff
It is impossible to allow dynamic shared state to change frequently and guarantee that all host simultaneously access identical versions of that state. [Singha/Zyda
1999]

Shared State
Absolute consistency:
Wait until everybody acknowledges before moving to a new position. Considering latency and available bandwidth, this may imply slowing down too much.

Frequent updates:
Users send the updates and move on. But: some users will receive the updates quickly and others wont. Inconsistent views!

Shared State
Centralized Repositories - File Repository
For pedagogical purpose: One file per entity Each update:
Open File Read/Updat Close File
User 1
User 2

User n

Shared State
Centralized Repositories - File Repository
Advantage:
absolute consistency. Avoids concurrency problems. Locks on objects easily implemented (when are locks necessary?)

Problems:
SLOW!

Shared State
Centralized Repositories - Repository in Server Memory
Entry in memory per entity. Users need to acquire a lock.
User 1 User 2

User n

Shared State
Centralized Repositories - Repository in Server Memory Advantages
Fast!

Disadvantages
No persistency. What happens if server crashes?

Shared State
Distributed Repositories - Virtual Repository
Different clients manage different parts of the world. Clients have a local cache of used objects.

Shared State
Distributed Repositories - Virtual Repository Advantages:
No bottleneck

Disadvantages:
Complex to program.

Shared State
Frequent State Regeneration Is absolute consistency always necessary? When is it NOT necessary?

Shared State
Frequent State Regeneration
For instance in a group flight simulator program: A temporal inaccuracy in the position of another plane is not too serious. Example: if updates are sent every 40 milliseconds, one lost package is almost imperceptible.

Shared State
Frequent State Regeneration
Producer
Broadcast location of local objects Either when they move or at a fixed rate.

Consumers (other participants)

Draw the scene using the locations in the local cache. Update the local cache with the remote events.

Shared State
Frequent State Regeneration
Producers and consumers are decoupled.

All interactions can be decoupled?

Independent movements: YES. Tight-collaborative tasks: NO.
Fast cycles Slow cycles

Shared State
Situation 1:
A slow computer controls a plane moving in a straight line. A participant in a fast computer perceives a jumpy movement.

Situation 2:
A fast computer floods a slow computer with messages of a car moving in a straight line.

Shared State
Issue 1 (slow updates - Fast computer sees jumpiness)

Issue 2 (fast updates - Slow computer overwhelmed)

Shared State
Solution to alleviate both issues: DEAD RECKONING. Instead of sending updates at frame rate, Send parameters that describe the trajectory of the object (example: initial position and velocity) Each participant displays the trajectory at its own rate.

Shared State
Advantages:
Accommodates heterogeneity. Bandwidth usage is reduced.

Cost:
More cycles to compute trajectory at each node. Need to re-synchronize.

Shared State
Dead Reckoning - Two parts:
Prediction
Predict where the object is based on the received parameters.

Convergence
Once an actual position is received: how to move the object from the predicted to the actual position.

Shared State
Prediction Algorithms
Derivative Polynomials: Order one:

pos1 pos0 vel * t

Order two:

1 pos1 pos0 vel * t acc * t 2 2

Part of DIS protocol.

Shared State
Other prediction algorithms:
Circles Spirals

Planes seem to follow many such curves.

Shared State
Other strategy:
Send an update when the actual position is very different from the predicted position. Advantage: the error in the system is kept under certain threshold. See paper by C. Faistnauer in IEEE VR 2000.

Shared State
Convergence Algorithms
Predicted Position
Convergence Position

Current Position

Last actual position

Shared State
Convergence Algorithms Snap: Just move the object to the most recent actual position. Linear: Linearly interpolate to a point in the new predicted path (convergence point). Cubic Spline: Create a path in the form of a cubic spline from the current position to the convergence point.

Scalability and Performance

Improve the size and performance of a Net-VE by reducing network bandwidth and processor resources. Less bandwidth requirements means more users in the network. Less processor requirements means free cycles for other purposes (better graphics,) and more heterogeneous participants.

Scalability and Performance

Required resources are a function of:
Number of transmitted messages Average number of destination hosts per messsage Bandwidth required per message Timeliness (minimal delays) Processor cycles needed to process each message.

Scalability and Performance

A reduction in any of these items is a gain. BUT usually a reduction in one means a penalty in another. Example:
Dead Reckoning reduces required bandwidth but more processor cycles are required.

Scalability and Performance

Compression & Aggregation Compression:
Aims at reducing bandwidth utilization by reducing the size of the messages. Internal: encode in a more efficient manner. External: Avoid retransmitting information that is identical to previous packages.

Scalability and Performance

Compression
Example: Send Snapshots at periodic rates. Send updates to the snapshots. Snapshots sent over reliable protocol. Updates sent over unreliable protocol.

Scalability and Performance

Packet Aggregation
Aims at reducing the number of packets by merging several packets together. The saving comes from the headers (25 bytes for UDP and 40 bytes for TCP). Which packets can be merged?
Packages from all entities managed by the node (at the client). Packages from several clients (at the host).

Scalability and Performance

Which packages to merge?
Based on a timeout (send a message every timeout units of time). Based on quorum (send a message only when the quorum has been reached). Combination of the previous two.

Scalability and Performance

Aggregation Servers
Define servers that aggregate messages for sets of entities having common characteristics:
Virtual World Location. Entity type.

Scalability and Performance

General model:
Each node has an Aura, or area of influence. Each node has a Nimbus, or a type of entities from which to receive data.
Aura
Source Dest1
Nimbus 1

Dest2
Nimbus 2

Scalability and Performance

An event is only send from the source to the destination if the sources aura intersects the destinations nimbus. In Massive-1 unicast, peer-to-peer, unreliable protocols are used to send events from the sender to the destination. This does not scale very well.

Scalability and Performance

Multicast
Key question: how to map groups of users to multicast groups. Approaches:
Group-per-Entity Group-per-Region

Scalability and Performance

Group-per-Entity [Abrams/Watsen/Zyda98]. There is a multicast group per entity. Each node subscribes to multicast groups of entities in its nimbus. A directory server is necessary to inform nodes about the position and multicast groups of other entities. Problem: The number of multicast groups grows rapidly.

Scalability and Performance

Group-per-region
The virtual world is partitioned into cells. Each cell is assigned a multicast group. Users subscribe/unsubscribe from multicast groups as they travel. As a user approaches a region, it needs to subscribe to the neighbor cells. Hexagonal cells are an advantage.

Scalability and Performance

Level of Detail perception Objects that are very far away will:
Appear smaller Probably not be the focus of attention.

Therefore:
Can be updated less frequently.

Scalability and Performance

Need for multiple channels. Different channels have different resolution (update frequencies, spacial granularity, etc) Who handles the channels?
A server Each client

Research Topics
Multimedia
Incorporating audio/video live streams to NVEs Questions:
What to display Where to display it A single rate or multiple rates?

Research Topics - Multimedia

FreeWalk [Nakanishi96].
Mapped a picture of the user in the front face of a pyramid (avatar). Users can move freely and establish voice conversations. The volume depends on the distance and mutual orientation. A community server informs users about positions and orientation of other users. Actual voice messages and pictures are exchanged among users.

Research Topics
FreeWalk

Research Topics - Multimedia

Awareness-Driven Video QOS [Reynard 98]
Three different QOS levels are established:
Porthole: 1 update every 5 minutes. At the top of the virtual office. Glance: 1 frame/sec. At the front of the office. Communications: 20 frames/sec. On top of users avatar.

Research Topics - Multimedia

Awareness-Driven Video QOS

Research Topics - Architectures

Hierarchy of Servers [Funkhouser95]
Compared Static Allocation of clients to servers vs. Dynamic Allocation (a server per room). The number of server-to-server messages was reduced.

Research Topics - Architectures

Courtesy of T. Funkhouser http://www.cs.princeton.edu/funk/ring.html

Research Topics - Architectures

Dynamic partitioning of space [Restrepo03] Partition the space dynamically as the users move in the world. Assign one partition to each server. If a space becomes to crowded, split in a quadtree fashion. Load among servers was better balanced.

Research Topics - Architectures

Courtesy if the Eafit University.

 How to accommodate heterogeneous nodes in a NetVE [Trefftz03] The system handles multiple variables:
Display Rate Frequency of remote updates Frequency of video updates

Research Topics Heterogeneity

The user can specify her preferences. The System Administrator can define minimum levels The system optimizes the function and defines fidelity levels for each client.

Research Topics - Heterogeneity

Proof-of-concept application:

Research Topics - Heterogeneity

The Switchboard architecture

Research Topics - Collaborative AR

Two or more users collaborating in a reality augmented with virtual objects. See Studio3D by Vienna University. What happens if the users are in different physical locations?

Research Topics
What are the big players doing?
Mike Zyda (NPS):
Very successful army game. Agents in VEs. Defense-Entertainment cooperation for Training.

Greenhalgh/Benford (Nottingham)
Heterogeneity Wireless

Steve Feiner
IEEE VR 2003: Merge of VR and wireless technologies.

Research Topics
Americas Army

From Americas Armys home page.

Commercial Systems
ActiveWorlds
Commercial system. Free sessions for guests. Used for:
Education Social interaction of groups

Provide avatars and connected virtual locales Communication: chat tool.

Toy Application (cubes)

Demonstration Will be downloadable from: http://arcadia.eafit.edu.co/CGIMtutorial/

Toy Application
Packages - Client Application

Toy Application
Packages - Server Application
Jus the Server Package.

Toy Application - server

Toy Application - send

Toy Application - receive

Toy Application - replicate

Toy Application - Documentation

JavaDoc
Documentation of classes, methods and atributes. Created by Java.

UML
Poseidon UML

SourceCode
JBuilder Personal Edition project.

Toy Application
Changes you can do:
Use UDP instead of TCP - Compare performance. Add audio-video streaming (JMF) Make it peer-to-peer - Compare performance. Try the techniques for scalability defined in the tutorial. Use a VRML loader to import prettier geometry.

Toy Application
Test your own ideas!

Conferences
IEEE Virtual Reality (formerly IEEE VRAIS) ACM VRST (Virtual Reality Software and Technology). ACM CVE (Collaborative Virtual Environments). (every two years). ACM CSCW (Computer Supported Collaborative Work). IASTED CGIM.

Journals
PRESENCE. MIT Press. In General networking or HumanComputer Interaction Journals have more and more articles on NVEs.

Summary and Conclusions

New field. Technology is here. The need & interest are growing. Many areas to be explored. Any idea for your own research? I will be happy to discuss. Enjoy!

Thank you!

Bibliography - BOOKS
[Singhal/Zyda 1999] Singhal, S. and Zyda, M. Networked Virtual Environments: Design and Implementation. AddisonWesley.1999. [Churchill/Snowdon/Munro 2001]Churchill, E., Snowdon, D. and Munro, A. Collaborative Virtual Enironments. Digital Places and Spaces for Interaction. Springer Verlag. 2001.

Bibliography - Papers and Articles

[Abrams/Wasten/Zyda98] Abrams, H., Waksen, K. and Zyda M. Three tiered interest management for largescale virtual environments. In Proceedings of VRST 1998. Taipei, Taiwan, November 1998. [Dawson98] Dawson, F. XDSL market blooming. Interactive Week 5(39):28-29. October, 1998. [Funkhouser96] Funkhouser, T. Network Topologies for Scalable Multi-User Environments. In Proceedings of IEEE

VRAIS. 1996.

Bibliography - Papers and Articles

[Greenhalgh/Benford95] Greenhalgh, C., Benford, S. Massive: a Virtual Reality System for Teleconferencing. ACM Transaction on Computer Human Interfaces. 2(3):239-261.
[Hagsand96] Hagsand, O. Interactive multiuser VEs in the DIVE system. IEEE Multimedia 3(1):30-39, 1996. [IEEE93] IEEE standard for information technology / protocols for distributed simulation applications. IEEE Standard 1278-1993. IEEE Computer Society, 1993.
[Macedonia95] Macedonia, M., Zyda, M., Pratt, D. and Barham, P. Exploiting reality with multicast groups: A network architecture for large-scale virtual environments. In Proceedings of IEEE VRAIS. 1995.

Bibliography - Papers and Articles

[Miller/Thorpe95] Miler, D., Thorpe J.A. SIMNET: The advent of Simulator Networking. In Proceedings of the IEEE 83(8):1114-1123, August 1995. [Nakanishi96] Nakanishi, H., Yoshida, C., Nishimura, T. and Ishida, T.

FreeWalk: Supporting Casual Meetings in a Network. In Proceedings of ACM CSCW. 1996. [Restrepo03] Restrepo A., Montoya, A. and Trefftz H. Dynamic Server Allocation in Virtual Environments using QuadTrees for Space Partitioning. In Proceedings of IASTED Computer Science and Technology, Cancun, Mexico, 2003. [Reynard98] Reynard, G., Benford, S., Greenhalgh, C. & Heath, C. Awareness Driven Video Quality of Service in Collaborative Virtual Environments. In Proceedings of ACM CHI 98.

Bibliography - Papers and Articles

[Shaw/Green93] Shaw, C. Green, M. The MR toolkit peers package and experiment. In Proceedings of the 1993 IEEE Virtual RealityAnnual International Symposium, 463/469. Seattle, September 1993. [Trefftz03] Trefftz, H., Marsic, I. and Zyda M. Handling Heterogeneity in Networked Virtual Environments. In Presence 12(1) 37. February 2003.

Bibliography - Web Pages

[DIVE] DIVE home page http://www.sics.se/dce/dive/ [idSoftware] idSoftware home page http://www.idsoftware.com/ [MASSIVE-3] MASSIVE home page http://www.crg.cs.nott.ac.uk/research/systems/MASSIVE -3/ [NPSNET] NPSNET home page http://www.npsnet.org/~npsnet/ Toy application: http://arcadia.eafit.edu.co/CGIMtutorial/