Simulation Modelling Practice and Theory 94 (2019) 237–249

Contents lists available at ScienceDirect

Simulation Modelling Practice and Theory

journal homepage: www.elsevier.com/locate/simpat

AR-PED: A framework of augmented reality enabled pedestrian-in-

T
the-loop simulation
Daniel Perez, Mahmud Hasan, Yuzhong Shen, Hong Yang
⁎

Department of Modeling, Simulation & Visualization Engineering, Old Dominion University, 4700 Elkhorn Ave, Norfolk, VA 23529, USA

ARTICLE INFO ABSTRACT

Keywords: Pedestrian simulators are in high demand for many research and industry applications, such as
Pedestrian simulation evaluating pedestrian safety and testing of autonomous vehicles. A recent trend on the devel-
Virtual reality opment of pedestrian simulators is to use virtual reality (VR) head-mounted display (HMD) de-
Augmented reality vices, since they offer a highly immersive and realistic view of the virtual environment (VE).
Virtual environment
However, existing pedestrian simulators that make use of this technology have a set of limitations
Head-mounted display
Traffic simulation
that make them unsuitable for the simulation of a large virtual scenario. Specifically, the
boundaries of the virtual scene set by these simulators encompass a relatively small area (e.g., a
single crosswalk). In light of these issues, this paper aims to develop an augmented reality (AR)
enabled pedestrian-in-the-loop simulation (AR-PED) framework that enables the simulation of
pedestrians in a large virtual scenario (e.g., a city) with an actual scale. The proposed AR-PED
simulation framework is empowered by the latest advances in cloud computing and AR to allow
multiple users to access the simulation at the same time. A client-server architecture allows users
to act as pedestrians, drive a car or simply visualize the scene simultaneously. Additionally, other
vehicles can be artificially generated by a microscopic traffic simulation (MTS) package that acts
as the server. A prototype of the proposed framework is implemented and demonstrated by si-
mulating a model of a city. The promising capabilities and performance of the AR-PED frame-
work are attested with the deployable experiments and potential improvements over the present
study to further excel the current framework are also discussed.

1. Introduction

Pedestrian simulators are in great demand in both research and industry fields for various reasons and applications. One of the
most important application is the study of pedestrian safety in a variety of scenarios. Among these scenarios, some of the popular
cases are the determination of the dangers of texting while crossing a road [1–3] and the teaching of safety to young pedestrians
[3–6]. Other than these conventional applications, the need for advanced pedestrian simulators pertaining to emerging transportation
innovations (e.g., connected and autonomous vehicles) is also demanding [7,8]. In general, the term “pedestrian simulator” may refer
to either the simulation of virtual pedestrians and their behavior in various traffic scenarios or the development and use of a virtual
simulation in which users can act as pedestrians to perform traffic experiments. In this paper, we refer strictly to the second scope
when we mention this term.
With the recent advances in virtual reality (VR), head mounted display (HMD) devices are a very helpful option when developing

⁎
Corresponding author.
E-mail addresses: dpere013@odu.edu (D. Perez), mhasa002@odu.edu (M. Hasan), yshen@odu.edu (Y. Shen), hyang@odu.edu,
yanghong108@gmail.com (H. Yang).

https://doi.org/10.1016/j.simpat.2019.03.005
Received 6 August 2018; Received in revised form 13 March 2019; Accepted 14 March 2019
Available online 15 March 2019
1569-190X/ © 2019 Elsevier B.V. All rights reserved.
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

pedestrian simulators. There are multiple implementations of pedestrian simulators that make use of HMD devices [7,9–14].
However, existing simulators are only capable of simulating a very small part of a city such as a midblock crosswalk or an inter-
section. This is due to the current limitation of HMD devices regarding the simulation area. For example, the most popular HMDs in
the market right now are the Oculus Rift and HTC Vive headsets, whereas both have limitations on the play area [15,16]. Another
challenge for pedestrian simulators is the realistic implementation of the vehicles’ behavior. Programming vehicles at a single in-
tersection or street crossing is a relatively simple task, but the implementation of traffic circulation in a large area is significantly
more challenging. Traffic in a city needs to handle many variables such as interactions between vehicles, stop signs, traffic lights, etc.
A possible solution to this issue is to make use of microscopic traffic simulation (MTS) packages for the running of traffic. However,
there was no successful practice in integrating pedestrian simulators with MTS.
In the past, we have presented a framework for M3 visualization of MTS models [17,18] and another work that deals with the
deployment of user-controlled vehicles in the framework [18]. In this paper, we expand over our previous work to develop a user-
controlled pedestrian simulation framework, offering a virtual environment in which user-controlled pedestrians can coexist with
user-controlled vehicles for realistically simulating pedestrian-driver interactions. The developed pedestrian simulator is named AR-
PED, and it can be used for multiple purposes, e.g., studying pedestrians’ behavior in the context of autonomous vehicles, in-
vestigating pedestrian-driver behavior at intersections, etc. The seamless connection between the AR-PED simulator and the MTS
server makes use of the latest advances in cloud networking. Specifically, a TCP/IP connection is established to synchronize the
vehicle and pedestrian data between the server (MTS software) and multiple clients (simulators) simultaneously. To allow the pe-
destrians the maximum freedom of movement, AR-PED allows pedestrian users to control their avatars with a wireless augmented
reality (AR) headset, namely Microsoft HoloLens. The headset is able to display a hologram of the target environment (e.g., a city, an
intersection, etc.) in a 1:1 scale to the user, while sending the pedestrian data to the server. To summarize, the major contributions of
this paper are:

• The development of a pedestrian simulator, AR-PED, with a wireless HMD device (Microsoft HoloLens) that allows the users total
freedom of movement without any physical boundary restrictions.
• The first implementation of adding user-controlled pedestrian to an MTS model that simulates traffic circulation in a large
network.
• A framework where user-controlled vehicles and pedestrians can coexist with artificial vehicles generated by an MTS model. In the
future, connected and autonomous vehicles can be deployed in this framework for testing pedestrian responses in the new traffic
environment.

The rest of the paper is organized as follows. Section 2 performs a review of the existing literature on pedestrian simulators. Then,
Section 3 describes our proposed methodology for the development of our AR-PED simulator, and the results of our developed
prototype are shown in Section 4. Finally, Section 5 draws the conclusions from the development of our project and sets directions for
the future work.

2. Literature review

Many traffic simulation packages offer modules for pedestrian simulation in which crowds of artificial pedestrian agents are
simulated in their virtual environment. While these simulations of artificial pedestrians are useful in traffic simulations, they lack the
realism of pedestrian behavior. Simulating a person is a significantly challenging task and the realism of artificial pedestrian si-
mulators is not enough in areas such as testing of autonomous vehicles, in which real behavioral responses are needed to asses and
improve their system.
Another mainstream of pedestrian simulation is to use human-controlled pedestrian simulators. These pedestrian simulators are
popular in the literature for diverse applications such as determining the effects of text messaging on pedestrians [1–3], improving
child pedestrian safety [3–6], cognitive monitoring and evaluation [12], testing of autonomous vehicles [7,8] and improving Stated
Preference (SP) experiments [14,19]. Different combinations of virtual reality devices are used in each work. The following para-
graphs offer a thoroughly examination of existing practices.
The simplest approach for VR pedestrian simulators consists of using desktop computers in which users typically move down-
scaled avatars in a 3D virtual world. This approach is frequently used in studies focused on working with child pedestrians because it
is an inexpensive method that can be easily installed in most schools and child centers. For instance, Thomson et al. [5] developed a
VR pedestrian simulator that was displayed in a single desktop monitor. In this study, the users observed the actions in a third-person
point of view from an elevated semi-aerial position so that the intersections could be fully displayed on the screen. The pedestrians
were able to cross the street with the press of a button. While this approach can be useful to understand where to cross a road, it does
not allow the users to feel like they are actually crossing the street. Alternatively, some approaches developed a first-person point of
view and more immersive pedestrian simulator by using multiple screens that surround the users, offering a peripheral view of the
scene. For example, McComas et al. [4] developed a simulator for teaching child pedestrian safety in which three 21″ screens
displayed a virtual environment consisting of different types of intersections. Additionally, they utilized a head tracking device to
determine whether the children looked left-right-left before crossing. In their study, they did not mention how the children moved in
the virtual environment, so it is assumed that they used a regular keyboard as the main input method. Using traditional input
methods such as the keyboard or the mouse might lack realism and make the users feel like they are not actually moving in the
simulated environment. To fix this, Schewbel et al. [6] proposed a three-screen desktop pedestrian simulator in which the users were

238
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

on top of a plywood structure. Participants observed the traffic in the three screens from a first-person perspective. When they
considered safe to cross the intersection, they stepped down of the structure, which triggered a pressure plate connected to the virtual
simulator. Once that signal was triggered, the view changed to a third-person perspective so that the participants could observe their
avatar crossing the street. This study offered a more realistic input method to cross an intersection. However, users can only perform
one action and they did not have the freedom of moving freely through the virtual environment.
Multi-screen pedestrian simulators offer the users a peripheral first-person view of the virtual environment, but they do not offer an
immersive experience where the users may feel like they are in a real traffic scenario. To solve this, a few studies developed VR pedestrian
simulators using a Cave Automatic Virtual Environment (CAVE) [20]. The CAVE consists of a set of three big screens which act as walls
where images are projected, giving the users the feeling of being surrounded by a virtual world. The most significant limitation of this
virtual reality device for pedestrian simulator is that it does not allow the users to move beyond the walls. The user may move in the
limited area between the screens, but this space is typically not big enough to encompass a traffic scenario. For instance, Rahimian et al.
[2] developed a VR pedestrian simulator using a CAVE in which the pedestrians could move in an area of 4.33 × 3.06 m. While this area
was big enough to conduct their study, the users were limited to move in a single crossing inside of a significantly limited space. To solve
this issue Banducci et al. [1] placed a manual treadmill that was linked to the virtual environment so that when the users walked on it, the
virtual world would move accordingly. While this upgrade allowed users to cross the street in a simulated environment, the pedestrians
had the limitation of only moving in a single direction. A slightly different approach was followed by Dommes et al. [21], where they built
a street-crossing simulator using a ten-screen image projection system that allowed the participants to walk in a two-lane intersection. In
this case, users could freely move in an area that was 5.7 m wide and 7 m long. However, the space was still limited to a single intersection
and many scenarios such as simulating a crosswalk spanning a two-way four-lane street are not feasible.
Lately, head-mounted display systems are a good VR alternative to the devices mentioned above. They provide an 360° view of the
virtual environment that offers a highly immersive experience of the simulation. Additionally, it has been demonstrated that HMD
devices provide more realistic VR experiences due to the capability of making users forget about their surroundings in the real
environment [22]. Another advantage with respect to CAVE devices is their low cost and portability. There exist multiple studies that
used this kind of devices to develop a pedestrian simulator. An early attempt on using HMD devices in pedestrian simulators is an
investigation of road crossing realized in 2002 by Simpson et al. [9], in which they used a HMD with a resolution of 640 × 480 pixels
that produced synoptic images with an horizontal and vertical field of view of 48° and 60° respectively. Although their HMD device
had immersion problems due to the early state of the technology (low resolution, synoptic images, small field of view, etc.), they
determined that using this type of devices was a promising method for pedestrian simulators. The latest advances in head-mounted
displays have produced high-resolution devices with stereoscopic images and greater fields of view that produce highly-immersive
VR environments. In an attempt to compare child pedestrian behavior with their parents’ expectations, Morrongiello and Corbett [10]
developed a pedestrian simulator using a Virtual Research systems HMD display with stereoscopic images and a resolution of
1280 × 1024 pixels that allowed the participants to move in a room of 8 m × 5 m. The virtual scene consisted of a two-lane street
with sidewalks. The same team used this device in a different study where they aimed to understand how injury risk arises [11].
Shuchisnigdha et al. [13] developed a pedestrian simulator using the HTC Vive and a virtual environment developed with Unity.
Their experiment simulated a single crosswalk and users could freely move in an area of 4 by 7 m approximately. Other studies have
come up with a similar setup using the Oculus Rift headset [7,12,14], which allows the users to freely move in an area of maximally
about 4 m by 4 m. While these devices offer a highly realistic environment, the boundaries of the scenario are still relatively small,
which limit researchers to do their studies at a small intersection.
The development of a virtual environment that encompasses a significantly big area (e.g., a corridor with multiple intersections or
a city) for pedestrian simulation has not been fully addressed. Hartmann et al. [7,8] have mentioned in their studies the need of
simulating a whole city with real pedestrians using HMD devices for the testing of autonomous vehicles. In their study [8], they
discussed the possibility of using an augmented reality (AR) display, the Microsoft HoloLens, to simulate pedestrian behavior; and
they provide a simple prototype in which the users wearing the headset can visualize the hologram of a car in the real world. To the
best of our knowledge, there does not exist a virtual/augmented reality framework that can enable simulation with both user-
controlled vehicles and pedestrians coexisting with other artificial vehicles in a large area.
To summarize the reviewed literature, Table 1 shows a comparison of the most relevant pedestrian simulators. The studies are
ordered by the level of immersion (from low to high). The table shows some useful information for the development of our study. It is
worth to note that most recent simulators that use HMDs as their VR technology develop their virtual environment with Unity.
Besides, there is no existing pedestrian simulator without the limits of space constraints. To the best of the authors’ knowledge, the
present study is the first to offer an AR-PED pedestrian simulator in which the users can freely move without any preset boundaries.

3. Methodology

This study proposes a novel simulation framework, AR-PED, where user-controlled pedestrians coexist with user-controlled ve-
hicles as well as with artificial vehicles generated by an MTS server. The framework is empowered by the latest advances in computer
hardware, software and networking. The following sections detail the system design and key components of the proposed framework.

3.1. AR-PED system architecture

The proposed AR-PED framework employs a distributed architecture based on the client-server model where the MTS is the server
and the clients can access the simulation to control a pedestrian, a vehicle or visualize the scene. The overall structure of the proposed

239
 D. Perez, et al.

Table 1
Comparison of existing pedestrian simulators.
Study Scope Technology Level of Input methods Space Boundaries Participants
immersion

[5] Training child pedestrians 1-screen PC Low Designated Button Users do not move 129 children of ages 7, 9 and 11
[4] Training child pedestrians 3-screen PC Low Not mentioned Users do not move 95 children from grades 4–6
[6] Training child pedestrians 3-screen PC, Plywood Low Plywood structure On/off the structure 102 children and 74 adults
structure
[2] Studying the impact of texting CAVE Medium User's movement 4.33 m × 3.06 m 48 undergraduate students
[1] Studying the impact of texting CAVE, Manual treadmill Medium Manual Treadmill No limit walking forward, but unable to 37 university students
move sideways
[21] Studying age-related difficulties in crossing 10-screen image projector Medium User's movement 5.7 m × 7 m 58 healthy aged participants

240
a street system
[9] Studying road crossing behavior Early VR HMD Medium-high User's movement 6 m (forward) 24 participants between 5 and 30
years old
[10] Comparing child pedestrian behaviors with Virtual Research systems High User's movement 8 m×5 m 139 children between 7 and 12
their parents’ expectations HMD years old
[11] Examining how risk of injury arises in child Virtual Research systems High User's movement 8m×5 m 95 children between 7 and 11 years
pedestrians HMD old
[13] Pedestrian safety research HTC Vive, Unity High User's movement 4m×7 m 26 university students
[12] Cognitive monitoring and evaluation Oculus Rift DK2, Unity, Leap High Robotic arm (Myo), Hands Users move according to arm input No participants
Motion, Myo (Leap Motion)
[14] Evaluation of VR State Preference Oculus Rift, Unity High User's movement 4.5 m × 2 m 42 university students
experiments
Simulation Modelling Practice and Theory 94 (2019) 237–249
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 1. AR-PED system architecture.

framework is shown in Fig. 1. The server hosts the back-end MTS engine, which can be any commercial, open-source or custom-made
MTS platform. The only requirement for the selected MTS engine is that it needs to have a customizable Application Programming
Interface (API) so that the communication code can be implemented on it. The MTS engine generates the artificial vehicle data, traffic
signal information, etc. Besides, the server has the function of synchronizing all the user-controlled data from pedestrian and vehicles
outside the MTS engine. The server processes all the information and sends it back to each client so that they can visualize it. The
clients present a realistic rendering of the 3D environment that contains the road network used by the MTS server and a graphical
user interface (GUI) for users to interact with the visualization. Multiple clients can connect to the server simultaneously and perform
different actions. Based on their purpose, there are three different types of clients: visualizers, drivers and pedestrians. The visualizer
clients present the first type and their details were presented in [17]. The clients allow a visualization in which multiple users can see
the same traffic simulation from multiple perspectives (bird's view, pedestrian view, driving view and car following view). It is
important to note that the pedestrian view differs significantly from the pedestrian client, since in this case the pedestrian data is not
exchanged with the server and, consequently, they will not be visualized in the rest of the clients. The visualization can be ex-
perienced from different devices (also referred as modes) such as desktop, laptops, mobile, and VR and AR devices. The second type
of clients are the driver clients. Each of these clients has the ability of selecting one of the cars generated by the MTS server and
controlling it using different IO devices. The development and deployment of this type of client has been presented in [18]. The third
type corresponds to the pedestrian clients, which are the focus of this paper. Pedestrian clients are controlled through the Microsoft
HoloLens headset, which provides them with a view in AR of the virtual city. The pedestrian data is sent from these clients to the
server and then, after it has been processed, the server is responsible for sending it back to the rest of the clients for visualization
purposes.
The server processes all the data from the MTS engine and the driver and pedestrian clients. This data is then sent back to the
clients to ensure a uniform visualization to all the users. Visualization clients do not send any data to the server, since their purpose is
solely to visualize the traffic simulation scene. The communication between server and clients is obtained by incorporating net-
working communications using the TCP/IP protocol. The connection between the server and each client is handled independently,
and it is maintained until the client or the server decide to close it. In this case, the server is always running, and the clients have the
ability to disconnect from the server, which closes their specific connection without affecting the communication with the other
clients.

241
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 2. Microsoft HoloLens with labeled parts.

3.2. Pedestrian clients

Pedestrian data is collected from the pedestrian clients and sent to the server, where it is processed and then sent back to all the
clients so that they can visualize the pedestrians. Each pedestrian client can control one pedestrian, which will be visualized by the
rest of the clients in the form of a virtual avatar with the proper dimensions. Once the connection with the server is established,
pedestrians can move freely through the virtual city, and their corresponding avatars will be updated in the rest of the clients.
Pedestrian clients are controlled using Microsoft HoloLens, which is the most popular AR headset in the market (Fig. 2). We
believe that this device is ideal for developing a pedestrian simulator for multiple reasons. First, since the users are able to see the real
world merged with the hologram of the virtual city, dizziness and other sickness induced by virtual reality [23] are not a problem.
Although the experience of the user can be lessened by obstructions produced by objects from the real world, the device can be
comfortably used in big and clear spaces such as basketball courts or parking lots, which are accessible by everyone. However, the
most important advantage that this headset presents is the absence of any kind of cables. Most popular VR devices like HTC Vive and
Oculus Rift need to be connected to a computer to work, which sets a limitation in the scene boundaries. Even in the case of using
significantly long cables, the sensors in these devices need to be placed at a distance [15,16], giving a maximum simulation area of
about 4 m × 4 m. In contrast, Microsoft HoloLens does not need a separate computer to work and it includes the appropriate sensors
and cameras inside the headsets, which guarantees the users the ability of moving through a virtual scenario without boundaries.

4. Prototype development and demo

A prototype AR-PED has been developed following the proposed framework. The virtual city where the simulation takes place was
built upon a model covering an area of 2.0 miles × 2.0 miles, with approximately 1000 buildings, road network, and other facilities
(e.g., bus stops, trees, signs, etc.) that are typically present in an actual city. It should be emphasized that users can freely replace it by
any city models of their interest. The AR-PED pedestrian simulator was specifically developed for the Microsoft HoloLens headset,
whereas the visualization can be experienced using different types of devices (desktop, laptop, VR, AR, etc.) as described in [17,18].
For the prototype, the MTS server currently runs on a laptop with the following configuration: (a) Intel® Core™ i7-4800MQ CPU @
2.70GHz; (b) 16.00GB RAM; and (c) Windows 7 Professional 64-bit Operating System. The following sections provide the detailed
description of the prototype development and demonstration.

4.1. Selection of MTS server

Quadstone Paramics (Parallel Microscopic Traffic Simulator) [24] has been utilized in this study as the MTS server. Specifically,
its Modeler is used in this study. It has built-in car-following model and lane-changing model that imitate the psycho-physical
feedback and gap acceptance behavior of various vehicles stochastically. Apart from the fundamental modeling and analysis features,
the most important attribute for us to consider Paramics in this study is its customizable API. Since the default traffic models and
functions may not always satisfy users’ needs, its API allows modelers to access and develop customized components. More im-
portantly, it allows the simulation model to interface with many external programs and toolkits easily. This is critical for the present
study because the simulation model, as a server, needs to communicate and exchange data with various visualization platforms in real
time.
It is worth to notice that other MTS packages can be easily adapted to work with our integrated framework. In the past, we have
implemented PTV Vissim as the MTS server and we have found that it is also a good option to consider [17,18]. Although the
prototype that we show uses Paramics as the MTS software, substituting it by Vissim or other similar traffic simulators would be a
significantly easy task. In addition, the developed AR-PED does not require use any pedestrian simulation modules in these traffic
simulators.

242
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 3. VE Development: (a) Network in Paramics, (b) 2D city model; and (c) 3D city model.

4.2. Virtual environment development

The game engine Unity has been utilized to create the virtual environment (VE) in the clients’ end. A set of advanced rendering
techniques are utilized in the VE, e.g., shadow maps and environment maps. The target road network in the city model matches the
node-link road network in the Paramics model. It is imperative that the geometry of the road network in both models aligns ap-
propriately. Fig. 3 shows the development of the virtual environment. The development process is reversible, depending on the initial
model at disposal. Besides, while we used Unity to develop our demo framework, it is worth to notice that any other tool for VE
development (i.e. Maya, Unreal Engine, SketchUp, etc.) can be used.

4.2.1. GUI in pedestrian clients

A critical factor in the pedestrian clients’ implementation on the HoloLens headset was the adaptation of the graphical user
interface to an AR device. In the desktop applications with traditional displays, user interfaces are commonly implemented in the
screen space, and a user UI is usually displayed at a fixed location on a screen. However, screen spaces no longer exist in AR
applications as there are no input devices to interact with such spaces. Instead, user interfaces are implemented directly in the world
space, and the users can interact with them using their voice or pointing with their finger. In our prototype, we developed a GUI for
AR in which users can press buttons and toggles using the “gaze-and-commit” interaction, which is a combination of gazing at an
element and then doing an air tap gesture to select it. Specifically, the GUI includes two buttons: one to select the type of avatar that
will be visualized for the user and another one to connect to the server. Once the users press the “Connect to Server” button, they will
not be able to change their avatar and the GUI will disappear from their environment, giving the users the freedom to walk through
the virtual city without obstructions in their view. Fig. 4 shows the view from the pedestrian client once the system is initialized and
the GUI panel that allows users to select their avatar.

4.3. Communication between clients and server

Fig. 5 shows the communication between the client and server through a sequence diagram. Once the pedestrian user runs the app
and connects the server, a connection request is sent to the main thread of the MTS simulation and a separate thread for the client is
created. The MTS main thread handles the creation and modification of the artificial vehicles’ data. In each iteration, the client's
thread obtains this data from the main thread and sends it to the pedestrian user so that he/she can visualize it accordingly. The client
app processes the data obtained from the user and then sends the pedestrian data to the server. Once this data is received in the
client's thread, it is updated in the MTS main thread accordingly. If the user decides to disconnect from the server, a close request is
sent to the server. At this point, the client's thread is closed by the server and the client is disconnecting without affecting other
clients.

Fig. 4. Pedestrian client view upon initialization of the system [(a). If the user clicks on “Select Avatar”, a list with the possible avatars (b) is shown
so that the user can select it.].

243
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 5. Sequence diagram of client-server communication.

The movement of the pedestrian avatars is implemented by changing the coordinates of the avatar's target in the visualization
client. The pedestrian will follow the coordinates given by the server, generating the impression that he/she is walking in the virtual
environment in the same way as the corresponding pedestrian user wearing the HoloLens headset. Because of this, each pedestrian
client needs to send the coordinates of the person walking through the city. The server receives these coordinates and sends them to
all the clients. The clients then update the position of each pedestrian's target appropriately. Fig. 6 shows how the visualization of the
pedestrian is updated according to the user's position. When the user starts to move, the target moves accordingly, updating the
position of the user's virtual avatar in all the clients connected to the network.
Apart from the user's location, the client also sends the avatar information selected by the user so that their virtual pedestrian is
visualized accordingly. Thus, for each pedestrian client, we send three pieces of information: X coordinate (horizontal position) of the
user's location; the Z coordinate (forward position) of the user's location; and the type of avatar selected by the user.

Fig. 6. The pedestrian's target is updated according to the user's position. [The target is represented by the red mark.].

244
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 7. A user wearing the HoloLens headset walks straight and the corresponding virtual pedestrian is visualized through a projector screen for
comparisons. (A top view of the virtual pedestrian is shown in (a)–(c), while (d)–(f) display the profile view of the virtual pedestrian.) [See the video
demonstration of the framework: click here].

4.4. Demonstration of the framework

To demonstrate our AR-PED simulation framework, we show different pictures of a user wearing the HoloLens headset with our
pedestrian client framework, while the corresponding visualization of the virtual pedestrian is also displayed on a projector screen
simultaneously. Readers can click on the picture to see a video of the demonstration. We used tape to measure a distance of 3 m and
placed 2 spheres on the crosswalk in the virtual scene that match the corresponding origin and destination points of our physical
space in Fig. 7(a)–(c). The spheres were separated by 3 units in Unity, which corresponds to 3 m in the physical world. Likewise, in
Fig. 7(d)–(f), the distance was increased to 3.75 m to exactly represent the width of a lane. Through these test scenarios, we can
determine that the measurements of our virtual scene correspond with the measurements in the real world. Fig. 7(a)–(c) show a
person using the pedestrian client framework and the corresponding visualization from top view, and Fig. 7(d)–(f) show the profile
view during the walk process.
Fig. 7 shows the virtual pedestrian in the projector screen from the perspective of a visualization client. Additionally, users can
access the visualization from the driving clients. Fig. 8 shows a pedestrian and a driver client visualizing the simulation simulta-
neously. The pedestrian can see the car controlled by the driver and the driver can observe the virtual avatar of the pedestrian. In this
case, the virtual distance between the pedestrian and the driver was about 16 m, while the physical distance was 1 m approximately.
Note that they are required to be close to each other. The actual pedestrian and the driver can physically stay at different locations
(e.g., different rooms). The virtual views from both clients are displayed in the figure, as well as a top-down view of the whole scene.
Those two screens are not needed in conducting any real experiments. The pedestrian and driver will see all the scenes from the
perspective of their own avatar through their headsets.

4.5. Evaluation of the framework

We perform different experiments to further evaluate the performance of the framework. First, we measure the latency of the
pedestrian's response. The latency is defined as the time elapsed between sending the pedestrian data from the user wearing the
HoloLens and another client visualizing the corresponding virtual pedestrian. We measured a best-case latency of 0.06 s and a worst-
case latency of 0.23 s. On average, we measured a latency of 0.15 s with a standard deviation of 0.05 s. We determined that this
latency is small enough for the purposes of our framework. When demonstrating the system, users do not notice a delay between the
pedestrian client and the visualization of the virtual pedestrian. This can be appreciated in the video demonstration that is linked to
Fig. 7.
In a different experiment, we also determine if users behave differently when being in the framework as pedestrians from when
they do in real world. To do this, we measure the time that it takes a person to cross a 14-meter crosswalk both in the framework and
in the real world. We performed this experiment with 5 different users. Each user crossed the road 4 times in the real world and 4 in
the framework. Fig. 9 shows the average time that it took each user to cross the intersection in the framework as well as in the real

245
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 8. A driver and a pedestrian participate the simulation simultaneously: (a) Pedestrian-driver participation; (b) and (c) show the virtual views
from the pedestrian and driver perspectives, respectively; and (d) shows a top view of the virtual scene.

Fig. 9. Average time (in seconds) that it takes to each user to cross a 14-meter crosswalk in the framework and in the real world.

world. On average, pedestrians took 1.15 s more to cross the intersection in the framework than in the real world. This can be due to
the fact that users are wearing a headset in the framework and thus they tend to walk relatively slower. However, we considered this
difference to be low enough and we determine that users behave closely as they do in the real world.
In the third experiment, we evaluate the accuracy of the virtual locations with respect to the actual location of the user wearing
the HoloLens. To do this, we set a path in the real world that users can follow while wearing the headset. In a perfect case scenario,
the pedestrian trajectory retrieved in the virtual world will match with our defined path. The path consists of five segments and four
90-degrees turns between them. Segments 1–4 are 3 m long, while segment 5 is 2 m. Fig. 10 shows the setup for the experiment.
We perform the experiment with 4 different users, and each user follows the path 3 different times. Fig. 11 shows the virtual
locations retrieved in each of the experiments and compares it with a reference line that corresponds with our physical path (as
shown in Fig. 10). For each user, we use a graph showing their virtual locations in the 3 different experiments, as well as a set of box
plots showing the deviations per segment. When calculating the deviations, we omitted the small portions of the path in which users
made turns, since a user might start performing the turn before reaching the end of a segment, and this behavior should not be
counted as an error. Based on the experiment results, it can be seen that the virtual trajectories retrieved by the framework were not
completely perfect, but they closely match the path defined in the real world. Additionally, it is observed that the deviation tends to
increase in the latter segments of the path. This might be due to calibration issues of the HoloLens headset. On average, we measure

246
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

Fig. 10. The user follows a physical path composed of 5 segments [The angle between adjacent segments is 90°.].

Fig. 11. Trajectories for each user in 3 different experiments (a–d) and their corresponding box plots of deviations for segments S1 to S5 (e–h).

an absolute error of 0.11, 0.07, 0.13 and 0.15 m for users 1–4, respectively. It is expected that different factors can contribute to such
errors. It has been reported that the location sensors in the HoloLens headset are more accurate at low movement speeds and in bright
conditions [25]. The walking pattern of each user (speed, movement of their head, rotations, etc.) also play a big role in the
estimation of the user location. On average we reported low deviations between the user's location and its virtual location. Since the
final virtual paths closely match with the physical path in a large space, we determine that the virtual locations are accurate enough
for the purpose of our framework.

247
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

5. Conclusions and discussion

In this paper, we presented the AR-PED pedestrian simulator that enables user-controlled pedestrians to coexist with user-con-
trolled vehicles and artificial vehicles generated by an MTS server. Multiple users can access the framework simultaneously to control
a virtual pedestrian that creates a pedestrian-in-the-loop simulation environment. Meanwhile, driver-controlled vehicle can also be
integrated to involve actual driving decision when perceiving pedestrians. We observed an average latency of 0.15 s, which it is low
enough to visualize the pedestrian's movement in real time. Additionally, we prove that both the virtual locations and crossing times
can closely match with those in the real world. The framework has been demonstrated in a sample city, but it can be easily deployed
to any 3D model of a traffic environment if its structure is matched with the road network in the MTS server. The MTS package
selected in this prototype was Quadstone Paramics, but it can be easily substituted with any MTS software such as Vissim that offers a
customizable API.
The developed AR-PED framework for pedestrian simulation can be used for different purposes and experiments, e.g., research of
pedestrian safety, training of young pedestrians, cognitive evaluation, or deployment and testing of autonomous vehicles interactions
with pedestrians. In the future, we plan to deploy autonomous vehicles to the framework so that we can further demonstrate the
advantages of our system for their testing and training in innovative transportation systems. Certainly, our AR-PED framework has
some limitations that need to be further improved in the future. The virtual pedestrians move according to the position of a target
(which is updated every time when the pedestrian client moves). This produces a movement that matches with the client. However,
the client's rotation is not kept track in the current stage of our project, and if the client pedestrian only rotates, it is not reflected on
the virtual pedestrian. Similarly, if the client moves backwards or laterally, the virtual pedestrian will move forward in the direction
that the client is moving. Another issue to be addressed in our future work is the better inclusion of multiple pedestrian clients
simultaneously. As of now, multiple pedestrian clients can be connected to the same framework. However, if these clients are in the
same room, the position of their virtual avatars will be different than their relative position in the real space. This can be solved using
hologram sharing techniques that are already available in the HoloLens SDK [26]. However, while sharing a hologram is a relatively
easy task, coordinating the hologram sharing with the connection with the server, as well as determining which users are in the same
room and which users are not, is a challenging task. In the future, we plan to address all these problems to enhance our framework.

Acknowledgemnts

The authors appreciate the support of the Department of Modeling, Simulation & Visualization (MSVE) at ODU, especially Dr.
Rick McKenzie in MSVE, for supporting this research work. The contents of this paper only reflect views of the authors who are
responsible for the facts and accuracy of the data and results presented herein. The contents of the paper do not necessarily reflect the
official views or policies of sponsoring agencies. The authors also deeply appreciate the editor and the anonymous reviewers that
have provided valuable comments and suggestions for improving the paper.

References

[1] S.E. Banducci, et al., The effects of cell phone and text message conversations on simulated street crossing, Hum. Factors 58 (1) (2016) 150–162.
[2] P. Rahimian, et al., Using a virtual environment to study the impact of sending traffic alerts to texting pedestrians, 2016 IEEE Virtual Reality (VR), Greenville, SC,
USA, 2016, pp. 141–149.
[3] D.C. Schwebel, et al., Virtual reality by mobile smartphone: improving child pedestrian safety, Inj. Prev. 23 (357) (2017) 1–5.
[4] J. McComas, M. MacKay, J. Pivik, Effectiveness of virtual reality for teaching pedestrian safety, CyberPsychol. Behav. 5 (3) (2002) 185–190.
[5] J.A. Thomson, et al., Influence of virtual reality training on the roadside crossing judgments of child pedestrians, J. Exp. Psychol. 11 (3) (2005) 175.
[6] D.C. Schwebel, J. Gaines, J. Severson, Validation of virtual reality as a tool to understand and prevent child pedestrian injury, Accid. Anal. Prev. 40 (4) (2008)
1394–1400.
[7] M. Hartmann, et al., “Pedestrian in the loop”: an approach using virtual reality, 2017 XXVI International Conference on Information, Communication and
Automation Technologies (ICAT), Sarajevo, Bosnia-Herzegovina, IEEE, 2017, pp. 1–8.
[8] M. Hartmann, et al., "Pedestrian in the loop": an approach using augmented reality, SAE World Congress Experience (WCX18), Detroit, USA, 2018.
[9] G. Simpson, L. Johnston, M. Richardson, An investigation of road crossing in a virtual environment, Accid. Anal. Prev. 35 (5) (2003) 787–796.
[10] B.A. Morrongiello, M. Corbett, Using a virtual environment to study child pedestrian behaviours: a comparison of parents’ expectations and children's street
crossing behaviour, Inj. Prev. 21 (5) (2015) 291–295.
[11] B.A. Morrongiello, et al., Using a virtual environment to examine how children cross streets: advancing our understanding of how injury risk arises, J. Pediatr.
Psychol. 41 (2) (2015) 265–275.
[12] J. Orlosky, et al., An interactive pedestrian environment simulator for cognitive monitoring and evaluation, Proceedings of the 20th International Conference on
Intelligent User Interfaces Companion, ACM, 2015.
[13] S. Deb, et al., Efficacy of virtual reality in pedestrian safety research, Appl. Ergon. 65 (2017) 449–460.
[14] B. Farooq, E. Cherchi, A. Sobhani, Virtual immersive reality for stated preference travel behavior experiments: a case study of autonomous vehicles on urban
roads, Transp. Res. Rec. 2672 (50) (2018) 35–45.
[15] Oculus VR. Oculus roomscale—tips for setting up a killer VR room. 2017 06/21/2018]; Available from: https://www.oculus.com/blog/oculus-roomscale-tips-
for-setting-up-a-killer-vr-room/.
[16] Steam VR. Steam VR support - what are the space requirements of the vive? 2018 06/21/2018]; Available from: https://support.steampowered.com/kb_article.
php?ref=7770-WRUP-5951.
[17] H. Yang, et al., A framework for interactive M3 visualization of microscopic traffic simulation, Transp. Res. Rec. 2672 (44) (2018) 62–71.
[18] M. Hassan, et al., Distributed microscopic traffic simulation with human in the loop and virtual reality technologies, Transportation Research Board's 98th
Annual Meeting, Washington, D.C., 2019.
[19] Z. Patterson, et al., Comparing text-only and virtual reality discrete choice experiments of neighbourhood choice, Landsc. Urban Plan. 157 (2017) 63–74.
[20] H Creagh, Cave automatic virtual environment, Electrical Insulation Conference and Electrical Manufacturing & Coil Winding Technology Conference, 2003.
Proceedings, IEEE, 2003.
[21] A. Dommès, et al., Towards an explanation of age-related difficulties in crossing a two-way street, Accid. Anal. Prev. 85 (2015) 229–238.

248
D. Perez, et al. Simulation Modelling Practice and Theory 94 (2019) 237–249

[22] B.G. Witmer, M.J. Singer, Measuring presence in virtual environments: a presence questionnaire, Presence 7 (3) (1998) 225–240.
[23] S. Nichols, H. Patel, Health and safety implications of virtual reality: a review of empirical evidence, Appl. Ergon. 33 (3) (2002) 251–271.
[24] G.D. Cameron, G.I. Duncan, PARAMICS—parallel microscopic simulation of road traffic, J. Supercomput. 10 (1) (1996) 25–53.
[25] Y. Liu, et al., Technical evaluation of HoloLens for multimedia: a first look, IEEE Multimed. 25 (4) (2018) 8–18.
[26] Microsoft. MR Sharing 240: mMultiple HoloLens devices. 2018 06/28/2018]; Available from: https://docs.microsoft.com/en-us/windows/mixed-reality/
holograms-240.

249