Computer Networks 49 (2005) 84–102

www.elsevier.com/locate/comnet

Adaptive server selection for large scale interactive

online games
a,*
Kang-Won Lee , Bong-Jun Ko b, Seraphin Calo a

a
IBM Thomas J. Watson Research Center, 19 Skyline Dr., Hawthorne, NY 10532, USA
b
Columbia University, New York, NY 10027, USA

Available online 25 May 2005

Abstract

Large scale interactive online games aim to support a very large number of game players simultaneously. To support
hundreds of thousands of concurrent players, game providers have so far focused on developing highly scalable game
server architectures and extensible network infrastructures. Recently, distributed online games are beginning to incor-
porate more interactive features and action sequences; thus, it becomes increasingly important to provision server
resources in an efficient manner to support real-time interaction between the users.
In this paper, we present a novel distributed algorithm to select game servers for a group of clients participating in a
large scale interactive online game session. The goal of server selection is to minimize the server resource usage while
satisfying a real-time delay constraint. We develop a synchronization delay model for interactive games and formulate
the server selection problem, and prove that the considered problem is NP-hard. The proposed algorithm, called zoom-
in–zoom-out, is adaptive to session dynamics (e.g., clients join) and lets the clients select appropriate servers in a
distributed manner such that the server resource is efficiently utilized. Using simulation, we study the performance
of the proposed algorithm and show that it is simple, yet effective in achieving its design goal. In particular, we show
that the performance of our algorithm is comparable to, or sometimes even better than, that of centralized greedy
algorithms, which require global information and extensive computations.

2005 Published by Elsevier B.V.

Keywords: Distributed systems; MMOG; On-line game infrastructure; Resource allocation; Distributed Algorithm; Synchronization
delay model

1. Introduction

Large scale interactive online games, such as

*
Corresponding author. Tel.: +1 914 784 7228; fax: +1 914
Massively Multi-player Online Games (MMOG),
784 6205. aim to support a very large number of game play-
E-mail address: kangwon@us.ibm.com (K.-W. Lee). ers simultaneously. In practice, MMOG providers

1389-1286/$ - see front matter
2005 Published by Elsevier B.V.

doi:10.1016/j.comnet.2005.04.006
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
often are required to support hundreds of thou-
sands of geographically distributed users at the
same time. For example, it has been reported that
about 165,000 concurrent users play Lineage [1] in
Taiwan [2]. To support such a large number of
players, MMOG providers have so far focused
on developing highly scalable game server archi-
tectures and supporting network infrastructures
that span wide geographical regions while satisfy-
ing loose real time requirements [3,4]. Recently,
however, large scale online games are beginning
to incorporate more interactive features and action
sequences (e.g., first person player-to-player com-
bat) to provide a realistic game experience [5].
Thus it becomes increasingly important to provi-
sion enough server resources to support real-time
interaction between users.
There are several challenges in augmenting
MMOGs with such interactive features. First,
unlike in online first-person-shooting (FPS)-type
games where a small number of users that are geo-
graphically close to each other get assigned to the
same game session, MMOG games must maintain
a persistent virtual world view for a large number
of game players that are distributed over the net-
work. Second, the maintenance of a long-lived per-
sistent world mandates a server-based game
architecture, where clients interact with a central
server that keeps track of the game states. How-
ever, it is well known that the conventional ser-
ver–client architecture does not scale well as the
number of clients increases [6]. To overcome this
limitation, a mirrored server architecture has been
proposed [7–9], where a set of distributed game
servers are allocated and orchestrated to support
a large number of distributed clients. In this archi-
tecture, the game servers are typically intercon-
nected via well provisioned network links, and
each game client is directed to connect to a nearby
game server. The game servers simulate the game
state according to the actions of the players and
maintain a consistent game state among them-
selves so that all the clients can have a consistent
view of the game world.
In parallel with this architectural development,
proposals have been made to dynamically provi-
sion game servers on the fly exploiting emerging
Grid technologies [10,11,3]. According to these
85
on-demand game architectures, game servers can
be dynamically provisioned to accommodate the
capacity requirements as the user demand changes.
In addition to efficient resource utilization, such
on-demand game architecture removes the burden
of server and network management from the game
service provider. Also by multiplexing and sharing
a large scale infrastructure across multiple games,
the service provider can try out new titles without
having to make a large initial investment in infra-
structure [10]. Thus it now becomes an important
issue to dynamically provision and utilize shared
server resources in an efficient manner.
In this paper, we present a novel distributed
algorithm that selects game servers for a group
of clients participating in large scale interactive on-
line games. The goal of the server selection algo-
rithm is to select the minimum number of servers
from all available servers, while satisfying the
real-time delay constraint of the game. To solve
this problem, we first develop a synchronization
delay model for interactive distributed games, for-
mulate the server selection problem, and prove
that finding an optimal solution of this problem
is NP-hard. We then propose a novel heuristic
algorithm for the considered server selection prob-
lem that can be easily implemented in distributed
environments. The proposed algorithm, called
zoom-in–zoom-out, is adaptive to session dynamics
(e.g., clients join) and lets the clients select appro-
priate servers in a distributed manner such that the
server resource is efficiently utilized. Through sim-
ulation-based evaluation, we study the perfor-
mance of the proposed algorithm with various
zoom-in–zoom-out techniques and show that it is
simple, yet effective in achieving its design goal.
In particular, we show that the performance of
our algorithm is comparable to, or sometimes even
better than, that of centralized greedy algorithms,
which require global information and excessive
computation.
The remainder of this paper is organized as fol-
lows. Section 2 presents synchronization delay
model and problem formulation. Section 3 pre-
sents the proposed server selection algorithm. Sec-
tion 4 presents the performance of the proposed
algorithm varying parameters and in comparison
with more expensive greedy algorithms. Section 5
86 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
presents an overview of related work, and finally
Section 6 concludes the paper.
2. Problem formulation
2.1. System model
We assume a mirrored-server architecture,
where multiple game servers are interconnected
via well-provisioned links. In our model, a server
refers to an entity that collectively maintains a
persistent virtual world, calculates the game state
based on the events generated by the players. A
client refers to an entity that renders and presents
the game states to the player. We call the network
interconnection between mirrored servers the ser-
ver network. The server network is assumed to be
private and dedicated to the game service pro-
vider.
We further assume that the virtual game world
is divided into multiple regions, wherein relatively
few game players (compared to the number of
users in the entire game world) directly interact
with each other. In general, a region is defined
by the geography or architectural structure in the
virtual world, and the game players in the same re-
gion may engage in combat or first person shoot-
ing actions. We call the persistent state of a
region a session. A single session may span multi-
ple game servers to accommodate geographically
distant clients. Each client is assigned to one of
the servers, called the contact server of the client,
which is responsible for forwarding the clientÕs ac-
tion events to all the other servers participating in
the same session. Upon receiving clientsÕ action,
each game server independently calculates a new
game state, and sends the updated state to the di-
rectly connected clients. We note that this client
and server relation effectively defines a one-to-
one mapping from the clients to the servers. We
call this mapping the server allocation or just allo-
cation. When a player moves from one region to
another region, the client may need to hand-off
from the current contact server to another server,
which hosts the session for the new region. Discov-
ering and subscribing to a new server can be facil-
itated by techniques presented in [12–14].
For accurate game simulation and presenta-
tion, game events generated by the players must
be time-stamped according to a global clock or
must be serialized in a total order. In practice,
however, having a global clock or determining a
total order of game playerÕs actions is infeasible.
Instead, most practical game systems adopt a syn-
chronization mechanism, which divides time into
discrete slots and lets each server wait for events
generated by remote clients to arrive before exe-
cuting game simulation [15,16,7,17]. Depending
on the scheme, inconsistencies due to late events
are resolved by rolling back the game state, and
lost events are dead-reckoned based on the action
history. In this paper, we do not assume a partic-
ular synchronization mechanism. Instead we just
assume that the synchronization delay must be
greater than the latency for the events generated
by clients to reach the farthest server, and for
the results of game simulation to reach all the
local clients. This condition must be met because
we assume that each server simulates the game
state independently and for that it has to receive
all the events from the farthest client before per-
forming the simulation. Based on this condition,
we define the synchronization delay as follows:
the synchronization delay between a client and a
server is the time difference between the instance
that the client sends its playersÕ actions and the
instance that the client renders a new game state
in response to the actions sent to the server. This
synchronization delay depends on the network
latency from the client to the server (upstream
latency), processing time at the server, and the
network latency from the server to the client
(downstream latency).
To synchronize game play and interaction
amongst all players participating in the same ses-
sion, we must take into account synchronization
delays between all clients in the session and the
corresponding servers. More specifically, a game
server must calculate a new game state after receiv-
ing the action events from the farthest client.
Otherwise, the action from the farthest client will
not be synchronized with others. Similarly, at the
client side, a new game state should not be pre-
sented to the players until the same game state is
delivered to the farthest client from the server.
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
Otherwise, the game simulation becomes unfair to
the farthest clients [18].
2.2. Delay model and problem statement
Let Cg denote a set of game clients that partic-
ipate in a game session g and let S denote a set of
available game servers. Servers in S form an undi-
rected connected graph, called the server network.
Let ds(j, k) denote the shortest distance (or latency)
between servers sj and sk, and let dc(i, j) denote the
distance between a client ci and a server sj. Suppose
client ci is mapped to server sj in some allocation
Ag. Then we say server sj serves client ci under
Ag. We define Cg,j
Cg as the set of clients that
are directly connected to the server sj, i.e.,
Cg,j = {ci 2 Cgjsj serves ci}. We also define the ses-
sion server set, S(Ag) under the allocation Ag as the
set of servers that serve at least one client in C, i.e.,
S(Ag) = {sj 2 SjCg,j 5 /}.
Let Du(ci, sk) be the upstream distance between a
client ci and a server sk. We note that Du(ci, sk) is
defined not only for a client and a server that are
directly connected, but also for a client and a ser-
ver connected indirectly via some other servers for-
warding the client ciÕs actions to server sk through
the shortest path in the server graph, i.e.,
Du(ci, sk) = dc(ci, sj) + ds(sj, sk), where sj is the
contact server for ci. On the other hand, the down-
stream distance from sj to a client ci 2 Cg,j is just
dc(sj, ci) as ci receives game states from its contact
server sj.
Fig. 1 presents an example. There are three game
sessions: (i) session g1 consisting of clients c1, c2, c3,
c7, and c8, represented in gray; (ii) session g2 con-
sisting of clients c4 and c5, represented by double
lines; and (iii) session g3 consisting of clients c6
and c9, represented in white. The server allocations
Fig. 1. Example of game server architecture with three sessions.
87
for each session are S(Ag1) = {S1, S2, S5}, S(Ag2) =
{S2, S4}, and S(Ag3) = {S3, S4}. To illustrate the
latency calculation, take the events generated by
client c5 in session g2. In this case, the upstream
latency from c5 to s2 is Du(c5, s2) = dc(c5, s4) +
ds(s4, s2). The downstream latency from s4 to c5 is
dc(s4, c5).
We now consider the overall synchronization
delay for a session. As previously discussed, for a
server to simulate game state in a fair manner it
must wait until all the action data from all the
clients in the session to arrive. In other words, a ser-
ver sj must wait for maxci 2Cg Du ðci ; sj Þ before game
simulation. Then it processes the action data and
sends out updates to all the clients it serves. This
game state update takes up to maxci 2Cg;j d c ðsj ; cj Þ
for the farthest client. Since this condition must
hold for all servers, the overall session synchroniza-
tion delay Ds(A) under allocation A can be suc-
cinctly written as follows:
 
Ds ðAÞ ¼ max max Du ðci ; sj Þ þ max d c ðci ; sj Þ .
sj 2SðAÞ ci 2C g ci 2C g;j
Based on this session synchronization model, we
now formulate our server selection problem. Our
goal in this paper is to find a server allocation that
minimizes the number of servers allocated to a ses-
sion, while satisfying a given synchronization
delay requirement.1 Restated formally, the prob-
lem is: given a network topology consisting of a
set of servers S, a set of clients C, a set of links be-
tween the servers and the clients that are annotated
with latency between the nodes, and a real-time
delay requirement D for a game session, find a ser-
ver allocation Amin that minimizes jS(A)j subject to
Ds(A) 6 D and jS(A)j P 1. This problem is NP-
hard as an efficient solution to this problem can
effectively solve other NP-hard problems.
Theorem 1. The minimum game server allocation
problem is NP-hard.
1
In this paper, we only consider the server selection problem
for a single game session. The problem of optimally allocating
servers for multiple dynamic game sessions is a very challenging
problem of its own and we leave it as a future research topic.
88 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
Proof. Consider, in our model, the case when the
latency between every pair of servers is zero. Fur-
ther assume that the latency between any client
and the closest server to it is D/2, where D is the
sync-delay bound in our model. Then the optimi-
zation goal in this specific case would be to mini-
mize the number of servers subject to the
condition that every client is within distance D/2
from its contact server, to which it is allocated.
This is exactly a set-covering problem, in which a
set is defined by the set of clients that are within
the distance D/2 from each server. Since our prob-
lem generalizes the set-covering problem, which is
a well-known NP-hard problem, our problem is
also NP-hard. h
3. Server selection algorithm
3.1. Impact of server allocation
In this section, we first analyze the impact of
server allocation using an example. Fig. 2 illus-
trates a simple example with three servers, s1, s2,
and s3, and two clients, c1 and c2. Fig. 2(a) shows
a network configuration, where the solid lines rep-
resent the latency between the servers in the server
Fig. 2. Examples of game server allocation. (a) The topology,
(b) allocation A1, (c) allocation A2 and (d) allocation A3.
network, and the dashed lines represent the latency
between each server–client pair.
Fig. 2(b)–(d) shows three different server alloca-
tions, A1, A2, and A3, where allocated session serv-
ers are marked in gray. According to the session
delay model, the overall delay for allocation A1
is 10 with both maximum upstream and down-
stream delays being 5 (between c2 and s1). Allocat-
ing s2 instead of s1 as the session server (allocation
A2) reduces the synchronization delay to 6 (up-
stream/downstream 3 each). Note that, though
the end-to-end distances between the clients are
the same in A1 and A2, the synchronization delay
is reduced by placing the server at the center of
the network. Finally, if we allocate two servers at
the edges of the server network as in allocation
A3, the synchronization delay decreases to 4 with
the upstream latency 3 (e.g., along the path c1–
s1–s2–s3) and the downstream latency reduced to
1. In this case, the reduction comes from placing
the servers near the clients at the expense of using
two servers.
From this example, we can draw the following
intuition to design our server selection algorithm:
• If we had to choose only one server to minimize
the overall synchronization delay, then it would
be optimal to select a server that minimizes the
maximum distance (not the average) to all cli-
ents. In graph theory, such a node is called
the center of a network.2 In this paper, we call
it a core server.
• If it is faster to forward packets via the contact
server over the server network than to send
packets to a remote server directly over the pub-
lic Internet, allocating servers near the ‘‘edge’’
of the server network (and close to the clients)
reduces the overall synchronization delay.
However, this comes at the expense of increas-
ing the number of servers in the session.
2
Formally, the eccentricity of the vertex v in a graph G is the
maximum distance from v to any other vertex. The center of a
graph G is the set of vertices of eccentricity equal to the radius,
where the radius of G is the minimum eccentricity among the
vertices of G.
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
Based on these observations, we design a distrib-
uted server selection algorithm as follows.
3.2. Server selection algorithm
We design our server selection algorithm based
on our observation presented in the previous sec-
tion: the synchronization delay of a session is small
when clients are connected to the ‘‘edge’’ servers;
and it increases as the clients are connected to serv-
ers near the core server. Since our goal is to reduce
the number of session servers, at a high level, our
algorithm attempts to reduce the ‘‘diameter’’ of
the network of session servers while keeping the
synchronization delay below the bound, D. To this
end, each client visits a sequence of servers that are
successively closer to the core server. This proce-
dure has an effect of increasing the chance for
more clients to share a common server, and there-
by decreasing the number of total session servers,
at the expense of increased synchronization delay.
Given a set of clients C, a set of servers S, link
configuration, and delay requirement D, our algo-
rithm performs the following procedure.
Step 1: Initially allocate each client ci 2 C to the
closest server in S.3 Denote this initial con-
tact server of ci by s0(ci). This produces an
initial allocation A. We assume that the
session has been provisioned to satisfy
the delay requirement D, i.e., Ds(A) 6 D.
Otherwise, the violation is marked, a high
level session regrouping module is called,
and then exit.
Step 2: Find the core server, s* 2 S, of the session
that minimizes the maximum distance to
the clients.4 Finding the core of a network
in a distributed system has many interest-
ing applications (e.g., core-based multicast
tree construction), and thus has been stud-
3
This can be done by roundtrip-time measurement based on
a game server directory in the region.
4
The core is not necessarily the same server for multiple game
sessions since the core is determined by the distribution of the
clients in a particular game session.
89
ied in various contexts. In this paper we
employ a tournament-based method stud-
ied in [19].
Step 3 (Zoom-In): At each client ci, do the fol-
lowing. Migrate each client, ci, from the
current contact server, sk(ci), to another
server, sk + 1(ci), that is further from ci
and is closer to s* than sk(ci). Using this
as a new contact server sk + 1(ci), calculate
the session synchronization delay. If the
delay is within the bound D, then mark
the server sk + 1(ci) to be the ‘‘new contact
server’’, and notify all clients in the session
of the successful migration. When notified
of a successful migration of some other cli-
ent, mark the current server to be the ‘‘new
contact server’’. Repeat this step for all cli-
ents until all clients arrive at the core server
s*, at which point each client migrates to
the ‘‘new contact server’’.
Step 4 (Zoom-Out): At each client ci, do the fol-
lowing. Let S 0 be the current set of contact
servers for all clients. Then among the
servers in S 0 that are further from the core
server s* than the current contact server,
find the closest server to ci. Then probe
the session synchronization delay to see
whether the session synchronization delay
would still be within the bound D if the cli-
ent migrated to this new server. If yes, ci
migrates to the new server. Otherwise, take
no action. Repeat this step for all clients
until no client can migrate to a server in
S 0 that is further from s* than the current
server s(ci).
To ensure correctness, we assume the existence of
a mechanism to enforce that clients do not per-
form Steps 3 and 4 concurrently. In practice, such
a serialization can be obtained by using a token-
passing scheme amongst clients.
The procedure in Step 3 has the effect of moving
a cluster of the session servers toward the core ser-
ver; hence we call it zoom-in. With this procedure,
the number of session servers tends to decrease,
but the overall synchronization delay tends to in-
crease because the latency between clients and
their contact servers increases. This zoom-in
90 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
process terminates when all clients have reached
the core server s*. At this point, each client ci con-
nects to the last server in the migration procedure
that makes the overall synchronization delay
below the bound.
Note that, in the zoom-in procedure, we at-
tempt to move all clients up to the core server even
after the synchronization delay exceeds the delay
bound. This is because the synchronization delay
does not always monotonically increase, but some-
times decreases as we move clients from the edge
to the core. Such fluctuations in the synchroniza-
tion delay are frequent particularly when the loca-
tion of a new contact server deviates from the
shortest path from the client to the core server.
Therefore, if we terminated the zoom-in process
when we first encounter increase in the synchroni-
zation delay, then the process may be trapped in a
local minimum point. By iterating the zoom-in
step all the way up to the core server, the client
can explore the entire search space and avoid stop-
ping at local minima.
By performing the procedure in Step 4, we seek
to further reduce the number of session servers by
‘‘freeing up’’ some servers among those obtained
in Step 3 that do not contribute to decreasing the
synchronization delay. This is possible because
after the zoom-in procedure, servers near the core
server are most likely to have been selected, but
they may be removed without increasing the syn-
chronization delay. This process is called zoom-
out since it tends to remove servers from the core
and proceed to the outside, i.e., to the boundary
of the session server cluster. Note that, in zoom-
ing-out process, since the clients attempt to mi-
grate to another server among the set of servers
that are currently selected, Step 4 can only reduce
the number of session servers. If all clients could
connect to the core server in Step 3, this is an
optimal selection and we do not need to perform
Step 4.
Note that, in practice, the information shared
by the session clients can be exchanged through
the ‘‘server network’’ rather than maintaining a
separate channel for signaling between the cli-
ents. For instance, in Step 3 of the algorithm,
the notification for successful migration of a cli-
ent can be sent to all the servers in the corre-
sponding session. More specifically, when a
client successfully migrates to a new server, that
server notifies of this migration to all the other
session servers in S(Ag). Then, each session ser-
ver, in turn, informs its clients that they can
mark the current server as the new contact ser-
ver. This approach is more scalable than having
each client individually contact all the session cli-
ents since the number of session servers is typi-
cally small compared to that of clients, and we
can minimize the session state information main-
tained by each client.
Fig. 3 illustrates the above procedure through
an example. In Fig. 3(a), each of five clients,
c0, . . ., c4 is initially allocated to the closest server.
In this allocation, we have five servers participat-
ing in a game session by the clients. Now, in Fig.
3(b), each client incrementally probes servers that
are closer to the core server, for example, client
c2 migrates twice toward the core server as long
as the delay bound is satisfied. As a result of the
zoom-in procedure, three servers near the core ser-
ver are selected. Now, clients further seek to re-
duce the number of session servers by moving
out from the core server. For example, in Fig.
3(c), two clients, c1 and c2, were able to migrate
to the session servers closer than the core server,
with s* no longer being allocated to any client.
The final server allocation is shown in Fig. 3(d),
with only two servers being selected by the
procedure.
In what follows, we present several issues in
implementing this algorithm in a scalable and effi-
cient manner.
3.3. Implementing the selection algorithm
3.3.1. Minimizing migration overhead
The first issue concerns game session migration.
Although a game architecture based on on-de-
mand technologies enables our dynamic server
selection, migrating a game session from one ser-
ver to another is a relatively expensive procedure.
As a result, probing a new server and trying to mi-
grate the game session in each zoom-in and zoom-
out step by a client is not a good idea. We address
this issue by handling the zoom-in and zoom-out
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102 91

c0 c0

c2 c2

S* S*
c1 c1

c3 c4 c3 c4

a b

c0 c0

c2 c2

S* S*
c1 c1

c3 c4 c3 c4

c d

Fig. 3. ZIZO algorithm. (a) Initial allocation, (b) Zoom-In, (c) Zoom-Out and (d) final allocation.

steps in the control and management plane, not in
the actual game session plane. To support this
operation, a server that has been probed by a client
runs a simulation on synchronization delay com-
putation and maintains a virtual state. The clients
keep probing the next possible server following the
server selection algorithm until they reach a steady
state. Only when the entire zoom-in and zoom-out
process terminates, do the clients coordinate and
move the game session to the newly allocated serv-
ers. In this way, we can minimize the management
and session migration cost. In addition, we note
that the search process need not occur every time
when there is a change in the session. A practical
implementation may control the frequency of ser-
ver search procedures by specifying a threshold
or a damping factor. For example, the proposed
server selection algorithm may be triggered only
when certain number of servers have been added
to the session.
3.3.2. Scalable server search method
The performance of the ‘‘zoom-in’’ and ‘‘zoom-
out’’ procedures in our proposed algorithm relies
on the effectiveness of the server search method.
We consider three types of search methods: (i) full
search, (ii) partial search, and (iii) tree-based
search.
In the full search method, each client searches
for the next server to migrate among all servers
in S. This may be costly if S is large, but will pro-
vide the most accurate result.
In the partial search method, when we discover
a core server s*, each server discovers up to k clos-
est servers that are closer to s* than itself, and up
to k closest servers that are further from s*, where
k < jSj. When a client searches for the next server,
it considers only the servers in these sets as the can-
didates. In order to provide an estimate of the dis-
tance from its clients to the other servers, each
server evaluates the distances to the other servers
92 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
based on the distance in the public network
domain, on which the clientsÕ data travel to the
servers.
The tree-based method is similar in spirit to the
partial search method in that each client utilizes
the set of servers discovered by the current server
in searching for the next server. The difference is
that each server, si, selects only one server, p(si),
which is the closest to itself among those closer
to the core server s*. Then si registers p(si) as its
parent server, and informs p(si) that it has done
so. This mechanism effectively forms a core-based
tree, with s* being the root of the tree, along which
each client selects the next server when it migrates
from one server to another. More specifically,
when zooming in, each client migrates to the par-
ent server of the current server; hence there is no
need to ‘‘search’’ for the next server. On the other
hand, when clients zoom-out, they search for the
next server that is further from the core server
among the children nodes of the current server in
the server tree.
The partial search and the tree-based method
enable the clients to narrow the set of the candi-
date servers within which the clients search for
the next server at each step. They are an approxi-
mation based on a hypothesis that when a client
selects the next server, it will be most likely on
the shortest path from the client to the core server.
The above three different search methods repre-
sent different levels of trade-off between the search
space and the cost: the full search method will
examine all possible alternatives at high complex-
ity, whereas the tree-based method will consider
only those servers that are on the shortest path,
but does not incur search cost. We expect that
the partial search method will perform similarly
to the full search method as we increase k.
3.3.3. Coordinating client migration
Next, we consider two different types of client
migration strategies for the zoom-in and zoom-
out process. In a naive implementation, each client
may individually probe candidate servers and may
make a decision on migration. In this case, it is
possible that, while some clients connected to a
server could change their contact servers, others
may fail to migrate because it would violate the
overall delay constraints. Therefore this uncoordi-
nated migration of clients may result in a subopti-
mal result by temporarily increasing the number of
servers allocated to the session. An alternative ap-
proach we consider in this paper is to coordinate
the migration of all clients on the same server
simultaneously, and move them only when all the
clients can migrate to new servers.
3.3.4. Server capacity
The delay model presented in Section 2 and our
base-line algorithm in Section 3.2 do not take into
account the processing time at the server. How-
ever, if a server needs to handle a lot of game cli-
ents, its processing time can be a bottleneck in
overall latency in practice. While we mostly leave
this issue as a future research topic (e.g., incorpo-
rating the impact of the number of clients in a ser-
ver into the delay model), we take a simple
heuristic approach to handle this case in this
paper.
When the number of clients at a server con-
cerns, we impose an upper bound on the number
of clients that can be connected to a server, up
to which we assume the processing time remains
constant. Then we slightly modify our algorithm
to deal with the bounded capacity of the servers
as follows. When a client performs zoom-in and
zoom-out process, it first searches the best server
to migrate to; however, if the server has already
reached its bound on the number of clients, then
it attempts to migrate to the next best server,
and so on, until it finds a server that can accom-
modate more clients. More specifically, in Step 3
of the algorithm, if a client ci is currently con-
nected to a server s(ci), it first attempts to migrate
to the closest server among the servers that are clo-
ser to the core server and further from ci. If this
server is full, ci attempts to migrate to the next
closest server, and so on. During a zoom-out pro-
cess, the client searches for a server to migrate to in
a similar manner, but only among the set of cur-
rent contact servers. We evaluate this ‘‘capaci-
tated’’ version of the algorithm in Section 4.
3.3.5. Overhead of the algorithm
Finally, we study the computational overhead
of the proposed algorithm. Overall we observe that
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
the computation overhead of the algorithm in-
curred at each client is moderate. The Step 1 re-
quires a communication overhead of O(jSj) for
each client for exact computation. In most practi-
cal situations, however, it will be O(l) where
l jSj is the number of servers in the clientÕs re-
gion. The Step 2 requires a total of log2jSj tourna-
ments for the entire session. The Step 3 requires
either O(1) or O(k) depending on the type of the
zoom-in algorithm (see Section 3.3). The Step 4
requires O(m) where m jSj is the number of
session servers.
In the next section, we present the effectiveness
of the design alternatives considered above using a
simulation-based study.
4. Performance evaluation
In this section, we evaluate the performance of
the proposed algorithm through simulations. For
simulation, we use a synthetic, two-level,
hierarchical topology generated by the BRITE
topology generator [20]. Specifically, we generate
50 AS domains derived by Barabási–Albert model
[21], and for each AS domain, we then generate
100 nodes connected by Waxman model. Out of
these 5000 nodes generated (50 AS · 100 nodes
per AS), we randomly select 100 servers and 50
clients to participate in the same game session
for each simulation scenario. We then measure
the performance of the server selection schemes
by averaging the results obtained from 50 simula-
tion runs using 50 different sets of servers and
clients selected randomly. We emphasize that we
select a relatively small number of clients (with
respect to that of the servers) in the simulation
since our goal is to evaluate the performance of
the algorithm for a ‘‘single’’ session. In reality,
however, there will be many such sessions and
the total number of clients will be much larger
than that of servers.
To emulate a well-provisioned server network
with little congestion, the inter-server latency is
set to be smaller than the client-to-server latency.
In particular, we study the cases when the latency
between two servers is 10%, 30%, and 50% of the
latency in the underlying topology produced by
93
the topology generator. We denote this latency
reduction factor by a parameter f(=0.1, 0.3, 0.5).
We also use some abbreviated notations to
represent different types of client migration
approaches during the Zoom-In and Zoom-Out
process as explained in Section 3.3. The first set
of indices represents the way clients select the next
server to migrate:
• Tree: Tree-based search (clients migrate along
the shortest-path tree to the core server).
• Part: Partial search (clients migrate by search-
ing in the candidate server set).
• Full: Full search (clients migrate by searching
for the next server among ‘‘all’’ servers).
Particularly in the partial search cases, we
denote by k the maximum number of candidate
servers given by the current server.
The second set of indices represents two differ-
ent ways of moving clients from one server to
another:
• Un: Uncoordinated migration (each client
migrates to a new server individually).
• Co: Coordinated migration (all clients con-
nected to a server migrate simultaneously).
We have also evaluated the effect of the migra-
tion order of the clients by comparing several ap-
proaches, e.g., round-robin, random order, and
furthest-client-first (from the core). Overall, we
find that the order in which clients migrate does
not affect the end result of the simulation. In this
section, we present the case of the round-robin
migration.
For evaluation, we compare the performance of
the proposed algorithm with two centralized server
selection algorithms, namely, greedy server selec-
tion and client clustering algorithms. In what
follows, we briefly describe these centralized algo-
rithms before showing the performance results of
the proposed scheme.
4.1. Reference centralized algorithms
The greedy selection algorithm attempts to
incrementally add a new server to the session server
94 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
set by exhaustively searching for a server that,
when added, results in the smallest synchroniza-
tion delay. More specifically, at each step of the
greedy selection, we calculate the synchronization
delay when a non-session server has been tempo-
rarily added to the session server set and each cli-
ent is connected to the closest session server. After
evaluating all non-session servers, we select a ser-
ver that produces the lowest synchronization delay
when added. We keep adding servers to the session
server set in this manner until the synchronization
delay becomes smaller than the bound. This algo-
rithm is similar to the ‘‘greedy algorithm’’ pre-
sented in [22] in the context of the Web server
replica placement problem.5 In [22], the greedy
algorithm has been shown to perform closely to
the optimal solution.
The second reference algorithm, called client
clustering algorithm, also attempts to select servers
in a greedy manner. But, this algorithm is different
from the first algorithm in that, while the greedy
algorithm keeps adding a server that would mini-
mize the synchronization delay, the clustering
algorithm adds the server that would maximize
the number of clients that will be served by it. Con-
ceptually, let us assume that each server is assigned
a disk of radius x. We say the server ‘‘covers’’ the
clients that are within the radius from the server if
the server–client distance is smaller than x, i.e.,
dc(ci, sj) 6 x. Using this model, we select a server
that covers the largest number of clients that are
not yet covered by any server in the session server
set. We keep adding servers until all clients are
covered. Now we repeat this process for all possi-
ble disk sizes for each server, and then select the
smallest size session server set for which the syn-
chronization delay is within bound.6 For the set-
covering problem, this selection strategy is known
to have a O(1 + log N) performance bound [23].
Note that these centralized algorithms are not
only impractical to be applied in a distributed net-
5
The goal of server replica placement problem in [22] is to
choose M replicas among N potential locations to minimize the
clientÕs access latency.
6
If the current disk radius is x, we increase the radius by the
minimum of the distance from each server to its closest client
that is not yet covered by a server.
working environment, but also are computation-
ally expensive: the greedy selection must evaluate
the new synchronization delay for each of the serv-
ers that have not yet been added to the session ser-
ver set, and the clustering algorithm must select
the session server set for all possible values of
server disk size.
4.2. Number of servers selected
Fig. 4 compares the number of servers allocated
by the centralized algorithms and the proposed
algorithm with full server search when the inter-
server latency is relatively small (f = 0.1 in Fig.
4(a)), moderate (f = 0.3 in Fig. 4(b)), and large
(f = 0.5 in Fig. 4(c)). The y-axis represents the
number of servers selected by each algorithm and
the x-axis represents the delay bound D. In gen-
eral, we observe that, when the full server search
method is employed, our proposed algorithm per-
forms as well as, or sometimes even better than the
centralized algorithms (when f = 0.5 and D < 30).
As the delay bound increases, all algorithms find
that fewer servers can meet the delay requirement.
In particular, when the delay bound is sufficiently
large, we note that all the algorithms can find the
optimal server allocation, which is to allocate only
one server.
When the inter-server latency is relatively large,
more servers are needed to satisfy the synchroniza-
tion delay bound, particularly when the delay
bound is tight. This is because the inter-server la-
tency becomes the bottleneck and therefore it is
better for the clients to connect to nearby servers
when the delay bound is small. On the other hand,
if the server-to-server delay is relatively small, it
leaves some ‘‘slack’’ in the client-to-server latency,
and it allows the clients to connect to a relatively
distant servers. As a result, the number of servers
required to meet the delay bound is reduced. Note
that our proposed algorithm sometimes finds bet-
ter solutions than the centralized algorithms do
(especially when the delay bound is tight). We ob-
serve this to be the case because the centralized
algorithms tend to select the servers near the center
of the network and keep them in the selected set,
while our algorithm can remove redundant servers
near the center during the zoom-out process. We
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102 95

25 25
ZIZO-Full-Un ZIZO-Full-Un
ZIZO-Full-Co ZIZO-Full-Co
Greedy Greedy
20 Clustering 20 Clustering
# of servers selected

# of servers selected
15 15

10 10

5 5

0 0
25 30 35 40 45 50 25 30 35 40 45 50
delay bound (msec) delay bound (msec)

a b

25
ZIZO-Full-Un
ZIZO-Full-Co
Greedy
20 Clustering
# of servers selected

15

10

0
25 30 35 40 45 50
delay bound (msec)

Fig. 4. The number of servers selected: full search vs centralized, server capacity unbounded. (a) f = 0.1, (b) f = 0.3 and (c) f = 0.5.

verify the effectiveness of the zoom-out in Section client results in as good a server allocation as the
4.4. case with explicit client coordination.
We further notice that the coordinated client In Fig. 5, we evaluate the effectiveness of the
migration offers little benefit. This observation im- partial server search mechanisms with different
plies that the independent decision made by each values of k, i.e., the maximum number of servers

20 20
ZIZO-Tree-Un ZIZO-Tree-Co
ZIZO-Part-Un (k=5) ZIZO-Part-Co (k=5)
ZIZO-Part-Un (k=10) ZIZO-Part-Co (k=10)
ZIZO-Part-Un (k=20) ZIZO-Part-Co (k=20)
# of servers selected

# of servers selected

15 15
ZIZO-Full-Un ZIZO-Full-Co

10 10

5 5

0 0
25 30 35 40 45 50 25 30 35 40 45 50
a delay bound (msec) b delay bound (msec)

Fig. 5. The number of servers selected: partial search, server capacity unbounded, f = 0.3. (a) Uncoordinated migration and
(b) coordinated migration.
96 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102

25
ZIZO-Tree-Un
tice, each server will be limited to handling up to
ZIZO-Tree-Co
ZIZO-Part-Un (k=5)
certain number of clients. Fig. 6 shows the case
20 ZIZO-Part-Co (k=5)
ZIZO-Part-Un (k=10)
where the maximum number of clients that each
ZIZO-Part-Co (k=10) server can handle is 10. In this case, our algorithm
# of servers

15
10
5 can accommodate more clients. From the figure,
0
25
Fig. 6. The number of servers selected: bounded server capac-
ity, f = 0.3.
that each client searches, when f is 0.3. Fig. 5(a)
shows the case when clients migrate individually,
while Fig. 5(b) shows coordinated migration. We
observe that, as k increases, the performance of
partial search will become close to that of full
search in both the figures. In particular, searching
for a next server among (maximum) 5 candidate
servers produces significant improvement over
the tree-based migration, and k = 20 (out of 100
servers) results in a performance that is very close
to the full search case. From this result, we can
conclude that it is worthwhile for each client to
search for the next server from a small set of
candidate servers instead of searching from a
tree.
So far we have seen the case when there is no
bound on the server capacity. However, in prac-
ZIZO-Full-Un
ZIZO-Full-Co first searches the best server to migrate to; how-
ever, if the server is full, it searches for the next
best server, and so on, until it finds a server that
the performance of the algorithm improves as we
30 35 40 45 50 expand the search space for each client by increas-
delay bound (msec) ing k. However, when the number of candidate
servers is very small as in tree-based migration
method, the algorithm performs quite poorly. In
particular, even when the delay bound is suffi-
ciently large, the tree-based algorithms fail to
reach the optimal number of servers (which is five
servers). This is because migration along a fixed
server tree severely limits the search space for each
client, and the chance of the same server being
contacted for candidate server for migration is rel-
atively high. However, we find that the partial
search with large k and full search can find the
optimal selection.
4.3. Number of migrations
Fig. 7(a) and (b) plot the average and maximum
number of migrations of 50 clients, respectively,
with different types of ZIZO algorithms. Each
point in the figures is obtained by averaging over
the values resulted from 50 simulation runs. Recall
that these migrations do not happen in the data

6 10
9
5
# of migrations per client

8
max # of migrations

7
4
6
3 5
4
2 ZIZO-Full-Un ZIZO-Full-Un
ZIZO-Part-Un (k=10) 3 ZIZO-Part-Un (k=10)
ZIZO-Tree-Un 2 ZIZO-Tree-Un
1 ZIZO-Full-Co ZIZO-Full-Co
ZIZO-Part-Co (k=10) 1 ZIZO-Part-Co (k=10)
ZIZO-Tree-Co ZIZO-Tree-Co
0 0
25 30 35 40 45 50 25 30 35 40 45 50
a # of clients b # of clients

Fig. 7. The number of server migrations of clients: server capacity unbounded, f = 0.3. (a) Average number of migrations per client
and (b) maximum number of migrations.
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
session domain but in the control domain. There-
fore, the average number of migrations indicates
the computation overhead and the message
exchange overhead of each algorithm rather than
the actual session migration overhead. On the
other hand, values in Fig. 7(b) (the number of ser-
ver migrations of the client that visited the most
servers) provide a rough indication of the time
required to create a session since a game session
can be started after all clients have finished the
migration steps and selected their servers to
connect.
In the figure, while the number of required
migration steps is slightly larger when the delay
bound is smaller, it is somewhat insensitive to
the difference in the synchronization delay. This
is because, during the zoom-in procedure, all cli-
ents first attempt to migrate up to the core servers
(regardless of the sync-delay bound), and relatively
few client migrations occur in the zoom-out
procedure.
When we compare different migration/searching
strategies, in general, the coordinated client migra-
tion methods incur less migration steps than the
uncoordinated ones, and the tree-based search
scheme results in less number of migrations than
the search-based ones. This result is expected be-
cause in the tree-based search algorithms at each
client there are only a few servers that the client
can choose from. On the other hand, we observe
that the non-tree search algorithms do not incur
an excessive number of migrations either. The
difference between the two is only a constant fac-
tor and does not change much due to the simula-
tion parameter. In particular, the coordinated
migration with partial search in moderate number
of candidate servers (e.g., ZIZO-Part-Co(k = 10))
appears to strike a good balance between the
complexity and the performance observed in
Fig. 5.
4.4. Impact of zoom-out
In Fig. 8, we quantify the additional benefit ob-
tained by the zoom-out procedure, compared to
the case when only zoom-in has been performed.
Here we plot the ratio of the number of servers
after performing the whole zoom-in–zoom-out
97
1.1
# of servers-ZIZO/# of servers-ZI
1
0.9
0.8
ZIZO-Tree-Un
ZIZO-Tree-Co
ZIZO-Part-Un (k=10)
0.7
ZIZO-Part-Co (k=10)
ZIZO-Full-Un
ZIZO-Full-Co
0.6
25 30 35 40 45 50
delay bound (msec)
Fig. 8. Further benefit of Zoom-Out: server capacity
unbounded, f = 0.3.
algorithm to the best case when all clients have mi-
grated up to the core server without zoom-out. We
observe that, when the delay bound is tight
(25–35 ms), 10–20% of unnecessary servers are fur-
ther removed from the selected server set by
zooming-out clients outward from the core server.
This result partially accounts for the sub-par
performance of the greedy algorithms presented
in Fig. 4.
4.5. Adaptivity to incremental client join
In this section, we evaluate the adaptivity of the
proposed algorithm by considering the case when
the clients incrementally join the gaming network.
We handle this incremental growth of game ses-
sion as follows. First, when a new client arrives,
this client connects to the closest server, and
probes the synchronization delay. If the synchroni-
zation delay is larger than the delay bound, the
new client is notified of delay violation and re-
jected. Otherwise, it is accepted and the client con-
nects to the closest server. We then perform our
proposed algorithm periodically after every p suc-
cessful joins to the session in an effort to reduce the
number of session servers. With this approach, the
number of servers would temporarily increase as
more clients join until the server selection algo-
rithm is performed in the next round. Thus we ex-
pect a tradeoff between the frequency of running
the algorithm (i.e., how many times the clients
migrate servers overall) and the level of keeping
the temporary growth of the set of session servers.
98 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102

25 25
ZIZO-Full-Un
ZIZO-Part-Un (k=10)
ZIZO-Tree-Un
20 20 ZIZO-Full-Co
# of session servers

# of session servers
ZIZO-Part-Co (k=10)
ZIZO-Tree-Co
15 15

10 10
ZIZO-Full-Un
ZIZO-Part-Un (k=10)
5 ZIZO-Tree-Un 5
ZIZO-Full-Co
ZIZO-Part-Co (k=10)
ZIZO-Tree-Co
0 0
10 20 30 40 50 10 20 30 40 50
a # of clients b # of clients

Fig. 9. Number of servers selected when clients join incrementally: p = 10, unbounded server capacity, f = 0.3. (a) Delay
bound = 25 ms and (b) delay bound = 35 ms.

4 4
# of migrations per client

# of migrations per client

3 3

2 2
ZIZO-Full-Un ZIZO-Full-Un
ZIZO-Part-Un (k=10) ZIZO-Part-Un (k=10)
1 ZIZO-Tree-Un 1 ZIZO-Tree-Un
ZIZO-Full-Co ZIZO-Full-Co
ZIZO-Part-Co (k=10) ZIZO-Part-Co (k=10)
ZIZO-Tree-Co ZIZO-Tree-Co
0 0
10 20 30 40 50 10 20 30 40 50
a # of clients b # of clients

Fig. 10. Number of migrations per client when clients join incrementally: p = 10, unbounded server capacity, f = 0.3. (a) Delay
bound = 25 ms and (b) delay bound = 35 ms.

In this section, we show the results when the recon- that the algorithm is performed. When we com-
figuration period p is 10.7 pare these results to those in Fig. 5, which present
In Fig. 9, we show the number of session servers the case where all the clients are given initially, we
allocated as new clients join the game session, can see that running the algorithm ‘‘on-line’’ pro-
when the delay bound is relatively tight (Fig. duces as efficient results as the running it ‘‘off-line’’
9(a)) and moderate (Fig. 9(b)). The x-axis of the since the numbers of servers allocated when the
figures represents the number of clients that have last client finally has joined is close to the corre-
joined thus far. We see that, as more clients join sponding values in Fig. 5.
the game, the number of selected servers temporar- Fig. 10 plots the average number of migration
ily grows, and then drops significantly at the point steps performed by clients. Here we observe that
the number of migrations per client tend to de-
crease as the game session grows. In particular,
7
An alternative approach to periodic server selection would when the delay bound is moderate (Fig. 10(b)),
be to have a threshold in the number of servers and to trigger the overhead of the session change monotonically
the ZIZO process whenever the size (or the increase in size) of decreases as more clients join. This is because,
the session server set exceeds the threshold. The net effect of the
periodic method and the threshold-based method is similar.
when a game has a loose delay bound, the server
Thus we only present the periodic recomputation scenario in selection by the existing clients tends to form a
this paper. cluster near the center of the network as more
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
clients join, and therefore, there is not much need
for the clients to move their servers to find a better
selection.
We conclude from these results that, when the
game session grows incrementally, our algorithm
is effective and scalable in terms of the server selec-
tion result and the overhead for session reconfigu-
ration and convergence.
5. Related work
In the context of real-time group communica-
tion, our problem shares some similarity with the
delay-constrained multicast tree construction
problem [24,25]. The goal of the delay-constrained
multicast tree construction problem is to construct
a lowest-cost multicast tree with an additional con-
straint on the upper bound of end-to-end delay be-
tween the sender and the receivers. Our work
differs in that the delay present in server–client
gaming networks is not just an end-to-end delay
but a nonlinear combination of client-to-server
and server-to-server delay as given in our delay
model.
Our work is related to the dual of the minimum
K-center problem [26], in which the number of
servers is to be minimized when the maximum dis-
tance between clients and their nearest servers is
given. In the server selection problem we have con-
sidered we must take the inter-server delay into ac-
count as well. The (log(N) + 1)-approximation
bound is known for this problem, and we are cur-
rently investigating the possibility of finding a for-
mal approximation bound for our problem.
Our server selection algorithm has been devel-
oped for a mirrored server architecture. Recently,
the mirrored server architecture has been proposed
for distributed gaming [7–9] in order to overcome
the limited scalability of the classic client–server
architecture. The mirrored server architecture pro-
vides many benefits of peer-to-peer architecture
without burdening the clients with game simula-
tion and administrative functions. To find the clos-
est server among the mirrors, a client can use ping
or Internet distance service to find servers in its
proximity. In the mirrored server architecture,
the servers connected via a well provisioned pri-
99
vate network to ensure low latency between them.
And since the server network is privately owned,
advanced features such as IP multicast can be
leveraged to facilitate efficient event delivery.
In large scale games, the virtual game world is
divided into multiple regions or zones [12–14]
and only the users in the same zone directly inter-
act with each other. Therefore, it is important to
logically group the users in the same zone together
and process the events and actions between them.
To support clustering of clients and communica-
tion between them one can leverage pub-sub tech-
nologies and distributed hashing. In [12], the
authors proposed to use a distributed hash table
(DHT) for rendezvous to find a server in a mir-
rored server architecture. In [13], the authors con-
struct a Voronoi diagram based on the coordinates
of each node in the virtual world and maintain
peer-to-peer connection between the nodes to han-
dle the dynamics of the game play. In [14], the
authors proposed a range query mechanism
that can support multiple attributes and explicit
load balancing. Such algorithms can be used for
communications in a game session once the server
allocation has been resolved by using our
algorithm.
For efficiency, in the mirrored server architec-
ture, each game server simulates the game state
independently based on the game events it receives.
Since there is a delay in event delivery in distrib-
uted game environments, the game state at each
server must be synchronized with each other. For
this many synchronization algorithms have been
proposed. One of the simplest approaches is the
lockstep synchronization. In the lockstep synchro-
nization, as the name suggests, all the events in the
system must be acknowledged by all members and
processed in lockstep. Since events are processed in
sequence there is no inconsistency in the game
state. However, since it cannot guarantee that
the game will play at the wall clock rate, the lock-
step synchronization is not applicable for online
games. Another simple mechanism, called the
bucket synchronization [16], delays the game sim-
ulation for a fixed time period, or bucket, to pre-
vent misordering of events due to network delay.
Bucket synchronization is widely used in
distributed games for its simplicity. However, its
100 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
performance and user experience is highly sensitive
to the bucket size.
Optimistic synchronization algorithms try to re-
duce the synchronization delay by simulating
game play with minimal delay and rolling back
the game state when inconsistency has been de-
tected. Time warp synchronization and trailing
state synchronization (TSS) [7] are examples of
the optimistic algorithm. In time warp synchroni-
zation, a snapshot of the game state is taken before
execution of each event. When a late event arrives
which requires re-computation, the game state gets
rolled back to one of the previous snapshots and
game simulation gets re-executed with the new
events. TSS maintains multiple game states with
different synchronization delays instead of keeping
snapshots. Each state processes the events after its
own delay and the ‘‘leading’’ state (the state with
the smallest delay) rolls back to one of the
‘‘trailing’’ states if an inconsistency has been dis-
covered. Since it does not need to maintain many
snapshots it is more efficient in terms of space
complexity.
Another interesting issue in distributed gaming
environment is to simulate the game play in a fair
manner. This problem arises from the fact that dif-
ferent clients may experience different latency to
the server, and the game players who are closer
to the server can react quickly before other players
who are connected over delayed links, thus have
unfair advantage. To address this issue, Guo
et al. have proposed a proxy-based algorithm to
ensure so-called fair order service [27]. The main
concept of this algorithm is to have a client side
proxy, which marks the reaction time of the user,
and forwards this information to the server. When
the server receives messages, it processes them con-
sidering both the reaction time and the arrival time
of the messages. The server selection algorithm
proposed in this paper does not depend on a par-
ticular synchronization technique. We just assume
that all servers must receive data from all clients
for computing a new game state, which is a neces-
sary condition for most synchronization models.
Shaikh et al. [10] and Saha et al. [11] have
proposed an on-line games hosting platform by
developing middleware based on existing grid
components. The proposed gaming architecture
is based on a utility computing model to support
multiple games by leveraging off-the-shelf soft-
ware components that were originally developed
for business applications. For example, they use
IBM Tivoli Intelligent Orchestrator (TIO) [28] as
the provisioning module for game servers. In addi-
tion to performing addition and removal of servers
according to the workload dynamics, the provi-
sioning module performs administrative tasks such
as (a) new game deployment, (b) directory man-
agement of games and players, (c) player redirec-
tion to server, and (d) game content distribution.
In the commercial world, Butterfly.net [3] is imple-
menting the idea of building a scalable gaming
infrastructure based on grid computing technol-
ogy. However, both these works focus on architec-
tural issues, leaving such issues as server allocation
unaddressed.
6. Conclusion
In this paper, we present a novel distributed
algorithm that dynamically selects game servers
for a group of game clients participating in large
scale interactive online games. The goal of server
selection is to minimize server resource usage while
satisfying the real-time delay constraint. The pro-
posed algorithm, called zoom-in–zoom-out, is
adaptive to session dynamics and lets the clients
select appropriate servers in a distributed manner
such that the number of servers used by the game
session is significantly reduced compared to the
case when clients select the closest servers. We
have considered various zoom-in techniques that
the clients can implement; namely (i) tree-based
search, (ii) partial server search, and (iii) full server
search. We have also considered two different
types of client migration; (i) coordinated migra-
tion and (ii) uncoordinated migration. Using
simulation, we have shown that the proposed
zoom-in–zoom-out algorithm results in compara-
ble or better performance than the centralized
greedy algorithms, which are based on global
information and expensive computation. We also
show that, in practice, a partial server search
algorithm with uncoordinated client migration
strikes a good balance between the performance
 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
and complexity. In particular, we have shown that
this scheme can find the optimal solution in many
cases and performs closely to the full search algo-
rithm. Furthermore, we have shown that the
zoom-out process is essential for reducing the
number of servers especially when the delay bound
is tight. In the scenarios we have considered, the
zoom-out process can save up to 20% of server re-
sources. Finally, we have shown that our proposed
algorithm is not only effective for a static configu-
ration but also is adaptive to the dynamics in the
client set.
References
[1] Lineage. Available from: <http://www.lineage.net/>.
[2] NCsoft. Available from: <http://www.ncsoft.net/kor/
ncnewsdata/View.asp>.
[3] Butterfly.net. Available from: <http://www.butterfly.net/>.
[4] Terazona. Available from: <http://www.zona.net/>.
[5] PlanetSide. Available from: <http://www.planetside.
com/>.
[6] J.D. Pellegrino, C. Dovrolis, Bandwidth requirement and
state consistency in three multiplayer game architectures,
in: Proceedings of NetGames, 2003.
[7] E. Cronin, B. Filstrup, A. Kurc, A Distributed Multiplayer
Game Server System, Technical Report, University of
Michigan, May 2001.
[8] D. Bauer, S. Rooney, P. Scotton, Network infrastructure
for massively distributed games, in: Proceedings of Net-
Games, 2002.
[9] M. Mauve, S. Fischer, J. Widmer, A generic proxy system
for networked computer games, in: Proceedings of Net-
Games, 2002.
[10] A. Shaikh, S. Sahu, M. Rosu, M. Shea, D. Saha,
Implementation of a service platform for online games,
in: Proceedings of NetGames, 2004.
[11] D. Saha, S. Sahu, A. Shaikh, A service platform for
OnLine games, in: Proceedings of NetGames, 2003.
[12] T. Iimura, H. Hazeyama, Y. Kadobayashi, Zoned feder-
ation of game servers: a peer-to-peer approach to scalable
multi-player online games, in: Proceedings of NetGame,
2004.
[13] S.-Y. Hu, G.-M. Liao, Scalable peer-to-peer networked
virtual environment, in: Proceedings of NetGame,
2004.
[14] A.R. Bharambe, M. Agrawal, S. Seshan, Mercury: sup-
porting scalable multi-attribute range queries, in: Proceed-
ings of ACM SIGCOMM, 2004.
[15] L. Gautier, C. Diot, J. Kurose, End-to-end transmission
control mechanisms for multiparty interactive applica-
tions on the Internet, in: Proceedings of IEEE Infocom,
1999.
101
[16] C. Diot, L. Gautier, A distributed architecture for multi-
player interactive applications on the Internet, IEEE
Networks Magazine 13 (4) (1999) 6–15.
[17] Y.-J. Lin, S.P.K. Guo, Sync-MS: synchronized messaging
service for real-time multi-player distributed games, in:
Proceedings of IEEE International Conference on Net-
work Protocols (ICNP), 2002.
[18] P. Bettner, M. Terrano, 1500 Archers on a 28.8: network
programming in age of empires and beyond, in: Game
Developers Conference, 2001.
[19] D.G. Thaler, C.V. Ravishankar, Distributed center-loca-
tion algorithms, IEEE Journal on Selected Areas in
Communications 15 (3) (1997) 291–303.
[20] BRITE. Available from: <http://www.cs.bu.edu/brite/>.
[21] A.L. Barabási, R. Albert, Emergence of scaling in random
networks, Science (1999) 509–512.
[22] L. Qiu, V. Padmanabham, G. Voelker, On the placement
of web server replicas, in: Proceedings of 20th IEEE
INFOCOM, 2001.
[23] T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein,
Introduction to algorithms, second ed., MIT Press, 2001.
[24] M. Parsa, Q. Zhu, J.J. Garcia-Luna-Aceves, An iterative
algorithm for delay-constrained minimum-cost multicast-
ing, IEEE/ACM Transactions on Networking 6 (4) (1998)
461–474.
[25] H.F. Salama, D.S. Reeves, Y. Viniotis, An Efficient Delay-
Constrained Minimum Spanning Tree Heuristic, TR-96/46,
Technical Report, North Carolina State University, 1996.
[26] J. Bar-Ilan, D. Peleg, Approximation algorithms for
selecting network centers, in: Proceedings of 2nd Work-
shop on Algorithms and Data Structures, Lecture Notes in
Computer Science, vol. 519, 1991, pp. 343–354.
[27] K. Guo, S. Mukerjee, S. Rangrajan, S. Paul, A fair
message exchange framework for distributed multi-player
games, in: Proceedings of NetGames, 2003.
[28] IBM, IBM Tivoli Intelligent Orchestrator. Available
from: <http://www-306.ibm.com/software/tivoli/products/
intell-orch/>.
Dr. Kang-Won Lee was born in Seoul,
Korea. He received a B.S. in 1992 a
M.S. in 1994 from Seoul National
University in computer engineering
and a Ph.D. in 2000 from the Univer-
sity of Illinois at Urbana–Champaign
(UIUC) in Computer Science. He
served the Republic of Korea Air
Force during 1994–1995 and worked as
a research assistant for the TIMELY
research group at the Center for Reli-
able and High Performance Computing, Urbana IL, during
1996–2000. In 2000, he joined IBM T. J. Watson Research
Center, Hawthorne, NY as a research staff member. His
research interest is in computer networks, large scale distributed
systems, and policy-based system management. He was awar-
ded a Magna Cum Laude from Seoul National University in
102 K.-W. Lee et al. / Computer Networks 49 (2005) 84–102
1992, the Korean Government Overseas Scholarship from the
Ministry of Education in 1996, the KFSA Scholarship from the
Korea Foundation for Advanced Studies in 1997, the C.W.
Gear Outstanding Graduate Student Award from the Com-
puter Science Department at UIUC in 1999, and the Best
Student Paper Award from Packet Video Workshop in 2000.
Recently, He received the IBM Research Division Award for
contributing to the architecture of IBM Autonomic Computing
Policy Infrastructure and Products.
Bong-Jun Ko received the B.S. and
M.S. degrees in electrical engineering
from Seoul National University,
Korea, and is currently working
toward a Ph.D. degree in electrical
engineering at Columbia University,
New York. He has worked as a
research engineer at LG Electronics in
Korea before joining Columbia Uni-
versity, and has interned at IBM T. J.
Watson Research Lab, NY. His
research interests include large-scale distributed systems and
mobile and wireless networks. He received the IEEE ICNP Best
Paper Award in 2003.
Dr. Seraphin Calo is a Research Staff
Member at IBM Research and cur-
rently manages the Policy Technologies
group within that organization. He
received the M.S., M.A., and Ph.D.
degrees in electrical engineering from
Princeton University, Princeton, New
Jersey. He has worked, published, and
managed research projects in a number
of technical areas, including: queueing
theory, data communications net-
works, multi-access protocols, expert systems, and complex
systems management. He has been very active in international
conferences, particularly in the systems management and policy
areas. His recent involvements include serving on the Orga-
nizing Committee of Policy 2004 (IEEE 5th International
Workshop on Policies for Distributed Systems and Networks)
and serving as the General Chair of IM 2005 (The Ninth IFIP/
IEEE International Symposium on Integrated Network Man-
agement). He has authored more than 50 technical papers and
has several United States patents (three issued and four pend-
ing). He has received two IBM Research Division awards, and
two IBM Invention Achievement awards. His current research
interests include: distributed applications, services management,
and policy based computing.