A Multicast RPC Implementation for Java

Philip Russell and Chase Covello

Computer Science Department
University of California, Los Angeles
{prussell,chase}@cs.ucla.edu
http://code.google.com/p/multicast-rpc/

than worry about the explicit memory

Abstract
We propose an extension of Remote
Procedure Call (RPC) semantics to a
multicast environment and discuss a
Java implementation thereof. The
implementation consists of a reliable,
FIFO multicast provider, five
interchangeable Total Order Broadcast
algorithms, and an RPC client and server
that make use of these technologies. We
demonstrate that the implementation
works correctly. Throughout the paper,
we discuss future work that could be
done on this project.
Introduction
Remote Procedure Calls (RPC) [1] are a
convenient paradigm to use when
programming a distributed system. A
well-designed RPC implementation like
Java’s RMI is transparent to the
programmer – that is, remote procedure
calls are made with identical syntax to
local procedure calls [4]. This brings the
ease of programming of shared memory
multiprocessing systems to inexpensive
multicomputer clusters built with
commercial off-the-shelf components.
Instead of explicitly passing messages
between processes, the programmer
simply sets the RPC mechanism up and
performs method calls on objects.
Threads can thus be moved to other
computers without affecting method
invocation semantics. This allows the
programmer to focus on the higher level
task of writing a correct program rather
management required in distributed
memory multiprocessors.
Point-to-point RPC between a client and
a server system is mostly a solved
problem [1], but RPC in a multicast
environment is still an active field of
research. RPC has been criticized for
being inherently point-to-point [3], but
we believe that the ability of a client
system to execute an RPC call on many
servers in parallel would be useful in
many situations. For example, a master
processor could send software updates or
remote administration commands to a
network of mobile devices with little
effort on the part of the programmer.
Thinking bigger, a network of mobile
devices could be treated as a “grid,”
bringing massively parallel computing
power to any client within wireless
range. Such a network could be scaled
up or down without requiring any
changes to client programs of the grid.
This paper demonstrates an
implementation of multicast RPC in
Java. Because it is written entirely in
software without any modification to the
language or compiler, our
implementation does not have the
transparency of Java RMI. However, it
would be straightforward to integrate
this code into the Java runtime system as
an extension to RMI, providing more
transparency. This project, then, is a
proof of concept.

1
Multicast Implementation
Correct RPC semantics require both the
sender and receiver to agree on the order
in which procedures are executed.
Multicast RPC is no different – in fact,
all receivers should agree on the same
ordering for procedure calls. This
property, called Total Order Broadcast
[2], is not provided in the Java standard
library. Java’s network library supports
the core transport-layer protocols of the
Internet: TCP, UDP, and multicast UDP.
Since multicast UDP does not provide
any guarantee of reliable or in-order
delivery, it is not a total order broadcast
algorithm, and is thus unsuitable for use
with RPC. Furthermore, we were unable
to correctly route multicast packets on
any of the networks we tested. Given the
limited amount of time available for this
project, we chose to implement a total
order broadcast protocol in software atop
TCP connections. This is not truly a
multicast protocol, and in fact it is
inefficient for production-quality
multicast RPC, but we believe that the
modular nature of the code lends itself to
being replaced with a better
implementation in the future, perhaps
taking advantage of advanced multicast
services in the operating system or
network hardware.
Our total order broadcast
implementation consists of the following
modules:
• The multicast simulator. This is a
reliable, FIFO multicast transport
layer. It has a set of message sender
and receiver classes, along with a
multicast manager that tracks
membership in the multicast group
and relays the information to group
members.
• The total order broadcast
modules. These use the multicast
simulator for their transport layer
and add a total ordering guarantee to
message transport. They are based on
the pseudocode in [2]. Five
implementations are available, which
allows the RPC user to choose
among the performance and
reliability tradeoffs inherent in the
algorithms.
Note that all the packages mentioned are
relative to edu.ucla.cs.rpc.multicast.
Multicast Simulator
Our multicast code is located in the
network package. Messages are sent
using instances of multicast sender,
found in MulticastMessage-
Sender.java. A message sender
maintains a set of message receiver
addresses. It sends a message by opening
a TCP connection to each receiver in the
set, serializing the message, and sending
the serialized byte stream over the
connection. The set of receiver addresses
is kept up to date with the help of a
separate thread that listens on a socket
for broadcast updates.
Unicast messages are received with
instances of MessageReceiver.java.
This class runs a thread that listens on a
socket for incoming messages. The
messages are passed to the receiver’s
message handler, a type of extensible
event handler that the user provides. A
handler might print the message to the
console, queue it, or hand it off to a
waiting processing thread, for example.
Multicast messages arrive at instances of
MulticastMessageReceiver.java. This
class extends MessageReceiver.java to
maintain membership in the multicast
group, including sets of the sender and
receiver unique identifiers. Updates to
2
these sets are accomplished in the same
way as for multicast senders.
A core concept in our multicast
simulator is that of the multicast
manager. It is implemented in
MulticastManager.java. The multicast
manager listens on a TCP socket for Join
and Leave messages from the senders
and receivers. Upon receiving such a
message, it updates its internal
membership sets and sends updates to
members of the group. Multicast senders
need to know the addresses of the
group’s receivers, so those are broadcast
to the senders whenever a receiver joins
or leaves the group. Similarly, sender
and receiver membership changes are
broadcast to all receivers in the group.
The combination of these four classes,
along with Message objects, provides a
reliable, FIFO multicast transport layer.
Any single sender will observe a
consistent ordering of its own messages
among all the receivers in the group. For
total ordering of all messages, however,
one of the five total order broadcast
algorithms is needed.
Fixed Sequencer
The simplest total ordering algorithm
uses a single process through which all
broadcast messages must be routed. This
process, called a sequencer [2], receives
one message at a time, so a total order is
imposed automatically. The sequencer
adds a sequence number to each
incoming message, then broadcasts them
to all receivers in the multicast group.
Our fixed sequencer resides in the
sequencer.fixed package, in Fixed-
Sequencer.java. It uses a message
receiver to accept incoming messages
and broadcasts them with a multicast
message sender. We have a test program
in test.FixedTester.java.
Moving Sequencer
The Fixed Sequencer scheme relies on a
single, fixed intermediary process to act
as a relay between the group of senders
(S) and the group of destinations (D).
The main advantage of FS is that it is
relatively simple, but suffers from a
single point of failure. If the FS process
ever crashes, then the entire system stops
processing messages. No message can be
sent from S, and no member of D can
receive a message. Instead of a single
intermediary process, we can use a
group of processes to relay messages.
This is the Moving Sequencer scheme. It
distributes both the responsibility of
relaying messages between S and D and
balances the load of sending messages
amongst its members.
As outlined in [2], the MS
implementation relies on a token ring. A
Token is an object that records a
sequence number and keeps a list of
messages that have been relayed
between S and D. The algorithm
proceeds thus:
1. Each Sender in S broadcasts to every
member of MS.
2. Each sequencer waits for both a
message and the token. As soon as it
holds both, of these objects, it
attempts to add the message to the
list of already relayed messages. If
this operation fails, then the
sequencer knows that the message
has already been relayed, and should
be dropped. Otherwise, it stamps the
message with a sequence number
and sends the message to D.
The token ring is implemented in the
util.token package. The primary files
are Token.java and NetworkToken-
Manager.java. A token manager is an
object that is responsible for both
3
initializing the token, receiving it from
the previous node, and sending it to the
subsequent node in the ring when either
a timeout occurs or it has been signaled
to do so by another process.
The Moving Sequencer algorithm is
implemented in the sequencer.moving
package, in the file Moving-
Sequencer.java. This class is tested by
the file MovingTest.java, found in test.
The current implementation successfully
constructs the token ring and relays only
one copy of a message between S and D.
There are no provisions for dealing with
the possibility of a member of the ring
failing. The current implementation can
be extended towards this functionality
by maintaining an on line registry of
token ring members. In the case of
member failure, each member of the
token ring is notified by the registry,
which then calculates a new token ring
by randomly assigning new previous and
adjacent nodes to each member.
Privilege-Based
The Privilege based scheme, like
Moving Sequencer, is based on creating
a token ring of processes. Instead of
sequencers forming the ring, the
members of S do so. The algorithm
given in [2] is:
1. For each members in S, wait for both
a message from the upper layer and
the token.
2. When both of these objects are held,
assign the message a sequence
number and multi-cast it to D.
PB is a relatively simple scheme that
distributes the load of sending messages
from the upper level amongst the
members of S. The token ring is
implemented in the same files mentioned
in Moving Sequencer. The algorithm
implementation is in Privileged-
Sender.java, package privilege. It is
tested in PrivilegeTest.java, found in
the testing package. The purpose of the
test is to create the token ring amongst S,
and to send messages from S to a
destination.
Communications History
In contrast to the Fixed/Moving
Sequencer and Privilege-based scheme,
the Communications History class of
algorithms do not differentiate between
S and D. Instead, all processes in the
system belong to the same group P. Of
the class of Communications History
algorithms, we implemented the causal
history algorithm, which proceeds as
follows [2]:
1. Every member p of P has an array of
logical clocks that record that
maximum clock value of process q
of P, as known by p. Initially, all
value are zero. An update occurs
when p receives a message from q
with a new clock value.
2. To broadcast, p update its own
logical clock, time-stamps the
message, and sends it across some
FIFO channel (a TCP stream in our
implementation).
3. When a process p receives a message
from q, p updates its array of logical
clocks to be the maximum of the
current logical clock value for q and
the value embedded in the message.
p then creates a set of deliverable
messages, defined as those that have
been received but have not been
delivered and the timestamp of each
message is less than the minimum
clock value for each process in the
logical clock array.
4. Each deliverable message is
delivered to the layer above, and
4
 added to the delivered set.
The algorithm imposes an order on the
messages by restricting the conditions
under which a message can be delivered
to the layer above. As described above
and detailed in [2], a message cannot be
added to the deliverable set unless the
process p has received a message from
each other process q with a later
timestamp than the message's. A system-
wide total order is thus imposed by
forcing logically earlier messages to be
delivered first.
The algorithm is implemented in the file
CausalProcess.java, in package
commhistory.
Destinations Agreement
The class of destinations agreement
algorithms work on the principal that the
Senders broadcast to the Destinations,
who re-circulate enhanced messages
amongst themselves to decide on the
total order of message delivery.
Our implementation focuses on
destination agreements with agreed-upon
sequence numbers [2]. The algorithm
proceeds as follows:
1. Each sender in S broadcasts the
message from the layer above in the
usual manner.
2. Each destination D records two
message sets, stamped and received.
As well, it initializes its logical clock
to be zero.
3. Upon receiving a non-timestamped
message, the destination adds it
received, timestamps the message,
and re-transmits the message to the
other destinations.
4. Upon receiving a timestamped
message, the checks to see if it has a
copy of the message from every
other destination. If not, it records
the sender id, keyed by the message.
If yes, then it finds the globally
maximum timestamp of the message,
stamps a copy of the message with
the global timestamp, and adds it to
stamped.
5. For each message in stamped, the
algorithm checks to see if every
message in received has a logically
later timestamp. If this is true, then
the stamped message is delivered to
the layer above.
The algorithm enforces a total order on
the message delivery by finding the
logically latest copy of each message,
and ensuring that it cannot be delivered
unless all other received messages are
logically later [2]. Our implementation is
found in the file AgreementDest.java,
package destination.agreement.
The algorithm assumes that there is a
way to discover only the set D of
destinations. We were not able to
implement this functionality in the given
time constraints. Possible extensions to
this project would be to use a different
MulticastManager to negotiate the
message channels amongst only D.
The RPC Layer
With a total order broadcast algorithm
available, the actual RPC code becomes
very simple. A program that wishes to
call a remote method constructs a new
RPC client instance, initializing it with a
Sender implementation from one of the
fire total order broadcast algorithms. The
relevant code is in RPCClient.java. A
client then invokes the call method,
which accepts an object or Class
instance, a method name, and a variable
number of arguments. The call method
packages these arguments into an
5
RPCMessage and total order broadcasts
it.
On the server side, an instance of
RPCServer.java receives the message,
unpacks its contents, and performs a
dynamic class and method lookup using
the Java reflection API. It then invokes
the method using the provided
arguments and returns.
Our implementation does not currently
support return values. Any method may
be invoked, but return values are
discarded on the server side. We can
think of two ways to support return
values: either allow multicast messages
to aggregate responses from the
receivers and pass them back to the
sender, or have the RPC client wait for
responses from all the servers. An earlier
version of the project tried the former
solution, but it didn’t really work well
and complicated much of the total order
broadcast code. Further work on this
project might include having the RPC
client wait for response messages from
the servers and aggregating their return
values.
Results
Our multicast RPC implementation
works for our test cases.
RPCClient.java contains a main
method that starts a multicast manager, a
fixed sequencer, a client, and several
servers. It then instantiates an RPCTest
class, which has an rpcTest method that
prints its string argument to the standard
output. The client calls this method with
the string “Hello, world!”, then shuts
down the servers, sequencer, and
multicast manager. Running the program
produces one “Hello, world!” on the
console for each RPC server running.
We are thus confident that we have a
working RPC implementation.
Conclusion
In this paper we discussed the problem
of making remote procedure calls to
multiple destinations as a multicast
operation. We outlined the architecture
to achieve this goal in Java. The two
primary components of the architecture
are the multi-cast simulator and the total
order algorithms. Together, these pieces
create a working implementation of RPC
semantics in a multicast environment. As
discussed throughout the paper, there is
future work to be done, starting with
implementing a true multicast routing
layer that the message senders, receiver
and multicast managers communicate
through. Some of the total order
broadcast modules, such as Moving
Sequencer, can be extended with further
functionality, and entirely new modules
can be created. We envision performing
experiments to obtain performance
evaluation data for the different
algorithms. We are confident that this
work forms the basis for a modular,
transparent testing and implementation
suite for multicast remote procedure
calls.
References
[1] Birrell, Andrew and Nelson, Bruce
Jay. “Implementing Remote
Procedure Calls.” ACM Trans.
Computer Systems, Feb 1984.
[2] Défago, Xavier, Schiper, André, et
al. “Total Order Broadcast and
Multicast Algorithms: Taxonomy
and Survey.” ACM Computing
Surveys, Dec 2004.
[3] Tanenbaum, Andrew S. and van
Renesse, Robbert. “A Critique of the
Remote Procedure Call Paradigm.”
Proc. EUTECO 88, Apr 1988.
[4] Waldo, Jim. “Remote procedure calls
and Java Remote Method
6
Invocation.” IEEE Concurrency, Jul
1999.