Automatic detection of internal queues and stages in message processing systems
Suman Karumuri, Steve Reiss, Brown University. {suman,spr@cs.brown.edu}
Abstract
Complex applications today involve multiple processes,
multiple threads of control, distributed processing, thread
pools, event handling, messages. The behaviors and misbe-
haviors of these nondeterministic, message-based systems
are difficult to capture and understand. The typical ap-
proach is to trace the behavior of the systems and track how
the different incoming messages are processed throughout
the system. While messages between processes can be cap-
tured automatically at the network or library level, tracing
the message processing within a system, which is often more
complex and error-prone, requires the programmer to man-
ually instrument the code by identifying the different mes-
sage handlers, thread states, processing stages, and shared
queues accurately and completely. In this paper we show
how dynamic analysis can be used to automatically iden-
tify the transactions, stages and shared queues in Java pro-
grams as a prelude to trace-based comprehension.
1 Introduction
Today’s complex systems typically handle a series of ex-
ternal requests or messages as a whole or by splitting the
requests into separate computational steps and processing
each of these stages using mechanisms such as thread pools.
The overall behavior of such systems can be nondeterminis-
tic, unpredictable, and difficult to understand. We call such
systems message processing systems (MPS), call the re-
quests transactions, call the various processing steps stages,
and call the mechanism used to save and allocate stages to
the different threads queues. In this paper, we present new
program analysis techniques to detect queues, stages and
transactions in a message processing system using minimal
programmer input, the list of messages in a system.
As an example of a MPS, consider a simple web crawler
we wrote for a class project. The crawler uses a pool of
threads to retrieve a queue of pages. Each thread takes a
page off the queue and processes it in the following order:
check if the URL is a valid, download the HTML page if
the URL is valid and finally add the new URLs found in the
downloaded page to the shared queue for further process-
ing. Here there is a single stage, processing a URL, and
a single queue through which the worker threads exchange
messages containing URLs. Some other examples of MPSs
we have looked at are a peer to peer system, the Haboob
HTTP server, the Hadoop Map-Reduce framework, a back
end for code search, a linguistic search web service, the en-
terprise messaging system ActiveMQ, the Rupy and Jetty
HTTP servers and Jabber tools JBother and Openfire.
The best way to understand the behavior of a MPS is by
analyzing a trace of its behavior, a trace that can be both pre-
cise and that can be generated with relatively low overhead.
However, such traces are difficult to obtain and thus are used
only as a last resort. The main problem here is that procur-
ing such a trace requires significant work on the part of the
programmer since the code to generate the trace needs to be
inserted manually in many portions of the system. Here it
is easy to overlook a queue or processing stage and to insert
bugs into the system along with the instrumentation code.
The manual instrumentation of a message processing sys-
tem is carried out in two steps, identifying the transactions,
stages and queues in the MPS, and manually adding trace
code in the form of instrumentation based on these identi-
fications. While frameworks like XTrace[6] ease the latter
step, no tools currently exist to automate the former step.
Prior work in the area of automatic trace generation of
message processing systems has concentrated on the exter-
nal messages rather than the internals of the systems. These
systems capture and interpret the data at well defined in-
terfaces like network sockets (Causeway[2]), Unix system
calls (BorderPatrol[4]), the libevent entries (WhoDunIt[3])
or J2EE calls (Pinpoint [5]). Since these interfaces are stan-
dard and well documented, the queues and stages in those
systems can be readily identified. These techniques do not
work on the various internal queues and threading mecha-
nisms that most complex MPSs utilize.
We are developing a system that attempts to completely
automate the process of tracing message processing systems
to facilitate understanding their behavior. In this paper we
describe our approach to the problem of automatically de-
termining what instrumentation is needed to generate ap-
propriate traces. Our approach uses minimal programmer
input and a combination of static and dynamic analysis to
obtain the appropriate information.
2 Identifying internal queues and stages
Our first step in automating the instrumentation process
was to understand the common techniques used in MPS. By
understanding what the programmer would need to know
to instrument the wide range of message processing sys-
tems (described in previous section), we looked at and de-
termined we could assume the following traits: (a) All mes-
sages in the system can be identified by a small set of Java
types. (b) Each message in the system can be uniquely iden-
tified. (c) The shared queues are implemented using Java
collections containing objects of type messages or by a field
(queue of size 1) of type message. (d) Each stage waits for
the next event that it can process from the queue. Once it
finds an event, it removes the event from the queue, pro-
cesses it and waits for another event. (e) The start stage of
a transaction always creates a message for the next stage to
process, without retrieving a new message. The last stage of
every transaction retrieves a message from a bin but doesn’t
insert any messages into the shared bin for further process-
ing. (f) Every stage, except the starting stage of a trans-
action, extracts a message from the shared queue and pro-
cesses it. While processing the message, the stage may up-
date the same message or may create a new message based
on existing message, that is inserted into the shared queue
for further processing.
While most message processing systems will satisfy
these assumptions, it should be relatively easy to augment
non-conforming systems either manually or automatically.
For example, if a system lacks a unique message identity,
either a new field could be inserted in the message objects
or the object hash code could be used as an approximate
identity for each message.
Our system requires two simple inputs from the pro-
grammer. The first is the name of the Java class or classes
that represent messages in the system. The second is a sam-
ple run of the program from which we can obtain dynamic
information. Using these, our system identifies the shared
bins as those fields that contain a message or a Java collec-
tion of messages and that have corresponding get-set oper-
ations (implicit or explicit) that are performed by different
threads ignoring initializations. Initializations typically in-
volve fields local to the worker threads that are initialized
once from the main thread. To identify and thus ignore such
fields, we only identify those fields as queues when they are
used to exchange more messages than the maximum num-
ber of threads in the JVM at any point of time. To imple-
ment this analysis, we used AspectJ to trace all get and set
calls on the fields that contain message objects and all add
and remove operations on collections 1 that hold the mes-
1 Since the interface to add and remove elements is not the same for all
the collections, we have segregated relevant Java Collection API calls into
add or remove calls. If a collection contains Java Objects (unspecialized
sage objects. The shared queues are identified by a python
program that walks the trace to match the corresponding
set/add and get/remove calls from different threads using
message IDs.
Our definition of a stage matches the definition of an
event handler by Reiss [7]. We thus use their dynamic event
handler detection system to identify various stages in the
system. All stages that created a message without retriev-
ing a message were identified as a start stage and all stages
that retrieved a message but didn’t insert new messages for
further processing were identified as end stages of the trans-
action. All other stages are treated as intermediate stages. It
should be noted that the same stage may act as an end stage
in one transaction and as an intermediate stage in another
because of conditional branching.
3 Results and Conclusions
To evaluate the effectiveness of our analysis, we used our
prototype system to automate automatically identify queues
and event handlers in our web crawler and in Rupy[1], an
open source HTTP server. The results of the analysis were
compared against the results obtained through manual code
inspection and were found to be identical. The analysis was
also part of an automatic causal graph generator (under con-
struction), using which we were able to generate a simple
causal graph of the execution of the crawler, another indica-
tion that the analysis works.
In this paper, we have presented new program analysis
techniques to automatically detect transactions, stages and
queues in a MPS. We have also shown that these techniques
are accurate by evaluating them on real world systems. We
are currently working on using these techniques along with
others to build a complete system for understanding mes-
sage processing applications based on automated trace gen-
eration and analysis.
References
[1] http://code.google.com/p/rupy/.
[2] A. e. a. Chanda. Causeway: operating system support for con-
trolling and analyzing the execution of distributed programs.
In HOTOS’05, pages 18–18.
[3] A. C. et al. Whodunit: transactional profiling for multi-tier
applications. In EuroSys ’07, pages 17–30.
[4] E. K. et al. Borderpatrol: isolating events for black-box trac-
ing. In Eurosys ’08, pages 191–203.
[5] M. Y. C. et al. Pinpoint: Problem determination in large, dy-
namic internet services. In DNS’02, pages 595–604.
[6] R. F. et al. X-trace: A pervasive network tracing framework.
In NSDI’07.
[7] S. P. Reiss. Dynamic detection of event handlers. In Proceed-
ings of WODA ’08, pages 1–7.
collection) we only trace operations on objects of type messages.